2019-10-13T03:36:58.984 INFO:root:teuthology version: 1.0.0-471e0e3 2019-10-13T03:36:58.985 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.U1IGBI.tmp -- --verbose --lock --description fs/traceless/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml traceless/50pc.yaml} --name pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711 2019-10-13T03:36:59.025 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T03:36:59.081 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711 branch: wip-pdonnell-testing-20191012.025926 description: fs/traceless/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml traceless/50pc.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4407711' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-pdonnell-testing-20191012.025926 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds inject traceless reply probability: 0.5 mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191012.025926 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd suite: fs suite_branch: wip-pdonnell-testing-20191012.025926 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - check-counter: counters: mds: - mds.dir_split - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.4306 2019-10-13T03:36:59.081 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa; will attempt to use it 2019-10-13T03:36:59.081 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks 2019-10-13T03:36:59.082 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-13T03:36:59.083 INFO:teuthology.task.internal:Checking packages... 2019-10-13T03:36:59.098 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd' 2019-10-13T03:36:59.098 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-13T03:36:59.098 INFO:teuthology.packaging:ref: None 2019-10-13T03:36:59.099 INFO:teuthology.packaging:tag: None 2019-10-13T03:36:59.099 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191012.025926 2019-10-13T03:36:59.099 INFO:teuthology.packaging:sha1: a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T03:36:59.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-pdonnell-testing-20191012.025926 2019-10-13T03:36:59.268 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T03:36:59.269 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-13T03:36:59.278 INFO:teuthology.task.internal:no buildpackages task found 2019-10-13T03:36:59.278 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-13T03:36:59.288 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-13T03:36:59.288 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T03:36:59.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2019-10-13T03:37:09.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2019-10-13T03:37:19.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2019-10-13T03:37:29.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2019-10-13T03:37:39.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2019-10-13T03:37:49.437 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:37:59.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:09.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:19.481 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:29.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:39.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:49.537 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:38:59.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:09.569 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:19.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:29.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:39.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:49.637 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:39:59.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:09.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:19.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:29.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:39.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:49.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:40:59.761 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:09.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:19.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:29.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:39.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:49.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:41:59.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:09.861 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:19.878 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:29.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:39.911 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:49.923 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:42:59.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:43:09.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:43:19.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:43:29.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:43:40.022 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:43:50.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:44:00.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-13T03:44:10.064 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:44:20.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:44:30.095 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:44:40.113 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:44:50.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:00.149 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:10.158 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:20.175 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:30.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:40.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:45:50.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:00.240 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:10.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:20.272 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:30.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:40.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:46:50.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:00.345 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:10.360 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:20.373 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:30.382 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:40.397 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:47:50.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:48:00.429 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-13T03:48:10.448 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_pdonnell@teuthology', 'description': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711', 'machine_type': 'smithi'} 2019-10-13T03:48:10.463 DEBUG:teuthology.lock.ops:locked ubuntu@smithi110.front.sepia.ceph.com, ubuntu@smithi204.front.sepia.ceph.com 2019-10-13T03:48:10.609 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T03:48:10.832 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of ubuntu 18.04 2019-10-13T03:48:10.835 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of ubuntu 18.04 2019-10-13T03:48:11.243 INFO:teuthology.orchestra.console:Power off smithi110 2019-10-13T03:48:11.244 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T03:48:11.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:11.352 INFO:teuthology.orchestra.console:Power off smithi204 2019-10-13T03:48:11.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-13T03:48:11.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:11.536 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2019-10-13T03:48:11.637 INFO:teuthology.orchestra.console:Power on smithi204 2019-10-13T03:48:11.637 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T03:48:11.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:15.384 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:15.742 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:19.532 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:19.763 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2019-10-13T03:48:19.864 INFO:teuthology.orchestra.console:Power on smithi110 2019-10-13T03:48:19.864 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-13T03:48:19.899 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:19.928 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:20.129 INFO:teuthology.orchestra.console:Power on for smithi204 completed 2019-10-13T03:48:20.232 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish 2019-10-13T03:48:23.976 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-13T03:48:24.208 INFO:teuthology.orchestra.console:Power on for smithi110 completed 2019-10-13T03:48:24.309 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish 2019-10-13T03:51:21.673 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:30.733 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:39.790 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:40.769 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:48.849 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:49.834 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:57.933 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:51:58.890 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:06.990 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:07.950 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:16.046 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:17.002 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:25.097 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:26.058 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:35.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:44.174 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:52:53.233 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:02.509 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:02.519 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:08.529 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:08.539 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:14.541 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:14.551 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:20.561 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:23.615 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:29.619 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:29.704 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:29.704 INFO:teuthology.orchestra.run.smithi204:> hostname 2019-10-13T03:53:30.113 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com 2019-10-13T03:53:30.114 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.114 INFO:teuthology.orchestra.run.smithi204:> grep smithi204.front.sepia.ceph.com /etc/hosts 2019-10-13T03:53:30.161 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204 2019-10-13T03:53:30.162 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.162 INFO:teuthology.orchestra.run.smithi204:> sudo hostname smithi204 2019-10-13T03:53:30.381 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.381 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hosts 2019-10-13T03:53:30.396 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.396 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/172.21.15.204/172.21.15.204/g' /etc/hosts 2019-10-13T03:53:30.476 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.476 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hostname 2019-10-13T03:53:30.556 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:30.557 INFO:teuthology.orchestra.run.smithi204:> sudo hostname smithi204 2019-10-13T03:53:30.634 INFO:teuthology.provision.fog.smithi204:Deploy complete! 2019-10-13T03:53:30.636 DEBUG:teuthology.parallel:result is None 2019-10-13T03:53:30.670 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:30.680 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:36.688 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:36.699 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:42.708 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:42.718 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:48.731 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:51.774 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:53:57.779 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:57.864 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:57.864 INFO:teuthology.orchestra.run.smithi110:> hostname 2019-10-13T03:53:58.438 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com 2019-10-13T03:53:58.439 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.439 INFO:teuthology.orchestra.run.smithi110:> grep smithi110.front.sepia.ceph.com /etc/hosts 2019-10-13T03:53:58.510 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110 2019-10-13T03:53:58.510 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.511 INFO:teuthology.orchestra.run.smithi110:> sudo hostname smithi110 2019-10-13T03:53:58.602 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.603 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/smithi110.front.sepia.ceph.com/smithi110/g' /etc/hosts 2019-10-13T03:53:58.684 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.684 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/172.21.15.110/172.21.15.110/g' /etc/hosts 2019-10-13T03:53:58.761 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.762 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/smithi110.front.sepia.ceph.com/smithi110/g' /etc/hostname 2019-10-13T03:53:58.841 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:58.841 INFO:teuthology.orchestra.run.smithi110:> sudo hostname smithi110 2019-10-13T03:53:58.919 INFO:teuthology.provision.fog.smithi110:Deploy complete! 2019-10-13T03:53:58.921 DEBUG:teuthology.parallel:result is None 2019-10-13T03:53:59.094 INFO:teuthology.lock.keys:Checking smithi110.front.sepia.ceph.com 2019-10-13T03:53:59.094 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T03:53:59.116 INFO:teuthology.lock.keys:Checking smithi204.front.sepia.ceph.com 2019-10-13T03:53:59.116 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-13T03:53:59.133 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server 2019-10-13T03:53:59.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:53:59.134 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:59.221 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:59.221 INFO:teuthology.orchestra.run.smithi110:> true 2019-10-13T03:53:59.392 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:59.393 INFO:teuthology.orchestra.run.smithi110:> uname -m 2019-10-13T03:53:59.462 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2019-10-13T03:53:59.463 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:53:59.463 INFO:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T03:53:59.533 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="18.04" 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=bionic 2019-10-13T03:53:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=bionic 2019-10-13T03:53:59.536 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2019-10-13T03:53:59.553 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server 2019-10-13T03:53:59.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:53:59.553 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:53:59.639 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:59.639 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-13T03:53:59.852 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:59.852 INFO:teuthology.orchestra.run.smithi204:> uname -m 2019-10-13T03:53:59.922 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2019-10-13T03:53:59.923 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:53:59.923 INFO:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T03:53:59.993 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="18.04" 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=bionic 2019-10-13T03:53:59.994 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=bionic 2019-10-13T03:53:59.995 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2019-10-13T03:54:00.009 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-10-13T03:54:00.051 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6EdIEDzZRdBRD0bLwvaqt1T2CWfYA4jgooF37gfsxlmNU8R7fsTcSSy4aIzep3lsgjtvbgfSe9phgjthGuFZn3BvGAZEQRt/PN2Jo+QAspYO/VWjU2DgRKNGwpPsJ9KplGabY0l3aAMPJqj1zySLSZ3fITcyCmFX2mnB1AeQcTaX50JpSiyxr3UZ2pRumFxhCb9aAmDZnLN3IsqEKZ83bMyRrXPG15KSyxCVHlSonFn6Ww2aO+jPB1W9RrJkJZdvzbEKZKOgd5YDA1jPltKKeWHYOfE2SKF1xNOFnyUut4UhQ5DbBK9yrkOa5yO0ksPW0LmKTM5UXCtvV7eQc8dRZ smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTlOcAA/Jqcpb/OnkOrrCmLB5vmc9IBTCJhMmorSjsCMf8qQOidO511p0AH330PF9U0wABWhkxY81XHPtDnTJ9s2n6kflGm2uj3F+qPocj4Z9UJ8XSUfLRtgBVWjcnyksNai26fo+xBDZK1cFxxDJp+T+L8FlQ9GUInuU0fmz3ohjpGL4WQFSfge5s4qFziOXMUqk8nONiddfXPZNJ/mvS1GYWcoG+aCTQ1Pqf+BfzINGwzYYpdJFqVCTS13iw2l3e/p7Hh8UeY2u/MCrE/lwCD1siKnYkToTJ7LPRbLrb7A0faOD0RUK0owHIwv1+owJ3ZfArM6LqjQKj7DFb6DF3 2019-10-13T03:54:00.051 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-13T03:54:00.108 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-13T03:54:00.118 INFO:teuthology.task.internal:Saving configuration 2019-10-13T03:54:00.135 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-13T03:54:00.145 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-13T03:54:00.160 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 03:48:10.454905', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi110.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIENNTfzaxZPXp5G3HQ8Me0aHgYmlQ/OP1D087AlS9r4I', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711'} 2019-10-13T03:54:00.175 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-13 03:48:10.454886', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi204.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3p/i/KhhpkGcL6P1N34frWMb5FUFYA73wYIwyexeS2', u'description': u'/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711'} 2019-10-13T03:54:00.175 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-13T03:54:00.185 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-10-13T03:54:00.185 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2019-10-13T03:54:00.185 INFO:teuthology.run_tasks:Running task console_log... 2019-10-13T03:54:00.328 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-13T03:54:00.328 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-13T03:54:00.338 INFO:teuthology.task.internal:Opening connections... 2019-10-13T03:54:00.339 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2019-10-13T03:54:00.340 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:54:00.426 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:54:00.427 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:54:00.513 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-13T03:54:00.521 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:00.522 INFO:teuthology.orchestra.run.smithi204:> uname -m 2019-10-13T03:54:00.593 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2019-10-13T03:54:00.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:00.594 INFO:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2019-10-13T03:54:00.663 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2019-10-13T03:54:00.664 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T03:54:00.664 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2019-10-13T03:54:00.664 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2019-10-13T03:54:00.664 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="18.04" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=bionic 2019-10-13T03:54:00.665 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=bionic 2019-10-13T03:54:00.666 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2019-10-13T03:54:00.678 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:00.678 INFO:teuthology.orchestra.run.smithi110:> uname -m 2019-10-13T03:54:00.690 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2019-10-13T03:54:00.691 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:00.691 INFO:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2019-10-13T03:54:00.761 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="18.04" 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-13T03:54:00.770 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-13T03:54:00.771 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-13T03:54:00.771 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-13T03:54:00.771 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=bionic 2019-10-13T03:54:00.771 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=bionic 2019-10-13T03:54:00.772 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2019-10-13T03:54:00.785 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-13T03:54:00.801 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-13T03:54:00.812 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-13T03:54:00.813 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:00.813 INFO:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2019-10-13T03:54:00.816 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:00.816 INFO:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2019-10-13T03:54:00.823 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-13T03:54:00.864 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-13T03:54:00.864 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:00.864 INFO:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2019-10-13T03:54:00.867 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:00.867 INFO:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2019-10-13T03:54:00.943 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-13T03:54:00.980 INFO:teuthology.run_tasks:Running task kernel... 2019-10-13T03:54:00.996 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'mds.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2019-10-13T03:54:00.996 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-13T03:54:00.996 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:00.997 INFO:teuthology.orchestra.run.smithi110:> uname -r 2019-10-13T03:54:01.003 INFO:teuthology.orchestra.run.smithi110.stdout:4.15.0-58-generic 2019-10-13T03:54:01.003 INFO:teuthology.task.kernel:Running kernel on smithi110: 4.15.0-58-generic 2019-10-13T03:54:01.003 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:01.003 INFO:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2019-10-13T03:54:01.124 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:01.125 INFO:teuthology.orchestra.run.smithi110:> sudo apt-get update 2019-10-13T03:54:01.254 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-13T03:54:01.361 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T03:54:01.508 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [529 kB] 2019-10-13T03:54:01.522 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-13T03:54:01.646 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB] 2019-10-13T03:54:01.770 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [177 kB] 2019-10-13T03:54:01.875 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-13T03:54:01.887 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-13T03:54:01.910 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-13T03:54:01.946 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-13T03:54:02.083 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-13T03:54:02.219 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-13T03:54:02.225 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [591 kB] 2019-10-13T03:54:02.326 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-13T03:54:02.362 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-13T03:54:02.397 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-13T03:54:02.829 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [752 kB] 2019-10-13T03:54:03.479 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [270 kB] 2019-10-13T03:54:04.005 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-13T03:54:04.251 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-13T03:54:04.413 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-13T03:54:05.097 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-13T03:54:05.778 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-13T03:54:06.318 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-13T03:54:06.479 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-13T03:54:06.641 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-13T03:54:06.803 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-13T03:54:06.964 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-13T03:54:07.126 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-13T03:54:07.232 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 6,739 kB in 6s (1,137 kB/s) 2019-10-13T03:54:08.542 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:54:08.586 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:08.586 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T03:54:08.670 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:54:08.890 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T03:54:08.890 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T03:54:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T03:54:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T03:54:08.964 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:08.964 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T03:54:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:54:09.371 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T03:54:09.371 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T03:54:09.539 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2019-10-13T03:54:09.539 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T03:54:09.540 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-13T03:54:09.540 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-13T03:54:09.541 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2019-10-13T03:54:09.541 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-13T03:54:09.542 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2019-10-13T03:54:09.542 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2019-10-13T03:54:09.572 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2019-10-13T03:54:09.572 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T03:54:09.572 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-13T03:54:09.572 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-13T03:54:09.574 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2019-10-13T03:54:09.574 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-13T03:54:09.755 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T03:54:09.755 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 66.3 MB of archives. 2019-10-13T03:54:09.755 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-13T03:54:09.755 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-13T03:54:10.841 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-13T03:54:11.791 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-13T03:54:13.433 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-13T03:54:13.687 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-13T03:54:13.859 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-13T03:54:14.894 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-13T03:54:15.607 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-13T03:54:16.029 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 66.3 MB in 6s (11.0 MB/s) 2019-10-13T03:54:16.184 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-13T03:54:16.222 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-13T03:54:16.224 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:54:16.272 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:17.940 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-13T03:54:17.952 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:54:18.062 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:18.543 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-13T03:54:18.555 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:54:18.593 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:22.530 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:54:22.659 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:54:22.975 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:54:23.071 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:54:23.399 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-13T03:54:23.410 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-13T03:54:23.449 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T03:54:28.867 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-13T03:54:28.886 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:54:28.926 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:31.410 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:54:31.539 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:54:31.879 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T03:54:32.003 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:32.562 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:32.688 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-13T03:54:32.814 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:33.308 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:54:33.308 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-13T03:54:33.449 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:34.001 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-13T03:54:34.127 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-13T03:54:34.253 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:54:34.423 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-13T03:54:34.424 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:54:47.742 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-13T03:54:47.742 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2019-10-13T03:54:47.777 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2019-10-13T03:54:48.164 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:54:48.191 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:54:48.535 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T03:54:48.578 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T03:54:49.443 INFO:teuthology.orchestra.run.smithi110.stdout:done 2019-10-13T03:54:49.881 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:54:49.882 INFO:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2019-10-13T03:54:49.906 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 15 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T03:54:49.907 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 4.15.0.65.67 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2019-10-13T03:54:49.908 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2019-10-13T03:54:49.909 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2019-10-13T03:54:49.909 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-13T03:54:49.909 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:49.909 INFO:teuthology.orchestra.run.smithi204:> uname -r 2019-10-13T03:54:49.917 INFO:teuthology.orchestra.run.smithi204.stdout:4.15.0-58-generic 2019-10-13T03:54:49.917 INFO:teuthology.task.kernel:Running kernel on smithi204: 4.15.0-58-generic 2019-10-13T03:54:49.918 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:49.918 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2019-10-13T03:54:50.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:50.065 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-13T03:54:50.331 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T03:54:50.331 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-13T03:54:50.473 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-13T03:54:50.785 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB] 2019-10-13T03:54:50.825 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-13T03:54:51.199 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [591 kB] 2019-10-13T03:54:51.335 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [529 kB] 2019-10-13T03:54:51.805 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [752 kB] 2019-10-13T03:54:51.939 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [177 kB] 2019-10-13T03:54:52.396 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-13T03:54:52.437 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [270 kB] 2019-10-13T03:54:52.557 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-13T03:54:52.719 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-13T03:54:52.949 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-13T03:54:53.190 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-13T03:54:53.332 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-13T03:54:53.351 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-13T03:54:53.937 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-13T03:54:54.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-13T03:54:54.425 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-13T03:54:54.587 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-13T03:54:54.726 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-13T03:54:54.748 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-13T03:54:55.266 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-13T03:54:55.428 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-13T03:54:55.590 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-13T03:54:55.751 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-13T03:54:55.913 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-13T03:54:56.074 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-13T03:54:56.183 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 6,739 kB in 6s (1,137 kB/s) 2019-10-13T03:54:57.478 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:54:57.525 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:57.525 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T03:54:57.604 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:54:57.823 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T03:54:57.823 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T03:54:57.899 INFO:teuthology.orchestra.run.smithi204.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T03:54:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T03:54:57.899 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:54:57.900 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T03:54:57.979 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:54:58.215 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T03:54:58.216 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T03:54:58.370 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2019-10-13T03:54:58.370 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T03:54:58.370 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-13T03:54:58.371 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-13T03:54:58.372 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2019-10-13T03:54:58.372 INFO:teuthology.orchestra.run.smithi204.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-13T03:54:58.372 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2019-10-13T03:54:58.372 INFO:teuthology.orchestra.run.smithi204.stdout: thermald 2019-10-13T03:54:58.410 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2019-10-13T03:54:58.410 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-13T03:54:58.410 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-13T03:54:58.411 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-13T03:54:58.413 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2019-10-13T03:54:58.414 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-13T03:54:58.596 INFO:teuthology.orchestra.run.smithi204.stdout:3 upgraded, 5 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T03:54:58.596 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 66.3 MB of archives. 2019-10-13T03:54:58.596 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-13T03:54:58.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-13T03:54:59.703 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-13T03:55:00.660 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-13T03:55:02.334 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-13T03:55:02.497 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-13T03:55:02.668 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-13T03:55:03.701 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-13T03:55:04.408 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-13T03:55:04.824 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 66.3 MB in 6s (11.1 MB/s) 2019-10-13T03:55:04.995 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-13T03:55:05.032 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-13T03:55:05.034 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:55:05.084 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:06.780 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-13T03:55:06.790 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:55:06.889 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:07.416 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-13T03:55:07.425 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:55:07.457 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:11.389 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:55:11.502 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:55:11.817 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:55:11.939 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:55:12.268 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-13T03:55:12.275 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-13T03:55:12.317 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T03:55:17.081 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-13T03:55:17.100 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-13T03:55:17.138 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:19.509 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-13T03:55:19.642 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-13T03:55:19.960 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-13T03:55:20.071 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:20.599 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:20.725 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-13T03:55:20.851 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:21.284 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:55:21.284 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-13T03:55:21.411 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:21.888 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-13T03:55:22.006 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-13T03:55:22.132 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-13T03:55:22.293 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-13T03:55:22.293 INFO:teuthology.orchestra.run.smithi204.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:55:35.125 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-13T03:55:35.126 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub' 2019-10-13T03:55:35.153 INFO:teuthology.orchestra.run.smithi204.stdout:Generating grub configuration file ... 2019-10-13T03:55:35.571 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:55:35.595 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:55:35.895 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T03:55:35.931 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T03:55:36.772 INFO:teuthology.orchestra.run.smithi204.stdout:done 2019-10-13T03:55:37.169 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:37.169 INFO:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2019-10-13T03:55:37.192 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2019-10-13T03:55:37.192 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2019-10-13T03:55:37.192 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 15 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 4.15.0.65.67 2019-10-13T03:55:37.193 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T03:55:37.194 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2019-10-13T03:55:37.194 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2019-10-13T03:55:37.194 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2019-10-13T03:55:37.194 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2019-10-13T03:55:37.195 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-13T03:55:37.195 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-10-13T03:55:37.195 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-13T03:55:37.195 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-13T03:55:37.195 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-10-13T03:55:37.195 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2019-10-13T03:55:37.196 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:37.196 INFO:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2019-10-13T03:55:37.257 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:37.258 INFO:teuthology.orchestra.run.smithi110:> sudo apt-get update 2019-10-13T03:55:37.371 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T03:55:37.487 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T03:55:37.648 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T03:55:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T03:55:39.134 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:55:39.180 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:39.180 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T03:55:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:55:39.501 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T03:55:39.502 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T03:55:39.577 INFO:teuthology.orchestra.run.smithi110.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T03:55:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T03:55:39.579 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:39.579 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T03:55:39.671 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T03:55:39.890 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T03:55:39.897 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T03:55:40.097 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-13T03:55:40.098 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T03:55:40.099 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.100 INFO:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2019-10-13T03:55:40.130 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2019-10-13T03:55:40.130 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2019-10-13T03:55:40.131 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2019-10-13T03:55:40.131 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2019-10-13T03:55:40.131 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 15 2019-10-13T03:55:40.131 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T03:55:40.131 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2019-10-13T03:55:40.132 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2019-10-13T03:55:40.132 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 4.15.0.65.67 2019-10-13T03:55:40.132 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T03:55:40.132 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2019-10-13T03:55:40.133 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2019-10-13T03:55:40.133 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2019-10-13T03:55:40.133 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2019-10-13T03:55:40.134 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.134 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T03:55:40.205 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.Kz5DLFXPuB 2019-10-13T03:55:40.205 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.206 INFO:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Kz5DLFXPuB 2019-10-13T03:55:40.294 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.294 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.Kz5DLFXPuB 2019-10-13T03:55:40.468 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.Kz5DLFXPuB is 8KB 2019-10-13T03:55:40.518 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.519 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.Kz5DLFXPuB 2019-10-13T03:55:40.526 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.526 INFO:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-13T03:55:40.604 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.604 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-13T03:55:40.729 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-13T03:55:40.729 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:40.729 INFO:teuthology.orchestra.run.smithi110:> sudo update-grub 2019-10-13T03:55:41.936 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2019-10-13T03:55:41.954 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2019-10-13T03:55:42.179 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:55:42.191 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:55:42.467 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T03:55:42.475 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T03:55:43.009 INFO:teuthology.orchestra.run.smithi110.stderr:done 2019-10-13T03:55:43.011 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:55:43.011 INFO:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2019-10-13T03:55:43.015 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-10-13T03:55:43.015 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2019-10-13T03:55:43.016 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:43.016 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2019-10-13T03:55:43.074 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:43.075 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-13T03:55:43.200 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T03:55:43.315 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T03:55:43.478 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T03:55:43.645 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T03:55:45.160 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:55:45.204 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:45.204 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-13T03:55:45.353 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:55:45.658 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T03:55:45.658 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T03:55:45.748 INFO:teuthology.orchestra.run.smithi204.stderr:E: Unable to locate package linux-image-current-generic 2019-10-13T03:55:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-13T03:55:45.750 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:45.750 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-13T03:55:45.840 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T03:55:46.077 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T03:55:46.077 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T03:55:46.255 INFO:teuthology.orchestra.run.smithi204.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-13T03:55:46.255 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 0 newly installed, 0 to remove and 98 not upgraded. 2019-10-13T03:55:46.257 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.257 INFO:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2019-10-13T03:55:46.272 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 15 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2019-10-13T03:55:46.273 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 4.15.0.65.67 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2019-10-13T03:55:46.274 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2019-10-13T03:55:46.275 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2019-10-13T03:55:46.275 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.275 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T03:55:46.345 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.OsycCB7aYE 2019-10-13T03:55:46.346 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.346 INFO:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub/grub.cfg /tmp/tmp.OsycCB7aYE 2019-10-13T03:55:46.421 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.422 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.OsycCB7aYE 2019-10-13T03:55:46.606 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.OsycCB7aYE is 8KB 2019-10-13T03:55:46.617 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.618 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.OsycCB7aYE 2019-10-13T03:55:46.627 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.627 INFO:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-13T03:55:46.707 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.707 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-13T03:55:46.832 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-13T03:55:46.832 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:46.832 INFO:teuthology.orchestra.run.smithi204:> sudo update-grub 2019-10-13T03:55:48.043 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub' 2019-10-13T03:55:48.060 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2019-10-13T03:55:48.285 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-13T03:55:48.295 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-13T03:55:48.557 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-13T03:55:48.567 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-13T03:55:49.023 INFO:teuthology.orchestra.run.smithi204.stderr:done 2019-10-13T03:55:49.024 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:49.024 INFO:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2019-10-13T03:55:49.027 INFO:teuthology.misc:Re-opening connections... 2019-10-13T03:55:49.027 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com 2019-10-13T03:55:49.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:55:49.029 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:55:49.112 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:55:49.113 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-13T03:55:49.739 DEBUG:teuthology.misc:waited 0.711385965347 2019-10-13T03:55:50.744 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:55:50.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:55:50.745 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:56:50.747 DEBUG:teuthology.orchestra.remote:timed out 2019-10-13T03:56:50.747 DEBUG:teuthology.misc:waited 61.7196509838 2019-10-13T03:56:51.748 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:56:51.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:56:51.749 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:56:54.808 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2019-10-13T03:56:54.809 DEBUG:teuthology.misc:waited 65.7813968658 2019-10-13T03:56:55.809 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:56:55.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:56:55.810 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:56:57.880 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2019-10-13T03:56:57.881 DEBUG:teuthology.misc:waited 68.8534579277 2019-10-13T03:56:58.882 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:56:58.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:56:58.883 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:57:00.952 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2019-10-13T03:57:00.953 DEBUG:teuthology.misc:waited 71.9254579544 2019-10-13T03:57:01.954 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:57:01.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:57:01.954 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:57:04.024 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2019-10-13T03:57:04.025 DEBUG:teuthology.misc:waited 74.9974658489 2019-10-13T03:57:05.026 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:57:05.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:57:05.028 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:58:05.029 DEBUG:teuthology.orchestra.remote:timed out 2019-10-13T03:58:05.029 DEBUG:teuthology.misc:waited 136.001914024 2019-10-13T03:58:06.031 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:58:06.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:58:06.032 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:58:06.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:06.118 INFO:teuthology.orchestra.run.smithi110:> true 2019-10-13T03:58:06.424 DEBUG:teuthology.misc:waited 137.397126913 2019-10-13T03:58:07.426 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-10-13T03:58:07.426 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-65-generic"... 2019-10-13T03:58:07.426 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:07.426 INFO:teuthology.orchestra.run.smithi110:> uname -r 2019-10-13T03:58:07.436 INFO:teuthology.orchestra.run.smithi110.stdout:4.15.0-65-generic 2019-10-13T03:58:07.436 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-13T03:58:07.436 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-13T03:58:07.436 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-10-13T03:58:07.436 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-65-generic"... 2019-10-13T03:58:07.437 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:07.437 INFO:teuthology.orchestra.run.smithi204:> uname -r 2019-10-13T03:58:07.437 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-13T03:58:07.439 ERROR:teuthology.task.kernel:Saw exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 704, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 183, in need_to_install stdout=uname_fp, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 432, in run r.execute() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 102, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 493, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 806, in open_session timeout=timeout, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 933, in open_channel raise e error: [Errno 104] Connection reset by peer 2019-10-13T03:58:08.442 INFO:teuthology.misc:Re-opening connections... 2019-10-13T03:58:08.442 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com 2019-10-13T03:58:08.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:58:08.443 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:58:08.595 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:08.596 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-13T03:58:08.903 DEBUG:teuthology.misc:waited 0.460911989212 2019-10-13T03:58:09.904 INFO:teuthology.misc:trying to connect to ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T03:58:09.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T03:58:09.906 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T03:58:09.990 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:09.990 INFO:teuthology.orchestra.run.smithi110:> true 2019-10-13T03:58:10.219 DEBUG:teuthology.misc:waited 1.77764296532 2019-10-13T03:58:11.220 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-10-13T03:58:11.220 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-65-generic"... 2019-10-13T03:58:11.221 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:11.221 INFO:teuthology.orchestra.run.smithi204:> uname -r 2019-10-13T03:58:11.228 INFO:teuthology.orchestra.run.smithi204.stdout:4.15.0-65-generic 2019-10-13T03:58:11.228 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-13T03:58:11.228 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-13T03:58:12.229 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-10-13T03:58:12.229 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.229 INFO:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T03:58:12.256 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2019-10-13T03:58:12.277 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-10-13T03:58:12.277 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.277 INFO:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-13T03:58:12.361 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2019-10-13T03:58:12.378 INFO:teuthology.run_tasks:Running task internal.base... 2019-10-13T03:58:12.388 INFO:teuthology.task.internal:Creating test directory... 2019-10-13T03:58:12.389 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.389 INFO:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T03:58:12.392 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.392 INFO:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-13T03:58:12.400 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-10-13T03:58:12.414 INFO:teuthology.run_tasks:Running task internal.archive... 2019-10-13T03:58:12.428 INFO:teuthology.task.internal:Creating archive directory... 2019-10-13T03:58:12.429 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.429 INFO:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T03:58:12.447 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.447 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-13T03:58:12.463 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-10-13T03:58:12.473 INFO:teuthology.task.internal:Enabling coredump saving... 2019-10-13T03:58:12.473 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.473 INFO:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T03:58:12.540 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.541 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T03:58:12.556 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T03:58:12.564 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-13T03:58:12.565 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-10-13T03:58:12.581 INFO:teuthology.task.internal:Configuring sudo... 2019-10-13T03:58:12.581 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.581 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T03:58:12.617 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.617 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-13T03:58:12.633 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-10-13T03:58:12.683 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-10-13T03:58:12.683 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.683 INFO:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T03:58:12.687 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.687 INFO:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-13T03:58:12.694 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.694 INFO:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T03:58:12.765 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.765 INFO:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T03:58:12.839 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:12.839 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T03:58:12.950 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.950 INFO:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-13T03:58:12.958 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:12.958 INFO:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-13T03:58:13.032 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:13.032 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-13T03:58:13.145 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T03:58:13.145 INFO:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2019-10-13T03:58:13.212 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T03:58:13.213 INFO:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2019-10-13T03:58:13.263 INFO:teuthology.run_tasks:Running task internal.timer... 2019-10-13T03:58:13.274 INFO:teuthology.task.internal:Starting timer... 2019-10-13T03:58:13.274 INFO:teuthology.run_tasks:Running task pcp... 2019-10-13T03:58:13.315 INFO:teuthology.run_tasks:Running task selinux... 2019-10-13T03:58:13.327 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2019-10-13T03:58:13.328 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2019-10-13T03:58:13.328 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-10-13T03:58:13.328 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2019-10-13T03:58:13.328 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-10-13T03:58:13.329 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-10-13T03:58:13.355 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-10-13T03:58:13.380 INFO:teuthology.repo_utils:Fetching master from upstream 2019-10-13T03:58:13.464 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch master 2019-10-13T03:58:13.499 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'set_python_path.yml'}, {'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-10-13T03:58:13.502 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi204.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-10-13T03:58:13.899 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-10-13T03:58:17.554 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T03:58:17.574 INFO:teuthology.task.ansible.out: TASK [Check for /usr/bin/python] *********************************************** Sunday 13 October 2019 03:58:17 +0000 (0:00:00.061) 0:00:00.061 ******** 2019-10-13T03:58:19.228 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi110.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:58:19.229 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.001739", "end": "2019-10-13 03:58:19.199380", "rc": 0, "start": "2019-10-13 03:58:19.197641", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-13T03:58:19.549 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi204.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.003380", "end": "2019-10-13 03:58:19.843808", "rc": 0, "start": "2019-10-13 03:58:19.840428", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-13T03:58:19.557 INFO:teuthology.task.ansible.out: TASK [Set ansible_python_interpreter=/usr/bin/python3] ************************* Sunday 13 October 2019 03:58:19 +0000 (0:00:01.982) 0:00:02.044 ******** 2019-10-13T03:58:19.584 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:19.596 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:19.603 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-13T03:58:19.614 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:19 +0000 (0:00:00.056) 0:00:02.101 ******** 2019-10-13T03:58:19.627 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:19 +0000 (0:00:00.013) 0:00:02.114 ******** 2019-10-13T03:58:20.845 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T03:58:20.853 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.204"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:77ab"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570939101", "hour": "03", "iso8601": "2019-10-13T03:58:21Z", "iso8601_basic": "20191013T035821095108", "iso8601_basic_short": "20191013T035821", "iso8601_micro": "2019-10-13T03:58:21.095187Z", "minute": "58", "month": "10", "second": "21", "time": "03:58:21", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.204", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:77ab", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShh 2019-10-13T03:58:20.855 INFO:teuthology.task.ansible.out:q6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "nvme0n1": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["97d94574-b350-4896-9285-2998fb12b1c3"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "labels": [], "masters": [], "uuids": ["97d94574-b350-4896-9285-2998fb12b1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 G 2019-10-13T03:58:20.856 INFO:teuthology.task.ansible.out:B", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0. 2019-10-13T03:58:20.858 INFO:teuthology.task.ansible.out:00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contr 2019-10-13T03:58:20.859 INFO:teuthology.task.ansible.out:oller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941a36"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "a 2019-10-13T03:58:20.862 INFO:teuthology.task.ansible.out:nsible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "p 2019-10-13T03:58:20.863 INFO:teuthology.task.ansible.out:romisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_i 2019-10-13T03:58:20.865 INFO:teuthology.task.ansible.out:pxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_i 2019-10-13T03:58:20.866 INFO:teuthology.task.ansible.out:pxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:77:aa", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "t 2019-10-13T03:58:20.867 INFO:teuthology.task.ansible.out:x_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.204", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:77ab", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:77:ab", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-zavrlhwnzklyzguvxtdhdomutlvymsjf ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "eno1", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-gene 2019-10-13T03:58:20.869 INFO:teuthology.task.ansible.out:ric", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, 2019-10-13T03:58:20.870 INFO:teuthology.task.ansible.out: "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31473, "ansible_memory_mb": {"nocache": {"free": 31781, "used": 273}, "real": {"free": 31473, "total": 32054, "used": 581}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440804, "block_size": 4096, "block_total": 240095108, "block_used": 13654304, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501533184, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi204.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel 2019-10-13T03:58:20.871 INFO:teuthology.task.ansible.out:(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928378", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9362A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANZkQ/ZjhsiZKZJ09/tZnoH/6R0D4QJ5NQhL1jbow5ed/pOt7drfgTwItQg8DZZcVmQ0oB0GTLEAB9GUTqAu4oeILnlG8+miuIHxpqa400POaQLxTZ7jdgSWsqGLbR0uChhK8hJlq0DhqiwsJ4gHrdTYWzCt8B4cwFjeUthoLmm9AAAAFQCn01UobLOvI49K43i8/4SsM2S1owAAAIEAo2GacMRcgogjKFMTo7RDCqun8to4Fhx7zZc3FTeh7NMDGc9XsXgneU6KLJfqQYej85ja1JB/hW6GjcDUltewshD+i/jZzgRQ0yDd31iJTwsx+WPD70l7BRaO3mclp+t34JgA5/RRUdfyYze2V2zGbrEdoa4h4vSNS8u240nqsMUAAACBAMCVr2m0CHX3LmmMMd2eIunMpsoJvQ3Kd2oDfKCe1vcTsqtIttZGzJr8uonWoGg1xMStQhamlSLfrT6EUDPtKwgpq2gTkxA786YSok2SXx2NRkXTD6QmpsxZGSjmooTDqA9QwQY0C5mHdwnGhUnsPmrt939u36g9HUB0qySY52OU", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLeCHyOBMMrKPsWWbrq7RDY+H50DIbYQYrPEf49A9p2agwRvrDkjxARHcYmIZ7UfBmiQkX9PUysIJcW2YwhKxUE=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZ 2019-10-13T03:58:20.873 INFO:teuthology.task.ansible.out:DI1NTE5AAAAIF3p/i/KhhpkGcL6P1N34frWMb5FUFYA73wYIwyexeS2", "ansible_ssh_host_key_rsa_public": "A 2019-10-13T03:58:20.875 INFO:teuthology.task.ansible.out:AAAB3NzaC1yc2EAAAADAQABAAABAQDTlOcAA/Jqcpb/OnkOrrCmLB5vmc9IBTCJhMmorSjsCMf8qQOidO511p0AH330PF9U0wABWhkxY81XHPtDnTJ9s2n6kflGm2uj3F+qPocj4Z9UJ8XSUfLRtgBVWjcnyksNai26fo+xBDZK1cFxxDJp+T+L8FlQ9GUInuU0fmz3ohjpGL4WQFSfge5s4qFziOXMUqk8nONiddfXPZNJ/mvS1GYWcoG+aCTQ1Pqf+BfzINGwzYYpdJFqVCTS13iw2l3e/p7Hh8UeY2u/MCrE/lwCD1siKnYkToTJ7LPRbLrb7A0faOD0RUK0owHIwv1+owJ3ZfArM6LqjQKj7DFb6DF3", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 64, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T03:58:20.963 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:20 +0000 (0:00:01.332) 0:00:03.447 ******** 2019-10-13T03:58:20.980 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.110"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:cc35"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570939100", "hour": "03", "iso8601": "2019-10-13T03:58:20Z", "iso8601_basic": "20191013T035820853216", "iso8601_basic_short": "20191013T035820", "iso8601_micro": "2019-10-13T03:58:20.853323Z", "minute": "58", "month": "10", "second": "20", "time": "03:58:20", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.110", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cc:35", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:cc35", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cc:35", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGl 2019-10-13T03:58:20.982 INFO:teuthology.task.ansible.out:Gre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "nvme0n1": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["40030898-e0dc-4263-932e-a36226be20b4"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "labels": [], "masters": [], "uuids": ["40030898-e0dc-4263-932e-a36226be20b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB 2019-10-13T03:58:20.983 INFO:teuthology.task.ansible.out:", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.0 2019-10-13T03:58:20.984 INFO:teuthology.task.ansible.out:0 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contro 2019-10-13T03:58:20.986 INFO:teuthology.task.ansible.out:ller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091dfc3c1"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "an 2019-10-13T03:58:20.987 INFO:teuthology.task.ansible.out:sible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]" 2019-10-13T03:58:20.988 INFO:teuthology.task.ansible.out:, "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:db:14", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp 2019-10-13T03:58:20.990 INFO:teuthology.task.ansible.out:_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:db:15", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "featur 2019-10-13T03:58:20.991 INFO:teuthology.task.ansible.out:es": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:cc:34", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": 2019-10-13T03:58:20.993 INFO:teuthology.task.ansible.out:true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.110", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:cc35", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8 2019-10-13T03:58:20.994 INFO:teuthology.task.ansible.out:f:cc:35", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-shmkwiqnlqgrotndbhfezoaxqilysdex ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi110.front.sepia.ceph.com", "ansible_hostname": "smithi110", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "lo", "eno1"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixe 2019-10-13T03:58:20.995 INFO:teuthology.task.ansible.out:d]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31473, "ansible_memory_mb": {"nocache": {"free": 31782, " 2019-10-13T03:58:20.997 INFO:teuthology.task.ansible.out:used": 272}, "real": {"free": 31473, "total": 32054, "used": 581}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440810, "block_size": 4096, "block_total": 240095108, "block_used": 13654298, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501557760, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi110.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", 2019-10-13T03:58:20.998 INFO:teuthology.task.ansible.out: "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731578", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8DB14", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAOUqhx+sV7P9h30R/09UDHtDtHrqhhWGaLNDlHnHksKhW7Ot7Wp5fpSTOIKceyLjb3cMDByrqCuVcezjWcjF4fpolqsZQTcOdIdGZa56XTl6CIMMBVFI/6G7AUTSDCbGjMKd350N3+GQCtVYKFBDNL/T80jCUbENe2MzmsHZImJLAAAAFQDVGz7784bagR+m2cSmjLM54AHr/QAAAIEAiULyaTxt8L95VLVHhxj3BW9dDwwuXF/yibygV//cCycb3PgmzZR5lWCFv6iOsR/QFA0Jwk9ARVnhSHQi9gWHucYUR1LFtysvx6tMNOL5w8gRHfbuh/Q3lR2SNdTEmcebsrvumxlaR7UKqH+8oQKZoqIrfCKgr504am0wxX8RWaAAAACAb7eDNMYH6mc39b8Sxu+tfDihb792g9s0kx+QliZFpBwapgcJa2101q5soov0aIkwWRwjTflhclv5RTTF4SWyONbExgZqVD7vPtwBehx9eVmL92Op7j0Hl4zL/elj4jJBYe4LVofPL1PQZCQCjdjsbicicM1pRJ4QCBKhuMVhrGs=", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMVXusGKTYEhqZY0lmW+nL1LOT4H2nPGwRSoNVf2S+WQETHog95SE4W205BeVOcVOdzznqMfawnr4JRM4oS/5KU=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIENNTfzaxZPXp5G3HQ8Me0aHgYmlQ/OP1D087AlS9r4I", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQD6EdIEDzZRdBRD0bLwvaqt1T2CWfYA4jgooF37gfsxlmNU8R7fsTcSSy4aIzep3lsgjtvbgfSe9phgjthGuFZn3BvGAZEQRt/PN2Jo+QAspYO/VWjU2Dg 2019-10-13T03:58:21.000 INFO:teuthology.task.ansible.out:RKNGwpPsJ9KplGabY0l3aAMPJqj1zySLSZ3fITcyCmFX2mnB1AeQcTaX50JpSiyxr3UZ2pRumFxhCb9aAmDZnLN3IsqEKZ83bMyRrXPG15KSyxCVHlSonFn6Ww2aO+jPB1W9RrJkJZdvzbEKZKOgd5YDA1jPltKKeWHYOfE2SKF1xNOFnyUut4UhQ5DbBK9yrkOa5yO0ksPW0LmKTM5UXCtvV7eQc8dRZ", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 59, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T03:58:21.100 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:21 +0000 (0:00:00.140) 0:00:03.587 ******** 2019-10-13T03:58:21.230 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T03:58:21.271 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:21 +0000 (0:00:00.171) 0:00:03.758 ******** 2019-10-13T03:58:21.272 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:21.278 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T03:58:21.279 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:21.324 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:21 +0000 (0:00:00.052) 0:00:03.811 ******** 2019-10-13T03:58:21.737 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-10-13T03:58:21.737 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T03:58:21.786 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:21 +0000 (0:00:00.462) 0:00:04.274 ******** 2019-10-13T03:58:21.793 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-13T03:58:21.803 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:58:21.833 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:21 +0000 (0:00:00.046) 0:00:04.321 ******** 2019-10-13T03:58:21.852 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:58:21.983 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-10-13T03:58:21.983 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.065212", "end": "2019-10-13 03:58:22.277663", "rc": 0, "start": "2019-10-13 03:58:22.212451", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-13T03:58:22.027 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:22 +0000 (0:00:00.194) 0:00:04.515 ******** 2019-10-13T03:58:22.044 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.062535", "end": "2019-10-13 03:58:22.010486", "rc": 0, "start": "2019-10-13 03:58:21.947951", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-13T03:58:22.088 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:22 +0000 (0:00:00.060) 0:00:04.575 ******** 2019-10-13T03:58:22.323 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-10-13T03:58:22.324 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:58:22.326 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:58:22.372 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:22 +0000 (0:00:00.284) 0:00:04.860 ******** 2019-10-13T03:58:22.411 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:22 +0000 (0:00:00.038) 0:00:04.899 ******** 2019-10-13T03:58:23.206 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-10-13T03:58:23.213 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570939102.39-253775593419188/source", "state": "file", "uid": 0} 2019-10-13T03:58:23.245 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:23 +0000 (0:00:00.834) 0:00:05.733 ******** 2019-10-13T03:58:23.256 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1570939102.43-106778068274133/source", "state": "file", "uid": 0} 2019-10-13T03:58:23.291 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:23 +0000 (0:00:00.045) 0:00:05.778 ******** 2019-10-13T03:58:23.604 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:23.606 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:23.607 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:23.733 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:23.745 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:23.868 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:23.890 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.007 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.030 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.151 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.193 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.276 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.318 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.417 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.463 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-13T03:58:24.593 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2019-10-13T03:58:24.594 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-10-13T03:58:24.605 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-10-13T03:58:24.818 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:24 +0000 (0:00:01.527) 0:00:07.306 ******** 2019-10-13T03:58:24.856 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:24 +0000 (0:00:00.037) 0:00:07.343 ******** 2019-10-13T03:58:25.580 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-13T03:58:25.589 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.204"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:77ab"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570939105", "hour": "03", "iso8601": "2019-10-13T03:58:25Z", "iso8601_basic": "20191013T035825818300", "iso8601_basic_short": "20191013T035825", "iso8601_micro": "2019-10-13T03:58:25.818357Z", "minute": "58", "month": "10", "second": "25", "time": "03:58:25", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.204", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:77ab", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShh 2019-10-13T03:58:25.591 INFO:teuthology.task.ansible.out:q6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "nvme0n1": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["97d94574-b350-4896-9285-2998fb12b1c3"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "labels": [], "masters": [], "uuids": ["97d94574-b350-4896-9285-2998fb12b1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 G 2019-10-13T03:58:25.592 INFO:teuthology.task.ansible.out:B", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0. 2019-10-13T03:58:25.594 INFO:teuthology.task.ansible.out:00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contr 2019-10-13T03:58:25.595 INFO:teuthology.task.ansible.out:oller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corpo 2019-10-13T03:58:25.596 INFO:teuthology.task.ansible.out:ration C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941a36"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filte 2019-10-13T03:58:25.598 INFO:teuthology.task.ansible.out:r": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offl 2019-10-13T03:58:25.599 INFO:teuthology.task.ansible.out:oad": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_u 2019-10-13T03:58:25.601 INFO:teuthology.task.ansible.out:dp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:77:aa", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": " 2019-10-13T03:58:25.602 INFO:teuthology.task.ansible.out:on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.204", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:77ab", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:77:ab", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", 2019-10-13T03:58:25.603 INFO:teuthology.task.ansible.out:"SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-gtmjzlnprotoqyparbazhcuozeccctiw ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_w 2019-10-13T03:58:25.605 INFO:teuthology.task.ansible.out:wn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "eno1", "lo"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", " 2019-10-13T03:58:25.606 INFO:teuthology.task.ansible.out:tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31467, "ansible_memory_mb": {"nocache": {"free": 31781, "used": 273}, "real": {"free": 31467, "total": 32054, "used": 587}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi204.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmd 2019-10-13T03:58:25.608 INFO:teuthology.task.ansible.out:line": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928378", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9362A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANZkQ/ZjhsiZKZJ09/tZnoH/6R0D4QJ5NQhL1jbow5ed/pOt7drfgTwItQg8DZZcVmQ0oB0GTLEAB9GUTqAu4oeILnlG8+miuIHxpqa400POaQLxTZ7jdgSWsqGLbR0uChhK8hJlq0DhqiwsJ4gHrdTYWzCt8B4cwFjeUthoLmm9AAAAFQCn01UobLOvI49K43i8/4SsM2S1owAAAIEAo2GacMRcgogjKFMTo7RDCqun8to4Fhx7zZc3FTeh7NMDGc9XsXgneU6KLJfqQYej85ja1JB/hW6GjcDUltewshD+i/jZzgRQ0yDd31iJTwsx+WPD70l7BRaO3mclp+t34JgA5/RRUdfyYze2V2zGbrEdoa4h4vSNS8u240nqsMUAAACBAMCVr2m0CHX3LmmMMd2eIunMpsoJvQ3Kd2oDfKCe1vcTsq 2019-10-13T03:58:25.609 INFO:teuthology.task.ansible.out:tIttZGzJr8uonWoGg1xMStQhamlSLfrT6EUDPtKwgpq2gTkxA786YSok2SXx2NRkXTD6QmpsxZGSjmooTDqA9QwQY0C5mHdwnGhUnsPmrt939u36g9HUB0qySY52OU", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLeCHyOBMMrKPsWWbrq7RDY+H50DIbYQYrPEf49A9p2agwRvrDkjxARHcYmIZ7UfBmiQkX9PUysIJcW2YwhKxUE=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIF3p/i/KhhpkGcL6P1N34frWMb5FUFYA73wYIwyexeS2", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDTlOcAA/Jqcpb/OnkOrrCmLB5vmc9IBTCJhMmorSjsCMf8qQOidO511p0AH330PF9U0wABWhkxY81XHPtDnTJ9s2n6kflGm2uj3F+qPocj4Z9UJ8XSUfLRtgBVWjcnyksNai26fo+xBDZK1cFxxDJp+T+L8FlQ9GUInuU0fmz3ohjpGL4WQFSfge5s4qFziOXMUqk8nONiddfXPZNJ/mvS1GYWcoG+aCTQ1Pqf+BfzINGwzYYpdJFqVCTS13iw2l3e/p7Hh8UeY2u/MCrE/lwCD1siKnYkToTJ7LPRbLrb7A0faOD0RUK0owHIwv1+owJ3ZfArM6LqjQKj7DFb6DF3", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 68, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": 2019-10-13T03:58:25.611 INFO:teuthology.task.ansible.out: "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T03:58:25.687 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.830) 0:00:08.174 ******** 2019-10-13T03:58:25.710 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.110"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:cc35"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-13", "day": "13", "epoch": "1570939105", "hour": "03", "iso8601": "2019-10-13T03:58:25Z", "iso8601_basic": "20191013T035825578809", "iso8601_basic_short": "20191013T035825", "iso8601_micro": "2019-10-13T03:58:25.578910Z", "minute": "58", "month": "10", "second": "25", "time": "03:58:25", "tz": "UTC", "tz_offset": "+0000", "weekday": "Sunday", "weekday_number": "0", "weeknumber": "40", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.110", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cc:35", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:cc35", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cc:35", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGl 2019-10-13T03:58:25.712 INFO:teuthology.task.ansible.out:Gre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "nvme0n1": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["40030898-e0dc-4263-932e-a36226be20b4"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "labels": [], "masters": [], "uuids": ["40030898-e0dc-4263-932e-a36226be20b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB 2019-10-13T03:58:25.713 INFO:teuthology.task.ansible.out:", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.0 2019-10-13T03:58:25.715 INFO:teuthology.task.ansible.out:0 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contro 2019-10-13T03:58:25.716 INFO:teuthology.task.ansible.out:ller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091dfc3c1"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "an 2019-10-13T03:58:25.718 INFO:teuthology.task.ansible.out:sible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:db:14", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "pr 2019-10-13T03:58:25.719 INFO:teuthology.task.ansible.out:omisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ip 2019-10-13T03:58:25.721 INFO:teuthology.task.ansible.out:xip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:db:15", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ip 2019-10-13T03:58:25.723 INFO:teuthology.task.ansible.out:xip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:cc:34", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx 2019-10-13T03:58:25.724 INFO:teuthology.task.ansible.out:_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.110", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:cc35", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8f:cc:35", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-fvznnlqqmwocgbfhakrszyeldluonnge ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi110.front.sepia.ceph.com", "ansible_hostname": "smithi110", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "lo", "eno1"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-gener 2019-10-13T03:58:25.725 INFO:teuthology.task.ansible.out:ic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broad 2019-10-13T03:58:25.728 INFO:teuthology.task.ansible.out:cast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31467, "ansible_memory_mb": {"nocache": {"free": 31782, "used": 272}, "real": {"free": 31467, "total": 32054, "used": 587}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440802, "block_size": 4096, "block_total": 240095108, "block_used": 13654306, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501524992, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi110.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) 2019-10-13T03:58:25.729 INFO:teuthology.task.ansible.out: Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731578", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8DB14", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAOUqhx+sV7P9h30R/09UDHtDtHrqhhWGaLNDlHnHksKhW7Ot7Wp5fpSTOIKceyLjb3cMDByrqCuVcezjWcjF4fpolqsZQTcOdIdGZa56XTl6CIMMBVFI/6G7AUTSDCbGjMKd350N3+GQCtVYKFBDNL/T80jCUbENe2MzmsHZImJLAAAAFQDVGz7784bagR+m2cSmjLM54AHr/QAAAIEAiULyaTxt8L95VLVHhxj3BW9dDwwuXF/yibygV//cCycb3PgmzZR5lWCFv6iOsR/QFA0Jwk9ARVnhSHQi9gWHucYUR1LFtysvx6tMNOL5w8gRHfbuh/Q3lR2SNdTEmcebsrvumxlaR7UKqH+8oQKZoqIrfCKgr504am0wxX8RWaAAAACAb7eDNMYH6mc39b8Sxu+tfDihb792g9s0kx+QliZFpBwapgcJa2101q5soov0aIkwWRwjTflhclv5RTTF4SWyONbExgZqVD7vPtwBehx9eVmL92Op7j0Hl4zL/elj4jJBYe4LVofPL1PQZCQCjdjsbicicM1pRJ4QCBKhuMVhrGs=", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMVXusGKTYEhqZY0lmW+nL1LOT4H2nPGwRSoNVf2S+WQETHog95SE4W205BeVOcVOdzznqMfawnr4JRM4 2019-10-13T03:58:25.731 INFO:teuthology.task.ansible.out:oS/5KU=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIENNTfzaxZPXp5G3HQ8Me0aHgYmlQ/OP1D087AlS9r4I", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQD6EdIEDzZRdBRD0bLwvaqt1T2CWfYA4jgooF37gfsxlmNU8R7fsTcSSy4aIzep3lsgjtvbgfSe9phgjthGuFZn3BvGAZEQRt/PN2Jo+QAspYO/VWjU2DgRKNGwpPsJ9KplGabY0l3aAMPJqj1zySLSZ3fITcyCmFX2mnB1AeQcTaX50JpSiyxr3UZ2pRumFxhCb9aAmDZnLN3IsqEKZ83bMyRrXPG15KSyxCVHlSonFn6Ww2aO+jPB1W9RrJkJZdvzbEKZKOgd5YDA1jPltKKeWHYOfE2SKF1xNOFnyUut4UhQ5DbBK9yrkOa5yO0ksPW0LmKTM5UXCtvV7eQc8dRZ", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 64, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-13T03:58:25.735 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:25.807 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.120) 0:00:08.295 ******** 2019-10-13T03:58:25.853 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.045) 0:00:08.340 ******** 2019-10-13T03:58:25.858 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-13T03:58:25.860 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:25.899 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.046) 0:00:08.387 ******** 2019-10-13T03:58:25.905 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-13T03:58:25.907 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:25.946 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.046) 0:00:08.434 ******** 2019-10-13T03:58:25.952 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:25.998 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:25 +0000 (0:00:00.051) 0:00:08.485 ******** 2019-10-13T03:58:26.003 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-13T03:58:26.004 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:26.043 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.045) 0:00:08.530 ******** 2019-10-13T03:58:26.050 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:26.094 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.050) 0:00:08.581 ******** 2019-10-13T03:58:26.098 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-13T03:58:26.100 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:26.140 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.046) 0:00:08.627 ******** 2019-10-13T03:58:26.177 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-13T03:58:26.179 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:26.224 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.084) 0:00:08.711 ******** 2019-10-13T03:58:26.230 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-10-13T03:58:26.231 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:58:26.272 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.047) 0:00:08.759 ******** 2019-10-13T03:58:26.352 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:58:26.420 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:26 +0000 (0:00:00.148) 0:00:08.907 ******** 2019-10-13T03:58:26.459 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T03:58:26.460 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T03:58:26.593 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-13T03:58:26.616 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-13T03:58:26.754 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T03:58:26.886 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-13T03:58:26.921 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T03:58:27.016 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-13T03:58:27.059 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T03:58:27.132 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-13T03:58:27.185 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T03:58:27.260 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-13T03:58:27.336 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T03:58:27.401 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-13T03:58:27.475 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T03:58:27.530 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-13T03:58:27.639 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T03:58:27.661 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-13T03:58:27.789 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T03:58:27.791 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-13T03:58:27.917 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T03:58:27.949 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-13T03:58:28.056 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T03:58:28.087 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-13T03:58:28.184 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T03:58:28.227 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-13T03:58:28.323 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T03:58:28.357 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-13T03:58:28.467 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T03:58:28.500 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-13T03:58:28.599 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T03:58:28.642 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-13T03:58:28.726 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T03:58:28.800 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-13T03:58:28.866 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T03:58:28.944 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-13T03:58:28.999 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T03:58:29.094 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-13T03:58:29.139 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T03:58:29.238 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-13T03:58:29.283 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T03:58:29.375 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-13T03:58:29.408 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T03:58:29.523 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-13T03:58:29.619 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T03:58:29.725 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-13T03:58:29.801 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T03:58:29.879 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-13T03:58:29.922 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T03:58:30.031 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-13T03:58:30.075 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T03:58:30.170 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-13T03:58:30.193 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T03:58:30.319 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-13T03:58:30.344 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T03:58:30.449 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-13T03:58:30.474 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T03:58:30.599 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-13T03:58:30.601 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T03:58:30.729 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T03:58:30.762 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-13T03:58:30.859 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T03:58:30.909 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-13T03:58:30.991 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T03:58:31.055 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-13T03:58:31.121 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T03:58:31.204 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-13T03:58:31.247 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T03:58:31.345 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-13T03:58:31.368 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T03:58:31.477 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-13T03:58:31.510 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T03:58:31.627 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-13T03:58:31.628 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T03:58:31.764 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-13T03:58:31.765 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T03:58:31.898 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-13T03:58:31.911 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T03:58:32.046 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T03:58:32.047 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-13T03:58:32.177 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T03:58:32.201 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-13T03:58:32.319 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T03:58:32.343 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-13T03:58:32.449 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T03:58:32.482 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T03:58:32.588 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T03:58:32.621 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-13T03:58:32.731 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T03:58:32.774 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-13T03:58:32.849 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T03:58:32.913 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-13T03:58:32.978 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T03:58:33.073 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-13T03:58:33.116 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T03:58:33.228 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-13T03:58:33.261 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T03:58:33.365 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-13T03:58:33.388 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T03:58:33.514 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-13T03:58:33.525 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-13T03:58:33.527 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:33.646 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T03:58:33.648 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:33.673 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-13T03:58:33.792 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T03:58:33.826 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-13T03:58:33.932 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T03:58:33.976 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-13T03:58:34.061 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T03:58:34.094 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-13T03:58:34.179 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T03:58:34.247 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-13T03:58:34.312 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-13T03:58:34.387 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-13T03:58:34.441 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T03:58:34.537 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-13T03:58:34.570 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T03:58:34.688 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-13T03:58:34.699 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T03:58:34.827 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T03:58:34.829 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-13T03:58:34.956 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T03:58:34.957 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-13T03:58:35.079 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T03:58:35.101 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-13T03:58:35.206 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T03:58:35.249 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-13T03:58:35.337 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T03:58:35.401 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-13T03:58:35.476 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T03:58:35.540 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-13T03:58:35.593 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T03:58:35.709 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-13T03:58:35.742 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T03:58:35.851 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-13T03:58:35.863 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-13T03:58:36.008 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-13T03:58:36.135 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-13T03:58:36.148 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-13T03:58:36.273 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-13T03:58:36.296 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-13T03:58:36.414 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-13T03:58:36.457 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-13T03:58:36.545 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-13T03:58:36.619 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-13T03:58:36.680 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T03:58:36.781 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-13T03:58:36.814 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-13T03:58:36.940 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-13T03:58:36.952 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-13T03:58:37.070 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T03:58:37.092 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-13T03:58:37.211 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T03:58:37.251 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-13T03:58:37.252 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:37.337 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T03:58:37.382 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-13T03:58:37.486 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T03:58:37.519 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-13T03:58:37.617 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T03:58:37.683 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-13T03:58:37.757 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T03:58:37.820 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-13T03:58:37.885 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T03:58:37.992 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-13T03:58:38.025 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-13T03:58:38.125 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-13T03:58:38.127 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:38.159 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T03:58:38.269 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-13T03:58:38.293 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T03:58:38.421 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-13T03:58:38.423 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T03:58:38.560 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-13T03:58:38.562 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T03:58:38.700 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T03:58:38.712 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-13T03:58:38.819 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T03:58:38.852 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-13T03:58:38.959 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T03:58:39.013 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-13T03:58:39.097 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T03:58:39.151 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-13T03:58:39.229 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T03:58:39.284 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-13T03:58:39.355 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T03:58:39.434 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-13T03:58:39.479 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T03:58:39.480 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:39.565 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-13T03:58:39.618 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T03:58:39.712 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-13T03:58:39.745 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T03:58:39.840 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-13T03:58:39.883 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T03:58:39.979 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-13T03:58:40.027 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T03:58:40.120 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-13T03:58:40.152 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T03:58:40.264 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-13T03:58:40.286 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T03:58:40.446 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-13T03:58:40.458 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T03:58:40.648 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T03:58:40.671 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-13T03:58:40.842 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T03:58:40.876 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-13T03:58:40.972 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T03:58:41.035 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-13T03:58:41.112 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T03:58:41.177 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-13T03:58:41.238 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T03:58:41.334 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-13T03:58:41.496 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-13T03:58:41.632 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-13T03:58:41.779 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-13T03:58:41.857 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-13T03:58:41.937 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-13T03:58:42.075 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-13T03:58:42.218 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-13T03:58:42.369 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-13T03:58:42.424 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-13T03:58:42.509 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-13T03:58:42.854 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-13T03:58:43.084 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-13T03:58:43.376 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-13T03:58:43.537 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-13T03:58:43.909 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-13T03:58:44.078 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-13T03:58:44.373 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-13T03:58:44.508 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:44 +0000 (0:00:18.088) 0:00:26.996 ******** 2019-10-13T03:58:44.515 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-13T03:58:44.531 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T03:58:44.582 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:44 +0000 (0:00:00.068) 0:00:27.064 ******** 2019-10-13T03:58:44.626 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T03:58:44.636 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T03:58:44.637 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T03:58:44.639 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T03:58:44.640 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T03:58:44.642 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T03:58:44.643 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364 2019-10-13T03:58:44.644 INFO:teuthology.task.ansible.out:b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin 2019-10-13T03:58:44.646 INFO:teuthology.task.ansible.out:@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cC 2019-10-13T03:58:44.647 INFO:teuthology.task.ansible.out:x1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e 2019-10-13T03:58:44.649 INFO:teuthology.task.ansible.out:0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116 2019-10-13T03:58:44.652 INFO:teuthology.task.ansible.out:f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d71 2019-10-13T03:58:44.654 INFO:teuthology.task.ansible.out:56c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T03:58:44.693 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-10-13T03:58:44.695 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:44 +0000 (0:00:00.116) 0:00:27.180 ******** 2019-10-13T03:58:45.040 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-13T03:58:45.041 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-13T03:58:45.525 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-13T03:58:45.642 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:58:45.682 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-10-13T03:58:45.682 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T03:58:45.749 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:45 +0000 (0:00:01.055) 0:00:28.236 ******** 2019-10-13T03:58:45.803 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:45 +0000 (0:00:00.053) 0:00:28.290 ******** 2019-10-13T03:58:45.833 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-13T03:58:45.877 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:45 +0000 (0:00:00.074) 0:00:28.364 ******** 2019-10-13T03:58:45.941 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-13T03:58:45.944 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-13T03:58:45.945 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-13T03:58:45.947 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-13T03:58:45.948 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-13T03:58:45.950 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-13T03:58:45.951 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwa 2019-10-13T03:58:45.956 INFO:teuthology.task.ansible.out:tkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 86 2019-10-13T03:58:45.958 INFO:teuthology.task.ansible.out:8a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de37 2019-10-13T03:58:45.959 INFO:teuthology.task.ansible.out:4405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841 2019-10-13T03:58:45.960 INFO:teuthology.task.ansible.out:a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb45 2019-10-13T03:58:45.962 INFO:teuthology.task.ansible.out:0ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfq 2019-10-13T03:58:45.963 INFO:teuthology.task.ansible.out:Q 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "s 2019-10-13T03:58:45.965 INFO:teuthology.task.ansible.out:koduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-13T03:58:45.974 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:58:45.975 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T03:58:46.016 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:46 +0000 (0:00:00.138) 0:00:28.503 ******** 2019-10-13T03:58:46.073 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T03:58:46.237 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T03:58:46.371 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T03:58:46.500 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T03:58:46.502 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T03:58:46.638 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T03:58:46.664 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:58:46.675 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi204.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-13T03:58:46.717 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:58:46 +0000 (0:00:00.701) 0:00:29.205 ******** 2019-10-13T03:58:46.797 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKS 2019-10-13T03:58:46.799 INFO:teuthology.task.ansible.out:zOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T03:58:46.911 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-13T03:58:46.925 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T03:58:47.065 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-13T03:58:47.066 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-13T03:58:47.088 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:47.216 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-13T03:58:47.228 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T03:58:47.230 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:47.240 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:47.251 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:47.252 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:47.383 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-13T03:58:47.385 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T03:58:47.509 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T03:58:47.511 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T03:58:47.521 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-13T03:58:47.522 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-13T03:58:47.635 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T03:58:47.657 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-13T03:58:47.777 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T03:58:47.817 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-13T03:58:47.821 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-13T03:58:47.914 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T03:58:47.956 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-13T03:58:48.042 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T03:58:48.114 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-13T03:58:48.125 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:48.178 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T03:58:48.271 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-13T03:58:48.273 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-13T03:58:48.283 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:48.293 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:48.305 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T03:58:48.316 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:48.327 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:58:48.453 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T03:58:48.454 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T03:58:48.464 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-13T03:58:48.612 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T03:58:48.616 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T03:58:48.634 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-13T03:58:48.635 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-13T03:58:48.732 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T03:58:48.790 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-13T03:58:48.854 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T03:58:48.939 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-13T03:58:48.992 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T03:58:48.994 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T03:58:49.107 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-13T03:58:49.108 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T03:58:49.240 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T03:58:49.262 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-13T03:58:49.378 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-13T03:58:49.420 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-13T03:58:49.515 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T03:58:49.568 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-13T03:58:49.649 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T03:58:49.724 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-13T03:58:49.725 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-13T03:58:49.780 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T03:58:49.782 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T03:58:49.865 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-13T03:58:49.867 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-13T03:58:49.936 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T03:58:49.938 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T03:58:50.005 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-13T03:58:50.056 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T03:58:50.140 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-13T03:58:50.189 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T03:58:50.313 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T03:58:50.314 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-13T03:58:50.316 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-13T03:58:50.444 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T03:58:50.464 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-13T03:58:50.596 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-13T03:58:50.597 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T03:58:50.733 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T03:58:50.745 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-13T03:58:50.881 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-13T03:58:50.892 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T03:58:51.034 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T03:58:51.038 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-13T03:58:51.166 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T03:58:51.180 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-13T03:58:51.181 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-13T03:58:51.306 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T03:58:51.327 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-13T03:58:51.329 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-13T03:58:51.463 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T03:58:51.464 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T03:58:51.495 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-13T03:58:51.610 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T03:58:51.612 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T03:58:51.657 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-13T03:58:51.752 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T03:58:51.797 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-13T03:58:51.882 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T03:58:51.939 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-13T03:58:52.014 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T03:58:52.080 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-13T03:58:52.133 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T03:58:52.242 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-13T03:58:52.266 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T03:58:52.360 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T03:58:52.416 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T03:58:52.510 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-13T03:58:52.548 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T03:58:52.653 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-13T03:58:52.693 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T03:58:52.798 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-13T03:58:52.831 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T03:58:52.832 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T03:58:52.834 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T03:58:52.947 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-13T03:58:52.949 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-13T03:58:52.986 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T03:58:53.081 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-13T03:58:53.082 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-13T03:58:53.113 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T03:58:53.229 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-13T03:58:53.240 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T03:58:53.368 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T03:58:53.370 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T03:58:53.390 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-13T03:58:53.496 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T03:58:53.528 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-13T03:58:53.633 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T03:58:53.666 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-13T03:58:53.749 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T03:58:53.792 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-13T03:58:53.886 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T03:58:53.949 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-13T03:58:54.017 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T03:58:54.112 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-13T03:58:54.148 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T03:58:54.255 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-13T03:58:54.288 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T03:58:54.289 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T03:58:54.404 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-13T03:58:54.406 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-13T03:58:54.408 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-13T03:58:54.419 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T03:58:54.533 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-13T03:58:54.566 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T03:58:54.664 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-13T03:58:54.696 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T03:58:54.698 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T03:58:54.822 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-13T03:58:54.833 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T03:58:54.950 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-13T03:58:54.952 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-13T03:58:54.983 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T03:58:55.098 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-13T03:58:55.110 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T03:58:55.111 INFO:teuthology.task.ansible.out: 2019-10-13T03:58:55.218 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-13T03:58:55.251 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T03:58:55.369 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-13T03:58:55.388 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T03:58:55.507 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-13T03:58:55.510 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T03:58:55.635 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-13T03:58:55.635 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T03:58:55.784 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T03:58:55.785 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-13T03:58:55.914 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T03:58:55.947 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-13T03:58:55.948 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-13T03:58:56.068 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T03:58:56.110 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-13T03:58:56.197 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T03:58:56.270 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-13T03:58:56.323 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T03:58:56.423 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-13T03:58:56.425 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-13T03:58:56.465 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T03:58:56.550 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-13T03:58:56.603 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T03:58:56.687 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-13T03:58:56.740 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T03:58:56.826 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-13T03:58:56.891 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T03:58:56.967 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-13T03:58:57.012 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T03:58:57.086 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-13T03:58:57.139 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T03:58:57.212 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-13T03:58:57.264 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T03:58:57.350 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-13T03:58:57.407 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T03:58:57.523 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-13T03:58:57.528 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T03:58:57.636 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-13T03:58:57.659 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T03:58:57.776 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-13T03:58:57.798 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T03:58:57.924 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T03:58:57.925 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-13T03:58:58.048 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T03:58:58.070 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-13T03:58:58.184 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T03:58:58.207 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-13T03:58:58.307 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-13T03:58:58.360 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-13T03:58:58.436 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T03:58:58.498 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-13T03:58:58.564 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T03:58:58.633 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-13T03:58:58.692 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-13T03:58:58.694 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T03:58:58.766 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-13T03:58:58.829 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T03:58:58.831 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T03:58:58.892 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-13T03:58:58.955 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T03:58:59.010 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-13T03:58:59.083 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T03:58:59.157 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-13T03:58:59.216 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T03:58:59.303 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-13T03:58:59.367 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T03:58:59.465 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-13T03:58:59.487 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T03:58:59.613 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T03:58:59.614 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-13T03:58:59.739 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-13T03:58:59.753 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T03:58:59.880 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-13T03:58:59.913 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-13T03:59:00.051 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-13T03:59:00.062 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-13T03:59:00.187 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T03:59:00.332 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-13T03:59:00.333 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T03:59:00.466 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-13T03:59:00.489 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T03:59:00.633 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-13T03:59:00.644 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-13T03:59:00.645 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T03:59:00.781 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T03:59:00.884 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-13T03:59:00.886 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-13T03:59:00.929 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:00 +0000 (0:00:14.211) 0:00:43.416 ******** 2019-10-13T03:59:00.971 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:00.973 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:00.983 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:00.995 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:00.995 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.006 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.022 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:01.022 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-13T03:59:01.023 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.025 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.036 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.161 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:01.163 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-13T03:59:01.183 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:01.184 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-13T03:59:01.195 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.295 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:01.295 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-13T03:59:01.329 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T03:59:01.447 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:01.449 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-13T03:59:01.481 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:01.482 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T03:59:01.484 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:01.633 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T03:59:01.695 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-13T03:59:01.795 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:01.797 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T03:59:01.806 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.808 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.819 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.820 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.832 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.833 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-13T03:59:01.835 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.846 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.846 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.857 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.870 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.871 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.881 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.882 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.894 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.895 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.906 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.908 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.919 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.920 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.933 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.934 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.946 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.946 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.958 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.959 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.970 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.971 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.984 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.985 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:01.986 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-13T03:59:01.988 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:01.999 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.010 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.011 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.022 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.023 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.042 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.065 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.067 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.068 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.069 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.081 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.081 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.083 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.095 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.108 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.110 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.111 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.123 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.124 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.126 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:02.139 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.140 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:02.141 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.152 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.153 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.164 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.176 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.177 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.189 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.190 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.201 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.212 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.213 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.224 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.225 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.241 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.243 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.244 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.254 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.267 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.268 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.279 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.292 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.294 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.306 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:02.307 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.318 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.329 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.341 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.352 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.363 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.374 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.375 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.386 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.397 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.407 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.419 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.430 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.430 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.446 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.452 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.464 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.465 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-13T03:59:02.467 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.477 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.478 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.496 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.500 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.511 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.512 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.523 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.524 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.534 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.535 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.654 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:02 +0000 (0:00:01.725) 0:00:45.142 ******** 2019-10-13T03:59:02.661 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:02.662 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-13T03:59:02.684 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-10-13T03:59:02.686 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:02.728 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:02 +0000 (0:00:00.072) 0:00:45.214 ******** 2019-10-13T03:59:02.732 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:02.733 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-13T03:59:02.886 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:02.887 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T03:59:02.909 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:02.911 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-13T03:59:02.915 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:03.040 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T03:59:03.057 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-13T03:59:03.065 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-13T03:59:03.161 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:03.163 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T03:59:03.204 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:03 +0000 (0:00:00.476) 0:00:45.690 ******** 2019-10-13T03:59:03.244 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.246 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.257 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.258 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.269 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.285 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.286 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.287 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.299 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T03:59:03.418 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T03:59:03.429 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.431 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf59 2019-10-13T03:59:03.432 INFO:teuthology.task.ansible.out:2607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:03.545 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:03.546 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T03:59:03.567 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.568 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-13T03:59:03.663 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:03.665 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T03:59:03.706 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.707 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-13T03:59:03.781 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:03.782 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T03:59:03.855 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.857 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-13T03:59:03.910 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:03.911 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T03:59:03.991 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:03.992 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-13T03:59:03.994 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.005 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.006 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.017 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.039 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.040 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.042 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T03:59:04.043 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.054 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.054 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.067 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.076 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.077 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.088 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.088 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.099 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.100 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.111 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.123 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.133 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.134 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.146 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.146 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.158 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.169 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.170 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:04.171 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.172 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.182 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.193 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.204 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.205 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.216 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.217 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.228 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.240 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.240 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.258 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.264 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.276 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.280 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.287 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.288 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.299 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.300 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:04.302 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.313 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.315 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.325 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.336 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.347 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.349 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.359 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.360 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.371 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.373 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.383 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.385 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.395 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.406 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.408 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.418 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.419 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.433 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi110.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-13T03:59:04.435 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.445 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.458 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.459 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.469 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.471 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.482 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.494 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.495 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.497 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.517 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.519 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.520 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.531 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.531 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.542 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.543 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:04.544 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.555 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.556 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.569 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.578 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.579 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.580 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.596 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.604 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.626 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.627 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.638 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.638 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.649 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.650 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.661 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.671 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.672 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.683 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:04.685 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.686 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.757 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:04 +0000 (0:00:01.554) 0:00:47.245 ******** 2019-10-13T03:59:04.784 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-10-13T03:59:04.786 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:04.827 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:04 +0000 (0:00:00.070) 0:00:47.315 ******** 2019-10-13T03:59:04.833 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:04.836 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T03:59:04.928 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T03:59:05.013 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-13T03:59:05.067 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T03:59:05.152 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-13T03:59:05.205 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T03:59:05.279 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-13T03:59:05.332 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-13T03:59:05.415 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-13T03:59:05.451 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T03:59:05.546 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-13T03:59:05.588 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T03:59:05.679 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-13T03:59:05.712 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T03:59:05.830 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-13T03:59:05.842 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T03:59:05.960 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-13T03:59:05.972 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T03:59:06.101 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T03:59:06.105 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-13T03:59:06.225 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T03:59:06.226 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-13T03:59:06.354 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T03:59:06.378 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-13T03:59:06.417 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:01.589) 0:00:48.905 ******** 2019-10-13T03:59:06.447 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T03:59:06.449 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => { "msg": "Host smithi110.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-13T03:59:06.492 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.074) 0:00:48.979 ******** 2019-10-13T03:59:06.500 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-13T03:59:06.522 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T03:59:06.524 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-13T03:59:06.563 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.071) 0:00:49.051 ******** 2019-10-13T03:59:06.580 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:06.581 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-13T03:59:06.592 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T03:59:06.593 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:06.640 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.074) 0:00:49.125 ******** 2019-10-13T03:59:06.668 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-13T03:59:06.670 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:06.711 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.073) 0:00:49.198 ******** 2019-10-13T03:59:06.717 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:06.718 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-13T03:59:06.741 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T03:59:06.742 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:06.799 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.087) 0:00:49.286 ******** 2019-10-13T03:59:06.827 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T03:59:06.828 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:06.889 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.090) 0:00:49.376 ******** 2019-10-13T03:59:06.895 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:06.896 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-13T03:59:06.908 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-13T03:59:06.909 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:06.949 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:06 +0000 (0:00:00.060) 0:00:49.436 ******** 2019-10-13T03:59:06.976 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:06.977 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-13T03:59:06.988 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-13T03:59:06.989 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.029 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.079) 0:00:49.516 ******** 2019-10-13T03:59:07.056 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T03:59:07.058 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.117 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.088) 0:00:49.605 ******** 2019-10-13T03:59:07.124 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:07.124 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-13T03:59:07.146 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T03:59:07.148 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.194 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.076) 0:00:49.681 ******** 2019-10-13T03:59:07.224 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-13T03:59:07.225 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:07.273 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.078) 0:00:49.760 ******** 2019-10-13T03:59:07.279 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:07.280 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-13T03:59:07.302 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T03:59:07.303 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.357 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.076) 0:00:49.837 ******** 2019-10-13T03:59:07.389 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-13T03:59:07.390 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.441 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.091) 0:00:49.928 ******** 2019-10-13T03:59:07.448 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-13T03:59:07.471 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-13T03:59:07.473 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:07.521 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.079) 0:00:50.008 ******** 2019-10-13T03:59:07.549 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T03:59:07.550 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.589 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.068) 0:00:50.077 ******** 2019-10-13T03:59:07.632 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-13T03:59:07.633 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:07.687 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.097) 0:00:50.175 ******** 2019-10-13T03:59:07.693 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:07.694 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-13T03:59:07.715 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-13T03:59:07.717 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.754 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.066) 0:00:50.241 ******** 2019-10-13T03:59:07.781 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.835 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.080) 0:00:50.322 ******** 2019-10-13T03:59:07.842 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:07.843 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-13T03:59:07.855 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.895 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.060) 0:00:50.382 ******** 2019-10-13T03:59:07.921 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T03:59:07.923 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:07.963 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:07 +0000 (0:00:00.068) 0:00:50.451 ******** 2019-10-13T03:59:07.969 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:07.971 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-13T03:59:07.994 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T03:59:07.995 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:08.032 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.068) 0:00:50.519 ******** 2019-10-13T03:59:08.058 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T03:59:08.060 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.113 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.081) 0:00:50.601 ******** 2019-10-13T03:59:08.119 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:08.120 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-13T03:59:08.152 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-13T03:59:08.154 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.199 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.085) 0:00:50.686 ******** 2019-10-13T03:59:08.227 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T03:59:08.229 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.267 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.068) 0:00:50.755 ******** 2019-10-13T03:59:08.276 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:08.277 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-13T03:59:08.299 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-13T03:59:08.300 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.338 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.070) 0:00:50.825 ******** 2019-10-13T03:59:08.364 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-13T03:59:08.365 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.403 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.065) 0:00:50.890 ******** 2019-10-13T03:59:08.409 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:08.409 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-13T03:59:08.421 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T03:59:08.422 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.460 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.056) 0:00:50.947 ******** 2019-10-13T03:59:08.461 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:08.497 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-13T03:59:08.499 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:08.541 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.081) 0:00:51.028 ******** 2019-10-13T03:59:08.547 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:08.548 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-13T03:59:08.560 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T03:59:08.601 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.060) 0:00:51.089 ******** 2019-10-13T03:59:08.628 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T03:59:08.630 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.673 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.071) 0:00:51.160 ******** 2019-10-13T03:59:08.675 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:08.687 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-13T03:59:08.689 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-13T03:59:08.711 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T03:59:08.714 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.752 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.078) 0:00:51.239 ******** 2019-10-13T03:59:08.789 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.036) 0:00:51.276 ******** 2019-10-13T03:59:08.794 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T03:59:08.795 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.797 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.840 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.051) 0:00:51.327 ******** 2019-10-13T03:59:08.845 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-13T03:59:08.846 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => { "msg": "Host smithi204.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-13T03:59:08.889 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.048) 0:00:51.376 ******** 2019-10-13T03:59:08.894 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T03:59:08.894 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.896 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:08.939 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.050) 0:00:51.427 ******** 2019-10-13T03:59:08.945 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-13T03:59:08.945 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-13T03:59:08.987 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:08 +0000 (0:00:00.047) 0:00:51.475 ******** 2019-10-13T03:59:08.992 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T03:59:08.993 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.034 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.046) 0:00:51.521 ******** 2019-10-13T03:59:09.038 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-13T03:59:09.039 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.080 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.046) 0:00:51.567 ******** 2019-10-13T03:59:09.106 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-13T03:59:09.108 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.147 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.067) 0:00:51.634 ******** 2019-10-13T03:59:09.174 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-13T03:59:09.175 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.219 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.071) 0:00:51.706 ******** 2019-10-13T03:59:09.245 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-13T03:59:09.246 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.300 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.071) 0:00:51.778 ******** TASK [common : Get the current timezone] *************************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.101213", "end": "2019-10-13 03:59:09.245445", "rc": 0, "start": "2019-10-13 03:59:09.144232", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T03:59:09.343 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.052) 0:00:51.830 ******** 2019-10-13T03:59:09.350 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-13T03:59:09.350 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.394 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.051) 0:00:51.882 ******** 2019-10-13T03:59:09.404 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.447 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.052) 0:00:51.935 ******** 2019-10-13T03:59:09.455 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.468 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:09.503 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.055) 0:00:51.990 ******** 2019-10-13T03:59:09.509 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-13T03:59:09.509 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.555 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.052) 0:00:52.043 ******** 2019-10-13T03:59:09.562 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-13T03:59:09.564 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.607 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.051) 0:00:52.094 ******** 2019-10-13T03:59:09.613 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T03:59:09.613 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.665 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.057) 0:00:52.152 ******** 2019-10-13T03:59:09.670 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-13T03:59:09.672 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.718 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.052) 0:00:52.205 ******** 2019-10-13T03:59:09.723 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-13T03:59:09.725 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.775 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.057) 0:00:52.263 ******** 2019-10-13T03:59:09.777 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:09.785 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-13T03:59:09.785 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:09.787 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:09.831 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.055) 0:00:52.319 ******** 2019-10-13T03:59:09.837 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-13T03:59:09.838 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.884 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.052) 0:00:52.371 ******** 2019-10-13T03:59:09.890 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-13T03:59:09.892 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.936 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.051) 0:00:52.423 ******** 2019-10-13T03:59:09.941 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-13T03:59:09.943 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:09.985 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:09 +0000 (0:00:00.049) 0:00:52.472 ******** 2019-10-13T03:59:09.991 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-13T03:59:09.992 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.036 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.050) 0:00:52.523 ******** 2019-10-13T03:59:10.041 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-13T03:59:10.043 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.098 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.060) 0:00:52.583 ******** 2019-10-13T03:59:10.102 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-13T03:59:10.104 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:10.149 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.053) 0:00:52.636 ******** 2019-10-13T03:59:10.175 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-13T03:59:10.176 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.216 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.066) 0:00:52.703 ******** 2019-10-13T03:59:10.242 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-13T03:59:10.243 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:10.282 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.066) 0:00:52.769 ******** 2019-10-13T03:59:10.312 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.381 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.098) 0:00:52.868 ******** 2019-10-13T03:59:10.407 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-10-13T03:59:10.408 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.451 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.069) 0:00:52.938 ******** 2019-10-13T03:59:10.479 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.527 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.076) 0:00:53.014 ******** 2019-10-13T03:59:10.532 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-10-13T03:59:10.533 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.303825", "end": "2019-10-13 03:59:10.511633", "rc": 0, "start": "2019-10-13 03:59:10.207808", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:10.578 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.051) 0:00:53.065 ******** 2019-10-13T03:59:10.585 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-13T03:59:10.586 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.644 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.065) 0:00:53.131 ******** 2019-10-13T03:59:10.650 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-13T03:59:10.654 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.694 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.049) 0:00:53.181 ******** 2019-10-13T03:59:10.699 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-13T03:59:10.701 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:10.742 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.048) 0:00:53.229 ******** 2019-10-13T03:59:10.768 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.827 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.085) 0:00:53.314 ******** 2019-10-13T03:59:10.868 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:10.912 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.084) 0:00:53.399 ******** 2019-10-13T03:59:10.917 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-13T03:59:10.918 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-10-13T03:59:10.919 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.067912", "end": "2019-10-13 03:59:10.874092", "rc": 0, "start": "2019-10-13 03:59:10.806180", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:10.967 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:10 +0000 (0:00:00.055) 0:00:53.455 ******** 2019-10-13T03:59:10.973 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-13T03:59:10.974 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.020 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.052) 0:00:53.507 ******** 2019-10-13T03:59:11.048 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.096 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.075) 0:00:53.583 ******** 2019-10-13T03:59:11.131 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.174 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.077) 0:00:53.661 ******** 2019-10-13T03:59:11.209 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-13T03:59:11.211 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.249 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.074) 0:00:53.736 ******** 2019-10-13T03:59:11.275 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-13T03:59:11.276 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:11.317 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.068) 0:00:53.805 ******** 2019-10-13T03:59:11.345 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-13T03:59:11.393 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.075) 0:00:53.881 ******** 2019-10-13T03:59:11.421 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-13T03:59:11.423 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.464 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.070) 0:00:53.951 ******** 2019-10-13T03:59:11.505 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-13T03:59:11.513 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.557 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.092) 0:00:54.044 ******** 2019-10-13T03:59:11.605 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-13T03:59:11.606 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.617 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.675 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.117) 0:00:54.162 ******** 2019-10-13T03:59:11.703 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-13T03:59:11.705 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.716 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.760 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.084) 0:00:54.247 ******** 2019-10-13T03:59:11.789 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-13T03:59:11.792 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:11.856 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:11 +0000 (0:00:00.096) 0:00:54.344 ******** 2019-10-13T03:59:12.102 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T03:59:12.103 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.101974", "end": "2019-10-13 03:59:12.368103", "rc": 0, "start": "2019-10-13 03:59:12.266129", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-13T03:59:12.144 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.287) 0:00:54.631 ******** 2019-10-13T03:59:12.170 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-13T03:59:12.172 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.220 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.076) 0:00:54.708 ******** 2019-10-13T03:59:12.242 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:12.246 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-13T03:59:12.248 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.292 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.067) 0:00:54.775 ******** 2019-10-13T03:59:12.314 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-13T03:59:12.315 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.354 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.066) 0:00:54.841 ******** 2019-10-13T03:59:12.382 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-13T03:59:12.384 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.425 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.070) 0:00:54.912 ******** 2019-10-13T03:59:12.452 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-13T03:59:12.453 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.499 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.074) 0:00:54.986 ******** 2019-10-13T03:59:12.525 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-13T03:59:12.527 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.568 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.068) 0:00:55.055 ******** 2019-10-13T03:59:12.594 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-13T03:59:12.596 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:12.639 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:12 +0000 (0:00:00.068) 0:00:55.124 ******** 2019-10-13T03:59:13.361 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.565528", "end": "2019-10-13 03:59:13.625103", "rc": 0, "start": "2019-10-13 03:59:13.059575", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:13.410 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:13 +0000 (0:00:00.773) 0:00:55.898 ******** 2019-10-13T03:59:13.555 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:13.612 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:13 +0000 (0:00:00.201) 0:00:56.099 ******** 2019-10-13T03:59:13.613 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:14.052 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-13T03:59:14.053 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.068294", "end": "2019-10-13 03:59:14.222383", "rc": 0, "start": "2019-10-13 03:59:14.154089", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:14.090 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:14 +0000 (0:00:00.478) 0:00:56.578 ******** 2019-10-13T03:59:14.763 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:14.772 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-13T03:59:14.774 INFO:teuthology.task.ansible.out: [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T03:59:14.814 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:14 +0000 (0:00:00.723) 0:00:57.301 ******** 2019-10-13T03:59:15.649 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T03:59:15.703 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:15 +0000 (0:00:00.889) 0:00:58.190 ******** 2019-10-13T03:59:15.731 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-13T03:59:15.732 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:15.781 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:15 +0000 (0:00:00.077) 0:00:58.268 ******** 2019-10-13T03:59:16.277 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-13T03:59:16.311 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:16 +0000 (0:00:00.530) 0:00:58.798 ******** 2019-10-13T03:59:17.110 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T03:59:17.151 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:17 +0000 (0:00:00.839) 0:00:59.638 ******** 2019-10-13T03:59:17.495 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:17.499 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-13T03:59:17.500 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T03:59:17.551 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:17 +0000 (0:00:00.399) 0:01:00.038 ******** 2019-10-13T03:59:17.556 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T03:59:17.557 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T03:59:17.872 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T03:59:18.177 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-13T03:59:18.221 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:18 +0000 (0:00:00.669) 0:01:00.708 ******** 2019-10-13T03:59:18.348 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-13T03:59:18.349 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T03:59:18.398 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:18 +0000 (0:00:00.177) 0:01:00.885 ******** 2019-10-13T03:59:18.405 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-13T03:59:18.407 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T03:59:18.455 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:18 +0000 (0:00:00.056) 0:01:00.942 ******** 2019-10-13T03:59:18.492 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-13T03:59:18.494 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:18.538 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:18 +0000 (0:00:00.083) 0:01:01.026 ******** 2019-10-13T03:59:18.713 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T03:59:18.714 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-13T03:59:18.716 INFO:teuthology.task.ansible.out: 2019-10-13T03:59:18.889 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-10-13T03:59:18.890 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-13T03:59:18.929 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:18 +0000 (0:00:00.390) 0:01:01.416 ******** 2019-10-13T03:59:19.017 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T03:59:19.017 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-13T03:59:19.295 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-13T03:59:19.338 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.408) 0:01:01.825 ******** 2019-10-13T03:59:19.376 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-13T03:59:19.378 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-13T03:59:19.416 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.078) 0:01:01.904 ******** 2019-10-13T03:59:19.463 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-13T03:59:19.464 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.503 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.086) 0:01:01.990 ******** 2019-10-13T03:59:19.530 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:19.571 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.067) 0:01:02.058 ******** 2019-10-13T03:59:19.609 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-13T03:59:19.611 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.620 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.631 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.643 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.688 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.117) 0:01:02.175 ******** 2019-10-13T03:59:19.729 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-13T03:59:19.731 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.740 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:19.817 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.129) 0:01:02.304 ******** 2019-10-13T03:59:19.825 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-10-13T03:59:19.826 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T03:59:19.880 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:19 +0000 (0:00:00.062) 0:01:02.367 ******** 2019-10-13T03:59:20.338 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T03:59:20.340 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-13T03:59:20.655 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-13T03:59:20.706 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** ok: [smithi110.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:20.932 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-13T03:59:20.934 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-13T03:59:20.979 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:20 +0000 (0:00:01.099) 0:01:03.467 ******** 2019-10-13T03:59:21.129 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-13T03:59:21.131 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T03:59:21.172 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:21 +0000 (0:00:00.192) 0:01:03.659 ******** 2019-10-13T03:59:21.505 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T03:59:21.506 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-13T03:59:21.518 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T03:59:21.519 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:21.576 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:21 +0000 (0:00:00.404) 0:01:04.064 ******** 2019-10-13T03:59:21.814 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T03:59:21.815 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-13T03:59:21.921 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-13T03:59:21.922 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-13T03:59:21.968 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:21 +0000 (0:00:00.391) 0:01:04.455 ******** 2019-10-13T03:59:22.131 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-13T03:59:22.132 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-13T03:59:22.184 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.216) 0:01:04.672 ******** 2019-10-13T03:59:22.232 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-13T03:59:22.234 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-13T03:59:22.278 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.093) 0:01:04.765 ******** 2019-10-13T03:59:22.304 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-13T03:59:22.306 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.345 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.067) 0:01:04.833 ******** 2019-10-13T03:59:22.374 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-13T03:59:22.376 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:22.416 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.070) 0:01:04.903 ******** 2019-10-13T03:59:22.452 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-13T03:59:22.454 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.464 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.475 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.488 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.529 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.113) 0:01:05.016 ******** 2019-10-13T03:59:22.545 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-13T03:59:22.547 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-13T03:59:22.593 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.063) 0:01:05.080 ******** 2019-10-13T03:59:22.599 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-13T03:59:22.600 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.656 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.062) 0:01:05.143 ******** 2019-10-13T03:59:22.760 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-13T03:59:22.760 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:59:22.802 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.146) 0:01:05.290 ******** 2019-10-13T03:59:22.834 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-13T03:59:22.836 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:22.882 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.079) 0:01:05.369 ******** 2019-10-13T03:59:22.919 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-13T03:59:22.921 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:22.964 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:22 +0000 (0:00:00.081) 0:01:05.451 ******** 2019-10-13T03:59:23.002 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-13T03:59:23.003 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:23.051 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:23 +0000 (0:00:00.087) 0:01:05.538 ******** 2019-10-13T03:59:23.415 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-13T03:59:23.416 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-13T03:59:23.456 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:23 +0000 (0:00:00.404) 0:01:05.943 ******** 2019-10-13T03:59:23.578 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T03:59:23.579 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:24.120 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 03:57:39 UTC", "ActiveEnterTimestampMonotonic": "18276330", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice local-fs.target var-run.mount -.mount systemd-tmpfiles-setup.service basic.target remote-fs.target nss-lookup.target time-sync.target systemd-journald.socket sysinit.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 03:57:39 UTC", "AssertTimestampMonotonic": "18275771", "Before": "shutdown.target xdm.service plymouth-quit.service multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 03:57:39 UTC", "ConditionTimestampMonotonic": "18275771", "ConfigurationDirectoryMode": "0755", "Confl 2019-10-13T03:59:24.122 INFO:teuthology.task.ansible.out:icts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "802", "ExecMainStartTimestamp": "Sun 2019-10-13 03:57:39 UTC", "ExecMainStartTimestampMonotonic": "18276304", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2019-10-13 03:57:39 UTC] ; stop_time=[n/a] ; pid=802 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 03:57:39 UTC", "InactiveExitTimestampMonotonic": "18276330", "InvocationID": "3a7a28f855694a4b83782b7a84ea9aa7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "Keyri 2019-10-13T03:59:24.129 INFO:teuthology.task.ansible.out:ngMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "802", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "Re 2019-10-13T03:59:24.132 INFO:teuthology.task.ansible.out:quiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 03:57:39 UTC", "StateChangeTimestampMonotonic": "18276330", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 03:57:39 UTC", "WatchdogTimestampMonotonic": "18276329", "WatchdogUSec": "0"}} 2019-10-13T03:59:24.167 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.710) 0:01:06.654 ******** 2019-10-13T03:59:24.193 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-13T03:59:24.195 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.234 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.067) 0:01:06.721 ******** 2019-10-13T03:59:24.271 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-13T03:59:24.272 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.283 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.344 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.109) 0:01:06.831 ******** 2019-10-13T03:59:24.374 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-13T03:59:24.375 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.432 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.088) 0:01:06.920 ******** 2019-10-13T03:59:24.492 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-13T03:59:24.494 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.557 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.124) 0:01:07.044 ******** 2019-10-13T03:59:24.567 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-13T03:59:24.568 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:24.626 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.068) 0:01:07.113 ******** 2019-10-13T03:59:24.631 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-13T03:59:24.631 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.675 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.049) 0:01:07.162 ******** 2019-10-13T03:59:24.712 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-13T03:59:24.714 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.763 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.087) 0:01:07.250 ******** 2019-10-13T03:59:24.790 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-13T03:59:24.792 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.832 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.069) 0:01:07.320 ******** 2019-10-13T03:59:24.860 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:24.924 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:24 +0000 (0:00:00.091) 0:01:07.412 ******** 2019-10-13T03:59:24.964 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-13T03:59:24.966 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:25.014 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.089) 0:01:07.502 ******** 2019-10-13T03:59:25.024 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-13T03:59:25.025 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-13T03:59:25.068 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.053) 0:01:07.555 ******** 2019-10-13T03:59:25.075 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-13T03:59:25.076 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:25.118 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.049) 0:01:07.605 ******** 2019-10-13T03:59:25.156 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-13T03:59:25.203 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.085) 0:01:07.690 ******** 2019-10-13T03:59:25.242 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-13T03:59:25.244 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-13T03:59:25.290 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.086) 0:01:07.777 ******** 2019-10-13T03:59:25.341 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-13T03:59:25.342 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-13T03:59:25.389 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.099) 0:01:07.876 ******** 2019-10-13T03:59:25.438 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-13T03:59:25.441 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T03:59:25.493 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.104) 0:01:07.981 ******** 2019-10-13T03:59:25.501 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-13T03:59:25.502 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-13T03:59:25.558 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.060) 0:01:08.041 ******** 2019-10-13T03:59:25.643 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-13T03:59:25.646 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T03:59:25.692 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.138) 0:01:08.180 ******** 2019-10-13T03:59:25.710 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-13T03:59:25.712 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T03:59:25.757 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.064) 0:01:08.244 ******** 2019-10-13T03:59:25.793 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-13T03:59:25.796 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:25.852 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.095) 0:01:08.339 ******** 2019-10-13T03:59:25.858 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-13T03:59:25.859 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T03:59:25.939 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.086) 0:01:08.426 ******** 2019-10-13T03:59:25.950 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-13T03:59:25.951 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-13T03:59:25.995 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:25 +0000 (0:00:00.055) 0:01:08.482 ******** 2019-10-13T03:59:26.001 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-13T03:59:26.003 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:26.048 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.052) 0:01:08.535 ******** 2019-10-13T03:59:26.053 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-13T03:59:26.055 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:26.074 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:26.110 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.062) 0:01:08.597 ******** 2019-10-13T03:59:26.116 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-13T03:59:26.118 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:26.160 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.048) 0:01:08.646 ******** 2019-10-13T03:59:26.318 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi110.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-13T03:59:26.362 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.202) 0:01:08.849 ******** 2019-10-13T03:59:26.477 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-13T03:59:26.478 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-13T03:59:26.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.158) 0:01:09.007 ******** 2019-10-13T03:59:26.777 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-13T03:59:26.780 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 03:57:34 UTC", "ActiveEnterTimestampMonotonic": "17406260", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount remote-fs.target systemd-tmpfiles-setup.service system.slice local-fs.target network.target time-sync.target -.mount nss-lookup.target systemd-journald.socket basic.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 03:57:34 UTC", "AssertTimestampMonotonic": "17405705", "Before": "shutdown.target multi-user.target plymouth-quit.service getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 03:57:34 UTC", "ConditionTimestampMonotonic": "17405705", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-10-13T03:59:26.781 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "823", "ExecMainStartTimestamp": "Sun 2019-10-13 03:57:34 UTC", "ExecMainStartTimestampMonotonic": "17406239", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2019-10-13 03:57:34 UTC] ; stop_time=[n/a] ; pid=823 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 03:57:34 UTC", "InactiveExitTimestampMonotonic": "17406260", "InvocationID": "b0d74300b0be4b2da0b62ec5a56b6bba", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "inf 2019-10-13T03:59:26.782 INFO:teuthology.task.ansible.out:inity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "823", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "R 2019-10-13T03:59:26.784 INFO:teuthology.task.ansible.out:estrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 03:57:34 UTC", "StateChangeTimestampMonotonic": "17406260", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 03:57:34 UTC", "WatchdogTimestampMonotonic": "17406259", "WatchdogUSec": "0"}} 2019-10-13T03:59:26.841 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.320) 0:01:09.328 ******** 2019-10-13T03:59:26.848 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-10-13T03:59:26.849 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-13T03:59:26.911 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.070) 0:01:09.399 ******** 2019-10-13T03:59:26.920 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-13T03:59:26.922 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:26.963 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:26 +0000 (0:00:00.051) 0:01:09.450 ******** 2019-10-13T03:59:26.970 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-13T03:59:26.970 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi110"}, "changed": false} 2019-10-13T03:59:27.016 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.052) 0:01:09.503 ******** 2019-10-13T03:59:27.021 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-13T03:59:27.022 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.023 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.065 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.049) 0:01:09.552 ******** 2019-10-13T03:59:27.102 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-13T03:59:27.104 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.142 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.076) 0:01:09.629 ******** 2019-10-13T03:59:27.189 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-13T03:59:27.190 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.236 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.094) 0:01:09.723 ******** 2019-10-13T03:59:27.267 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-13T03:59:27.268 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.315 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.079) 0:01:09.803 ******** 2019-10-13T03:59:27.355 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-13T03:59:27.357 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.399 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.083) 0:01:09.886 ******** 2019-10-13T03:59:27.425 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.472 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.072) 0:01:09.959 ******** 2019-10-13T03:59:27.498 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-13T03:59:27.500 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.549 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.076) 0:01:10.036 ******** 2019-10-13T03:59:27.574 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-13T03:59:27.576 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.619 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.068) 0:01:10.104 ******** 2019-10-13T03:59:27.664 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-13T03:59:27.665 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:27.709 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.091) 0:01:10.196 ******** 2019-10-13T03:59:27.748 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-13T03:59:27.748 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-13T03:59:27.789 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.079) 0:01:10.275 ******** 2019-10-13T03:59:27.827 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-13T03:59:27.828 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-13T03:59:27.888 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.099) 0:01:10.375 ******** 2019-10-13T03:59:27.950 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-13T03:59:27.952 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-13T03:59:27.996 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:27 +0000 (0:00:00.108) 0:01:10.484 ******** 2019-10-13T03:59:28.046 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-13T03:59:28.117 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.120) 0:01:10.604 ******** 2019-10-13T03:59:28.123 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-13T03:59:28.124 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi110.front.sepia.ceph.com", "ansible_hostname": "smithi110", "ansible_nodename": "smithi110"}, "changed": true, "name": "smithi110"} 2019-10-13T03:59:28.166 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.049) 0:01:10.653 ******** 2019-10-13T03:59:28.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-13T03:59:28.246 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-13T03:59:28.311 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.144) 0:01:10.798 ******** 2019-10-13T03:59:28.462 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-13T03:59:28.463 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-13T03:59:28.511 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.199) 0:01:10.998 ******** 2019-10-13T03:59:28.537 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-10-13T03:59:28.538 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-13T03:59:28.539 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:28.581 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.070) 0:01:11.069 ******** 2019-10-13T03:59:28.634 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.052) 0:01:11.121 ******** 2019-10-13T03:59:28.664 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:28.712 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.077) 0:01:11.199 ******** 2019-10-13T03:59:28.886 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-13T03:59:28.886 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-13T03:59:28.939 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.219) 0:01:11.419 ******** TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-13T03:59:28.984 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:28 +0000 (0:00:00.052) 0:01:11.471 ******** 2019-10-13T03:59:29.392 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-13T03:59:29.436 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.450) 0:01:11.924 ******** 2019-10-13T03:59:29.464 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:29.468 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-13T03:59:29.469 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually 2019-10-13T03:59:29.471 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T03:59:29.520 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.083) 0:01:12.007 ******** 2019-10-13T03:59:29.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-13T03:59:29.528 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi204"}, "changed": false} 2019-10-13T03:59:29.580 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.059) 0:01:12.067 ******** 2019-10-13T03:59:29.588 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-13T03:59:29.589 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:29.652 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.071) 0:01:12.139 ******** 2019-10-13T03:59:29.681 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T03:59:29.683 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:29.734 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.081) 0:01:12.221 ******** 2019-10-13T03:59:29.761 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T03:59:29.762 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:29.810 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.076) 0:01:12.297 ******** 2019-10-13T03:59:29.851 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:29.894 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:29 +0000 (0:00:00.083) 0:01:12.381 ******** 2019-10-13T03:59:29.945 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T03:59:29.947 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.001 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.107) 0:01:12.488 ******** 2019-10-13T03:59:30.040 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.081 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.080) 0:01:12.568 ******** 2019-10-13T03:59:30.087 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-13T03:59:30.088 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_nodename": "smithi204"}, "changed": true, "name": "smithi204"} 2019-10-13T03:59:30.136 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.049) 0:01:12.618 ******** TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T03:59:30.141 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.188 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.056) 0:01:12.675 ******** 2019-10-13T03:59:30.227 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-13T03:59:30.287 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.099) 0:01:12.775 ******** 2019-10-13T03:59:30.315 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T03:59:30.358 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.070) 0:01:12.845 ******** 2019-10-13T03:59:30.403 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.464 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.103) 0:01:12.949 ******** 2019-10-13T03:59:30.489 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-10-13T03:59:30.495 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-13T03:59:30.538 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.076) 0:01:13.026 ******** 2019-10-13T03:59:30.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-13T03:59:30.548 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.550 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.623 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.084) 0:01:13.110 ******** 2019-10-13T03:59:30.650 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.689 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.065) 0:01:13.176 ******** 2019-10-13T03:59:30.725 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T03:59:30.727 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.769 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.078) 0:01:13.254 ******** 2019-10-13T03:59:30.808 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.853 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.085) 0:01:13.340 ******** 2019-10-13T03:59:30.890 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T03:59:30.891 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:30.936 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.083) 0:01:13.423 ******** 2019-10-13T03:59:30.945 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-10-13T03:59:30.946 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-13T03:59:30.996 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:30 +0000 (0:00:00.060) 0:01:13.484 ******** 2019-10-13T03:59:31.004 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-13T03:59:31.006 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.067 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.069) 0:01:13.554 ******** 2019-10-13T03:59:31.094 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-13T03:59:31.095 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.133 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.066) 0:01:13.621 ******** 2019-10-13T03:59:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-13T03:59:31.173 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.184 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.185 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.195 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.206 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.220 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.231 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.233 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.243 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.250 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.255 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.295 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.162) 0:01:13.783 ******** 2019-10-13T03:59:31.322 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T03:59:31.367 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.067) 0:01:13.851 ******** 2019-10-13T03:59:31.418 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T03:59:31.420 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.430 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.442 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.487 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.123) 0:01:13.974 ******** 2019-10-13T03:59:31.499 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:31.517 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-13T03:59:31.518 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-13T03:59:31.520 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T03:59:31.569 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.082) 0:01:14.056 ******** 2019-10-13T03:59:31.610 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.041) 0:01:14.098 ******** 2019-10-13T03:59:31.619 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T03:59:31.621 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.665 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.054) 0:01:14.152 ******** 2019-10-13T03:59:31.671 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-13T03:59:31.673 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.718 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.053) 0:01:14.205 ******** 2019-10-13T03:59:31.724 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T03:59:31.726 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.728 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.771 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.052) 0:01:14.258 ******** 2019-10-13T03:59:31.777 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T03:59:31.779 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.819 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.047) 0:01:14.306 ******** 2019-10-13T03:59:31.826 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T03:59:31.828 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.869 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.049) 0:01:14.356 ******** 2019-10-13T03:59:31.875 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T03:59:31.876 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.918 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.049) 0:01:14.405 ******** 2019-10-13T03:59:31.923 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-13T03:59:31.924 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:31.970 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:31 +0000 (0:00:00.051) 0:01:14.457 ******** 2019-10-13T03:59:31.975 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T03:59:31.976 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.033 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.055) 0:01:14.513 ******** TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.078 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.052) 0:01:14.565 ******** 2019-10-13T03:59:32.085 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T03:59:32.085 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.127 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.049) 0:01:14.615 ******** 2019-10-13T03:59:32.154 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T03:59:32.156 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.208 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.080) 0:01:14.695 ******** 2019-10-13T03:59:32.236 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-13T03:59:32.238 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005105", "end": "2019-10-13 03:59:32.209407", "rc": 0, "start": "2019-10-13 03:59:32.204302", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.277 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.069) 0:01:14.765 ******** 2019-10-13T03:59:32.318 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.040) 0:01:14.805 ******** 2019-10-13T03:59:32.335 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T03:59:32.370 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.051) 0:01:14.857 ******** 2019-10-13T03:59:32.377 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-13T03:59:32.423 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.051) 0:01:14.909 ******** 2019-10-13T03:59:32.449 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-13T03:59:32.501 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.079) 0:01:14.988 ******** 2019-10-13T03:59:32.529 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-13T03:59:32.530 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.595 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.093) 0:01:15.082 ******** 2019-10-13T03:59:32.633 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-13T03:59:32.634 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.644 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.684 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.089) 0:01:15.171 ******** 2019-10-13T03:59:32.710 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-10-13T03:59:32.712 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.752 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.066) 0:01:15.238 ******** 2019-10-13T03:59:32.787 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T03:59:32.789 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.838 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.087) 0:01:15.325 ******** 2019-10-13T03:59:32.875 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T03:59:32.877 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:32.930 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.091) 0:01:15.417 ******** 2019-10-13T03:59:32.972 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:32 +0000 (0:00:00.042) 0:01:15.459 ******** 2019-10-13T03:59:32.983 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-13T03:59:32.985 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.041 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.069) 0:01:15.528 ******** 2019-10-13T03:59:33.049 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-13T03:59:33.051 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.093 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.051) 0:01:15.580 ******** 2019-10-13T03:59:33.119 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-13T03:59:33.121 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.160 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.067) 0:01:15.648 ******** 2019-10-13T03:59:33.188 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-13T03:59:33.188 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.228 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.067) 0:01:15.715 ******** 2019-10-13T03:59:33.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-13T03:59:33.269 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.279 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.280 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.291 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.302 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.302 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.315 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.317 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.327 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.328 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.339 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.339 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.382 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.153) 0:01:15.869 ******** 2019-10-13T03:59:33.419 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T03:59:33.460 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.078) 0:01:15.947 ******** 2019-10-13T03:59:33.497 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T03:59:33.499 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.510 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.511 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.512 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.560 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.099) 0:01:16.047 ******** 2019-10-13T03:59:33.586 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-13T03:59:33.631 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.071) 0:01:16.119 ******** 2019-10-13T03:59:33.668 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T03:59:33.670 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.718 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.086) 0:01:16.205 ******** 2019-10-13T03:59:33.765 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T03:59:33.766 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.768 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.820 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.102) 0:01:16.307 ******** 2019-10-13T03:59:33.847 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-13T03:59:33.849 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.889 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.069) 0:01:16.376 ******** 2019-10-13T03:59:33.916 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:33.958 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:33 +0000 (0:00:00.068) 0:01:16.445 ******** 2019-10-13T03:59:33.984 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-13T03:59:33.986 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:34.025 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:34 +0000 (0:00:00.067) 0:01:16.513 ******** 2019-10-13T03:59:34.189 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-13T03:59:34.191 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004621", "end": "2019-10-13 03:59:34.457565", "rc": 0, "start": "2019-10-13 03:59:34.452944", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:34.232 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:34 +0000 (0:00:00.205) 0:01:16.719 ******** 2019-10-13T03:59:34.259 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-13T03:59:34.301 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:34 +0000 (0:00:00.069) 0:01:16.788 ******** 2019-10-13T03:59:34.433 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-13T03:59:34.435 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-13T03:59:34.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-13T03:59:34.650 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-10-13T03:59:34.697 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:34 +0000 (0:00:00.396) 0:01:17.184 ******** 2019-10-13T03:59:34.758 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-13T03:59:34.765 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T03:59:34.817 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:34 +0000 (0:00:00.120) 0:01:17.305 ******** 2019-10-13T03:59:35.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-13T03:59:35.307 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-13T03:59:35.359 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:35 +0000 (0:00:00.541) 0:01:17.846 ******** 2019-10-13T03:59:36.007 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-13T03:59:36.254 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-13T03:59:36.256 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T03:59:36.302 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:36 +0000 (0:00:00.942) 0:01:18.789 ******** 2019-10-13T03:59:36.332 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-13T03:59:36.371 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:36 +0000 (0:00:00.069) 0:01:18.858 ******** 2019-10-13T03:59:36.544 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2019-10-13T03:59:36.545 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035450", "end": "2019-10-13 03:59:36.521593", "rc": 0, "start": "2019-10-13 03:59:36.486143", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:36.585 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:36 +0000 (0:00:00.213) 0:01:19.072 ******** 2019-10-13T03:59:36.895 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi204.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-13T03:59:36.958 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:36 +0000 (0:00:00.372) 0:01:19.445 ******** 2019-10-13T03:59:37.855 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-13T03:59:37.855 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T03:59:37.895 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:37 +0000 (0:00:00.937) 0:01:20.383 ******** 2019-10-13T03:59:37.921 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-13T03:59:37.961 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:37 +0000 (0:00:00.066) 0:01:20.449 ******** 2019-10-13T03:59:38.133 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-13T03:59:38.134 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032656", "end": "2019-10-13 03:59:38.397850", "rc": 0, "start": "2019-10-13 03:59:38.365194", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T03:59:38.182 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:38 +0000 (0:00:00.220) 0:01:20.669 ******** 2019-10-13T03:59:39.270 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T03:59:39.311 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:39 +0000 (0:00:01.129) 0:01:21.798 ******** 2019-10-13T03:59:39.337 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:40.915 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T03:59:40.962 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:40 +0000 (0:00:01.650) 0:01:23.449 ******** 2019-10-13T03:59:40.983 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:49.459 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-13T03:59:49.460 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 92 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-13T03:59:49.462 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-13T03:59:49.463 INFO:teuthology.task.ansible.out:6 to remove and 92 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:49.536 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:49 +0000 (0:00:08.573) 0:01:32.024 ******** 2019-10-13T03:59:49.560 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T03:59:50.785 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 92 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-13T03:59:50.788 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-13T03:59:50.789 INFO:teuthology.task.ansible.out:6 to remove and 92 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T03:59:50.841 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 03:59:50 +0000 (0:00:01.300) 0:01:33.324 ******** 2019-10-13T03:59:50.866 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:13.099 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:13.100 INFO:teuthology.task.ansible.out:TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T04:00:13.102 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-13T04:00:13.103 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (948 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-13T04:00:13.105 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-13T04:00:13.106 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu. 2019-10-13T04:00:13.107 INFO:teuthology.task.ansible.out:com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (948 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting u 2019-10-13T04:00:13.109 INFO:teuthology.task.ansible.out:p apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find ap 2019-10-13T04:00:13.110 INFO:teuthology.task.ansible.out:titude. Using apt-get instead"]} 2019-10-13T04:00:13.149 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:13 +0000 (0:00:22.312) 0:01:55.637 ******** 2019-10-13T04:00:13.177 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T04:00:13.237 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:13 +0000 (0:00:00.087) 0:01:55.725 ******** 2019-10-13T04:00:13.259 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:13.818 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T04:00:13.820 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-13T04:00:13.821 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (945 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-13T04:00:13.823 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-13T04:00:13.824 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 85 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu. 2019-10-13T04:00:13.827 INFO:teuthology.task.ansible.out:com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (945 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting u 2019-10-13T04:00:13.829 INFO:teuthology.task.ansible.out:p apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find ap 2019-10-13T04:00:13.830 INFO:teuthology.task.ansible.out:titude. Using apt-get instead"]} 2019-10-13T04:00:13.881 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:13 +0000 (0:00:00.643) 0:01:56.369 ******** 2019-10-13T04:00:13.916 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-13T04:00:13.963 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:13 +0000 (0:00:00.081) 0:01:56.450 ******** 2019-10-13T04:00:13.986 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:25.304 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T04:00:25.306 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-13T04:00:25.307 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 4s (2702 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-13T04:00:25.309 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-13T04:00:25.310 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 4s (2702 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files a 2019-10-13T04:00:25.312 INFO:teuthology.task.ansible.out:nd directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:25.353 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:25 +0000 (0:00:11.390) 0:02:07.840 ******** 2019-10-13T04:00:25.376 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via 2019-10-13T04:00:25.378 INFO:teuthology.task.ansible.out: squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:26.062 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-13T04:00:26.064 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 4s (2610 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-13T04:00:26.065 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-13T04:00:26.067 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 85 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 4s (2610 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files a 2019-10-13T04:00:26.068 INFO:teuthology.task.ansible.out:nd directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:26.111 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:26 +0000 (0:00:00.758) 0:02:08.598 ******** 2019-10-13T04:00:26.135 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:26.270 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-13T04:00:26.271 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:26.327 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:26 +0000 (0:00:00.216) 0:02:08.814 ******** 2019-10-13T04:00:26.454 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-13T04:00:26.455 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-13T04:00:26.497 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:26 +0000 (0:00:00.169) 0:02:08.984 ******** 2019-10-13T04:00:26.521 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:26.524 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-13T04:00:26.526 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:26.566 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:26 +0000 (0:00:00.068) 0:02:09.053 ******** 2019-10-13T04:00:26.729 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-13T04:00:26.731 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-13T04:00:26.772 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:26 +0000 (0:00:00.206) 0:02:09.260 ******** 2019-10-13T04:00:27.035 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-13T04:00:27.036 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:27.090 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.317) 0:02:09.578 ******** 2019-10-13T04:00:27.122 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-13T04:00:27.123 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-13T04:00:27.168 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.077) 0:02:09.655 ******** 2019-10-13T04:00:27.242 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-13T04:00:27.294 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.126) 0:02:09.782 ******** 2019-10-13T04:00:27.316 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:27.321 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-13T04:00:27.322 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T04:00:27.323 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:27.363 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.068) 0:02:09.850 ******** 2019-10-13T04:00:27.399 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.036) 0:02:09.886 ******** 2019-10-13T04:00:27.500 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-13T04:00:27.501 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:27.574 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.175) 0:02:10.062 ******** 2019-10-13T04:00:27.584 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-13T04:00:27.585 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-13T04:00:27.654 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.079) 0:02:10.142 ******** 2019-10-13T04:00:27.885 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T04:00:27.888 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 04:00:12 UTC", "ActiveEnterTimestampMonotonic": "171002703", "ActiveExitTimestamp": "Sun 2019-10-13 03:59:58 UTC", "ActiveExitTimestampMonotonic": "157562233", "ActiveState": "active", "After": "system.slice network.target systemd-journald.socket remote-fs.target sysinit.target basic.target nss-lookup.target systemd-tmpfiles-setup.service -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 04:00:12 UTC", "AssertTimestampMonotonic": "170977016", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 04:00:12 UTC", "ConditionTimestampMonotonic": "170977016", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-13T04:00:27.889 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6664", "ExecMainStartTimestamp": "Sun 2019-10-13 04:00:12 UTC", "ExecMainStartTimestampMonotonic": "171002692", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sun 2019-10-13 03:59:58 UTC", "InactiveEnterTimestampMonotonic": "157636449", "InactiveExitTimestamp": "Sun 2019-10-13 04:00:12 UTC", "InactiveExitTimestampMonotonic": "170977783", "InvocationID": "5efd4cd65bde4d42a2729efee1301cda", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-13T04:00:27.891 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6664", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-13T04:00:27.892 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 04:00:12 UTC", "StateChangeTimestampMonotonic": "171002703", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 04:00:12 UTC", "WatchdogTimestampMonotonic": "171002703", "WatchdogUSec": "0"}} 2019-10-13T04:00:27.940 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:27 +0000 (0:00:00.285) 0:02:10.427 ******** 2019-10-13T04:00:27.975 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.013 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.073) 0:02:10.501 ******** 2019-10-13T04:00:28.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-13T04:00:28.045 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-13T04:00:28.046 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.112 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.098) 0:02:10.599 ******** 2019-10-13T04:00:28.172 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.059) 0:02:10.659 ******** 2019-10-13T04:00:28.178 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-13T04:00:28.180 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.227 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.055) 0:02:10.714 ******** 2019-10-13T04:00:28.278 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T04:00:28.279 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.318 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.090) 0:02:10.805 ******** 2019-10-13T04:00:28.346 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-13T04:00:28.347 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-13T04:00:28.348 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.391 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.072) 0:02:10.878 ******** 2019-10-13T04:00:28.433 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.042) 0:02:10.920 ******** 2019-10-13T04:00:28.439 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T04:00:28.441 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.483 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.048) 0:02:10.969 ******** 2019-10-13T04:00:28.511 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:28.560 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi110.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.611 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.127) 0:02:11.097 ******** 2019-10-13T04:00:28.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-13T04:00:28.617 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:28.660 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.049) 0:02:11.147 ******** 2019-10-13T04:00:28.665 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T04:00:28.708 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.048) 0:02:11.195 ******** 2019-10-13T04:00:28.731 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:28.757 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T04:00:28.759 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.806 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.097) 0:02:11.293 ******** 2019-10-13T04:00:28.830 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:28.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T04:00:28.862 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.903 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.097) 0:02:11.390 ******** 2019-10-13T04:00:28.924 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:28.941 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-13T04:00:28.944 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 04:00:13 UTC", "ActiveEnterTimestampMonotonic": "176483541", "ActiveExitTimestamp": "Sun 2019-10-13 04:00:00 UTC", "ActiveExitTimestampMonotonic": "163640675", "ActiveState": "active", "After": "nss-lookup.target network.target -.mount basic.target system.slice systemd-journald.socket sysinit.target remote-fs.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 04:00:13 UTC", "AssertTimestampMonotonic": "176461358", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 04:00:13 UTC", "ConditionTimestampMonotonic": "176461358", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-13T04:00:28.945 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6649", "ExecMainStartTimestamp": "Sun 2019-10-13 04:00:13 UTC", "ExecMainStartTimestampMonotonic": "176483520", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Sun 2019-10-13 04:00:00 UTC", "InactiveEnterTimestampMonotonic": "163708177", "InactiveExitTimestamp": "Sun 2019-10-13 04:00:13 UTC", "InactiveExitTimestampMonotonic": "176461936", "InvocationID": "0659c0c7b2a0478a8a93637a4105bf37", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-13T04:00:28.947 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6649", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-13T04:00:28.948 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 04:00:13 UTC", "StateChangeTimestampMonotonic": "176483541", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 04:00:13 UTC", "WatchdogTimestampMonotonic": "176483540", "WatchdogUSec": "0"}} TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi110.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:28.991 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:28 +0000 (0:00:00.088) 0:02:11.479 ******** 2019-10-13T04:00:29.041 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.528 ******** 2019-10-13T04:00:29.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-13T04:00:29.049 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.091 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.578 ******** 2019-10-13T04:00:29.097 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T04:00:29.099 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.139 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.048) 0:02:11.626 ******** 2019-10-13T04:00:29.145 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-13T04:00:29.147 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.189 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.676 ******** 2019-10-13T04:00:29.195 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T04:00:29.262 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.726 ******** TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.287 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.047) 0:02:11.774 ******** 2019-10-13T04:00:29.293 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T04:00:29.295 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.336 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.048) 0:02:11.823 ******** 2019-10-13T04:00:29.342 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-13T04:00:29.344 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.385 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.872 ******** 2019-10-13T04:00:29.390 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T04:00:29.392 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.436 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.047) 0:02:11.920 ******** 2019-10-13T04:00:29.438 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-13T04:00:29.440 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.481 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:11.969 ******** 2019-10-13T04:00:29.488 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T04:00:29.489 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.533 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.051) 0:02:12.020 ******** 2019-10-13T04:00:29.539 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-13T04:00:29.540 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.581 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.047) 0:02:12.068 ******** 2019-10-13T04:00:29.588 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.603 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:29.628 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.047) 0:02:12.115 ******** 2019-10-13T04:00:29.665 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.696 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.067) 0:02:12.183 ******** 2019-10-13T04:00:29.732 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.036) 0:02:12.219 ******** 2019-10-13T04:00:29.738 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T04:00:29.739 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.787 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.055) 0:02:12.274 ******** 2019-10-13T04:00:29.794 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-13T04:00:29.849 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.061) 0:02:12.336 ******** 2019-10-13T04:00:29.856 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.882 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:29.899 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.049) 0:02:12.386 ******** 2019-10-13T04:00:29.904 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-13T04:00:29.906 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.956 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:29 +0000 (0:00:00.057) 0:02:12.443 ******** 2019-10-13T04:00:29.962 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T04:00:29.963 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:29.977 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:30.016 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.059) 0:02:12.503 ******** 2019-10-13T04:00:30.022 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-13T04:00:30.024 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.070 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.053) 0:02:12.557 ******** 2019-10-13T04:00:30.080 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.104 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:30.120 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.050) 0:02:12.607 ******** 2019-10-13T04:00:30.128 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-13T04:00:30.129 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.172 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.051) 0:02:12.659 ******** 2019-10-13T04:00:30.177 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T04:00:30.179 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.223 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.050) 0:02:12.710 ******** 2019-10-13T04:00:30.230 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:30.232 INFO:teuthology.task.ansible.out:TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.277 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.053) 0:02:12.764 ******** 2019-10-13T04:00:30.278 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:30.284 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:30.286 INFO:teuthology.task.ansible.out:TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.352 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.074) 0:02:12.839 ******** 2019-10-13T04:00:30.361 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:30.363 INFO:teuthology.task.ansible.out:TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.406 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.053) 0:02:12.893 ******** 2019-10-13T04:00:30.412 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-13T04:00:30.412 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.457 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.051) 0:02:12.945 ******** 2019-10-13T04:00:30.464 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T04:00:30.465 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.508 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.050) 0:02:12.995 ******** 2019-10-13T04:00:30.536 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-13T04:00:30.537 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.578 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.069) 0:02:13.065 ******** 2019-10-13T04:00:30.596 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T04:00:30.598 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-13T04:00:30.638 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.059) 0:02:13.125 ******** 2019-10-13T04:00:30.648 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.688 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.050) 0:02:13.176 ******** 2019-10-13T04:00:30.716 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T04:00:30.721 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.762 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.073) 0:02:13.250 ******** 2019-10-13T04:00:30.791 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-13T04:00:30.793 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.834 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.072) 0:02:13.322 ******** 2019-10-13T04:00:30.863 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T04:00:30.865 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.909 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.074) 0:02:13.396 ******** 2019-10-13T04:00:30.946 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T04:00:30.948 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:30.989 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:30 +0000 (0:00:00.079) 0:02:13.476 ******** 2019-10-13T04:00:30.994 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T04:00:30.995 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-13T04:00:31.042 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.052) 0:02:13.529 ******** 2019-10-13T04:00:31.052 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.106 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.064) 0:02:13.594 ******** 2019-10-13T04:00:31.112 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T04:00:31.114 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.168 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.058) 0:02:13.652 ******** 2019-10-13T04:00:31.172 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-13T04:00:31.174 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.223 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.058) 0:02:13.710 ******** 2019-10-13T04:00:31.252 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-13T04:00:31.254 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.310 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.086) 0:02:13.797 ******** 2019-10-13T04:00:31.340 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.380 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.070) 0:02:13.868 ******** 2019-10-13T04:00:31.420 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:31.461 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.080) 0:02:13.948 ******** 2019-10-13T04:00:31.532 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T04:00:31.536 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-13T04:00:31.537 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:31.599 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.137) 0:02:14.086 ******** 2019-10-13T04:00:31.626 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-13T04:00:31.628 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-13T04:00:31.674 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:31 +0000 (0:00:00.075) 0:02:14.161 ******** 2019-10-13T04:00:31.982 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-13T04:00:31.983 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-13T04:00:32.051 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.376) 0:02:14.538 ******** 2019-10-13T04:00:32.067 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-13T04:00:32.068 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-13T04:00:32.111 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.060) 0:02:14.599 ******** 2019-10-13T04:00:32.142 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-13T04:00:32.143 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:32.192 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.080) 0:02:14.679 ******** 2019-10-13T04:00:32.413 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-13T04:00:32.414 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-13T04:00:32.465 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.273) 0:02:14.953 ******** 2019-10-13T04:00:32.540 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-13T04:00:32.541 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-13T04:00:32.602 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.136) 0:02:15.089 ******** 2019-10-13T04:00:32.610 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-13T04:00:32.612 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:32.666 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.064) 0:02:15.153 ******** 2019-10-13T04:00:32.695 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-13T04:00:32.697 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:32.738 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.072) 0:02:15.225 ******** 2019-10-13T04:00:32.766 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T04:00:32.767 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:32.829 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.091) 0:02:15.317 ******** 2019-10-13T04:00:32.867 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T04:00:32.868 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:32.912 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.082) 0:02:15.399 ******** 2019-10-13T04:00:32.941 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T04:00:32.943 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:32.997 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:32 +0000 (0:00:00.085) 0:02:15.484 ******** 2019-10-13T04:00:33.008 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-13T04:00:33.051 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.053) 0:02:15.538 ******** 2019-10-13T04:00:33.058 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-13T04:00:33.060 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.106 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.055) 0:02:15.594 ******** 2019-10-13T04:00:33.135 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:33.137 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T04:00:33.187 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.080) 0:02:15.675 ******** 2019-10-13T04:00:33.240 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi110.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.248 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.263 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.284 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.285 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.296 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.316 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.317 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.328 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.330 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-13T04:00:33.340 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.351 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.365 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.371 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.376 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.425 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.237) 0:02:15.912 ******** 2019-10-13T04:00:33.432 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-13T04:00:33.434 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-13T04:00:33.481 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.055) 0:02:15.968 ******** 2019-10-13T04:00:33.614 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-13T04:00:33.617 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005010", "end": "2019-10-13 04:00:33.594493", "rc": 0, "start": "2019-10-13 04:00:33.589483", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:33.676 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.195) 0:02:16.164 ******** 2019-10-13T04:00:33.721 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.044) 0:02:16.208 ******** 2019-10-13T04:00:33.727 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T04:00:33.728 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.776 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.055) 0:02:16.263 ******** 2019-10-13T04:00:33.783 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-13T04:00:33.784 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.832 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.055) 0:02:16.319 ******** 2019-10-13T04:00:33.838 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-13T04:00:33.839 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.887 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.054) 0:02:16.374 ******** 2019-10-13T04:00:33.894 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-13T04:00:33.895 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.941 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.054) 0:02:16.428 ******** 2019-10-13T04:00:33.952 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:33.993 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:33 +0000 (0:00:00.052) 0:02:16.481 ******** 2019-10-13T04:00:34.000 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-13T04:00:34.007 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.019 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:34.045 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.051) 0:02:16.532 ******** 2019-10-13T04:00:34.076 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-13T04:00:34.078 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.115 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.070) 0:02:16.603 ******** 2019-10-13T04:00:34.145 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.185 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.069) 0:02:16.672 ******** 2019-10-13T04:00:34.236 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:34.238 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-13T04:00:34.280 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.094) 0:02:16.767 ******** 2019-10-13T04:00:34.328 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi204.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.339 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.353 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.367 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.368 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.380 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.391 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.402 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.403 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.429 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi204.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.439 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.440 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.457 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.461 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.538 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.248) 0:02:17.016 ******** 2019-10-13T04:00:34.671 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-13T04:00:34.673 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004407", "end": "2019-10-13 04:00:34.653680", "rc": 0, "start": "2019-10-13 04:00:34.649273", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-13T04:00:34.759 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.230) 0:02:17.247 ******** 2019-10-13T04:00:34.797 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-13T04:00:34.798 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.840 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.080) 0:02:17.328 ******** 2019-10-13T04:00:34.878 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-13T04:00:34.879 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:34.932 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.091) 0:02:17.419 ******** 2019-10-13T04:00:34.937 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-13T04:00:34.938 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1570938847, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:34.985 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:34 +0000 (0:00:00.052) 0:02:17.472 ******** 2019-10-13T04:00:34.993 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.038 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.053) 0:02:17.525 ******** 2019-10-13T04:00:35.044 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T04:00:35.046 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.052 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.058 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:35.087 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.049) 0:02:17.575 ******** 2019-10-13T04:00:35.136 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T04:00:35.137 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item={u'block_used': 13654306, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440802, u'size_available': 927501524992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440802, "block_size": 4096, "block_total": 240095108, "block_used": 13654306, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501524992, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-13T04:00:35.183 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.095) 0:02:17.670 ******** 2019-10-13T04:00:35.379 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009872", "end": "2019-10-13 04:00:35.355348", "rc": 0, "start": "2019-10-13 04:00:35.345476", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:35.420 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.237) 0:02:17.908 ******** 2019-10-13T04:00:35.468 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T04:00:35.470 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'block_used': 13654306, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440802, u'size_available': 927501524992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440802, "block_size": 4096, "block_total": 240095108, "block_used": 13654306, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501524992, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.510 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.089) 0:02:17.997 ******** 2019-10-13T04:00:35.537 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T04:00:35.578 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:35 +0000 (0:00:00.067) 0:02:18.065 ******** 2019-10-13T04:00:35.637 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T04:00:35.638 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QFRD', u'wwn-0x5000c50091dfc3c1'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QFRD', u'wwn': u'0x5000c50091dfc3c1', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1', u'wwn-0x5000c50091dfc3c1-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T04:00:35.640 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50091dfc3c1"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.650 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.662 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.681 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.696 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.720 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.744 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:35.755 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.035 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-13T04:00:36.036 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1570938896, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-13T04:00:36.090 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.511) 0:02:18.577 ******** 2019-10-13T04:00:36.128 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-13T04:00:36.130 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.132 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.192 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.102) 0:02:18.679 ******** 2019-10-13T04:00:36.241 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-13T04:00:36.286 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.093) 0:02:18.773 ******** 2019-10-13T04:00:36.444 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-13T04:00:36.445 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008517", "end": "2019-10-13 04:00:36.672408", "rc": 0, "start": "2019-10-13 04:00:36.663891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:36.500 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.205) 0:02:18.979 ******** 2019-10-13T04:00:36.531 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-13T04:00:36.533 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.576 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.084) 0:02:19.063 ******** 2019-10-13T04:00:36.603 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-13T04:00:36.651 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:36 +0000 (0:00:00.074) 0:02:19.138 ******** 2019-10-13T04:00:36.701 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T04:00:36.702 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E', u'wwn-0x5000c50092941a36'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XP2E', u'wwn': u'0x5000c50092941a36', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1', u'wwn-0x5000c50092941a36-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T04:00:36.705 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50092941a36"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.714 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.715 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.739 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.752 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.754 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.767 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.779 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:36.791 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:37.529 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN', u'nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.626117", "end": "2019-10-13 04:00:37.502671", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:35.876554", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T04:00:37.530 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:38.474 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN', u'nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.557372", "end": "2019-10-13 04:00:38.705508", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:37.148136", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-13T04:00:38.476 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:38.861 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.184077", "end": "2019-10-13 04:00:38.824087", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:37.640010", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:39.740 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.119281", "end": "2019-10-13 04:00:39.964812", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:38.845531", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:40.109 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.118104", "end": "2019-10-13 04:00:40.083218", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:38.965114", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:41.026 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.149631", "end": "2019-10-13 04:00:41.240817", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:40.091186", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:41.344 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.114535", "end": "2019-10-13 04:00:41.319747", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:40.205212", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:42.254 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.101547", "end": "2019-10-13 04:00:42.483008", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:41.381461", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:42.628 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P'], u'uuids': [u'40030898-e0dc-4263-932e-a36226be20b4']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.141630", "end": "2019-10-13 04:00:42.581395", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "labels": [], "masters": [], "uuids": ["40030898-e0dc-4263-932e-a36226be20b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:41.439765", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:43.504 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs'], u'uuids': [u'97d94574-b350-4896-9285-2998fb12b1c3']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.127246", "end": "2019-10-13 04:00:43.733146", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "labels": [], "masters": [], "uuids": ["97d94574-b350-4896-9285-2998fb12b1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:42.605900", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:43.879 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.090103", "end": "2019-10-13 04:00:43.848282", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:42.758179", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:43.938 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:43 +0000 (0:00:07.287) 0:02:26.426 ******** 2019-10-13T04:00:44.003 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T04:00:44.005 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QFRD', u'wwn-0x5000c50091dfc3c1'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QFRD', u'wwn': u'0x5000c50091dfc3c1', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1', u'wwn-0x5000c50091dfc3c1-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD", "wwn-0x5000c50091dfc3c1"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QFRD-part1", "wwn-0x5000c50091dfc3c1-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QFRD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T04:00:44.006 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50091dfc3c1"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.016 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.029 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.041 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.042 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.067 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.077 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.093 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.286 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj', u'nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN', u'nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053831", "end": "2019-10-13 04:00:44.248083", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ywmbtb-wQI3-pWyq-tDdK-LOGk-eacd-tNk2Dj", "nvme-INTEL_SSDPEDMD400G4_PHFT6204013Z400BGN", "nvme-nvme.8086-50484654363230343031335a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:44.194252", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-13T04:00:44.288 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0404463 s, 2.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0404463 s, 2.9 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:44.574 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149862", "end": "2019-10-13 04:00:44.543464", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwyurtCb3EBEv4PwpObliTEidFT2hhMJhP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:44.393602", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147368 s, 783 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147368 s, 783 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:44.743 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-13T04:00:44.744 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.102603", "end": "2019-10-13 04:00:44.969621", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:43.867018", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-13T04:00:44.815 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:44 +0000 (0:00:00.876) 0:02:27.302 ******** 2019-10-13T04:00:44.863 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-13T04:00:44.865 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153878", "end": "2019-10-13 04:00:44.822323", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwC4y6st6ggmnLpi8oQdcd1McgC28bgjv7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:44.668445", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149284 s, 773 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149284 s, 773 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:44.868 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E', u'wwn-0x5000c50092941a36'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XP2E', u'wwn': u'0x5000c50092941a36', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1', u'wwn-0x5000c50092941a36-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-13T04:00:44.870 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50092941a36"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.881 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.893 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.908 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.927 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.938 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.941 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:44.941 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:44.953 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:45.130 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN', u'nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049101", "end": "2019-10-13 04:00:45.353408", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gbEgNK-8E4s-k6CS-lcU9-iRSB-xyFw-DUTpxa", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:45.304307", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-13T04:00:45.137 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0329042 s, 3.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0329042 s, 3.5 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.156 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.151850", "end": "2019-10-13 04:00:45.104876", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwWc2XPy1gqeDDcjTm9Xkp5RCgR04Ne9Eb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:44.953026", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148998 s, 774 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148998 s, 774 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.447 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.150654", "end": "2019-10-13 04:00:45.659395", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqunMFSWZ5c1V26dgiJrdhXfDHf0xaetOW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:45.508741", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14788 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14788 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.457 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P'], u'uuids': [u'40030898-e0dc-4263-932e-a36226be20b4']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.152830", "end": "2019-10-13 04:00:45.413377", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3Gw3Q7Gc7eX8QogtcdQlea1esr9MHNNHR6P"], "labels": [], "masters": [], "uuids": ["40030898-e0dc-4263-932e-a36226be20b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:45.260547", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148348 s, 778 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148348 s, 778 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.724 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142493", "end": "2019-10-13 04:00:45.939616", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqbuYZm82pUd5kMKO5PXucdXniU3E7YIOc"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:45.797123", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140221 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140221 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.736 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152801", "end": "2019-10-13 04:00:45.711908", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HjY61VYZNs0TysCmqWUfGlGre5e6H3GwFLiM7VYUWrBWzGOqENHxEvPThRpmOrd8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:45.559107", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148252 s, 778 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148252 s, 778 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:45.804 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:45 +0000 (0:00:00.988) 0:02:28.291 ******** 2019-10-13T04:00:46.000 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.147545", "end": "2019-10-13 04:00:46.217580", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqNQhAmyJ6pF341dFElbPsOtiJqkggxhdd"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:46.070035", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144642 s, 797 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144642 s, 797 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.122 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-13T04:00:46.123 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.163983", "end": "2019-10-13 04:00:46.082688", "rc": 0, "start": "2019-10-13 04:00:45.918705", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.167 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.362) 0:02:28.654 ******** 2019-10-13T04:00:46.291 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs'], u'uuids': [u'97d94574-b350-4896-9285-2998fb12b1c3']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.149838", "end": "2019-10-13 04:00:46.506340", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqXO1wSJlpVatTKFtslg37RouykDwO6eqs"], "labels": [], "masters": [], "uuids": ["97d94574-b350-4896-9285-2998fb12b1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:46.356502", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14519 s, 794 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14519 s, 794 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.367 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050982", "end": "2019-10-13 04:00:46.336267", "rc": 0, "start": "2019-10-13 04:00:46.285285", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.417 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.249) 0:02:28.904 ******** 2019-10-13T04:00:46.446 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T04:00:46.493 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.076) 0:02:28.981 ******** 2019-10-13T04:00:46.534 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-13T04:00:46.536 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:46.588 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.094) 0:02:29.075 ******** 2019-10-13T04:00:46.597 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.144110", "end": "2019-10-13 04:00:46.789779", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-j8DOCfDJ3jf8PmplSGShhq6mGWmZvfyqw2oNMlhG44l2XtiPubs6a3sZ3AYydRrW"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-13 04:00:46.645669", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139687 s, 826 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139687 s, 826 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.662 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.074) 0:02:29.149 ******** 2019-10-13T04:00:46.668 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-13T04:00:46.668 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:46.730 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.067) 0:02:29.217 ******** 2019-10-13T04:00:46.768 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:46.822 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.090) 0:02:29.308 ******** 2019-10-13T04:00:46.849 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-13T04:00:46.851 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:46.909 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.088) 0:02:29.396 ******** 2019-10-13T04:00:46.918 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-13T04:00:46.920 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.088746", "end": "2019-10-13 04:00:47.136932", "rc": 0, "start": "2019-10-13 04:00:47.048186", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:46.967 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:46 +0000 (0:00:00.053) 0:02:29.450 ******** 2019-10-13T04:00:47.059 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T04:00:47.061 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:47.105 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.142) 0:02:29.592 ******** 2019-10-13T04:00:47.168 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T04:00:47.170 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={u'block_used': 13654306, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440802, u'size_available': 927501524992, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440802, "block_size": 4096, "block_total": 240095108, "block_used": 13654306, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501524992, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.213 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.107) 0:02:29.701 ******** 2019-10-13T04:00:47.232 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-13T04:00:47.233 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.105655", "end": "2019-10-13 04:00:47.464211", "rc": 0, "start": "2019-10-13 04:00:47.358556", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:00:47.285 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.071) 0:02:29.772 ******** 2019-10-13T04:00:47.296 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.341 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.055) 0:02:29.827 ******** 2019-10-13T04:00:47.352 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-13T04:00:47.408 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.067) 0:02:29.895 ******** 2019-10-13T04:00:47.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:47.416 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.487 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.078) 0:02:29.974 ******** 2019-10-13T04:00:47.496 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-13T04:00:47.497 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.551 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.064) 0:02:30.038 ******** 2019-10-13T04:00:47.558 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:47.561 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.623 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.071) 0:02:30.110 ******** 2019-10-13T04:00:47.629 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.671 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.047) 0:02:30.158 ******** 2019-10-13T04:00:47.676 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T04:00:47.677 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.719 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.048) 0:02:30.206 ******** 2019-10-13T04:00:47.724 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T04:00:47.726 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.798 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.078) 0:02:30.285 ******** 2019-10-13T04:00:47.806 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-13T04:00:47.807 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.849 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.050) 0:02:30.336 ******** 2019-10-13T04:00:47.854 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-13T04:00:47.856 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.903 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.054) 0:02:30.391 ******** 2019-10-13T04:00:47.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-13T04:00:47.912 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:47.964 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:47 +0000 (0:00:00.060) 0:02:30.452 ******** 2019-10-13T04:00:48.060 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-13T04:00:48.062 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-13T04:00:48.128 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.162) 0:02:30.614 ******** 2019-10-13T04:00:48.179 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-13T04:00:48.179 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654312, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440796, u'size_available': 927501500416, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440796, "block_size": 4096, "block_total": 240095108, "block_used": 13654312, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501500416, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.243 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.115) 0:02:30.730 ******** 2019-10-13T04:00:48.271 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-13T04:00:48.273 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.324 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.079) 0:02:30.809 ******** 2019-10-13T04:00:48.368 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:48.370 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.416 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.094) 0:02:30.904 ******** 2019-10-13T04:00:48.449 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:00:48.450 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.493 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.077) 0:02:30.981 ******** 2019-10-13T04:00:48.541 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-13T04:00:48.543 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.594 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.100) 0:02:31.081 ******** 2019-10-13T04:00:48.635 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T04:00:48.636 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} [WARNING]: The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. 2019-10-13T04:00:48.638 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.677 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.083) 0:02:31.164 ******** 2019-10-13T04:00:48.721 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.037) 0:02:31.202 ******** 2019-10-13T04:00:48.752 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-13T04:00:48.753 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:48.795 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:48 +0000 (0:00:00.080) 0:02:31.282 ******** 2019-10-13T04:00:49.190 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-13T04:00:49.192 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} 2019-10-13T04:00:49.271 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:49 +0000 (0:00:00.467) 0:02:31.749 ******** TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T04:00:49.273 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:49.520 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:49.616 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:49.784 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:49.837 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.051 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.072 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.293 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.297 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:50.340 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:50 +0000 (0:00:01.077) 0:02:32.827 ******** 2019-10-13T04:00:50.341 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:50.349 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.351 INFO:teuthology.task.ansible.out: 2019-10-13T04:00:50.470 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T04:00:50.471 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T04:00:50.529 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:50 +0000 (0:00:00.188) 0:02:33.016 ******** 2019-10-13T04:00:50.572 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-13T04:00:50.573 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-13T04:00:50.648 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:50 +0000 (0:00:00.114) 0:02:33.130 ******** 2019-10-13T04:00:50.649 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T04:00:50.651 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:50.740 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:50.798 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-13T04:00:50.799 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-13T04:00:50.855 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:50 +0000 (0:00:00.210) 0:02:33.341 ******** 2019-10-13T04:00:50.892 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T04:00:50.893 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:50.915 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:51.001 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.055 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.151 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.231 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.376) 0:02:33.718 ******** 2019-10-13T04:00:51.242 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.329 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.383 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-13T04:00:51.385 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T04:00:51.427 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.196) 0:02:33.915 ******** 2019-10-13T04:00:51.456 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-13T04:00:51.457 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-13T04:00:51.462 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-13T04:00:51.512 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.083) 0:02:33.999 ******** 2019-10-13T04:00:51.550 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.038) 0:02:34.037 ******** 2019-10-13T04:00:51.700 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-13T04:00:51.701 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T04:00:51.760 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.210) 0:02:34.247 ******** 2019-10-13T04:00:51.800 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-13T04:00:51.837 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:51 +0000 (0:00:00.076) 0:02:34.324 ******** 2019-10-13T04:00:52.028 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-10-13T04:00:52.030 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true} 2019-10-13T04:00:52.078 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:52 +0000 (0:00:00.241) 0:02:34.566 ******** 2019-10-13T04:00:52.084 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true} 2019-10-13T04:00:52.126 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:52 +0000 (0:00:00.047) 0:02:34.613 ******** 2019-10-13T04:00:52.582 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-13T04:00:52.584 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-13T04:00:52.643 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:52 +0000 (0:00:00.516) 0:02:35.130 ******** 2019-10-13T04:00:52.800 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-13T04:00:52.801 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T04:00:52.849 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:52 +0000 (0:00:00.205) 0:02:35.335 ******** 2019-10-13T04:00:52.854 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-13T04:00:52.855 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-13T04:00:52.898 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:52 +0000 (0:00:00.049) 0:02:35.385 ******** 2019-10-13T04:00:52.996 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-13T04:00:52.997 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T04:00:53.052 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.154) 0:02:35.539 ******** 2019-10-13T04:00:53.060 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-13T04:00:53.061 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T04:00:53.107 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.052) 0:02:35.592 ******** 2019-10-13T04:00:53.249 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-13T04:00:53.250 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-13T04:00:53.290 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.184) 0:02:35.777 ******** 2019-10-13T04:00:53.413 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-13T04:00:53.415 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-13T04:00:53.456 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.166) 0:02:35.943 ******** 2019-10-13T04:00:53.689 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-13T04:00:53.755 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.296) 0:02:36.240 ******** 2019-10-13T04:00:53.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-13T04:00:53.828 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-13T04:00:53.905 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:53 +0000 (0:00:00.151) 0:02:36.392 ******** 2019-10-13T04:00:54.158 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-13T04:00:54.234 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:54 +0000 (0:00:00.328) 0:02:36.721 ******** 2019-10-13T04:00:54.778 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-13T04:00:54.779 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T04:00:54.840 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:54 +0000 (0:00:00.606) 0:02:37.327 ******** 2019-10-13T04:00:55.022 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-13T04:00:55.022 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.038595", "end": "2019-10-13 04:00:55.005531", "rc": 0, "start": "2019-10-13 04:00:54.966936", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-13T04:00:55.068 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.227) 0:02:37.555 ******** 2019-10-13T04:00:55.136 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-13T04:00:55.138 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:55.181 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.112) 0:02:37.668 ******** 2019-10-13T04:00:55.187 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-13T04:00:55.188 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T04:00:55.230 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.049) 0:02:37.717 ******** 2019-10-13T04:00:55.235 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-13T04:00:55.236 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:55.280 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.049) 0:02:37.767 ******** 2019-10-13T04:00:55.423 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-13T04:00:55.424 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048455", "end": "2019-10-13 04:00:55.651099", "rc": 0, "start": "2019-10-13 04:00:55.602644", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-13T04:00:55.465 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.185) 0:02:37.952 ******** 2019-10-13T04:00:55.504 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-13T04:00:55.505 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:55.550 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.084) 0:02:38.037 ******** 2019-10-13T04:00:55.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-13T04:00:55.581 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:55.634 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:55 +0000 (0:00:00.084) 0:02:38.122 ******** 2019-10-13T04:00:56.130 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-10-13T04:00:56.131 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T04:00:56.170 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.536) 0:02:38.658 ******** 2019-10-13T04:00:56.198 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-13T04:00:56.199 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:56.237 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.066) 0:02:38.724 ******** 2019-10-13T04:00:56.424 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T04:00:56.482 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.244) 0:02:38.969 ******** 2019-10-13T04:00:56.510 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-13T04:00:56.511 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:56.558 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.076) 0:02:39.046 ******** 2019-10-13T04:00:56.604 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-13T04:00:56.645 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.086) 0:02:39.132 ******** 2019-10-13T04:00:56.672 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-13T04:00:56.674 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:56.713 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.068) 0:02:39.201 ******** 2019-10-13T04:00:56.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-13T04:00:56.896 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-13T04:00:56.942 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:56 +0000 (0:00:00.228) 0:02:39.430 ******** 2019-10-13T04:00:56.950 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-13T04:00:56.953 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 03:57:47 UTC", "ActiveEnterTimestampMonotonic": "26561519", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service systemd-journald.socket -.mount system.slice network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 03:57:39 UTC", "AssertTimestampMonotonic": "18072104", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 03:57:39 UTC", "ConditionTimestampMonotonic": "18072103", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-10-13T04:00:56.954 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1114", "ExecMainStartTimestamp": "Sun 2019-10-13 03:57:47 UTC", "ExecMainStartTimestampMonotonic": "26561492", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 03:57:39 UTC", "InactiveExitTimestampMonotonic": "18072591", "InvocationID": "cfefc1f871464b108a853b4bed6fa578", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-10-13T04:00:56.956 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1114", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 03:57:47 UTC", "StateChangeTimestampMonoton 2019-10-13T04:00:56.958 INFO:teuthology.task.ansible.out:ic": "26561519", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 03:57:47 UTC", "WatchdogTimestampMonotonic": "26561517", "WatchdogUSec": "0"}} 2019-10-13T04:00:57.007 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:57 +0000 (0:00:00.062) 0:02:39.493 ******** 2019-10-13T04:00:57.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-13T04:00:57.014 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:57.068 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:57 +0000 (0:00:00.062) 0:02:39.555 ******** 2019-10-13T04:00:57.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:57.132 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:57 +0000 (0:00:00.063) 0:02:39.619 ******** 2019-10-13T04:00:57.301 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-13T04:00:57.305 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2019-10-13 03:57:42 UTC", "ActiveEnterTimestampMonotonic": "25586862", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket systemd-tmpfiles-setup.service basic.target sysinit.target network.target system.slice -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2019-10-13 03:57:33 UTC", "AssertTimestampMonotonic": "17156102", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2019-10-13 03:57:33 UTC", "ConditionTimestampMonotonic": "17156102", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-10-13T04:00:57.307 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1135", "ExecMainStartTimestamp": "Sun 2019-10-13 03:57:42 UTC", "ExecMainStartTimestampMonotonic": "25586835", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2019-10-13 03:57:33 UTC", "InactiveExitTimestampMonotonic": "17156678", "InvocationID": "2742b1bcc03147b78917f1c4b6916017", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-10-13T04:00:57.308 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1135", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2019-10-13 03:57:42 UTC", "StateChangeTimestampMonoton 2019-10-13T04:00:57.310 INFO:teuthology.task.ansible.out:ic": "25586862", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2019-10-13 03:57:42 UTC", "WatchdogTimestampMonotonic": "25586860", "WatchdogUSec": "0"}} 2019-10-13T04:00:57.379 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:57 +0000 (0:00:00.247) 0:02:39.867 ******** 2019-10-13T04:00:57.406 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-13T04:00:57.408 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:00:57.454 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:57 +0000 (0:00:00.075) 0:02:39.942 ******** 2019-10-13T04:00:57.972 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-13T04:00:57.974 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"cache_update_time": 1570938847, "cache_updated": false, "changed": false} 2019-10-13T04:00:58.022 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:58 +0000 (0:00:00.566) 0:02:40.509 ******** 2019-10-13T04:00:58.170 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-13T04:00:58.171 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T04:00:58.252 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:58 +0000 (0:00:00.229) 0:02:40.738 ******** 2019-10-13T04:00:58.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-13T04:00:58.334 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1570938896, "cache_updated": false, "changed": false} 2019-10-13T04:00:58.385 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:58 +0000 (0:00:00.133) 0:02:40.872 ******** 2019-10-13T04:00:58.531 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-13T04:00:58.532 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-13T04:00:58.572 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:58 +0000 (0:00:00.187) 0:02:41.060 ******** 2019-10-13T04:00:58.598 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-13T04:00:58.598 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-13T04:00:58.663 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:58 +0000 (0:00:00.083) 0:02:41.143 ******** 2019-10-13T04:00:58.678 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:59.055 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-13T04:00:59.144 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:59 +0000 (0:00:00.487) 0:02:41.630 ******** 2019-10-13T04:00:59.177 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:00:59.681 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-13T04:00:59.682 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T04:00:59.732 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:00:59 +0000 (0:00:00.589) 0:02:42.220 ******** 2019-10-13T04:00:59.962 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-13T04:01:00.015 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.282) 0:02:42.502 ******** 2019-10-13T04:01:00.236 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi110.front.sepia.ceph.com] => {"changed": false} 2019-10-13T04:01:00.287 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.271) 0:02:42.774 ******** 2019-10-13T04:01:00.326 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T04:01:00.389 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.102) 0:02:42.876 ******** 2019-10-13T04:01:00.545 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1570938828.0093212, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1570938826.242205, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1570938826.242205, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi204.front.sepia.ceph.com] => {"changed": false} 2019-10-13T04:01:00.585 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.195) 0:02:43.072 ******** 2019-10-13T04:01:00.623 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.038) 0:02:43.110 ******** 2019-10-13T04:01:00.661 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-13T04:01:00.663 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-13T04:01:00.716 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.092) 0:02:43.203 ******** 2019-10-13T04:01:00.865 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-13T04:01:00.866 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1570938800.064657, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1570938798.2735527, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1570938798.2735527, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-10-13T04:01:00.923 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:00 +0000 (0:00:00.207) 0:02:43.410 ******** 2019-10-13T04:01:00.982 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-13T04:01:00.984 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-13T04:01:01.023 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.100) 0:02:43.510 ******** 2019-10-13T04:01:01.040 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:01:01.061 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi110.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T04:01:01.109 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.085) 0:02:43.596 ******** 2019-10-13T04:01:01.115 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-13T04:01:01.116 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-13T04:01:01.170 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.060) 0:02:43.657 ******** 2019-10-13T04:01:01.176 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-13T04:01:01.177 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.199 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-13T04:01:01.232 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.061) 0:02:43.719 ******** 2019-10-13T04:01:01.238 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-13T04:01:01.239 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-13T04:01:01.281 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.049) 0:02:43.768 ******** 2019-10-13T04:01:01.308 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-13T04:01:01.310 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.348 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.066) 0:02:43.835 ******** 2019-10-13T04:01:01.370 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T04:01:01.412 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.064) 0:02:43.899 ******** 2019-10-13T04:01:01.492 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-13T04:01:01.543 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.125) 0:02:44.025 ******** 2019-10-13T04:01:01.545 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi110.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T04:01:01.590 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.052) 0:02:44.078 ******** 2019-10-13T04:01:01.621 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.690 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.099) 0:02:44.177 ******** 2019-10-13T04:01:01.698 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-13T04:01:01.699 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-13T04:01:01.775 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.084) 0:02:44.262 ******** 2019-10-13T04:01:01.782 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi110.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.783 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.785 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.786 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.788 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.843 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.067) 0:02:44.330 ******** 2019-10-13T04:01:01.854 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.899 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.056) 0:02:44.386 ******** 2019-10-13T04:01:01.906 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-13T04:01:01.907 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.909 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.910 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.912 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.916 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:01.997 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:01 +0000 (0:00:00.097) 0:02:44.484 ******** 2019-10-13T04:01:02.008 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-13T04:01:02.010 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.012 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.014 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.060 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.063) 0:02:44.547 ******** 2019-10-13T04:01:02.068 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.124 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.057) 0:02:44.605 ******** 2019-10-13T04:01:02.127 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.138 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.139 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.150 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-13T04:01:02.152 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.201 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.075) 0:02:44.680 ******** 2019-10-13T04:01:02.235 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.041) 0:02:44.722 ******** 2019-10-13T04:01:02.250 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.288 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.051) 0:02:44.774 ******** 2019-10-13T04:01:02.294 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-13T04:01:02.295 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.349 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.061) 0:02:44.836 ******** 2019-10-13T04:01:02.357 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-13T04:01:02.359 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.407 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.057) 0:02:44.894 ******** 2019-10-13T04:01:02.414 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-13T04:01:02.416 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.459 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.052) 0:02:44.946 ******** 2019-10-13T04:01:02.466 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-13T04:01:02.468 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.526 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.066) 0:02:45.013 ******** 2019-10-13T04:01:02.536 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-13T04:01:02.537 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.596 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.069) 0:02:45.083 ******** 2019-10-13T04:01:02.605 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T04:01:02.605 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.661 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.065) 0:02:45.148 ******** 2019-10-13T04:01:02.672 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.726 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.064) 0:02:45.213 ******** 2019-10-13T04:01:02.736 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.791 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.065) 0:02:45.278 ******** 2019-10-13T04:01:02.797 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-13T04:01:02.799 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.856 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.064) 0:02:45.343 ******** 2019-10-13T04:01:02.862 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T04:01:02.864 INFO:teuthology.task.ansible.out:skipping: [smithi110.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.911 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.053) 0:02:45.396 ******** 2019-10-13T04:01:02.919 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T04:01:02.920 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:02.978 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:02 +0000 (0:00:00.065) 0:02:45.462 ******** 2019-10-13T04:01:03.016 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:03.063 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.088) 0:02:45.550 ******** 2019-10-13T04:01:03.090 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-13T04:01:03.092 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-13T04:01:03.134 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.069) 0:02:45.620 ******** 2019-10-13T04:01:03.139 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T04:01:03.139 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009525", "end": "2019-10-13 04:01:03.053867", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 04:01:03.044342", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.237 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004235", "end": "2019-10-13 04:01:03.203820", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 04:01:03.199585", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.301 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010908", "end": "2019-10-13 04:01:03.258637", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-13 04:01:03.247729", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.302 INFO:teuthology.task.ansible.out: 2019-10-13T04:01:03.381 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005434", "end": "2019-10-13 04:01:03.343906", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 04:01:03.338472", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.438 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004254", "end": "2019-10-13 04:01:03.402976", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-13 04:01:03.398722", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.511 INFO:teuthology.task.ansible.out:ok: [smithi110.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.008037", "end": "2019-10-13 04:01:03.475811", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 04:01:03.467774", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.560 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.427) 0:02:46.047 ******** 2019-10-13T04:01:03.569 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005163", "end": "2019-10-13 04:01:03.530286", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-13 04:01:03.525123", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.592 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T04:01:03.630 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.070) 0:02:46.118 ******** 2019-10-13T04:01:03.733 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-13T04:01:03.734 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.038488", "end": "2019-10-13 04:01:03.692665", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-13 04:01:03.654177", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-13T04:01:03.789 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.158) 0:02:46.276 ******** 2019-10-13T04:01:03.791 INFO:teuthology.task.ansible.out: 2019-10-13T04:01:03.801 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-13T04:01:03.802 INFO:teuthology.task.ansible.out:changed: [smithi110.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-13T04:01:03.856 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-13T04:01:03.919 INFO:teuthology.task.ansible.out:Sunday 13 October 2019 04:01:03 +0000 (0:00:00.129) 0:02:46.406 ******** 2019-10-13T04:01:04.041 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-13T04:01:04.042 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-13T04:01:04.169 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched PLAY [paddles] ***************************************************************** skipping: no hosts matched 2019-10-13T04:01:04.177 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi110.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 smithi204.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 Sunday 13 October 2019 04:01:04 +0000 (0:00:00.253) 0:02:46.660 ******** =============================================================================== testnode : Upgrade packages -------------------------------------------- 22.31s users : Create all admin users with sudo access. ----------------------- 18.09s users : Update authorized_keys using the keys repo --------------------- 14.21s testnode : Install packages -------------------------------------------- 11.39s testnode : Ensure packages are not present. ----------------------------- 8.57s testnode : Zap all non-root disks --------------------------------------- 7.29s Check for /usr/bin/python ----------------------------------------------- 1.98s users : Remove revoked users -------------------------------------------- 1.59s users : Update authorized_keys for each user with literal keys ---------- 1.55s ansible-managed : Add authorized keys for the ansible user. ------------- 1.53s Gathering Facts --------------------------------------------------------- 1.33s testnode : Update apt cache. -------------------------------------------- 1.13s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.10s testnode : Create logical volume(s) ------------------------------------- 1.07s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.99s testnode : Install python-apt ------------------------------------------- 0.94s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.83s Gat 2019-10-13T04:01:04.180 INFO:teuthology.task.ansible.out:hering Facts --------------------------------------------------------- 0.83s common : Set the hardware clock ----------------------------------------- 0.77s common : Update apt cache ----------------------------------------------- 0.72s 2019-10-13T04:01:04.446 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi204.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2019-10-13T04:01:04.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T04:01:04.449 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T04:01:04.534 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:04.535 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-13T04:01:04.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-13T04:01:04.744 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2019-10-13T04:01:04.832 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:04.832 INFO:teuthology.orchestra.run.smithi110:> true 2019-10-13T04:01:05.008 INFO:teuthology.run_tasks:Running task clock... 2019-10-13T04:01:05.023 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-10-13T04:01:05.023 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:05.023 INFO:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T04:01:05.095 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-13T04:01:05.096 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Command line: ntpd -gq 2019-10-13T04:01:05.096 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: proto: precision = 0.123 usec (-23) 2019-10-13T04:01:05.098 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T04:01:05.099 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: restrict ::: KOD does nothing without LIMITED. 2019-10-13T04:01:05.099 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen and drop on 0 v6wildcard [::]:123 2019-10-13T04:01:05.099 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-13T04:01:05.099 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen normally on 2 lo 127.0.0.1:123 2019-10-13T04:01:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2019-10-13T04:01:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen normally on 4 lo [::1]:123 2019-10-13T04:01:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2019-10-13T04:01:05.100 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:05 ntpd[7869]: Listening on routing socket on fd #22 for interface updates 2019-10-13T04:01:05.101 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T04:01:05.101 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-13T04:01:08.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:08 ntpd[7869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:08.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:08 ntpd[7869]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:08.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:08 ntpd[7869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:08.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:08 ntpd[7869]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:10.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:10 ntpd[7869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:10.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:10 ntpd[7869]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:10.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:10 ntpd[7869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:10.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:10 ntpd[7869]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:12.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:12 ntpd[7869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:12.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:12 ntpd[7869]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:12.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:12 ntpd[7869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:12.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:12 ntpd[7869]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:14.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:14 ntpd[7869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:14.088 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:14 ntpd[7869]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:14.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:14 ntpd[7869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:14.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:14 ntpd[7869]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:14.089 INFO:teuthology.orchestra.run.smithi204.stderr:13 Oct 04:01:14 ntpd[7869]: can't open /var/log/ntpstats/loopstats.20191013: Permission denied 2019-10-13T04:01:14.089 INFO:teuthology.orchestra.run.smithi204.stdout:13 Oct 04:01:14 ntpd[7869]: ntpd: time slew -0.000196 s 2019-10-13T04:01:14.089 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time slew -0.000196s 2019-10-13T04:01:14.137 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:14.138 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:14.139 INFO:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-13T04:01:14.201 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-13T04:01:14.201 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Command line: ntpd -gq 2019-10-13T04:01:14.202 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: proto: precision = 0.123 usec (-23) 2019-10-13T04:01:14.203 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T04:01:14.203 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-13T04:01:14.203 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: restrict ::: KOD does nothing without LIMITED. 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen and drop on 0 v6wildcard [::]:123 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen normally on 2 lo 127.0.0.1:123 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2019-10-13T04:01:14.204 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen normally on 4 lo [::1]:123 2019-10-13T04:01:14.205 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2019-10-13T04:01:14.205 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:14 ntpd[7880]: Listening on routing socket on fd #22 for interface updates 2019-10-13T04:01:19.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:19 ntpd[7880]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:19.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:19 ntpd[7880]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:19.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:19 ntpd[7880]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:19.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:19 ntpd[7880]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:21.201 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:21 ntpd[7880]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:21.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:21 ntpd[7880]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:21.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:21 ntpd[7880]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:21.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:21 ntpd[7880]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:23.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:23 ntpd[7880]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:23.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:23 ntpd[7880]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:23.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:23 ntpd[7880]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:23.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:23 ntpd[7880]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-13T04:01:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: can't open /var/log/ntpstats/rawstats.20191013: Permission denied 2019-10-13T04:01:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-13T04:01:25.202 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: can't open /var/log/ntpstats/peerstats.20191013: Permission denied 2019-10-13T04:01:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2019-10-13T04:01:25.203 INFO:teuthology.orchestra.run.smithi110.stderr:13 Oct 04:01:25 ntpd[7880]: can't open /var/log/ntpstats/loopstats.20191013: Permission denied 2019-10-13T04:01:25.203 INFO:teuthology.orchestra.run.smithi110.stdout:13 Oct 04:01:25 ntpd[7880]: ntpd: time slew -0.000270 s 2019-10-13T04:01:25.203 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew -0.000270s 2019-10-13T04:01:25.253 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2019-10-13T04:01:25.254 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2019-10-13T04:01:25.254 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:25.254 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:25.254 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:25.254 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-13T04:01:25.255 INFO:teuthology.run_tasks:Running task install... 2019-10-13T04:01:25.267 DEBUG:teuthology.task.install:project ceph 2019-10-13T04:01:25.267 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}} 2019-10-13T04:01:25.267 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T04:01:25.268 INFO:teuthology.task.install:Using flavor: basic 2019-10-13T04:01:25.273 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-10-13T04:01:25.273 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2019-10-13T04:01:25.273 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:25.274 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-13T04:01:25.329 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-13T04:01:25.333 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:01:25.334 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:25.334 INFO:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2019-10-13T04:01:25.334 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:25.335 INFO:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2019-10-13T04:01:25.387 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2019-10-13T04:01:25.393 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:01:25.393 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:25.393 INFO:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2019-10-13T04:01:25.425 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-13T04:01:25.472 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-13T04:01:25.472 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2019-10-13T04:01:25.473 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:25.473 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-13T04:01:25.525 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-13T04:01:25.532 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:01:25.532 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-10-13T04:01:25.533 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-10-13T04:01:25.533 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:01:25.554 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-13T04:01:25.603 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-13T04:01:25.604 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2019-10-13T04:01:25.607 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:25.617 INFO:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2019-10-13T04:01:25.680 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2019-10-13T04:01:25.685 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:01:25.685 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-10-13T04:01:25.685 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-10-13T04:01:25.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:01:25.711 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default/ 2019-10-13T04:01:25.713 INFO:teuthology.task.install.deb:Package version is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:01:25.832 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:25.833 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-13T04:01:25.884 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:25.884 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-13T04:01:25.890 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default/ 2019-10-13T04:01:25.890 INFO:teuthology.task.install.deb:Package version is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:01:26.014 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:26.015 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-13T04:01:26.065 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:26.066 INFO:teuthology.orchestra.run.smithi110:> sudo apt-get update 2019-10-13T04:01:26.100 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T04:01:26.168 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic InRelease 2019-10-13T04:01:26.189 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-13T04:01:26.211 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-13T04:01:26.219 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T04:01:26.232 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,858 B] 2019-10-13T04:01:26.264 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-13T04:01:26.351 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic InRelease 2019-10-13T04:01:26.372 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-13T04:01:26.377 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T04:01:26.397 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-13T04:01:26.397 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-13T04:01:26.398 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-13T04:01:26.419 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,858 B] 2019-10-13T04:01:26.450 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-13T04:01:26.538 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T04:01:26.547 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-13T04:01:26.708 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-13T04:01:26.739 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 26.8 kB in 1s (43.9 kB/s) 2019-10-13T04:01:26.944 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 26.8 kB in 1s (41.4 kB/s) 2019-10-13T04:01:27.869 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T04:01:27.916 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:01:27.916 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6132-ga8ee8ac-1bionic ceph-daemon=15.0.0-6132-ga8ee8ac-1bionic ceph-mds=15.0.0-6132-ga8ee8ac-1bionic ceph-mgr=15.0.0-6132-ga8ee8ac-1bionic ceph-common=15.0.0-6132-ga8ee8ac-1bionic ceph-fuse=15.0.0-6132-ga8ee8ac-1bionic ceph-test=15.0.0-6132-ga8ee8ac-1bionic radosgw=15.0.0-6132-ga8ee8ac-1bionic python-ceph=15.0.0-6132-ga8ee8ac-1bionic libcephfs2=15.0.0-6132-ga8ee8ac-1bionic libcephfs-dev=15.0.0-6132-ga8ee8ac-1bionic librados2=15.0.0-6132-ga8ee8ac-1bionic librbd1=15.0.0-6132-ga8ee8ac-1bionic rbd-fuse=15.0.0-6132-ga8ee8ac-1bionic python3-cephfs=15.0.0-6132-ga8ee8ac-1bionic cephfs-shell=15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:01:27.994 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T04:01:28.064 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T04:01:28.108 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:01:28.108 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6132-ga8ee8ac-1bionic ceph-daemon=15.0.0-6132-ga8ee8ac-1bionic ceph-mds=15.0.0-6132-ga8ee8ac-1bionic ceph-mgr=15.0.0-6132-ga8ee8ac-1bionic ceph-common=15.0.0-6132-ga8ee8ac-1bionic ceph-fuse=15.0.0-6132-ga8ee8ac-1bionic ceph-test=15.0.0-6132-ga8ee8ac-1bionic radosgw=15.0.0-6132-ga8ee8ac-1bionic python-ceph=15.0.0-6132-ga8ee8ac-1bionic libcephfs2=15.0.0-6132-ga8ee8ac-1bionic libcephfs-dev=15.0.0-6132-ga8ee8ac-1bionic librados2=15.0.0-6132-ga8ee8ac-1bionic librbd1=15.0.0-6132-ga8ee8ac-1bionic rbd-fuse=15.0.0-6132-ga8ee8ac-1bionic python3-cephfs=15.0.0-6132-ga8ee8ac-1bionic cephfs-shell=15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:01:28.190 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T04:01:28.237 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T04:01:28.238 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T04:01:28.386 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2019-10-13T04:01:28.387 INFO:teuthology.orchestra.run.smithi204.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T04:01:28.387 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T04:01:28.387 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-13T04:01:28.392 INFO:teuthology.orchestra.run.smithi204.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-13T04:01:28.392 INFO:teuthology.orchestra.run.smithi204.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-13T04:01:28.392 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T04:01:28.392 INFO:teuthology.orchestra.run.smithi204.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-13T04:01:28.392 INFO:teuthology.orchestra.run.smithi204.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T04:01:28.393 INFO:teuthology.orchestra.run.smithi204.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T04:01:28.394 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados runc socat ubuntu-fan xmlstarlet 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-13T04:01:28.395 INFO:teuthology.orchestra.run.smithi204.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-13T04:01:28.396 INFO:teuthology.orchestra.run.smithi204.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-13T04:01:28.396 INFO:teuthology.orchestra.run.smithi204.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-13T04:01:28.397 INFO:teuthology.orchestra.run.smithi204.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-13T04:01:28.398 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-13T04:01:28.398 INFO:teuthology.orchestra.run.smithi204.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-13T04:01:28.398 INFO:teuthology.orchestra.run.smithi204.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-13T04:01:28.398 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-13T04:01:28.398 INFO:teuthology.orchestra.run.smithi204.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-13T04:01:28.399 INFO:teuthology.orchestra.run.smithi204.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-10-13T04:01:28.435 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T04:01:28.435 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T04:01:28.494 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2019-10-13T04:01:28.494 INFO:teuthology.orchestra.run.smithi204.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-13T04:01:28.494 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T04:01:28.494 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T04:01:28.495 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon ceph-osd ceph-test cephfs-shell cgroupfs-mount containerd 2019-10-13T04:01:28.495 INFO:teuthology.orchestra.run.smithi204.stdout: dns-root-data dnsmasq-base docker.io formencode-i18n jq libcephfs-dev 2019-10-13T04:01:28.495 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-10-13T04:01:28.495 INFO:teuthology.orchestra.run.smithi204.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librgw2 2019-10-13T04:01:28.496 INFO:teuthology.orchestra.run.smithi204.stdout: libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T04:01:28.496 INFO:teuthology.orchestra.run.smithi204.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-10-13T04:01:28.496 INFO:teuthology.orchestra.run.smithi204.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-10-13T04:01:28.496 INFO:teuthology.orchestra.run.smithi204.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-10-13T04:01:28.496 INFO:teuthology.orchestra.run.smithi204.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-10-13T04:01:28.497 INFO:teuthology.orchestra.run.smithi204.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T04:01:28.497 INFO:teuthology.orchestra.run.smithi204.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T04:01:28.497 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T04:01:28.497 INFO:teuthology.orchestra.run.smithi204.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T04:01:28.498 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T04:01:28.498 INFO:teuthology.orchestra.run.smithi204.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T04:01:28.498 INFO:teuthology.orchestra.run.smithi204.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T04:01:28.498 INFO:teuthology.orchestra.run.smithi204.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T04:01:28.499 INFO:teuthology.orchestra.run.smithi204.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T04:01:28.499 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T04:01:28.499 INFO:teuthology.orchestra.run.smithi204.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T04:01:28.499 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-10-13T04:01:28.500 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip python3-rados radosgw rbd-fuse runc socat ubuntu-fan 2019-10-13T04:01:28.500 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet 2019-10-13T04:01:28.501 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2019-10-13T04:01:28.501 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 librbd1 2019-10-13T04:01:28.595 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2019-10-13T04:01:28.595 INFO:teuthology.orchestra.run.smithi110.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T04:01:28.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T04:01:28.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-13T04:01:28.597 INFO:teuthology.orchestra.run.smithi110.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-13T04:01:28.598 INFO:teuthology.orchestra.run.smithi110.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-13T04:01:28.598 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T04:01:28.598 INFO:teuthology.orchestra.run.smithi110.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-13T04:01:28.599 INFO:teuthology.orchestra.run.smithi110.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-13T04:01:28.599 INFO:teuthology.orchestra.run.smithi110.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-13T04:01:28.599 INFO:teuthology.orchestra.run.smithi110.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-13T04:01:28.599 INFO:teuthology.orchestra.run.smithi110.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T04:01:28.599 INFO:teuthology.orchestra.run.smithi110.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T04:01:28.600 INFO:teuthology.orchestra.run.smithi110.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T04:01:28.600 INFO:teuthology.orchestra.run.smithi110.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T04:01:28.600 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T04:01:28.600 INFO:teuthology.orchestra.run.smithi110.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T04:01:28.600 INFO:teuthology.orchestra.run.smithi110.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T04:01:28.601 INFO:teuthology.orchestra.run.smithi110.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T04:01:28.601 INFO:teuthology.orchestra.run.smithi110.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T04:01:28.601 INFO:teuthology.orchestra.run.smithi110.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T04:01:28.602 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T04:01:28.602 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-10-13T04:01:28.602 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados runc socat ubuntu-fan xmlstarlet 2019-10-13T04:01:28.603 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2019-10-13T04:01:28.603 INFO:teuthology.orchestra.run.smithi110.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-13T04:01:28.603 INFO:teuthology.orchestra.run.smithi110.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-13T04:01:28.603 INFO:teuthology.orchestra.run.smithi110.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-13T04:01:28.604 INFO:teuthology.orchestra.run.smithi110.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-13T04:01:28.605 INFO:teuthology.orchestra.run.smithi110.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-13T04:01:28.605 INFO:teuthology.orchestra.run.smithi110.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-13T04:01:28.605 INFO:teuthology.orchestra.run.smithi110.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-10-13T04:01:28.642 INFO:teuthology.orchestra.run.smithi204.stdout:2 upgraded, 119 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T04:01:28.642 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 183 MB of archives. 2019-10-13T04:01:28.642 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 869 MB of additional disk space will be used. 2019-10-13T04:01:28.642 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6132-ga8ee8ac-1bionic [1,489 kB] 2019-10-13T04:01:28.685 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-13T04:01:28.702 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2019-10-13T04:01:28.702 INFO:teuthology.orchestra.run.smithi110.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-13T04:01:28.702 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-13T04:01:28.702 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-13T04:01:28.702 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon ceph-osd ceph-test cephfs-shell cgroupfs-mount containerd 2019-10-13T04:01:28.703 INFO:teuthology.orchestra.run.smithi110.stdout: dns-root-data dnsmasq-base docker.io formencode-i18n jq libcephfs-dev 2019-10-13T04:01:28.704 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-10-13T04:01:28.705 INFO:teuthology.orchestra.run.smithi110.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librgw2 2019-10-13T04:01:28.705 INFO:teuthology.orchestra.run.smithi110.stdout: libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-13T04:01:28.705 INFO:teuthology.orchestra.run.smithi110.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-10-13T04:01:28.705 INFO:teuthology.orchestra.run.smithi110.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-10-13T04:01:28.705 INFO:teuthology.orchestra.run.smithi110.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-10-13T04:01:28.706 INFO:teuthology.orchestra.run.smithi110.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-10-13T04:01:28.706 INFO:teuthology.orchestra.run.smithi110.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-10-13T04:01:28.706 INFO:teuthology.orchestra.run.smithi110.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-13T04:01:28.706 INFO:teuthology.orchestra.run.smithi110.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-13T04:01:28.706 INFO:teuthology.orchestra.run.smithi110.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-13T04:01:28.707 INFO:teuthology.orchestra.run.smithi110.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-13T04:01:28.708 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-10-13T04:01:28.708 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-10-13T04:01:28.708 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip python3-rados radosgw rbd-fuse runc socat ubuntu-fan 2019-10-13T04:01:28.708 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet 2019-10-13T04:01:28.708 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2019-10-13T04:01:28.709 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 librbd1 2019-10-13T04:01:28.785 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6132-ga8ee8ac-1bionic [2,966 kB] 2019-10-13T04:01:28.846 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-10-13T04:01:28.874 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6132-ga8ee8ac-1bionic [426 kB] 2019-10-13T04:01:28.888 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [303 kB] 2019-10-13T04:01:28.895 INFO:teuthology.orchestra.run.smithi110.stdout:2 upgraded, 119 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T04:01:28.895 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 183 MB of archives. 2019-10-13T04:01:28.896 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 869 MB of additional disk space will be used. 2019-10-13T04:01:28.896 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-13T04:01:28.897 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.2 kB] 2019-10-13T04:01:28.898 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [111 kB] 2019-10-13T04:01:28.902 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6132-ga8ee8ac-1bionic [10.5 kB] 2019-10-13T04:01:28.902 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6132-ga8ee8ac-1bionic [248 kB] 2019-10-13T04:01:28.911 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6132-ga8ee8ac-1bionic [4,490 kB] 2019-10-13T04:01:28.947 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6132-ga8ee8ac-1bionic [1,489 kB] 2019-10-13T04:01:29.058 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-10-13T04:01:29.062 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6132-ga8ee8ac-1bionic [98.8 kB] 2019-10-13T04:01:29.065 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6132-ga8ee8ac-1bionic [329 kB] 2019-10-13T04:01:29.077 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6132-ga8ee8ac-1bionic [17.8 MB] 2019-10-13T04:01:29.182 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-10-13T04:01:29.344 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-10-13T04:01:29.383 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-10-13T04:01:29.545 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-10-13T04:01:29.586 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-13T04:01:29.694 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6132-ga8ee8ac-1bionic [4,972 kB] 2019-10-13T04:01:29.813 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-13T04:01:29.867 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6132-ga8ee8ac-1bionic [1,324 kB] 2019-10-13T04:01:29.909 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6132-ga8ee8ac-1bionic [5,895 kB] 2019-10-13T04:01:29.934 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-13T04:01:30.092 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6132-ga8ee8ac-1bionic [2,966 kB] 2019-10-13T04:01:30.116 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6132-ga8ee8ac-1bionic [22.4 MB] 2019-10-13T04:01:30.150 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-13T04:01:30.175 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-13T04:01:30.391 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-13T04:01:30.626 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-13T04:01:30.837 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-13T04:01:30.867 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-13T04:01:30.926 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,800 B] 2019-10-13T04:01:30.926 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6132-ga8ee8ac-1bionic [12.5 kB] 2019-10-13T04:01:30.926 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [503 kB] 2019-10-13T04:01:30.937 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6132-ga8ee8ac-1bionic [1,822 kB] 2019-10-13T04:01:31.005 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6132-ga8ee8ac-1bionic [3,353 kB] 2019-10-13T04:01:31.078 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-13T04:01:31.105 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6132-ga8ee8ac-1bionic [426 kB] 2019-10-13T04:01:31.121 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-13T04:01:31.132 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6132-ga8ee8ac-1bionic [34.3 kB] 2019-10-13T04:01:31.133 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6132-ga8ee8ac-1bionic [1,093 kB] 2019-10-13T04:01:31.171 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6132-ga8ee8ac-1bionic [3,912 B] 2019-10-13T04:01:31.178 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6132-ga8ee8ac-1bionic [13.6 kB] 2019-10-13T04:01:31.178 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6132-ga8ee8ac-1bionic [12.8 kB] 2019-10-13T04:01:31.179 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6132-ga8ee8ac-1bionic [35.3 MB] 2019-10-13T04:01:31.245 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [303 kB] 2019-10-13T04:01:31.320 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.2 kB] 2019-10-13T04:01:31.321 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [111 kB] 2019-10-13T04:01:31.324 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6132-ga8ee8ac-1bionic [10.5 kB] 2019-10-13T04:01:31.324 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6132-ga8ee8ac-1bionic [248 kB] 2019-10-13T04:01:31.340 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-13T04:01:31.362 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-13T04:01:31.398 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6132-ga8ee8ac-1bionic [4,490 kB] 2019-10-13T04:01:31.581 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-13T04:01:31.803 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-13T04:01:32.009 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-13T04:01:32.124 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-13T04:01:32.264 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6132-ga8ee8ac-1bionic [98.8 kB] 2019-10-13T04:01:32.313 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6132-ga8ee8ac-1bionic [329 kB] 2019-10-13T04:01:32.333 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-13T04:01:32.372 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6132-ga8ee8ac-1bionic [17.8 MB] 2019-10-13T04:01:32.393 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-13T04:01:32.569 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [320 kB] 2019-10-13T04:01:32.574 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-13T04:01:32.576 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.5 kB] 2019-10-13T04:01:32.576 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [115 kB] 2019-10-13T04:01:32.581 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-6132-ga8ee8ac-1bionic [14.3 kB] 2019-10-13T04:01:32.581 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6132-ga8ee8ac-1bionic [17.3 kB] 2019-10-13T04:01:32.582 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,856 B] 2019-10-13T04:01:32.582 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6132-ga8ee8ac-1bionic [5,151 kB] 2019-10-13T04:01:32.785 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [61.9 kB] 2019-10-13T04:01:32.837 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-13T04:01:33.033 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-13T04:01:33.078 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-13T04:01:33.269 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-13T04:01:33.405 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-13T04:01:33.591 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-13T04:01:33.647 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-13T04:01:33.664 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6132-ga8ee8ac-1bionic [4,972 kB] 2019-10-13T04:01:33.809 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-13T04:01:33.832 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-13T04:01:33.937 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6132-ga8ee8ac-1bionic [1,324 kB] 2019-10-13T04:01:33.994 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-13T04:01:34.001 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6132-ga8ee8ac-1bionic [5,895 kB] 2019-10-13T04:01:34.155 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-13T04:01:34.309 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6132-ga8ee8ac-1bionic [22.4 MB] 2019-10-13T04:01:34.317 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-13T04:01:34.334 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-13T04:01:34.484 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-13T04:01:34.495 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-13T04:01:34.657 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-13T04:01:34.841 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-13T04:01:34.985 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-13T04:01:35.178 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-13T04:01:35.333 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-13T04:01:35.448 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,800 B] 2019-10-13T04:01:35.449 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6132-ga8ee8ac-1bionic [12.5 kB] 2019-10-13T04:01:35.449 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [503 kB] 2019-10-13T04:01:35.470 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6132-ga8ee8ac-1bionic [1,822 kB] 2019-10-13T04:01:35.504 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-13T04:01:35.564 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6132-ga8ee8ac-1bionic [3,353 kB] 2019-10-13T04:01:35.692 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-13T04:01:35.729 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6132-ga8ee8ac-1bionic [34.3 kB] 2019-10-13T04:01:35.730 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6132-ga8ee8ac-1bionic [1,093 kB] 2019-10-13T04:01:35.786 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6132-ga8ee8ac-1bionic [3,912 B] 2019-10-13T04:01:35.786 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6132-ga8ee8ac-1bionic [13.6 kB] 2019-10-13T04:01:35.787 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6132-ga8ee8ac-1bionic [12.8 kB] 2019-10-13T04:01:35.787 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6132-ga8ee8ac-1bionic [35.3 MB] 2019-10-13T04:01:35.829 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-13T04:01:36.030 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-13T04:01:36.210 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-13T04:01:36.373 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-13T04:01:36.414 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-13T04:01:36.566 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-13T04:01:36.733 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-13T04:01:36.907 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-13T04:01:36.975 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-13T04:01:37.150 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-13T04:01:37.429 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-6132-ga8ee8ac-1bionic [320 kB] 2019-10-13T04:01:37.435 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-6132-ga8ee8ac-1bionic [20.5 kB] 2019-10-13T04:01:37.439 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-6132-ga8ee8ac-1bionic [115 kB] 2019-10-13T04:01:37.442 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-6132-ga8ee8ac-1bionic [14.3 kB] 2019-10-13T04:01:37.443 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6132-ga8ee8ac-1bionic [17.3 kB] 2019-10-13T04:01:37.443 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6132-ga8ee8ac-1bionic [3,856 B] 2019-10-13T04:01:37.449 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6132-ga8ee8ac-1bionic [5,151 kB] 2019-10-13T04:01:37.563 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-13T04:01:37.670 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191012.025926/a8ee8ac95ca822967c9dffe2a705f1d5502812bd/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6132-ga8ee8ac-1bionic [61.9 kB] 2019-10-13T04:01:37.724 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-13T04:01:37.725 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-13T04:01:37.886 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-13T04:01:37.966 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-13T04:01:38.128 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-13T04:01:38.208 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-13T04:01:38.369 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-13T04:01:38.543 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-13T04:01:38.758 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-13T04:01:38.983 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-13T04:01:39.183 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-13T04:01:39.329 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-13T04:01:39.561 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-13T04:01:39.682 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-13T04:01:39.883 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-13T04:01:40.015 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-13T04:01:40.205 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-13T04:01:40.465 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-13T04:01:40.649 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-13T04:01:41.208 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-13T04:01:41.390 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-13T04:01:42.456 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-13T04:01:42.652 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-13T04:01:44.037 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-13T04:01:44.199 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-13T04:01:44.217 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-13T04:01:44.379 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-13T04:01:44.557 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-13T04:01:44.715 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-13T04:01:44.920 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-13T04:01:45.077 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-13T04:01:45.256 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-13T04:01:45.418 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-13T04:01:45.428 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-13T04:01:45.589 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-13T04:01:46.397 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-13T04:01:46.573 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-13T04:01:47.132 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-13T04:01:47.292 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-13T04:01:47.543 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-13T04:01:47.706 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-13T04:01:48.251 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-13T04:01:48.412 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-13T04:01:48.421 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-13T04:01:48.574 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-13T04:01:48.582 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-13T04:01:48.744 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-13T04:01:48.816 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-13T04:01:48.987 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-13T04:01:49.138 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-13T04:01:49.316 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-13T04:01:49.379 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-13T04:01:49.569 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-13T04:01:49.640 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-13T04:01:49.811 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-13T04:01:50.281 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-13T04:01:50.441 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-13T04:01:50.648 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-13T04:01:50.770 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-13T04:01:50.890 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-13T04:01:51.092 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-13T04:01:51.314 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-13T04:01:51.520 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-13T04:01:51.774 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-13T04:01:51.986 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-13T04:01:52.186 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-13T04:01:52.408 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-13T04:01:52.595 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-13T04:01:52.820 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-13T04:01:52.920 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-13T04:01:53.143 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-13T04:01:53.463 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-13T04:01:53.691 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-13T04:01:53.905 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-13T04:01:54.139 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-13T04:01:54.459 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-10-13T04:01:54.700 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-13T04:01:54.713 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-10-13T04:01:54.861 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-13T04:01:54.954 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-13T04:01:55.023 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-13T04:01:55.116 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-13T04:01:55.277 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-13T04:01:55.550 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-13T04:01:55.792 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-13T04:01:55.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-13T04:01:55.954 INFO:teuthology.orchestra.run.smithi204.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-13T04:01:56.042 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-13T04:01:56.115 INFO:teuthology.orchestra.run.smithi204.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-13T04:01:56.203 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-13T04:01:56.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-13T04:01:56.465 INFO:teuthology.orchestra.run.smithi204.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-13T04:01:56.722 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-13T04:01:57.137 INFO:teuthology.orchestra.run.smithi204.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-13T04:01:57.427 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-13T04:01:57.541 INFO:teuthology.orchestra.run.smithi204.stdout:Get:114 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-13T04:01:57.830 INFO:teuthology.orchestra.run.smithi110.stdout:Get:114 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-13T04:01:57.877 INFO:teuthology.orchestra.run.smithi204.stdout:Get:115 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-13T04:01:58.039 INFO:teuthology.orchestra.run.smithi204.stdout:Get:116 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-13T04:01:58.166 INFO:teuthology.orchestra.run.smithi110.stdout:Get:115 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-13T04:01:58.328 INFO:teuthology.orchestra.run.smithi110.stdout:Get:116 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-13T04:01:58.364 INFO:teuthology.orchestra.run.smithi204.stdout:Get:117 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-13T04:01:58.605 INFO:teuthology.orchestra.run.smithi204.stdout:Get:118 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-13T04:01:58.689 INFO:teuthology.orchestra.run.smithi110.stdout:Get:117 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-13T04:01:58.930 INFO:teuthology.orchestra.run.smithi110.stdout:Get:118 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-13T04:01:59.034 INFO:teuthology.orchestra.run.smithi204.stdout:Get:119 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-13T04:01:59.275 INFO:teuthology.orchestra.run.smithi204.stdout:Get:120 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-13T04:01:59.358 INFO:teuthology.orchestra.run.smithi110.stdout:Get:119 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-13T04:01:59.437 INFO:teuthology.orchestra.run.smithi204.stdout:Get:121 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-13T04:01:59.599 INFO:teuthology.orchestra.run.smithi110.stdout:Get:120 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-13T04:01:59.761 INFO:teuthology.orchestra.run.smithi110.stdout:Get:121 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-13T04:02:00.253 INFO:teuthology.orchestra.run.smithi204.stdout:Preconfiguring packages ... 2019-10-13T04:02:00.407 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 183 MB in 31s (5,893 kB/s) 2019-10-13T04:02:00.553 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-uritemplate. 2019-10-13T04:02:00.576 INFO:teuthology.orchestra.run.smithi110.stdout:Preconfiguring packages ... 2019-10-13T04:02:00.602 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-13T04:02:00.606 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-13T04:02:00.643 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T04:02:00.741 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 183 MB in 31s (5,871 kB/s) 2019-10-13T04:02:00.861 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-uritemplate. 2019-10-13T04:02:00.905 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyparsing. 2019-10-13T04:02:00.912 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-13T04:02:00.916 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-13T04:02:00.918 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-10-13T04:02:00.955 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T04:02:00.959 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T04:02:01.253 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyparsing. 2019-10-13T04:02:01.267 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-10-13T04:02:01.312 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T04:02:01.648 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyperclip. 2019-10-13T04:02:01.660 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-10-13T04:02:01.698 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-10-13T04:02:01.933 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyperclip. 2019-10-13T04:02:01.946 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-10-13T04:02:01.984 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cmd2. 2019-10-13T04:02:01.991 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-10-13T04:02:01.998 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-10-13T04:02:02.043 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T04:02:02.285 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cmd2. 2019-10-13T04:02:02.293 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-10-13T04:02:02.295 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package pigz. 2019-10-13T04:02:02.311 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-pigz_2.4-1_amd64.deb ... 2019-10-13T04:02:02.338 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T04:02:02.354 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking pigz (2.4-1) ... 2019-10-13T04:02:02.580 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package pigz. 2019-10-13T04:02:02.593 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-pigz_2.4-1_amd64.deb ... 2019-10-13T04:02:02.622 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking pigz (2.4-1) ... 2019-10-13T04:02:02.715 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package bridge-utils. 2019-10-13T04:02:02.730 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-13T04:02:02.774 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-13T04:02:02.958 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package bridge-utils. 2019-10-13T04:02:02.966 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-13T04:02:03.008 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-13T04:02:03.111 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-13T04:02:03.127 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-13T04:02:03.211 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T04:02:03.353 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-13T04:02:03.370 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-13T04:02:03.445 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T04:02:03.554 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-librbd1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:03.682 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T04:02:03.790 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-librbd1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:03.907 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T04:02:04.208 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-librados2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:04.312 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T04:02:04.434 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-librados2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:04.571 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (15.0.0-6132-ga8ee8ac-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-13T04:02:04.948 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2019-10-13T04:02:04.963 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-libcephfs2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:05.007 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.191 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2019-10-13T04:02:05.204 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-libcephfs2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:05.241 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.309 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-rados. 2019-10-13T04:02:05.322 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-python-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:05.360 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.519 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rados. 2019-10-13T04:02:05.533 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-python-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:05.577 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.620 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-13T04:02:05.635 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-python-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:05.671 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.873 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-13T04:02:05.888 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-python-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:05.932 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:05.964 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-cephfs. 2019-10-13T04:02:05.979 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-python-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:06.023 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:06.263 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-cephfs. 2019-10-13T04:02:06.273 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-python-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:06.300 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-ceph-common. 2019-10-13T04:02:06.309 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:06.315 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-python-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:06.359 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:06.578 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph-common. 2019-10-13T04:02:06.591 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-python-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:06.628 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:06.670 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-prettytable. 2019-10-13T04:02:06.683 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-python-prettytable_0.7.2-3_all.deb ... 2019-10-13T04:02:06.728 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-13T04:02:06.948 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-rbd. 2019-10-13T04:02:06.948 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-prettytable. 2019-10-13T04:02:06.962 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-python-prettytable_0.7.2-3_all.deb ... 2019-10-13T04:02:06.963 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-python-rbd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:07.006 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:07.007 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-13T04:02:07.254 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rbd. 2019-10-13T04:02:07.271 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python-rbd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:07.304 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:07.369 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-13T04:02:07.391 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-13T04:02:07.432 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T04:02:07.620 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-13T04:02:07.636 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-13T04:02:07.674 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T04:02:07.737 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2019-10-13T04:02:07.751 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-librgw2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:07.795 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:07.981 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2019-10-13T04:02:07.996 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-librgw2_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:08.042 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:08.389 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-rgw. 2019-10-13T04:02:08.404 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-python-rgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:08.444 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:08.659 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rgw. 2019-10-13T04:02:08.673 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-python-rgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:08.717 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:08.759 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-13T04:02:08.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-13T04:02:08.808 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T04:02:08.987 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-13T04:02:09.000 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-13T04:02:09.029 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T04:02:09.120 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-13T04:02:09.134 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-13T04:02:09.178 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T04:02:09.332 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-13T04:02:09.346 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-13T04:02:09.390 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T04:02:09.498 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0. 2019-10-13T04:02:09.511 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-liboath0_2.6.1-1_amd64.deb ... 2019-10-13T04:02:09.548 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-13T04:02:09.760 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0. 2019-10-13T04:02:09.774 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-liboath0_2.6.1-1_amd64.deb ... 2019-10-13T04:02:09.822 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-13T04:02:09.859 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2019-10-13T04:02:09.872 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-libradosstriper1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:09.909 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:10.113 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2019-10-13T04:02:10.118 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-libradosstriper1_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:10.154 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:10.345 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2019-10-13T04:02:10.359 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:10.404 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:10.583 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2019-10-13T04:02:10.598 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-ceph-common_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:10.641 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:12.051 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2019-10-13T04:02:12.067 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-ceph-base_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:12.118 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:12.305 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2019-10-13T04:02:12.320 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-ceph-base_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:12.364 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:13.014 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-bcrypt. 2019-10-13T04:02:13.037 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-13T04:02:13.072 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-13T04:02:13.310 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-bcrypt. 2019-10-13T04:02:13.325 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-13T04:02:13.368 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-13T04:02:13.417 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-13T04:02:13.431 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-13T04:02:13.475 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-13T04:02:13.713 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-13T04:02:13.726 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-13T04:02:13.771 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-13T04:02:13.904 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-jwt. 2019-10-13T04:02:13.919 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-13T04:02:13.962 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-13T04:02:14.175 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-jwt. 2019-10-13T04:02:14.191 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-13T04:02:14.234 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-13T04:02:14.257 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-openssl. 2019-10-13T04:02:14.270 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-13T04:02:14.299 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T04:02:14.545 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-openssl. 2019-10-13T04:02:14.560 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-13T04:02:14.603 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T04:02:14.621 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-logutils. 2019-10-13T04:02:14.632 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-python-logutils_0.3.3-5_all.deb ... 2019-10-13T04:02:14.668 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-13T04:02:14.915 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-logutils. 2019-10-13T04:02:14.929 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-python-logutils_0.3.3-5_all.deb ... 2019-10-13T04:02:14.973 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-13T04:02:14.979 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-markupsafe. 2019-10-13T04:02:14.992 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-13T04:02:15.029 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-13T04:02:15.276 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-markupsafe. 2019-10-13T04:02:15.290 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-13T04:02:15.334 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-13T04:02:15.357 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-mako. 2019-10-13T04:02:15.371 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-13T04:02:15.416 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-13T04:02:15.629 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-mako. 2019-10-13T04:02:15.643 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-13T04:02:15.687 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-13T04:02:15.761 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-13T04:02:15.775 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-13T04:02:15.819 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-13T04:02:16.024 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-13T04:02:16.038 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-13T04:02:16.082 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-13T04:02:16.105 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-singledispatch. 2019-10-13T04:02:16.118 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-13T04:02:16.155 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-13T04:02:16.368 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-singledispatch. 2019-10-13T04:02:16.384 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-13T04:02:16.427 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-13T04:02:16.474 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-simplejson. 2019-10-13T04:02:16.488 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-13T04:02:16.524 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-13T04:02:16.747 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-simplejson. 2019-10-13T04:02:16.761 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-13T04:02:16.799 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-13T04:02:16.873 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-webob. 2019-10-13T04:02:16.889 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-13T04:02:16.938 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T04:02:17.158 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webob. 2019-10-13T04:02:17.174 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-13T04:02:17.217 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T04:02:17.260 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-bs4. 2019-10-13T04:02:17.275 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python-bs4_4.6.0-1_all.deb ... 2019-10-13T04:02:17.322 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-13T04:02:17.543 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-bs4. 2019-10-13T04:02:17.556 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-python-bs4_4.6.0-1_all.deb ... 2019-10-13T04:02:17.593 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-13T04:02:17.632 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-waitress. 2019-10-13T04:02:17.648 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python-waitress_1.0.1-1_all.deb ... 2019-10-13T04:02:17.690 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-13T04:02:17.913 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-waitress. 2019-10-13T04:02:17.926 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python-waitress_1.0.1-1_all.deb ... 2019-10-13T04:02:17.970 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-13T04:02:18.018 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-dnspython. 2019-10-13T04:02:18.031 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python-dnspython_1.15.0-1_all.deb ... 2019-10-13T04:02:18.077 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-13T04:02:18.285 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-dnspython. 2019-10-13T04:02:18.301 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python-dnspython_1.15.0-1_all.deb ... 2019-10-13T04:02:18.343 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-13T04:02:18.389 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package formencode-i18n. 2019-10-13T04:02:18.404 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-13T04:02:18.450 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T04:02:18.630 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package formencode-i18n. 2019-10-13T04:02:18.644 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-13T04:02:18.686 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T04:02:18.725 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-formencode. 2019-10-13T04:02:18.741 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-13T04:02:18.783 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T04:02:18.964 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-formencode. 2019-10-13T04:02:18.978 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-13T04:02:19.022 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T04:02:19.111 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-tempita. 2019-10-13T04:02:19.126 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python-tempita_0.5.2-2_all.deb ... 2019-10-13T04:02:19.169 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-13T04:02:19.367 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-tempita. 2019-10-13T04:02:19.383 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python-tempita_0.5.2-2_all.deb ... 2019-10-13T04:02:19.426 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-13T04:02:19.472 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-paste. 2019-10-13T04:02:19.485 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-13T04:02:19.522 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T04:02:19.728 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-paste. 2019-10-13T04:02:19.742 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-13T04:02:19.787 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T04:02:19.834 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-13T04:02:19.847 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-13T04:02:19.892 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T04:02:20.098 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-13T04:02:20.113 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-13T04:02:20.157 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T04:02:20.195 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-13T04:02:20.208 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-13T04:02:20.245 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-13T04:02:20.476 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-13T04:02:20.490 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-13T04:02:20.526 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-13T04:02:20.573 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-webtest. 2019-10-13T04:02:20.587 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-13T04:02:20.631 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T04:02:20.838 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webtest. 2019-10-13T04:02:20.852 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-13T04:02:20.896 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T04:02:20.993 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pecan. 2019-10-13T04:02:21.007 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python-pecan_1.2.1-2_all.deb ... 2019-10-13T04:02:21.064 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-13T04:02:21.249 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pecan. 2019-10-13T04:02:21.264 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python-pecan_1.2.1-2_all.deb ... 2019-10-13T04:02:21.310 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-13T04:02:21.381 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-certifi. 2019-10-13T04:02:21.396 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python-certifi_2018.1.18-2_all.deb ... 2019-10-13T04:02:21.447 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-13T04:02:21.671 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-certifi. 2019-10-13T04:02:21.685 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python-certifi_2018.1.18-2_all.deb ... 2019-10-13T04:02:21.729 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-13T04:02:21.792 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-chardet. 2019-10-13T04:02:21.808 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-python-chardet_3.0.4-1_all.deb ... 2019-10-13T04:02:21.850 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-13T04:02:22.082 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-chardet. 2019-10-13T04:02:22.097 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python-chardet_3.0.4-1_all.deb ... 2019-10-13T04:02:22.140 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-13T04:02:22.179 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-urllib3. 2019-10-13T04:02:22.193 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-13T04:02:22.229 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T04:02:22.469 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-urllib3. 2019-10-13T04:02:22.484 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-13T04:02:22.528 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T04:02:22.582 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-requests. 2019-10-13T04:02:22.597 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-13T04:02:22.642 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T04:02:22.858 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-requests. 2019-10-13T04:02:22.869 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-13T04:02:22.906 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T04:02:22.972 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-werkzeug. 2019-10-13T04:02:22.985 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-13T04:02:23.062 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T04:02:23.251 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-werkzeug. 2019-10-13T04:02:23.269 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-13T04:02:23.323 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T04:02:23.437 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2019-10-13T04:02:23.451 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-ceph-mgr_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:23.495 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:23.645 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2019-10-13T04:02:23.655 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-ceph-mgr_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:23.705 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:23.998 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2019-10-13T04:02:24.010 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-ceph-mon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:24.051 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:24.192 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2019-10-13T04:02:24.205 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-ceph-mon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:24.250 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:24.963 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2019-10-13T04:02:24.976 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-ceph-osd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:25.015 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:25.137 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2019-10-13T04:02:25.153 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-ceph-osd_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:25.195 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:26.884 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2019-10-13T04:02:26.898 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:26.942 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:27.008 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2019-10-13T04:02:27.022 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:27.066 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:27.186 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package runc. 2019-10-13T04:02:27.201 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T04:02:27.245 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T04:02:27.319 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package runc. 2019-10-13T04:02:27.333 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T04:02:27.369 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T04:02:27.773 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package containerd. 2019-10-13T04:02:27.786 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T04:02:27.823 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T04:02:27.873 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package containerd. 2019-10-13T04:02:27.886 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-13T04:02:27.931 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T04:02:29.727 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package docker.io. 2019-10-13T04:02:29.742 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-13T04:02:29.819 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package docker.io. 2019-10-13T04:02:29.820 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T04:02:29.833 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-13T04:02:29.903 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T04:02:32.683 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-daemon. 2019-10-13T04:02:32.696 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-ceph-daemon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:32.732 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:32.991 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-daemon. 2019-10-13T04:02:33.005 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-ceph-daemon_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:33.049 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:33.060 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2019-10-13T04:02:33.073 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-ceph-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:33.110 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:33.377 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2019-10-13T04:02:33.390 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-ceph-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:33.427 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:33.505 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2019-10-13T04:02:33.519 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-ceph-mds_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:33.563 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:33.838 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2019-10-13T04:02:33.853 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-ceph-mds_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:33.896 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:34.050 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-13T04:02:34.063 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-python-repoze.lru_0.7-1_all.deb ... 2019-10-13T04:02:34.100 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-13T04:02:34.353 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-underscore. 2019-10-13T04:02:34.370 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-13T04:02:34.391 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-13T04:02:34.403 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T04:02:34.405 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-python-repoze.lru_0.7-1_all.deb ... 2019-10-13T04:02:34.450 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-13T04:02:34.680 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-13T04:02:34.696 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-13T04:02:34.703 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjs-underscore. 2019-10-13T04:02:34.717 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-13T04:02:34.738 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T04:02:34.761 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T04:02:35.022 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-13T04:02:35.038 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-13T04:02:35.066 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-routes. 2019-10-13T04:02:35.080 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T04:02:35.083 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-python-routes_2.4.1-1_all.deb ... 2019-10-13T04:02:35.116 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-13T04:02:35.400 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-routes. 2019-10-13T04:02:35.413 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-python-routes_2.4.1-1_all.deb ... 2019-10-13T04:02:35.445 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-13T04:02:35.458 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-13T04:02:35.460 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-ceph-mgr-dashboard_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:35.503 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:35.778 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-13T04:02:35.793 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-ceph-mgr-dashboard_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:35.836 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:36.376 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-13T04:02:36.391 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-ceph-mgr-diskprediction-cloud_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:36.434 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:36.729 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-13T04:02:36.744 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-ceph-mgr-diskprediction-cloud_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:36.745 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-decorator. 2019-10-13T04:02:36.759 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-python-decorator_4.1.2-1_all.deb ... 2019-10-13T04:02:36.784 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:36.803 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-13T04:02:37.113 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-decorator. 2019-10-13T04:02:37.114 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-scipy. 2019-10-13T04:02:37.128 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-13T04:02:37.129 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-python-decorator_4.1.2-1_all.deb ... 2019-10-13T04:02:37.165 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T04:02:37.170 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-13T04:02:37.498 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-scipy. 2019-10-13T04:02:37.513 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-13T04:02:37.557 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T04:02:38.289 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-13T04:02:38.303 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-13T04:02:38.347 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-13T04:02:38.716 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-13T04:02:38.729 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-13T04:02:38.773 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-13T04:02:38.818 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-joblib. 2019-10-13T04:02:38.833 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python-joblib_0.11-1_all.deb ... 2019-10-13T04:02:38.876 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-13T04:02:39.213 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-sklearn. 2019-10-13T04:02:39.228 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python-sklearn_0.19.1-3_all.deb ... 2019-10-13T04:02:39.237 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-joblib. 2019-10-13T04:02:39.251 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-python-joblib_0.11-1_all.deb ... 2019-10-13T04:02:39.271 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-13T04:02:39.288 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-13T04:02:39.649 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sklearn. 2019-10-13T04:02:39.664 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python-sklearn_0.19.1-3_all.deb ... 2019-10-13T04:02:39.707 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-13T04:02:39.820 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-13T04:02:39.835 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-ceph-mgr-diskprediction-local_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:39.878 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:40.256 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-13T04:02:40.270 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-ceph-mgr-diskprediction-local_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:40.314 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:40.581 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-monotonic. 2019-10-13T04:02:40.595 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-python-monotonic_1.1-2_all.deb ... 2019-10-13T04:02:40.639 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-13T04:02:40.934 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-fasteners. 2019-10-13T04:02:40.949 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python-fasteners_0.12.0-3_all.deb ... 2019-10-13T04:02:40.997 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-13T04:02:41.042 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-monotonic. 2019-10-13T04:02:41.057 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-python-monotonic_1.1-2_all.deb ... 2019-10-13T04:02:41.111 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-13T04:02:41.295 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-httplib2. 2019-10-13T04:02:41.311 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-13T04:02:41.353 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T04:02:41.412 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-fasteners. 2019-10-13T04:02:41.426 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python-fasteners_0.12.0-3_all.deb ... 2019-10-13T04:02:41.470 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-13T04:02:41.698 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pyasn1. 2019-10-13T04:02:41.712 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-python-pyasn1_0.4.2-3_all.deb ... 2019-10-13T04:02:41.756 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-13T04:02:41.774 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-httplib2. 2019-10-13T04:02:41.787 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-13T04:02:41.841 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T04:02:42.103 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-13T04:02:42.116 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-13T04:02:42.118 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyasn1. 2019-10-13T04:02:42.133 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-python-pyasn1_0.4.2-3_all.deb ... 2019-10-13T04:02:42.161 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T04:02:42.185 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-13T04:02:42.472 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-rsa. 2019-10-13T04:02:42.487 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python-rsa_3.4.2-1_all.deb ... 2019-10-13T04:02:42.530 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-13T04:02:42.531 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-13T04:02:42.546 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-13T04:02:42.589 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T04:02:42.901 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rsa. 2019-10-13T04:02:42.915 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-python-rsa_3.4.2-1_all.deb ... 2019-10-13T04:02:42.919 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-13T04:02:42.933 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-13T04:02:42.959 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-13T04:02:42.984 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:02:43.373 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-13T04:02:43.387 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-13T04:02:43.431 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:02:43.449 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-oauth2client. 2019-10-13T04:02:43.463 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-python-oauth2client_4.1.2-3_all.deb ... 2019-10-13T04:02:43.507 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-13T04:02:43.900 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-websocket. 2019-10-13T04:02:43.917 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-13T04:02:43.935 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-oauth2client. 2019-10-13T04:02:43.952 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-python-oauth2client_4.1.2-3_all.deb ... 2019-10-13T04:02:43.959 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T04:02:43.994 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-13T04:02:44.270 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-yaml. 2019-10-13T04:02:44.283 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-python-yaml_3.12-1build2_amd64.deb ... 2019-10-13T04:02:44.328 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-13T04:02:44.380 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-websocket. 2019-10-13T04:02:44.394 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-13T04:02:44.438 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T04:02:44.656 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-kubernetes. 2019-10-13T04:02:44.673 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-13T04:02:44.714 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T04:02:44.749 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-yaml. 2019-10-13T04:02:44.765 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-python-yaml_3.12-1build2_amd64.deb ... 2019-10-13T04:02:44.808 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-13T04:02:45.144 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-kubernetes. 2019-10-13T04:02:45.152 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-13T04:02:45.158 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-13T04:02:45.167 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:45.202 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T04:02:45.210 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:45.513 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-13T04:02:45.527 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-ceph-mgr-rook_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:45.579 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:45.607 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-13T04:02:45.621 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:45.665 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:45.890 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-13T04:02:45.906 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-ceph-mgr-ssh_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:45.949 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:45.993 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-13T04:02:46.007 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-ceph-mgr-rook_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:46.051 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:46.285 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-13T04:02:46.301 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-libonig4_6.7.0-1_amd64.deb ... 2019-10-13T04:02:46.343 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-13T04:02:46.368 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-13T04:02:46.379 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-ceph-mgr-ssh_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:46.421 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:46.696 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-13T04:02:46.711 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-13T04:02:46.749 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-13T04:02:46.754 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T04:02:46.762 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-libonig4_6.7.0-1_amd64.deb ... 2019-10-13T04:02:46.807 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-13T04:02:47.032 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2019-10-13T04:02:47.046 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-jq_1.5+dfsg-2_amd64.deb ... 2019-10-13T04:02:47.090 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-13T04:02:47.135 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-13T04:02:47.198 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-13T04:02:47.198 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T04:02:47.343 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2019-10-13T04:02:47.360 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-13T04:02:47.402 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-13T04:02:47.497 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2019-10-13T04:02:47.512 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-jq_1.5+dfsg-2_amd64.deb ... 2019-10-13T04:02:47.555 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-13T04:02:47.797 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-13T04:02:47.807 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2019-10-13T04:02:47.811 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-13T04:02:47.821 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-13T04:02:47.855 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T04:02:47.866 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-13T04:02:48.141 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2019-10-13T04:02:48.158 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-13T04:02:48.191 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-13T04:02:48.211 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-13T04:02:48.226 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-13T04:02:48.261 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T04:02:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2019-10-13T04:02:48.537 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2019-10-13T04:02:48.544 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-13T04:02:48.552 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-ceph-test_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:48.589 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-13T04:02:48.596 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:48.897 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2019-10-13T04:02:48.913 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-ceph-test_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:48.947 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:51.332 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2019-10-13T04:02:51.346 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-python3-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:51.390 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:51.672 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rados. 2019-10-13T04:02:51.688 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-python3-rados_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:51.712 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-10-13T04:02:51.728 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-python3-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:51.730 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:51.770 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:52.058 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-10-13T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2019-10-13T04:02:52.075 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-python3-ceph-argparse_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:52.087 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-python3-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:52.116 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:52.122 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:52.369 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cephfs. 2019-10-13T04:02:52.385 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-python3-cephfs_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:52.427 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:52.442 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-colorama. 2019-10-13T04:02:52.456 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-python3-colorama_0.3.7-1_all.deb ... 2019-10-13T04:02:52.500 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-10-13T04:02:52.762 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-shell. 2019-10-13T04:02:52.764 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-colorama. 2019-10-13T04:02:52.776 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-cephfs-shell_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:52.778 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-python3-colorama_0.3.7-1_all.deb ... 2019-10-13T04:02:52.820 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:52.822 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-10-13T04:02:53.061 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephfs-shell. 2019-10-13T04:02:53.072 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-cephfs-shell_15.0.0-6132-ga8ee8ac-1bionic_all.deb ... 2019-10-13T04:02:53.108 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:53.197 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-13T04:02:53.212 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-cgroupfs-mount_1.4_all.deb ... 2019-10-13T04:02:53.265 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-13T04:02:53.461 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-13T04:02:53.477 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-cgroupfs-mount_1.4_all.deb ... 2019-10-13T04:02:53.500 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package dns-root-data. 2019-10-13T04:02:53.514 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-dns-root-data_2018013001_all.deb ... 2019-10-13T04:02:53.529 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-13T04:02:53.550 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-13T04:02:53.766 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package dns-root-data. 2019-10-13T04:02:53.778 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-dns-root-data_2018013001_all.deb ... 2019-10-13T04:02:53.823 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-13T04:02:53.920 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-13T04:02:53.934 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-13T04:02:53.978 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-13T04:02:54.100 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-13T04:02:54.114 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-13T04:02:54.150 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-13T04:02:54.281 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-13T04:02:54.296 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-libcephfs-dev_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:54.340 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:54.462 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-13T04:02:54.479 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-libcephfs-dev_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:54.520 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:54.626 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-attr. 2019-10-13T04:02:54.640 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python-attr_17.4.0-2_all.deb ... 2019-10-13T04:02:54.675 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-13T04:02:54.849 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-attr. 2019-10-13T04:02:54.866 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-python-attr_17.4.0-2_all.deb ... 2019-10-13T04:02:54.907 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-13T04:02:54.945 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-ceph. 2019-10-13T04:02:54.959 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:55.003 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:55.151 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph. 2019-10-13T04:02:55.165 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-python-ceph_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:02:55.211 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:02:55.297 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-funcsigs. 2019-10-13T04:02:55.312 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python-funcsigs_1.0.2-4_all.deb ... 2019-10-13T04:02:55.355 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-13T04:02:55.445 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-funcsigs. 2019-10-13T04:02:55.459 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-python-funcsigs_1.0.2-4_all.deb ... 2019-10-13T04:02:55.495 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-13T04:02:55.658 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-webencodings. 2019-10-13T04:02:55.675 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python-webencodings_0.5-2_all.deb ... 2019-10-13T04:02:55.716 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-13T04:02:55.790 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webencodings. 2019-10-13T04:02:55.807 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-python-webencodings_0.5-2_all.deb ... 2019-10-13T04:02:55.848 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-13T04:02:56.003 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-html5lib. 2019-10-13T04:02:56.018 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python-html5lib_0.999999999-1_all.deb ... 2019-10-13T04:02:56.061 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-13T04:02:56.151 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-html5lib. 2019-10-13T04:02:56.165 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-python-html5lib_0.999999999-1_all.deb ... 2019-10-13T04:02:56.209 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-13T04:02:56.406 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-13T04:02:56.419 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-13T04:02:56.464 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T04:02:56.563 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-13T04:02:56.577 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../109-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-13T04:02:56.612 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T04:02:56.909 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-openid. 2019-10-13T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python-openid_2.2.5-6_all.deb ... 2019-10-13T04:02:56.967 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-13T04:02:57.050 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-openid. 2019-10-13T04:02:57.065 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../110-python-openid_2.2.5-6_all.deb ... 2019-10-13T04:02:57.108 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-13T04:02:57.304 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastescript. 2019-10-13T04:02:57.320 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-python-pastescript_2.0.2-2_all.deb ... 2019-10-13T04:02:57.362 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-13T04:02:57.453 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastescript. 2019-10-13T04:02:57.468 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../111-python-pastescript_2.0.2-2_all.deb ... 2019-10-13T04:02:57.511 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-13T04:02:57.690 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pluggy. 2019-10-13T04:02:57.706 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-python-pluggy_0.6.0-1_all.deb ... 2019-10-13T04:02:57.749 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-13T04:02:57.823 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pluggy. 2019-10-13T04:02:57.840 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../112-python-pluggy_0.6.0-1_all.deb ... 2019-10-13T04:02:57.873 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-13T04:02:58.051 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-py. 2019-10-13T04:02:58.066 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../113-python-py_1.5.2-1_all.deb ... 2019-10-13T04:02:58.109 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-13T04:02:58.160 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-py. 2019-10-13T04:02:58.174 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../113-python-py_1.5.2-1_all.deb ... 2019-10-13T04:02:58.218 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-13T04:02:58.421 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pyinotify. 2019-10-13T04:02:58.437 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../114-python-pyinotify_0.9.6-1_all.deb ... 2019-10-13T04:02:58.479 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-13T04:02:58.539 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyinotify. 2019-10-13T04:02:58.553 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../114-python-pyinotify_0.9.6-1_all.deb ... 2019-10-13T04:02:58.596 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-13T04:02:58.832 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pytest. 2019-10-13T04:02:58.848 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../115-python-pytest_3.3.2-2_all.deb ... 2019-10-13T04:02:58.890 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-13T04:02:58.907 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pytest. 2019-10-13T04:02:58.921 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../115-python-pytest_3.3.2-2_all.deb ... 2019-10-13T04:02:58.966 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-13T04:02:59.235 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-scgi. 2019-10-13T04:02:59.249 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../116-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-13T04:02:59.294 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-13T04:02:59.319 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-scgi. 2019-10-13T04:02:59.334 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../116-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-13T04:02:59.378 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-13T04:02:59.538 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-13T04:02:59.553 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../117-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-13T04:02:59.596 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:02:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-13T04:02:59.645 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../117-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-13T04:02:59.689 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:02:59.957 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2019-10-13T04:02:59.973 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../118-radosgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:03:00.015 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:00.052 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2019-10-13T04:03:00.065 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../118-radosgw_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:03:00.109 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:00.674 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2019-10-13T04:03:00.688 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../119-rbd-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:03:00.732 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:00.804 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2019-10-13T04:03:00.820 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../119-rbd-fuse_15.0.0-6132-ga8ee8ac-1bionic_amd64.deb ... 2019-10-13T04:03:00.864 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:01.111 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-13T04:03:01.128 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../120-ubuntu-fan_0.12.10_all.deb ... 2019-10-13T04:03:01.245 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-13T04:03:01.257 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../120-ubuntu-fan_0.12.10_all.deb ... 2019-10-13T04:03:01.322 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-13T04:03:01.569 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-13T04:03:01.618 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-13T04:03:01.864 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-13T04:03:01.874 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-13T04:03:02.080 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-13T04:03:02.117 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-13T04:03:02.286 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:03:02.335 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-13T04:03:02.549 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:03:02.776 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-13T04:03:03.022 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-13T04:03:03.259 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-13T04:03:03.447 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-13T04:03:03.472 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T04:03:03.644 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-13T04:03:03.724 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-13T04:03:03.880 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-13T04:03:03.923 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-13T04:03:04.112 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-13T04:03:04.121 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-13T04:03:04.307 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-13T04:03:04.397 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-13T04:03:04.597 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T04:03:04.605 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-13T04:03:04.732 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T04:03:04.802 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-13T04:03:04.854 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T04:03:04.941 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-13T04:03:04.956 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T04:03:05.073 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T04:03:05.169 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-13T04:03:05.179 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:05.376 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:05.485 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-13T04:03:05.633 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-13T04:03:05.681 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-13T04:03:05.820 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-13T04:03:05.863 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-py (1.5.2-1) ... 2019-10-13T04:03:06.051 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-py (1.5.2-1) ... 2019-10-13T04:03:06.096 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-13T04:03:06.287 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-13T04:03:06.294 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-joblib (0.11-1) ... 2019-10-13T04:03:06.485 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-joblib (0.11-1) ... 2019-10-13T04:03:06.558 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-13T04:03:06.681 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-13T04:03:06.748 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-13T04:03:06.875 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-13T04:03:06.884 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-13T04:03:07.079 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-10-13T04:03:07.089 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-13T04:03:07.292 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-10-13T04:03:07.382 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up dns-root-data (2018013001) ... 2019-10-13T04:03:07.511 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T04:03:07.587 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up dns-root-data (2018013001) ... 2019-10-13T04:03:07.715 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-10-13T04:03:07.816 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T04:03:07.943 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T04:03:08.046 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-13T04:03:08.069 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T04:03:08.208 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-13T04:03:08.304 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:08.333 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-13T04:03:08.438 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-13T04:03:08.560 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph-argparse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:08.611 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-13T04:03:08.688 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-13T04:03:08.816 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-13T04:03:08.850 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-13T04:03:09.043 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-13T04:03:09.297 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T04:03:09.512 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-13T04:03:09.550 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-13T04:03:09.760 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-13T04:03:09.875 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T04:03:10.124 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-13T04:03:10.174 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-13T04:03:10.390 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-13T04:03:10.409 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-13T04:03:10.590 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-13T04:03:10.617 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-13T04:03:10.794 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T04:03:10.816 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-13T04:03:11.011 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-13T04:03:11.074 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-13T04:03:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-13T04:03:11.336 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-13T04:03:11.547 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-13T04:03:11.570 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-13T04:03:11.756 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-13T04:03:11.797 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T04:03:11.887 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-13T04:03:11.949 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-13T04:03:12.032 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T04:03:12.039 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-13T04:03:12.159 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T04:03:12.184 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-13T04:03:12.310 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-13T04:03:12.359 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T04:03:12.485 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:12.511 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-13T04:03:12.612 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:12.646 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:12.763 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-13T04:03:12.772 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:12.898 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-13T04:03:13.006 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-13T04:03:13.124 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-13T04:03:13.224 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-13T04:03:13.342 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-13T04:03:13.350 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T04:03:13.477 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-13T04:03:13.510 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T04:03:13.654 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T04:03:13.702 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T04:03:13.854 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-13T04:03:13.903 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:03:14.013 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-13T04:03:14.080 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-13T04:03:14.205 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-13T04:03:14.206 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-13T04:03:14.399 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-13T04:03:14.861 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T04:03:15.046 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T04:03:15.118 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-13T04:03:15.323 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T04:03:19.041 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T04:03:19.234 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-13T04:03:19.360 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-13T04:03:19.427 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-13T04:03:19.561 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-13T04:03:19.570 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-13T04:03:19.756 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T04:03:19.770 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-13T04:03:19.868 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T04:03:19.936 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-13T04:03:19.992 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-13T04:03:20.085 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-13T04:03:20.117 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-13T04:03:20.241 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-13T04:03:20.284 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T04:03:20.367 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-13T04:03:20.410 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up pigz (2.4-1) ... 2019-10-13T04:03:20.468 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-13T04:03:20.519 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-13T04:03:20.669 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-13T04:03:20.687 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-13T04:03:20.795 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up pigz (2.4-1) ... 2019-10-13T04:03:20.921 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-13T04:03:20.922 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-13T04:03:21.114 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-13T04:03:21.131 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-13T04:03:21.333 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-10-13T04:03:21.374 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-13T04:03:21.567 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-13T04:03:21.625 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-13T04:03:21.776 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-10-13T04:03:21.818 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T04:03:22.036 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-13T04:03:22.086 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-13T04:03:22.137 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T04:03:22.296 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-13T04:03:22.530 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-13T04:03:22.548 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group `docker' (GID 122) ... 2019-10-13T04:03:22.656 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-13T04:03:22.660 INFO:teuthology.orchestra.run.smithi110.stdout:Done. 2019-10-13T04:03:22.794 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-13T04:03:23.082 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group `docker' (GID 122) ... 2019-10-13T04:03:23.196 INFO:teuthology.orchestra.run.smithi204.stdout:Done. 2019-10-13T04:03:23.328 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-13T04:03:25.983 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T04:03:26.519 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-13T04:03:26.979 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-13T04:03:27.164 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T04:03:27.456 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T04:03:27.523 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-13T04:03:27.724 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-10-13T04:03:27.828 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-13T04:03:28.033 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-13T04:03:28.078 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-13T04:03:28.262 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T04:03:28.363 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-13T04:03:28.395 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-13T04:03:28.490 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:28.616 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:28.646 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-13T04:03:28.742 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-13T04:03:28.864 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-13T04:03:29.007 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-13T04:03:29.133 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:29.259 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:29.394 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-13T04:03:29.420 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-13T04:03:29.995 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-13T04:03:30.073 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-13T04:03:30.435 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:30.545 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:30.638 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-13T04:03:30.646 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:30.747 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T04:03:31.144 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.270 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.274 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-13T04:03:31.396 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-daemon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.522 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-13T04:03:31.526 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.652 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.845 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:31.946 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.058 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-13T04:03:32.205 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.293 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rados (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.323 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.449 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-13T04:03:32.460 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.667 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.678 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cephfs (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:32.888 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T04:03:32.889 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T04:03:33.097 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.153 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.224 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.269 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.349 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-13T04:03:33.370 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-13T04:03:33.551 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.572 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.681 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:33.769 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T04:03:33.769 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-13T04:03:34.052 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:34.187 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephfs-shell (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:34.207 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2019-10-13T04:03:34.313 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-13T04:03:34.510 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2019-10-13T04:03:34.539 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rbd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:34.666 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:34.695 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2019-10-13T04:03:34.856 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-13T04:03:35.084 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-13T04:03:35.217 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2019-10-13T04:03:35.384 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:35.628 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2019-10-13T04:03:35.643 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T04:03:35.643 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T04:03:35.829 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2019-10-13T04:03:35.994 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-13T04:03:36.061 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:36.187 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:36.233 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-13T04:03:36.288 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T04:03:36.464 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:36.534 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:36.573 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:36.795 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T04:03:36.795 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-13T04:03:36.899 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-13T04:03:37.184 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-13T04:03:37.211 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:37.274 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-13T04:03:37.297 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-13T04:03:37.346 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:37.472 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-13T04:03:37.503 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:37.690 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:37.816 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:38.034 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T04:03:38.034 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T04:03:38.152 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-13T04:03:38.306 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:38.468 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-13T04:03:38.500 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T04:03:38.501 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T04:03:38.558 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-13T04:03:38.590 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-13T04:03:38.774 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:38.787 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:38.994 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T04:03:38.995 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T04:03:39.301 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:39.337 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T04:03:39.337 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-13T04:03:39.519 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T04:03:39.519 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T04:03:39.615 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:39.810 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:39.833 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T04:03:39.833 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-13T04:03:39.953 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.088 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.150 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.205 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.307 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.373 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T04:03:40.374 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-13T04:03:40.418 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.567 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.676 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:40.718 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T04:03:40.811 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:40.879 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T04:03:40.879 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-13T04:03:40.946 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T04:03:41.160 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.287 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.430 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-rook (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.463 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T04:03:41.465 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:41.465 INFO:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-10-13T04:03:41.546 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2019-10-13T04:03:41.573 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.699 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.794 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2019-10-13T04:03:41.795 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2019-10-13T04:03:41.825 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-ssh (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.945 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2019-10-13T04:03:41.946 INFO:teuthology.orchestra.run.smithi110.stdout: libbison-dev libfl-dev libfl2 zlib1g-dev 2019-10-13T04:03:41.947 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2019-10-13T04:03:41.948 INFO:teuthology.orchestra.run.smithi110.stdout: bison-doc flex-doc libssl-doc 2019-10-13T04:03:41.959 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (15.0.0-6132-ga8ee8ac-1bionic) ... 2019-10-13T04:03:41.972 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2019-10-13T04:03:41.973 INFO:teuthology.orchestra.run.smithi110.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-10-13T04:03:42.085 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-13T04:03:42.169 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:42.171 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T04:03:42.171 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 2,737 kB of archives. 2019-10-13T04:03:42.172 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-10-13T04:03:42.172 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-10-13T04:03:42.304 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-13T04:03:42.718 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-10-13T04:03:42.813 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T04:03:42.814 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:42.814 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-10-13T04:03:42.965 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-13T04:03:43.213 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-13T04:03:43.214 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-13T04:03:43.261 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-10-13T04:03:43.373 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2019-10-13T04:03:43.374 INFO:teuthology.orchestra.run.smithi204.stdout: libbison-dev libfl-dev libfl2 zlib1g-dev 2019-10-13T04:03:43.376 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2019-10-13T04:03:43.376 INFO:teuthology.orchestra.run.smithi204.stdout: bison-doc flex-doc libssl-doc 2019-10-13T04:03:43.401 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2019-10-13T04:03:43.402 INFO:teuthology.orchestra.run.smithi204.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-10-13T04:03:43.585 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2019-10-13T04:03:43.585 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 2,737 kB of archives. 2019-10-13T04:03:43.585 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-10-13T04:03:43.586 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-10-13T04:03:43.778 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-10-13T04:03:44.109 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-10-13T04:03:44.237 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-10-13T04:03:44.574 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-10-13T04:03:44.665 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-10-13T04:03:44.736 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-10-13T04:03:44.898 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.4 [1,566 kB] 2019-10-13T04:03:45.173 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-10-13T04:03:45.632 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-10-13T04:03:45.851 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 2,737 kB in 3s (787 kB/s) 2019-10-13T04:03:45.971 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-10-13T04:03:46.000 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package flex. 2019-10-13T04:03:46.051 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143618 files and directories currently installed.) 2019-10-13T04:03:46.055 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-10-13T04:03:46.099 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking flex (2.6.4-6) ... 2019-10-13T04:03:46.132 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-10-13T04:03:46.294 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.4 [1,566 kB] 2019-10-13T04:03:46.528 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-10-13T04:03:46.542 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T04:03:46.595 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:46.939 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package bison. 2019-10-13T04:03:46.953 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T04:03:47.002 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:47.238 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 2,737 kB in 3s (796 kB/s) 2019-10-13T04:03:47.318 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-10-13T04:03:47.333 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-10-13T04:03:47.376 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T04:03:47.384 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package flex. 2019-10-13T04:03:47.439 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143618 files and directories currently installed.) 2019-10-13T04:03:47.443 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-10-13T04:03:47.482 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking flex (2.6.4-6) ... 2019-10-13T04:03:47.696 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-10-13T04:03:47.710 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-10-13T04:03:47.780 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T04:03:47.953 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-10-13T04:03:47.970 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T04:03:48.003 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:48.149 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libfl2:amd64. 2019-10-13T04:03:48.164 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-libfl2_2.6.4-6_amd64.deb ... 2019-10-13T04:03:48.250 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-10-13T04:03:48.298 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package bison. 2019-10-13T04:03:48.312 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-10-13T04:03:48.352 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:48.505 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-10-13T04:03:48.518 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-libfl-dev_2.6.4-6_amd64.deb ... 2019-10-13T04:03:48.561 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T04:03:48.643 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-10-13T04:03:48.656 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-10-13T04:03:48.701 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T04:03:48.805 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-10-13T04:03:48.822 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-libssl-dev_1.1.1-1ubuntu2.1~18.04.4_amd64.deb ... 2019-10-13T04:03:48.872 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T04:03:49.021 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-10-13T04:03:49.038 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-10-13T04:03:49.071 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T04:03:49.341 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up flex (2.6.4-6) ... 2019-10-13T04:03:49.391 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfl2:amd64. 2019-10-13T04:03:49.405 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-libfl2_2.6.4-6_amd64.deb ... 2019-10-13T04:03:49.469 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-10-13T04:03:49.491 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-10-13T04:03:49.769 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-10-13T04:03:49.784 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-libfl-dev_2.6.4-6_amd64.deb ... 2019-10-13T04:03:49.827 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T04:03:49.887 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T04:03:50.021 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:50.063 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-10-13T04:03:50.076 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-libssl-dev_1.1.1-1ubuntu2.1~18.04.4_amd64.deb ... 2019-10-13T04:03:50.113 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T04:03:50.156 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:50.282 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-10-13T04:03:50.459 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T04:03:50.580 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up flex (2.6.4-6) ... 2019-10-13T04:03:50.701 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-10-13T04:03:51.136 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.4) ... 2019-10-13T04:03:51.237 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:51.368 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:51.473 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-10-13T04:03:51.624 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-13T04:03:52.127 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:52.216 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-10-13T04:03:52.287 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T04:03:52.413 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T04:03:52.539 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T04:03:52.674 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:53.108 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T04:03:53.110 DEBUG:teuthology.parallel:result is None 2019-10-13T04:03:53.284 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-10-13T04:03:53.374 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-10-13T04:03:53.444 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-10-13T04:03:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-10-13T04:03:53.688 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-10-13T04:03:53.789 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-13T04:03:54.221 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-13T04:03:54.223 DEBUG:teuthology.parallel:result is None 2019-10-13T04:03:54.223 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.223 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-13T04:03:54.268 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-13T04:03:54.273 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:03:54.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:03:54.438 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.438 INFO:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2019-10-13T04:03:54.462 INFO:teuthology.orchestra.run.smithi204.stdout:15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:03:54.463 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:03:54.463 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132-ga8ee8ac-1bionic is installed. 2019-10-13T04:03:54.464 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:54.464 INFO:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2019-10-13T04:03:54.517 INFO:teuthology.orchestra.run.smithi110.stdout:Ubuntu 2019-10-13T04:03:54.523 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-13T04:03:54.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:03:54.700 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:54.700 INFO:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2019-10-13T04:03:54.730 INFO:teuthology.orchestra.run.smithi110.stdout:15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:03:54.730 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6132-ga8ee8ac-1bionic 2019-10-13T04:03:54.731 INFO:teuthology.task.install:The correct ceph version 15.0.0-6132-ga8ee8ac-1bionic is installed. 2019-10-13T04:03:54.731 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-10-13T04:03:54.731 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.732 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T04:03:54.748 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:54.749 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-13T04:03:54.797 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-10-13T04:03:54.797 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.798 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T04:03:54.815 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.815 INFO:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T04:03:54.895 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:54.895 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-13T04:03:54.912 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:54.912 INFO:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-13T04:03:54.991 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-10-13T04:03:54.992 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:54.992 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T04:03:55.041 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.041 INFO:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T04:03:55.120 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.120 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-13T04:03:55.170 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.170 INFO:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-13T04:03:55.254 INFO:teuthology.run_tasks:Running task ceph... 2019-10-13T04:03:55.329 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-10-13T04:03:55.329 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.329 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2019-10-13T04:03:55.333 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.333 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2019-10-13T04:03:55.348 INFO:tasks.ceph:Disabling ceph logrotate... 2019-10-13T04:03:55.349 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.349 INFO:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T04:03:55.415 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.415 INFO:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-13T04:03:55.431 INFO:tasks.ceph:Creating extra log directories... 2019-10-13T04:03:55.431 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.431 INFO:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T04:03:55.498 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.498 INFO:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-13T04:03:55.515 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2019-10-13T04:03:55.516 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2019-10-13T04:03:55.516 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2019-10-13T04:03:55.516 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.516 INFO:teuthology.orchestra.run.smithi204:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T04:03:55.628 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.628 INFO:teuthology.orchestra.run.smithi204:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:55.728 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.728 INFO:teuthology.orchestra.run.smithi110:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-10-13T04:03:55.774 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.774 INFO:teuthology.orchestra.run.smithi110:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-10-13T04:03:55.874 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-10-13T04:03:55.874 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds inject traceless reply probability': 0.5, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}}} 2019-10-13T04:03:55.875 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, {'ceph': None}, {'ceph-fuse': None}, {'check-counter': {'counters': {'mds': ['mds.dir_split']}}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.4306', 'owner': 'scheduled_pdonnell@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'targets': {'smithi110.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6EdIEDzZRdBRD0bLwvaqt1T2CWfYA4jgooF37gfsxlmNU8R7fsTcSSy4aIzep3lsgjtvbgfSe9phgjthGuFZn3BvGAZEQRt/PN2Jo+QAspYO/VWjU2DgRKNGwpPsJ9KplGabY0l3aAMPJqj1zySLSZ3fITcyCmFX2mnB1AeQcTaX50JpSiyxr3UZ2pRumFxhCb9aAmDZnLN3IsqEKZ83bMyRrXPG15KSyxCVHlSonFn6Ww2aO+jPB1W9RrJkJZdvzbEKZKOgd5YDA1jPltKKeWHYOfE2SKF1xNOFnyUut4UhQ5DbBK9yrkOa5yO0ksPW0LmKTM5UXCtvV7eQc8dRZ', 'smithi204.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTlOcAA/Jqcpb/OnkOrrCmLB5vmc9IBTCJhMmorSjsCMf8qQOidO511p0AH330PF9U0wABWhkxY81XHPtDnTJ9s2n6kflGm2uj3F+qPocj4Z9UJ8XSUfLRtgBVWjcnyksNai26fo+xBDZK1cFxxDJp+T+L8FlQ9GUInuU0fmz3ohjpGL4WQFSfge5s4qFziOXMUqk8nONiddfXPZNJ/mvS1GYWcoG+aCTQ1Pqf+BfzINGwzYYpdJFqVCTS13iw2l3e/p7Hh8UeY2u/MCrE/lwCD1siKnYkToTJ7LPRbLrb7A0faOD0RUK0owHIwv1+owJ3ZfArM6LqjQKj7DFb6DF3'}, 'job_id': '4407711', 'tube': 'smithi', 'suite_branch': 'wip-pdonnell-testing-20191012.025926', 'priority': 95, 'os_version': '18.04', 'first_in_suite': False, 'branch': 'wip-pdonnell-testing-20191012.025926', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': 'pdonnell@redhat.com', 'archive_path': '/home/teuthworker/archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407711', 'description': 'fs/traceless/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml traceless/50pc.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'name': 'pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds inject traceless reply probability': 0.5, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'filestore flush min': 0, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}}, 'fs': 'xfs', 'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'}}, 'admin_socket': {'branch': 'wip-pdonnell-testing-20191012.025926'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd', 'branch': 'wip-pdonnell-testing-20191012.025926'}}, 'os_type': 'ubuntu', 'suite_sha1': 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'} 2019-10-13T04:03:55.875 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.875 INFO:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T04:03:55.941 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:55.942 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-13T04:03:55.949 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:55.950 INFO:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T04:03:56.016 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.016 INFO:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2019-10-13T04:03:56.105 DEBUG:teuthology.orchestra.remote:smithi204:/scratch_devs is 72B 2019-10-13T04:03:56.117 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T04:03:56.118 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.118 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T04:03:56.126 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-13T04:03:56.127 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.127 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 449 Links: 1 2019-10-13T04:03:56.127 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.127 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-13 04:00:50.206828141 +0000 2019-10-13T04:03:56.128 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-13 04:00:50.078892144 +0000 2019-10-13T04:03:56.128 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-13 04:00:50.078892144 +0000 2019-10-13T04:03:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-13T04:03:56.134 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-13T04:03:56.134 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-13T04:03:56.134 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000443457 s, 1.2 MB/s 2019-10-13T04:03:56.138 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.138 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T04:03:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-13T04:03:56.210 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.210 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 455 Links: 1 2019-10-13T04:03:56.210 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.210 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-13 04:00:50.430716136 +0000 2019-10-13T04:03:56.211 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-13 04:00:50.310776138 +0000 2019-10-13T04:03:56.211 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-13 04:00:50.310776138 +0000 2019-10-13T04:03:56.211 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-13T04:03:56.220 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-13T04:03:56.220 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-13T04:03:56.220 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000382104 s, 1.3 MB/s 2019-10-13T04:03:56.225 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.225 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T04:03:56.296 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 461 Links: 1 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-13 04:00:50.706578129 +0000 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-13 04:00:50.570646133 +0000 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-13 04:00:50.570646133 +0000 2019-10-13T04:03:56.297 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-13T04:03:56.306 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-13T04:03:56.306 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-13T04:03:56.307 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.0004182 s, 1.2 MB/s 2019-10-13T04:03:56.310 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.311 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T04:03:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 467 Links: 1 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-13 04:03:09.320114153 +0000 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-13 04:00:50.818522126 +0000 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-13 04:00:50.818522126 +0000 2019-10-13T04:03:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-13T04:03:56.392 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-13T04:03:56.392 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-13T04:03:56.393 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000422368 s, 1.2 MB/s 2019-10-13T04:03:56.397 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T04:03:56.397 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T04:03:56.398 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:56.398 INFO:teuthology.orchestra.run.smithi204:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T04:03:56.470 INFO:teuthology.orchestra.run.smithi204.stdout:lrwxrwxrwx 1 root root 9 Oct 13 03:57 /dev/disk/by-id/wwn-0x5000c50092941a36 -> ../../sda 2019-10-13T04:03:56.470 INFO:teuthology.orchestra.run.smithi204.stdout:lrwxrwxrwx 1 root root 10 Oct 13 03:57 /dev/disk/by-id/wwn-0x5000c50092941a36-part1 -> ../../sda1 2019-10-13T04:03:56.470 INFO:tasks.ceph:dev map: {} 2019-10-13T04:03:56.479 DEBUG:teuthology.orchestra.remote:smithi110:/scratch_devs is 72B 2019-10-13T04:03:56.496 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T04:03:56.497 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.497 INFO:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-13T04:03:56.505 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-13T04:03:56.506 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.506 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 437 Links: 1 2019-10-13T04:03:56.506 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.506 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2019-10-13 04:00:49.644462961 +0000 2019-10-13T04:03:56.506 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2019-10-13 04:00:49.520462964 +0000 2019-10-13T04:03:56.507 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2019-10-13 04:00:49.520462964 +0000 2019-10-13T04:03:56.507 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2019-10-13T04:03:56.514 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2019-10-13T04:03:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2019-10-13T04:03:56.521 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000327013 s, 1.6 MB/s 2019-10-13T04:03:56.522 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.522 INFO:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-13T04:03:56.573 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-13T04:03:56.573 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.573 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 443 Links: 1 2019-10-13T04:03:56.574 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.574 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2019-10-13 04:00:49.916462955 +0000 2019-10-13T04:03:56.574 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2019-10-13 04:00:49.772462958 +0000 2019-10-13T04:03:56.574 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2019-10-13 04:00:49.772462958 +0000 2019-10-13T04:03:56.574 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2019-10-13T04:03:56.583 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2019-10-13T04:03:56.583 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2019-10-13T04:03:56.583 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000383298 s, 1.3 MB/s 2019-10-13T04:03:56.588 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.588 INFO:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-13T04:03:56.659 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-13T04:03:56.659 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 449 Links: 1 2019-10-13T04:03:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2019-10-13 04:00:50.156462950 +0000 2019-10-13T04:03:56.660 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2019-10-13 04:00:50.044462952 +0000 2019-10-13T04:03:56.660 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2019-10-13 04:00:50.044462952 +0000 2019-10-13T04:03:56.660 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2019-10-13T04:03:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2019-10-13T04:03:56.669 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2019-10-13T04:03:56.669 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000417411 s, 1.2 MB/s 2019-10-13T04:03:56.676 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.676 INFO:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-13T04:03:56.748 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-13T04:03:56.748 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-13T04:03:56.748 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 455 Links: 1 2019-10-13T04:03:56.748 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:03:56.749 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2019-10-13 04:00:50.304462946 +0000 2019-10-13T04:03:56.749 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2019-10-13 04:00:50.300462946 +0000 2019-10-13T04:03:56.749 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2019-10-13 04:00:50.300462946 +0000 2019-10-13T04:03:56.749 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2019-10-13T04:03:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2019-10-13T04:03:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2019-10-13T04:03:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000425677 s, 1.2 MB/s 2019-10-13T04:03:56.762 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-13T04:03:56.763 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-13T04:03:56.763 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.763 INFO:teuthology.orchestra.run.smithi110:> ls -l /dev/disk/by-id/wwn-* 2019-10-13T04:03:56.837 INFO:teuthology.orchestra.run.smithi110.stdout:lrwxrwxrwx 1 root root 9 Oct 13 03:57 /dev/disk/by-id/wwn-0x5000c50091dfc3c1 -> ../../sda 2019-10-13T04:03:56.837 INFO:teuthology.orchestra.run.smithi110.stdout:lrwxrwxrwx 1 root root 10 Oct 13 03:57 /dev/disk/by-id/wwn-0x5000c50091dfc3c1-part1 -> ../../sda1 2019-10-13T04:03:56.837 INFO:tasks.ceph:dev map: {} 2019-10-13T04:03:56.838 INFO:tasks.ceph:Generating config... 2019-10-13T04:03:56.839 INFO:tasks.ceph:[mds] mds debug frag = True 2019-10-13T04:03:56.839 INFO:tasks.ceph:[mds] mds bal split bits = 3 2019-10-13T04:03:56.840 INFO:tasks.ceph:[mds] mds bal split size = 100 2019-10-13T04:03:56.840 INFO:tasks.ceph:[mds] mds inject traceless reply probability = 0.5 2019-10-13T04:03:56.840 INFO:tasks.ceph:[mds] debug mds = 20 2019-10-13T04:03:56.840 INFO:tasks.ceph:[mds] mds bal merge size = 5 2019-10-13T04:03:56.840 INFO:tasks.ceph:[mds] debug ms = 1 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] mds bal frag = True 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] mds op complaint time = 180 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] osd op complaint time = 180 2019-10-13T04:03:56.841 INFO:tasks.ceph:[mds] mds verify scatter = True 2019-10-13T04:03:56.842 INFO:tasks.ceph:[client] debug ms = 1 2019-10-13T04:03:56.842 INFO:tasks.ceph:[client] debug client = 20 2019-10-13T04:03:56.842 INFO:tasks.ceph:[client] client mount timeout = 600 2019-10-13T04:03:56.842 INFO:tasks.ceph:[mon] debug mon = 20 2019-10-13T04:03:56.842 INFO:tasks.ceph:[mon] debug paxos = 20 2019-10-13T04:03:56.842 INFO:tasks.ceph:[mon] mon op complaint time = 120 2019-10-13T04:03:56.843 INFO:tasks.ceph:[mon] debug ms = 1 2019-10-13T04:03:56.843 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-10-13T04:03:56.843 INFO:tasks.ceph:[osd] debug ms = 1 2019-10-13T04:03:56.843 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-10-13T04:03:56.843 INFO:tasks.ceph:[osd] filestore flush min = 0 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] debug osd = 25 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-10-13T04:03:56.844 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] debug journal = 20 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-10-13T04:03:56.845 INFO:tasks.ceph:[osd] debug filestore = 20 2019-10-13T04:03:56.846 INFO:tasks.ceph:[osd] osd op complaint time = 180 2019-10-13T04:03:56.846 INFO:tasks.ceph:Setting up mon.a... 2019-10-13T04:03:56.847 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.847 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-10-13T04:03:56.944 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2019-10-13T04:03:56.946 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:56.946 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-10-13T04:03:57.046 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.046 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T04:03:57.126 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.c', '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]'), ('mon.b', '172.21.15.204')] 2019-10-13T04:03:57.126 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.126 INFO:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.110 --addv c '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]' --add b 172.21.15.204 --print /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:03:57.224 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:03:57.224 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 7c497bad-e78c-4b28-a8fa-4ca4e4cb6052 2019-10-13T04:03:57.224 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 7c497bad-e78c-4b28-a8fa-4ca4e4cb6052 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2019-10-13T04:03:57.225106+0000 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:created 2019-10-13T04:03:57.225106+0000 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 0 (unknown) 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2019-10-13T04:03:57.225 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2019-10-13T04:03:57.226 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] mon.c 2019-10-13T04:03:57.226 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-10-13T04:03:57.227 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7c497bad-e78c-4b28-a8fa-4ca4e4cb6052... 2019-10-13T04:03:57.229 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.229 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T04:03:57.297 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:57.297 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-13T04:03:57.360 INFO:tasks.ceph:Creating admin key on mon.a... 2019-10-13T04:03:57.361 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.361 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-10-13T04:03:57.407 INFO:tasks.ceph:Copying monmap to all nodes... 2019-10-13T04:03:57.480 DEBUG:teuthology.orchestra.remote:smithi110:/etc/ceph/ceph.keyring is 206B 2019-10-13T04:03:57.547 DEBUG:teuthology.orchestra.remote:smithi110:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-10-13T04:03:57.560 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2019-10-13T04:03:57.560 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:57.561 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T04:03:57.618 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:57.619 INFO:teuthology.orchestra.run.smithi204:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:03:57.728 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2019-10-13T04:03:57.728 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.729 INFO:teuthology.orchestra.run.smithi110:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-13T04:03:57.793 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.793 INFO:teuthology.orchestra.run.smithi110:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:03:57.906 INFO:tasks.ceph:Setting up mon nodes... 2019-10-13T04:03:57.906 INFO:tasks.ceph:Setting up mgr nodes... 2019-10-13T04:03:57.906 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:57.906 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T04:03:57.949 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-10-13T04:03:57.951 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:57.951 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T04:03:57.994 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-10-13T04:03:57.996 INFO:tasks.ceph:Setting up mds nodes... 2019-10-13T04:03:57.996 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:57.997 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T04:03:58.038 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2019-10-13T04:03:58.041 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.041 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2019-10-13T04:03:58.121 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:58.121 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T04:03:58.167 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2019-10-13T04:03:58.169 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:58.169 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2019-10-13T04:03:58.248 INFO:tasks.ceph_client:Setting up client nodes... 2019-10-13T04:03:58.249 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:03:58.249 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-10-13T04:03:58.348 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-10-13T04:03:58.359 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-10-13T04:03:58.359 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {}} 2019-10-13T04:03:58.360 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.360 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-10-13T04:03:58.376 INFO:tasks.ceph:{} 2019-10-13T04:03:58.377 INFO:tasks.ceph:{} 2019-10-13T04:03:58.377 INFO:tasks.ceph:osd.4 2019-10-13T04:03:58.377 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.377 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-10-13T04:03:58.456 INFO:tasks.ceph:{} 2019-10-13T04:03:58.456 INFO:tasks.ceph:{} 2019-10-13T04:03:58.457 INFO:tasks.ceph:osd.5 2019-10-13T04:03:58.457 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.457 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-10-13T04:03:58.535 INFO:tasks.ceph:{} 2019-10-13T04:03:58.535 INFO:tasks.ceph:{} 2019-10-13T04:03:58.535 INFO:tasks.ceph:osd.6 2019-10-13T04:03:58.536 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.536 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-10-13T04:03:58.615 INFO:tasks.ceph:{} 2019-10-13T04:03:58.615 INFO:tasks.ceph:{} 2019-10-13T04:03:58.615 INFO:tasks.ceph:osd.7 2019-10-13T04:03:58.616 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:03:58.616 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:03:58.752 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-10-13T04:03:58.752 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-10-13T04:03:58.753 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T04:03:58.753 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T04:03:58.753 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-13T04:03:58.753 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:03:58.748+0000 7f7019243d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-10-13T04:04:01.234 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:01.235 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-10-13T04:04:01.271 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:01.272 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:01.379 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-10-13T04:04:01.380 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-10-13T04:04:01.380 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T04:04:01.380 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T04:04:01.380 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-13T04:04:01.381 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:01.376+0000 7f70633b0d80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-10-13T04:04:03.845 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:03.846 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-10-13T04:04:03.926 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:03.926 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:04.029 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-10-13T04:04:04.029 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-10-13T04:04:04.030 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T04:04:04.030 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T04:04:04.030 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-10-13T04:04:04.030 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:04.024+0000 7f2d4d5c1d80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-10-13T04:04:06.469 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:06.469 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-10-13T04:04:06.550 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:06.550 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:06.653 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-10-13T04:04:06.653 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-10-13T04:04:06.653 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T04:04:06.653 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T04:04:06.653 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-10-13T04:04:06.654 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-13T04:04:06.652+0000 7f94010e3d80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-10-13T04:04:09.105 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:09.105 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-10-13T04:04:09.185 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:09.185 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-10-13T04:04:09.202 INFO:tasks.ceph:{} 2019-10-13T04:04:09.202 INFO:tasks.ceph:{} 2019-10-13T04:04:09.203 INFO:tasks.ceph:osd.0 2019-10-13T04:04:09.203 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:09.203 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-10-13T04:04:09.285 INFO:tasks.ceph:{} 2019-10-13T04:04:09.285 INFO:tasks.ceph:{} 2019-10-13T04:04:09.285 INFO:tasks.ceph:osd.1 2019-10-13T04:04:09.286 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:09.286 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-10-13T04:04:09.371 INFO:tasks.ceph:{} 2019-10-13T04:04:09.371 INFO:tasks.ceph:{} 2019-10-13T04:04:09.371 INFO:tasks.ceph:osd.2 2019-10-13T04:04:09.372 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:09.372 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-10-13T04:04:09.452 INFO:tasks.ceph:{} 2019-10-13T04:04:09.452 INFO:tasks.ceph:{} 2019-10-13T04:04:09.453 INFO:tasks.ceph:osd.3 2019-10-13T04:04:09.453 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:09.453 INFO:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-10-13T04:04:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-10-13T04:04:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T04:04:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T04:04:09.572 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-13T04:04:09.572 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:09.568+0000 7f2a69bf3d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-10-13T04:04:11.999 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:11.999 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-10-13T04:04:12.015 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:12.015 INFO:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.112+0000 7ff1b11f0d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-10-13T04:04:12.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.112+0000 7ff1b11f0d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-10-13T04:04:12.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.116+0000 7ff1b11f0d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T04:04:12.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.116+0000 7ff1b11f0d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T04:04:12.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.116+0000 7ff1b11f0d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-13T04:04:12.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:12.116+0000 7ff1b11f0d80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-10-13T04:04:14.574 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:14.574 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-10-13T04:04:14.657 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:14.657 INFO:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-10-13T04:04:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-10-13T04:04:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T04:04:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T04:04:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-13T04:04:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:14.756+0000 7f8123f51d80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-10-13T04:04:17.238 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:17.238 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-10-13T04:04:17.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:17.318 INFO:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-10-13T04:04:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-10-13T04:04:17.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T04:04:17.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T04:04:17.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-13T04:04:17.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:17.428+0000 7f76c2441d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-10-13T04:04:19.886 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:19.886 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-10-13T04:04:19.966 INFO:tasks.ceph:Reading keys from all nodes... 2019-10-13T04:04:19.966 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:19.967 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:19.974 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.LtrXx0wJ9y 2019-10-13T04:04:19.975 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:19.975 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.LtrXx0wJ9y 2019-10-13T04:04:20.061 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.061 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.LtrXx0wJ9y 2019-10-13T04:04:20.221 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.LtrXx0wJ9y is 56B 2019-10-13T04:04:20.250 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.250 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.LtrXx0wJ9y 2019-10-13T04:04:20.257 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.258 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:20.328 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.S1yF2NoO7Q 2019-10-13T04:04:20.329 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.329 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.S1yF2NoO7Q 2019-10-13T04:04:20.409 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.409 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.S1yF2NoO7Q 2019-10-13T04:04:20.564 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.S1yF2NoO7Q is 56B 2019-10-13T04:04:20.576 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.576 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.S1yF2NoO7Q 2019-10-13T04:04:20.583 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.583 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:20.653 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.MqCKu2hwJH 2019-10-13T04:04:20.653 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.654 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.MqCKu2hwJH 2019-10-13T04:04:20.733 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.734 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.MqCKu2hwJH 2019-10-13T04:04:20.900 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.MqCKu2hwJH is 56B 2019-10-13T04:04:20.912 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.912 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.MqCKu2hwJH 2019-10-13T04:04:20.919 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.919 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:20.989 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.iNhicgNyBZ 2019-10-13T04:04:20.990 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:20.990 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.iNhicgNyBZ 2019-10-13T04:04:21.076 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.076 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.iNhicgNyBZ 2019-10-13T04:04:21.231 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.iNhicgNyBZ is 56B 2019-10-13T04:04:21.244 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.244 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.iNhicgNyBZ 2019-10-13T04:04:21.253 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.253 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:21.328 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.FfcCTOsfYX 2019-10-13T04:04:21.328 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.329 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.FfcCTOsfYX 2019-10-13T04:04:21.407 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.408 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.FfcCTOsfYX 2019-10-13T04:04:21.574 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.FfcCTOsfYX is 56B 2019-10-13T04:04:21.615 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.615 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.FfcCTOsfYX 2019-10-13T04:04:21.622 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.622 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:21.692 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.3NFvGshSDH 2019-10-13T04:04:21.692 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.693 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.3NFvGshSDH 2019-10-13T04:04:21.772 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:21.772 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.3NFvGshSDH 2019-10-13T04:04:21.963 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.3NFvGshSDH is 56B 2019-10-13T04:04:22.027 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:22.027 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.3NFvGshSDH 2019-10-13T04:04:22.049 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.049 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:22.064 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.io4SZqzp2y 2019-10-13T04:04:22.064 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.064 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.io4SZqzp2y 2019-10-13T04:04:22.153 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.153 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.io4SZqzp2y 2019-10-13T04:04:22.307 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.io4SZqzp2y is 56B 2019-10-13T04:04:22.338 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.338 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.io4SZqzp2y 2019-10-13T04:04:22.346 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.346 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:22.417 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.xmuVKCCqYj 2019-10-13T04:04:22.417 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.417 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.xmuVKCCqYj 2019-10-13T04:04:22.500 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.500 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.xmuVKCCqYj 2019-10-13T04:04:22.653 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.xmuVKCCqYj is 56B 2019-10-13T04:04:22.665 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.665 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.xmuVKCCqYj 2019-10-13T04:04:22.672 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.672 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:22.756 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.875cyPb6vS 2019-10-13T04:04:22.757 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.757 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.875cyPb6vS 2019-10-13T04:04:22.837 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:22.837 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.875cyPb6vS 2019-10-13T04:04:22.996 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.875cyPb6vS is 56B 2019-10-13T04:04:23.009 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.009 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.875cyPb6vS 2019-10-13T04:04:23.015 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.015 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:23.086 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.wz6NQYAao4 2019-10-13T04:04:23.086 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.086 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.wz6NQYAao4 2019-10-13T04:04:23.152 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.152 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.wz6NQYAao4 2019-10-13T04:04:23.313 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.wz6NQYAao4 is 56B 2019-10-13T04:04:23.326 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.326 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.wz6NQYAao4 2019-10-13T04:04:23.333 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.336 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:23.412 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.X5Jkjzdk43 2019-10-13T04:04:23.412 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.412 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.X5Jkjzdk43 2019-10-13T04:04:23.498 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.498 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.X5Jkjzdk43 2019-10-13T04:04:23.650 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.X5Jkjzdk43 is 56B 2019-10-13T04:04:23.663 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.663 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.X5Jkjzdk43 2019-10-13T04:04:23.671 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.671 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:23.756 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.8HPR4OTRGx 2019-10-13T04:04:23.757 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.757 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.8HPR4OTRGx 2019-10-13T04:04:23.844 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:23.845 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.8HPR4OTRGx 2019-10-13T04:04:23.996 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.8HPR4OTRGx is 56B 2019-10-13T04:04:24.008 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.008 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.8HPR4OTRGx 2019-10-13T04:04:24.092 DEBUG:teuthology.orchestra.remote:smithi110:/etc/ceph/ceph.client.0.keyring is 59B 2019-10-13T04:04:24.104 INFO:tasks.ceph:Adding keys to all mons... 2019-10-13T04:04:24.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.105 INFO:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T04:04:24.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.109 INFO:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAtoqJd05qhOBAAWPK5Xp2VYyYqMM4gsqKTGg== 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.b] 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAuoqJdGWxcAhAAH4z5zLjYAzG0cfGU86KPfQ== 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2019-10-13T04:04:24.131 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAuoqJd4LPfLBAAVGJ3FMeKlZeeGORpy74cUw== 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAxoqJdZHWnFhAAWeBQQs/b/Pm0kuSEADZEfg== 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA0oqJdiUTHARAAyGnJjnb9W9PPYWzrg86baw== 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA2oqJd6R/yJhAAtUgREMWLTcxvE+g2J4kVag== 2019-10-13T04:04:24.132 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2019-10-13T04:04:24.133 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAtoqJdPkxVOxAAHCmdbx1O/f5acBicsuaoig== 2019-10-13T04:04:24.133 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.a] 2019-10-13T04:04:24.133 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAuoqJdY5UKChAAb9IyWoeNxg0ddwfU+owh+g== 2019-10-13T04:04:24.133 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2019-10-13T04:04:24.133 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA5oqJdAl8YIhAAq4woSIOvr1OrF3wdNbIEDA== 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA8oqJd9oUBBxAAF1ST1mX69y2hfqtMsuogEA== 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA+oqJdHb07LRAAHdL4orMSY/Ohuqkv+YyU5w== 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2019-10-13T04:04:24.134 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBoqJdU+OqGRAAEYlP+wA04pbPSmTSUpRRBQ== 2019-10-13T04:04:24.135 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2019-10-13T04:04:24.135 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAuoqJdQp3SFBAAoxmcAWBoxlcA0j+dJBDwdg== 2019-10-13T04:04:24.157 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2019-10-13T04:04:24.159 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAtoqJd05qhOBAAWPK5Xp2VYyYqMM4gsqKTGg== 2019-10-13T04:04:24.160 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2019-10-13T04:04:24.160 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAuoqJdGWxcAhAAH4z5zLjYAzG0cfGU86KPfQ== 2019-10-13T04:04:24.160 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2019-10-13T04:04:24.160 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAuoqJd4LPfLBAAVGJ3FMeKlZeeGORpy74cUw== 2019-10-13T04:04:24.160 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2019-10-13T04:04:24.161 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAxoqJdZHWnFhAAWeBQQs/b/Pm0kuSEADZEfg== 2019-10-13T04:04:24.161 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2019-10-13T04:04:24.161 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA0oqJdiUTHARAAyGnJjnb9W9PPYWzrg86baw== 2019-10-13T04:04:24.162 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2019-10-13T04:04:24.162 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA2oqJd6R/yJhAAtUgREMWLTcxvE+g2J4kVag== 2019-10-13T04:04:24.162 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2019-10-13T04:04:24.162 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAtoqJdPkxVOxAAHCmdbx1O/f5acBicsuaoig== 2019-10-13T04:04:24.162 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAuoqJdY5UKChAAb9IyWoeNxg0ddwfU+owh+g== 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA5oqJdAl8YIhAAq4woSIOvr1OrF3wdNbIEDA== 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA8oqJd9oUBBxAAF1ST1mX69y2hfqtMsuogEA== 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2019-10-13T04:04:24.163 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA+oqJdHb07LRAAHdL4orMSY/Ohuqkv+YyU5w== 2019-10-13T04:04:24.164 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2019-10-13T04:04:24.164 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBBoqJdU+OqGRAAEYlP+wA04pbPSmTSUpRRBQ== 2019-10-13T04:04:24.164 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2019-10-13T04:04:24.164 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAuoqJdQp3SFBAAoxmcAWBoxlcA0j+dJBDwdg== 2019-10-13T04:04:24.164 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.165 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T04:04:24.232 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.232 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T04:04:24.275 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.275 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.344 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.344 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.389 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.389 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.456 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.456 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.499 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.499 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.569 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.570 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.614 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.615 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.682 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.682 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.727 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.727 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.795 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.795 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:24.838 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.838 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T04:04:24.905 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:24.905 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-13T04:04:24.947 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:24.947 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.015 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.015 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.062 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.129 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.129 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.171 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.171 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.238 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.238 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.283 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.283 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.350 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.350 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.394 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.394 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.461 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.462 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-13T04:04:25.506 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.506 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T04:04:25.546 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.546 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-13T04:04:25.593 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-10-13T04:04:25.594 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.594 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-10-13T04:04:25.673 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.674 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T04:04:25.794 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.794 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-10-13T04:04:25.809 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.809 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-10-13T04:04:25.877 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:25.877 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:25.881 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.881 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:25.894 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:25.894 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T04:04:26.040 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.040 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-10-13T04:04:26.054 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.054 INFO:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-10-13T04:04:26.070 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.071 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-13T04:04:26.213 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.213 INFO:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-10-13T04:04:26.229 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.229 INFO:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:26.300 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.300 INFO:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-13T04:04:26.307 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-10-13T04:04:26.307 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.307 INFO:teuthology.orchestra.run.smithi204:> which systemctl 2019-10-13T04:04:26.381 INFO:teuthology.orchestra.run.smithi204.stdout:/bin/systemctl 2019-10-13T04:04:26.382 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-13T04:04:26.382 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.382 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-13T04:04:26.449 INFO:tasks.ceph.mon.b:Started 2019-10-13T04:04:26.450 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-13T04:04:26.450 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.450 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-13T04:04:26.453 INFO:tasks.ceph.mon.c:Started 2019-10-13T04:04:26.453 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.454 INFO:teuthology.orchestra.run.smithi110:> which systemctl 2019-10-13T04:04:26.461 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/systemctl 2019-10-13T04:04:26.461 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-13T04:04:26.462 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.462 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-13T04:04:26.528 INFO:tasks.ceph.mon.a:Started 2019-10-13T04:04:26.529 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-10-13T04:04:26.529 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-10-13T04:04:26.529 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.529 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-10-13T04:04:26.532 INFO:tasks.ceph.mgr.x:Started 2019-10-13T04:04:26.533 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-10-13T04:04:26.533 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.533 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-10-13T04:04:26.539 INFO:tasks.ceph.mgr.y:Started 2019-10-13T04:04:26.539 INFO:tasks.ceph:Setting crush tunables to default 2019-10-13T04:04:26.539 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:26.539 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2019-10-13T04:04:26.698 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.636+0000 7f236ed79700 1 Processor -- start 2019-10-13T04:04:26.698 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.636+0000 7f236ed79700 1 -- start start 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236ed79700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236ed79700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 0x7f236814e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236ed79700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236805a200 con 0x7f23680a61a0 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236ed79700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680a8820 con 0x7f23680a75f0 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236ed79700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2368059f60 con 0x7f23680aa2b0 2019-10-13T04:04:26.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42082/0 (socket says 172.21.15.110:42082) 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 -- 172.21.15.110:0/3891949675 learned_addr learned my addr 172.21.15.110:0/3891949675 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 -- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 msgr2=0x7f23680a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 -- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 msgr2=0x7f23680a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 --2- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T04:04:26.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236d316700 1 --2- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236db17700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 0x7f236814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236db17700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 msgr2=0x7f236814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236db17700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 msgr2=0x7f236814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236db17700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 0x7f236814e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.640+0000 7f236db17700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 0x7f236814e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.680+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174154246 0 0) 0x7f236805a200 con 0x7f23680a61a0 2019-10-13T04:04:26.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.680+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23540030a0 con 0x7f23680a61a0 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1343846173 0 0) 0x7f23540030a0 con 0x7f23680a61a0 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 msgr2=0x7f236814e900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23680aa2b0 0x7f236814e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 msgr2=0x7f23680a12f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 --2- 172.21.15.110:0/3891949675 >> v2:172.21.15.110:3300/0 conn(0x7f23680a75f0 0x7f23680a12f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:26.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236807ec90 con 0x7f23680a61a0 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.684+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f2364001750 con 0x7f23680a61a0 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 >> v1:172.21.15.204:6789/0 conn(0x7f23680a61a0 legacy=0x7f236807e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23540036d0 0x7f2354003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2354004d80 0x7f235404f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23680a8820 con 0x7f2354004410 2019-10-13T04:04:26.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2368059f60 con 0x7f2354004d80 2019-10-13T04:04:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2354005220 con 0x7f23540036d0 2019-10-13T04:04:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236db17700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2354004d80 0x7f235404f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236d316700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23540036d0 0x7f2354003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23540036d0 msgr2=0x7f2354003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23540036d0 0x7f2354003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2354004d80 msgr2=0x7f235404f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2354004d80 0x7f235404f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 -- 172.21.15.110:0/3891949675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235404fcd0 con 0x7f2354004410 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236db17700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2354004d80 0x7f235404f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f23540053e0 tx=0x7f2360008120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f23540053e0 tx=0x7f2360008120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.688+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=READY pgs=4 cs=0 l=1 rx=0x7f23540053e0 tx=0x7f2360008120).ready entity=mon.0 client_cookie=e7449ca77602be47 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=THROTTLE_DONE pgs=4 cs=0 l=1 rx=0x7f23540053e0 tx=0x7f2360008120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f236400ed70 con 0x7f2354004410 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=THROTTLE_DONE pgs=4 cs=0 l=1 rx=0x7f23540053e0 tx=0x7f2360008120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236400e990 con 0x7f2354004410 2019-10-13T04:04:26.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236cb15700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=THROTTLE_DONE pgs=4 cs=0 l=1 rx=0x7f23540053e0 tx=0x7f2360008120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f235bfff700 1 -- 172.21.15.110:0/3891949675 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2364006750 con 0x7f2354004410 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 -- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 msgr2=0x7f2354004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 --2- 172.21.15.110:0/3891949675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2354004410 0x7f2354004830 secure :-1 s=READY pgs=4 cs=0 l=1 rx=0x7f23540053e0 tx=0x7f2360008120).stop 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236db17700 1 -- 172.21.15.110:0/3891949675 reap_dead start 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 -- 172.21.15.110:0/3891949675 shutdown_connections 2019-10-13T04:04:26.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 -- 172.21.15.110:0/3891949675 shutdown_connections 2019-10-13T04:04:26.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 -- 172.21.15.110:0/3891949675 wait complete. 2019-10-13T04:04:26.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.692+0000 7f236ed79700 1 -- 172.21.15.110:0/3891949675 >> 172.21.15.110:0/3891949675 conn(0x7f236805e980 msgr2=0x7f23680a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.696+0000 7f236ed79700 1 Processor -- start 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.696+0000 7f236ed79700 1 -- start start 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2368129e90 0x7f23680641e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2368143aa0 0x7f236812d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:26.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2368064730 con 0x7f2368143aa0 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680651c0 con 0x7f2368143680 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2368063f40 con 0x7f2368129e90 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2368143aa0 0x7f236812d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236cb15700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2368143aa0 0x7f236812d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42094/0 (socket says 172.21.15.110:42094) 2019-10-13T04:04:26.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236cb15700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.110:51504/0 (socket says 172.21.15.110:51504) 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 -- 172.21.15.110:0/785701758 learned_addr learned my addr 172.21.15.110:0/785701758 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236d316700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2368129e90 0x7f23680641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 -- 172.21.15.110:0/785701758 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2368129e90 msgr2=0x7f23680641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2368129e90 0x7f23680641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 -- 172.21.15.110:0/785701758 >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 msgr2=0x7f2368121ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:26.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 -- 172.21.15.110:0/785701758 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368142360 con 0x7f2368143aa0 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236cb15700 1 --2- 172.21.15.110:0/785701758 >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> v2:172.21.15.110:3300/0 conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> v2:172.21.15.110:3300/0 conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=READY pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2360005c90 con 0x7f2368143aa0 2019-10-13T04:04:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23600058b0 con 0x7f2368143aa0 2019-10-13T04:04:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2360008c60 con 0x7f2368143aa0 2019-10-13T04:04:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2368064a50 con 0x7f2368143aa0 2019-10-13T04:04:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2368122010 con 0x7f2368143aa0 2019-10-13T04:04:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.725 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 341+0+0 (secure 0 0 0) 0x7f23600074e0 con 0x7f2368143aa0 2019-10-13T04:04:26.726 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.726 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.700+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2360008620 con 0x7f2368143aa0 2019-10-13T04:04:26.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.752+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23680555c0 con 0x7f2368143aa0 2019-10-13T04:04:26.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.760+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.760+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7f2360013060 con 0x7f2368143aa0 2019-10-13T04:04:26.856 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.852+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f2368066560 con 0x7f2368143aa0 2019-10-13T04:04:26.863 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.860+0000 7f236db17700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=THROTTLE_DONE pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:26.863 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.860+0000 7f2359ffb700 1 -- 172.21.15.110:0/785701758 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2360012aa0 con 0x7f2368143aa0 2019-10-13T04:04:26.863 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 msgr2=0x7f236812d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 secure :-1 s=READY pgs=5 cs=0 l=1 rx=0x7f2368065380 tx=0x7f236400ba10).stop 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 shutdown_connections 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2368143aa0 0x7f236812d450 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 --2- 172.21.15.110:0/785701758 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2368129e90 0x7f23680641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:26.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 --2- 172.21.15.110:0/785701758 >> v2:172.21.15.204:3300/0 conn(0x7f2368143680 0x7f2368121ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 shutdown_connections 2019-10-13T04:04:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 wait complete. 2019-10-13T04:04:26.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:26.864+0000 7f236ed79700 1 -- 172.21.15.110:0/785701758 >> 172.21.15.110:0/785701758 conn(0x7f23680a5ce0 msgr2=0x7f23680a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:26.877 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-10-13T04:04:26.878 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.878 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:26.884 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.MuMckdAPqd 2019-10-13T04:04:26.884 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.885 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.MuMckdAPqd 2019-10-13T04:04:26.965 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:26.965 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.MuMckdAPqd 2019-10-13T04:04:27.121 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.MuMckdAPqd is 37B 2019-10-13T04:04:27.133 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.133 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.MuMckdAPqd 2019-10-13T04:04:27.148 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.148 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:27.218 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.oeQBxvVTcl 2019-10-13T04:04:27.218 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.219 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.oeQBxvVTcl 2019-10-13T04:04:27.298 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.298 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.oeQBxvVTcl 2019-10-13T04:04:27.450 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.oeQBxvVTcl is 37B 2019-10-13T04:04:27.461 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.461 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.oeQBxvVTcl 2019-10-13T04:04:27.633 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.633 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:27.641 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.g4jGyuVRry 2019-10-13T04:04:27.641 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.641 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.g4jGyuVRry 2019-10-13T04:04:27.720 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.720 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.g4jGyuVRry 2019-10-13T04:04:27.875 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.g4jGyuVRry is 37B 2019-10-13T04:04:27.887 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.887 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.g4jGyuVRry 2019-10-13T04:04:27.894 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.894 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-13T04:04:27.964 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.IOFezPAvok 2019-10-13T04:04:27.964 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:27.964 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.IOFezPAvok 2019-10-13T04:04:28.046 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:28.046 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.IOFezPAvok 2019-10-13T04:04:28.204 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.IOFezPAvok is 37B 2019-10-13T04:04:28.215 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:28.215 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.IOFezPAvok 2019-10-13T04:04:28.222 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.222 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:28.231 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.495I8TNdCA 2019-10-13T04:04:28.231 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.232 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.495I8TNdCA 2019-10-13T04:04:28.319 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.319 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.495I8TNdCA 2019-10-13T04:04:28.471 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.495I8TNdCA is 37B 2019-10-13T04:04:28.485 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.486 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.495I8TNdCA 2019-10-13T04:04:28.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.493 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:28.569 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.52Oztf8gI3 2019-10-13T04:04:28.569 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.569 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.52Oztf8gI3 2019-10-13T04:04:28.649 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.650 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.52Oztf8gI3 2019-10-13T04:04:28.802 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.52Oztf8gI3 is 37B 2019-10-13T04:04:28.858 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.859 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.52Oztf8gI3 2019-10-13T04:04:28.870 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.870 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:28.942 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.VgNpHduiIx 2019-10-13T04:04:28.942 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:28.942 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.VgNpHduiIx 2019-10-13T04:04:29.024 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.024 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.VgNpHduiIx 2019-10-13T04:04:29.182 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.VgNpHduiIx is 37B 2019-10-13T04:04:29.235 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.236 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.VgNpHduiIx 2019-10-13T04:04:29.243 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.243 INFO:teuthology.orchestra.run.smithi110:> mktemp 2019-10-13T04:04:29.316 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.yvAP8OMxRo 2019-10-13T04:04:29.316 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.317 INFO:teuthology.orchestra.run.smithi110:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.yvAP8OMxRo 2019-10-13T04:04:29.398 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.398 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.yvAP8OMxRo 2019-10-13T04:04:29.549 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.yvAP8OMxRo is 37B 2019-10-13T04:04:29.600 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.601 INFO:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.yvAP8OMxRo 2019-10-13T04:04:29.607 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.608 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 91e5f49d-d1d1-4381-9d98-0003d595cfcb 0 2019-10-13T04:04:29.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 Processor -- start 2019-10-13T04:04:29.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 -- start start 2019-10-13T04:04:29.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0a75f0 0x7f1c6c0a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 0x7f1c6c07e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c0a8820 con 0x7f1c6c0aa2b0 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c059f60 con 0x7f1c6c0a61a0 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c05a200 con 0x7f1c6c0a75f0 2019-10-13T04:04:29.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42114/0 (socket says 172.21.15.110:42114) 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 -- 172.21.15.110:0/235453295 learned_addr learned my addr 172.21.15.110:0/235453295 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6ad9d700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0a75f0 0x7f1c6c0a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 0x7f1c6c07e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 -- 172.21.15.110:0/235453295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0a75f0 msgr2=0x7f1c6c0a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0a75f0 0x7f1c6c0a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 -- 172.21.15.110:0/235453295 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 msgr2=0x7f1c6c07e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 0x7f1c6c07e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 -- 172.21.15.110:0/235453295 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c0a1840 con 0x7f1c6c0aa2b0 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 0x7f1c6c07e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.110:3300/0 conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1c640032b0 tx=0x7f1c60007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.110:3300/0 conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1c640032b0 tx=0x7f1c60007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f1c640032b0 tx=0x7f1c60007710).ready entity=mon.0 client_cookie=30ab75374e8bfdc4 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:29.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f1c640032b0 tx=0x7f1c60007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c69d9b700 1 -- 172.21.15.110:0/235453295 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c640109e0 con 0x7f1c6c0aa2b0 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f1c640032b0 tx=0x7f1c60007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c6b59e700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f1c640032b0 tx=0x7f1c60007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c69d9b700 1 -- 172.21.15.110:0/235453295 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64010600 con 0x7f1c6c0aa2b0 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c69d9b700 1 -- 172.21.15.110:0/235453295 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c640110c0 con 0x7f1c6c0aa2b0 2019-10-13T04:04:29.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 -- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 msgr2=0x7f1c6c07e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.756+0000 7f1c708ba700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f1c640032b0 tx=0x7f1c60007710).stop 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 -- 172.21.15.110:0/235453295 shutdown_connections 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c0aa2b0 0x7f1c6c07e9e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 --2- 172.21.15.110:0/235453295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0a75f0 0x7f1c6c0a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 --2- 172.21.15.110:0/235453295 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0a61a0 0x7f1c6c07e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.768 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 -- 172.21.15.110:0/235453295 shutdown_connections 2019-10-13T04:04:29.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 -- 172.21.15.110:0/235453295 wait complete. 2019-10-13T04:04:29.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 -- 172.21.15.110:0/235453295 >> 172.21.15.110:0/235453295 conn(0x7f1c6c05e980 msgr2=0x7f1c6c0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:29.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 Processor -- start 2019-10-13T04:04:29.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.760+0000 7f1c708ba700 1 -- start start 2019-10-13T04:04:29.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1c6c12c870 0x7f1c6c0a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c118510 0x7f1c6c09e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c09e160 con 0x7f1c6c11cef0 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c0a0730 con 0x7f1c6c118510 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c0a0990 con 0x7f1c6c12c870 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c118510 0x7f1c6c09e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c6c118510 0x7f1c6c09e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42118/0 (socket says 172.21.15.110:42118) 2019-10-13T04:04:29.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 -- 172.21.15.110:0/1690868187 learned_addr learned my addr 172.21.15.110:0/1690868187 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6ad9d700 1 --2- 172.21.15.110:0/1690868187 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c12c870 0x7f1c6c0a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 -- 172.21.15.110:0/1690868187 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c12c870 msgr2=0x7f1c6c0a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c12c870 0x7f1c6c0a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 -- 172.21.15.110:0/1690868187 >> v1:172.21.15.204:6789/0 conn(0x7f1c6c11cef0 legacy=0x7f1c6c11d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 -- 172.21.15.110:0/1690868187 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c07de90 con 0x7f1c6c118510 2019-10-13T04:04:29.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> v2:172.21.15.110:3300/0 conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> v2:172.21.15.110:3300/0 conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c60005920 con 0x7f1c6c118510 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.775 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c60007160 con 0x7f1c6c118510 2019-10-13T04:04:29.776 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c0a5e00 con 0x7f1c6c118510 2019-10-13T04:04:29.776 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c60012260 con 0x7f1c6c118510 2019-10-13T04:04:29.779 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c11db10 con 0x7f1c6c118510 2019-10-13T04:04:29.779 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.779 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.779 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 2) v1 ==== 16220+0+0 (secure 0 0 0) 0x7f1c60012570 con 0x7f1c6c118510 2019-10-13T04:04:29.779 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.764+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1c6001f140 con 0x7f1c6c118510 2019-10-13T04:04:29.780 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.772+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c0555c0 con 0x7f1c6c118510 2019-10-13T04:04:29.780 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.776+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.780 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.776+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7f1c6001e4e0 con 0x7f1c6c118510 2019-10-13T04:04:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.864+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "91e5f49d-d1d1-4381-9d98-0003d595cfcb", "id": 0} v 0) v1 -- 0x7f1c6c05f050 con 0x7f1c6c118510 2019-10-13T04:04:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.872+0000 7f1c6a59c700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.874 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.872+0000 7f1c5b7fe700 1 -- 172.21.15.110:0/1690868187 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "91e5f49d-d1d1-4381-9d98-0003d595cfcb", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1c60012020 con 0x7f1c6c118510 2019-10-13T04:04:29.875 INFO:teuthology.orchestra.run.smithi110.stdout:0 2019-10-13T04:04:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.872+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 msgr2=0x7f1c6c09e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.872+0000 7f1c708ba700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f1c6c09d780 tx=0x7f1c5c007660).stop 2019-10-13T04:04:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 shutdown_connections 2019-10-13T04:04:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 --2- 172.21.15.110:0/1690868187 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c12c870 0x7f1c6c0a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 --2- 172.21.15.110:0/1690868187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c118510 0x7f1c6c09e4e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 shutdown_connections 2019-10-13T04:04:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 wait complete. 2019-10-13T04:04:29.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.876+0000 7f1c708ba700 1 -- 172.21.15.110:0/1690868187 >> 172.21.15.110:0/1690868187 conn(0x7f1c6c0a75f0 msgr2=0x7f1c6c0a7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:29.885 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:29.885 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 1ae65af5-8cf4-42f4-bbed-edc4a2b31b80 1 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 Processor -- start 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 -- start start 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba9c0a61a0 0x7fba9c07e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9c05a200 con 0x7fba9c0a75f0 2019-10-13T04:04:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c0a8820 con 0x7fba9c0aa2b0 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fbaa1750700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c059f60 con 0x7fba9c0a61a0 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42126/0 (socket says 172.21.15.110:42126) 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 -- 172.21.15.110:0/578538456 learned_addr learned my addr 172.21.15.110:0/578538456 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9b7fe700 1 --1- 172.21.15.110:0/578538456 >> v1:172.21.15.110:6789/0 conn(0x7fba9c0a75f0 0x7fba9c0a12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44834/0 (socket says 172.21.15.110:44834) 2019-10-13T04:04:29.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9affd700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba9c0a61a0 0x7fba9c07e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 -- 172.21.15.110:0/578538456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba9c0a61a0 msgr2=0x7fba9c07e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba9c0a61a0 0x7fba9c07e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 -- 172.21.15.110:0/578538456 >> v1:172.21.15.110:6789/0 conn(0x7fba9c0a75f0 legacy=0x7fba9c0a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.972+0000 7fba9bfff700 1 -- 172.21.15.110:0/578538456 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c059a20 con 0x7fba9c0aa2b0 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> v2:172.21.15.110:3300/0 conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fba94008b00 tx=0x7fba90002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> v2:172.21.15.110:3300/0 conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fba94008b00 tx=0x7fba90002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7fba94008b00 tx=0x7fba90002ff0).ready entity=mon.0 client_cookie=c95c2a28de3a89e3 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fba94008b00 tx=0x7fba90002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9a7fc700 1 -- 172.21.15.110:0/578538456 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba94010d00 con 0x7fba9c0aa2b0 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fba94008b00 tx=0x7fba90002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9bfff700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7fba94008b00 tx=0x7fba90002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9a7fc700 1 -- 172.21.15.110:0/578538456 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba94010920 con 0x7fba9c0aa2b0 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fba9a7fc700 1 -- 172.21.15.110:0/578538456 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba940114a0 con 0x7fba9c0aa2b0 2019-10-13T04:04:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 -- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 msgr2=0x7fba9c07e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7fba94008b00 tx=0x7fba90002ff0).stop 2019-10-13T04:04:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 -- 172.21.15.110:0/578538456 shutdown_connections 2019-10-13T04:04:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c0aa2b0 0x7fba9c07e9e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 --2- 172.21.15.110:0/578538456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba9c0a61a0 0x7fba9c07e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 -- 172.21.15.110:0/578538456 shutdown_connections 2019-10-13T04:04:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 -- 172.21.15.110:0/578538456 wait complete. 2019-10-13T04:04:29.986 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.976+0000 7fbaa1750700 1 -- 172.21.15.110:0/578538456 >> 172.21.15.110:0/578538456 conn(0x7fba9c05e980 msgr2=0x7fba9c0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:29.988 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 Processor -- start 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- start start 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fba9c09fa10 0x7fba9c0a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba9c09ea80 0x7fba9c129dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9c09eea0 con 0x7fba9c111410 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c09f190 con 0x7fba9c09ea80 2019-10-13T04:04:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c111fa0 con 0x7fba9c09fa10 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9b7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fba9c111410 0x7fba9c112240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44840/0 (socket says 172.21.15.110:44840) 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9b7fe700 1 -- 172.21.15.110:0/1596049358 learned_addr learned my addr 172.21.15.110:0/1596049358 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.110:3300/0 conn(0x7fba9c09ea80 0x7fba9c129dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9affd700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.204:3300/0 conn(0x7fba9c09fa10 0x7fba9c0a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320975829 0 0) 0x7fba9c09eea0 con 0x7fba9c111410 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c0030a0 con 0x7fba9c111410 2019-10-13T04:04:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 -- 172.21.15.110:0/1596049358 >> v2:172.21.15.204:3300/0 conn(0x7fba9c09fa10 msgr2=0x7fba9c0a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.204:3300/0 conn(0x7fba9c09fa10 0x7fba9c0a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 -- 172.21.15.110:0/1596049358 >> v1:172.21.15.110:6789/0 conn(0x7fba9c111410 legacy=0x7fba9c112240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 -- 172.21.15.110:0/1596049358 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c12a320 con 0x7fba9c09ea80 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.110:3300/0 conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.110:3300/0 conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba94005aa0 con 0x7fba9c09ea80 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba940110c0 con 0x7fba9c09ea80 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c12a5f0 con 0x7fba9c09ea80 2019-10-13T04:04:29.992 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba9400f680 con 0x7fba9c09ea80 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.980+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c12ab40 con 0x7fba9c09ea80 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.984+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.984+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 2) v1 ==== 16220+0+0 (secure 0 0 0) 0x7fba9400f990 con 0x7fba9c09ea80 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.984+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.984+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fba9401f0d0 con 0x7fba9c09ea80 2019-10-13T04:04:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.988+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c0555c0 con 0x7fba9c09ea80 2019-10-13T04:04:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.992+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:29.992+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7fba9400fc80 con 0x7fba9c09ea80 2019-10-13T04:04:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.088+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ae65af5-8cf4-42f4-bbed-edc4a2b31b80", "id": 1} v 0) v1 -- 0x7fba9c111d50 con 0x7fba9c09ea80 2019-10-13T04:04:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.096+0000 7fba9bfff700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.096+0000 7fba83fff700 1 -- 172.21.15.110:0/1596049358 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ae65af5-8cf4-42f4-bbed-edc4a2b31b80", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba94012060 con 0x7fba9c09ea80 2019-10-13T04:04:30.104 INFO:teuthology.orchestra.run.smithi110.stdout:1 2019-10-13T04:04:30.106 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 msgr2=0x7fba9c129dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.106 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7fba9c09f350 tx=0x7fba90007a50).stop 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 shutdown_connections 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 --2- 172.21.15.110:0/1596049358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba9c09ea80 0x7fba9c129dd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 --2- 172.21.15.110:0/1596049358 >> v2:172.21.15.204:3300/0 conn(0x7fba9c09fa10 0x7fba9c0a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 shutdown_connections 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 wait complete. 2019-10-13T04:04:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.104+0000 7fbaa1750700 1 -- 172.21.15.110:0/1596049358 >> 172.21.15.110:0/1596049358 conn(0x7fba9c0aa2b0 msgr2=0x7fba9c09ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:30.118 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5 2 2019-10-13T04:04:30.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 Processor -- start 2019-10-13T04:04:30.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 -- start start 2019-10-13T04:04:30.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 0x7fe0b00a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:30.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a61a0 0x7fe0b007e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:30.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b005a200 con 0x7fe0b00aa2b0 2019-10-13T04:04:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b00a8820 con 0x7fe0b00a61a0 2019-10-13T04:04:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b6eda700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b0059f60 con 0x7fe0b00a75f0 2019-10-13T04:04:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a61a0 0x7fe0b007e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a61a0 0x7fe0b007e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42136/0 (socket says 172.21.15.110:42136) 2019-10-13T04:04:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 -- 172.21.15.110:0/2172596459 learned_addr learned my addr 172.21.15.110:0/2172596459 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:30.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b5477700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 0x7fe0b00a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 -- 172.21.15.110:0/2172596459 >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 msgr2=0x7fe0b00a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:30.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 0x7fe0b00a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:30.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 -- 172.21.15.110:0/2172596459 >> v1:172.21.15.204:6789/0 conn(0x7fe0b00aa2b0 legacy=0x7fe0b0147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:30.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 -- 172.21.15.110:0/2172596459 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00a1840 con 0x7fe0b00a61a0 2019-10-13T04:04:30.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b5477700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 0x7fe0b00a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:30.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).ready entity=mon.0 client_cookie=968af38d9ece355d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0a7fff700 1 -- 172.21.15.110:0/2172596459 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a00111e0 con 0x7fe0b00a61a0 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.216+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0a7fff700 1 -- 172.21.15.110:0/2172596459 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a0008a80 con 0x7fe0b00a61a0 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0a7fff700 1 -- 172.21.15.110:0/2172596459 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a00117c0 con 0x7fe0b00a61a0 2019-10-13T04:04:30.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 msgr2=0x7fe0b007e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7fe0b005a120 tx=0x7fe0ac002ff0).stop 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- 172.21.15.110:0/2172596459 shutdown_connections 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/2172596459 >> v2:172.21.15.204:3300/0 conn(0x7fe0b00a75f0 0x7fe0b00a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/2172596459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a61a0 0x7fe0b007e490 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- 172.21.15.110:0/2172596459 shutdown_connections 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- 172.21.15.110:0/2172596459 wait complete. 2019-10-13T04:04:30.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- 172.21.15.110:0/2172596459 >> 172.21.15.110:0/2172596459 conn(0x7fe0b005e980 msgr2=0x7fe0b00a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 Processor -- start 2019-10-13T04:04:30.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.220+0000 7fe0b6eda700 1 -- start start 2019-10-13T04:04:30.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a5620 0x7fe0b009dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0b01406c0 0x7fe0b0140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe0b0141030 0x7fe0b007b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b005f050 con 0x7fe0b00a5620 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b0148220 con 0x7fe0b0141030 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b009e750 con 0x7fe0b01406c0 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a5620 0x7fe0b009dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a5620 0x7fe0b009dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42140/0 (socket says 172.21.15.110:42140) 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 -- 172.21.15.110:0/3022275499 learned_addr learned my addr 172.21.15.110:0/3022275499 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0b01406c0 0x7fe0b0140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5c78700 1 --2- 172.21.15.110:0/3022275499 >> v2:172.21.15.204:3300/0 conn(0x7fe0b0141030 0x7fe0b007b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 -- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0b01406c0 msgr2=0x7fe0b0140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0b01406c0 0x7fe0b0140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 -- 172.21.15.110:0/3022275499 >> v2:172.21.15.204:3300/0 conn(0x7fe0b0141030 msgr2=0x7fe0b007b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> v2:172.21.15.204:3300/0 conn(0x7fe0b0141030 0x7fe0b007b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 -- 172.21.15.110:0/3022275499 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b009e330 con 0x7fe0b00a5620 2019-10-13T04:04:30.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> v2:172.21.15.110:3300/0 conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0ac00a160 con 0x7fe0b00a5620 2019-10-13T04:04:30.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac008ab0 con 0x7fe0b00a5620 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b007b750 con 0x7fe0b00a5620 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0ac00a740 con 0x7fe0b00a5620 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b007bc80 con 0x7fe0b00a5620 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 2) v1 ==== 16220+0+0 (secure 0 0 0) 0x7fe0ac00aa50 con 0x7fe0b00a5620 2019-10-13T04:04:30.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.224+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fe0ac01e520 con 0x7fe0b00a5620 2019-10-13T04:04:30.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.232+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b00555c0 con 0x7fe0b00a5620 2019-10-13T04:04:30.239 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.236+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.239 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.236+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67206 (secure 0 0 0) 0x7fe0ac002560 con 0x7fe0b00a5620 2019-10-13T04:04:30.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.240+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.240+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fe0ac001ec0 con 0x7fe0b00a5620 2019-10-13T04:04:30.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.240+0000 7fe0a5ffb700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 0x7fe090029760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.240+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 0x7fe090029760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.240+0000 7fe0b4c76700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 0x7fe090029760 crc :-1 s=READY pgs=2 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.332+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5", "id": 2} v 0) v1 -- 0x7fe0b0066560 con 0x7fe0b00a5620 2019-10-13T04:04:30.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.336+0000 7fe0b5477700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.336+0000 7fe0a5ffb700 1 -- 172.21.15.110:0/3022275499 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe0ac0021b0 con 0x7fe0b00a5620 2019-10-13T04:04:30.340 INFO:teuthology.orchestra.run.smithi110.stdout:2 2019-10-13T04:04:30.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 msgr2=0x7fe090029760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 0x7fe090029760 crc :-1 s=READY pgs=2 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 msgr2=0x7fe0b009dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fe0b01483e0 tx=0x7fe0a8007960).stop 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 shutdown_connections 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> v2:172.21.15.204:3300/0 conn(0x7fe0b0141030 0x7fe0b007b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe090027310 0x7fe090029760 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0b00a5620 0x7fe0b009dde0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 --2- 172.21.15.110:0/3022275499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0b01406c0 0x7fe0b0140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 shutdown_connections 2019-10-13T04:04:30.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 wait complete. 2019-10-13T04:04:30.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.340+0000 7fe0b6eda700 1 -- 172.21.15.110:0/3022275499 >> 172.21.15.110:0/3022275499 conn(0x7fe0b00a2c20 msgr2=0x7fe0b007e260 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.352 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:30.352 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new f38f8d49-36c3-4dfa-8bb5-057288982141 3 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 Processor -- start 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 -- start start 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f480aa2b0 0x7f9f4814f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4805a200 con 0x7f9f480a75f0 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4807ec90 con 0x7f9f480a61a0 2019-10-13T04:04:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f480a8820 con 0x7f9f480aa2b0 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f480aa2b0 0x7f9f4814f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e187700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9f480a75f0 0x7f9f480a12f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44866/0 (socket says 172.21.15.110:44866) 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f480aa2b0 0x7f9f4814f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42156/0 (socket says 172.21.15.110:42156) 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e187700 1 -- 172.21.15.110:0/3474422512 learned_addr learned my addr 172.21.15.110:0/3474422512 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 -- 172.21.15.110:0/3474422512 >> v1:172.21.15.204:6789/0 conn(0x7f9f480a61a0 legacy=0x7f9f4807e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 -- 172.21.15.110:0/3474422512 >> v1:172.21.15.110:6789/0 conn(0x7f9f480a75f0 legacy=0x7f9f480a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.446 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4d185700 1 -- 172.21.15.110:0/3474422512 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706362011 0 0) 0x7f9f4805a200 con 0x7f9f480a75f0 2019-10-13T04:04:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 -- 172.21.15.110:0/3474422512 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48059a20 con 0x7f9f480aa2b0 2019-10-13T04:04:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> v2:172.21.15.110:3300/0 conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> v2:172.21.15.110:3300/0 conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).ready entity=mon.0 client_cookie=b97c4de339ccb9b7 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4d185700 1 -- 172.21.15.110:0/3474422512 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f38011260 con 0x7f9f480aa2b0 2019-10-13T04:04:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4d185700 1 -- 172.21.15.110:0/3474422512 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f38008bb0 con 0x7f9f480aa2b0 2019-10-13T04:04:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4e988700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4d185700 1 -- 172.21.15.110:0/3474422512 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f38011840 con 0x7f9f480aa2b0 2019-10-13T04:04:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.436+0000 7f9f4fbea700 1 -- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 msgr2=0x7f9f4814f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f9f38008780 tx=0x7f9f3800fbe0).stop 2019-10-13T04:04:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 -- 172.21.15.110:0/3474422512 shutdown_connections 2019-10-13T04:04:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/3474422512 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f480aa2b0 0x7f9f4814f910 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 -- 172.21.15.110:0/3474422512 shutdown_connections 2019-10-13T04:04:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 -- 172.21.15.110:0/3474422512 wait complete. 2019-10-13T04:04:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 -- 172.21.15.110:0/3474422512 >> 172.21.15.110:0/3474422512 conn(0x7f9f4805e980 msgr2=0x7f9f480a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 Processor -- start 2019-10-13T04:04:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.440+0000 7f9f4fbea700 1 -- start start 2019-10-13T04:04:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480a0840 con 0x7f9f48072580 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f48121e20 con 0x7f9f48122ee0 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f481249b0 con 0x7f9f480a4160 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4e988700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9f48072580 0x7f9f480749b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44874/0 (socket says 172.21.15.110:44874) 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4e988700 1 -- 172.21.15.110:0/92260606 learned_addr learned my addr 172.21.15.110:0/92260606 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362209239 0 0) 0x7f9f480a0840 con 0x7f9f48072580 2019-10-13T04:04:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f2c0030a0 con 0x7f9f48072580 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101730329 0 0) 0x7f9f48121e20 con 0x7f9f48122ee0 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f480a0840 con 0x7f9f48122ee0 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 -- 172.21.15.110:0/92260606 >> v1:172.21.15.204:6789/0 conn(0x7f9f48122ee0 legacy=0x7f9f48116c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 -- 172.21.15.110:0/92260606 >> v1:172.21.15.110:6789/0 conn(0x7f9f48072580 legacy=0x7f9f480749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 -- 172.21.15.110:0/92260606 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480a4580 con 0x7f9f480a4160 2019-10-13T04:04:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f400101c0 con 0x7f9f480a4160 2019-10-13T04:04:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4000cb20 con 0x7f9f480a4160 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f40012450 con 0x7f9f480a4160 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f48122500 con 0x7f9f480a4160 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.444+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f480761c0 con 0x7f9f480a4160 2019-10-13T04:04:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9f40012760 con 0x7f9f480a4160 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f46ffd700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 0x7f9f2c02a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f9f40034cd0 con 0x7f9f480a4160 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f4e187700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 0x7f9f2c02a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.448+0000 7f9f4e187700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 0x7f9f2c02a530 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.452+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f480555c0 con 0x7f9f480a4160 2019-10-13T04:04:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.460+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.460+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f9f40026080 con 0x7f9f480a4160 2019-10-13T04:04:30.615 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.612+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f38f8d49-36c3-4dfa-8bb5-057288982141", "id": 3} v 0) v1 -- 0x7f9f48122050 con 0x7f9f480a4160 2019-10-13T04:04:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.624+0000 7f9f4d986700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.624+0000 7f9f46ffd700 1 -- 172.21.15.110:0/92260606 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f38f8d49-36c3-4dfa-8bb5-057288982141", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9f40012a50 con 0x7f9f480a4160 2019-10-13T04:04:30.630 INFO:teuthology.orchestra.run.smithi110.stdout:3 2019-10-13T04:04:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 msgr2=0x7f9f2c02a530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 0x7f9f2c02a530 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 msgr2=0x7f9f48072030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f9f40000f00 tx=0x7f9f4000c660).stop 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 shutdown_connections 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9f2c028030 0x7f9f2c02a530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 --2- 172.21.15.110:0/92260606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f480a4160 0x7f9f48072030 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 shutdown_connections 2019-10-13T04:04:30.634 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 wait complete. 2019-10-13T04:04:30.635 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.632+0000 7f9f4fbea700 1 -- 172.21.15.110:0/92260606 >> 172.21.15.110:0/92260606 conn(0x7f9f480a75f0 msgr2=0x7f9f4809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.645 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:30.645 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 1f731a59-e70a-4f9d-8114-c14e183b8b11 4 2019-10-13T04:04:30.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 Processor -- start 2019-10-13T04:04:30.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 -- start start 2019-10-13T04:04:30.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb2cc0a75f0 0x7fb2cc0a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:30.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc0a61a0 0x7fb2cc07e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:30.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc05a200 con 0x7fb2cc0aa2b0 2019-10-13T04:04:30.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0a8820 con 0x7fb2cc0a61a0 2019-10-13T04:04:30.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc059f60 con 0x7fb2cc0a75f0 2019-10-13T04:04:30.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc0a61a0 0x7fb2cc07e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc0a61a0 0x7fb2cc07e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42166/0 (socket says 172.21.15.110:42166) 2019-10-13T04:04:30.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 -- 172.21.15.110:0/3561437496 learned_addr learned my addr 172.21.15.110:0/3561437496 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:30.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/3561437496 >> v2:172.21.15.204:3300/0 conn(0x7fb2cc0a75f0 0x7fb2cc0a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 -- 172.21.15.110:0/3561437496 >> v2:172.21.15.204:3300/0 conn(0x7fb2cc0a75f0 msgr2=0x7fb2cc0a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> v2:172.21.15.204:3300/0 conn(0x7fb2cc0a75f0 0x7fb2cc0a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 -- 172.21.15.110:0/3561437496 >> v1:172.21.15.204:6789/0 conn(0x7fb2cc0aa2b0 legacy=0x7fb2cc147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 -- 172.21.15.110:0/3561437496 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0a1840 con 0x7fb2cc0a61a0 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> v2:172.21.15.110:3300/0 conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> v2:172.21.15.110:3300/0 conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).ready entity=mon.0 client_cookie=24c651551cf2aa97 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cb7fe700 1 -- 172.21.15.110:0/3561437496 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c40111e0 con 0x7fb2cc0a61a0 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cb7fe700 1 -- 172.21.15.110:0/3561437496 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c4008a80 con 0x7fb2cc0a61a0 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2cb7fe700 1 -- 172.21.15.110:0/3561437496 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c40117c0 con 0x7fb2cc0a61a0 2019-10-13T04:04:30.737 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 -- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 msgr2=0x7fb2cc07e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.728+0000 7fb2d2446700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fb2cc05a120 tx=0x7fb2b4002ff0).stop 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- 172.21.15.110:0/3561437496 shutdown_connections 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 --2- 172.21.15.110:0/3561437496 >> v2:172.21.15.204:3300/0 conn(0x7fb2cc0a75f0 0x7fb2cc0a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 --2- 172.21.15.110:0/3561437496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc0a61a0 0x7fb2cc07e490 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- 172.21.15.110:0/3561437496 shutdown_connections 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- 172.21.15.110:0/3561437496 wait complete. 2019-10-13T04:04:30.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- 172.21.15.110:0/3561437496 >> 172.21.15.110:0/3561437496 conn(0x7fb2cc05e980 msgr2=0x7fb2cc0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 Processor -- start 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- start start 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc09e040 0x7fb2cc13f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2cc07b0e0 0x7fb2cc07b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc140c70 con 0x7fb2cc09e460 2019-10-13T04:04:30.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc140ea0 con 0x7fb2cc09e040 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d2446700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc141110 con 0x7fb2cc07b0e0 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d09e3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc09e040 0x7fb2cc13f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2cbfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb2cc09e460 0x7fb2cc13fc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44884/0 (socket says 172.21.15.110:44884) 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d09e3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb2cc09e040 0x7fb2cc13f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42174/0 (socket says 172.21.15.110:42174) 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2cbfff700 1 -- 172.21.15.110:0/806887774 learned_addr learned my addr 172.21.15.110:0/806887774 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.732+0000 7fb2d11e4700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2cc07b0e0 0x7fb2cc07b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 -- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2cc07b0e0 msgr2=0x7fb2cc07b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2cc07b0e0 0x7fb2cc07b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 -- 172.21.15.110:0/806887774 >> v1:172.21.15.110:6789/0 conn(0x7fb2cc09e460 legacy=0x7fb2cc13fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194966625 0 0) 0x7fb2cc140c70 con 0x7fb2cc09e460 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 -- 172.21.15.110:0/806887774 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc148010 con 0x7fb2cc09e040 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> v2:172.21.15.110:3300/0 conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> v2:172.21.15.110:3300/0 conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b40091a0 con 0x7fb2cc09e040 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b4009b90 con 0x7fb2cc09e040 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc07ba50 con 0x7fb2cc09e040 2019-10-13T04:04:30.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b4009780 con 0x7fb2cc09e040 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc07bf80 con 0x7fb2cc09e040 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb2b4007170 con 0x7fb2cc09e040 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 0x7fb2a4029c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7fb2b4034520 con 0x7fb2cc09e040 2019-10-13T04:04:30.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 0x7fb2a4029c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:30.747 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.736+0000 7fb2cbfff700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 0x7fb2a4029c80 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:30.747 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.744+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc0555c0 con 0x7fb2cc09e040 2019-10-13T04:04:30.751 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.748+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.752 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.748+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb2b40058f0 con 0x7fb2cc09e040 2019-10-13T04:04:30.899 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.896+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1f731a59-e70a-4f9d-8114-c14e183b8b11", "id": 4} v 0) v1 -- 0x7fb2cc066560 con 0x7fb2cc09e040 2019-10-13T04:04:30.911 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.904+0000 7fb2d09e3700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:30.911 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.904+0000 7fb2c97fa700 1 -- 172.21.15.110:0/806887774 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f731a59-e70a-4f9d-8114-c14e183b8b11", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2b4005b10 con 0x7fb2cc09e040 2019-10-13T04:04:30.914 INFO:teuthology.orchestra.run.smithi110.stdout:4 2019-10-13T04:04:30.917 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 msgr2=0x7fb2a4029c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.917 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 0x7fb2a4029c80 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.917 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 msgr2=0x7fb2cc13f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fb2cc07e1f0 tx=0x7fb2bc007960).stop 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 shutdown_connections 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2cc07b0e0 0x7fb2cc07b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb2a4027780 0x7fb2a4029c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 --2- 172.21.15.110:0/806887774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2cc09e040 0x7fb2cc13f6c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:30.918 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 shutdown_connections 2019-10-13T04:04:30.919 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 wait complete. 2019-10-13T04:04:30.919 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:30.916+0000 7fb2d2446700 1 -- 172.21.15.110:0/806887774 >> 172.21.15.110:0/806887774 conn(0x7fb2cc0a8020 msgr2=0x7fb2cc09ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:30.929 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:30.929 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 566b8f0d-601e-451d-901c-3b5aa1f50b49 5 2019-10-13T04:04:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.012+0000 7f2e55653700 1 Processor -- start 2019-10-13T04:04:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.012+0000 7f2e55653700 1 -- start start 2019-10-13T04:04:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e500a75f0 0x7f2e500a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5005a200 con 0x7f2e500a61a0 2019-10-13T04:04:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5007ec90 con 0x7f2e500aa2b0 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e500a8820 con 0x7f2e500a75f0 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e500a75f0 0x7f2e500a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4effd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2e500a61a0 0x7f2e5007e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44892/0 (socket says 172.21.15.110:44892) 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4effd700 1 -- 172.21.15.110:0/646868499 learned_addr learned my addr 172.21.15.110:0/646868499 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4e7fc700 1 -- 172.21.15.110:0/646868499 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22237904 0 0) 0x7f2e5005a200 con 0x7f2e500a61a0 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4e7fc700 1 -- 172.21.15.110:0/646868499 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e380030a0 con 0x7f2e500a61a0 2019-10-13T04:04:31.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 -- 172.21.15.110:0/646868499 >> v1:172.21.15.204:6789/0 conn(0x7f2e500aa2b0 legacy=0x7f2e5014e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 -- 172.21.15.110:0/646868499 >> v1:172.21.15.110:6789/0 conn(0x7f2e500a61a0 legacy=0x7f2e5007e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 -- 172.21.15.110:0/646868499 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e500a1840 con 0x7f2e500a75f0 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> v2:172.21.15.110:3300/0 conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> v2:172.21.15.110:3300/0 conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).ready entity=mon.0 client_cookie=b886a36bbc8e4bb0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4e7fc700 1 -- 172.21.15.110:0/646868499 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e400119c0 con 0x7f2e500a75f0 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4e7fc700 1 -- 172.21.15.110:0/646868499 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e400115e0 con 0x7f2e500a75f0 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e4e7fc700 1 -- 172.21.15.110:0/646868499 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4000a160 con 0x7f2e500a75f0 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 msgr2=0x7f2e500a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f2e40008b00 tx=0x7f2e4000ff60).stop 2019-10-13T04:04:31.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- 172.21.15.110:0/646868499 shutdown_connections 2019-10-13T04:04:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 --2- 172.21.15.110:0/646868499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a75f0 0x7f2e500a12f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- 172.21.15.110:0/646868499 shutdown_connections 2019-10-13T04:04:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- 172.21.15.110:0/646868499 wait complete. 2019-10-13T04:04:31.025 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.016+0000 7f2e55653700 1 -- 172.21.15.110:0/646868499 >> 172.21.15.110:0/646868499 conn(0x7f2e5005e980 msgr2=0x7f2e500a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 Processor -- start 2019-10-13T04:04:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 -- start start 2019-10-13T04:04:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e500a8130 0x7f2e5011f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e50143870 con 0x7f2e5009f9a0 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e50144de0 con 0x7f2e5011f8c0 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e55653700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e50143090 con 0x7f2e500a8130 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e500a8130 0x7f2e5011f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e500a8130 0x7f2e5011f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42188/0 (socket says 172.21.15.110:42188) 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 -- 172.21.15.110:0/200793773 learned_addr learned my addr 172.21.15.110:0/200793773 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:31.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 -- 172.21.15.110:0/200793773 >> v1:172.21.15.204:6789/0 conn(0x7f2e5011f8c0 legacy=0x7f2e50130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 -- 172.21.15.110:0/200793773 >> v1:172.21.15.110:6789/0 conn(0x7f2e5009f9a0 legacy=0x7f2e5009e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003323373 0 0) 0x7f2e50144de0 con 0x7f2e5011f8c0 2019-10-13T04:04:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 -- 172.21.15.110:0/200793773 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5009f120 con 0x7f2e500a8130 2019-10-13T04:04:31.030 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.020+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> v2:172.21.15.110:3300/0 conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> v2:172.21.15.110:3300/0 conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4400caa0 con 0x7f2e500a8130 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e44010250 con 0x7f2e500a8130 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e500a02e0 con 0x7f2e500a8130 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4400d2b0 con 0x7f2e500a8130 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5011fce0 con 0x7f2e500a8130 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f2e4400d4b0 con 0x7f2e500a8130 2019-10-13T04:04:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 0x7f2e30029ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f2e440345f0 con 0x7f2e500a8130 2019-10-13T04:04:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 0x7f2e30029ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.024+0000 7f2e4f7fe700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 0x7f2e30029ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.028+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e500555c0 con 0x7f2e500a8130 2019-10-13T04:04:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.036+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.036+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f2e44025980 con 0x7f2e500a8130 2019-10-13T04:04:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.196+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "566b8f0d-601e-451d-901c-3b5aa1f50b49", "id": 5} v 0) v1 -- 0x7f2e50066560 con 0x7f2e500a8130 2019-10-13T04:04:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.200+0000 7f2e4effd700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.200+0000 7f2e37fff700 1 -- 172.21.15.110:0/200793773 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "566b8f0d-601e-451d-901c-3b5aa1f50b49", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e44029020 con 0x7f2e500a8130 2019-10-13T04:04:31.203 INFO:teuthology.orchestra.run.smithi110.stdout:5 2019-10-13T04:04:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 msgr2=0x7f2e30029ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 0x7f2e30029ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 msgr2=0x7f2e5011f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f2e440013e0 tx=0x7f2e4400c3f0).stop 2019-10-13T04:04:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 shutdown_connections 2019-10-13T04:04:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2e300277e0 0x7f2e30029ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 --2- 172.21.15.110:0/200793773 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e500a8130 0x7f2e5011f370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 shutdown_connections 2019-10-13T04:04:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 wait complete. 2019-10-13T04:04:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.204+0000 7f2e55653700 1 -- 172.21.15.110:0/200793773 >> 172.21.15.110:0/200793773 conn(0x7f2e500a75f0 msgr2=0x7f2e500a9440 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.222 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.222 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 8cc0914d-13a9-4b3b-9df1-ba5519ef1a42 6 2019-10-13T04:04:31.327 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 Processor -- start 2019-10-13T04:04:31.327 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 -- start start 2019-10-13T04:04:31.327 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57240aa2b0 0x7f572414f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.327 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572405a200 con 0x7f57240a61a0 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572407ec90 con 0x7f57240a75f0 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f572b254700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57240a8820 con 0x7f57240aa2b0 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f5729ff2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57240aa2b0 0x7f572414f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f5729ff2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57240aa2b0 0x7f572414f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42200/0 (socket says 172.21.15.110:42200) 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f5729ff2700 1 -- 172.21.15.110:0/8237966 learned_addr learned my addr 172.21.15.110:0/8237966 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f5729ff2700 1 -- 172.21.15.110:0/8237966 >> v1:172.21.15.204:6789/0 conn(0x7f57240a75f0 legacy=0x7f57240a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.320+0000 7f5729ff2700 1 -- 172.21.15.110:0/8237966 >> v1:172.21.15.110:6789/0 conn(0x7f57240a61a0 legacy=0x7f572407e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f571bfff700 1 -- 172.21.15.110:0/8237966 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917830334 0 0) 0x7f572405a200 con 0x7f57240a61a0 2019-10-13T04:04:31.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 -- 172.21.15.110:0/8237966 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5724059a20 con 0x7f57240aa2b0 2019-10-13T04:04:31.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> v2:172.21.15.110:3300/0 conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f571c008b00 tx=0x7f571c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> v2:172.21.15.110:3300/0 conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f571c008b00 tx=0x7f571c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f571c008b00 tx=0x7f571c010020).ready entity=mon.0 client_cookie=aa5810b54e042f3d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f571c008b00 tx=0x7f571c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f571bfff700 1 -- 172.21.15.110:0/8237966 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c00a160 con 0x7f57240aa2b0 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f571c008b00 tx=0x7f571c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f5729ff2700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f571c008b00 tx=0x7f571c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f571bfff700 1 -- 172.21.15.110:0/8237966 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c011a90 con 0x7f57240aa2b0 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f571bfff700 1 -- 172.21.15.110:0/8237966 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c00a740 con 0x7f57240aa2b0 2019-10-13T04:04:31.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 -- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 msgr2=0x7f572414f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f571c008b00 tx=0x7f571c010020).stop 2019-10-13T04:04:31.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 -- 172.21.15.110:0/8237966 shutdown_connections 2019-10-13T04:04:31.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 --2- 172.21.15.110:0/8237966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57240aa2b0 0x7f572414f910 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 -- 172.21.15.110:0/8237966 shutdown_connections 2019-10-13T04:04:31.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 -- 172.21.15.110:0/8237966 wait complete. 2019-10-13T04:04:31.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.324+0000 7f572b254700 1 -- 172.21.15.110:0/8237966 >> 172.21.15.110:0/8237966 conn(0x7f572405e980 msgr2=0x7f57240a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 Processor -- start 2019-10-13T04:04:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- start start 2019-10-13T04:04:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f572413f2d0 0x7f5724112b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57241438a0 con 0x7f57241130c0 2019-10-13T04:04:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57241134e0 con 0x7f572413eeb0 2019-10-13T04:04:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572411ee00 con 0x7f572413f2d0 2019-10-13T04:04:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5729ff2700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f57241130c0 0x7f5724129180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44918/0 (socket says 172.21.15.110:44918) 2019-10-13T04:04:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5729ff2700 1 -- 172.21.15.110:0/329024206 learned_addr learned my addr 172.21.15.110:0/329024206 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:31.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> v2:172.21.15.204:3300/0 conn(0x7f572413f2d0 0x7f5724112b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255277757 0 0) 0x7f57241438a0 con 0x7f57241130c0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57040030a0 con 0x7f57241130c0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1772587305 0 0) 0x7f57040030a0 con 0x7f57241130c0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57241438a0 con 0x7f57241130c0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f571c00a160 con 0x7f57241130c0 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 -- 172.21.15.110:0/329024206 >> v1:172.21.15.204:6789/0 conn(0x7f572413eeb0 legacy=0x7f57241430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 -- 172.21.15.110:0/329024206 >> v1:172.21.15.110:6789/0 conn(0x7f57241130c0 legacy=0x7f5724129180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 -- 172.21.15.110:0/329024206 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572411f5e0 con 0x7f572413f2d0 2019-10-13T04:04:31.339 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> v2:172.21.15.204:3300/0 conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> v2:172.21.15.204:3300/0 conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5714006910 con 0x7f572413f2d0 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57240a07b0 con 0x7f572413f2d0 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.340 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714006530 con 0x7f572413f2d0 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57240a4160 con 0x7f572413f2d0 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.328+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5714010c40 con 0x7f572413f2d0 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5714010ee0 con 0x7f572413f2d0 2019-10-13T04:04:31.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f5719ffb700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 0x7f570402a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f5714034980 con 0x7f572413f2d0 2019-10-13T04:04:31.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f57297f1700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 0x7f570402a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.332+0000 7f57297f1700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 0x7f570402a680 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.336+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57240555c0 con 0x7f572413f2d0 2019-10-13T04:04:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.344+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.344+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f5714025d30 con 0x7f572413f2d0 2019-10-13T04:04:31.502 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.500+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8cc0914d-13a9-4b3b-9df1-ba5519ef1a42", "id": 6} v 0) v1 -- 0x7f5724066560 con 0x7f572413f2d0 2019-10-13T04:04:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.512+0000 7f5728ff0700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.512+0000 7f5719ffb700 1 -- 172.21.15.110:0/329024206 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cc0914d-13a9-4b3b-9df1-ba5519ef1a42", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f57140111d0 con 0x7f572413f2d0 2019-10-13T04:04:31.517 INFO:teuthology.orchestra.run.smithi110.stdout:6 2019-10-13T04:04:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 msgr2=0x7f570402a680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 0x7f570402a680 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 msgr2=0x7f5724112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f57200010d0 tx=0x7f571c0104b0).stop 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 shutdown_connections 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5704028180 0x7f570402a680 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 --2- 172.21.15.110:0/329024206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f572413f2d0 0x7f5724112b70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 shutdown_connections 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 wait complete. 2019-10-13T04:04:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.520+0000 7f572b254700 1 -- 172.21.15.110:0/329024206 >> 172.21.15.110:0/329024206 conn(0x7f57240a75f0 msgr2=0x7f57240a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.534 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.534 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa 7 2019-10-13T04:04:31.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 Processor -- start 2019-10-13T04:04:31.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 -- start start 2019-10-13T04:04:31.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f13040a75f0 0x7f13040a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13040aa2b0 0x7f130414e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130405a200 con 0x7f13040a61a0 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040a8820 con 0x7f13040a75f0 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1308beb700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1304059f60 con 0x7f13040aa2b0 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f130259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f13040a61a0 0x7f130407e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44924/0 (socket says 172.21.15.110:44924) 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f130259c700 1 -- 172.21.15.110:0/2973710097 learned_addr learned my addr 172.21.15.110:0/2973710097 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:31.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1302d9d700 1 --2- 172.21.15.110:0/2973710097 >> v2:172.21.15.204:3300/0 conn(0x7f13040a75f0 0x7f13040a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f130359e700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13040aa2b0 0x7f130414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666099743 0 0) 0x7f130405a200 con 0x7f13040a61a0 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec0030a0 con 0x7f13040a61a0 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3146124277 0 0) 0x7f12ec0030a0 con 0x7f13040a61a0 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13040aa2b0 msgr2=0x7f130414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:31.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13040aa2b0 0x7f130414e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:31.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 >> v2:172.21.15.204:3300/0 conn(0x7f13040a75f0 msgr2=0x7f13040a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:31.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 --2- 172.21.15.110:0/2973710097 >> v2:172.21.15.204:3300/0 conn(0x7f13040a75f0 0x7f13040a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:31.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130407ec90 con 0x7f13040a61a0 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f12f8001750 con 0x7f13040a61a0 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 >> v1:172.21.15.110:6789/0 conn(0x7f13040a61a0 legacy=0x7f130407e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12ec0036d0 0x7f12ec003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12ec004d80 0x7f12ec04f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040a8820 con 0x7f12ec004410 2019-10-13T04:04:31.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304059f60 con 0x7f12ec0036d0 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec0051a0 con 0x7f12ec004d80 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f1302d9d700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12ec0036d0 0x7f12ec003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.632+0000 7f130359e700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12ec004d80 0x7f12ec04f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 -- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12ec004d80 msgr2=0x7f12ec04f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:31.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12ec004d80 0x7f12ec04f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 -- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12ec0036d0 msgr2=0x7f12ec003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12ec0036d0 0x7f12ec003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 -- 172.21.15.110:0/2973710097 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec04fc80 con 0x7f12ec004410 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130359e700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12ec004d80 0x7f12ec04f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1302d9d700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12ec0036d0 0x7f12ec003af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f12f8001950 tx=0x7f12f4007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f12f8001950 tx=0x7f12f4007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f12f8001950 tx=0x7f12f4007960).ready entity=mon.0 client_cookie=fb1096168f9cb9ab server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f12f8001950 tx=0x7f12f4007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f800ed60 con 0x7f12ec004410 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f12f8001950 tx=0x7f12f4007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f800e980 con 0x7f12ec004410 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130259c700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f12f8001950 tx=0x7f12f4007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1301d9b700 1 -- 172.21.15.110:0/2973710097 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f8010530 con 0x7f12ec004410 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 -- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 msgr2=0x7f12ec004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 --2- 172.21.15.110:0/2973710097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12ec004410 0x7f12ec004830 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f12f8001950 tx=0x7f12f4007960).stop 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f130359e700 1 -- 172.21.15.110:0/2973710097 reap_dead start 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 -- 172.21.15.110:0/2973710097 shutdown_connections 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 -- 172.21.15.110:0/2973710097 shutdown_connections 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 -- 172.21.15.110:0/2973710097 wait complete. 2019-10-13T04:04:31.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.636+0000 7f1308beb700 1 -- 172.21.15.110:0/2973710097 >> 172.21.15.110:0/2973710097 conn(0x7f130405e980 msgr2=0x7f13040a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.651 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 Processor -- start 2019-10-13T04:04:31.651 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- start start 2019-10-13T04:04:31.651 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13041200a0 0x7f130409fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.651 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 0x7f130407dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1304128970 0x7f1304128d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130405fa90 con 0x7f1304128970 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040a0730 con 0x7f130407d830 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130405fcf0 con 0x7f13041200a0 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1304128970 0x7f1304128d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1304128970 0x7f1304128d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42224/0 (socket says 172.21.15.110:42224) 2019-10-13T04:04:31.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 -- 172.21.15.110:0/3632452593 learned_addr learned my addr 172.21.15.110:0/3632452593 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130259c700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 0x7f130407dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1302d9d700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13041200a0 0x7f130409fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 -- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13041200a0 msgr2=0x7f130409fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13041200a0 0x7f130409fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 -- 172.21.15.110:0/3632452593 >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 msgr2=0x7f130407dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 0x7f130407dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 -- 172.21.15.110:0/3632452593 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13041480a0 con 0x7f1304128970 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130259c700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 0x7f130407dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.110:3300/0 conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.110:3300/0 conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12fc008970 con 0x7f1304128970 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc008590 con 0x7f1304128970 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304060230 con 0x7f1304128970 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12fc0123c0 con 0x7f1304128970 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1304120bf0 con 0x7f1304128970 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f12fc00b220 con 0x7f1304128970 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.640+0000 7f12f37fe700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 0x7f12e8029e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.644+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f12fc034670 con 0x7f1304128970 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.644+0000 7f1302d9d700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 0x7f12e8029e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.644+0000 7f1302d9d700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 0x7f12e8029e40 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:31.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.648+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13040555c0 con 0x7f1304128970 2019-10-13T04:04:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.656+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.656+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f12fc00c5d0 con 0x7f1304128970 2019-10-13T04:04:31.810 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.808+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa", "id": 7} v 0) v1 -- 0x7f1304066560 con 0x7f1304128970 2019-10-13T04:04:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.816+0000 7f130359e700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:31.819 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.816+0000 7f12f37fe700 1 -- 172.21.15.110:0/3632452593 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f12fc010220 con 0x7f1304128970 2019-10-13T04:04:31.825 INFO:teuthology.orchestra.run.smithi110.stdout:7 2019-10-13T04:04:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 msgr2=0x7f12e8029e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 0x7f12e8029e40 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 msgr2=0x7f1304128d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f130405feb0 tx=0x7f12f800b8e0).stop 2019-10-13T04:04:31.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 shutdown_connections 2019-10-13T04:04:31.829 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f12e8027940 0x7f12e8029e40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.829 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1304128970 0x7f1304128d90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.829 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13041200a0 0x7f130409fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 --2- 172.21.15.110:0/3632452593 >> v2:172.21.15.204:3300/0 conn(0x7f130407d830 0x7f130407dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:31.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.824+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 shutdown_connections 2019-10-13T04:04:31.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.828+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 wait complete. 2019-10-13T04:04:31.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:31.828+0000 7f1308beb700 1 -- 172.21.15.110:0/3632452593 >> 172.21.15.110:0/3632452593 conn(0x7f13040a5450 msgr2=0x7f130411b540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:31.840 INFO:tasks.ceph.osd.4:Restarting daemon 2019-10-13T04:04:31.840 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:31.840 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-10-13T04:04:31.844 INFO:tasks.ceph.osd.4:Started 2019-10-13T04:04:31.844 INFO:tasks.ceph.osd.5:Restarting daemon 2019-10-13T04:04:31.844 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:31.844 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-10-13T04:04:31.848 INFO:tasks.ceph.osd.5:Started 2019-10-13T04:04:31.848 INFO:tasks.ceph.osd.6:Restarting daemon 2019-10-13T04:04:31.848 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:31.848 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-10-13T04:04:31.852 INFO:tasks.ceph.osd.6:Started 2019-10-13T04:04:31.852 INFO:tasks.ceph.osd.7:Restarting daemon 2019-10-13T04:04:31.852 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:31.852 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-10-13T04:04:31.855 INFO:tasks.ceph.osd.7:Started 2019-10-13T04:04:31.855 INFO:tasks.ceph.osd.0:Restarting daemon 2019-10-13T04:04:31.856 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.856 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-10-13T04:04:31.859 INFO:tasks.ceph.osd.0:Started 2019-10-13T04:04:31.859 INFO:tasks.ceph.osd.1:Restarting daemon 2019-10-13T04:04:31.859 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.859 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-10-13T04:04:31.863 INFO:tasks.ceph.osd.1:Started 2019-10-13T04:04:31.863 INFO:tasks.ceph.osd.2:Restarting daemon 2019-10-13T04:04:31.863 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.863 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-10-13T04:04:31.867 INFO:tasks.ceph.osd.2:Started 2019-10-13T04:04:31.867 INFO:tasks.ceph.osd.3:Restarting daemon 2019-10-13T04:04:31.867 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:31.867 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-10-13T04:04:31.871 INFO:tasks.ceph.osd.3:Started 2019-10-13T04:04:31.871 INFO:tasks.ceph:Waiting for OSDs to come up 2019-10-13T04:04:32.216 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:04:32.216+0000 7f8b2ba85d80 -1 Falling back to public interface 2019-10-13T04:04:32.441 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:04:32.440+0000 7f9e89e6ed80 -1 Falling back to public interface 2019-10-13T04:04:32.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T04:04:32.448+0000 7f140f2f8d80 -1 Falling back to public interface 2019-10-13T04:04:32.456 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:04:32.452+0000 7fac2a1e8d80 -1 Falling back to public interface 2019-10-13T04:04:32.464 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:04:32.464+0000 7f6c96449d80 -1 Falling back to public interface 2019-10-13T04:04:32.465 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:04:32.460+0000 7fa1615ddd80 -1 Falling back to public interface 2019-10-13T04:04:32.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:04:32.476+0000 7f8a049acd80 -1 Falling back to public interface 2019-10-13T04:04:32.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:04:32.488+0000 7f29332b1d80 -1 Falling back to public interface 2019-10-13T04:04:32.685 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:32.685 INFO:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:32.779 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 Processor -- start 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 -- start start 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f18a00a6410 0x7f18a00a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18a0065060 0x7f18a0144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a0145010 0x7f18a00a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a00a20e0 con 0x7f18a0065060 2019-10-13T04:04:32.780 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a00a2380 con 0x7f18a00a6410 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.772+0000 7f18a6c5e700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00a77f0 con 0x7f18a0145010 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18a0065060 0x7f18a0144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a51fb700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f18a00a6410 0x7f18a00a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18a0065060 0x7f18a0144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42256/0 (socket says 172.21.15.110:42256) 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 -- 172.21.15.110:0/375313708 learned_addr learned my addr 172.21.15.110:0/375313708 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:32.781 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a59fc700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a0145010 0x7f18a00a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 -- 172.21.15.110:0/375313708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a0145010 msgr2=0x7f18a00a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a0145010 0x7f18a00a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 -- 172.21.15.110:0/375313708 >> v2:172.21.15.204:3300/0 conn(0x7f18a00a6410 msgr2=0x7f18a00a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> v2:172.21.15.204:3300/0 conn(0x7f18a00a6410 0x7f18a00a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 -- 172.21.15.110:0/375313708 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00a9160 con 0x7f18a0065060 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> v2:172.21.15.110:3300/0 conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f18a00a22a0 tx=0x7f189c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.782 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> v2:172.21.15.110:3300/0 conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f18a00a22a0 tx=0x7f189c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f18a00a22a0 tx=0x7f189c007710).ready entity=mon.0 client_cookie=2f90e030e29a9541 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f18a00a22a0 tx=0x7f189c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f18a00a22a0 tx=0x7f189c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f1897fff700 1 -- 172.21.15.110:0/375313708 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18900088a0 con 0x7f18a0065060 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a49fa700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f18a00a22a0 tx=0x7f189c007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f1897fff700 1 -- 172.21.15.110:0/375313708 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18900073c0 con 0x7f18a0065060 2019-10-13T04:04:32.783 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f1897fff700 1 -- 172.21.15.110:0/375313708 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1890006560 con 0x7f18a0065060 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 -- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 msgr2=0x7f18a0144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f18a00a22a0 tx=0x7f189c007710).stop 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 -- 172.21.15.110:0/375313708 shutdown_connections 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a0145010 0x7f18a00a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/375313708 >> v2:172.21.15.204:3300/0 conn(0x7f18a00a6410 0x7f18a00a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/375313708 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a0065060 0x7f18a0144ac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:32.784 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 -- 172.21.15.110:0/375313708 shutdown_connections 2019-10-13T04:04:32.785 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 -- 172.21.15.110:0/375313708 wait complete. 2019-10-13T04:04:32.785 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.776+0000 7f18a6c5e700 1 -- 172.21.15.110:0/375313708 >> 172.21.15.110:0/375313708 conn(0x7f18a005afb0 msgr2=0x7f18a009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:32.788 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 Processor -- start 2019-10-13T04:04:32.789 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- start start 2019-10-13T04:04:32.789 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18a01234c0 0x7f18a00a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:32.789 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a00a0780 con 0x7f18a0121300 2019-10-13T04:04:32.789 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a011f8b0 con 0x7f18a011ec90 2019-10-13T04:04:32.789 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a01227c0 con 0x7f18a01234c0 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18a01234c0 0x7f18a00a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a51fb700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f18a0121300 0x7f18a0122cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:44972/0 (socket says 172.21.15.110:44972) 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a51fb700 1 -- 172.21.15.110:0/3605792784 learned_addr learned my addr 172.21.15.110:0/3605792784 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 -- 172.21.15.110:0/3605792784 >> v1:172.21.15.204:6789/0 conn(0x7f18a011ec90 legacy=0x7f18a011f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 -- 172.21.15.110:0/3605792784 >> v1:172.21.15.110:6789/0 conn(0x7f18a0121300 legacy=0x7f18a0122cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645043073 0 0) 0x7f18a011f8b0 con 0x7f18a011ec90 2019-10-13T04:04:32.790 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 -- 172.21.15.110:0/3605792784 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a011fae0 con 0x7f18a01234c0 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> v2:172.21.15.110:3300/0 conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> v2:172.21.15.110:3300/0 conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f189000f5f0 con 0x7f18a01234c0 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.791 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1890008aa0 con 0x7f18a01234c0 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a013c6c0 con 0x7f18a01234c0 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18900114b0 con 0x7f18a01234c0 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a013c8d0 con 0x7f18a01234c0 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.780+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1890011790 con 0x7f18a01234c0 2019-10-13T04:04:32.792 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.793 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f1895ffb700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 0x7f1880029ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:32.793 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f1890078790 con 0x7f18a01234c0 2019-10-13T04:04:32.793 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f18a51fb700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 0x7f1880029ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:32.793 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.784+0000 7f18a51fb700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 0x7f1880029ce0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:32.793 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.788+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0042510 con 0x7f18a01234c0 2019-10-13T04:04:32.795 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.792+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.795 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.792+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f1890060e30 con 0x7f18a01234c0 2019-10-13T04:04:32.983 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.980+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18a00555c0 con 0x7f18a01234c0 2019-10-13T04:04:32.984 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.980+0000 7f18a49fa700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:32.984 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.980+0000 7f1895ffb700 1 -- 172.21.15.110:0/3605792784 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7f1890061090 con 0x7f18a01234c0 2019-10-13T04:04:32.984 INFO:teuthology.misc.health.smithi110.stdout: 2019-10-13T04:04:32.984 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:31.812105+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:32.986 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 msgr2=0x7f1880029ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:32.986 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 0x7f1880029ce0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:32.987 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 msgr2=0x7f18a00a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:32.987 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f189c007930 tx=0x7f189c007a40).stop 2019-10-13T04:04:32.987 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 shutdown_connections 2019-10-13T04:04:32.987 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f18800277e0 0x7f1880029ce0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:32.987 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 --2- 172.21.15.110:0/3605792784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18a01234c0 0x7f18a00a4ca0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:32.988 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 shutdown_connections 2019-10-13T04:04:32.988 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 wait complete. 2019-10-13T04:04:32.988 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:32.984+0000 7f18a6c5e700 1 -- 172.21.15.110:0/3605792784 >> 172.21.15.110:0/3605792784 conn(0x7f18a009c500 msgr2=0x7f18a011d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:32.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-10-13T04:04:33.372 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:04:33.368+0000 7f8b2ba85d80 -1 osd.7 0 log_to_monitors {default=true} 2019-10-13T04:04:33.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T04:04:33.580+0000 7f140f2f8d80 -1 osd.2 0 log_to_monitors {default=true} 2019-10-13T04:04:33.586 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:04:33.580+0000 7f9e89e6ed80 -1 osd.5 0 log_to_monitors {default=true} 2019-10-13T04:04:33.633 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:04:33.628+0000 7fac2a1e8d80 -1 osd.6 0 log_to_monitors {default=true} 2019-10-13T04:04:33.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:04:33.644+0000 7f6c96449d80 -1 osd.1 0 log_to_monitors {default=true} 2019-10-13T04:04:33.648 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:04:33.644+0000 7fa1615ddd80 -1 osd.4 0 log_to_monitors {default=true} 2019-10-13T04:04:33.652 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:04:33.652+0000 7f8a049acd80 -1 osd.0 0 log_to_monitors {default=true} 2019-10-13T04:04:33.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:04:33.672+0000 7f29332b1d80 -1 osd.3 0 log_to_monitors {default=true} 2019-10-13T04:04:35.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T04:04:35.312+0000 7f13f852d700 -1 osd.2 0 waiting for initial osdmap 2019-10-13T04:04:35.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:04:35.312+0000 7f6c7f67e700 -1 osd.1 0 waiting for initial osdmap 2019-10-13T04:04:35.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:04:35.311+0000 7f8b14cba700 -1 osd.7 0 waiting for initial osdmap 2019-10-13T04:04:35.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:04:35.312+0000 7f291c4e6700 -1 osd.3 0 waiting for initial osdmap 2019-10-13T04:04:35.323 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:04:35.319+0000 7f9e730a3700 -1 osd.5 0 waiting for initial osdmap 2019-10-13T04:04:35.323 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:04:35.319+0000 7fac1341d700 -1 osd.6 0 waiting for initial osdmap 2019-10-13T04:04:35.323 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:04:35.319+0000 7fa14a812700 -1 osd.4 0 waiting for initial osdmap 2019-10-13T04:04:35.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:04:35.320+0000 7f89edbe1700 -1 osd.0 0 waiting for initial osdmap 2019-10-13T04:04:39.802 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:39.803 INFO:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:39.896 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 Processor -- start 2019-10-13T04:04:39.896 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 -- start start 2019-10-13T04:04:39.896 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4e700a1970 0x7f4e7014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700a9410 con 0x7f4e700a6410 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700a1d90 con 0x7f4e70065060 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e74e6d700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700a20e0 con 0x7f4e700a1970 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6ed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4e700a6410 0x7f4e70144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45198/0 (socket says 172.21.15.110:45198) 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6ed9d700 1 -- 172.21.15.110:0/359062536 learned_addr learned my addr 172.21.15.110:0/359062536 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:39.897 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> v2:172.21.15.204:3300/0 conn(0x7f4e700a1970 0x7f4e7014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522857182 0 0) 0x7f4e700a9410 con 0x7f4e700a6410 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c0030a0 con 0x7f4e700a6410 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 5119354 0 0) 0x7f4e5c0030a0 con 0x7f4e700a6410 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 >> v2:172.21.15.204:3300/0 conn(0x7f4e700a1970 msgr2=0x7f4e7014f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 --2- 172.21.15.110:0/359062536 >> v2:172.21.15.204:3300/0 conn(0x7f4e700a1970 0x7f4e7014f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:39.898 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 >> v1:172.21.15.204:6789/0 conn(0x7f4e70065060 legacy=0x7f4e700a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700a77f0 con 0x7f4e700a6410 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f4e60001750 con 0x7f4e700a6410 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 >> v1:172.21.15.110:6789/0 conn(0x7f4e700a6410 legacy=0x7f4e70144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e5c0036d0 0x7f4e5c003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e5c004410 0x7f4e5c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:39.899 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700a20e0 con 0x7f4e5c004d80 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c04fbf0 con 0x7f4e5c004410 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.888+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c04fe30 con 0x7f4e5c0036d0 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6ed9d700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e5c0036d0 0x7f4e5c003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6659c700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e5c004410 0x7f4e5c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.900 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 -- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e5c0036d0 msgr2=0x7f4e5c003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e5c0036d0 0x7f4e5c003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 -- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e5c004410 msgr2=0x7f4e5c004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e5c004410 0x7f4e5c004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 -- 172.21.15.110:0/359062536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c050130 con 0x7f4e5c004d80 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e680032b0 tx=0x7f4e58001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.901 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 -- 172.21.15.110:0/359062536 reap_dead start 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e680032b0 tx=0x7f4e58001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f4e680032b0 tx=0x7f4e58001310).ready entity=mon.0 client_cookie=3ab7c84d04a00cc9 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f4e680032b0 tx=0x7f4e58001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e68005aa0 con 0x7f4e5c004d80 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f4e680032b0 tx=0x7f4e58001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68008d70 con 0x7f4e5c004d80 2019-10-13T04:04:39.902 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f4e680032b0 tx=0x7f4e58001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e6e59c700 1 -- 172.21.15.110:0/359062536 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e680092b0 con 0x7f4e5c004d80 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 -- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 msgr2=0x7f4e5c00b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f4e680032b0 tx=0x7f4e58001310).stop 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 -- 172.21.15.110:0/359062536 shutdown_connections 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/359062536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e5c004d80 0x7f4e5c00b660 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:39.903 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 -- 172.21.15.110:0/359062536 shutdown_connections 2019-10-13T04:04:39.904 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 -- 172.21.15.110:0/359062536 wait complete. 2019-10-13T04:04:39.904 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.892+0000 7f4e74e6d700 1 -- 172.21.15.110:0/359062536 >> 172.21.15.110:0/359062536 conn(0x7f4e7005afb0 msgr2=0x7f4e7009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:39.907 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 Processor -- start 2019-10-13T04:04:39.907 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 -- start start 2019-10-13T04:04:39.907 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4e70123080 0x7f4e7011eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e7011f030 0x7f4e7013b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700a3470 con 0x7f4e70121440 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700a8700 con 0x7f4e7011f030 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e74e6d700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e70124970 con 0x7f4e70123080 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e6f59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e7011f030 0x7f4e7013b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.908 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e6f59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e7011f030 0x7f4e7013b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42500/0 (socket says 172.21.15.110:42500) 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e6f59e700 1 -- 172.21.15.110:0/3583750314 learned_addr learned my addr 172.21.15.110:0/3583750314 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.896+0000 7f4e6659c700 1 --2- 172.21.15.110:0/3583750314 >> v2:172.21.15.204:3300/0 conn(0x7f4e70123080 0x7f4e7011eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 -- 172.21.15.110:0/3583750314 >> v2:172.21.15.204:3300/0 conn(0x7f4e70123080 msgr2=0x7f4e7011eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> v2:172.21.15.204:3300/0 conn(0x7f4e70123080 0x7f4e7011eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 -- 172.21.15.110:0/3583750314 >> v1:172.21.15.204:6789/0 conn(0x7f4e70121440 legacy=0x7f4e7007b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:39.909 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 -- 172.21.15.110:0/3583750314 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70124eb0 con 0x7f4e7011f030 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> v2:172.21.15.110:3300/0 conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> v2:172.21.15.110:3300/0 conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.910 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e6800b290 con 0x7f4e7011f030 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68005ca0 con 0x7f4e7011f030 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e70133f10 con 0x7f4e7011f030 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e6800b870 con 0x7f4e7011f030 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e701234a0 con 0x7f4e7011f030 2019-10-13T04:04:39.911 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4e680152b0 con 0x7f4e7011f030 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 0x7f4e54029e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f4e68034bb0 con 0x7f4e7011f030 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6ed9d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 0x7f4e54029e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:39.912 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.900+0000 7f4e6ed9d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 0x7f4e54029e60 crc :-1 s=READY pgs=38 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:39.913 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.904+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e700555c0 con 0x7f4e7011f030 2019-10-13T04:04:39.913 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.908+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:39.913 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:39.908+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4e6801db10 con 0x7f4e7011f030 2019-10-13T04:04:40.069 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.068+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e70124730 con 0x7f4e7011f030 2019-10-13T04:04:40.071 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.068+0000 7f4e6f59e700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.071 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.068+0000 7f4e67fff700 1 -- 172.21.15.110:0/3583750314 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10487 (secure 0 0 0) 0x7f4e6801dd70 con 0x7f4e7011f030 2019-10-13T04:04:40.072 INFO:teuthology.misc.health.smithi110.stdout: 2019-10-13T04:04:40.073 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":14,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:37.331541+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:40.076 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 msgr2=0x7f4e54029e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.076 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 0x7f4e54029e60 crc :-1 s=READY pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:40.076 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 msgr2=0x7f4e7013b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.077 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f4e700a36a0 tx=0x7f4e58001170).stop 2019-10-13T04:04:40.077 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 shutdown_connections 2019-10-13T04:04:40.077 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4e54027960 0x7f4e54029e60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:40.077 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/3583750314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e7011f030 0x7f4e7013b300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:40.078 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 --2- 172.21.15.110:0/3583750314 >> v2:172.21.15.204:3300/0 conn(0x7f4e70123080 0x7f4e7011eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:40.078 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 shutdown_connections 2019-10-13T04:04:40.078 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 wait complete. 2019-10-13T04:04:40.078 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:40.072+0000 7f4e74e6d700 1 -- 172.21.15.110:0/3583750314 >> 172.21.15.110:0/3583750314 conn(0x7f4e7007e0c0 msgr2=0x7f4e700a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:40.084 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T04:04:40.085 INFO:tasks.ceph:Creating RBD pool 2019-10-13T04:04:40.085 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:40.085 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2019-10-13T04:04:40.191 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 Processor -- start 2019-10-13T04:04:40.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 -- start start 2019-10-13T04:04:40.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:40.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540a0fc0 con 0x7f6b540aa2b0 2019-10-13T04:04:40.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540a75f0 con 0x7f6b540a17e0 2019-10-13T04:04:40.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5c00e700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54059e10 con 0x7f6b540a88e0 2019-10-13T04:04:40.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5a5ab700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6b540aa2b0 0x7f6b5407e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45216/0 (socket says 172.21.15.110:45216) 2019-10-13T04:04:40.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b5a5ab700 1 -- 172.21.15.110:0/4060041183 learned_addr learned my addr 172.21.15.110:0/4060041183 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:40.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:40.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497621652 0 0) 0x7f6b540a0fc0 con 0x7f6b540aa2b0 2019-10-13T04:04:40.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b400030a0 con 0x7f6b540aa2b0 2019-10-13T04:04:40.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276722459 0 0) 0x7f6b540a75f0 con 0x7f6b540a17e0 2019-10-13T04:04:40.195 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b540a0fc0 con 0x7f6b540a17e0 2019-10-13T04:04:40.195 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b59daa700 1 -- 172.21.15.110:0/4060041183 >> v1:172.21.15.204:6789/0 conn(0x7f6b540a17e0 legacy=0x7f6b5414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.195 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b59daa700 1 -- 172.21.15.110:0/4060041183 >> v1:172.21.15.110:6789/0 conn(0x7f6b540aa2b0 legacy=0x7f6b5407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.195 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.184+0000 7f6b59daa700 1 -- 172.21.15.110:0/4060041183 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b54059a20 con 0x7f6b540a88e0 2019-10-13T04:04:40.196 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.196 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.196 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).ready entity=mon.2 client_cookie=3a48d8509fc0b980 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:40.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b440112a0 con 0x7f6b540a88e0 2019-10-13T04:04:40.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b59daa700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.197 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b44008bb0 con 0x7f6b540a88e0 2019-10-13T04:04:40.198 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b595a9700 1 -- 172.21.15.110:0/4060041183 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b44011880 con 0x7f6b540a88e0 2019-10-13T04:04:40.198 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 -- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 msgr2=0x7f6b540a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.198 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7f6b44002f30 tx=0x7f6b4400fbe0).stop 2019-10-13T04:04:40.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 -- 172.21.15.110:0/4060041183 shutdown_connections 2019-10-13T04:04:40.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/4060041183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b540a88e0 0x7f6b540a1290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:40.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 -- 172.21.15.110:0/4060041183 shutdown_connections 2019-10-13T04:04:40.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 -- 172.21.15.110:0/4060041183 wait complete. 2019-10-13T04:04:40.200 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.188+0000 7f6b5c00e700 1 -- 172.21.15.110:0/4060041183 >> 172.21.15.110:0/4060041183 conn(0x7f6b5405e980 msgr2=0x7f6b540a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:40.206 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 Processor -- start 2019-10-13T04:04:40.206 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- start start 2019-10-13T04:04:40.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6b54128f20 0x7f6b540a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:40.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b541283c0 con 0x7f6b5407b190 2019-10-13T04:04:40.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5414f200 con 0x7f6b54115b60 2019-10-13T04:04:40.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b54128740 con 0x7f6b54128f20 2019-10-13T04:04:40.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5adac700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6b5407b190 0x7f6b5407b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45222/0 (socket says 172.21.15.110:45222) 2019-10-13T04:04:40.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5adac700 1 -- 172.21.15.110:0/2117646024 learned_addr learned my addr 172.21.15.110:0/2117646024 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:40.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> v2:172.21.15.204:3300/0 conn(0x7f6b54128f20 0x7f6b540a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:40.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089143218 0 0) 0x7f6b541283c0 con 0x7f6b5407b190 2019-10-13T04:04:40.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c0030a0 con 0x7f6b5407b190 2019-10-13T04:04:40.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 -- 172.21.15.110:0/2117646024 >> v1:172.21.15.204:6789/0 conn(0x7f6b54115b60 legacy=0x7f6b54124e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 -- 172.21.15.110:0/2117646024 >> v1:172.21.15.110:6789/0 conn(0x7f6b5407b190 legacy=0x7f6b5407b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:40.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 -- 172.21.15.110:0/2117646024 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5407bdb0 con 0x7f6b54128f20 2019-10-13T04:04:40.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> v2:172.21.15.204:3300/0 conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> v2:172.21.15.204:3300/0 conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:40.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b44008de0 con 0x7f6b54128f20 2019-10-13T04:04:40.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b54130af0 con 0x7f6b54128f20 2019-10-13T04:04:40.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b440114a0 con 0x7f6b54128f20 2019-10-13T04:04:40.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b44007340 con 0x7f6b54128f20 2019-10-13T04:04:40.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.192+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b54131020 con 0x7f6b54128f20 2019-10-13T04:04:40.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f6b4400f850 con 0x7f6b54128f20 2019-10-13T04:04:40.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b4affd700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 0x7f6b3c02a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:40.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f6b44035240 con 0x7f6b54128f20 2019-10-13T04:04:40.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b5a5ab700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 0x7f6b3c02a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:40.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.196+0000 7f6b5a5ab700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 0x7f6b3c02a1b0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:40.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.200+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b540555c0 con 0x7f6b54128f20 2019-10-13T04:04:40.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.212+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:40.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.212+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f6b4401dc10 con 0x7f6b54128f20 2019-10-13T04:04:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:40.364+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f6b540a79c0 con 0x7f6b54128f20 2019-10-13T04:04:41.365 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.364+0000 7f6b59daa700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.364+0000 7f6b4affd700 1 -- 172.21.15.110:0/2117646024 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f6b4400fc20 con 0x7f6b54128f20 2019-10-13T04:04:41.380 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2019-10-13T04:04:41.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.380+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 msgr2=0x7f6b3c02a1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.380+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 0x7f6b3c02a1b0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:41.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 msgr2=0x7f6b540a5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f6b500010d0 tx=0x7f6b50001110).stop 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 shutdown_connections 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f6b3c027cb0 0x7f6b3c02a1b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 --2- 172.21.15.110:0/2117646024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b54128f20 0x7f6b540a5d30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 shutdown_connections 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 wait complete. 2019-10-13T04:04:41.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.384+0000 7f6b5c00e700 1 -- 172.21.15.110:0/2117646024 >> 172.21.15.110:0/2117646024 conn(0x7f6b5405e950 msgr2=0x7f6b5409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:41.399 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:41.400 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-10-13T04:04:41.504 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 Processor -- start 2019-10-13T04:04:41.504 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 -- start start 2019-10-13T04:04:41.504 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc144430 0x7f08bc144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0a3a80 con 0x7f08bc144da0 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0a2530 con 0x7f08bc0a9890 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.496+0000 7f08c1288700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0aa570 con 0x7f08bc144430 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc144430 0x7f08bc144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc144430 0x7f08bc144850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42520/0 (socket says 172.21.15.110:42520) 2019-10-13T04:04:41.505 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 -- 172.21.15.110:0/361154739 learned_addr learned my addr 172.21.15.110:0/361154739 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 -- 172.21.15.110:0/361154739 >> v1:172.21.15.204:6789/0 conn(0x7f08bc0a9890 legacy=0x7f08bc0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 -- 172.21.15.110:0/361154739 >> v1:172.21.15.110:6789/0 conn(0x7f08bc144da0 legacy=0x7f08bc14ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 -- 172.21.15.110:0/361154739 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0a9270 con 0x7f08bc144430 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> v2:172.21.15.110:3300/0 conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08a4008b00 tx=0x7f08a4010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> v2:172.21.15.110:3300/0 conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08a4008b00 tx=0x7f08a4010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f08a4008b00 tx=0x7f08a4010020).ready entity=mon.0 client_cookie=f4e987524bc422c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:41.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f08a4008b00 tx=0x7f08a4010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08ba7fc700 1 -- 172.21.15.110:0/361154739 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a400a160 con 0x7f08bc144430 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f08a4008b00 tx=0x7f08a4010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08ba7fc700 1 -- 172.21.15.110:0/361154739 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a4011a90 con 0x7f08bc144430 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08baffd700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f08a4008b00 tx=0x7f08a4010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08ba7fc700 1 -- 172.21.15.110:0/361154739 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a400a740 con 0x7f08bc144430 2019-10-13T04:04:41.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 -- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 msgr2=0x7f08bc144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f08a4008b00 tx=0x7f08a4010020).stop 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 -- 172.21.15.110:0/361154739 shutdown_connections 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 --2- 172.21.15.110:0/361154739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc144430 0x7f08bc144850 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 -- 172.21.15.110:0/361154739 shutdown_connections 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 -- 172.21.15.110:0/361154739 wait complete. 2019-10-13T04:04:41.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.500+0000 7f08c1288700 1 -- 172.21.15.110:0/361154739 >> 172.21.15.110:0/361154739 conn(0x7f08bc09c500 msgr2=0x7f08bc09c920 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:41.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 Processor -- start 2019-10-13T04:04:41.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 -- start start 2019-10-13T04:04:41.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc120d50 0x7f08bc0a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:41.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f08bc0a6350 0x7f08bc0a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:41.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0a5bb0 con 0x7f08bc0a27a0 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0a3d40 con 0x7f08bc120d50 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08c1288700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0a6cc0 con 0x7f08bc0a6350 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc120d50 0x7f08bc0a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08bc120d50 0x7f08bc0a3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42524/0 (socket says 172.21.15.110:42524) 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 -- 172.21.15.110:0/2365429497 learned_addr learned my addr 172.21.15.110:0/2365429497 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:41.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08baffd700 1 --2- 172.21.15.110:0/2365429497 >> v2:172.21.15.204:3300/0 conn(0x7f08bc0a6350 0x7f08bc0a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 -- 172.21.15.110:0/2365429497 >> v2:172.21.15.204:3300/0 conn(0x7f08bc0a6350 msgr2=0x7f08bc0a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> v2:172.21.15.204:3300/0 conn(0x7f08bc0a6350 0x7f08bc0a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 -- 172.21.15.110:0/2365429497 >> v1:172.21.15.204:6789/0 conn(0x7f08bc0a27a0 legacy=0x7f08bc0a2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 -- 172.21.15.110:0/2365429497 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0a33c0 con 0x7f08bc120d50 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> v2:172.21.15.110:3300/0 conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> v2:172.21.15.110:3300/0 conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.504+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac00ca90 con 0x7f08bc120d50 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac00c6b0 con 0x7f08bc120d50 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc13dc40 con 0x7f08bc120d50 2019-10-13T04:04:41.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac010f60 con 0x7f08bc120d50 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc13e140 con 0x7f08bc120d50 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f08ac007420 con 0x7f08bc120d50 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 0x7f08a0029e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:41.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7f08ac034b60 con 0x7f08bc120d50 2019-10-13T04:04:41.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08baffd700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 0x7f08a0029e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:41.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.508+0000 7f08baffd700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 0x7f08a0029e40 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:41.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.512+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc11dcf0 con 0x7f08bc120d50 2019-10-13T04:04:41.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.520+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:41.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.520+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f08ac025f70 con 0x7f08bc120d50 2019-10-13T04:04:41.683 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:41.680+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08bc0555c0 con 0x7f08bc120d50 2019-10-13T04:04:42.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.376+0000 7f08bb7fe700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.376+0000 7f08b3fff700 1 -- 172.21.15.110:0/2365429497 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f08ac007710 con 0x7f08bc120d50 2019-10-13T04:04:42.398 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2019-10-13T04:04:42.401 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 msgr2=0x7f08a0029e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 0x7f08a0029e40 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 msgr2=0x7f08bc0a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7f08ac000e90 tx=0x7f08a4010430).stop 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 shutdown_connections 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f08a0027940 0x7f08a0029e40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.402 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 --2- 172.21.15.110:0/2365429497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08bc120d50 0x7f08bc0a3fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.403 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 --2- 172.21.15.110:0/2365429497 >> v2:172.21.15.204:3300/0 conn(0x7f08bc0a6350 0x7f08bc0a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.403 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 shutdown_connections 2019-10-13T04:04:42.403 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 wait complete. 2019-10-13T04:04:42.403 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.400+0000 7f08c1288700 1 -- 172.21.15.110:0/2365429497 >> 172.21.15.110:0/2365429497 conn(0x7f08bc0a8180 msgr2=0x7f08bc0a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:42.413 INFO:tasks.ceph:Setting up CephFS filesystem... 2019-10-13T04:04:42.413 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:42.413 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:42.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.500+0000 7f19610dd700 1 Processor -- start 2019-10-13T04:04:42.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.500+0000 7f19610dd700 1 -- start start 2019-10-13T04:04:42.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 0x7f195c0a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f195c0a8c10 0x7f195c0a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c0a77f0 con 0x7f195c0a6410 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c0a20e0 con 0x7f195c0a8c10 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0a2380 con 0x7f195c065060 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195b59e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f195c0a6410 0x7f195c144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45244/0 (socket says 172.21.15.110:45244) 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195b59e700 1 -- 172.21.15.110:0/1873979179 learned_addr learned my addr 172.21.15.110:0/1873979179 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:42.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195ad9d700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 0x7f195c0a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> v2:172.21.15.204:3300/0 conn(0x7f195c0a8c10 0x7f195c0a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784664216 0 0) 0x7f195c0a77f0 con 0x7f195c0a6410 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19480030a0 con 0x7f195c0a6410 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 -- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 msgr2=0x7f195c0a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 502824848 0 0) 0x7f19480030a0 con 0x7f195c0a6410 2019-10-13T04:04:42.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 0x7f195c0a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 -- 172.21.15.110:0/1873979179 >> v1:172.21.15.110:6789/0 conn(0x7f195c0a6410 legacy=0x7f195c144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 -- 172.21.15.110:0/1873979179 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c14f9f0 con 0x7f195c0a8c10 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195ad9d700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 0x7f195c0a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> v2:172.21.15.204:3300/0 conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f194c001d10 tx=0x7f194c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> v2:172.21.15.204:3300/0 conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f194c001d10 tx=0x7f194c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f194c001d10 tx=0x7f194c001d50).ready entity=mon.1 client_cookie=5f0e841967cca3f9 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f194c001d10 tx=0x7f194c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c001ad0 con 0x7f195c0a8c10 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f194c001d10 tx=0x7f194c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19608dc700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f194c001d10 tx=0x7f194c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c0018e0 con 0x7f195c0a8c10 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f195a59c700 1 -- 172.21.15.110:0/1873979179 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1954005740 con 0x7f195c0a8c10 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 msgr2=0x7f195c0a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f194c001d10 tx=0x7f194c001d50).stop 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- 172.21.15.110:0/1873979179 shutdown_connections 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f195c0a8c10 0x7f195c0a9030 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 --2- 172.21.15.110:0/1873979179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f195c065060 0x7f195c0a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- 172.21.15.110:0/1873979179 shutdown_connections 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- 172.21.15.110:0/1873979179 wait complete. 2019-10-13T04:04:42.513 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.504+0000 7f19610dd700 1 -- 172.21.15.110:0/1873979179 >> 172.21.15.110:0/1873979179 conn(0x7f195c05afb0 msgr2=0x7f195c09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:42.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 Processor -- start 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- start start 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f195c113550 0x7f195c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c118100 con 0x7f195c14f050 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c1175e0 con 0x7f195c115ff0 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c113f20 con 0x7f195c113550 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f195c113550 0x7f195c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.517 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f195c113550 0x7f195c0788b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42542/0 (socket says 172.21.15.110:42542) 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 -- 172.21.15.110:0/50294718 learned_addr learned my addr 172.21.15.110:0/50294718 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773526692 0 0) 0x7f195c1175e0 con 0x7f195c115ff0 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19400030a0 con 0x7f195c115ff0 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 -- 172.21.15.110:0/50294718 >> v1:172.21.15.204:6789/0 conn(0x7f195c115ff0 legacy=0x7f195c07bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 -- 172.21.15.110:0/50294718 >> v1:172.21.15.110:6789/0 conn(0x7f195c14f050 legacy=0x7f195c0780b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 -- 172.21.15.110:0/50294718 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c078e00 con 0x7f195c113550 2019-10-13T04:04:42.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> v2:172.21.15.110:3300/0 conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> v2:172.21.15.110:3300/0 conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f195400ccc0 con 0x7f195c113550 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c113b80 con 0x7f195c113550 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195400f430 con 0x7f195c113550 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1954008b60 con 0x7f195c113550 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.508+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c1316c0 con 0x7f195c113550 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f195400c310 con 0x7f195c113550 2019-10-13T04:04:42.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f1953fff700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 0x7f194002a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:42.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f1954034690 con 0x7f195c113550 2019-10-13T04:04:42.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f195b59e700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 0x7f194002a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.512+0000 7f195b59e700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 0x7f194002a250 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.516+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c113970 con 0x7f195c113550 2019-10-13T04:04:42.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.524+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.524+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f195401d150 con 0x7f195c113550 2019-10-13T04:04:42.694 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.692+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f195c0555c0 con 0x7f195c113550 2019-10-13T04:04:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.692+0000 7f19608dc700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.695 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.692+0000 7f1953fff700 1 -- 172.21.15.110:0/50294718 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11874 (secure 0 0 0) 0x7f195401d3b0 con 0x7f195c113550 2019-10-13T04:04:42.697 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:42.698 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:42.361630+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:42.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 msgr2=0x7f194002a250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 0x7f194002a250 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 msgr2=0x7f195c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f195c114370 tx=0x7f1954007130).stop 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 shutdown_connections 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1940027d50 0x7f194002a250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 --2- 172.21.15.110:0/50294718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f195c113550 0x7f195c0788b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 shutdown_connections 2019-10-13T04:04:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 wait complete. 2019-10-13T04:04:42.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.696+0000 7f19610dd700 1 -- 172.21.15.110:0/50294718 >> 172.21.15.110:0/50294718 conn(0x7f195c0a7cf0 msgr2=0x7f195c09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:40.397189+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T04:04:42.713 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:42.714 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T04:04:42.811 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 Processor -- start 2019-10-13T04:04:42.811 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 -- start start 2019-10-13T04:04:42.811 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a540aa2b0 0x7f1a5407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:42.811 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a88e0 0x7f1a5414e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:42.812 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a540a0fc0 con 0x7f1a5407ebf0 2019-10-13T04:04:42.812 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a540a1a90 con 0x7f1a540aa2b0 2019-10-13T04:04:42.812 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a5a85a700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54059e10 con 0x7f1a540a88e0 2019-10-13T04:04:42.812 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.804+0000 7f1a58df7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a540aa2b0 0x7f1a5407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.812 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a53fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1a5407ebf0 0x7f1a540a1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45260/0 (socket says 172.21.15.110:45260) 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a53fff700 1 -- 172.21.15.110:0/2422163110 learned_addr learned my addr 172.21.15.110:0/2422163110 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a595f8700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a88e0 0x7f1a5414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 -- 172.21.15.110:0/2422163110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a88e0 msgr2=0x7f1a5414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a88e0 0x7f1a5414e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 -- 172.21.15.110:0/2422163110 >> v1:172.21.15.110:6789/0 conn(0x7f1a5407ebf0 legacy=0x7f1a540a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a537fe700 1 -- 172.21.15.110:0/2422163110 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414738944 0 0) 0x7f1a540a0fc0 con 0x7f1a5407ebf0 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 -- 172.21.15.110:0/2422163110 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a540a75f0 con 0x7f1a540aa2b0 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> v2:172.21.15.110:3300/0 conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a4c008780 tx=0x7f1a44003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> v2:172.21.15.110:3300/0 conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a4c008780 tx=0x7f1a44003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f1a4c008780 tx=0x7f1a44003370).ready entity=mon.0 client_cookie=b94d3d8116cf958d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f1a4c008780 tx=0x7f1a44003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a537fe700 1 -- 172.21.15.110:0/2422163110 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a4c00a160 con 0x7f1a540aa2b0 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f1a4c008780 tx=0x7f1a44003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a537fe700 1 -- 172.21.15.110:0/2422163110 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c008bb0 con 0x7f1a540aa2b0 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a58df7700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f1a4c008780 tx=0x7f1a44003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 msgr2=0x7f1a5407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f1a4c008780 tx=0x7f1a44003370).stop 2019-10-13T04:04:42.815 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a537fe700 1 -- 172.21.15.110:0/2422163110 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 0 0 0) 0x7f1a4c00a740 con 0x7f1a540aa2b0 2019-10-13T04:04:42.816 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- 172.21.15.110:0/2422163110 shutdown_connections 2019-10-13T04:04:42.816 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a88e0 0x7f1a5414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:42.816 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/2422163110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a540aa2b0 0x7f1a5407e6a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.816 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- 172.21.15.110:0/2422163110 shutdown_connections 2019-10-13T04:04:42.816 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- 172.21.15.110:0/2422163110 wait complete. 2019-10-13T04:04:42.817 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- 172.21.15.110:0/2422163110 >> 172.21.15.110:0/2422163110 conn(0x7f1a5405e980 msgr2=0x7f1a540a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:42.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 Processor -- start 2019-10-13T04:04:42.822 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.808+0000 7f1a5a85a700 1 -- start start 2019-10-13T04:04:42.822 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 0x7f1a54079760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:42.822 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a54079260 0x7f1a54079cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:42.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5407b300 con 0x7f1a5409c500 2019-10-13T04:04:42.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a54078760 con 0x7f1a54079260 2019-10-13T04:04:42.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54078a00 con 0x7f1a540a65d0 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a54079260 0x7f1a54079cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a54079260 0x7f1a54079cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42556/0 (socket says 172.21.15.110:42556) 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 -- 172.21.15.110:0/3402517091 learned_addr learned my addr 172.21.15.110:0/3402517091 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a58df7700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 0x7f1a54079760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 -- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 msgr2=0x7f1a54079760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 0x7f1a54079760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 -- 172.21.15.110:0/3402517091 >> v1:172.21.15.204:6789/0 conn(0x7f1a5409c500 legacy=0x7f1a540a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 -- 172.21.15.110:0/3402517091 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54078ca0 con 0x7f1a54079260 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a58df7700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 0x7f1a54079760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> v2:172.21.15.110:3300/0 conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> v2:172.21.15.110:3300/0 conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a44012be0 con 0x7f1a54079260 2019-10-13T04:04:42.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a54127800 con 0x7f1a54079260 2019-10-13T04:04:42.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44012800 con 0x7f1a54079260 2019-10-13T04:04:42.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a44011510 con 0x7f1a54079260 2019-10-13T04:04:42.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a540a2c30 con 0x7f1a54079260 2019-10-13T04:04:42.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1a440058b0 con 0x7f1a54079260 2019-10-13T04:04:42.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 0x7f1a38029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:42.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f1a440351a0 con 0x7f1a54079260 2019-10-13T04:04:42.830 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.812+0000 7f1a58df7700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 0x7f1a38029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:42.831 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.816+0000 7f1a58df7700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 0x7f1a38029d00 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:42.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.820+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a540555c0 con 0x7f1a54079260 2019-10-13T04:04:42.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.828+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.828+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f1a44026530 con 0x7f1a54079260 2019-10-13T04:04:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.976+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1a5407aa10 con 0x7f1a54079260 2019-10-13T04:04:42.978 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.976+0000 7f1a595f8700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:42.979 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.976+0000 7f1a517fa700 1 -- 172.21.15.110:0/3402517091 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1a44005ba0 con 0x7f1a54079260 2019-10-13T04:04:42.979 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:04:42.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 msgr2=0x7f1a38029d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 0x7f1a38029d00 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 msgr2=0x7f1a54079cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:42.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f1a54078920 tx=0x7f1a4c008180).stop 2019-10-13T04:04:42.981 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 shutdown_connections 2019-10-13T04:04:42.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1a38027800 0x7f1a38029d00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a54079260 0x7f1a54079cb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.982 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 --2- 172.21.15.110:0/3402517091 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a540a65d0 0x7f1a54079760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 shutdown_connections 2019-10-13T04:04:42.984 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 wait complete. 2019-10-13T04:04:42.985 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:42.980+0000 7f1a5a85a700 1 -- 172.21.15.110:0/3402517091 >> 172.21.15.110:0/3402517091 conn(0x7f1a540a1290 msgr2=0x7f1a5407d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:42.991 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:42.991 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2019-10-13T04:04:43.088 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.080+0000 7f30c88d8700 1 Processor -- start 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.080+0000 7f30c88d8700 1 -- start start 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c40a77f0 con 0x7f30c40a8c10 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c40a9030 con 0x7f30c40a6410 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40a20e0 con 0x7f30c4065060 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c359e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f30c40a8c10 0x7f30c414e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45276/0 (socket says 172.21.15.110:45276) 2019-10-13T04:04:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c359e700 1 -- 172.21.15.110:0/3725919335 learned_addr learned my addr 172.21.15.110:0/3725919335 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894465656 0 0) 0x7f30c40a77f0 con 0x7f30c40a8c10 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00030a0 con 0x7f30c40a8c10 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187482753 0 0) 0x7f30c40a9030 con 0x7f30c40a6410 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c40a77f0 con 0x7f30c40a6410 2019-10-13T04:04:43.090 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 -- 172.21.15.110:0/3725919335 >> v1:172.21.15.204:6789/0 conn(0x7f30c40a6410 legacy=0x7f30c4144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 -- 172.21.15.110:0/3725919335 >> v1:172.21.15.110:6789/0 conn(0x7f30c40a8c10 legacy=0x7f30c414e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 -- 172.21.15.110:0/3725919335 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40a9260 con 0x7f30c4065060 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).ready entity=mon.2 client_cookie=699ea5c5b6b1b02c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac0112a0 con 0x7f30c4065060 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c259c700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac008bb0 con 0x7f30c4065060 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c1d9b700 1 -- 172.21.15.110:0/3725919335 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac011880 con 0x7f30c4065060 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 -- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 msgr2=0x7f30c40a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f30ac002f30 tx=0x7f30ac00fbe0).stop 2019-10-13T04:04:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 -- 172.21.15.110:0/3725919335 shutdown_connections 2019-10-13T04:04:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.084+0000 7f30c88d8700 1 --2- 172.21.15.110:0/3725919335 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c4065060 0x7f30c40a1970 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.088+0000 7f30c88d8700 1 -- 172.21.15.110:0/3725919335 shutdown_connections 2019-10-13T04:04:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.088+0000 7f30c88d8700 1 -- 172.21.15.110:0/3725919335 wait complete. 2019-10-13T04:04:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.088+0000 7f30c88d8700 1 -- 172.21.15.110:0/3725919335 >> 172.21.15.110:0/3725919335 conn(0x7f30c405afb0 msgr2=0x7f30c409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.088+0000 7f30c88d8700 1 Processor -- start 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.088+0000 7f30c88d8700 1 -- start start 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30c407d330 0x7f30c407d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c407dca0 0x7f30c4130110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c40a0140 con 0x7f30c409fab0 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c4119d60 con 0x7f30c407d330 2019-10-13T04:04:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c411a000 con 0x7f30c407dca0 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c2d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f30c409fab0 0x7f30c41128e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45278/0 (socket says 172.21.15.110:45278) 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c2d9d700 1 -- 172.21.15.110:0/1134118499 learned_addr learned my addr 172.21.15.110:0/1134118499 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> v2:172.21.15.110:3300/0 conn(0x7f30c407d330 0x7f30c407d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c359e700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c407dca0 0x7f30c4130110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362026253 0 0) 0x7f30c40a0140 con 0x7f30c409fab0 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a80030a0 con 0x7f30c409fab0 2019-10-13T04:04:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 -- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c407dca0 msgr2=0x7f30c4130110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c407dca0 0x7f30c4130110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 -- 172.21.15.110:0/1134118499 >> v1:172.21.15.110:6789/0 conn(0x7f30c409fab0 legacy=0x7f30c41128e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1152534018 0 0) 0x7f30a80030a0 con 0x7f30c409fab0 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 -- 172.21.15.110:0/1134118499 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40a6920 con 0x7f30c407d330 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> v2:172.21.15.110:3300/0 conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> v2:172.21.15.110:3300/0 conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30b4001dc0 con 0x7f30c407d330 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b4001bd0 con 0x7f30c407d330 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c4119700 con 0x7f30c407d330 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac0112a0 con 0x7f30c407d330 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c411a2a0 con 0x7f30c407d330 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f30ac007150 con 0x7f30c407d330 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.092+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.096+0000 7f30bb7fe700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 0x7f30a802a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.103 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.096+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f30ac034890 con 0x7f30c407d330 2019-10-13T04:04:43.104 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.096+0000 7f30c2d9d700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 0x7f30a802a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.104 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.096+0000 7f30c2d9d700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 0x7f30a802a310 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.105 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.096+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c40555c0 con 0x7f30c407d330 2019-10-13T04:04:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.104+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.104+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f30ac011de0 con 0x7f30c407d330 2019-10-13T04:04:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.256+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f30c4119b30 con 0x7f30c407d330 2019-10-13T04:04:43.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c259c700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30bb7fe700 1 -- 172.21.15.110:0/1134118499 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7f30c4119700 con 0x7f30c407d330 2019-10-13T04:04:43.261 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:43.261 INFO:teuthology.orchestra.run.smithi110.stdout:[ 2019-10-13T04:04:43.261 INFO:teuthology.orchestra.run.smithi110.stdout: { 2019-10-13T04:04:43.262 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 1, 2019-10-13T04:04:43.262 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": "rbd" 2019-10-13T04:04:43.262 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T04:04:43.262 INFO:teuthology.orchestra.run.smithi110.stdout:] 2019-10-13T04:04:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 msgr2=0x7f30a802a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 0x7f30a802a310 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 msgr2=0x7f30c407d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.264 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f30b4002000 tx=0x7f30b4001b10).stop 2019-10-13T04:04:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 shutdown_connections 2019-10-13T04:04:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c407dca0 0x7f30c4130110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f30a8027e10 0x7f30a802a310 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 --2- 172.21.15.110:0/1134118499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30c407d330 0x7f30c407d750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 shutdown_connections 2019-10-13T04:04:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 wait complete. 2019-10-13T04:04:43.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.260+0000 7f30c88d8700 1 -- 172.21.15.110:0/1134118499 >> 172.21.15.110:0/1134118499 conn(0x7f30c40a6410 msgr2=0x7f30c4059300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:43.272 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2019-10-13T04:04:43.273 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:43.273 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2019-10-13T04:04:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.364+0000 7fbf9d4ad700 1 Processor -- start 2019-10-13T04:04:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.364+0000 7fbf9d4ad700 1 -- start start 2019-10-13T04:04:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf980aa2b0 0x7fbf9807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:43.374 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf980a88e0 0x7fbf9814e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf980a0fc0 con 0x7fbf9807ebf0 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980a1a90 con 0x7fbf980a88e0 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf98059e10 con 0x7fbf980aa2b0 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf980a88e0 0x7fbf9814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf980a88e0 0x7fbf9814e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42578/0 (socket says 172.21.15.110:42578) 2019-10-13T04:04:43.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 -- 172.21.15.110:0/2560814239 learned_addr learned my addr 172.21.15.110:0/2560814239 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf977fe700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf980aa2b0 0x7fbf9807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 -- 172.21.15.110:0/2560814239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf980aa2b0 msgr2=0x7fbf9807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf980aa2b0 0x7fbf9807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 -- 172.21.15.110:0/2560814239 >> v1:172.21.15.110:6789/0 conn(0x7fbf9807ebf0 legacy=0x7fbf980a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 -- 172.21.15.110:0/2560814239 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980a75f0 con 0x7fbf980a88e0 2019-10-13T04:04:43.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> v2:172.21.15.110:3300/0 conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf98067e60 tx=0x7fbf88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> v2:172.21.15.110:3300/0 conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf98067e60 tx=0x7fbf88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7fbf98067e60 tx=0x7fbf88007710).ready entity=mon.0 client_cookie=2f31250cca2b1077 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7fbf98067e60 tx=0x7fbf88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7fbf98067e60 tx=0x7fbf88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf967fc700 1 -- 172.21.15.110:0/2560814239 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf90010b90 con 0x7fbf980a88e0 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf97fff700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7fbf98067e60 tx=0x7fbf88007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf967fc700 1 -- 172.21.15.110:0/2560814239 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf900107b0 con 0x7fbf980a88e0 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf967fc700 1 -- 172.21.15.110:0/2560814239 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf900112b0 con 0x7fbf980a88e0 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 msgr2=0x7fbf9814e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.368+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7fbf98067e60 tx=0x7fbf88007710).stop 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/2560814239 shutdown_connections 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf980a88e0 0x7fbf9814e900 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/2560814239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf980aa2b0 0x7fbf9807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:43.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/2560814239 shutdown_connections 2019-10-13T04:04:43.380 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/2560814239 wait complete. 2019-10-13T04:04:43.380 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/2560814239 >> 172.21.15.110:0/2560814239 conn(0x7fbf9805e980 msgr2=0x7fbf980a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:43.380 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 Processor -- start 2019-10-13T04:04:43.380 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- start start 2019-10-13T04:04:43.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9811d2b0 0x7fbf9811a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf9811b540 0x7fbf980a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9811cd00 con 0x7fbf981227b0 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9811ae50 con 0x7fbf9811b540 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf9d4ad700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9811d010 con 0x7fbf9811d2b0 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf9811b540 0x7fbf980a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf9811b540 0x7fbf980a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42584/0 (socket says 172.21.15.110:42584) 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 -- 172.21.15.110:0/158827804 learned_addr learned my addr 172.21.15.110:0/158827804 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:43.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf96ffd700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9811d2b0 0x7fbf9811a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 -- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9811d2b0 msgr2=0x7fbf9811a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9811d2b0 0x7fbf9811a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:43.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 -- 172.21.15.110:0/158827804 >> v1:172.21.15.204:6789/0 conn(0x7fbf981227b0 legacy=0x7fbf980a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:43.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 -- 172.21.15.110:0/158827804 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980a29a0 con 0x7fbf9811b540 2019-10-13T04:04:43.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.372+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> v2:172.21.15.110:3300/0 conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.376+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> v2:172.21.15.110:3300/0 conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.376+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=READY pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf900058b0 con 0x7fbf9811b540 2019-10-13T04:04:43.390 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf90010d90 con 0x7fbf9811b540 2019-10-13T04:04:43.390 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9811b960 con 0x7fbf9811b540 2019-10-13T04:04:43.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9000f280 con 0x7fbf9811b540 2019-10-13T04:04:43.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf980a39f0 con 0x7fbf9811b540 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fbf90015250 con 0x7fbf9811b540 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 0x7fbf78029da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fbf90034d40 con 0x7fbf9811b540 2019-10-13T04:04:43.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf977fe700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 0x7fbf78029da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:43.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf977fe700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 0x7fbf78029da0 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:43.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.380+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf980555c0 con 0x7fbf9811b540 2019-10-13T04:04:43.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.388+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:43.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.388+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fbf9001d8f0 con 0x7fbf9811b540 2019-10-13T04:04:43.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:43.536+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fbf9811b1a0 con 0x7fbf9811b540 2019-10-13T04:04:44.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.388+0000 7fbf97fff700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=THROTTLE_DONE pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.388+0000 7fbf7ffff700 1 -- 172.21.15.110:0/158827804 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbf9001db50 con 0x7fbf9811b540 2019-10-13T04:04:44.402 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' created 2019-10-13T04:04:44.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 msgr2=0x7fbf78029da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 0x7fbf78029da0 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:44.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 msgr2=0x7fbf980a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 secure :-1 s=READY pgs=67 cs=0 l=1 rx=0x7fbf9811cf30 tx=0x7fbf8c001000).stop 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 shutdown_connections 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbf780278a0 0x7fbf78029da0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9811b540 0x7fbf980a2450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 --2- 172.21.15.110:0/158827804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9811d2b0 0x7fbf9811a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 shutdown_connections 2019-10-13T04:04:44.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 wait complete. 2019-10-13T04:04:44.408 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.404+0000 7fbf9d4ad700 1 -- 172.21.15.110:0/158827804 >> 172.21.15.110:0/158827804 conn(0x7fbf9807dce0 msgr2=0x7fbf980a86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:44.415 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2019-10-13T04:04:44.416 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:44.416 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2019-10-13T04:04:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.508+0000 7f2ba8e36700 1 Processor -- start 2019-10-13T04:04:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.508+0000 7f2ba8e36700 1 -- start start 2019-10-13T04:04:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba8e36700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba414dc70 0x7f2ba41500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba8e36700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40a3a80 con 0x7f2ba4144430 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba8e36700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40a2530 con 0x7f2ba40a9890 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba8e36700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40aa570 con 0x7f2ba414dc70 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2ba4144430 0x7f2ba4144850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45300/0 (socket says 172.21.15.110:45300) 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba259c700 1 -- 172.21.15.110:0/2570172448 learned_addr learned my addr 172.21.15.110:0/2570172448 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba359e700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba414dc70 0x7f2ba41500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787270697 0 0) 0x7f2b980010d0 con 0x7f2ba4144430 2019-10-13T04:04:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8c0030a0 con 0x7f2ba4144430 2019-10-13T04:04:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1465195866 0 0) 0x7f2b8c0030a0 con 0x7f2ba4144430 2019-10-13T04:04:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba414dc70 msgr2=0x7f2ba41500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba414dc70 0x7f2ba41500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 >> v1:172.21.15.204:6789/0 conn(0x7f2ba40a9890 legacy=0x7f2ba40a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40a9270 con 0x7f2ba4144430 2019-10-13T04:04:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f2b98001980 con 0x7f2ba4144430 2019-10-13T04:04:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 >> v1:172.21.15.110:6789/0 conn(0x7f2ba4144430 legacy=0x7f2ba4144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b8c004410 0x7f2b8c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b8c004d80 0x7f2b8c00b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40aa570 con 0x7f2b8c003700 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0051a0 con 0x7f2b8c004d80 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c04fbc0 con 0x7f2b8c004410 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba259c700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b8c004410 0x7f2b8c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba359e700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b8c004d80 0x7f2b8c00b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b8c004410 msgr2=0x7f2b8c004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b8c004410 0x7f2b8c004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b8c004d80 msgr2=0x7f2b8c00b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b8c004d80 0x7f2b8c00b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/2570172448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c04fe40 con 0x7f2b8c003700 2019-10-13T04:04:44.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).ready entity=mon.0 client_cookie=3461c363ab3f1e4a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b94006ab0 con 0x7f2b8c003700 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b940066d0 con 0x7f2b8c003700 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba1d9b700 1 -- 172.21.15.110:0/2570172448 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b94002590 con 0x7f2b8c003700 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.512+0000 7f2ba8e36700 1 -- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 msgr2=0x7f2b8c003b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/2570172448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8c003700 0x7f2b8c003b20 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f2b8c0053e0 tx=0x7f2b9c007930).stop 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba359e700 1 -- 172.21.15.110:0/2570172448 reap_dead start 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 -- 172.21.15.110:0/2570172448 shutdown_connections 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 -- 172.21.15.110:0/2570172448 shutdown_connections 2019-10-13T04:04:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 -- 172.21.15.110:0/2570172448 wait complete. 2019-10-13T04:04:44.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 -- 172.21.15.110:0/2570172448 >> 172.21.15.110:0/2570172448 conn(0x7f2ba409c500 msgr2=0x7f2ba409c920 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 Processor -- start 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.516+0000 7f2ba8e36700 1 -- start start 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2ba41218b0 0x7f2ba409f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 0x7f2ba413b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba409fc80 con 0x7f2ba409f860 2019-10-13T04:04:44.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba409e390 con 0x7f2ba41218b0 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba4064970 con 0x7f2ba4134600 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2ba409f860 0x7f2ba4133e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45314/0 (socket says 172.21.15.110:45314) 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba259c700 1 -- 172.21.15.110:0/764888019 learned_addr learned my addr 172.21.15.110:0/764888019 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba359e700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 0x7f2ba413b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> v2:172.21.15.204:3300/0 conn(0x7f2ba41218b0 0x7f2ba409f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666592864 0 0) 0x7f2ba409fc80 con 0x7f2ba409f860 2019-10-13T04:04:44.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b880030a0 con 0x7f2ba409f860 2019-10-13T04:04:44.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1461945904 0 0) 0x7f2b880030a0 con 0x7f2ba409f860 2019-10-13T04:04:44.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba409fc80 con 0x7f2ba409f860 2019-10-13T04:04:44.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f2b98001790 con 0x7f2ba409f860 2019-10-13T04:04:44.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 msgr2=0x7f2ba413b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 0x7f2ba413b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:44.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/764888019 >> v1:172.21.15.110:6789/0 conn(0x7f2ba409f860 legacy=0x7f2ba4133e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:44.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 -- 172.21.15.110:0/764888019 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba41222d0 con 0x7f2ba41218b0 2019-10-13T04:04:44.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba359e700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 0x7f2ba413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:44.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> v2:172.21.15.204:3300/0 conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> v2:172.21.15.204:3300/0 conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b94011a00 con 0x7f2ba41218b0 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b94006cf0 con 0x7f2ba41218b0 2019-10-13T04:04:44.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b940023a0 con 0x7f2ba41218b0 2019-10-13T04:04:44.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba4124650 con 0x7f2ba41218b0 2019-10-13T04:04:44.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.520+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba409daa0 con 0x7f2ba41218b0 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f2b94002640 con 0x7f2ba41218b0 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2b937fe700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 0x7f2b8802a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f2b94034610 con 0x7f2ba41218b0 2019-10-13T04:04:44.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2ba259c700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 0x7f2b8802a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:44.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2ba259c700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 0x7f2b8802a6d0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:44.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.524+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba40555c0 con 0x7f2ba41218b0 2019-10-13T04:04:44.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.536+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:44.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.536+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f2b9401cf70 con 0x7f2ba41218b0 2019-10-13T04:04:44.699 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:44.692+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"} v 0) v1 -- 0x7f2ba405cf50 con 0x7f2ba41218b0 2019-10-13T04:04:45.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.396+0000 7f2ba2d9d700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.396+0000 7f2b937fe700 1 -- 172.21.15.110:0/764888019 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"}]=0 v19) v1 ==== 154+0+0 (secure 0 0 0) 0x7f2b9401d1d0 con 0x7f2ba41218b0 2019-10-13T04:04:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 msgr2=0x7f2b8802a6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 0x7f2b8802a6d0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 msgr2=0x7f2ba409f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f2b94000f00 tx=0x7f2b98001110).stop 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 shutdown_connections 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f2b880281d0 0x7f2b8802a6d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ba4134600 0x7f2ba413b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 --2- 172.21.15.110:0/764888019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ba41218b0 0x7f2ba409f1b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 shutdown_connections 2019-10-13T04:04:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 wait complete. 2019-10-13T04:04:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.412+0000 7f2ba8e36700 1 -- 172.21.15.110:0/764888019 >> 172.21.15.110:0/764888019 conn(0x7f2ba40a8180 msgr2=0x7f2ba40a1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:45.425 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:45.425 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2019-10-13T04:04:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 Processor -- start 2019-10-13T04:04:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 -- start start 2019-10-13T04:04:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f50140a9890 0x7f50140a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50140a2280 0x7f50140a6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140a3a80 con 0x7f50140a6590 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140aa570 con 0x7f50140a9890 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5014064f80 con 0x7f50140a2280 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5018962700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f50140a6590 0x7f5014144430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45324/0 (socket says 172.21.15.110:45324) 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5018962700 1 -- 172.21.15.110:0/2162052150 learned_addr learned my addr 172.21.15.110:0/2162052150 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50140a2280 0x7f50140a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501359e700 1 --2- 172.21.15.110:0/2162052150 >> v2:172.21.15.204:3300/0 conn(0x7f50140a9890 0x7f50140a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405707067 0 0) 0x7f500c001450 con 0x7f50140a6590 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc0030a0 con 0x7f50140a6590 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2175997444 0 0) 0x7f4ffc0030a0 con 0x7f50140a6590 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50140a2280 msgr2=0x7f50140a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50140a2280 0x7f50140a6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 >> v2:172.21.15.204:3300/0 conn(0x7f50140a9890 msgr2=0x7f50140a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 --2- 172.21.15.110:0/2162052150 >> v2:172.21.15.204:3300/0 conn(0x7f50140a9890 0x7f50140a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140a9270 con 0x7f50140a6590 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f500c001d00 con 0x7f50140a6590 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 >> v1:172.21.15.110:6789/0 conn(0x7f50140a6590 legacy=0x7f5014144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ffc0036d0 0x7f4ffc003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ffc004d80 0x7f4ffc04f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140aa570 con 0x7f4ffc004410 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014064f80 con 0x7f4ffc004d80 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc04fbc0 con 0x7f4ffc0036d0 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501359e700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ffc0036d0 0x7f4ffc003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5018962700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ffc004d80 0x7f4ffc04f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 -- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ffc0036d0 msgr2=0x7f4ffc003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ffc0036d0 0x7f4ffc003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 -- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ffc004d80 msgr2=0x7f4ffc04f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ffc004d80 0x7f4ffc04f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 -- 172.21.15.110:0/2162052150 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc04fd80 con 0x7f4ffc004410 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5018962700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ffc004d80 0x7f4ffc04f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5008002f30 tx=0x7f500c008800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5008002f30 tx=0x7f500c008800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f5008002f30 tx=0x7f500c008800).ready entity=mon.0 client_cookie=656f6b5fd6289865 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f5008002f30 tx=0x7f500c008800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5008008bc0 con 0x7f4ffc004410 2019-10-13T04:04:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f5008002f30 tx=0x7f500c008800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5012d9d700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f5008002f30 tx=0x7f500c008800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50080087e0 con 0x7f4ffc004410 2019-10-13T04:04:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f501259c700 1 -- 172.21.15.110:0/2162052150 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5008005910 con 0x7f4ffc004410 2019-10-13T04:04:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 -- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 msgr2=0x7f4ffc004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5019163700 1 --2- 172.21.15.110:0/2162052150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ffc004410 0x7f4ffc004830 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f5008002f30 tx=0x7f500c008800).stop 2019-10-13T04:04:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.516+0000 7f5018962700 1 -- 172.21.15.110:0/2162052150 reap_dead start 2019-10-13T04:04:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- 172.21.15.110:0/2162052150 shutdown_connections 2019-10-13T04:04:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- 172.21.15.110:0/2162052150 shutdown_connections 2019-10-13T04:04:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- 172.21.15.110:0/2162052150 wait complete. 2019-10-13T04:04:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- 172.21.15.110:0/2162052150 >> 172.21.15.110:0/2162052150 conn(0x7f501409c500 msgr2=0x7f501409c920 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 Processor -- start 2019-10-13T04:04:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- start start 2019-10-13T04:04:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f501405ff70 0x7f50140a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5014133e70 0x7f5014134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140620d0 con 0x7f5014061cb0 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5014060480 con 0x7f5014133e70 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5019163700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5014060720 con 0x7f501405ff70 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5018962700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5014133e70 0x7f5014134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5012d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5014061cb0 0x7f501407db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45334/0 (socket says 172.21.15.110:45334) 2019-10-13T04:04:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5018962700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5014133e70 0x7f5014134290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42624/0 (socket says 172.21.15.110:42624) 2019-10-13T04:04:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f5012d9d700 1 -- 172.21.15.110:0/244763860 learned_addr learned my addr 172.21.15.110:0/244763860 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.520+0000 7f501359e700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f501405ff70 0x7f50140a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 -- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f501405ff70 msgr2=0x7f50140a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f501405ff70 0x7f50140a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 -- 172.21.15.110:0/244763860 >> v1:172.21.15.110:6789/0 conn(0x7f5014061cb0 legacy=0x7f501407db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705800184 0 0) 0x7f50140620d0 con 0x7f5014061cb0 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 -- 172.21.15.110:0/244763860 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501413c270 con 0x7f5014133e70 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> v2:172.21.15.110:3300/0 conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> v2:172.21.15.110:3300/0 conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c008d80 con 0x7f5014133e70 2019-10-13T04:04:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c0089a0 con 0x7f5014133e70 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501413b260 con 0x7f5014133e70 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c012950 con 0x7f5014133e70 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501413b760 con 0x7f5014133e70 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f500c012b50 con 0x7f5014133e70 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 0x7f4ff4029c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5463+0+0 (secure 0 0 0) 0x7f50140a3a80 con 0x7f5014133e70 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f501359e700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 0x7f4ff4029c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.524+0000 7f501359e700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 0x7f4ff4029c80 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.532+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50140555c0 con 0x7f5014133e70 2019-10-13T04:04:45.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.540+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:45.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.540+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f500c025ee0 con 0x7f5014133e70 2019-10-13T04:04:45.697 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:45.696+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"} v 0) v1 -- 0x7f50140664b0 con 0x7f5014133e70 2019-10-13T04:04:47.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.412+0000 7f5018962700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.414 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.412+0000 7f4ffbfff700 1 -- 172.21.15.110:0/244763860 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v21) v1 ==== 188+0+0 (secure 0 0 0) 0x7f500c00b3d0 con 0x7f5014133e70 2019-10-13T04:04:47.416 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' created 2019-10-13T04:04:47.421 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 msgr2=0x7f4ff4029c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.421 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 0x7f4ff4029c80 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.421 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 msgr2=0x7f5014134290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.422 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f5014065350 tx=0x7f5004007780).stop 2019-10-13T04:04:47.422 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 shutdown_connections 2019-10-13T04:04:47.422 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4ff4027780 0x7f4ff4029c80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.423 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5014133e70 0x7f5014134290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.423 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 --2- 172.21.15.110:0/244763860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f501405ff70 0x7f50140a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.423 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 shutdown_connections 2019-10-13T04:04:47.423 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 wait complete. 2019-10-13T04:04:47.423 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.420+0000 7f5019163700 1 -- 172.21.15.110:0/244763860 >> 172.21.15.110:0/244763860 conn(0x7f50140a7c80 msgr2=0x7f5014124bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:47.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:47.432 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2019-10-13T04:04:47.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.528+0000 7faef94ed700 1 Processor -- start 2019-10-13T04:04:47.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.528+0000 7faef94ed700 1 -- start start 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef40a9880 0x7faef40a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef40a6830 0x7faef414ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef40a3a70 con 0x7faef40a2270 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef40aa560 con 0x7faef40a6830 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef4064f70 con 0x7faef40a9880 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef40a6830 0x7faef414ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:47.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef40a6830 0x7faef414ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42632/0 (socket says 172.21.15.110:42632) 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 -- 172.21.15.110:0/3576705069 learned_addr learned my addr 172.21.15.110:0/3576705069 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef37fe700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef40a9880 0x7faef40a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 -- 172.21.15.110:0/3576705069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef40a9880 msgr2=0x7faef40a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef40a9880 0x7faef40a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 -- 172.21.15.110:0/3576705069 >> v1:172.21.15.204:6789/0 conn(0x7faef40a2270 legacy=0x7faef40a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 -- 172.21.15.110:0/3576705069 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef40a9260 con 0x7faef40a6830 2019-10-13T04:04:47.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> v2:172.21.15.110:3300/0 conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faef407e7c0 tx=0x7faee4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> v2:172.21.15.110:3300/0 conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faef407e7c0 tx=0x7faee4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7faef407e7c0 tx=0x7faee4007710).ready entity=mon.0 client_cookie=3cc57a9f9ed613e3 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:47.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7faef407e7c0 tx=0x7faee4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7faef407e7c0 tx=0x7faee4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef27fc700 1 -- 172.21.15.110:0/3576705069 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeec010b50 con 0x7faef40a6830 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef3fff700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7faef407e7c0 tx=0x7faee4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef27fc700 1 -- 172.21.15.110:0/3576705069 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec010770 con 0x7faef40a6830 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef27fc700 1 -- 172.21.15.110:0/3576705069 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeec0112b0 con 0x7faef40a6830 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 msgr2=0x7faef414ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7faef407e7c0 tx=0x7faee4007710).stop 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- 172.21.15.110:0/3576705069 shutdown_connections 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef40a6830 0x7faef414ec60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 --2- 172.21.15.110:0/3576705069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef40a9880 0x7faef40a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- 172.21.15.110:0/3576705069 shutdown_connections 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- 172.21.15.110:0/3576705069 wait complete. 2019-10-13T04:04:47.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.532+0000 7faef94ed700 1 -- 172.21.15.110:0/3576705069 >> 172.21.15.110:0/3576705069 conn(0x7faef409c4f0 msgr2=0x7faef409c910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:47.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 Processor -- start 2019-10-13T04:04:47.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 -- start start 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef405ec20 0x7faef409f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef407d2f0 0x7faef407d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef40a06f0 con 0x7faef4124100 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef409ef10 con 0x7faef405ec20 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef94ed700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef409f1b0 con 0x7faef407d2f0 2019-10-13T04:04:47.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef405ec20 0x7faef409f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faef405ec20 0x7faef409f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42638/0 (socket says 172.21.15.110:42638) 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 -- 172.21.15.110:0/3702849623 learned_addr learned my addr 172.21.15.110:0/3702849623 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef3fff700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef407d2f0 0x7faef407d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 -- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef407d2f0 msgr2=0x7faef407d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef407d2f0 0x7faef407d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 -- 172.21.15.110:0/3702849623 >> v1:172.21.15.204:6789/0 conn(0x7faef4124100 legacy=0x7faef409d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.536+0000 7faef2ffd700 1 -- 172.21.15.110:0/3702849623 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef4064520 con 0x7faef405ec20 2019-10-13T04:04:47.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> v2:172.21.15.110:3300/0 conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> v2:172.21.15.110:3300/0 conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee800c9a0 con 0x7faef405ec20 2019-10-13T04:04:47.548 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee800c5c0 con 0x7faef405ec20 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef40592f0 con 0x7faef405ec20 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee8010d70 con 0x7faef405ec20 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef4059870 con 0x7faef405ec20 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7faee8007420 con 0x7faef405ec20 2019-10-13T04:04:47.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 0x7faed4029d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:47.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5915+0+0 (secure 0 0 0) 0x7faef40a3a70 con 0x7faef405ec20 2019-10-13T04:04:47.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef37fe700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 0x7faed4029d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:47.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.540+0000 7faef37fe700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 0x7faed4029d60 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:47.553 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.544+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef40555b0 con 0x7faef405ec20 2019-10-13T04:04:47.554 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.548+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:47.555 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.552+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7faee8025f10 con 0x7faef405ec20 2019-10-13T04:04:47.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:47.708+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"} v 0) v1 -- 0x7faef409ece0 con 0x7faef405ec20 2019-10-13T04:04:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.415+0000 7faef2ffd700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.418 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.415+0000 7faedbfff700 1 -- 172.21.15.110:0/3702849623 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"}]=0 set pool 3 allow_ec_overwrites to true v22) v1 ==== 166+0+0 (secure 0 0 0) 0x7faee8007710 con 0x7faef405ec20 2019-10-13T04:04:48.423 INFO:teuthology.orchestra.run.smithi110.stderr:set pool 3 allow_ec_overwrites to true 2019-10-13T04:04:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 msgr2=0x7faed4029d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 0x7faed4029d60 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 msgr2=0x7faef409f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7faef409f0d0 tx=0x7faee4002ff0).stop 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 shutdown_connections 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef407d2f0 0x7faef407d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7faed4027860 0x7faed4029d60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 --2- 172.21.15.110:0/3702849623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faef405ec20 0x7faef409f6f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 shutdown_connections 2019-10-13T04:04:48.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 wait complete. 2019-10-13T04:04:48.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.423+0000 7faef94ed700 1 -- 172.21.15.110:0/3702849623 >> 172.21.15.110:0/3702849623 conn(0x7faef4059b80 msgr2=0x7faef40a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:48.439 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:48.439 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2019-10-13T04:04:48.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.515+0000 7f8d559fe700 1 Processor -- start 2019-10-13T04:04:48.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.515+0000 7f8d559fe700 1 -- start start 2019-10-13T04:04:48.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8d5014d900 0x7f8d5014fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d500a0fc0 con 0x7f8d500a88e0 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d500a75f0 con 0x7f8d500aa2b0 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d500a1a90 con 0x7f8d5014d900 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8d500a88e0 0x7f8d500a1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45356/0 (socket says 172.21.15.110:45356) 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 -- 172.21.15.110:0/1690616645 learned_addr learned my addr 172.21.15.110:0/1690616645 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:48.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4ffff700 1 --2- 172.21.15.110:0/1690616645 >> v2:172.21.15.204:3300/0 conn(0x7f8d5014d900 0x7f8d5014fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758556058 0 0) 0x7f8d500a0fc0 con 0x7f8d500a88e0 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d380030a0 con 0x7f8d500a88e0 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3133610976 0 0) 0x7f8d380030a0 con 0x7f8d500a88e0 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 >> v2:172.21.15.204:3300/0 conn(0x7f8d5014d900 msgr2=0x7f8d5014fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 --2- 172.21.15.110:0/1690616645 >> v2:172.21.15.204:3300/0 conn(0x7f8d5014d900 0x7f8d5014fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 >> v1:172.21.15.204:6789/0 conn(0x7f8d500aa2b0 legacy=0x7f8d5007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:48.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50059a20 con 0x7f8d500a88e0 2019-10-13T04:04:48.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f8d44001750 con 0x7f8d500a88e0 2019-10-13T04:04:48.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 >> v1:172.21.15.110:6789/0 conn(0x7f8d500a88e0 legacy=0x7f8d500a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d380036d0 0x7f8d38003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d38004d80 0x7f8d3800b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d500a1a90 con 0x7f8d38004410 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3804fbf0 con 0x7f8d38004d80 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3804fe30 con 0x7f8d380036d0 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4ffff700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d38004d80 0x7f8d3800b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4f7fe700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d380036d0 0x7f8d38003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.529 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 -- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d380036d0 msgr2=0x7f8d38003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:48.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d380036d0 0x7f8d38003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:48.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 -- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d38004d80 msgr2=0x7f8d3800b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:48.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d38004d80 0x7f8d3800b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:48.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 -- 172.21.15.110:0/1690616645 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d38050130 con 0x7f8d38004410 2019-10-13T04:04:48.530 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4ffff700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d38004d80 0x7f8d3800b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4ffff700 1 -- 172.21.15.110:0/1690616645 reap_dead start 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4f7fe700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d380036d0 0x7f8d38003af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8d44001a50 tx=0x7f8d48007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8d44001a50 tx=0x7f8d48007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f8d44001a50 tx=0x7f8d48007c30).ready entity=mon.0 client_cookie=bee7a608b8ef6a9d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.531 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f8d44001a50 tx=0x7f8d48007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4400b070 con 0x7f8d38004410 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f8d44001a50 tx=0x7f8d48007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4effd700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f8d44001a50 tx=0x7f8d48007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4400eb60 con 0x7f8d38004410 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d4e7fc700 1 -- 172.21.15.110:0/1690616645 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4400a6d0 con 0x7f8d38004410 2019-10-13T04:04:48.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 msgr2=0x7f8d38004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f8d44001a50 tx=0x7f8d48007c30).stop 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- 172.21.15.110:0/1690616645 shutdown_connections 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 --2- 172.21.15.110:0/1690616645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d38004410 0x7f8d38004830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- 172.21.15.110:0/1690616645 shutdown_connections 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- 172.21.15.110:0/1690616645 wait complete. 2019-10-13T04:04:48.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.519+0000 7f8d559fe700 1 -- 172.21.15.110:0/1690616645 >> 172.21.15.110:0/1690616645 conn(0x7f8d5005e980 msgr2=0x7f8d500a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 Processor -- start 2019-10-13T04:04:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- start start 2019-10-13T04:04:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5011f080 0x7f8d50126bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8d50127110 0x7f8d5011cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d50125890 con 0x7f8d5011d140 2019-10-13T04:04:48.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d50065530 con 0x7f8d50127110 2019-10-13T04:04:48.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50125ac0 con 0x7f8d5011f080 2019-10-13T04:04:48.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8d50127110 0x7f8d5011cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8d5011d140 0x7f8d50134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45370/0 (socket says 172.21.15.110:45370) 2019-10-13T04:04:48.538 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4ffff700 1 -- 172.21.15.110:0/2619670586 learned_addr learned my addr 172.21.15.110:0/2619670586 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4f7fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5011f080 0x7f8d50126bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 -- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5011f080 msgr2=0x7f8d50126bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5011f080 0x7f8d50126bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369170829 0 0) 0x7f8d50125890 con 0x7f8d5011d140 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 -- 172.21.15.110:0/2619670586 >> v1:172.21.15.110:6789/0 conn(0x7f8d5011d140 legacy=0x7f8d50134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.539 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 -- 172.21.15.110:0/2619670586 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50134810 con 0x7f8d50127110 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> v2:172.21.15.110:3300/0 conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> v2:172.21.15.110:3300/0 conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d44011690 con 0x7f8d50127110 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d50134ae0 con 0x7f8d50127110 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4400b270 con 0x7f8d50127110 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d44011c70 con 0x7f8d50127110 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.523+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5013b120 con 0x7f8d50127110 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f8d4400bd40 con 0x7f8d50127110 2019-10-13T04:04:48.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d37fff700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 0x7f8d30029d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f8d44034d80 con 0x7f8d50127110 2019-10-13T04:04:48.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d4f7fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 0x7f8d30029d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.527+0000 7f8d4f7fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 0x7f8d30029d10 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.535+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d500555c0 con 0x7f8d50127110 2019-10-13T04:04:48.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.539+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.539+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f8d4401d670 con 0x7f8d50127110 2019-10-13T04:04:48.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.707+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f8d500a79c0 con 0x7f8d50127110 2019-10-13T04:04:48.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:04:48.711+0000 7fe525a0c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-10-13T04:04:48.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.715+0000 7f8d4effd700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.715+0000 7f8d37fff700 1 -- 172.21.15.110:0/2619670586 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8d4400c030 con 0x7f8d50127110 2019-10-13T04:04:48.727 INFO:teuthology.orchestra.run.smithi110.stderr:new fs with metadata pool 2 and data pool 3 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 msgr2=0x7f8d30029d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 0x7f8d30029d10 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 msgr2=0x7f8d5011cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f8d5012e280 tx=0x7f8d400033b0).stop 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 shutdown_connections 2019-10-13T04:04:48.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5011f080 0x7f8d50126bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d30027810 0x7f8d30029d10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 --2- 172.21.15.110:0/2619670586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d50127110 0x7f8d5011cbf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 shutdown_connections 2019-10-13T04:04:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 wait complete. 2019-10-13T04:04:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.727+0000 7f8d559fe700 1 -- 172.21.15.110:0/2619670586 >> 172.21.15.110:0/2619670586 conn(0x7f8d500a6db0 msgr2=0x7f8d50124650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:48.744 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:48.744 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:48.839 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 Processor -- start 2019-10-13T04:04:48.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 -- start start 2019-10-13T04:04:48.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa3f80a6410 0x7fa3f80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a8c10 0x7fa3f80a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:48.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f80a77f0 con 0x7fa3f8065060 2019-10-13T04:04:48.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f80a20e0 con 0x7fa3f80a8c10 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa40076f700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f80a2380 con 0x7fa3f80a6410 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa3ff50d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a8c10 0x7fa3f80a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa3ff50d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a8c10 0x7fa3f80a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42666/0 (socket says 172.21.15.110:42666) 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa3ff50d700 1 -- 172.21.15.110:0/1213874230 learned_addr learned my addr 172.21.15.110:0/1213874230 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.831+0000 7fa3fed0c700 1 --2- 172.21.15.110:0/1213874230 >> v2:172.21.15.204:3300/0 conn(0x7fa3f80a6410 0x7fa3f80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 -- 172.21.15.110:0/1213874230 >> v2:172.21.15.204:3300/0 conn(0x7fa3f80a6410 msgr2=0x7fa3f80a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:48.841 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> v2:172.21.15.204:3300/0 conn(0x7fa3f80a6410 0x7fa3f80a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 -- 172.21.15.110:0/1213874230 >> v1:172.21.15.204:6789/0 conn(0x7fa3f8065060 legacy=0x7fa3f8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 -- 172.21.15.110:0/1213874230 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f814f9f0 con 0x7fa3f80a8c10 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).ready entity=mon.0 client_cookie=551890b7ab29196d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3fdd0a700 1 -- 172.21.15.110:0/1213874230 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3f0008c80 con 0x7fa3f80a8c10 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3fdd0a700 1 -- 172.21.15.110:0/1213874230 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f00088a0 con 0x7fa3f80a8c10 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa3fdd0a700 1 -- 172.21.15.110:0/1213874230 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3f00118c0 con 0x7fa3f80a8c10 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 msgr2=0x7fa3f80a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7fa3f80a15e0 tx=0x7fa3f4007710).stop 2019-10-13T04:04:48.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- 172.21.15.110:0/1213874230 shutdown_connections 2019-10-13T04:04:48.844 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 --2- 172.21.15.110:0/1213874230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a8c10 0x7fa3f80a9030 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.844 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 --2- 172.21.15.110:0/1213874230 >> v2:172.21.15.204:3300/0 conn(0x7fa3f80a6410 0x7fa3f80a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:48.844 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- 172.21.15.110:0/1213874230 shutdown_connections 2019-10-13T04:04:48.844 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- 172.21.15.110:0/1213874230 wait complete. 2019-10-13T04:04:48.844 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- 172.21.15.110:0/1213874230 >> 172.21.15.110:0/1213874230 conn(0x7fa3f805afb0 msgr2=0x7fa3f809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:48.847 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 Processor -- start 2019-10-13T04:04:48.847 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.835+0000 7fa40076f700 1 -- start start 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3f809e6e0 0x7fa3f80a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a3260 0x7fa3f809eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa3f814ee20 0x7fa3f814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f809e1a0 con 0x7fa3f80a3260 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f809e440 con 0x7fa3f814ee20 2019-10-13T04:04:48.848 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f809de60 con 0x7fa3f809e6e0 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a3260 0x7fa3f809eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a3260 0x7fa3f809eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42670/0 (socket says 172.21.15.110:42670) 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 -- 172.21.15.110:0/755354448 learned_addr learned my addr 172.21.15.110:0/755354448 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3ff50d700 1 --2- 172.21.15.110:0/755354448 >> v2:172.21.15.204:3300/0 conn(0x7fa3f814ee20 0x7fa3f814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fed0c700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3f809e6e0 0x7fa3f80a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.849 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 -- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3f809e6e0 msgr2=0x7fa3f80a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3f809e6e0 0x7fa3f80a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 -- 172.21.15.110:0/755354448 >> v2:172.21.15.204:3300/0 conn(0x7fa3f814ee20 msgr2=0x7fa3f814f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> v2:172.21.15.204:3300/0 conn(0x7fa3f814ee20 0x7fa3f814f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 -- 172.21.15.110:0/755354448 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80757e0 con 0x7fa3f80a3260 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> v2:172.21.15.110:3300/0 conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3ec00c940 con 0x7fa3f80a3260 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec0101c0 con 0x7fa3f80a3260 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f80a4320 con 0x7fa3f80a3260 2019-10-13T04:04:48.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3ec0021f0 con 0x7fa3f80a3260 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8062c50 con 0x7fa3f80a3260 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.839+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa3ec0066d0 con 0x7fa3f80a3260 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3eb7fe700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 0x7fa3e0029db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:48.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa3ec034550 con 0x7fa3f80a3260 2019-10-13T04:04:48.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3fed0c700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 0x7fa3e0029db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:48.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.843+0000 7fa3fed0c700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 0x7fa3e0029db0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:48.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.847+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f80555c0 con 0x7fa3f80a3260 2019-10-13T04:04:48.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.855+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:48.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:48.855+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa3ec003070 con 0x7fa3f80a3260 2019-10-13T04:04:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.023+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f80a2730 con 0x7fa3f80a3260 2019-10-13T04:04:49.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.023+0000 7fa3fe50b700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa3eb7fe700 1 -- 172.21.15.110:0/755354448 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7fa3ec0258e0 con 0x7fa3f80a3260 2019-10-13T04:04:49.030 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:49.030 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 msgr2=0x7fa3e0029db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 0x7fa3e0029db0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 msgr2=0x7fa3f809eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7fa3f807e340 tx=0x7fa3f4002ff0).stop 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 shutdown_connections 2019-10-13T04:04:49.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> v2:172.21.15.204:3300/0 conn(0x7fa3f814ee20 0x7fa3f814f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3f809e6e0 0x7fa3f80a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa3e00278b0 0x7fa3e0029db0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 --2- 172.21.15.110:0/755354448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3f80a3260 0x7fa3f809eb40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 shutdown_connections 2019-10-13T04:04:49.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 wait complete. 2019-10-13T04:04:49.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.027+0000 7fa40076f700 1 -- 172.21.15.110:0/755354448 >> 172.21.15.110:0/755354448 conn(0x7fa3f80a0030 msgr2=0x7fa3f8059300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:49.046 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:49.046 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:49.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 Processor -- start 2019-10-13T04:04:49.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 -- start start 2019-10-13T04:04:49.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6f8065060 0x7fd6f80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:49.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6f80a8c10 0x7fd6f80a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f80a77f0 con 0x7fd6f80a6410 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80a20e0 con 0x7fd6f8065060 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fe4d9700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80a2380 con 0x7fd6f80a8c10 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6f8065060 0x7fd6f80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6f8065060 0x7fd6f80a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42678/0 (socket says 172.21.15.110:42678) 2019-10-13T04:04:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 -- 172.21.15.110:0/384788400 learned_addr learned my addr 172.21.15.110:0/384788400 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6fd277700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6f80a8c10 0x7fd6f80a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 -- 172.21.15.110:0/384788400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6f80a8c10 msgr2=0x7fd6f80a9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6f80a8c10 0x7fd6f80a9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 -- 172.21.15.110:0/384788400 >> v1:172.21.15.204:6789/0 conn(0x7fd6f80a6410 legacy=0x7fd6f8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 -- 172.21.15.110:0/384788400 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f814f9f0 con 0x7fd6f8065060 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> v2:172.21.15.110:3300/0 conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> v2:172.21.15.110:3300/0 conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).ready entity=mon.0 client_cookie=3e2f98d854f9bb2b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f77fe700 1 -- 172.21.15.110:0/384788400 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6ec011410 con 0x7fd6f8065060 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f77fe700 1 -- 172.21.15.110:0/384788400 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec008c90 con 0x7fd6f8065060 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.131+0000 7fd6f77fe700 1 -- 172.21.15.110:0/384788400 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6ec0119f0 con 0x7fd6f8065060 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 msgr2=0x7fd6f80a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fd6f80a22a0 tx=0x7fd6e8002ff0).stop 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/384788400 shutdown_connections 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6f80a8c10 0x7fd6f80a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/384788400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f8065060 0x7fd6f80a1970 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.142 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/384788400 shutdown_connections 2019-10-13T04:04:49.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/384788400 wait complete. 2019-10-13T04:04:49.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/384788400 >> 172.21.15.110:0/384788400 conn(0x7fd6f805afb0 msgr2=0x7fd6f809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:49.146 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 Processor -- start 2019-10-13T04:04:49.146 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- start start 2019-10-13T04:04:49.146 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6f809d720 0x7fd6f814dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:49.146 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f80a02f0 con 0x7fd6f814e2d0 2019-10-13T04:04:49.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f809e8e0 con 0x7fd6f80a52b0 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fe4d9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f809fb50 con 0x7fd6f809d720 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6f7fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6f809d720 0x7fd6f814dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fd277700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd6f814e2d0 0x7fd6f812f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45396/0 (socket says 172.21.15.110:45396) 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.135+0000 7fd6fd277700 1 -- 172.21.15.110:0/961429902 learned_addr learned my addr 172.21.15.110:0/961429902 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739995037 0 0) 0x7fd6f80a02f0 con 0x7fd6f814e2d0 2019-10-13T04:04:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d80030a0 con 0x7fd6f814e2d0 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 -- 172.21.15.110:0/961429902 >> v1:172.21.15.204:6789/0 conn(0x7fd6f80a52b0 legacy=0x7fd6f809e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 -- 172.21.15.110:0/961429902 >> v1:172.21.15.110:6789/0 conn(0x7fd6f814e2d0 legacy=0x7fd6f812f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449515931 0 0) 0x7fd6f809e8e0 con 0x7fd6f80a52b0 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 -- 172.21.15.110:0/961429902 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f80a3dd0 con 0x7fd6f809d720 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> v2:172.21.15.110:3300/0 conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> v2:172.21.15.110:3300/0 conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6ec007340 con 0x7fd6f809d720 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8079fe0 con 0x7fd6f809d720 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec011610 con 0x7fd6f809d720 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f814e6f0 con 0x7fd6f809d720 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6ec005910 con 0x7fd6f809d720 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd6ec005b60 con 0x7fd6f809d720 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 0x7fd6d802a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd6ec035400 con 0x7fd6f809d720 2019-10-13T04:04:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6fca76700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 0x7fd6d802a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.139+0000 7fd6fca76700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 0x7fd6d802a240 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.147+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f80555c0 con 0x7fd6f809d720 2019-10-13T04:04:49.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.155+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.155+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd6ec01df00 con 0x7fd6f809d720 2019-10-13T04:04:49.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.323+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6f8066520 con 0x7fd6f809d720 2019-10-13T04:04:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.327+0000 7fd6f7fff700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.327+0000 7fd6f57fa700 1 -- 172.21.15.110:0/961429902 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7fd6ec00fc30 con 0x7fd6f809d720 2019-10-13T04:04:49.332 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:49.332 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:49.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 msgr2=0x7fd6d802a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 0x7fd6d802a240 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 msgr2=0x7fd6f814dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fd6f809f7d0 tx=0x7fd6ec008f80).stop 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 shutdown_connections 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd6d8027d40 0x7fd6d802a240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 --2- 172.21.15.110:0/961429902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6f809d720 0x7fd6f814dd80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 shutdown_connections 2019-10-13T04:04:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 wait complete. 2019-10-13T04:04:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.331+0000 7fd6fe4d9700 1 -- 172.21.15.110:0/961429902 >> 172.21.15.110:0/961429902 conn(0x7fd6f80a85a0 msgr2=0x7fd6f80a8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:49.345 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:49.345 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2019-10-13T04:04:49.452 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 Processor -- start 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 -- start start 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55840aa2b0 0x7f558407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 0x7f55840a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 0x7f5584147510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5584059e10 con 0x7f55840aa2b0 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558405a0b0 con 0x7f558407ebf0 2019-10-13T04:04:49.453 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f558abd8700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55840a0fc0 con 0x7f55840a17e0 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55840aa2b0 0x7f558407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55840aa2b0 0x7f558407e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42692/0 (socket says 172.21.15.110:42692) 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 -- 172.21.15.110:0/2364776476 learned_addr learned my addr 172.21.15.110:0/2364776476 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5588974700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 0x7f55840a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589976700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 0x7f5584147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 -- 172.21.15.110:0/2364776476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 msgr2=0x7f5584147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 0x7f5584147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 -- 172.21.15.110:0/2364776476 >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 msgr2=0x7f55840a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.435+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 0x7f55840a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 -- 172.21.15.110:0/2364776476 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55840a75f0 con 0x7f55840aa2b0 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5588974700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 0x7f55840a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589976700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 0x7f5584147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:49.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.110:3300/0 conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5584067e60 tx=0x7f5580007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.110:3300/0 conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5584067e60 tx=0x7f5580007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7f5584067e60 tx=0x7f5580007710).ready entity=mon.0 client_cookie=f65febaccd5775dd server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7f5584067e60 tx=0x7f5580007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5577fff700 1 -- 172.21.15.110:0/2364776476 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55780088a0 con 0x7f55840aa2b0 2019-10-13T04:04:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7f5584067e60 tx=0x7f5580007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5589175700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7f5584067e60 tx=0x7f5580007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5577fff700 1 -- 172.21.15.110:0/2364776476 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55780073c0 con 0x7f55840aa2b0 2019-10-13T04:04:49.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f5577fff700 1 -- 172.21.15.110:0/2364776476 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5578006560 con 0x7f55840aa2b0 2019-10-13T04:04:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 msgr2=0x7f558407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7f5584067e60 tx=0x7f5580007710).stop 2019-10-13T04:04:49.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- 172.21.15.110:0/2364776476 shutdown_connections 2019-10-13T04:04:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55840a17e0 0x7f5584147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 --2- 172.21.15.110:0/2364776476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55840aa2b0 0x7f558407e6a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 --2- 172.21.15.110:0/2364776476 >> v2:172.21.15.204:3300/0 conn(0x7f558407ebf0 0x7f55840a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- 172.21.15.110:0/2364776476 shutdown_connections 2019-10-13T04:04:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- 172.21.15.110:0/2364776476 wait complete. 2019-10-13T04:04:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- 172.21.15.110:0/2364776476 >> 172.21.15.110:0/2364776476 conn(0x7f558405e980 msgr2=0x7f55840a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:49.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 Processor -- start 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.439+0000 7f558abd8700 1 -- start start 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f558411f420 0x7f55841215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f558411d140 0x7f558411d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55840a2a90 con 0x7f55841271b0 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5584121d10 con 0x7f558411d140 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5584121fb0 con 0x7f558411f420 2019-10-13T04:04:49.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f558411d140 0x7f558411d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f558411d140 0x7f558411d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42696/0 (socket says 172.21.15.110:42696) 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5589976700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f55841271b0 0x7f55841275d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45412/0 (socket says 172.21.15.110:45412) 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 -- 172.21.15.110:0/3197046423 learned_addr learned my addr 172.21.15.110:0/3197046423 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5589175700 1 --2- 172.21.15.110:0/3197046423 >> v2:172.21.15.204:3300/0 conn(0x7f558411f420 0x7f55841215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 -- 172.21.15.110:0/3197046423 >> v2:172.21.15.204:3300/0 conn(0x7f558411f420 msgr2=0x7f55841215f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> v2:172.21.15.204:3300/0 conn(0x7f558411f420 0x7f55841215f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 -- 172.21.15.110:0/3197046423 >> v1:172.21.15.110:6789/0 conn(0x7f55841271b0 legacy=0x7f55841275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 -- 172.21.15.110:0/3197046423 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55840a1d90 con 0x7f558411d140 2019-10-13T04:04:49.465 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> v2:172.21.15.110:3300/0 conn(0x7f558411d140 0x7f558411d560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> v2:172.21.15.110:3300/0 conn(0x7f558411d140 0x7f558411d560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5580008ab0 con 0x7f558411d140 2019-10-13T04:04:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5580007160 con 0x7f558411d140 2019-10-13T04:04:49.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55840a21f0 con 0x7f558411d140 2019-10-13T04:04:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5580005b10 con 0x7f558411d140 2019-10-13T04:04:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558413b8a0 con 0x7f558411d140 2019-10-13T04:04:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f558000a180 con 0x7f558411d140 2019-10-13T04:04:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 0x7f5564029e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5580034930 con 0x7f558411d140 2019-10-13T04:04:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.443+0000 7f5589175700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 0x7f5564029e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:49.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.447+0000 7f5589175700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 0x7f5564029e80 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.451+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5584042510 con 0x7f558411d140 2019-10-13T04:04:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.459+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:49.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.459+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f55840a1d90 con 0x7f558411d140 2019-10-13T04:04:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:49.615+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f55840555c0 con 0x7f558411d140 2019-10-13T04:04:50.628 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.622+0000 7f5588974700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.629 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.622+0000 7f5575ffb700 1 -- 172.21.15.110:0/3197046423 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7f558000ac90 con 0x7f558411d140 2019-10-13T04:04:50.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 msgr2=0x7f5564029e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 0x7f5564029e80 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 msgr2=0x7f558411d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f55841352f0 tx=0x7f557c003370).stop 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 shutdown_connections 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 --2- 172.21.15.110:0/3197046423 >> v2:172.21.15.204:3300/0 conn(0x7f558411f420 0x7f55841215f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5564027980 0x7f5564029e80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 --2- 172.21.15.110:0/3197046423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558411d140 0x7f558411d560 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 shutdown_connections 2019-10-13T04:04:50.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 wait complete. 2019-10-13T04:04:50.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.634+0000 7f558abd8700 1 -- 172.21.15.110:0/3197046423 >> 172.21.15.110:0/3197046423 conn(0x7f558407ddc0 msgr2=0x7f55840a7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:50.651 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:50.651 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 Processor -- start 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 -- start start 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35b80a6410 0x7f35b80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b80a77f0 con 0x7f35b80a8c10 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b80a9030 con 0x7f35b8065060 2019-10-13T04:04:50.753 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bea06700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80a20e0 con 0x7f35b80a6410 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bd7a4700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35b80a8c10 0x7f35b814e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45420/0 (socket says 172.21.15.110:45420) 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bd7a4700 1 -- 172.21.15.110:0/1799548726 learned_addr learned my addr 172.21.15.110:0/1799548726 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.742+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> v2:172.21.15.110:3300/0 conn(0x7f35b80a6410 0x7f35b80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869254000 0 0) 0x7f35b80a77f0 con 0x7f35b80a8c10 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a00030a0 con 0x7f35b80a8c10 2019-10-13T04:04:50.754 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 -- 172.21.15.110:0/1799548726 >> v1:172.21.15.204:6789/0 conn(0x7f35b8065060 legacy=0x7f35b8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756653291 0 0) 0x7f35b80a9030 con 0x7f35b8065060 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 -- 172.21.15.110:0/1799548726 >> v1:172.21.15.110:6789/0 conn(0x7f35b80a8c10 legacy=0x7f35b814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 -- 172.21.15.110:0/1799548726 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80a9260 con 0x7f35b80a6410 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> v2:172.21.15.110:3300/0 conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35a8008780 tx=0x7f35a800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> v2:172.21.15.110:3300/0 conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35a8008780 tx=0x7f35a800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.755 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f35a8008780 tx=0x7f35a800fbe0).ready entity=mon.0 client_cookie=3eb2e672aba439db server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f35a8008780 tx=0x7f35a800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a8011260 con 0x7f35b80a6410 2019-10-13T04:04:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f35a8008780 tx=0x7f35a800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.756 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f35a8008780 tx=0x7f35a800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8008bb0 con 0x7f35b80a6410 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35b77fe700 1 -- 172.21.15.110:0/1799548726 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a8011840 con 0x7f35b80a6410 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 -- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 msgr2=0x7f35b80a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f35a8008780 tx=0x7f35a800fbe0).stop 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 -- 172.21.15.110:0/1799548726 shutdown_connections 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 --2- 172.21.15.110:0/1799548726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b80a6410 0x7f35b80a1970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 -- 172.21.15.110:0/1799548726 shutdown_connections 2019-10-13T04:04:50.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 -- 172.21.15.110:0/1799548726 wait complete. 2019-10-13T04:04:50.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.746+0000 7f35bea06700 1 -- 172.21.15.110:0/1799548726 >> 172.21.15.110:0/1799548726 conn(0x7f35b805afb0 msgr2=0x7f35b809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:50.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 Processor -- start 2019-10-13T04:04:50.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 -- start start 2019-10-13T04:04:50.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35b813c670 0x7f35b813eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:50.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b811f660 con 0x7f35b807bb60 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b81232f0 con 0x7f35b807a790 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bea06700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b81225b0 con 0x7f35b813c670 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35b813c670 0x7f35b813eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35b813c670 0x7f35b813eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42714/0 (socket says 172.21.15.110:42714) 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 -- 172.21.15.110:0/3014381842 learned_addr learned my addr 172.21.15.110:0/3014381842 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:50.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35b7fff700 1 --1- 172.21.15.110:0/3014381842 >> v1:172.21.15.110:6789/0 conn(0x7f35b807bb60 0x7f35b811ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45424/0 (socket says 172.21.15.110:45424) 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 -- 172.21.15.110:0/3014381842 >> v1:172.21.15.204:6789/0 conn(0x7f35b807a790 legacy=0x7f35b8064140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 -- 172.21.15.110:0/3014381842 >> v1:172.21.15.110:6789/0 conn(0x7f35b807bb60 legacy=0x7f35b811ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519455361 0 0) 0x7f35b811f660 con 0x7f35b807bb60 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 -- 172.21.15.110:0/3014381842 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80a6920 con 0x7f35b813c670 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> v2:172.21.15.110:3300/0 conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> v2:172.21.15.110:3300/0 conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.750+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35b00121a0 con 0x7f35b813c670 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b8122d20 con 0x7f35b813c670 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0005bc0 con 0x7f35b813c670 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.764 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35b0012780 con 0x7f35b813c670 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b813eff0 con 0x7f35b813c670 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f35b0012a60 con 0x7f35b813c670 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 0x7f3598029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f35b0034f70 con 0x7f35b813c670 2019-10-13T04:04:50.765 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 0x7f3598029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:50.766 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.754+0000 7f35bcfa3700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 0x7f3598029c90 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:50.766 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.758+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8042510 con 0x7f35b813c670 2019-10-13T04:04:50.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.762+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.767 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.762+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f35b000a320 con 0x7f35b813c670 2019-10-13T04:04:50.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.918+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35b80555c0 con 0x7f35b813c670 2019-10-13T04:04:50.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.918+0000 7f35bd7a4700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:50.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.918+0000 7f35b57fa700 1 -- 172.21.15.110:0/3014381842 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7f35b0026300 con 0x7f35b813c670 2019-10-13T04:04:50.924 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 3 2019-10-13T04:04:50.924 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:50.925 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T04:04:48.713885+0000","modified":"2019-10-13T04:04:50.613987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T04:04:50.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 msgr2=0x7f3598029c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 0x7f3598029c90 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 msgr2=0x7f35b813eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f35b0004710 tx=0x7f35b0010ae0).stop 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 shutdown_connections 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3598027790 0x7f3598029c90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 --2- 172.21.15.110:0/3014381842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35b813c670 0x7f35b813eaa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 shutdown_connections 2019-10-13T04:04:50.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 wait complete. 2019-10-13T04:04:50.928 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:50.922+0000 7f35bea06700 1 -- 172.21.15.110:0/3014381842 >> 172.21.15.110:0/3014381842 conn(0x7f35b80a2750 msgr2=0x7f35b807dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:50.936 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:50.936 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.030+0000 7f3df4922700 1 Processor -- start 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.030+0000 7f3df4922700 1 -- start start 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3df0065060 0x7f3df00a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00a77f0 con 0x7f3df00a8c10 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00a9030 con 0x7f3df00a6410 2019-10-13T04:04:51.042 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df00a20e0 con 0x7f3df0065060 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3df0065060 0x7f3df00a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3df0065060 0x7f3df00a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42720/0 (socket says 172.21.15.110:42720) 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 -- 172.21.15.110:0/3860678450 learned_addr learned my addr 172.21.15.110:0/3860678450 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dedd9b700 1 -- 172.21.15.110:0/3860678450 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235920401 0 0) 0x7f3df00a9030 con 0x7f3df00a6410 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dedd9b700 1 -- 172.21.15.110:0/3860678450 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ddc0030a0 con 0x7f3df00a6410 2019-10-13T04:04:51.043 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 -- 172.21.15.110:0/3860678450 >> v1:172.21.15.204:6789/0 conn(0x7f3df00a6410 legacy=0x7f3df0144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 -- 172.21.15.110:0/3860678450 >> v1:172.21.15.110:6789/0 conn(0x7f3df00a8c10 legacy=0x7f3df014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 -- 172.21.15.110:0/3860678450 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00a9260 con 0x7f3df0065060 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> v2:172.21.15.110:3300/0 conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> v2:172.21.15.110:3300/0 conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).ready entity=mon.0 client_cookie=75f83ca0a0c96925 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.044 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dedd9b700 1 -- 172.21.15.110:0/3860678450 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd8011510 con 0x7f3df0065060 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dee59c700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dedd9b700 1 -- 172.21.15.110:0/3860678450 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd8008c90 con 0x7f3df0065060 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3dedd9b700 1 -- 172.21.15.110:0/3860678450 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd8011af0 con 0x7f3df0065060 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 msgr2=0x7f3df00a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.045 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f3dd8008780 tx=0x7f3dd800fca0).stop 2019-10-13T04:04:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- 172.21.15.110:0/3860678450 shutdown_connections 2019-10-13T04:04:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 --2- 172.21.15.110:0/3860678450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df0065060 0x7f3df00a1970 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.046 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- 172.21.15.110:0/3860678450 shutdown_connections 2019-10-13T04:04:51.047 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- 172.21.15.110:0/3860678450 wait complete. 2019-10-13T04:04:51.047 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.034+0000 7f3df4922700 1 -- 172.21.15.110:0/3860678450 >> 172.21.15.110:0/3860678450 conn(0x7f3df005afb0 msgr2=0x7f3df009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 Processor -- start 2019-10-13T04:04:51.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 -- start start 2019-10-13T04:04:51.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3df0119140 0x7f3df01169f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0079110 con 0x7f3df011a2f0 2019-10-13T04:04:51.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0117040 con 0x7f3df007c010 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3df4922700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df01196a0 con 0x7f3df0119140 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3dee59c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3df011a2f0 0x7f3df01161f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45436/0 (socket says 172.21.15.110:45436) 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3dee59c700 1 -- 172.21.15.110:0/1700834835 learned_addr learned my addr 172.21.15.110:0/1700834835 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> v2:172.21.15.204:3300/0 conn(0x7f3df0119140 0x7f3df01169f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074691520 0 0) 0x7f3dd8008460 con 0x7f3df011a2f0 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd40030a0 con 0x7f3df011a2f0 2019-10-13T04:04:51.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1060840943 0 0) 0x7f3dd40030a0 con 0x7f3df011a2f0 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3def59e700 1 -- 172.21.15.110:0/1700834835 >> v1:172.21.15.204:6789/0 conn(0x7f3df007c010 legacy=0x7f3df0078910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3def59e700 1 -- 172.21.15.110:0/1700834835 >> v1:172.21.15.110:6789/0 conn(0x7f3df011a2f0 legacy=0x7f3df01161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3def59e700 1 -- 172.21.15.110:0/1700834835 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0134e10 con 0x7f3df0119140 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.038+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> v2:172.21.15.204:3300/0 conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> v2:172.21.15.204:3300/0 conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd8011e00 con 0x7f3df0119140 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd8011710 con 0x7f3df0119140 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df0132df0 con 0x7f3df0119140 2019-10-13T04:04:51.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3de0006d10 con 0x7f3df0119140 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df0133370 con 0x7f3df0119140 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f3de000a180 con 0x7f3df0119140 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 0x7f3dd402a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3de00346f0 con 0x7f3df0119140 2019-10-13T04:04:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3deed9d700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 0x7f3dd402a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.042+0000 7f3deed9d700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 0x7f3dd402a230 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.046+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df00555c0 con 0x7f3df0119140 2019-10-13T04:04:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.054+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.054+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f3de00248b0 con 0x7f3df0119140 2019-10-13T04:04:51.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.218+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3df0119860 con 0x7f3df0119140 2019-10-13T04:04:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.218+0000 7f3def59e700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.218+0000 7f3de77fe700 1 -- 172.21.15.110:0/1700834835 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7f3de000ac90 con 0x7f3df0119140 2019-10-13T04:04:51.225 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:51.226 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 msgr2=0x7f3dd402a230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 0x7f3dd402a230 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 msgr2=0x7f3df01169f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f3dd80081c0 tx=0x7f3dd800fca0).stop 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 shutdown_connections 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f3dd4027d30 0x7f3dd402a230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 --2- 172.21.15.110:0/1700834835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3df0119140 0x7f3df01169f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 shutdown_connections 2019-10-13T04:04:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 wait complete. 2019-10-13T04:04:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.222+0000 7f3df4922700 1 -- 172.21.15.110:0/1700834835 >> 172.21.15.110:0/1700834835 conn(0x7f3df009c500 msgr2=0x7f3df00a9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.239 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-10-13T04:04:51.239 INFO:tasks.ceph.mds.b:Restarting daemon 2019-10-13T04:04:51.240 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:51.240 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-10-13T04:04:51.243 INFO:tasks.ceph.mds.b:Started 2019-10-13T04:04:51.243 INFO:tasks.ceph.mds.a:Restarting daemon 2019-10-13T04:04:51.244 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:51.244 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-10-13T04:04:51.247 INFO:tasks.ceph.mds.a:Started 2019-10-13T04:04:51.248 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:51.248 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:51.249 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-10-13T04:04:51.299 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2019-10-13T04:04:51.300 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2019-10-13T04:04:51.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.338+0000 7f8568e81700 1 Processor -- start 2019-10-13T04:04:51.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.338+0000 7f8568e81700 1 -- start start 2019-10-13T04:04:51.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85640a6410 0x7f85640a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 0x7f85640a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640a77f0 con 0x7f8564065060 2019-10-13T04:04:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640a20e0 con 0x7f85640a6410 2019-10-13T04:04:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640a2380 con 0x7f85640a8c10 2019-10-13T04:04:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85640a6410 0x7f85640a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85640a6410 0x7f85640a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42746/0 (socket says 172.21.15.110:42746) 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 -- 172.21.15.110:0/3604998278 learned_addr learned my addr 172.21.15.110:0/3604998278 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 -- 172.21.15.110:0/3604998278 >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 msgr2=0x7f85640a9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f856359e700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 0x7f85640a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 0x7f85640a9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 -- 172.21.15.110:0/3604998278 >> v1:172.21.15.204:6789/0 conn(0x7f8564065060 legacy=0x7f8564144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:51.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 -- 172.21.15.110:0/3604998278 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856414f9f0 con 0x7f85640a6410 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f856359e700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 0x7f85640a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.110:3300/0 conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f85640a15e0 tx=0x7f855c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.110:3300/0 conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f85640a15e0 tx=0x7f855c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f85640a15e0 tx=0x7f855c003370).ready entity=mon.0 client_cookie=bd5c65610498c077 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f85640a15e0 tx=0x7f855c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8561d9b700 1 -- 172.21.15.110:0/3604998278 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85540111e0 con 0x7f85640a6410 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f85640a15e0 tx=0x7f855c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8562d9d700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f85640a15e0 tx=0x7f855c003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8561d9b700 1 -- 172.21.15.110:0/3604998278 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554008a80 con 0x7f85640a6410 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8561d9b700 1 -- 172.21.15.110:0/3604998278 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85540117c0 con 0x7f85640a6410 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 msgr2=0x7f85640a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f85640a15e0 tx=0x7f855c003370).stop 2019-10-13T04:04:51.359 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- 172.21.15.110:0/3604998278 shutdown_connections 2019-10-13T04:04:51.359 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 --2- 172.21.15.110:0/3604998278 >> v2:172.21.15.204:3300/0 conn(0x7f85640a8c10 0x7f85640a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.360 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 --2- 172.21.15.110:0/3604998278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85640a6410 0x7f85640a1970 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.360 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- 172.21.15.110:0/3604998278 shutdown_connections 2019-10-13T04:04:51.361 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- 172.21.15.110:0/3604998278 wait complete. 2019-10-13T04:04:51.361 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.342+0000 7f8568e81700 1 -- 172.21.15.110:0/3604998278 >> 172.21.15.110:0/3604998278 conn(0x7f856405afb0 msgr2=0x7f856409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.361 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 Processor -- start 2019-10-13T04:04:51.361 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 -- start start 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8564059640 0x7f85640a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856409e6f0 con 0x7f856409d560 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640a3dd0 con 0x7f856414ef60 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.346+0000 7f8568e81700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856409ea90 con 0x7f8564059640 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8564059640 0x7f85640a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f856259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f856409d560 0x7f85640a4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45466/0 (socket says 172.21.15.110:45466) 2019-10-13T04:04:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f856259c700 1 -- 172.21.15.110:0/105380086 learned_addr learned my addr 172.21.15.110:0/105380086 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453297002 0 0) 0x7f85640a3dd0 con 0x7f856414ef60 2019-10-13T04:04:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85480030a0 con 0x7f856414ef60 2019-10-13T04:04:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235794413 0 0) 0x7f856409e6f0 con 0x7f856409d560 2019-10-13T04:04:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 -- 172.21.15.110:0/105380086 >> v1:172.21.15.204:6789/0 conn(0x7f856414ef60 legacy=0x7f8564130420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 -- 172.21.15.110:0/105380086 >> v1:172.21.15.110:6789/0 conn(0x7f856409d560 legacy=0x7f85640a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 -- 172.21.15.110:0/105380086 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856409efd0 con 0x7f8564059640 2019-10-13T04:04:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> v2:172.21.15.110:3300/0 conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> v2:172.21.15.110:3300/0 conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8554007150 con 0x7f8564059640 2019-10-13T04:04:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85540113e0 con 0x7f8564059640 2019-10-13T04:04:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856414f380 con 0x7f8564059640 2019-10-13T04:04:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8554007350 con 0x7f8564059640 2019-10-13T04:04:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8564131c30 con 0x7f8564059640 2019-10-13T04:04:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f855400f910 con 0x7f8564059640 2019-10-13T04:04:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 0x7f854802a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8554035090 con 0x7f8564059640 2019-10-13T04:04:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f856259c700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 0x7f854802a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.350+0000 7f856259c700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 0x7f854802a2b0 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.354+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85640555c0 con 0x7f8564059640 2019-10-13T04:04:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.362+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.362+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f855401db00 con 0x7f8564059640 2019-10-13T04:04:51.527 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.522+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8564066520 con 0x7f8564059640 2019-10-13T04:04:51.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.522+0000 7f8562d9d700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.528 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.522+0000 7f85537fe700 1 -- 172.21.15.110:0/105380086 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7f855401dd60 con 0x7f8564059640 2019-10-13T04:04:51.530 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:51.530 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:51.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 msgr2=0x7f854802a2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 0x7f854802a2b0 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 msgr2=0x7f85640a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f856407d990 tx=0x7f8554011340).stop 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 shutdown_connections 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8548027db0 0x7f854802a2b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 --2- 172.21.15.110:0/105380086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8564059640 0x7f85640a5b50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 shutdown_connections 2019-10-13T04:04:51.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 wait complete. 2019-10-13T04:04:51.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.526+0000 7f8568e81700 1 -- 172.21.15.110:0/105380086 >> 172.21.15.110:0/105380086 conn(0x7f85640a7cf0 msgr2=0x7f85640a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.543 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:40.397189+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:43.569877+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:46.432662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T04:04:51.543 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:51.543 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T04:04:51.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 Processor -- start 2019-10-13T04:04:51.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 -- start start 2019-10-13T04:04:51.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f29d00aa2b0 0x7f29d007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29d007ebf0 0x7f29d00a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d00a0fc0 con 0x7f29d00a17e0 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0059e10 con 0x7f29d00aa2b0 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d8105700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d005a0b0 con 0x7f29d007ebf0 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f29d00aa2b0 0x7f29d007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f29d00aa2b0 0x7f29d007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42760/0 (socket says 172.21.15.110:42760) 2019-10-13T04:04:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 -- 172.21.15.110:0/1407655022 learned_addr learned my addr 172.21.15.110:0/1407655022 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d5ea1700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29d007ebf0 0x7f29d00a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 -- 172.21.15.110:0/1407655022 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29d007ebf0 msgr2=0x7f29d00a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29d007ebf0 0x7f29d00a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 -- 172.21.15.110:0/1407655022 >> v1:172.21.15.204:6789/0 conn(0x7f29d00a17e0 legacy=0x7f29d0147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 -- 172.21.15.110:0/1407655022 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d00a75f0 con 0x7f29d00aa2b0 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> v2:172.21.15.110:3300/0 conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> v2:172.21.15.110:3300/0 conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.634+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).ready entity=mon.0 client_cookie=e42c2f71111045d9 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d56a0700 1 -- 172.21.15.110:0/1407655022 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29c8011d70 con 0x7f29d00aa2b0 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d56a0700 1 -- 172.21.15.110:0/1407655022 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c8011990 con 0x7f29d00aa2b0 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d56a0700 1 -- 172.21.15.110:0/1407655022 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29c800a540 con 0x7f29d00aa2b0 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 msgr2=0x7f29d007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f29d005ccf0 tx=0x7f29c0002ff0).stop 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1407655022 shutdown_connections 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d00aa2b0 0x7f29d007e6a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 --2- 172.21.15.110:0/1407655022 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29d007ebf0 0x7f29d00a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1407655022 shutdown_connections 2019-10-13T04:04:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1407655022 wait complete. 2019-10-13T04:04:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1407655022 >> 172.21.15.110:0/1407655022 conn(0x7f29d005e980 msgr2=0x7f29d00a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 Processor -- start 2019-10-13T04:04:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- start start 2019-10-13T04:04:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f29d0064f00 0x7f29d0063290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f29d014ee50 0x7f29d012cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d0061b20 con 0x7f29d014ea30 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0061340 con 0x7f29d014ee50 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d00615e0 con 0x7f29d0064f00 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d5ea1700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f29d014ea30 0x7f29d005ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45480/0 (socket says 172.21.15.110:45480) 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d5ea1700 1 -- 172.21.15.110:0/1845315681 learned_addr learned my addr 172.21.15.110:0/1845315681 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.110:3300/0 conn(0x7f29d014ee50 0x7f29d012cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.204:3300/0 conn(0x7f29d0064f00 0x7f29d0063290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.653 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397507484 0 0) 0x7f29d0061b20 con 0x7f29d014ea30 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b80030a0 con 0x7f29d014ea30 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 -- 172.21.15.110:0/1845315681 >> v2:172.21.15.204:3300/0 conn(0x7f29d0064f00 msgr2=0x7f29d0063290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.204:3300/0 conn(0x7f29d0064f00 0x7f29d0063290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 -- 172.21.15.110:0/1845315681 >> v1:172.21.15.110:6789/0 conn(0x7f29d014ea30 legacy=0x7f29d005ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 -- 172.21.15.110:0/1845315681 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0060700 con 0x7f29d014ee50 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.110:3300/0 conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.654 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.110:3300/0 conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29cc003540 con 0x7f29d014ee50 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc003160 con 0x7f29d014ee50 2019-10-13T04:04:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d0060f10 con 0x7f29d014ee50 2019-10-13T04:04:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.638+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29cc003b20 con 0x7f29d014ee50 2019-10-13T04:04:51.664 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d012d0f0 con 0x7f29d014ee50 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f29cc009450 con 0x7f29d014ee50 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29c6ffd700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 0x7f29b802a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f29cc034480 con 0x7f29d014ee50 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 0x7f29b802a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d66a2700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 0x7f29b802a310 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.642+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d0042510 con 0x7f29d014ee50 2019-10-13T04:04:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.646+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.646+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f29cc025810 con 0x7f29d014ee50 2019-10-13T04:04:51.804 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.798+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f29d00555c0 con 0x7f29d014ee50 2019-10-13T04:04:51.804 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.798+0000 7f29d6ea3700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.805 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.798+0000 7f29c6ffd700 1 -- 172.21.15.110:0/1845315681 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f29cc025bb0 con 0x7f29d014ee50 2019-10-13T04:04:51.805 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:04:51.807 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 msgr2=0x7f29b802a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.807 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 0x7f29b802a310 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.807 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 msgr2=0x7f29d012cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f29d0061500 tx=0x7f29c0002f70).stop 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 shutdown_connections 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f29b8027e10 0x7f29b802a310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 --2- 172.21.15.110:0/1845315681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d014ee50 0x7f29d012cba0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 --2- 172.21.15.110:0/1845315681 >> v2:172.21.15.204:3300/0 conn(0x7f29d0064f00 0x7f29d0063290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 shutdown_connections 2019-10-13T04:04:51.808 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 wait complete. 2019-10-13T04:04:51.809 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.802+0000 7f29d8105700 1 -- 172.21.15.110:0/1845315681 >> 172.21.15.110:0/1845315681 conn(0x7f29d005e8c0 msgr2=0x7f29d007d880 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.817 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:51.817 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T04:04:51.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 Processor -- start 2019-10-13T04:04:51.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 -- start start 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fea000a88e0 0x7fea000a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea000a17e0 0x7fea0014e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea000a0fc0 con 0x7fea000aa2b0 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea00059e10 con 0x7fea000a17e0 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea0618d700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0005a0b0 con 0x7fea000a88e0 2019-10-13T04:04:51.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9fffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fea000aa2b0 0x7fea0007e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45486/0 (socket says 172.21.15.110:45486) 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9fffff700 1 -- 172.21.15.110:0/1581777650 learned_addr learned my addr 172.21.15.110:0/1581777650 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.110:3300/0 conn(0x7fea000a17e0 0x7fea0014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.204:3300/0 conn(0x7fea000a88e0 0x7fea000a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026388983 0 0) 0x7fea000a0fc0 con 0x7fea000aa2b0 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ec0030a0 con 0x7fea000aa2b0 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 -- 172.21.15.110:0/1581777650 >> v2:172.21.15.204:3300/0 conn(0x7fea000a88e0 msgr2=0x7fea000a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:51.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.204:3300/0 conn(0x7fea000a88e0 0x7fea000a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 -- 172.21.15.110:0/1581777650 >> v1:172.21.15.110:6789/0 conn(0x7fea000aa2b0 legacy=0x7fea0007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 198977673 0 0) 0x7fe9ec0030a0 con 0x7fea000aa2b0 2019-10-13T04:04:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 -- 172.21.15.110:0/1581777650 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea000a75f0 con 0x7fea000a17e0 2019-10-13T04:04:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.914+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.110:3300/0 conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.110:3300/0 conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).ready entity=mon.0 client_cookie=bf3174c893116e69 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f8001750 con 0x7fea000a17e0 2019-10-13T04:04:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea04f2b700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8001560 con 0x7fea000a17e0 2019-10-13T04:04:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9feffd700 1 -- 172.21.15.110:0/1581777650 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9f0005830 con 0x7fea000a17e0 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 msgr2=0x7fea0014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fe9f8001990 tx=0x7fe9f80014a0).stop 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- 172.21.15.110:0/1581777650 shutdown_connections 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 --2- 172.21.15.110:0/1581777650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a17e0 0x7fea0014e900 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 --2- 172.21.15.110:0/1581777650 >> v2:172.21.15.204:3300/0 conn(0x7fea000a88e0 0x7fea000a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- 172.21.15.110:0/1581777650 shutdown_connections 2019-10-13T04:04:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- 172.21.15.110:0/1581777650 wait complete. 2019-10-13T04:04:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- 172.21.15.110:0/1581777650 >> 172.21.15.110:0/1581777650 conn(0x7fea0005e980 msgr2=0x7fea000a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 Processor -- start 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- start start 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea000a4770 0x7fea0011bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 0x7fea0013a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea000a35f0 con 0x7fea0011ed00 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea000a3ea0 con 0x7fea000a4770 2019-10-13T04:04:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fea0618d700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea000a4ca0 con 0x7fea00079a70 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9ff7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea000a4770 0x7fea0011bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9ff7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fea000a4770 0x7fea0011bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42784/0 (socket says 172.21.15.110:42784) 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.918+0000 7fe9ff7fe700 1 -- 172.21.15.110:0/2819859241 learned_addr learned my addr 172.21.15.110:0/2819859241 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fea04f2b700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 0x7fea0013a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 -- 172.21.15.110:0/2819859241 >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 msgr2=0x7fea0013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 0x7fea0013a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 -- 172.21.15.110:0/2819859241 >> v1:172.21.15.204:6789/0 conn(0x7fea0011ed00 legacy=0x7fea000a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 -- 172.21.15.110:0/2819859241 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea000a4f40 con 0x7fea000a4770 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fea04f2b700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 0x7fea0013a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.110:3300/0 conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.110:3300/0 conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9e80058b0 con 0x7fea000a4770 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e8011ad0 con 0x7fea000a4770 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0011c0f0 con 0x7fea000a4770 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9e800c6b0 con 0x7fea000a4770 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea000a5f80 con 0x7fea000a4770 2019-10-13T04:04:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fe9e800c9c0 con 0x7fea000a4770 2019-10-13T04:04:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 0x7fe9e4029d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fe9e8034bd0 con 0x7fea000a4770 2019-10-13T04:04:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fffff700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 0x7fe9e4029d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.922+0000 7fe9fffff700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 0x7fe9e4029d90 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.930+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea000555c0 con 0x7fea000a4770 2019-10-13T04:04:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.934+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:51.934+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fe9e800e9b0 con 0x7fea000a4770 2019-10-13T04:04:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.082+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fea000a79c0 con 0x7fea000a4770 2019-10-13T04:04:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.082+0000 7fe9ff7fe700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.089 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.082+0000 7fe9fcff9700 1 -- 172.21.15.110:0/2819859241 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe9e800b130 con 0x7fea000a4770 2019-10-13T04:04:52.089 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:04:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 msgr2=0x7fe9e4029d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 0x7fe9e4029d90 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 msgr2=0x7fea0011bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fea000a4060 tx=0x7fe9e8007a90).stop 2019-10-13T04:04:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 shutdown_connections 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 --2- 172.21.15.110:0/2819859241 >> v2:172.21.15.204:3300/0 conn(0x7fea00079a70 0x7fea0013a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe9e4027890 0x7fe9e4029d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 --2- 172.21.15.110:0/2819859241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea000a4770 0x7fea0011bba0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 shutdown_connections 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 wait complete. 2019-10-13T04:04:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.086+0000 7fea0618d700 1 -- 172.21.15.110:0/2819859241 >> 172.21.15.110:0/2819859241 conn(0x7fea0007bc80 msgr2=0x7fea000a7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:52.101 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:52.101 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T04:04:52.206 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 Processor -- start 2019-10-13T04:04:52.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 -- start start 2019-10-13T04:04:52.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:52.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80a0fc0 con 0x7f8bc80aa2b0 2019-10-13T04:04:52.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80a75f0 con 0x7f8bc80a17e0 2019-10-13T04:04:52.207 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bd02f3700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8059e10 con 0x7f8bc80a88e0 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bce890700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8bc80aa2b0 0x7f8bc807e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45500/0 (socket says 172.21.15.110:45500) 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bce890700 1 -- 172.21.15.110:0/2598333809 learned_addr learned my addr 172.21.15.110:0/2598333809 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.190+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748708174 0 0) 0x7f8bc80a0fc0 con 0x7f8bc80aa2b0 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb40030a0 con 0x7f8bc80aa2b0 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600257838 0 0) 0x7f8bc80a75f0 con 0x7f8bc80a17e0 2019-10-13T04:04:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 -- 172.21.15.110:0/2598333809 >> v1:172.21.15.204:6789/0 conn(0x7f8bc80a17e0 legacy=0x7f8bc814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 -- 172.21.15.110:0/2598333809 >> v1:172.21.15.110:6789/0 conn(0x7f8bc80aa2b0 legacy=0x7f8bc807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 -- 172.21.15.110:0/2598333809 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8059a20 con 0x7f8bc80a88e0 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).ready entity=mon.2 client_cookie=eb880bce33619747 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bbc0112a0 con 0x7f8bc80a88e0 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc008bb0 con 0x7f8bc80a88e0 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bce08f700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 msgr2=0x7f8bc80a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bcd88e700 1 -- 172.21.15.110:0/2598333809 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bbc011880 con 0x7f8bc80a88e0 2019-10-13T04:04:52.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f8bbc002f30 tx=0x7f8bbc00fbe0).stop 2019-10-13T04:04:52.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- 172.21.15.110:0/2598333809 shutdown_connections 2019-10-13T04:04:52.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/2598333809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc80a88e0 0x7f8bc80a1290 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- 172.21.15.110:0/2598333809 shutdown_connections 2019-10-13T04:04:52.211 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- 172.21.15.110:0/2598333809 wait complete. 2019-10-13T04:04:52.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- 172.21.15.110:0/2598333809 >> 172.21.15.110:0/2598333809 conn(0x7f8bc805e980 msgr2=0x7f8bc80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:52.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 Processor -- start 2019-10-13T04:04:52.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.194+0000 7f8bd02f3700 1 -- start start 2019-10-13T04:04:52.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 0x7f8bc809c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.212 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8bc8123630 0x7f8bc8126920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc81123f0 con 0x7f8bc8126500 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc8111920 con 0x7f8bc8123630 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8114100 con 0x7f8bc8111fd0 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bce08f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8bc8126500 0x7f8bc8122e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45508/0 (socket says 172.21.15.110:45508) 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bce08f700 1 -- 172.21.15.110:0/1305743897 learned_addr learned my addr 172.21.15.110:0/1305743897 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:52.213 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bce890700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 0x7f8bc809c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> v2:172.21.15.204:3300/0 conn(0x7f8bc8123630 0x7f8bc8126920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096184820 0 0) 0x7f8bc81123f0 con 0x7f8bc8126500 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb00030a0 con 0x7f8bc8126500 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1338109074 0 0) 0x7f8bb00030a0 con 0x7f8bc8126500 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 -- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 msgr2=0x7f8bc809c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 0x7f8bc809c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.214 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 -- 172.21.15.110:0/1305743897 >> v1:172.21.15.110:6789/0 conn(0x7f8bc8126500 legacy=0x7f8bc8122e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 -- 172.21.15.110:0/1305743897 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc809cf10 con 0x7f8bc8123630 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bce890700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 0x7f8bc809c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> v2:172.21.15.204:3300/0 conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> v2:172.21.15.204:3300/0 conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.215 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bbc0112a0 con 0x7f8bc8123630 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc813a4e0 con 0x7f8bc8123630 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc0114a0 con 0x7f8bc8123630 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc813a9e0 con 0x7f8bc8123630 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc0002220 con 0x7f8bc8123630 2019-10-13T04:04:52.216 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.217 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f8bc00024c0 con 0x7f8bc8123630 2019-10-13T04:04:52.217 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.198+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.217 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.202+0000 7f8bbb7fe700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 0x7f8bb002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.217 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.202+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8bc0034430 con 0x7f8bc8123630 2019-10-13T04:04:52.217 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.202+0000 7f8bce890700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 0x7f8bb002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.219 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.202+0000 7f8bce890700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 0x7f8bb002a240 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.219 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.206+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8122a00 con 0x7f8bc8123630 2019-10-13T04:04:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.218+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.218+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f8bc00257c0 con 0x7f8bc8123630 2019-10-13T04:04:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.373+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8bc80555c0 con 0x7f8bc8123630 2019-10-13T04:04:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.377+0000 7f8bcf091700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.377+0000 7f8bbb7fe700 1 -- 172.21.15.110:0/1305743897 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8bc0025b60 con 0x7f8bc8123630 2019-10-13T04:04:52.390 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:04:52.390 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 msgr2=0x7f8bb002a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 0x7f8bb002a240 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 msgr2=0x7f8bc8126920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f8bbc007730 tx=0x7f8bbc00fff0).stop 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 shutdown_connections 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8bb0027d40 0x7f8bb002a240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.391 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bc8111fd0 0x7f8bc809c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 --2- 172.21.15.110:0/1305743897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bc8123630 0x7f8bc8126920 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 shutdown_connections 2019-10-13T04:04:52.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 wait complete. 2019-10-13T04:04:52.392 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.381+0000 7f8bd02f3700 1 -- 172.21.15.110:0/1305743897 >> 172.21.15.110:0/1305743897 conn(0x7f8bc80a7db0 msgr2=0x7f8bc80a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:52.402 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-10-13T04:04:52.402 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-10-13T04:04:52.402 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:52.402 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-10-13T04:04:52.494 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 Processor -- start 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- start start 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 0x7fee080a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08065060 0x7fee08144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 0x7fee080a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080a20e0 con 0x7fee08065060 2019-10-13T04:04:52.495 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080a2380 con 0x7fee08145010 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080a77f0 con 0x7fee080a6410 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08065060 0x7fee08144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08065060 0x7fee08144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42804/0 (socket says 172.21.15.110:42804) 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 -- 172.21.15.110:0/3011581723 learned_addr learned my addr 172.21.15.110:0/3011581723 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0d08f700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 0x7fee080a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0c88e700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 0x7fee080a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.496 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 -- 172.21.15.110:0/3011581723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 msgr2=0x7fee080a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:52.497 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 0x7fee080a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:52.497 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 -- 172.21.15.110:0/3011581723 >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 msgr2=0x7fee080a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:52.497 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 0x7fee080a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:52.497 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 -- 172.21.15.110:0/3011581723 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080a9160 con 0x7fee08065060 2019-10-13T04:04:52.497 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0d08f700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 0x7fee080a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0c88e700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 0x7fee080a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.110:3300/0 conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.110:3300/0 conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).ready entity=mon.0 client_cookie=21c58030d3b08575 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.498 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee077fe700 1 -- 172.21.15.110:0/3011581723 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee00011c30 con 0x7fee08065060 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee07fff700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee077fe700 1 -- 172.21.15.110:0/3011581723 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00011850 con 0x7fee08065060 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee077fe700 1 -- 172.21.15.110:0/3011581723 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000a540 con 0x7fee08065060 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 msgr2=0x7fee08144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.499 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7fee0805ccf0 tx=0x7fedf8002ff0).stop 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- 172.21.15.110:0/3011581723 shutdown_connections 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/3011581723 >> v2:172.21.15.204:3300/0 conn(0x7fee08145010 0x7fee080a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee080a6410 0x7fee080a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/3011581723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08065060 0x7fee08144ac0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- 172.21.15.110:0/3011581723 shutdown_connections 2019-10-13T04:04:52.500 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- 172.21.15.110:0/3011581723 wait complete. 2019-10-13T04:04:52.501 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.485+0000 7fee0e2f1700 1 -- 172.21.15.110:0/3011581723 >> 172.21.15.110:0/3011581723 conn(0x7fee0805afb0 msgr2=0x7fee0809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:52.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 Processor -- start 2019-10-13T04:04:52.506 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- start start 2019-10-13T04:04:52.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 0x7fee080a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08143bb0 0x7fee08143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.507 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080a4720 con 0x7fee08059740 2019-10-13T04:04:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080a5a10 con 0x7fee08143bb0 2019-10-13T04:04:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee08122d80 con 0x7fee08136400 2019-10-13T04:04:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08143bb0 0x7fee08143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee08143bb0 0x7fee08143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42810/0 (socket says 172.21.15.110:42810) 2019-10-13T04:04:52.508 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 -- 172.21.15.110:0/1696530287 learned_addr learned my addr 172.21.15.110:0/1696530287 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0c88e700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 0x7fee080a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 -- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 msgr2=0x7fee080a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 0x7fee080a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 -- 172.21.15.110:0/1696530287 >> v1:172.21.15.204:6789/0 conn(0x7fee08059740 legacy=0x7fee080a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 -- 172.21.15.110:0/1696530287 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee08144520 con 0x7fee08143bb0 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0c88e700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 0x7fee080a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:52.509 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> v2:172.21.15.110:3300/0 conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> v2:172.21.15.110:3300/0 conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedf800a160 con 0x7fee08143bb0 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.510 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8008b90 con 0x7fee08143bb0 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee081228e0 con 0x7fee08143bb0 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedf800a780 con 0x7fee08143bb0 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee08075050 con 0x7fee08143bb0 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.489+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fedf800a980 con 0x7fee08143bb0 2019-10-13T04:04:52.511 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.512 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee057fa700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 0x7fedec029dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:52.514 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fedf8034cc0 con 0x7fee08143bb0 2019-10-13T04:04:52.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee080555c0 con 0x7fee08143bb0 2019-10-13T04:04:52.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee0c88e700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 0x7fedec029dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:52.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.493+0000 7fee0c88e700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 0x7fedec029dd0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:52.515 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.497+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.516 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.497+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fee08144520 con 0x7fee08143bb0 2019-10-13T04:04:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.689+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fee081256a0 con 0x7fee08143bb0 2019-10-13T04:04:52.697 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.689+0000 7fee0d08f700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.689+0000 7fee057fa700 1 -- 172.21.15.110:0/1696530287 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+68498 (secure 0 0 0) 0x7fedf800c680 con 0x7fee08143bb0 2019-10-13T04:04:52.699 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:52.700 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 msgr2=0x7fedec029dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 0x7fedec029dd0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 msgr2=0x7fee08143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fee08122f40 tx=0x7fedf0007770).stop 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 shutdown_connections 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fedec0278d0 0x7fedec029dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee08143bb0 0x7fee08143fd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 --2- 172.21.15.110:0/1696530287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee08136400 0x7fee080a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 shutdown_connections 2019-10-13T04:04:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 wait complete. 2019-10-13T04:04:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:52.693+0000 7fee0e2f1700 1 -- 172.21.15.110:0/1696530287 >> 172.21.15.110:0/1696530287 conn(0x7fee080a7a90 msgr2=0x7fee0809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:52.711 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":13789},{"type":"v1","addr":"172.21.15.204:6801","nonce":13789}]},"active_addr":"172.21.15.204:6801/13789","active_change":"2019-10-13T04:04:29.230601+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-10-13T04:04:52.720 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-10-13T04:04:53.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T04:04:53.241+0000 7f2369b2e700 -1 mgr.server _prune_pending_service_map missing key mds.0 2019-10-13T04:04:53.526 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:53.526 INFO:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:53.630 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 Processor -- start 2019-10-13T04:04:53.630 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 -- start start 2019-10-13T04:04:53.630 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8d480a1970 0x7f8d4814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.630 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480a9410 con 0x7f8d480a6410 2019-10-13T04:04:53.631 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480a1d90 con 0x7f8d48065060 2019-10-13T04:04:53.631 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d4d26e700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d480a20e0 con 0x7f8d480a1970 2019-10-13T04:04:53.631 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d477fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8d480a6410 0x7f8d48144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45532/0 (socket says 172.21.15.110:45532) 2019-10-13T04:04:53.631 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d477fe700 1 -- 172.21.15.110:0/2132188993 learned_addr learned my addr 172.21.15.110:0/2132188993 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:53.632 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.613+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> v2:172.21.15.204:3300/0 conn(0x7f8d480a1970 0x7f8d4814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.632 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907515977 0 0) 0x7f8d480a9410 con 0x7f8d480a6410 2019-10-13T04:04:53.632 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d300030a0 con 0x7f8d480a6410 2019-10-13T04:04:53.632 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1237322090 0 0) 0x7f8d300030a0 con 0x7f8d480a6410 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 >> v2:172.21.15.204:3300/0 conn(0x7f8d480a1970 msgr2=0x7f8d4814f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 --2- 172.21.15.110:0/2132188993 >> v2:172.21.15.204:3300/0 conn(0x7f8d480a1970 0x7f8d4814f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 >> v1:172.21.15.204:6789/0 conn(0x7f8d48065060 legacy=0x7f8d480a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480a77f0 con 0x7f8d480a6410 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f8d38001ad0 con 0x7f8d480a6410 2019-10-13T04:04:53.633 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 >> v1:172.21.15.110:6789/0 conn(0x7f8d480a6410 legacy=0x7f8d48144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d300036d0 0x7f8d30003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d30004410 0x7f8d30004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480a20e0 con 0x7f8d30004d80 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3004fbf0 con 0x7f8d30004410 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3004fe30 con 0x7f8d300036d0 2019-10-13T04:04:53.634 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d46ffd700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d30004410 0x7f8d30004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d477fe700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d300036d0 0x7f8d30003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 -- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d300036d0 msgr2=0x7f8d30003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d300036d0 0x7f8d30003af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 -- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d30004410 msgr2=0x7f8d30004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d30004410 0x7f8d30004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:53.635 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 -- 172.21.15.110:0/2132188993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30050130 con 0x7f8d30004d80 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d477fe700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d300036d0 0x7f8d30003af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8d40002f30 tx=0x7f8d3c001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 -- 172.21.15.110:0/2132188993 reap_dead start 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8d40002f30 tx=0x7f8d3c001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f8d40002f30 tx=0x7f8d3c001310).ready entity=mon.0 client_cookie=b10808c7c8d3a498 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.636 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f8d40002f30 tx=0x7f8d3c001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d40008ba0 con 0x7f8d30004d80 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f8d40002f30 tx=0x7f8d3c001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d400087c0 con 0x7f8d30004d80 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d47fff700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f8d40002f30 tx=0x7f8d3c001310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d467fc700 1 -- 172.21.15.110:0/2132188993 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d40005910 con 0x7f8d30004d80 2019-10-13T04:04:53.637 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 -- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 msgr2=0x7f8d3000b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f8d40002f30 tx=0x7f8d3c001310).stop 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 -- 172.21.15.110:0/2132188993 shutdown_connections 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/2132188993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d30004d80 0x7f8d3000b660 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 -- 172.21.15.110:0/2132188993 shutdown_connections 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 -- 172.21.15.110:0/2132188993 wait complete. 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.617+0000 7f8d4d26e700 1 -- 172.21.15.110:0/2132188993 >> 172.21.15.110:0/2132188993 conn(0x7f8d4805afb0 msgr2=0x7f8d4809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:53.638 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 Processor -- start 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 -- start start 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8d48059e10 0x7f8d4807bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480a5ce0 con 0x7f8d480a5270 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d481219d0 con 0x7f8d4811bad0 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d4d26e700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4807c010 con 0x7f8d48059e10 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d47fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8d480a5270 0x7f8d4813a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45548/0 (socket says 172.21.15.110:45548) 2019-10-13T04:04:53.639 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d47fff700 1 -- 172.21.15.110:0/1556517423 learned_addr learned my addr 172.21.15.110:0/1556517423 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.621+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> v2:172.21.15.204:3300/0 conn(0x7f8d48059e10 0x7f8d4807bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64326061 0 0) 0x7f8d480a5ce0 con 0x7f8d480a5270 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d280030a0 con 0x7f8d480a5270 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 -- 172.21.15.110:0/1556517423 >> v1:172.21.15.204:6789/0 conn(0x7f8d4811bad0 legacy=0x7f8d480a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 -- 172.21.15.110:0/1556517423 >> v1:172.21.15.110:6789/0 conn(0x7f8d480a5270 legacy=0x7f8d4813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 -- 172.21.15.110:0/1556517423 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4811bef0 con 0x7f8d48059e10 2019-10-13T04:04:53.640 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> v2:172.21.15.204:3300/0 conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.641 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> v2:172.21.15.204:3300/0 conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.641 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.641 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.641 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d38008ad0 con 0x7f8d48059e10 2019-10-13T04:04:53.641 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.642 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.642 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3800ccf0 con 0x7f8d48059e10 2019-10-13T04:04:53.642 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d3800d2b0 con 0x7f8d48059e10 2019-10-13T04:04:53.642 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4811c1c0 con 0x7f8d48059e10 2019-10-13T04:04:53.642 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d480a2620 con 0x7f8d48059e10 2019-10-13T04:04:53.643 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.643 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f8d38010280 con 0x7f8d48059e10 2019-10-13T04:04:53.643 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.643 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 0x7f8d2802a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.643 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8d38036c80 con 0x7f8d48059e10 2019-10-13T04:04:53.644 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.625+0000 7f8d46ffd700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 0x7f8d2802a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.644 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.629+0000 7f8d46ffd700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 0x7f8d2802a310 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.644 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.629+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d48042510 con 0x7f8d48059e10 2019-10-13T04:04:53.645 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.637+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.645 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.637+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f8d38010570 con 0x7f8d48059e10 2019-10-13T04:04:53.804 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.801+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d4807c470 con 0x7f8d48059e10 2019-10-13T04:04:53.806 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.801+0000 7f8d477fe700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.806 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.801+0000 7f8d2ffff700 1 -- 172.21.15.110:0/1556517423 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7f8d38008180 con 0x7f8d48059e10 2019-10-13T04:04:53.808 INFO:teuthology.misc.health.smithi110.stdout: 2019-10-13T04:04:53.808 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:53.810 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 msgr2=0x7f8d2802a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.810 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 0x7f8d2802a310 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.810 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 msgr2=0x7f8d4807bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.810 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f8d38001cd0 tx=0x7f8d3800c970).stop 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 shutdown_connections 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8d28027e10 0x7f8d2802a310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 --2- 172.21.15.110:0/1556517423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d48059e10 0x7f8d4807bac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 shutdown_connections 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 wait complete. 2019-10-13T04:04:53.811 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:53.805+0000 7f8d4d26e700 1 -- 172.21.15.110:0/1556517423 >> 172.21.15.110:0/1556517423 conn(0x7f8d4807eaf0 msgr2=0x7f8d48125280 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:53.820 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-13T04:04:53.820 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:53.820 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:04:53.922 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.913+0000 7f7d7ef26700 1 Processor -- start 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.913+0000 7f7d7ef26700 1 -- start start 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d780a6410 0x7f7d780a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d780a8c10 0x7f7d780a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d780a77f0 con 0x7f7d78065060 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780a20e0 con 0x7f7d780a6410 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780a2380 con 0x7f7d780a8c10 2019-10-13T04:04:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d780a6410 0x7f7d780a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7dcc4700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d780a8c10 0x7f7d780a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d780a6410 0x7f7d780a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.110:52252/0 (socket says 172.21.15.110:52252) 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/476785489 learned_addr learned my addr 172.21.15.110:0/476785489 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ccc2700 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f7d78065060 0x7f7d78144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.110:53370/0 (socket says 172.21.15.110:53370) 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d780a8c10 msgr2=0x7f7d780a9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d780a8c10 0x7f7d780a9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/476785489 >> v1:172.21.15.204:6789/0 conn(0x7f7d78065060 legacy=0x7f7d78144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/476785489 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7814f9f0 con 0x7f7d780a6410 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> v2:172.21.15.204:3300/0 conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d780a22a0 tx=0x7f7d68007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> v2:172.21.15.204:3300/0 conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d780a22a0 tx=0x7f7d68007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f7d780a22a0 tx=0x7f7d68007710).ready entity=mon.1 client_cookie=76d06bb5ff5f06ba server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f7d780a22a0 tx=0x7f7d68007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.925 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d6ffff700 1 -- 172.21.15.110:0/476785489 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d70008c80 con 0x7f7d780a6410 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f7d780a22a0 tx=0x7f7d68007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d6ffff700 1 -- 172.21.15.110:0/476785489 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d700088a0 con 0x7f7d780a6410 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f7d780a22a0 tx=0x7f7d68007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d6ffff700 1 -- 172.21.15.110:0/476785489 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d700119f0 con 0x7f7d780a6410 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 msgr2=0x7f7d780a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f7d780a22a0 tx=0x7f7d68007710).stop 2019-10-13T04:04:53.926 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- 172.21.15.110:0/476785489 shutdown_connections 2019-10-13T04:04:53.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d780a8c10 0x7f7d780a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:53.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/476785489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d780a6410 0x7f7d780a1970 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- 172.21.15.110:0/476785489 shutdown_connections 2019-10-13T04:04:53.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- 172.21.15.110:0/476785489 wait complete. 2019-10-13T04:04:53.927 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 -- 172.21.15.110:0/476785489 >> 172.21.15.110:0/476785489 conn(0x7f7d7805afb0 msgr2=0x7f7d7809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.917+0000 7f7d7ef26700 1 Processor -- start 2019-10-13T04:04:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- start start 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7d7809f6a0 0x7f7d7805f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d7807a900 0x7f7d780a2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d7805fa30 0x7f7d780a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7809e6e0 con 0x7f7d7809f6a0 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d7809fc00 con 0x7f7d7807a900 2019-10-13T04:04:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7809fea0 con 0x7f7d7805fa30 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7d7809f6a0 0x7f7d7805f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7d7809f6a0 0x7f7d7805f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42846/0 (socket says 172.21.15.110:42846) 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/977721311 learned_addr learned my addr 172.21.15.110:0/977721311 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7dcc4700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d7805fa30 0x7f7d780a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ccc2700 1 --2- 172.21.15.110:0/977721311 >> v2:172.21.15.204:3300/0 conn(0x7f7d7807a900 0x7f7d780a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d7805fa30 msgr2=0x7f7d780a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d7805fa30 0x7f7d780a2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/977721311 >> v2:172.21.15.204:3300/0 conn(0x7f7d7807a900 msgr2=0x7f7d780a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> v2:172.21.15.204:3300/0 conn(0x7f7d7807a900 0x7f7d780a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 -- 172.21.15.110:0/977721311 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7807b1c0 con 0x7f7d7809f6a0 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> v2:172.21.15.110:3300/0 conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> v2:172.21.15.110:3300/0 conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d70007350 con 0x7f7d7809f6a0 2019-10-13T04:04:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70011610 con 0x7f7d7809f6a0 2019-10-13T04:04:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d780a31f0 con 0x7f7d7809f6a0 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d70005910 con 0x7f7d7809f6a0 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7814e380 con 0x7f7d7809f6a0 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f7d70005be0 con 0x7f7d7809f6a0 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 0x7f7d58029d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f7d70034e10 con 0x7f7d7809f6a0 2019-10-13T04:04:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d780555c0 con 0x7f7d7809f6a0 2019-10-13T04:04:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ccc2700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 0x7f7d58029d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.921+0000 7f7d7ccc2700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 0x7f7d58029d30 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.929+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:53.929+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f7d7001dc80 con 0x7f7d7809f6a0 2019-10-13T04:04:54.112 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.109+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d7809f490 con 0x7f7d7809f6a0 2019-10-13T04:04:54.114 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.109+0000 7f7d7d4c3700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.114 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.109+0000 7f7d6dffb700 1 -- 172.21.15.110:0/977721311 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7f7d7001dee0 con 0x7f7d7809f6a0 2019-10-13T04:04:54.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:54.116 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:04:54.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 msgr2=0x7f7d58029d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 0x7f7d58029d30 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 msgr2=0x7f7d7805f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f7d7809e8a0 tx=0x7f7d74001380).stop 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 shutdown_connections 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d7805fa30 0x7f7d780a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7d58027830 0x7f7d58029d30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.118 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d7809f6a0 0x7f7d7805f4e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.119 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 --2- 172.21.15.110:0/977721311 >> v2:172.21.15.204:3300/0 conn(0x7f7d7807a900 0x7f7d780a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.119 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 shutdown_connections 2019-10-13T04:04:54.119 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 wait complete. 2019-10-13T04:04:54.119 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.113+0000 7f7d7ef26700 1 -- 172.21.15.110:0/977721311 >> 172.21.15.110:0/977721311 conn(0x7f7d780a9f90 msgr2=0x7f7d7809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.139 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.139 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-10-13T04:04:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.225+0000 7f050f3b2700 1 Processor -- start 2019-10-13T04:04:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.225+0000 7f050f3b2700 1 -- start start 2019-10-13T04:04:54.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a6400 0x7f05080a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080a8c00 0x7f05080a9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080a77e0 con 0x7f0508065050 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05080a20d0 con 0x7f05080a8c00 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05080a2370 con 0x7f05080a6400 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080a8c00 0x7f05080a9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080a8c00 0x7f05080a9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42858/0 (socket says 172.21.15.110:42858) 2019-10-13T04:04:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 -- 172.21.15.110:0/2983776116 learned_addr learned my addr 172.21.15.110:0/2983776116 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050d94f700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a6400 0x7f05080a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 -- 172.21.15.110:0/2983776116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a6400 msgr2=0x7f05080a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a6400 0x7f05080a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 -- 172.21.15.110:0/2983776116 >> v1:172.21.15.204:6789/0 conn(0x7f0508065050 legacy=0x7f0508144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 -- 172.21.15.110:0/2983776116 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050814f9e0 con 0x7f05080a8c00 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> v2:172.21.15.110:3300/0 conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f05080a2290 tx=0x7f0504002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> v2:172.21.15.110:3300/0 conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f05080a2290 tx=0x7f0504002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f05080a2290 tx=0x7f0504002ff0).ready entity=mon.0 client_cookie=9e8a6abe28a4c4bc server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f05080a2290 tx=0x7f0504002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050c94d700 1 -- 172.21.15.110:0/2983776116 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0500010cc0 con 0x7f05080a8c00 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f05080a2290 tx=0x7f0504002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050e150700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f05080a2290 tx=0x7f0504002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050c94d700 1 -- 172.21.15.110:0/2983776116 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05000108e0 con 0x7f05080a8c00 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050c94d700 1 -- 172.21.15.110:0/2983776116 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05000114a0 con 0x7f05080a8c00 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 msgr2=0x7f05080a9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f05080a2290 tx=0x7f0504002ff0).stop 2019-10-13T04:04:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- 172.21.15.110:0/2983776116 shutdown_connections 2019-10-13T04:04:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080a8c00 0x7f05080a9020 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 --2- 172.21.15.110:0/2983776116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a6400 0x7f05080a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- 172.21.15.110:0/2983776116 shutdown_connections 2019-10-13T04:04:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- 172.21.15.110:0/2983776116 wait complete. 2019-10-13T04:04:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.229+0000 7f050f3b2700 1 -- 172.21.15.110:0/2983776116 >> 172.21.15.110:0/2983776116 conn(0x7f050805afa0 msgr2=0x7f050809c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.245 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 Processor -- start 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 -- start start 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a5490 0x7f050807a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080637c0 0x7f050814ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f050807ad40 0x7f050814f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0508063080 con 0x7f05080637c0 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0508064150 con 0x7f050807ad40 2019-10-13T04:04:54.246 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050f3b2700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050809daf0 con 0x7f05080a5490 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050d14e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080637c0 0x7f050814ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050d14e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05080637c0 0x7f050814ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42864/0 (socket says 172.21.15.110:42864) 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050d14e700 1 -- 172.21.15.110:0/3051060958 learned_addr learned my addr 172.21.15.110:0/3051060958 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.233+0000 7f050d94f700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a5490 0x7f050807a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050e150700 1 --2- 172.21.15.110:0/3051060958 >> v2:172.21.15.204:3300/0 conn(0x7f050807ad40 0x7f050814f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 -- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a5490 msgr2=0x7f050807a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05080a5490 0x7f050807a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 -- 172.21.15.110:0/3051060958 >> v2:172.21.15.204:3300/0 conn(0x7f050807ad40 msgr2=0x7f050814f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> v2:172.21.15.204:3300/0 conn(0x7f050807ad40 0x7f050814f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 -- 172.21.15.110:0/3051060958 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050809e030 con 0x7f05080637c0 2019-10-13T04:04:54.249 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050e150700 1 --2- 172.21.15.110:0/3051060958 >> v2:172.21.15.204:3300/0 conn(0x7f050807ad40 0x7f050814f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:54.264 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> v2:172.21.15.110:3300/0 conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> v2:172.21.15.110:3300/0 conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04fc00ca90 con 0x7f05080637c0 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc00c6b0 con 0x7f05080637c0 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0508062c40 con 0x7f05080637c0 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04fc010e60 con 0x7f05080637c0 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050809d670 con 0x7f05080637c0 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d14e700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f04fc007420 con 0x7f05080637c0 2019-10-13T04:04:54.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f04f0027880 0x7f04f0029d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f04fc034d90 con 0x7f05080637c0 2019-10-13T04:04:54.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d94f700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f04f0027880 0x7f04f0029d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.237+0000 7f050d94f700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f04f0027880 0x7f04f0029d80 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.241+0000 7f050f3b2700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] conn(0x7f05080a28e0 0x7f05080a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.241+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 --> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05080555b0 con 0x7f05080a28e0 2019-10-13T04:04:54.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.241+0000 7f050e150700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] conn(0x7f05080a28e0 0x7f05080a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.241+0000 7f050e150700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] conn(0x7f05080a28e0 0x7f05080a2d00 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.245+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== osd.0 v2:172.21.15.110:6816/14117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f05080555b0 con 0x7f05080a28e0 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.265+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 --> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f05080a0790 con 0x7f05080a28e0 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.265+0000 7f04fa7fc700 1 -- 172.21.15.110:0/3051060958 <== osd.0 v2:172.21.15.110:6816/14117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f05080a0790 con 0x7f05080a28e0 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] conn(0x7f05080a28e0 msgr2=0x7f05080a2d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:6816/14117,v1:172.21.15.110:6817/14117] conn(0x7f05080a28e0 0x7f05080a2d00 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f04f0027880 msgr2=0x7f04f0029d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.273 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f04f0027880 0x7f04f0029d80 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 msgr2=0x7f050814ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f0508063dc0 tx=0x7f05000103f0).stop 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050e150700 1 -- 172.21.15.110:0/3051060958 reap_dead start 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 shutdown_connections 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 --2- 172.21.15.110:0/3051060958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05080637c0 0x7f050814ed40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 shutdown_connections 2019-10-13T04:04:54.274 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 wait complete. 2019-10-13T04:04:54.275 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.269+0000 7f050f3b2700 1 -- 172.21.15.110:0/3051060958 >> 172.21.15.110:0/3051060958 conn(0x7f05080a6400 msgr2=0x7f05080a9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.281 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:54.282 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.282 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-10-13T04:04:54.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.373+0000 7fc597444700 1 Processor -- start 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.373+0000 7fc597444700 1 -- start start 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc590065060 0x7fc5900a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5900a77f0 con 0x7fc5900a6410 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5900a9030 con 0x7fc5900a8c10 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900a20e0 con 0x7fc590065060 2019-10-13T04:04:54.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5959e1700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc5900a6410 0x7fc590144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45584/0 (socket says 172.21.15.110:45584) 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5959e1700 1 -- 172.21.15.110:0/1326252635 learned_addr learned my addr 172.21.15.110:0/1326252635 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> v2:172.21.15.204:3300/0 conn(0x7fc590065060 0x7fc5900a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632594393 0 0) 0x7fc5900a77f0 con 0x7fc5900a6410 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c0030a0 con 0x7fc5900a6410 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436464205 0 0) 0x7fc5900a9030 con 0x7fc5900a8c10 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5900a77f0 con 0x7fc5900a8c10 2019-10-13T04:04:54.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3008394047 0 0) 0x7fc57c0030a0 con 0x7fc5900a6410 2019-10-13T04:04:54.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 -- 172.21.15.110:0/1326252635 >> v1:172.21.15.204:6789/0 conn(0x7fc5900a8c10 legacy=0x7fc59014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 -- 172.21.15.110:0/1326252635 >> v1:172.21.15.110:6789/0 conn(0x7fc5900a6410 legacy=0x7fc590144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 -- 172.21.15.110:0/1326252635 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5900a9260 con 0x7fc590065060 2019-10-13T04:04:54.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> v2:172.21.15.204:3300/0 conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc580008b00 tx=0x7fc5780014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> v2:172.21.15.204:3300/0 conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc580008b00 tx=0x7fc5780014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7fc580008b00 tx=0x7fc5780014a0).ready entity=mon.1 client_cookie=b2ef1e5415a6851 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fc580008b00 tx=0x7fc5780014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fc580008b00 tx=0x7fc5780014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc578001750 con 0x7fc590065060 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5951e0700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fc580008b00 tx=0x7fc5780014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578001560 con 0x7fc590065060 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc5949df700 1 -- 172.21.15.110:0/1326252635 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc580011ab0 con 0x7fc590065060 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 -- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 msgr2=0x7fc5900a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7fc580008b00 tx=0x7fc5780014a0).stop 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 -- 172.21.15.110:0/1326252635 shutdown_connections 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.377+0000 7fc597444700 1 --2- 172.21.15.110:0/1326252635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc590065060 0x7fc5900a1970 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.389 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- 172.21.15.110:0/1326252635 shutdown_connections 2019-10-13T04:04:54.390 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- 172.21.15.110:0/1326252635 wait complete. 2019-10-13T04:04:54.390 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- 172.21.15.110:0/1326252635 >> 172.21.15.110:0/1326252635 conn(0x7fc59005afb0 msgr2=0x7fc59009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 Processor -- start 2019-10-13T04:04:54.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- start start 2019-10-13T04:04:54.393 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc590059e40 0x7fc590060c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc59005efe0 0x7fc59005f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc590061740 con 0x7fc590112350 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900a0830 con 0x7fc590059e40 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc597444700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900614a0 con 0x7fc59005efe0 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc5959e1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc590059e40 0x7fc590060c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc5959e1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc590059e40 0x7fc590060c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42878/0 (socket says 172.21.15.110:42878) 2019-10-13T04:04:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.381+0000 7fc5959e1700 1 -- 172.21.15.110:0/675106237 learned_addr learned my addr 172.21.15.110:0/675106237 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5951e0700 1 --2- 172.21.15.110:0/675106237 >> v2:172.21.15.204:3300/0 conn(0x7fc59005efe0 0x7fc59005f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 -- 172.21.15.110:0/675106237 >> v2:172.21.15.204:3300/0 conn(0x7fc59005efe0 msgr2=0x7fc59005f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> v2:172.21.15.204:3300/0 conn(0x7fc59005efe0 0x7fc59005f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 -- 172.21.15.110:0/675106237 >> v1:172.21.15.204:6789/0 conn(0x7fc590112350 legacy=0x7fc590112770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 -- 172.21.15.110:0/675106237 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5900a3dd0 con 0x7fc590059e40 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> v2:172.21.15.110:3300/0 conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> v2:172.21.15.110:3300/0 conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.396 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.396 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.396 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.396 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5780113a0 con 0x7fc590059e40 2019-10-13T04:04:54.396 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578010fc0 con 0x7fc590059e40 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5900611e0 con 0x7fc590059e40 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc578011980 con 0x7fc590059e40 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc590113110 con 0x7fc590059e40 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc578007180 con 0x7fc590059e40 2019-10-13T04:04:54.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5959e1700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc574027920 0x7fc574029e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc578034570 con 0x7fc590059e40 2019-10-13T04:04:54.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.385+0000 7fc5951e0700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc574027920 0x7fc574029e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.389+0000 7fc5951e0700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc574027920 0x7fc574029e20 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.399 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.389+0000 7fc597444700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] conn(0x7fc590054340 0x7fc590054760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.399 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.389+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 --> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5900555c0 con 0x7fc590054340 2019-10-13T04:04:54.399 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.389+0000 7fc5961e2700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] conn(0x7fc590054340 0x7fc590054760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.400 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.393+0000 7fc5961e2700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] conn(0x7fc590054340 0x7fc590054760 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.400 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.393+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== osd.1 v2:172.21.15.110:6808/14116 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fc5900555c0 con 0x7fc590054340 2019-10-13T04:04:54.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 --> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc590054cb0 con 0x7fc590054340 2019-10-13T04:04:54.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc58e7fc700 1 -- 172.21.15.110:0/675106237 <== osd.1 v2:172.21.15.110:6808/14116 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc590054cb0 con 0x7fc590054340 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] conn(0x7fc590054340 msgr2=0x7fc590054760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:6808/14116,v1:172.21.15.110:6809/14116] conn(0x7fc590054340 0x7fc590054760 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc574027920 msgr2=0x7fc574029e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc574027920 0x7fc574029e20 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 msgr2=0x7fc590060c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7fc5900a09f0 tx=0x7fc5800102f0).stop 2019-10-13T04:04:54.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc5961e2700 1 -- 172.21.15.110:0/675106237 reap_dead start 2019-10-13T04:04:54.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 shutdown_connections 2019-10-13T04:04:54.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 --2- 172.21.15.110:0/675106237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc590059e40 0x7fc590060c90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 shutdown_connections 2019-10-13T04:04:54.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 wait complete. 2019-10-13T04:04:54.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.417+0000 7fc597444700 1 -- 172.21.15.110:0/675106237 >> 172.21.15.110:0/675106237 conn(0x7fc59009c500 msgr2=0x7fc590059300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.432 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:54.433 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.433 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-10-13T04:04:54.532 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.521+0000 7f87e901c700 1 Processor -- start 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- start start 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40a77f0 con 0x7f87e40a8c10 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40a9030 con 0x7f87e40a6410 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40a20e0 con 0x7f87e4065060 2019-10-13T04:04:54.533 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e881b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f87e40a8c10 0x7f87e414e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45604/0 (socket says 172.21.15.110:45604) 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e881b700 1 -- 172.21.15.110:0/2796360769 learned_addr learned my addr 172.21.15.110:0/2796360769 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636183588 0 0) 0x7f87e40a77f0 con 0x7f87e40a8c10 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc0030a0 con 0x7f87e40a8c10 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341732663 0 0) 0x7f87e40a9030 con 0x7f87e40a6410 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e40a77f0 con 0x7f87e40a6410 2019-10-13T04:04:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 -- 172.21.15.110:0/2796360769 >> v1:172.21.15.204:6789/0 conn(0x7f87e40a6410 legacy=0x7f87e4144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 -- 172.21.15.110:0/2796360769 >> v1:172.21.15.110:6789/0 conn(0x7f87e40a8c10 legacy=0x7f87e414e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 -- 172.21.15.110:0/2796360769 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40a9260 con 0x7f87e4065060 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f87d80032b0 tx=0x7f87d800ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f87d80032b0 tx=0x7f87d800ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f87d80032b0 tx=0x7f87d800ff60).ready entity=mon.2 client_cookie=9f9c565fa3668704 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f87d80032b0 tx=0x7f87d800ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f87d80032b0 tx=0x7f87d800ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d8011a00 con 0x7f87e4065060 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f87d80032b0 tx=0x7f87d800ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8011620 con 0x7f87e4065060 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e259c700 1 -- 172.21.15.110:0/2796360769 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d800a160 con 0x7f87e4065060 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 msgr2=0x7f87e40a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f87d80032b0 tx=0x7f87d800ff60).stop 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- 172.21.15.110:0/2796360769 shutdown_connections 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 --2- 172.21.15.110:0/2796360769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4065060 0x7f87e40a1970 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- 172.21.15.110:0/2796360769 shutdown_connections 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- 172.21.15.110:0/2796360769 wait complete. 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.525+0000 7f87e901c700 1 -- 172.21.15.110:0/2796360769 >> 172.21.15.110:0/2796360769 conn(0x7f87e405afb0 msgr2=0x7f87e409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 Processor -- start 2019-10-13T04:04:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 -- start start 2019-10-13T04:04:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4064280 0x7f87e412ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87e412b180 0x7f87e4062c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e411b090 con 0x7f87e40631a0 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e411a8b0 con 0x7f87e412b180 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e901c700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e411ab50 con 0x7f87e4064280 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87e412b180 0x7f87e4062c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f87e412b180 0x7f87e4062c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42894/0 (socket says 172.21.15.110:42894) 2019-10-13T04:04:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 -- 172.21.15.110:0/3889897327 learned_addr learned my addr 172.21.15.110:0/3889897327 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e881b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f87e40631a0 0x7f87e407d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45610/0 (socket says 172.21.15.110:45610) 2019-10-13T04:04:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e359e700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4064280 0x7f87e412ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 -- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4064280 msgr2=0x7f87e412ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4064280 0x7f87e412ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 -- 172.21.15.110:0/3889897327 >> v1:172.21.15.110:6789/0 conn(0x7f87e40631a0 legacy=0x7f87e407d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706708656 0 0) 0x7f87e411b090 con 0x7f87e40631a0 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 -- 172.21.15.110:0/3889897327 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40646a0 con 0x7f87e412b180 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e359e700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87e4064280 0x7f87e412ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> v2:172.21.15.110:3300/0 conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.529+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> v2:172.21.15.110:3300/0 conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d8005aa0 con 0x7f87e412b180 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8011c00 con 0x7f87e412b180 2019-10-13T04:04:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e411a530 con 0x7f87e412b180 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d8007340 con 0x7f87e412b180 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e412b5a0 con 0x7f87e412b180 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f87d800ffe0 con 0x7f87e412b180 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e2d9d700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f87c40277c0 0x7f87c4029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f87d8034f30 con 0x7f87e412b180 2019-10-13T04:04:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e359e700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f87c40277c0 0x7f87c4029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.533+0000 7f87e359e700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f87c40277c0 0x7f87c4029cc0 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.541+0000 7f87e901c700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] conn(0x7f87e4068890 0x7f87e407da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.541+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 --> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87e40555c0 con 0x7f87e4068890 2019-10-13T04:04:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.541+0000 7f87e881b700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] conn(0x7f87e4068890 0x7f87e407da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.541+0000 7f87e881b700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] conn(0x7f87e4068890 0x7f87e407da40 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.541+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== osd.2 v2:172.21.15.110:6800/14118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f87e40555c0 con 0x7f87e4068890 2019-10-13T04:04:54.570 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.565+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 --> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87e4068cb0 con 0x7f87e4068890 2019-10-13T04:04:54.570 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.565+0000 7f87cbfff700 1 -- 172.21.15.110:0/3889897327 <== osd.2 v2:172.21.15.110:6800/14118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f87e4068cb0 con 0x7f87e4068890 2019-10-13T04:04:54.572 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] conn(0x7f87e4068890 msgr2=0x7f87e407da40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.572 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:6800/14118,v1:172.21.15.110:6801/14118] conn(0x7f87e4068890 0x7f87e407da40 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.573 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f87c40277c0 msgr2=0x7f87c4029cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.573 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f87c40277c0 0x7f87c4029cc0 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.573 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 msgr2=0x7f87e4062c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.573 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f87e411aa70 tx=0x7f87dc001270).stop 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e881b700 1 -- 172.21.15.110:0/3889897327 reap_dead start 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 shutdown_connections 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 --2- 172.21.15.110:0/3889897327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e412b180 0x7f87e4062c50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 shutdown_connections 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 wait complete. 2019-10-13T04:04:54.574 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.569+0000 7f87e901c700 1 -- 172.21.15.110:0/3889897327 >> 172.21.15.110:0/3889897327 conn(0x7f87e40a6410 msgr2=0x7f87e4059300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.582 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:54.589 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.589 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-10-13T04:04:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.693+0000 7f1b26c0c700 1 Processor -- start 2019-10-13T04:04:54.703 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.693+0000 7f1b26c0c700 1 -- start start 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b20065060 0x7f1b200a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200a77f0 con 0x7f1b200a6410 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200a9030 con 0x7f1b200a8c10 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b200a20e0 con 0x7f1b20065060 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b251a9700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1b200a6410 0x7f1b20144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45616/0 (socket says 172.21.15.110:45616) 2019-10-13T04:04:54.704 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b251a9700 1 -- 172.21.15.110:0/1990129007 learned_addr learned my addr 172.21.15.110:0/1990129007 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> v2:172.21.15.204:3300/0 conn(0x7f1b20065060 0x7f1b200a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b1ffff700 1 -- 172.21.15.110:0/1990129007 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898918141 0 0) 0x7f1b200a77f0 con 0x7f1b200a6410 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b1ffff700 1 -- 172.21.15.110:0/1990129007 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b080030a0 con 0x7f1b200a6410 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 -- 172.21.15.110:0/1990129007 >> v1:172.21.15.204:6789/0 conn(0x7f1b200a8c10 legacy=0x7f1b2014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 -- 172.21.15.110:0/1990129007 >> v1:172.21.15.110:6789/0 conn(0x7f1b200a6410 legacy=0x7f1b20144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.705 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 -- 172.21.15.110:0/1990129007 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200a9260 con 0x7f1b20065060 2019-10-13T04:04:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> v2:172.21.15.204:3300/0 conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> v2:172.21.15.204:3300/0 conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).ready entity=mon.1 client_cookie=a604fa6209933dd0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.706 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b1ffff700 1 -- 172.21.15.110:0/1990129007 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b140119c0 con 0x7f1b20065060 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b249a8700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b1ffff700 1 -- 172.21.15.110:0/1990129007 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b140115e0 con 0x7f1b20065060 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b1ffff700 1 -- 172.21.15.110:0/1990129007 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b1400a160 con 0x7f1b20065060 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 msgr2=0x7f1b200a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f1b14008b00 tx=0x7f1b1400ff60).stop 2019-10-13T04:04:54.707 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- 172.21.15.110:0/1990129007 shutdown_connections 2019-10-13T04:04:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/1990129007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b20065060 0x7f1b200a1970 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- 172.21.15.110:0/1990129007 shutdown_connections 2019-10-13T04:04:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- 172.21.15.110:0/1990129007 wait complete. 2019-10-13T04:04:54.708 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.697+0000 7f1b26c0c700 1 -- 172.21.15.110:0/1990129007 >> 172.21.15.110:0/1990129007 conn(0x7f1b2005afb0 msgr2=0x7f1b2009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 Processor -- start 2019-10-13T04:04:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- start start 2019-10-13T04:04:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b2014ed70 0x7f1b2014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.712 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20075270 con 0x7f1b20063440 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b20063fc0 con 0x7f1b2014ed70 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b200a56e0 con 0x7f1b2014f6e0 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b249a8700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b2014ed70 0x7f1b2014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b251a9700 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f1b20063440 0x7f1b2009ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.110:53440/0 (socket says 172.21.15.110:53440) 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b251a9700 1 -- 172.21.15.110:0/498770247 learned_addr learned my addr 172.21.15.110:0/498770247 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.713 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 -- 172.21.15.110:0/498770247 >> v2:172.21.15.204:3300/0 conn(0x7f1b2014ed70 msgr2=0x7f1b2014f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> v2:172.21.15.204:3300/0 conn(0x7f1b2014ed70 0x7f1b2014f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777041313 0 0) 0x7f1b080030a0 con 0x7f1b20063440 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 -- 172.21.15.110:0/498770247 >> v1:172.21.15.204:6789/0 conn(0x7f1b20063440 legacy=0x7f1b2009ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 -- 172.21.15.110:0/498770247 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200a2c10 con 0x7f1b2014f6e0 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b249a8700 1 --2- 172.21.15.110:0/498770247 >> v2:172.21.15.204:3300/0 conn(0x7f1b2014ed70 0x7f1b2014f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b10003540 con 0x7f1b2014f6e0 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b10003160 con 0x7f1b2014f6e0 2019-10-13T04:04:54.715 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b10003b20 con 0x7f1b2014f6e0 2019-10-13T04:04:54.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2014fb00 con 0x7f1b2014f6e0 2019-10-13T04:04:54.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.701+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2009f3d0 con 0x7f1b2014f6e0 2019-10-13T04:04:54.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.716 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1b10009450 con 0x7f1b2014f6e0 2019-10-13T04:04:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b259aa700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b1dffb700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1b000277e0 0x7f1b00029ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1b100344d0 con 0x7f1b2014f6e0 2019-10-13T04:04:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b251a9700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1b000277e0 0x7f1b00029ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.705+0000 7f1b251a9700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1b000277e0 0x7f1b00029ce0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.709+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] conn(0x7f1b2007c100 0x7f1b2007c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.709+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 --> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b200a54d0 con 0x7f1b2007c100 2019-10-13T04:04:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.709+0000 7f1b249a8700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] conn(0x7f1b2007c100 0x7f1b2007c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.709+0000 7f1b249a8700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] conn(0x7f1b2007c100 0x7f1b2007c520 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.713+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== osd.3 v2:172.21.15.110:6824/14157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f1b200a54d0 con 0x7f1b2007c100 2019-10-13T04:04:54.738 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.733+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 --> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1b200555c0 con 0x7f1b2007c100 2019-10-13T04:04:54.739 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b1dffb700 1 -- 172.21.15.110:0/498770247 <== osd.3 v2:172.21.15.110:6824/14157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1b200555c0 con 0x7f1b2007c100 2019-10-13T04:04:54.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 >> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] conn(0x7f1b2007c100 msgr2=0x7f1b2007c520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.110:6824/14157,v1:172.21.15.110:6825/14157] conn(0x7f1b2007c100 0x7f1b2007c520 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1b000277e0 msgr2=0x7f1b00029ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1b000277e0 0x7f1b00029ce0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 msgr2=0x7f1b201320a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 --2- 172.21.15.110:0/498770247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b2014f6e0 0x7f1b201320a0 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f1b10000e90 tx=0x7f1b14007fd0).stop 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b259aa700 1 -- 172.21.15.110:0/498770247 reap_dead start 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 shutdown_connections 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 shutdown_connections 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 wait complete. 2019-10-13T04:04:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.737+0000 7f1b26c0c700 1 -- 172.21.15.110:0/498770247 >> 172.21.15.110:0/498770247 conn(0x7f1b200a9530 msgr2=0x7f1b2009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.751 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:54.751 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.751 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.837+0000 7f247d084700 1 Processor -- start 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.837+0000 7f247d084700 1 -- start start 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247d084700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f24780a1970 0x7f247814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247d084700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780a9410 con 0x7f2478065060 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247d084700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780a1d90 con 0x7f24780a6410 2019-10-13T04:04:54.850 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247d084700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780a20e0 con 0x7f24780a1970 2019-10-13T04:04:54.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f246ed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2478065060 0x7f24780a8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45634/0 (socket says 172.21.15.110:45634) 2019-10-13T04:04:54.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f246ed9d700 1 -- 172.21.15.110:0/3047163677 learned_addr learned my addr 172.21.15.110:0/3047163677 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:54.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247c883700 1 --2- 172.21.15.110:0/3047163677 >> v2:172.21.15.204:3300/0 conn(0x7f24780a1970 0x7f247814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646607269 0 0) 0x7f24780a9410 con 0x7f2478065060 2019-10-13T04:04:54.851 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24640030a0 con 0x7f2478065060 2019-10-13T04:04:54.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1233142643 0 0) 0x7f24640030a0 con 0x7f2478065060 2019-10-13T04:04:54.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 >> v2:172.21.15.204:3300/0 conn(0x7f24780a1970 msgr2=0x7f247814f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 --2- 172.21.15.110:0/3047163677 >> v2:172.21.15.204:3300/0 conn(0x7f24780a1970 0x7f247814f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.852 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 >> v1:172.21.15.204:6789/0 conn(0x7f24780a6410 legacy=0x7f2478144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780a77f0 con 0x7f2478065060 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f2460001750 con 0x7f2478065060 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247c883700 1 --2- 172.21.15.110:0/3047163677 >> v2:172.21.15.204:3300/0 conn(0x7f24780a1970 0x7f247814f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 >> v1:172.21.15.110:6789/0 conn(0x7f2478065060 legacy=0x7f24780a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.853 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2464004410 0x7f2464004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.855 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2464004d80 0x7f246400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.855 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780a20e0 con 0x7f2464003720 2019-10-13T04:04:54.855 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246404fc20 con 0x7f2464004d80 2019-10-13T04:04:54.855 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246404fe60 con 0x7f2464004410 2019-10-13T04:04:54.856 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.856 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f246ed9d700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2464004410 0x7f2464004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.856 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247c883700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2464004d80 0x7f246400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.856 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 -- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2464004410 msgr2=0x7f2464004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.857 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2464004410 0x7f2464004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.857 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 -- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2464004d80 msgr2=0x7f246400b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.857 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2464004d80 0x7f246400b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.857 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247c883700 1 -- 172.21.15.110:0/3047163677 reap_dead start 2019-10-13T04:04:54.857 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 -- 172.21.15.110:0/3047163677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464050160 con 0x7f2464003720 2019-10-13T04:04:54.858 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2468001000 tx=0x7f2470007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.858 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2468001000 tx=0x7f2470007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.858 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.841+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f2468001000 tx=0x7f2470007cb0).ready entity=mon.0 client_cookie=b12f9fdd96955177 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f2468001000 tx=0x7f2470007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468011490 con 0x7f2464003720 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f2468001000 tx=0x7f2470007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24680110b0 con 0x7f2464003720 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247759e700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f2468001000 tx=0x7f2470007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f2476d9d700 1 -- 172.21.15.110:0/3047163677 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468011a90 con 0x7f2464003720 2019-10-13T04:04:54.859 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 msgr2=0x7f2464003b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f2468001000 tx=0x7f2470007cb0).stop 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- 172.21.15.110:0/3047163677 shutdown_connections 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 --2- 172.21.15.110:0/3047163677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2464003720 0x7f2464003b40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- 172.21.15.110:0/3047163677 shutdown_connections 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- 172.21.15.110:0/3047163677 wait complete. 2019-10-13T04:04:54.860 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- 172.21.15.110:0/3047163677 >> 172.21.15.110:0/3047163677 conn(0x7f247805afb0 msgr2=0x7f247809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.865 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 Processor -- start 2019-10-13T04:04:54.865 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.845+0000 7f247d084700 1 -- start start 2019-10-13T04:04:54.865 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247811c780 0x7f247811f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.865 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24780780b0 0x7f24780784d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2478122c70 con 0x7f24781246e0 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247811dca0 con 0x7f24780780b0 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2478122490 con 0x7f247811c780 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24780780b0 0x7f24780784d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247759e700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247811c780 0x7f247811f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24780780b0 0x7f24780784d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42936/0 (socket says 172.21.15.110:42936) 2019-10-13T04:04:54.866 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 -- 172.21.15.110:0/3594396278 learned_addr learned my addr 172.21.15.110:0/3594396278 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203325468 0 0) 0x7f2478122c70 con 0x7f24781246e0 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c0030a0 con 0x7f24781246e0 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 -- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247811c780 msgr2=0x7f247811f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247811c780 0x7f247811f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 -- 172.21.15.110:0/3594396278 >> v1:172.21.15.110:6789/0 conn(0x7f24781246e0 legacy=0x7f247811d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 -- 172.21.15.110:0/3594396278 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2478078a20 con 0x7f24780780b0 2019-10-13T04:04:54.867 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> v2:172.21.15.110:3300/0 conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> v2:172.21.15.110:3300/0 conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f247000ca20 con 0x7f24780780b0 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.868 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247000eb90 con 0x7f24780780b0 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2478078cf0 con 0x7f24780780b0 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24700117b0 con 0x7f24780780b0 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247813b660 con 0x7f24780780b0 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.849+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f24700119b0 con 0x7f24780780b0 2019-10-13T04:04:54.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f247c883700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:54.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f2474d99700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f245c027e10 0x7f245c02a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f24700082c0 con 0x7f24780780b0 2019-10-13T04:04:54.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f247759e700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f245c027e10 0x7f245c02a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f247759e700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f245c027e10 0x7f245c02a310 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f247d084700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] conn(0x7f2478123220 0x7f2478123640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 --> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24780555c0 con 0x7f2478123220 2019-10-13T04:04:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f246ed9d700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] conn(0x7f2478123220 0x7f2478123640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.853+0000 7f246ed9d700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] conn(0x7f2478123220 0x7f2478123640 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:54.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.857+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== osd.4 v2:172.21.15.204:6826/13912 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f24780555c0 con 0x7f2478123220 2019-10-13T04:04:54.884 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 --> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f24780a0840 con 0x7f2478123220 2019-10-13T04:04:54.885 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f2474d99700 1 -- 172.21.15.110:0/3594396278 <== osd.4 v2:172.21.15.204:6826/13912 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f24780a0840 con 0x7f2478123220 2019-10-13T04:04:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] conn(0x7f2478123220 msgr2=0x7f2478123640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6826/13912,v1:172.21.15.204:6827/13912] conn(0x7f2478123220 0x7f2478123640 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f245c027e10 msgr2=0x7f245c02a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.887 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f245c027e10 0x7f245c02a310 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 msgr2=0x7f24780784d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f2478122650 tx=0x7f2468001d30).stop 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247c883700 1 -- 172.21.15.110:0/3594396278 reap_dead start 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 shutdown_connections 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 --2- 172.21.15.110:0/3594396278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24780780b0 0x7f24780784d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 shutdown_connections 2019-10-13T04:04:54.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 wait complete. 2019-10-13T04:04:54.889 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.881+0000 7f247d084700 1 -- 172.21.15.110:0/3594396278 >> 172.21.15.110:0/3594396278 conn(0x7f24780a3860 msgr2=0x7f247807dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:54.897 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:54.897 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:54.897 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-10-13T04:04:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.985+0000 7f0f5c9bf700 1 Processor -- start 2019-10-13T04:04:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.985+0000 7f0f5c9bf700 1 -- start start 2019-10-13T04:04:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0f580a6410 0x7f0f580a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.997 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f58065060 0x7f0f58144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58145010 0x7f0f580a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580a20e0 con 0x7f0f58145010 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580a2380 con 0x7f0f580a6410 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580a77f0 con 0x7f0f58065060 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58145010 0x7f0f580a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.998 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58145010 0x7f0f580a8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42946/0 (socket says 172.21.15.110:42946) 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 -- 172.21.15.110:0/3598792307 learned_addr learned my addr 172.21.15.110:0/3598792307 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5659c700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f58065060 0x7f0f58144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/3598792307 >> v2:172.21.15.204:3300/0 conn(0x7f0f580a6410 0x7f0f580a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 -- 172.21.15.110:0/3598792307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f58065060 msgr2=0x7f0f58144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f58065060 0x7f0f58144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 -- 172.21.15.110:0/3598792307 >> v2:172.21.15.204:3300/0 conn(0x7f0f580a6410 msgr2=0x7f0f580a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:54.999 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> v2:172.21.15.204:3300/0 conn(0x7f0f580a6410 0x7f0f580a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 -- 172.21.15.110:0/3598792307 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580a9160 con 0x7f0f58145010 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> v2:172.21.15.110:3300/0 conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> v2:172.21.15.110:3300/0 conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).ready entity=mon.0 client_cookie=2f94503625eb85b6 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.000 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f55d9b700 1 -- 172.21.15.110:0/3598792307 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f500070c0 con 0x7f0f58145010 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5759e700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f55d9b700 1 -- 172.21.15.110:0/3598792307 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50006be0 con 0x7f0f58145010 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f55d9b700 1 -- 172.21.15.110:0/3598792307 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f500076a0 con 0x7f0f58145010 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 msgr2=0x7f0f580a8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.989+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7f0f5805ccf0 tx=0x7f0f48007710).stop 2019-10-13T04:04:55.001 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/3598792307 shutdown_connections 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58145010 0x7f0f580a8c10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/3598792307 >> v2:172.21.15.204:3300/0 conn(0x7f0f580a6410 0x7f0f580a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/3598792307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f58065060 0x7f0f58144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/3598792307 shutdown_connections 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/3598792307 wait complete. 2019-10-13T04:04:55.002 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/3598792307 >> 172.21.15.110:0/3598792307 conn(0x7f0f5805afb0 msgr2=0x7f0f5809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.005 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 Processor -- start 2019-10-13T04:04:55.005 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.993+0000 7f0f5c9bf700 1 -- start start 2019-10-13T04:04:55.005 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58120280 0x7f0f58122990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f5811b6c0 0x7f0f5811bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f581209e0 con 0x7f0f58123b20 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f58124aa0 con 0x7f0f58120280 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f58124d40 con 0x7f0f5811b6c0 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58120280 0x7f0f58122990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f58120280 0x7f0f58122990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42950/0 (socket says 172.21.15.110:42950) 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 -- 172.21.15.110:0/1265185312 learned_addr learned my addr 172.21.15.110:0/1265185312 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5759e700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f5811b6c0 0x7f0f5811bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 -- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f5811b6c0 msgr2=0x7f0f5811bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0f5811b6c0 0x7f0f5811bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 -- 172.21.15.110:0/1265185312 >> v1:172.21.15.204:6789/0 conn(0x7f0f58123b20 legacy=0x7f0f58123f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 -- 172.21.15.110:0/1265185312 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f58124740 con 0x7f0f58120280 2019-10-13T04:04:55.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> v2:172.21.15.110:3300/0 conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> v2:172.21.15.110:3300/0 conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f48012640 con 0x7f0f58120280 2019-10-13T04:04:55.008 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f48007150 con 0x7f0f58120280 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f58122ee0 con 0x7f0f58120280 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f48012c20 con 0x7f0f58120280 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5811c030 con 0x7f0f58120280 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f0f48005530 con 0x7f0f58120280 2019-10-13T04:04:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f56d9d700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f0f34027890 0x7f0f34029d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:54.997+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0f48034f60 con 0x7f0f58120280 2019-10-13T04:04:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5659c700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f0f34027890 0x7f0f34029d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5659c700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f0f34027890 0x7f0f34029d90 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] conn(0x7f0f5807b510 0x7f0f5807b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 --> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f580555c0 con 0x7f0f5807b510 2019-10-13T04:04:55.011 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5759e700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] conn(0x7f0f5807b510 0x7f0f5807b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.001+0000 7f0f5759e700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] conn(0x7f0f5807b510 0x7f0f5807b930 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.012 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.005+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== osd.5 v2:172.21.15.204:6810/13910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f0f580555c0 con 0x7f0f5807b510 2019-10-13T04:04:55.032 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 --> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f5807be80 con 0x7f0f5807b510 2019-10-13T04:04:55.033 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f4f7fe700 1 -- 172.21.15.110:0/1265185312 <== osd.5 v2:172.21.15.204:6810/13910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f5807be80 con 0x7f0f5807b510 2019-10-13T04:04:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] conn(0x7f0f5807b510 msgr2=0x7f0f5807b930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6810/13910,v1:172.21.15.204:6811/13910] conn(0x7f0f5807b510 0x7f0f5807b930 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f0f34027890 msgr2=0x7f0f34029d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f0f34027890 0x7f0f34029d90 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 msgr2=0x7f0f58122990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f0f58124c60 tx=0x7f0f44002ff0).stop 2019-10-13T04:04:55.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5759e700 1 -- 172.21.15.110:0/1265185312 reap_dead start 2019-10-13T04:04:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 shutdown_connections 2019-10-13T04:04:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 --2- 172.21.15.110:0/1265185312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58120280 0x7f0f58122990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 shutdown_connections 2019-10-13T04:04:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 wait complete. 2019-10-13T04:04:55.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.029+0000 7f0f5c9bf700 1 -- 172.21.15.110:0/1265185312 >> 172.21.15.110:0/1265185312 conn(0x7f0f580a6410 msgr2=0x7f0f580a3200 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.045 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:55.045 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:55.045 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-10-13T04:04:55.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 Processor -- start 2019-10-13T04:04:55.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 -- start start 2019-10-13T04:04:55.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 0x7fe2c00a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.152 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00a77f0 con 0x7fe2c00a6410 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c00a20e0 con 0x7fe2c0065060 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c720c700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00a2380 con 0x7fe2c00a8c10 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c57a9700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe2c00a6410 0x7fe2c0144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45670/0 (socket says 172.21.15.110:45670) 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c57a9700 1 -- 172.21.15.110:0/1176797975 learned_addr learned my addr 172.21.15.110:0/1176797975 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.153 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c4fa8700 1 --2- 172.21.15.110:0/1176797975 >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 0x7fe2c00a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393035381 0 0) 0x7fe2c00a77f0 con 0x7fe2c00a6410 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a80030a0 con 0x7fe2c00a6410 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 598116846 0 0) 0x7fe2a80030a0 con 0x7fe2c00a6410 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1176797975 >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 msgr2=0x7fe2c00a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 0x7fe2c00a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1176797975 >> v1:172.21.15.110:6789/0 conn(0x7fe2c00a6410 legacy=0x7fe2c0144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.154 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1176797975 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c014f9f0 con 0x7fe2c00a8c10 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c4fa8700 1 --2- 172.21.15.110:0/1176797975 >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 0x7fe2c00a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.141+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).ready entity=mon.2 client_cookie=cca46549c75d4625 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2bc001750 con 0x7fe2c00a8c10 2019-10-13T04:04:55.155 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc001560 con 0x7fe2c00a8c10 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 msgr2=0x7fe2c00a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2b7fff700 1 -- 172.21.15.110:0/1176797975 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2b8005740 con 0x7fe2c00a8c10 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fe2b8008780 tx=0x7fe2bc0019d0).stop 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- 172.21.15.110:0/1176797975 shutdown_connections 2019-10-13T04:04:55.156 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1176797975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c00a8c10 0x7fe2c00a9030 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1176797975 >> v2:172.21.15.204:3300/0 conn(0x7fe2c0065060 0x7fe2c00a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- 172.21.15.110:0/1176797975 shutdown_connections 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- 172.21.15.110:0/1176797975 wait complete. 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- 172.21.15.110:0/1176797975 >> 172.21.15.110:0/1176797975 conn(0x7fe2c005afb0 msgr2=0x7fe2c009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 Processor -- start 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- start start 2019-10-13T04:04:55.157 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe2c00a9680 0x7fe2c0061fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c009d950 con 0x7fe2c0062510 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c00a3dd0 con 0x7fe2c00a9680 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c720c700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00a5a90 con 0x7fe2c009fdd0 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c4fa8700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe2c0062510 0x7fe2c009f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45680/0 (socket says 172.21.15.110:45680) 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c4fa8700 1 -- 172.21.15.110:0/1126796962 learned_addr learned my addr 172.21.15.110:0/1126796962 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:55.158 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c57a9700 1 --2- 172.21.15.110:0/1126796962 >> v2:172.21.15.204:3300/0 conn(0x7fe2c00a9680 0x7fe2c0061fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.145+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863929577 0 0) 0x7fe2c009d950 con 0x7fe2c0062510 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a00030a0 con 0x7fe2c0062510 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1126796962 >> v2:172.21.15.204:3300/0 conn(0x7fe2c00a9680 msgr2=0x7fe2c0061fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> v2:172.21.15.204:3300/0 conn(0x7fe2c00a9680 0x7fe2c0061fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.159 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1126796962 >> v1:172.21.15.110:6789/0 conn(0x7fe2c0062510 legacy=0x7fe2c009f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 209529475 0 0) 0x7fe2a00030a0 con 0x7fe2c0062510 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1126796962 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00a5d10 con 0x7fe2c009fdd0 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c57a9700 1 --2- 172.21.15.110:0/1126796962 >> v2:172.21.15.204:3300/0 conn(0x7fe2c00a9680 0x7fe2c0061fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.160 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.162 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.162 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2b0001850 con 0x7fe2c009fdd0 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0001660 con 0x7fe2c009fdd0 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c00a01f0 con 0x7fe2c009fdd0 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2b8008ba0 con 0x7fe2c009fdd0 2019-10-13T04:04:55.163 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c0130960 con 0x7fe2c009fdd0 2019-10-13T04:04:55.164 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.164 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fe2b800c310 con 0x7fe2c009fdd0 2019-10-13T04:04:55.164 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c5faa700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.164 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe2a0027e10 0x7fe2a002a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.165 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fe2b8033d10 con 0x7fe2c009fdd0 2019-10-13T04:04:55.165 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.149+0000 7fe2c57a9700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe2a0027e10 0x7fe2a002a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.165 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.153+0000 7fe2c57a9700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe2a0027e10 0x7fe2a002a310 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.165 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.157+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] conn(0x7fe2c009e130 0x7fe2c00a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.165 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.157+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 --> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2c00555c0 con 0x7fe2c009e130 2019-10-13T04:04:55.166 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.157+0000 7fe2c4fa8700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] conn(0x7fe2c009e130 0x7fe2c00a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.166 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.157+0000 7fe2c4fa8700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] conn(0x7fe2c009e130 0x7fe2c00a3fe0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.166 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.157+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== osd.6 v2:172.21.15.204:6818/13911 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fe2c00555c0 con 0x7fe2c009e130 2019-10-13T04:04:55.189 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 --> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2c00a5880 con 0x7fe2c009e130 2019-10-13T04:04:55.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2b5ffb700 1 -- 172.21.15.110:0/1126796962 <== osd.6 v2:172.21.15.204:6818/13911 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2c00a5880 con 0x7fe2c009e130 2019-10-13T04:04:55.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] conn(0x7fe2c009e130 msgr2=0x7fe2c00a3fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.192 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6818/13911,v1:172.21.15.204:6819/13911] conn(0x7fe2c009e130 0x7fe2c00a3fe0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe2a0027e10 msgr2=0x7fe2a002a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fe2a0027e10 0x7fe2a002a310 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 msgr2=0x7fe2c0130410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 --2- 172.21.15.110:0/1126796962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2c009fdd0 0x7fe2c0130410 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fe2b0001a90 tx=0x7fe2b0007e40).stop 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c5faa700 1 -- 172.21.15.110:0/1126796962 reap_dead start 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 shutdown_connections 2019-10-13T04:04:55.193 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 shutdown_connections 2019-10-13T04:04:55.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 wait complete. 2019-10-13T04:04:55.194 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.185+0000 7fe2c720c700 1 -- 172.21.15.110:0/1126796962 >> 172.21.15.110:0/1126796962 conn(0x7fe2c00a2ba0 msgr2=0x7fe2c009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.200 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:55.201 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:55.201 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-10-13T04:04:55.310 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.293+0000 7fa5be021700 1 Processor -- start 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.293+0000 7fa5be021700 1 -- start start 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa5b80a6410 0x7fa5b80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5b8065060 0x7fa5b8144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 0x7fa5b80a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80a20e0 con 0x7fa5b8065060 2019-10-13T04:04:55.311 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80a2380 con 0x7fa5b80a6410 2019-10-13T04:04:55.312 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80a77f0 con 0x7fa5b8145010 2019-10-13T04:04:55.312 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5b8065060 0x7fa5b8144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.312 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5b8065060 0x7fa5b8144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42976/0 (socket says 172.21.15.110:42976) 2019-10-13T04:04:55.312 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 -- 172.21.15.110:0/161302586 learned_addr learned my addr 172.21.15.110:0/161302586 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:55.322 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/161302586 >> v2:172.21.15.204:3300/0 conn(0x7fa5b80a6410 0x7fa5b80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.322 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5bcdbf700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 0x7fa5b80a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.323 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 -- 172.21.15.110:0/161302586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 msgr2=0x7fa5b80a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:55.323 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 0x7fa5b80a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.323 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 -- 172.21.15.110:0/161302586 >> v2:172.21.15.204:3300/0 conn(0x7fa5b80a6410 msgr2=0x7fa5b80a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:55.323 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> v2:172.21.15.204:3300/0 conn(0x7fa5b80a6410 0x7fa5b80a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.328 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 -- 172.21.15.110:0/161302586 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80a9160 con 0x7fa5b8065060 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5bcdbf700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 0x7fa5b80a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> v2:172.21.15.110:3300/0 conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> v2:172.21.15.110:3300/0 conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).ready entity=mon.0 client_cookie=b33f5ababbc0b01c server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.329 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b6ffd700 1 -- 172.21.15.110:0/161302586 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ac008c80 con 0x7fa5b8065060 2019-10-13T04:04:55.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b6ffd700 1 -- 172.21.15.110:0/161302586 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac0088a0 con 0x7fa5b8065060 2019-10-13T04:04:55.330 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5b6ffd700 1 -- 172.21.15.110:0/161302586 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ac011930 con 0x7fa5b8065060 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 msgr2=0x7fa5b8144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7fa5b80a22a0 tx=0x7fa5a8002ff0).stop 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- 172.21.15.110:0/161302586 shutdown_connections 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5b8145010 0x7fa5b80a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- 172.21.15.110:0/161302586 >> v2:172.21.15.204:3300/0 conn(0x7fa5b80a6410 0x7fa5b80a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.331 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 --2- 172.21.15.110:0/161302586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b8065060 0x7fa5b8144ac0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- 172.21.15.110:0/161302586 shutdown_connections 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- 172.21.15.110:0/161302586 wait complete. 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.297+0000 7fa5be021700 1 -- 172.21.15.110:0/161302586 >> 172.21.15.110:0/161302586 conn(0x7fa5b805afb0 msgr2=0x7fa5b809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 Processor -- start 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 -- start start 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5b807be10 0x7fa5b807ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.332 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b81229f0 con 0x7fa5b8124790 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b8121e90 con 0x7fa5b807b2a0 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.301+0000 7fa5be021700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b8122210 con 0x7fa5b807be10 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b7fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa5b8124790 0x7fa5b8134300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45692/0 (socket says 172.21.15.110:45692) 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b7fff700 1 -- 172.21.15.110:0/4196637065 learned_addr learned my addr 172.21.15.110:0/4196637065 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> v2:172.21.15.110:3300/0 conn(0x7fa5b807be10 0x7fa5b807ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.333 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 -- 172.21.15.110:0/4196637065 >> v1:172.21.15.204:6789/0 conn(0x7fa5b807b2a0 legacy=0x7fa5b813b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 -- 172.21.15.110:0/4196637065 >> v1:172.21.15.110:6789/0 conn(0x7fa5b8124790 legacy=0x7fa5b8134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360945123 0 0) 0x7fa5b81229f0 con 0x7fa5b8124790 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 -- 172.21.15.110:0/4196637065 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b807b6c0 con 0x7fa5b807be10 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> v2:172.21.15.110:3300/0 conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> v2:172.21.15.110:3300/0 conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.334 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ac007350 con 0x7fa5b807be10 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac011550 con 0x7fa5b807be10 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b80a06d0 con 0x7fa5b807be10 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ac005910 con 0x7fa5b807be10 2019-10-13T04:04:55.335 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b8134b00 con 0x7fa5b807be10 2019-10-13T04:04:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa5ac005c20 con 0x7fa5b807be10 2019-10-13T04:04:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa5980277b0 0x7fa598029cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.336 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa5ac034cb0 con 0x7fa5b807be10 2019-10-13T04:04:55.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.305+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa5980277b0 0x7fa598029cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.309+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa5980277b0 0x7fa598029cb0 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.309+0000 7fa5be021700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] conn(0x7fa5b81232d0 0x7fa5b81236f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.309+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 --> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5b80555c0 con 0x7fa5b81232d0 2019-10-13T04:04:55.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.309+0000 7fa5bcdbf700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] conn(0x7fa5b81232d0 0x7fa5b81236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.313+0000 7fa5bcdbf700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] conn(0x7fa5b81232d0 0x7fa5b81236f0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.313+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== osd.7 v2:172.21.15.204:6802/13913 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fa5b80555c0 con 0x7fa5b81232d0 2019-10-13T04:04:55.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 --> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5b807bbf0 con 0x7fa5b81232d0 2019-10-13T04:04:55.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5b4ff9700 1 -- 172.21.15.110:0/4196637065 <== osd.7 v2:172.21.15.204:6802/13913 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa5b807bbf0 con 0x7fa5b81232d0 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] conn(0x7fa5b81232d0 msgr2=0x7fa5b81236f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6802/13913,v1:172.21.15.204:6803/13913] conn(0x7fa5b81232d0 0x7fa5b81236f0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa5980277b0 msgr2=0x7fa598029cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa5980277b0 0x7fa598029cb0 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 msgr2=0x7fa5b807ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7fa5ac008360 tx=0x7fa5ac00f480).stop 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5bcdbf700 1 -- 172.21.15.110:0/4196637065 reap_dead start 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 shutdown_connections 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 --2- 172.21.15.110:0/4196637065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b807be10 0x7fa5b807ad50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 shutdown_connections 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 wait complete. 2019-10-13T04:04:55.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.337+0000 7fa5be021700 1 -- 172.21.15.110:0/4196637065 >> 172.21.15.110:0/4196637065 conn(0x7fa5b80a27c0 msgr2=0x7fa5b807db30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.353 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:55.354 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:55.354 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-13T04:04:55.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.445+0000 7fa575929700 1 Processor -- start 2019-10-13T04:04:55.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.445+0000 7fa575929700 1 -- start start 2019-10-13T04:04:55.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5700a6410 0x7fa5700a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5700a8c10 0x7fa5700a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:55.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700a77f0 con 0x7fa570065060 2019-10-13T04:04:55.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5700a20e0 con 0x7fa5700a8c10 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700a2380 con 0x7fa5700a6410 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5700a8c10 0x7fa5700a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5700a8c10 0x7fa5700a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42996/0 (socket says 172.21.15.110:42996) 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 -- 172.21.15.110:0/343431779 learned_addr learned my addr 172.21.15.110:0/343431779 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5700a6410 0x7fa5700a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 -- 172.21.15.110:0/343431779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5700a6410 msgr2=0x7fa5700a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:55.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5700a6410 0x7fa5700a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56e7fc700 1 -- 172.21.15.110:0/343431779 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798403768 0 0) 0x7fa5700a77f0 con 0x7fa570065060 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 -- 172.21.15.110:0/343431779 >> v1:172.21.15.204:6789/0 conn(0x7fa570065060 legacy=0x7fa570144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 -- 172.21.15.110:0/343431779 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57014f9f0 con 0x7fa5700a8c10 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> v2:172.21.15.110:3300/0 conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa5700a22a0 tx=0x7fa568007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> v2:172.21.15.110:3300/0 conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa5700a22a0 tx=0x7fa568007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7fa5700a22a0 tx=0x7fa568007a90).ready entity=mon.0 client_cookie=4239f4a4ccf22f94 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fa5700a22a0 tx=0x7fa568007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56e7fc700 1 -- 172.21.15.110:0/343431779 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa560008c80 con 0x7fa5700a8c10 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fa5700a22a0 tx=0x7fa568007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56e7fc700 1 -- 172.21.15.110:0/343431779 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5600088a0 con 0x7fa5700a8c10 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56ffff700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7fa5700a22a0 tx=0x7fa568007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 msgr2=0x7fa5700a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa56e7fc700 1 -- 172.21.15.110:0/343431779 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5600118c0 con 0x7fa5700a8c10 2019-10-13T04:04:55.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7fa5700a22a0 tx=0x7fa568007a90).stop 2019-10-13T04:04:55.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- 172.21.15.110:0/343431779 shutdown_connections 2019-10-13T04:04:55.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5700a8c10 0x7fa5700a9030 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 --2- 172.21.15.110:0/343431779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5700a6410 0x7fa5700a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:55.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- 172.21.15.110:0/343431779 shutdown_connections 2019-10-13T04:04:55.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- 172.21.15.110:0/343431779 wait complete. 2019-10-13T04:04:55.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.449+0000 7fa575929700 1 -- 172.21.15.110:0/343431779 >> 172.21.15.110:0/343431779 conn(0x7fa57005afb0 msgr2=0x7fa57009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 Processor -- start 2019-10-13T04:04:55.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 -- start start 2019-10-13T04:04:55.465 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa57007a790 0x7fa5701174e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570117a30 con 0x7fa5700643b0 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570118070 con 0x7fa570063f90 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa575929700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57007be80 con 0x7fa57007a790 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa5700643b0 0x7fa570132aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45714/0 (socket says 172.21.15.110:45714) 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56ffff700 1 -- 172.21.15.110:0/3797754092 learned_addr learned my addr 172.21.15.110:0/3797754092 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> v2:172.21.15.204:3300/0 conn(0x7fa57007a790 0x7fa5701174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.466 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056485184 0 0) 0x7fa570117a30 con 0x7fa5700643b0 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5540030a0 con 0x7fa5700643b0 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3713618295 0 0) 0x7fa5540030a0 con 0x7fa5700643b0 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 -- 172.21.15.110:0/3797754092 >> v1:172.21.15.204:6789/0 conn(0x7fa570063f90 legacy=0x7fa5701195e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 -- 172.21.15.110:0/3797754092 >> v1:172.21.15.110:6789/0 conn(0x7fa5700643b0 legacy=0x7fa570132aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 -- 172.21.15.110:0/3797754092 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57012b650 con 0x7fa57007a790 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> v2:172.21.15.204:3300/0 conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.467 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> v2:172.21.15.204:3300/0 conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.453+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa568005c80 con 0x7fa57007a790 2019-10-13T04:04:55.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5600114e0 con 0x7fa57007a790 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570119de0 con 0x7fa57007a790 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa568007540 con 0x7fa57007a790 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5701342b0 con 0x7fa57007a790 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa56800e180 con 0x7fa57007a790 2019-10-13T04:04:55.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 0x7fa55402a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:55.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa568034980 con 0x7fa57007a790 2019-10-13T04:04:55.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56effd700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 0x7fa55402a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:55.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.457+0000 7fa56effd700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 0x7fa55402a240 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:55.476 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.461+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5700555c0 con 0x7fa57007a790 2019-10-13T04:04:55.476 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.473+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.477 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.473+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa56800e730 con 0x7fa57007a790 2019-10-13T04:04:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa5700a07a0 con 0x7fa57007a790 2019-10-13T04:04:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa56f7fe700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa567fff700 1 -- 172.21.15.110:0/3797754092 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa56800ebe0 con 0x7fa57007a790 2019-10-13T04:04:55.641 INFO:teuthology.orchestra.run.smithi110.stdout:55834574854 2019-10-13T04:04:55.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 msgr2=0x7fa55402a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa575929700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 0x7fa55402a240 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 msgr2=0x7fa5701174e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.637+0000 7fa575929700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7fa560007700 tx=0x7fa5680110e0).stop 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 shutdown_connections 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa554027d40 0x7fa55402a240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 --2- 172.21.15.110:0/3797754092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa57007a790 0x7fa5701174e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 shutdown_connections 2019-10-13T04:04:55.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 wait complete. 2019-10-13T04:04:55.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:55.641+0000 7fa575929700 1 -- 172.21.15.110:0/3797754092 >> 172.21.15.110:0/3797754092 conn(0x7fa5700a6410 msgr2=0x7fa5700a1620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:55.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2019-10-13T04:04:55.946 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:55.946 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:55.950 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:04:55.950 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:04:56.654 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:56.654 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-13T04:04:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 Processor -- start 2019-10-13T04:04:56.757 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 -- start start 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 0x7f47700a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 0x7f4770144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4770145010 0x7f47700a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47700a20e0 con 0x7f4770145010 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47700a2380 con 0x7f47700a6410 2019-10-13T04:04:56.758 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f47777f0700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47700a77f0 con 0x7f4770065060 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4770145010 0x7f47700a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4770145010 0x7f47700a8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43012/0 (socket says 172.21.15.110:43012) 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 -- 172.21.15.110:0/403754962 learned_addr learned my addr 172.21.15.110:0/403754962 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477558c700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 0x7f4770144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f4775d8d700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 0x7f47700a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 -- 172.21.15.110:0/403754962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 msgr2=0x7f4770144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 0x7f4770144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 -- 172.21.15.110:0/403754962 >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 msgr2=0x7f47700a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 0x7f47700a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 -- 172.21.15.110:0/403754962 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47700a9160 con 0x7f4770145010 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477558c700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 0x7f4770144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f4775d8d700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 0x7f47700a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.749+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.110:3300/0 conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f477005ccf0 tx=0x7f4760003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.760 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.110:3300/0 conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f477005ccf0 tx=0x7f4760003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f477005ccf0 tx=0x7f4760003370).ready entity=mon.0 client_cookie=e918f6baa77c82a5 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f477005ccf0 tx=0x7f4760003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f477005ccf0 tx=0x7f4760003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f4774d8b700 1 -- 172.21.15.110:0/403754962 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47680088a0 con 0x7f4770145010 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f477658e700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f477005ccf0 tx=0x7f4760003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.761 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f4774d8b700 1 -- 172.21.15.110:0/403754962 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47680073c0 con 0x7f4770145010 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f4774d8b700 1 -- 172.21.15.110:0/403754962 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4768006560 con 0x7f4770145010 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 msgr2=0x7f47700a8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f477005ccf0 tx=0x7f4760003370).stop 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- 172.21.15.110:0/403754962 shutdown_connections 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4770145010 0x7f47700a8c10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 --2- 172.21.15.110:0/403754962 >> v2:172.21.15.204:3300/0 conn(0x7f47700a6410 0x7f47700a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:56.762 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 --2- 172.21.15.110:0/403754962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4770065060 0x7f4770144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- 172.21.15.110:0/403754962 shutdown_connections 2019-10-13T04:04:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- 172.21.15.110:0/403754962 wait complete. 2019-10-13T04:04:56.763 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- 172.21.15.110:0/403754962 >> 172.21.15.110:0/403754962 conn(0x7f477005afb0 msgr2=0x7f477009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 Processor -- start 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.753+0000 7f47777f0700 1 -- start start 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4770079e10 con 0x7f4770122270 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477007d1b0 con 0x7f4770121bb0 2019-10-13T04:04:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47701236b0 con 0x7f47700a3dd0 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f4775d8d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4770122270 0x7f4770124d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45726/0 (socket says 172.21.15.110:45726) 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f4775d8d700 1 -- 172.21.15.110:0/3220349165 learned_addr learned my addr 172.21.15.110:0/3220349165 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185927923 0 0) 0x7f4770079e10 con 0x7f4770122270 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47540030a0 con 0x7f4770122270 2019-10-13T04:04:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160879990 0 0) 0x7f477007d1b0 con 0x7f4770121bb0 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770079e10 con 0x7f4770121bb0 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 -- 172.21.15.110:0/3220349165 >> v1:172.21.15.204:6789/0 conn(0x7f4770121bb0 legacy=0x7f4770079610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 -- 172.21.15.110:0/3220349165 >> v1:172.21.15.110:6789/0 conn(0x7f4770122270 legacy=0x7f4770124d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 -- 172.21.15.110:0/3220349165 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47700a0660 con 0x7f47700a3dd0 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758012a00 con 0x7f47700a3dd0 2019-10-13T04:04:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477013cb80 con 0x7f47700a3dd0 2019-10-13T04:04:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4758007160 con 0x7f47700a3dd0 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758005530 con 0x7f47700a3dd0 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477013d050 con 0x7f47700a3dd0 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.757+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4758005810 con 0x7f47700a3dd0 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.773 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f476e7fc700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 0x7f475402a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:56.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4758034b60 con 0x7f47700a3dd0 2019-10-13T04:04:56.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f4775d8d700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 0x7f475402a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:56.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.761+0000 7f4775d8d700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 0x7f475402a4b0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:56.774 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.765+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4770125580 con 0x7f47700a3dd0 2019-10-13T04:04:56.777 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.773+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.777 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.773+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4758005b00 con 0x7f47700a3dd0 2019-10-13T04:04:56.947 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.945+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f47700555c0 con 0x7f47700a3dd0 2019-10-13T04:04:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.945+0000 7f477558c700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.945+0000 7f476e7fc700 1 -- 172.21.15.110:0/3220349165 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4758005d20 con 0x7f47700a3dd0 2019-10-13T04:04:56.948 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:56.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 msgr2=0x7f475402a4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:56.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 0x7f475402a4b0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:56.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 msgr2=0x7f47700790c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:56.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4758002f30 tx=0x7f4758010f50).stop 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 shutdown_connections 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4754027fb0 0x7f475402a4b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 --2- 172.21.15.110:0/3220349165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47700a3dd0 0x7f47700790c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 shutdown_connections 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 wait complete. 2019-10-13T04:04:56.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:56.949+0000 7f47777f0700 1 -- 172.21.15.110:0/3220349165 >> 172.21.15.110:0/3220349165 conn(0x7f47700a6c80 msgr2=0x7f47700a1620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:56.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2019-10-13T04:04:56.967 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:56.967 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-13T04:04:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 Processor -- start 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 -- start start 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae00a1970 0x7fbae014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae00a9410 con 0x7fbae0065060 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae00a1d90 con 0x7fbae00a6410 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae72d2700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00a20e0 con 0x7fbae00a1970 2019-10-13T04:04:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae506e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbae0065060 0x7fbae00a8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45736/0 (socket says 172.21.15.110:45736) 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae506e700 1 -- 172.21.15.110:0/2007142317 learned_addr learned my addr 172.21.15.110:0/2007142317 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae00a1970 0x7fbae014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483738605 0 0) 0x7fbae00a9410 con 0x7fbae0065060 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.061+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbacc0030a0 con 0x7fbae0065060 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 643428050 0 0) 0x7fbacc0030a0 con 0x7fbae0065060 2019-10-13T04:04:57.072 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae00a1970 msgr2=0x7fbae014f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae00a1970 0x7fbae014f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 >> v1:172.21.15.204:6789/0 conn(0x7fbae00a6410 legacy=0x7fbae0144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae00a77f0 con 0x7fbae0065060 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7fbad0001750 con 0x7fbae0065060 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 >> v1:172.21.15.110:6789/0 conn(0x7fbae0065060 legacy=0x7fbae00a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.073 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbacc003700 0x7fbacc003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbacc004410 0x7fbacc004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00a20e0 con 0x7fbacc004d80 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc0051a0 con 0x7fbacc004410 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc04fbc0 con 0x7fbacc003700 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae506e700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbacc004410 0x7fbacc004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae586f700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbacc003700 0x7fbacc003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 -- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbacc003700 msgr2=0x7fbacc003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbacc003700 0x7fbacc003b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 -- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbacc004410 msgr2=0x7fbacc004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbacc004410 0x7fbacc004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 -- 172.21.15.110:0/2007142317 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc04fe90 con 0x7fbacc004d80 2019-10-13T04:04:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).ready entity=mon.0 client_cookie=65531c24d16e803a server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad8007160 con 0x7fbacc004d80 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad8005ac0 con 0x7fbacc004d80 2019-10-13T04:04:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae486d700 1 -- 172.21.15.110:0/2007142317 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad80060c0 con 0x7fbacc004d80 2019-10-13T04:04:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 -- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 msgr2=0x7fbacc00b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 --2- 172.21.15.110:0/2007142317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbacc004d80 0x7fbacc00b660 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7fbacc0053e0 tx=0x7fbad0001c00).stop 2019-10-13T04:04:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae6070700 1 -- 172.21.15.110:0/2007142317 reap_dead start 2019-10-13T04:04:57.078 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 -- 172.21.15.110:0/2007142317 shutdown_connections 2019-10-13T04:04:57.078 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 -- 172.21.15.110:0/2007142317 shutdown_connections 2019-10-13T04:04:57.078 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 -- 172.21.15.110:0/2007142317 wait complete. 2019-10-13T04:04:57.078 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.065+0000 7fbae72d2700 1 -- 172.21.15.110:0/2007142317 >> 172.21.15.110:0/2007142317 conn(0x7fbae005afb0 msgr2=0x7fbae009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.081 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 Processor -- start 2019-10-13T04:04:57.081 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- start start 2019-10-13T04:04:57.081 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbae01210f0 0x7fbae0124710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae0124c60 0x7fbae00790b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae0079600 con 0x7fbae01229f0 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae0122210 con 0x7fbae01210f0 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01224b0 con 0x7fbae0124c60 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbae01210f0 0x7fbae0124710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.082 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbae01210f0 0x7fbae0124710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.110:52448/0 (socket says 172.21.15.110:52448) 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 -- 172.21.15.110:0/655414877 learned_addr learned my addr 172.21.15.110:0/655414877 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae6070700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae0124c60 0x7fbae00790b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 -- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae0124c60 msgr2=0x7fbae00790b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae0124c60 0x7fbae00790b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 -- 172.21.15.110:0/655414877 >> v1:172.21.15.204:6789/0 conn(0x7fbae01229f0 legacy=0x7fbae0123120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 -- 172.21.15.110:0/655414877 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0079830 con 0x7fbae01210f0 2019-10-13T04:04:57.083 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307282381 0 0) 0x7fbae0079600 con 0x7fbae01229f0 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> v2:172.21.15.204:3300/0 conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> v2:172.21.15.204:3300/0 conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad0012ad0 con 0x7fbae01210f0 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.084 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad00126f0 con 0x7fbae01210f0 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad00055f0 con 0x7fbae01210f0 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae013b3a0 con 0x7fbae01210f0 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.069+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae013b950 con 0x7fbae01210f0 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fbad0005890 con 0x7fbae01210f0 2019-10-13T04:04:57.085 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.086 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbad67fc700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 0x7fbac8029c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.086 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbad0035380 con 0x7fbae01210f0 2019-10-13T04:04:57.086 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbae586f700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 0x7fbac8029c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.086 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.073+0000 7fbae586f700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 0x7fbac8029c70 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.086 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.077+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae00555c0 con 0x7fbae01210f0 2019-10-13T04:04:57.088 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.085+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.088 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.085+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fbad0005b80 con 0x7fbae01210f0 2019-10-13T04:04:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.245+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbae0122670 con 0x7fbae01210f0 2019-10-13T04:04:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.245+0000 7fbae506e700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.245+0000 7fbad67fc700 1 -- 172.21.15.110:0/655414877 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbad00261c0 con 0x7fbae01210f0 2019-10-13T04:04:57.250 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 msgr2=0x7fbac8029c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 0x7fbac8029c70 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 msgr2=0x7fbae0124710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fbae00a0920 tx=0x7fbad000b540).stop 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 shutdown_connections 2019-10-13T04:04:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fbac8027770 0x7fbac8029c70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbae0124c60 0x7fbae00790b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 --2- 172.21.15.110:0/655414877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae01210f0 0x7fbae0124710 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 shutdown_connections 2019-10-13T04:04:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 wait complete. 2019-10-13T04:04:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.249+0000 7fbae72d2700 1 -- 172.21.15.110:0/655414877 >> 172.21.15.110:0/655414877 conn(0x7fbae00a8a60 msgr2=0x7fbae00a9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2019-10-13T04:04:57.263 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:57.263 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-10-13T04:04:57.366 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 Processor -- start 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 -- start start 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a6410 0x7f45cc0a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f45cc065060 0x7f45cc144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45cc145010 0x7f45cc0a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45cc0a20e0 con 0x7f45cc0a6410 2019-10-13T04:04:57.367 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45cc0a2380 con 0x7f45cc065060 2019-10-13T04:04:57.368 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d32f8700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc0a77f0 con 0x7f45cc145010 2019-10-13T04:04:57.368 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d1895700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a6410 0x7f45cc0a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.368 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d1895700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a6410 0x7f45cc0a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43042/0 (socket says 172.21.15.110:43042) 2019-10-13T04:04:57.368 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d1895700 1 -- 172.21.15.110:0/2408620557 learned_addr learned my addr 172.21.15.110:0/2408620557 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d1094700 1 --2- 172.21.15.110:0/2408620557 >> v2:172.21.15.204:3300/0 conn(0x7f45cc065060 0x7f45cc144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.357+0000 7f45d2096700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45cc145010 0x7f45cc0a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 -- 172.21.15.110:0/2408620557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45cc145010 msgr2=0x7f45cc0a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45cc145010 0x7f45cc0a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 -- 172.21.15.110:0/2408620557 >> v2:172.21.15.204:3300/0 conn(0x7f45cc065060 msgr2=0x7f45cc144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> v2:172.21.15.204:3300/0 conn(0x7f45cc065060 0x7f45cc144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.369 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 -- 172.21.15.110:0/2408620557 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc0a9160 con 0x7f45cc0a6410 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).ready entity=mon.0 client_cookie=671e12edb262b32b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.370 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d0893700 1 -- 172.21.15.110:0/2408620557 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45c80088a0 con 0x7f45cc0a6410 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d1895700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d0893700 1 -- 172.21.15.110:0/2408620557 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c80073c0 con 0x7f45cc0a6410 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d0893700 1 -- 172.21.15.110:0/2408620557 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45c8006560 con 0x7f45cc0a6410 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 -- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 msgr2=0x7f45cc0a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f45cc0a22a0 tx=0x7f45bc007710).stop 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 -- 172.21.15.110:0/2408620557 shutdown_connections 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45cc145010 0x7f45cc0a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.371 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 --2- 172.21.15.110:0/2408620557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a6410 0x7f45cc0a1970 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 --2- 172.21.15.110:0/2408620557 >> v2:172.21.15.204:3300/0 conn(0x7f45cc065060 0x7f45cc144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 -- 172.21.15.110:0/2408620557 shutdown_connections 2019-10-13T04:04:57.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 -- 172.21.15.110:0/2408620557 wait complete. 2019-10-13T04:04:57.372 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.361+0000 7f45d32f8700 1 -- 172.21.15.110:0/2408620557 >> 172.21.15.110:0/2408620557 conn(0x7f45cc05afb0 msgr2=0x7f45cc09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 Processor -- start 2019-10-13T04:04:57.375 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 -- start start 2019-10-13T04:04:57.376 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a4390 0x7f45cc13b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc123e30 con 0x7f45cc1215e0 2019-10-13T04:04:57.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc0a47b0 con 0x7f45cc121fd0 2019-10-13T04:04:57.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d32f8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45cc0a3dd0 con 0x7f45cc0a4390 2019-10-13T04:04:57.382 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a4390 0x7f45cc13b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a4390 0x7f45cc13b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43052/0 (socket says 172.21.15.110:43052) 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 -- 172.21.15.110:0/3759705774 learned_addr learned my addr 172.21.15.110:0/3759705774 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 -- 172.21.15.110:0/3759705774 >> v1:172.21.15.204:6789/0 conn(0x7f45cc121fd0 legacy=0x7f45cc11fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 -- 172.21.15.110:0/3759705774 >> v1:172.21.15.110:6789/0 conn(0x7f45cc1215e0 legacy=0x7f45cc1203e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 -- 172.21.15.110:0/3759705774 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc0a6920 con 0x7f45cc0a4390 2019-10-13T04:04:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646823182 0 0) 0x7f45cc123e30 con 0x7f45cc1215e0 2019-10-13T04:04:57.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> v2:172.21.15.110:3300/0 conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.384 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.365+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45c4008d50 con 0x7f45cc0a4390 2019-10-13T04:04:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c4013c60 con 0x7f45cc0a4390 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc123160 con 0x7f45cc0a4390 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45c4005c90 con 0x7f45cc0a4390 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc0a9430 con 0x7f45cc0a4390 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f45c400b250 con 0x7f45cc0a4390 2019-10-13T04:04:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 0x7f45b4029c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f45c40370b0 con 0x7f45cc0a4390 2019-10-13T04:04:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d1895700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 0x7f45b4029c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.369+0000 7f45d1895700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 0x7f45b4029c60 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.373+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc121a00 con 0x7f45cc0a4390 2019-10-13T04:04:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.377+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.377+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f45c4028460 con 0x7f45cc0a4390 2019-10-13T04:04:57.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f45cc0555c0 con 0x7f45cc0a4390 2019-10-13T04:04:57.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d2096700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.540 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45c27fc700 1 -- 172.21.15.110:0/3759705774 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45c400b540 con 0x7f45cc0a4390 2019-10-13T04:04:57.541 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:57.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 msgr2=0x7f45b4029c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 0x7f45b4029c60 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 msgr2=0x7f45cc13b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f45c40032b0 tx=0x7f45c400cb00).stop 2019-10-13T04:04:57.543 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 shutdown_connections 2019-10-13T04:04:57.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f45b4027760 0x7f45b4029c60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.537+0000 7f45d32f8700 1 --2- 172.21.15.110:0/3759705774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45cc0a4390 0x7f45cc13b620 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.541+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 shutdown_connections 2019-10-13T04:04:57.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.541+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 wait complete. 2019-10-13T04:04:57.544 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.541+0000 7f45d32f8700 1 -- 172.21.15.110:0/3759705774 >> 172.21.15.110:0/3759705774 conn(0x7f45cc07e0a0 msgr2=0x7f45cc125250 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2019-10-13T04:04:57.553 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:57.553 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-10-13T04:04:57.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 Processor -- start 2019-10-13T04:04:57.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 -- start start 2019-10-13T04:04:57.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 0x7f8f640a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8f640a8c10 0x7f8f640a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640a77f0 con 0x7f8f64065060 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640a20e0 con 0x7f8f640a8c10 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.645+0000 7f8f6975a700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640a2380 con 0x7f8f640a6410 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8f640a8c10 0x7f8f640a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8f640a8c10 0x7f8f640a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43060/0 (socket says 172.21.15.110:43060) 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 -- 172.21.15.110:0/3689259532 learned_addr learned my addr 172.21.15.110:0/3689259532 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f637fe700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 0x7f8f640a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.657 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 -- 172.21.15.110:0/3689259532 >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 msgr2=0x7f8f640a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 0x7f8f640a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 -- 172.21.15.110:0/3689259532 >> v1:172.21.15.204:6789/0 conn(0x7f8f64065060 legacy=0x7f8f64144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 -- 172.21.15.110:0/3689259532 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6414f9f0 con 0x7f8f640a8c10 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f637fe700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 0x7f8f640a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.110:3300/0 conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8f640a15e0 tx=0x7f8f54007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.658 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.110:3300/0 conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8f640a15e0 tx=0x7f8f54007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f8f640a15e0 tx=0x7f8f54007710).ready entity=mon.0 client_cookie=e610ffd4b3d76f0d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f8f640a15e0 tx=0x7f8f54007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f8f640a15e0 tx=0x7f8f54007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f627fc700 1 -- 172.21.15.110:0/3689259532 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5c010c40 con 0x7f8f640a8c10 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f63fff700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f8f640a15e0 tx=0x7f8f54007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f627fc700 1 -- 172.21.15.110:0/3689259532 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c010860 con 0x7f8f640a8c10 2019-10-13T04:04:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f627fc700 1 -- 172.21.15.110:0/3689259532 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5c0114a0 con 0x7f8f640a8c10 2019-10-13T04:04:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 -- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 msgr2=0x7f8f640a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f8f640a15e0 tx=0x7f8f54007710).stop 2019-10-13T04:04:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 -- 172.21.15.110:0/3689259532 shutdown_connections 2019-10-13T04:04:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 --2- 172.21.15.110:0/3689259532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f640a8c10 0x7f8f640a9030 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 --2- 172.21.15.110:0/3689259532 >> v2:172.21.15.204:3300/0 conn(0x7f8f640a6410 0x7f8f640a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.663 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 -- 172.21.15.110:0/3689259532 shutdown_connections 2019-10-13T04:04:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 -- 172.21.15.110:0/3689259532 wait complete. 2019-10-13T04:04:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.649+0000 7f8f6975a700 1 -- 172.21.15.110:0/3689259532 >> 172.21.15.110:0/3689259532 conn(0x7f8f6405afb0 msgr2=0x7f8f6409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 Processor -- start 2019-10-13T04:04:57.664 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 -- start start 2019-10-13T04:04:57.668 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.668 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640a3dd0 con 0x7f8f64075a80 2019-10-13T04:04:57.668 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6411c0a0 con 0x7f8f6411bc80 2019-10-13T04:04:57.668 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6414f770 con 0x7f8f640a4400 2019-10-13T04:04:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f637fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8f64075a80 0x7f8f640a5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45774/0 (socket says 172.21.15.110:45774) 2019-10-13T04:04:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f637fe700 1 -- 172.21.15.110:0/56476572 learned_addr learned my addr 172.21.15.110:0/56476572 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294101175 0 0) 0x7f8f640a3dd0 con 0x7f8f64075a80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f440030a0 con 0x7f8f64075a80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074549310 0 0) 0x7f8f6411c0a0 con 0x7f8f6411bc80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f640a3dd0 con 0x7f8f6411bc80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4169283691 0 0) 0x7f8f440030a0 con 0x7f8f64075a80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f6411c0a0 con 0x7f8f64075a80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f8f54001760 con 0x7f8f64075a80 2019-10-13T04:04:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 -- 172.21.15.110:0/56476572 >> v1:172.21.15.204:6789/0 conn(0x7f8f6411bc80 legacy=0x7f8f6414ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 -- 172.21.15.110:0/56476572 >> v1:172.21.15.110:6789/0 conn(0x7f8f64075a80 legacy=0x7f8f640a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 -- 172.21.15.110:0/56476572 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64075640 con 0x7f8f640a4400 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f5800c950 con 0x7f8f640a4400 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5800c570 con 0x7f8f640a4400 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64131b90 con 0x7f8f640a4400 2019-10-13T04:04:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.653+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f58010dc0 con 0x7f8f640a4400 2019-10-13T04:04:57.674 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f64132090 con 0x7f8f640a4400 2019-10-13T04:04:57.674 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.674 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f8f58007420 con 0x7f8f640a4400 2019-10-13T04:04:57.674 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f4bfff700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 0x7f8f4402a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8f58034920 con 0x7f8f640a4400 2019-10-13T04:04:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f637fe700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 0x7f8f4402a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.657+0000 7f8f637fe700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 0x7f8f4402a8d0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.661+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640555c0 con 0x7f8f640a4400 2019-10-13T04:04:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.669+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.669+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f8f58007710 con 0x7f8f640a4400 2019-10-13T04:04:57.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.833+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8f640a3970 con 0x7f8f640a4400 2019-10-13T04:04:57.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.833+0000 7f8f62ffd700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.833+0000 7f8f4bfff700 1 -- 172.21.15.110:0/56476572 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f58007930 con 0x7f8f640a4400 2019-10-13T04:04:57.838 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 msgr2=0x7f8f4402a8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 0x7f8f4402a8d0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 msgr2=0x7f8f6411a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.842 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f8f58000e90 tx=0x7f8f540014b0).stop 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 shutdown_connections 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f8f440283d0 0x7f8f4402a8d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 --2- 172.21.15.110:0/56476572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f640a4400 0x7f8f6411a0e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 shutdown_connections 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 wait complete. 2019-10-13T04:04:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.837+0000 7f8f6975a700 1 -- 172.21.15.110:0/56476572 >> 172.21.15.110:0/56476572 conn(0x7f8f640a6410 msgr2=0x7f8f6407dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2019-10-13T04:04:57.854 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:57.854 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-13T04:04:57.959 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 Processor -- start 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 -- start start 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb6440a6410 0x7fb6440a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644065060 0x7fb644144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb644145010 0x7fb6440a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6440a20e0 con 0x7fb644145010 2019-10-13T04:04:57.960 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6440a2380 con 0x7fb6440a6410 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.949+0000 7fb64b4da700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440a77f0 con 0x7fb644065060 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb644145010 0x7fb6440a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb649276700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644065060 0x7fb644144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb644145010 0x7fb6440a8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43074/0 (socket says 172.21.15.110:43074) 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb649a77700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb6440a6410 0x7fb6440a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 -- 172.21.15.110:0/3064634652 learned_addr learned my addr 172.21.15.110:0/3064634652 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:57.961 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 -- 172.21.15.110:0/3064634652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644065060 msgr2=0x7fb644144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644065060 0x7fb644144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 -- 172.21.15.110:0/3064634652 >> v2:172.21.15.204:3300/0 conn(0x7fb6440a6410 msgr2=0x7fb6440a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> v2:172.21.15.204:3300/0 conn(0x7fb6440a6410 0x7fb6440a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 -- 172.21.15.110:0/3064634652 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440a9160 con 0x7fb644145010 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> v2:172.21.15.110:3300/0 conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb64405ccf0 tx=0x7fb638007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> v2:172.21.15.110:3300/0 conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb64405ccf0 tx=0x7fb638007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.962 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7fb64405ccf0 tx=0x7fb638007710).ready entity=mon.0 client_cookie=91fdb2b005c8182b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fb64405ccf0 tx=0x7fb638007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fb64405ccf0 tx=0x7fb638007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb648a75700 1 -- 172.21.15.110:0/3064634652 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb63c0070c0 con 0x7fb644145010 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64a278700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7fb64405ccf0 tx=0x7fb638007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb648a75700 1 -- 172.21.15.110:0/3064634652 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c006be0 con 0x7fb644145010 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb648a75700 1 -- 172.21.15.110:0/3064634652 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb63c0076a0 con 0x7fb644145010 2019-10-13T04:04:57.963 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 -- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 msgr2=0x7fb6440a8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.964 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7fb64405ccf0 tx=0x7fb638007710).stop 2019-10-13T04:04:57.964 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 -- 172.21.15.110:0/3064634652 shutdown_connections 2019-10-13T04:04:57.964 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb644145010 0x7fb6440a8c10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:57.964 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 --2- 172.21.15.110:0/3064634652 >> v2:172.21.15.204:3300/0 conn(0x7fb6440a6410 0x7fb6440a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.965 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 --2- 172.21.15.110:0/3064634652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644065060 0x7fb644144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:57.965 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 -- 172.21.15.110:0/3064634652 shutdown_connections 2019-10-13T04:04:57.965 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 -- 172.21.15.110:0/3064634652 wait complete. 2019-10-13T04:04:57.965 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.953+0000 7fb64b4da700 1 -- 172.21.15.110:0/3064634652 >> 172.21.15.110:0/3064634652 conn(0x7fb64405afb0 msgr2=0x7fb64409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:57.966 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 Processor -- start 2019-10-13T04:04:57.966 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- start start 2019-10-13T04:04:57.969 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.969 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644063790 con 0x7fb644063370 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440a0620 con 0x7fb6441208a0 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb644120cc0 con 0x7fb644123430 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649276700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb644063370 0x7fb6441200a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45788/0 (socket says 172.21.15.110:45788) 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649276700 1 -- 172.21.15.110:0/4212303575 learned_addr learned my addr 172.21.15.110:0/4212303575 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188541346 0 0) 0x7fb644063790 con 0x7fb644063370 2019-10-13T04:04:57.970 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c0030a0 con 0x7fb644063370 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16172405 0 0) 0x7fb6440a0620 con 0x7fb6441208a0 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644063790 con 0x7fb6441208a0 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 -- 172.21.15.110:0/4212303575 >> v1:172.21.15.204:6789/0 conn(0x7fb6441208a0 legacy=0x7fb644118380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 -- 172.21.15.110:0/4212303575 >> v1:172.21.15.110:6789/0 conn(0x7fb644063370 legacy=0x7fb6441200a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 -- 172.21.15.110:0/4212303575 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64411bbb0 con 0x7fb644123430 2019-10-13T04:04:57.971 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb638005920 con 0x7fb644123430 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb644119b90 con 0x7fb644123430 2019-10-13T04:04:57.972 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb638007160 con 0x7fb644123430 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6380122f0 con 0x7fb644123430 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.957+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64411a060 con 0x7fb644123430 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fb6380125d0 con 0x7fb644123430 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.973 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb6367fc700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 0x7fb62c02a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:57.974 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb638034e30 con 0x7fb644123430 2019-10-13T04:04:57.974 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb649276700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 0x7fb62c02a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:57.974 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.961+0000 7fb649276700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 0x7fb62c02a4d0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:57.974 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.965+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6440555c0 con 0x7fb644123430 2019-10-13T04:04:57.976 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.973+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:57.977 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:57.973+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fb6380128c0 con 0x7fb644123430 2019-10-13T04:04:58.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb644063a10 con 0x7fb644123430 2019-10-13T04:04:58.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb649a77700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb6367fc700 1 -- 172.21.15.110:0/4212303575 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb638012ae0 con 0x7fb644123430 2019-10-13T04:04:58.140 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:58.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 msgr2=0x7fb62c02a4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 0x7fb62c02a4d0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 msgr2=0x7fb6440a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7fb638002f30 tx=0x7fb638010c30).stop 2019-10-13T04:04:58.143 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 shutdown_connections 2019-10-13T04:04:58.144 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fb62c027fd0 0x7fb62c02a4d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.144 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 --2- 172.21.15.110:0/4212303575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb644123430 0x7fb6440a9600 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.144 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 shutdown_connections 2019-10-13T04:04:58.144 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 wait complete. 2019-10-13T04:04:58.144 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.137+0000 7fb64b4da700 1 -- 172.21.15.110:0/4212303575 >> 172.21.15.110:0/4212303575 conn(0x7fb6440a6410 msgr2=0x7fb6440a3200 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2019-10-13T04:04:58.152 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:58.153 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-10-13T04:04:58.251 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.241+0000 7f14b03b0700 1 Processor -- start 2019-10-13T04:04:58.251 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.241+0000 7f14b03b0700 1 -- start start 2019-10-13T04:04:58.251 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.241+0000 7f14b03b0700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f14a80a6410 0x7f14a80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a80a77f0 con 0x7f14a8065060 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a80a9030 con 0x7f14a80a8c10 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a80a20e0 con 0x7f14a80a6410 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae14c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f14a8065060 0x7f14a8144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45798/0 (socket says 172.21.15.110:45798) 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae14c700 1 -- 172.21.15.110:0/669459350 learned_addr learned my addr 172.21.15.110:0/669459350 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:58.252 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> v2:172.21.15.204:3300/0 conn(0x7f14a80a6410 0x7f14a80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774906760 0 0) 0x7f14a80a77f0 con 0x7f14a8065060 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14940030a0 con 0x7f14a8065060 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2933561278 0 0) 0x7f14940030a0 con 0x7f14a8065060 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 >> v2:172.21.15.204:3300/0 conn(0x7f14a80a6410 msgr2=0x7f14a80a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 --2- 172.21.15.110:0/669459350 >> v2:172.21.15.204:3300/0 conn(0x7f14a80a6410 0x7f14a80a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 >> v1:172.21.15.204:6789/0 conn(0x7f14a80a8c10 legacy=0x7f14a814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80a9260 con 0x7f14a8065060 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f1498001750 con 0x7f14a8065060 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> v2:172.21.15.204:3300/0 conn(0x7f14a80a6410 0x7f14a80a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 >> v1:172.21.15.110:6789/0 conn(0x7f14a8065060 legacy=0x7f14a8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1494004410 0x7f1494004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1494004d80 0x7f149400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80a20e0 con 0x7f14940036d0 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149404fbf0 con 0x7f1494004410 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149404fe30 con 0x7f1494004d80 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae14c700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1494004410 0x7f1494004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14af14e700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1494004d80 0x7f149400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 -- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1494004d80 msgr2=0x7f149400b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1494004d80 0x7f149400b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 -- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1494004410 msgr2=0x7f1494004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1494004410 0x7f1494004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 -- 172.21.15.110:0/669459350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1494050130 con 0x7f14940036d0 2019-10-13T04:04:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14af14e700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1494004d80 0x7f149400b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae14c700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1494004410 0x7f1494004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14af14e700 1 -- 172.21.15.110:0/669459350 reap_dead start 2019-10-13T04:04:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f14a4003070 tx=0x7f1498001c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f14a4003070 tx=0x7f1498001c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f14a4003070 tx=0x7f1498001c20).ready entity=mon.0 client_cookie=537e5cc14bff30ad server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f14a4003070 tx=0x7f1498001c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a400c590 con 0x7f14940036d0 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f14a4003070 tx=0x7f1498001c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ae94d700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f14a4003070 tx=0x7f1498001c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a400ea40 con 0x7f14940036d0 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14ad94b700 1 -- 172.21.15.110:0/669459350 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a400cb70 con 0x7f14940036d0 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 msgr2=0x7f1494003af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f14a4003070 tx=0x7f1498001c20).stop 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- 172.21.15.110:0/669459350 shutdown_connections 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 --2- 172.21.15.110:0/669459350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14940036d0 0x7f1494003af0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- 172.21.15.110:0/669459350 shutdown_connections 2019-10-13T04:04:58.262 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- 172.21.15.110:0/669459350 wait complete. 2019-10-13T04:04:58.262 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.245+0000 7f14b03b0700 1 -- 172.21.15.110:0/669459350 >> 172.21.15.110:0/669459350 conn(0x7f14a805afb0 msgr2=0x7f14a809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 Processor -- start 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 -- start start 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f14a8063220 0x7f14a8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a81250f0 con 0x7f14a807a660 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a8123be0 con 0x7f14a80a9870 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14b03b0700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a80a0960 con 0x7f14a8063220 2019-10-13T04:04:58.266 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14af14e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f14a807a660 0x7f14a813d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45812/0 (socket says 172.21.15.110:45812) 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f14a8063220 0x7f14a8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14af14e700 1 -- 172.21.15.110:0/2785660601 learned_addr learned my addr 172.21.15.110:0/2785660601 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085886230 0 0) 0x7f14a81250f0 con 0x7f14a807a660 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14900030a0 con 0x7f14a807a660 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2912155338 0 0) 0x7f14900030a0 con 0x7f14a807a660 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 -- 172.21.15.110:0/2785660601 >> v1:172.21.15.204:6789/0 conn(0x7f14a80a9870 legacy=0x7f14a81248f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 -- 172.21.15.110:0/2785660601 >> v1:172.21.15.110:6789/0 conn(0x7f14a807a660 legacy=0x7f14a813d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 -- 172.21.15.110:0/2785660601 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a807aa80 con 0x7f14a8063220 2019-10-13T04:04:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> v2:172.21.15.204:3300/0 conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> v2:172.21.15.204:3300/0 conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.249+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a00020e0 con 0x7f14a8063220 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a0001ef0 con 0x7f14a8063220 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1498004360 con 0x7f14a8063220 2019-10-13T04:04:58.269 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a807ad50 con 0x7f14a8063220 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a813ecd0 con 0x7f14a8063220 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f1498010240 con 0x7f14a8063220 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 0x7f149002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1498034200 con 0x7f14a8063220 2019-10-13T04:04:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae94d700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 0x7f149002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.253+0000 7f14ae94d700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 0x7f149002a240 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.257+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a80555c0 con 0x7f14a8063220 2019-10-13T04:04:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.265+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.265+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f1498004920 con 0x7f14a8063220 2019-10-13T04:04:58.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.425+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f14a8066520 con 0x7f14a8063220 2019-10-13T04:04:58.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.425+0000 7f14ae14c700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.430 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.425+0000 7f149f7fe700 1 -- 172.21.15.110:0/2785660601 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1498004b40 con 0x7f14a8063220 2019-10-13T04:04:58.430 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 msgr2=0x7f149002a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 0x7f149002a240 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 msgr2=0x7f14a8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f14a0001250 tx=0x7f14a00022e0).stop 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 shutdown_connections 2019-10-13T04:04:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f1490027d40 0x7f149002a240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 --2- 172.21.15.110:0/2785660601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14a8063220 0x7f14a8063640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 shutdown_connections 2019-10-13T04:04:58.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 wait complete. 2019-10-13T04:04:58.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.429+0000 7f14b03b0700 1 -- 172.21.15.110:0/2785660601 >> 172.21.15.110:0/2785660601 conn(0x7f14a80a6800 msgr2=0x7f14a80a7520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2019-10-13T04:04:58.442 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:58.442 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-10-13T04:04:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 Processor -- start 2019-10-13T04:04:58.546 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 -- start start 2019-10-13T04:04:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fec0a6410 0x7f4fec0a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0a77f0 con 0x7f4fec0a8c10 2019-10-13T04:04:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0a9030 con 0x7f4fec065060 2019-10-13T04:04:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0a20e0 con 0x7f4fec0a6410 2019-10-13T04:04:58.547 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fec0a6410 0x7f4fec0a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fec0a6410 0x7f4fec0a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43104/0 (socket says 172.21.15.110:43104) 2019-10-13T04:04:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 -- 172.21.15.110:0/2051102100 learned_addr learned my addr 172.21.15.110:0/2051102100 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 -- 172.21.15.110:0/2051102100 >> v1:172.21.15.204:6789/0 conn(0x7f4fec065060 legacy=0x7f4fec144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.549 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 -- 172.21.15.110:0/2051102100 >> v1:172.21.15.110:6789/0 conn(0x7f4fec0a8c10 legacy=0x7f4fec14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 -- 172.21.15.110:0/2051102100 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0a9260 con 0x7f4fec0a6410 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> v2:172.21.15.110:3300/0 conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> v2:172.21.15.110:3300/0 conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).ready entity=mon.0 client_cookie=164efd40de6dd053 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff0b71700 1 -- 172.21.15.110:0/2051102100 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe8011510 con 0x7f4fec0a6410 2019-10-13T04:04:58.550 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff0b71700 1 -- 172.21.15.110:0/2051102100 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8008c90 con 0x7f4fec0a6410 2019-10-13T04:04:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff0b71700 1 -- 172.21.15.110:0/2051102100 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe8011af0 con 0x7f4fec0a6410 2019-10-13T04:04:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 -- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 msgr2=0x7f4fec0a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.551 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.537+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f4fe8008780 tx=0x7f4fe800fca0).stop 2019-10-13T04:04:58.552 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 -- 172.21.15.110:0/2051102100 shutdown_connections 2019-10-13T04:04:58.552 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/2051102100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec0a6410 0x7f4fec0a1970 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.552 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 -- 172.21.15.110:0/2051102100 shutdown_connections 2019-10-13T04:04:58.552 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 -- 172.21.15.110:0/2051102100 wait complete. 2019-10-13T04:04:58.553 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 -- 172.21.15.110:0/2051102100 >> 172.21.15.110:0/2051102100 conn(0x7f4fec05afb0 msgr2=0x7f4fec09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.553 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 Processor -- start 2019-10-13T04:04:58.553 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.541+0000 7f4ff35d6700 1 -- start start 2019-10-13T04:04:58.557 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4fec126a70 0x7f4fec136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.557 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fec127bf0 0x7f4fec07d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.558 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec07e1c0 con 0x7f4fec128db0 2019-10-13T04:04:58.558 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec127410 con 0x7f4fec127bf0 2019-10-13T04:04:58.559 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1276b0 con 0x7f4fec126a70 2019-10-13T04:04:58.559 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff2374700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4fec128db0 0x7f4fec07d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45822/0 (socket says 172.21.15.110:45822) 2019-10-13T04:04:58.559 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff2374700 1 -- 172.21.15.110:0/1021751879 learned_addr learned my addr 172.21.15.110:0/1021751879 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:58.559 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> v2:172.21.15.110:3300/0 conn(0x7f4fec127bf0 0x7f4fec07d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4fec126a70 0x7f4fec136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57836219 0 0) 0x7f4fec07e1c0 con 0x7f4fec128db0 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd40030a0 con 0x7f4fec128db0 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 -- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4fec126a70 msgr2=0x7f4fec136830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4fec126a70 0x7f4fec136830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.560 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 -- 172.21.15.110:0/1021751879 >> v1:172.21.15.110:6789/0 conn(0x7f4fec128db0 legacy=0x7f4fec07d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 -- 172.21.15.110:0/1021751879 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec126f70 con 0x7f4fec127bf0 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> v2:172.21.15.110:3300/0 conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> v2:172.21.15.110:3300/0 conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fdc00cb60 con 0x7f4fec127bf0 2019-10-13T04:04:58.561 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00c780 con 0x7f4fec127bf0 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fdc010e70 con 0x7f4fec127bf0 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec126120 con 0x7f4fec127bf0 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec136d80 con 0x7f4fec127bf0 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.562 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f4fdc011140 con 0x7f4fec127bf0 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 0x7f4fd402a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4fdc034b80 con 0x7f4fec127bf0 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.545+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 0x7f4fd402a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.549+0000 7f4ff1b73700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 0x7f4fd402a260 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.563 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.553+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec0555c0 con 0x7f4fec127bf0 2019-10-13T04:04:58.564 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.557+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.564 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.557+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f4fdc025f10 con 0x7f4fec127bf0 2019-10-13T04:04:58.718 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.713+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4fec1282a0 con 0x7f4fec127bf0 2019-10-13T04:04:58.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.713+0000 7f4ff1372700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.719 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.713+0000 7f4fe27fc700 1 -- 172.21.15.110:0/1021751879 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4fdc010100 con 0x7f4fec127bf0 2019-10-13T04:04:58.719 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:58.721 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 msgr2=0x7f4fd402a260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 0x7f4fd402a260 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 msgr2=0x7f4fec07d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7f4fec1291f0 tx=0x7f4fe80100b0).stop 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 shutdown_connections 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4fec126a70 0x7f4fec136830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.722 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f4fd4027d60 0x7f4fd402a260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.723 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 --2- 172.21.15.110:0/1021751879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fec127bf0 0x7f4fec07d470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.723 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 shutdown_connections 2019-10-13T04:04:58.723 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 wait complete. 2019-10-13T04:04:58.723 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.717+0000 7f4ff35d6700 1 -- 172.21.15.110:0/1021751879 >> 172.21.15.110:0/1021751879 conn(0x7f4fec05aef0 msgr2=0x7f4fec059300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2019-10-13T04:04:58.731 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:58.731 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-10-13T04:04:58.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.813+0000 7f702272f700 1 Processor -- start 2019-10-13T04:04:58.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.813+0000 7f702272f700 1 -- start start 2019-10-13T04:04:58.823 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f701c0a6410 0x7f701c0a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f701c065060 0x7f701c144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0a9410 con 0x7f701c145010 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c0a20e0 con 0x7f701c0a6410 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0a2380 con 0x7f701c065060 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f701c0a6410 0x7f701c0a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.824 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f701c0a6410 0x7f701c0a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43118/0 (socket says 172.21.15.110:43118) 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 -- 172.21.15.110:0/1414399705 learned_addr learned my addr 172.21.15.110:0/1414399705 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f701bfff700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f701c065060 0x7f701c144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 -- 172.21.15.110:0/1414399705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f701c065060 msgr2=0x7f701c144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f701c065060 0x7f701c144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 -- 172.21.15.110:0/1414399705 >> v1:172.21.15.110:6789/0 conn(0x7f701c145010 legacy=0x7f701c0a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 -- 172.21.15.110:0/1414399705 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c0a77f0 con 0x7f701c0a6410 2019-10-13T04:04:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> v2:172.21.15.110:3300/0 conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f700c008b00 tx=0x7f7004007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> v2:172.21.15.110:3300/0 conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f700c008b00 tx=0x7f7004007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f700c008b00 tx=0x7f7004007710).ready entity=mon.0 client_cookie=3d7065124636712b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f700c008b00 tx=0x7f7004007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f701b7fe700 1 -- 172.21.15.110:0/1414399705 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c010c40 con 0x7f701c0a6410 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f700c008b00 tx=0x7f7004007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f701b7fe700 1 -- 172.21.15.110:0/1414399705 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c010860 con 0x7f701c0a6410 2019-10-13T04:04:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f7020ccc700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f700c008b00 tx=0x7f7004007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f701b7fe700 1 -- 172.21.15.110:0/1414399705 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c0114a0 con 0x7f701c0a6410 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 msgr2=0x7f701c0a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f700c008b00 tx=0x7f7004007710).stop 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- 172.21.15.110:0/1414399705 shutdown_connections 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c0a6410 0x7f701c0a1970 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 --2- 172.21.15.110:0/1414399705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f701c065060 0x7f701c144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- 172.21.15.110:0/1414399705 shutdown_connections 2019-10-13T04:04:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- 172.21.15.110:0/1414399705 wait complete. 2019-10-13T04:04:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.817+0000 7f702272f700 1 -- 172.21.15.110:0/1414399705 >> 172.21.15.110:0/1414399705 conn(0x7f701c05afb0 msgr2=0x7f701c09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:58.831 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 Processor -- start 2019-10-13T04:04:58.831 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 -- start start 2019-10-13T04:04:58.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f701c121490 0x7f701c0a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f701c0a5ac0 0x7f701c134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0a3620 con 0x7f701c134230 2019-10-13T04:04:58.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c123880 con 0x7f701c121490 2019-10-13T04:04:58.832 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f702272f700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c124220 con 0x7f701c0a5ac0 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f7020ccc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f701c121490 0x7f701c0a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f701bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f701c134230 0x7f701c0a52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45838/0 (socket says 172.21.15.110:45838) 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f701bfff700 1 -- 172.21.15.110:0/495232151 learned_addr learned my addr 172.21.15.110:0/495232151 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.821+0000 7f70214cd700 1 --2- 172.21.15.110:0/495232151 >> v2:172.21.15.204:3300/0 conn(0x7f701c0a5ac0 0x7f701c134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 -- 172.21.15.110:0/495232151 >> v2:172.21.15.204:3300/0 conn(0x7f701c0a5ac0 msgr2=0x7f701c134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486067420 0 0) 0x7f7004000e90 con 0x7f701c134230 2019-10-13T04:04:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> v2:172.21.15.204:3300/0 conn(0x7f701c0a5ac0 0x7f701c134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 -- 172.21.15.110:0/495232151 >> v1:172.21.15.110:6789/0 conn(0x7f701c134230 legacy=0x7f701c0a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 -- 172.21.15.110:0/495232151 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c123f60 con 0x7f701c121490 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> v2:172.21.15.110:3300/0 conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> v2:172.21.15.110:3300/0 conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c005aa0 con 0x7f701c121490 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0110c0 con 0x7f701c121490 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c13b680 con 0x7f701c121490 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c00f680 con 0x7f701c121490 2019-10-13T04:04:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c13b890 con 0x7f701c121490 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f700c00f9a0 con 0x7f701c121490 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 0x7f7000029c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f700c034fc0 con 0x7f701c121490 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f701bfff700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 0x7f7000029c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.825+0000 7f701bfff700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 0x7f7000029c70 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.829+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c1218b0 con 0x7f701c121490 2019-10-13T04:04:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.837+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:58.837+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f701c123f60 con 0x7f701c121490 2019-10-13T04:04:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.001+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f701c0555c0 con 0x7f701c121490 2019-10-13T04:04:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.001+0000 7f7020ccc700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.001+0000 7f70197fa700 1 -- 172.21.15.110:0/495232151 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f700c01dcb0 con 0x7f701c121490 2019-10-13T04:04:59.007 INFO:teuthology.orchestra.run.smithi110.stdout:55834574855 2019-10-13T04:04:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 msgr2=0x7f7000029c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 0x7f7000029c70 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 msgr2=0x7f701c0a30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7f701c123a40 tx=0x7f70140011c0).stop 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 shutdown_connections 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f7000027770 0x7f7000029c70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 --2- 172.21.15.110:0/495232151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f701c121490 0x7f701c0a30d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 --2- 172.21.15.110:0/495232151 >> v2:172.21.15.204:3300/0 conn(0x7f701c0a5ac0 0x7f701c134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.010 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 shutdown_connections 2019-10-13T04:04:59.011 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 wait complete. 2019-10-13T04:04:59.011 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.005+0000 7f702272f700 1 -- 172.21.15.110:0/495232151 >> 172.21.15.110:0/495232151 conn(0x7f701c0a0aa0 msgr2=0x7f701c09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2019-10-13T04:04:59.018 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-10-13T04:04:59.019 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:59.019 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T04:04:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.105+0000 7f22ae223700 1 Processor -- start 2019-10-13T04:04:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.105+0000 7f22ae223700 1 -- start start 2019-10-13T04:04:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22a80a1970 0x7f22a814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.123 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80a9410 con 0x7f22a8065060 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80a1d90 con 0x7f22a80a6410 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80a20e0 con 0x7f22a80a1970 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a77fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f22a8065060 0x7f22a80a8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45846/0 (socket says 172.21.15.110:45846) 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a77fe700 1 -- 172.21.15.110:0/2829568997 learned_addr learned my addr 172.21.15.110:0/2829568997 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22acfc1700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22a80a1970 0x7f22a814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.124 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799330677 0 0) 0x7f22a80a9410 con 0x7f22a8065060 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22940030a0 con 0x7f22a8065060 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258672927 0 0) 0x7f22a80a1d90 con 0x7f22a80a6410 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a80a9410 con 0x7f22a80a6410 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1785103350 0 0) 0x7f22940030a0 con 0x7f22a8065060 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22a80a1970 msgr2=0x7f22a814f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:59.125 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22a80a1970 0x7f22a814f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 >> v1:172.21.15.204:6789/0 conn(0x7f22a80a6410 legacy=0x7f22a8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80a77f0 con 0x7f22a8065060 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f2290001750 con 0x7f22a8065060 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 >> v1:172.21.15.110:6789/0 conn(0x7f22a8065060 legacy=0x7f22a80a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2294004410 0x7f2294004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.126 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2294004e60 0x7f229400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80a20e0 con 0x7f2294003930 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229404fbc0 con 0x7f2294004e60 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229404fe00 con 0x7f2294004410 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a77fe700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2294004410 0x7f2294004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22acfc1700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2294004e60 0x7f229400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.127 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 -- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2294004410 msgr2=0x7f2294004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2294004410 0x7f2294004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 -- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2294004e60 msgr2=0x7f229400b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2294004e60 0x7f229400b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 -- 172.21.15.110:0/2829568997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22940500a0 con 0x7f2294003930 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a77fe700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2294004410 0x7f2294004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:04:59.128 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f22a0001000 tx=0x7f2298007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f22a0001000 tx=0x7f2298007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f22a0001000 tx=0x7f2298007c30).ready entity=mon.0 client_cookie=d820a2e52ccdf506 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f22a0001000 tx=0x7f2298007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a000c600 con 0x7f2294003930 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f22a0001000 tx=0x7f2298007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.129 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a0001340 con 0x7f2294003930 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a7fff700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f22a0001000 tx=0x7f2298007c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 -- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 msgr2=0x7f2294003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22a6ffd700 1 -- 172.21.15.110:0/2829568997 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a000cbe0 con 0x7f2294003930 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22ae223700 1 --2- 172.21.15.110:0/2829568997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2294003930 0x7f2294003d50 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f22a0001000 tx=0x7f2298007c30).stop 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.109+0000 7f22acfc1700 1 -- 172.21.15.110:0/2829568997 reap_dead start 2019-10-13T04:04:59.130 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- 172.21.15.110:0/2829568997 shutdown_connections 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- 172.21.15.110:0/2829568997 shutdown_connections 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- 172.21.15.110:0/2829568997 wait complete. 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- 172.21.15.110:0/2829568997 >> 172.21.15.110:0/2829568997 conn(0x7f22a805afb0 msgr2=0x7f22a809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 Processor -- start 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- start start 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f22a81247c0 0x7f22a813c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.131 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81306e0 con 0x7f22a81236a0 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a8124dc0 con 0x7f22a81243a0 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22ae223700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a8133d40 con 0x7f22a81247c0 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22a7fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f22a81236a0 0x7f22a8130f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45856/0 (socket says 172.21.15.110:45856) 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22a7fff700 1 -- 172.21.15.110:0/777442236 learned_addr learned my addr 172.21.15.110:0/777442236 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22acfc1700 1 --2- 172.21.15.110:0/777442236 >> v2:172.21.15.204:3300/0 conn(0x7f22a81247c0 0x7f22a813c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.132 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115703259 0 0) 0x7f22a81306e0 con 0x7f22a81236a0 2019-10-13T04:04:59.133 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.113+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0030a0 con 0x7f22a81236a0 2019-10-13T04:04:59.133 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470388922 0 0) 0x7f22a8124dc0 con 0x7f22a81243a0 2019-10-13T04:04:59.133 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a81306e0 con 0x7f22a81243a0 2019-10-13T04:04:59.133 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1650927830 0 0) 0x7f228c0030a0 con 0x7f22a81236a0 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> v1:172.21.15.110:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a8124dc0 con 0x7f22a81236a0 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f22a000ce00 con 0x7f22a81236a0 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 >> v2:172.21.15.204:3300/0 conn(0x7f22a81247c0 msgr2=0x7f22a813c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 --2- 172.21.15.110:0/777442236 >> v2:172.21.15.204:3300/0 conn(0x7f22a81247c0 0x7f22a813c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 >> v1:172.21.15.204:6789/0 conn(0x7f22a81243a0 legacy=0x7f22a8134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.134 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 >> v1:172.21.15.110:6789/0 conn(0x7f22a81236a0 legacy=0x7f22a8130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f228c003fb0 0x7f228c004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f228c009650 0x7f228c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a8133d40 con 0x7f228c003aa0 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c00bfd0 con 0x7f228c003fb0 2019-10-13T04:04:59.135 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c00c190 con 0x7f228c009650 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a77fe700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f228c003fb0 0x7f228c004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22acfc1700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f228c009650 0x7f228c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 -- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f228c009650 msgr2=0x7f228c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f228c009650 0x7f228c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 -- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f228c003fb0 msgr2=0x7f228c004a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.136 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f228c003fb0 0x7f228c004a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a8134a80 con 0x7f228c003aa0 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22acfc1700 1 -- 172.21.15.110:0/777442236 reap_dead start 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a80a35b0 con 0x7f228c003aa0 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a81236a0 con 0x7f228c003aa0 2019-10-13T04:04:59.137 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a00113e0 con 0x7f228c003aa0 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.138 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a000bc60 con 0x7f228c003aa0 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a00119c0 con 0x7f228c003aa0 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f22a0011bc0 con 0x7f228c003aa0 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 0x7f228c02da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.117+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f22a0034f20 con 0x7f228c003aa0 2019-10-13T04:04:59.139 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.121+0000 7f22a77fe700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 0x7f228c02da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.121+0000 7f22a77fe700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 0x7f228c02da10 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.140 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.121+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22800031f0 con 0x7f228c003aa0 2019-10-13T04:04:59.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.129+0000 7f22a7fff700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.141 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.129+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f22a001db10 con 0x7f228c003aa0 2019-10-13T04:04:59.289 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.285+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 --> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2280002000 con 0x7f228c02b510 2019-10-13T04:04:59.296 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.293+0000 7f22a4ff9700 1 -- 172.21.15.110:0/777442236 <== mgr.4105 v2:172.21.15.204:6800/13789 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80438 (crc 0 0 0) 0x7f2280002000 con 0x7f228c02b510 2019-10-13T04:04:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2019-10-13T04:04:59.305 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:59.307 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 msgr2=0x7f228c02da10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.307 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 0x7f228c02da10 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 msgr2=0x7f228c004410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7f22a0000f10 tx=0x7f2290000f20).stop 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 shutdown_connections 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f228c02b510 0x7f228c02da10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 --2- 172.21.15.110:0/777442236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f228c003aa0 0x7f228c004410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.308 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 shutdown_connections 2019-10-13T04:04:59.309 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 wait complete. 2019-10-13T04:04:59.309 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.305+0000 7f22ae223700 1 -- 172.21.15.110:0/777442236 >> 172.21.15.110:0/777442236 conn(0x7f22a80a6cf0 msgr2=0x7f22a809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.319 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-10-13T04:04:59.244554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8395136,"kb_used_data":6400,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746579584,"statfs":{"total":773094113280,"available":764497494016,"internally_reserved":8589934592,"allocated":6553600,"data_stored":719508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001979"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432636+0000","last_change":"2019-10-13T04:04:48.431642+0000","last_active":"2019-10-13T04:04:48.432636+0000","last_peered":"2019-10-13T04:04:48.432636+0000","last_clean":"2019-10-13T04:04:48.432636+0000","last_became_active":"2019-10-13T04:04:48.431123+0000","last_became_peered":"2019-10-13T04:04:48.431123+0000","last_unstale":"2019-10-13T04:04:48.432636+0000","last_undegraded":"2019-10-13T04:04:48.432636+0000","last_fullsized":"2019-10-13T04:04:48.432636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.417799+0000","last_change":"2019-10-13T04:04:42.618332+0000","last_active":"2019-10-13T04:04:48.417799+0000","last_peered":"2019-10-13T04:04:48.417799+0000","last_clean":"2019-10-13T04:04:48.417799+0000","last_became_active":"2019-10-13T04:04:42.617512+0000","last_became_peered":"2019-10-13T04:04:42.617512+0000","last_unstale":"2019-10-13T04:04:48.417799+0000","last_undegraded":"2019-10-13T04:04:48.417799+0000","last_fullsized":"2019-10-13T04:04:48.417799+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'4","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.622734+0000","last_change":"2019-10-13T04:04:45.404135+0000","last_active":"2019-10-13T04:04:51.622734+0000","last_peered":"2019-10-13T04:04:51.622734+0000","last_clean":"2019-10-13T04:04:51.622734+0000","last_became_active":"2019-10-13T04:04:45.403719+0000","last_became_peered":"2019-10-13T04:04:45.403719+0000","last_unstale":"2019-10-13T04:04:51.622734+0000","last_undegraded":"2019-10-13T04:04:51.622734+0000","last_fullsized":"2019-10-13T04:04:51.622734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432219+0000","last_change":"2019-10-13T04:04:48.430799+0000","last_active":"2019-10-13T04:04:48.432219+0000","last_peered":"2019-10-13T04:04:48.432219+0000","last_clean":"2019-10-13T04:04:48.432219+0000","last_became_active":"2019-10-13T04:04:48.430286+0000","last_became_peered":"2019-10-13T04:04:48.430286+0000","last_unstale":"2019-10-13T04:04:48.432219+0000","last_undegraded":"2019-10-13T04:04:48.432219+0000","last_fullsized":"2019-10-13T04:04:48.432219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.420869+0000","last_change":"2019-10-13T04:04:42.383533+0000","last_active":"2019-10-13T04:04:48.420869+0000","last_peered":"2019-10-13T04:04:48.420869+0000","last_clean":"2019-10-13T04:04:48.420869+0000","last_became_active":"2019-10-13T04:04:42.383201+0000","last_became_peered":"2019-10-13T04:04:42.383201+0000","last_unstale":"2019-10-13T04:04:48.420869+0000","last_undegraded":"2019-10-13T04:04:48.420869+0000","last_fullsized":"2019-10-13T04:04:48.420869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'7","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.624734+0000","last_change":"2019-10-13T04:04:45.398996+0000","last_active":"2019-10-13T04:04:51.624734+0000","last_peered":"2019-10-13T04:04:51.624734+0000","last_clean":"2019-10-13T04:04:51.624734+0000","last_became_active":"2019-10-13T04:04:45.398640+0000","last_became_peered":"2019-10-13T04:04:45.398640+0000","last_unstale":"2019-10-13T04:04:51.624734+0000","last_undegraded":"2019-10-13T04:04:51.624734+0000","last_fullsized":"2019-10-13T04:04:51.624734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.431909+0000","last_change":"2019-10-13T04:04:48.431132+0000","last_active":"2019-10-13T04:04:48.431909+0000","last_peered":"2019-10-13T04:04:48.431909+0000","last_clean":"2019-10-13T04:04:48.431909+0000","last_became_active":"2019-10-13T04:04:48.430617+0000","last_became_peered":"2019-10-13T04:04:48.430617+0000","last_unstale":"2019-10-13T04:04:48.431909+0000","last_undegraded":"2019-10-13T04:04:48.431909+0000","last_fullsized":"2019-10-13T04:04:48.431909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.421933+0000","last_change":"2019-10-13T04:04:42.384347+0000","last_active":"2019-10-13T04:04:48.421933+0000","last_peered":"2019-10-13T04:04:48.421933+0000","last_clean":"2019-10-13T04:04:48.421933+0000","last_became_active":"2019-10-13T04:04:42.383983+0000","last_became_peered":"2019-10-13T04:04:42.383983+0000","last_unstale":"2019-10-13T04:04:48.421933+0000","last_undegraded":"2019-10-13T04:04:48.421933+0000","last_fullsized":"2019-10-13T04:04:48.421933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.625379+0000","last_change":"2019-10-13T04:04:45.398273+0000","last_active":"2019-10-13T04:04:51.625379+0000","last_peered":"2019-10-13T04:04:51.625379+0000","last_clean":"2019-10-13T04:04:51.625379+0000","last_became_active":"2019-10-13T04:04:45.397947+0000","last_became_peered":"2019-10-13T04:04:45.397947+0000","last_unstale":"2019-10-13T04:04:51.625379+0000","last_undegraded":"2019-10-13T04:04:51.625379+0000","last_fullsized":"2019-10-13T04:04:51.625379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.563853+0000","last_change":"2019-10-13T04:04:48.563287+0000","last_active":"2019-10-13T04:04:48.563853+0000","last_peered":"2019-10-13T04:04:48.563853+0000","last_clean":"2019-10-13T04:04:48.563853+0000","last_became_active":"2019-10-13T04:04:48.562589+0000","last_became_peered":"2019-10-13T04:04:48.562589+0000","last_unstale":"2019-10-13T04:04:48.563853+0000","last_undegraded":"2019-10-13T04:04:48.563853+0000","last_fullsized":"2019-10-13T04:04:48.563853+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.419827+0000","last_change":"2019-10-13T04:04:42.483969+0000","last_active":"2019-10-13T04:04:48.419827+0000","last_peered":"2019-10-13T04:04:48.419827+0000","last_clean":"2019-10-13T04:04:48.419827+0000","last_became_active":"2019-10-13T04:04:42.483339+0000","last_became_peered":"2019-10-13T04:04:42.483339+0000","last_unstale":"2019-10-13T04:04:48.419827+0000","last_undegraded":"2019-10-13T04:04:48.419827+0000","last_fullsized":"2019-10-13T04:04:48.419827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.417706+0000","last_change":"2019-10-13T04:04:45.488583+0000","last_active":"2019-10-13T04:04:48.417706+0000","last_peered":"2019-10-13T04:04:48.417706+0000","last_clean":"2019-10-13T04:04:48.417706+0000","last_became_active":"2019-10-13T04:04:45.488145+0000","last_became_peered":"2019-10-13T04:04:45.488145+0000","last_unstale":"2019-10-13T04:04:48.417706+0000","last_undegraded":"2019-10-13T04:04:48.417706+0000","last_fullsized":"2019-10-13T04:04:48.417706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.564756+0000","last_change":"2019-10-13T04:04:48.564321+0000","last_active":"2019-10-13T04:04:48.564756+0000","last_peered":"2019-10-13T04:04:48.564756+0000","last_clean":"2019-10-13T04:04:48.564756+0000","last_became_active":"2019-10-13T04:04:48.562968+0000","last_became_peered":"2019-10-13T04:04:48.562968+0000","last_unstale":"2019-10-13T04:04:48.564756+0000","last_undegraded":"2019-10-13T04:04:48.564756+0000","last_fullsized":"2019-10-13T04:04:48.564756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.419991+0000","last_change":"2019-10-13T04:04:42.482494+0000","last_active":"2019-10-13T04:04:48.419991+0000","last_peered":"2019-10-13T04:04:48.419991+0000","last_clean":"2019-10-13T04:04:48.419991+0000","last_became_active":"2019-10-13T04:04:42.481467+0000","last_became_peered":"2019-10-13T04:04:42.481467+0000","last_unstale":"2019-10-13T04:04:48.419991+0000","last_undegraded":"2019-10-13T04:04:48.419991+0000","last_fullsized":"2019-10-13T04:04:48.419991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'5","reported_seq":"17","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.625409+0000","last_change":"2019-10-13T04:04:45.404217+0000","last_active":"2019-10-13T04:04:51.625409+0000","last_peered":"2019-10-13T04:04:51.625409+0000","last_clean":"2019-10-13T04:04:51.625409+0000","last_became_active":"2019-10-13T04:04:45.403737+0000","last_became_peered":"2019-10-13T04:04:45.403737+0000","last_unstale":"2019-10-13T04:04:51.625409+0000","last_undegraded":"2019-10-13T04:04:51.625409+0000","last_fullsized":"2019-10-13T04:04:51.625409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.561909+0000","last_change":"2019-10-13T04:04:48.560674+0000","last_active":"2019-10-13T04:04:48.561909+0000","last_peered":"2019-10-13T04:04:48.561909+0000","last_clean":"2019-10-13T04:04:48.561909+0000","last_became_active":"2019-10-13T04:04:48.560121+0000","last_became_peered":"2019-10-13T04:04:48.560121+0000","last_unstale":"2019-10-13T04:04:48.561909+0000","last_undegraded":"2019-10-13T04:04:48.561909+0000","last_fullsized":"2019-10-13T04:04:48.561909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.424872+0000","last_change":"2019-10-13T04:04:42.482487+0000","last_active":"2019-10-13T04:04:48.424872+0000","last_peered":"2019-10-13T04:04:48.424872+0000","last_clean":"2019-10-13T04:04:48.424872+0000","last_became_active":"2019-10-13T04:04:42.481484+0000","last_became_peered":"2019-10-13T04:04:42.481484+0000","last_unstale":"2019-10-13T04:04:48.424872+0000","last_undegraded":"2019-10-13T04:04:48.424872+0000","last_fullsized":"2019-10-13T04:04:48.424872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:48.721512+0000","last_change":"2019-10-13T04:04:45.404034+0000","last_active":"2019-10-13T04:04:48.721512+0000","last_peered":"2019-10-13T04:04:48.721512+0000","last_clean":"2019-10-13T04:04:48.721512+0000","last_became_active":"2019-10-13T04:04:45.403650+0000","last_became_peered":"2019-10-13T04:04:45.403650+0000","last_unstale":"2019-10-13T04:04:48.721512+0000","last_undegraded":"2019-10-13T04:04:48.721512+0000","last_fullsized":"2019-10-13T04:04:48.721512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432181+0000","last_change":"2019-10-13T04:04:48.431396+0000","last_active":"2019-10-13T04:04:48.432181+0000","last_peered":"2019-10-13T04:04:48.432181+0000","last_clean":"2019-10-13T04:04:48.432181+0000","last_became_active":"2019-10-13T04:04:48.430913+0000","last_became_peered":"2019-10-13T04:04:48.430913+0000","last_unstale":"2019-10-13T04:04:48.432181+0000","last_undegraded":"2019-10-13T04:04:48.432181+0000","last_fullsized":"2019-10-13T04:04:48.432181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.558186+0000","last_change":"2019-10-13T04:04:42.484231+0000","last_active":"2019-10-13T04:04:48.558186+0000","last_peered":"2019-10-13T04:04:48.558186+0000","last_clean":"2019-10-13T04:04:48.558186+0000","last_became_active":"2019-10-13T04:04:42.483381+0000","last_became_peered":"2019-10-13T04:04:42.483381+0000","last_unstale":"2019-10-13T04:04:48.558186+0000","last_undegraded":"2019-10-13T04:04:48.558186+0000","last_fullsized":"2019-10-13T04:04:48.558186+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"23'6","reported_seq":"18","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.626065+0000","last_change":"2019-10-13T04:04:45.485691+0000","last_active":"2019-10-13T04:04:51.626065+0000","last_peered":"2019-10-13T04:04:51.626065+0000","last_clean":"2019-10-13T04:04:51.626065+0000","last_became_active":"2019-10-13T04:04:45.485029+0000","last_became_peered":"2019-10-13T04:04:45.485029+0000","last_unstale":"2019-10-13T04:04:51.626065+0000","last_undegraded":"2019-10-13T04:04:51.626065+0000","last_fullsized":"2019-10-13T04:04:51.626065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.563919+0000","last_change":"2019-10-13T04:04:48.561119+0000","last_active":"2019-10-13T04:04:48.563919+0000","last_peered":"2019-10-13T04:04:48.563919+0000","last_clean":"2019-10-13T04:04:48.563919+0000","last_became_active":"2019-10-13T04:04:48.560473+0000","last_became_peered":"2019-10-13T04:04:48.560473+0000","last_unstale":"2019-10-13T04:04:48.563919+0000","last_undegraded":"2019-10-13T04:04:48.563919+0000","last_fullsized":"2019-10-13T04:04:48.563919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.558728+0000","last_change":"2019-10-13T04:04:42.381380+0000","last_active":"2019-10-13T04:04:48.558728+0000","last_peered":"2019-10-13T04:04:48.558728+0000","last_clean":"2019-10-13T04:04:48.558728+0000","last_became_active":"2019-10-13T04:04:42.381046+0000","last_became_peered":"2019-10-13T04:04:42.381046+0000","last_unstale":"2019-10-13T04:04:48.558728+0000","last_undegraded":"2019-10-13T04:04:48.558728+0000","last_fullsized":"2019-10-13T04:04:48.558728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.626109+0000","last_change":"2019-10-13T04:04:45.488504+0000","last_active":"2019-10-13T04:04:51.626109+0000","last_peered":"2019-10-13T04:04:51.626109+0000","last_clean":"2019-10-13T04:04:51.626109+0000","last_became_active":"2019-10-13T04:04:45.488054+0000","last_became_peered":"2019-10-13T04:04:45.488054+0000","last_unstale":"2019-10-13T04:04:51.626109+0000","last_undegraded":"2019-10-13T04:04:51.626109+0000","last_fullsized":"2019-10-13T04:04:51.626109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049424,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322416,"statfs":{"total":96636764160,"available":95562153984,"internally_reserved":1073741824,"allocated":851968,"data_stored":90093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.198}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.154}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.283}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.107}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.059}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.239}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.035}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.633}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.569}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.320}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.259}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.897}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.593}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.684}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.564}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.750}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.988}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.818}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.643}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049440,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322400,"statfs":{"total":96636764160,"available":95562137600,"internally_reserved":1073741824,"allocated":868352,"data_stored":91053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.976}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.937}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.205}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.279}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.525}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.162}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.204}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.125}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.366}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.429}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.802}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.862}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.718}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.068}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.131}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.274}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.477}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.813}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.661}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.733}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.138}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.094}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.356}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.042}]}]}]}} 2019-10-13T04:04:59.322 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:59.322 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-13T04:04:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 Processor -- start 2019-10-13T04:04:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 -- start start 2019-10-13T04:04:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.424 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80a77f0 con 0x7ff3e80a8c10 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80a9030 con 0x7ff3e8065060 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ee3c5700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80a20e0 con 0x7ff3e80a6410 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ed163700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff3e80a8c10 0x7ff3e814e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45874/0 (socket says 172.21.15.110:45874) 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.413+0000 7ff3ed163700 1 -- 172.21.15.110:0/1957041133 learned_addr learned my addr 172.21.15.110:0/1957041133 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930300603 0 0) 0x7ff3e80a77f0 con 0x7ff3e80a8c10 2019-10-13T04:04:59.425 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d00030a0 con 0x7ff3e80a8c10 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293285631 0 0) 0x7ff3e80a9030 con 0x7ff3e8065060 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e80a77f0 con 0x7ff3e8065060 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 -- 172.21.15.110:0/1957041133 >> v1:172.21.15.204:6789/0 conn(0x7ff3e8065060 legacy=0x7ff3e8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 -- 172.21.15.110:0/1957041133 >> v1:172.21.15.110:6789/0 conn(0x7ff3e80a8c10 legacy=0x7ff3e814e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 -- 172.21.15.110:0/1957041133 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e80a9260 con 0x7ff3e80a6410 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.426 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).ready entity=mon.2 client_cookie=cd66fbba92989697 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d8011260 con 0x7ff3e80a6410 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ec962700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d8008bb0 con 0x7ff3e80a6410 2019-10-13T04:04:59.427 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3e77fe700 1 -- 172.21.15.110:0/1957041133 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d8011840 con 0x7ff3e80a6410 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 msgr2=0x7ff3e80a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7ff3d8008780 tx=0x7ff3d800fbe0).stop 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/1957041133 shutdown_connections 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/1957041133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e80a6410 0x7ff3e80a1970 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/1957041133 shutdown_connections 2019-10-13T04:04:59.428 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/1957041133 wait complete. 2019-10-13T04:04:59.429 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.417+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/1957041133 >> 172.21.15.110:0/1957041133 conn(0x7ff3e805afb0 msgr2=0x7ff3e809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 Processor -- start 2019-10-13T04:04:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 -- start start 2019-10-13T04:04:59.432 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80a05e0 con 0x7ff3e8068a30 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e809fcb0 con 0x7ff3e809f890 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3ee3c5700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e8064670 con 0x7ff3e8063640 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3e7fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff3e8068a30 0x7ff3e807dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45878/0 (socket says 172.21.15.110:45878) 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.421+0000 7ff3e7fff700 1 -- 172.21.15.110:0/629952846 learned_addr learned my addr 172.21.15.110:0/629952846 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:59.433 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644619155 0 0) 0x7ff3e80a05e0 con 0x7ff3e8068a30 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c80030a0 con 0x7ff3e8068a30 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986626948 0 0) 0x7ff3e809fcb0 con 0x7ff3e809f890 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e80a05e0 con 0x7ff3e809f890 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2571312893 0 0) 0x7ff3c80030a0 con 0x7ff3e8068a30 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 -- 172.21.15.110:0/629952846 >> v1:172.21.15.204:6789/0 conn(0x7ff3e809f890 legacy=0x7ff3e813d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.434 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 -- 172.21.15.110:0/629952846 >> v1:172.21.15.110:6789/0 conn(0x7ff3e8068a30 legacy=0x7ff3e807dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 -- 172.21.15.110:0/629952846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e81410b0 con 0x7ff3e8063640 2019-10-13T04:04:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.435 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3dc001a40 con 0x7ff3e8063640 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e813f090 con 0x7ff3e8063640 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc001850 con 0x7ff3e8063640 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.436 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d8007340 con 0x7ff3e8063640 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e813f610 con 0x7ff3e8063640 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff3d800f850 con 0x7ff3e8063640 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 0x7ff3c802a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.425+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff3d8034e70 con 0x7ff3e8063640 2019-10-13T04:04:59.437 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.429+0000 7ff3e7fff700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 0x7ff3c802a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.438 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.429+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8068e50 con 0x7ff3e8063640 2019-10-13T04:04:59.438 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.429+0000 7ff3e7fff700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 0x7ff3c802a4d0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.440 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.437+0000 7ff3ec962700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.441 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.437+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff3d801dbd0 con 0x7ff3e8063640 2019-10-13T04:04:59.606 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.601+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 --> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3e80555c0 con 0x7ff3c8027fd0 2019-10-13T04:04:59.613 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.609+0000 7ff3e57fa700 1 -- 172.21.15.110:0/629952846 <== mgr.4105 v2:172.21.15.204:6800/13789 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80438 (crc 0 0 0) 0x7ff3e80555c0 con 0x7ff3c8027fd0 2019-10-13T04:04:59.623 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:04:59.623 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2019-10-13T04:04:59.625 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 msgr2=0x7ff3c802a4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 0x7ff3c802a4d0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 msgr2=0x7ff3e809e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7ff3dc001c80 tx=0x7ff3dc001790).stop 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 shutdown_connections 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff3c8027fd0 0x7ff3c802a4d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.626 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 --2- 172.21.15.110:0/629952846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3e8063640 0x7ff3e809e4f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 shutdown_connections 2019-10-13T04:04:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 wait complete. 2019-10-13T04:04:59.627 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:04:59.621+0000 7ff3ee3c5700 1 -- 172.21.15.110:0/629952846 >> 172.21.15.110:0/629952846 conn(0x7ff3e80a2a30 msgr2=0x7ff3e8122b40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.637 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-10-13T04:04:59.244554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8395136,"kb_used_data":6400,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746579584,"statfs":{"total":773094113280,"available":764497494016,"internally_reserved":8589934592,"allocated":6553600,"data_stored":719508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001979"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432636+0000","last_change":"2019-10-13T04:04:48.431642+0000","last_active":"2019-10-13T04:04:48.432636+0000","last_peered":"2019-10-13T04:04:48.432636+0000","last_clean":"2019-10-13T04:04:48.432636+0000","last_became_active":"2019-10-13T04:04:48.431123+0000","last_became_peered":"2019-10-13T04:04:48.431123+0000","last_unstale":"2019-10-13T04:04:48.432636+0000","last_undegraded":"2019-10-13T04:04:48.432636+0000","last_fullsized":"2019-10-13T04:04:48.432636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.417799+0000","last_change":"2019-10-13T04:04:42.618332+0000","last_active":"2019-10-13T04:04:48.417799+0000","last_peered":"2019-10-13T04:04:48.417799+0000","last_clean":"2019-10-13T04:04:48.417799+0000","last_became_active":"2019-10-13T04:04:42.617512+0000","last_became_peered":"2019-10-13T04:04:42.617512+0000","last_unstale":"2019-10-13T04:04:48.417799+0000","last_undegraded":"2019-10-13T04:04:48.417799+0000","last_fullsized":"2019-10-13T04:04:48.417799+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'4","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.622734+0000","last_change":"2019-10-13T04:04:45.404135+0000","last_active":"2019-10-13T04:04:51.622734+0000","last_peered":"2019-10-13T04:04:51.622734+0000","last_clean":"2019-10-13T04:04:51.622734+0000","last_became_active":"2019-10-13T04:04:45.403719+0000","last_became_peered":"2019-10-13T04:04:45.403719+0000","last_unstale":"2019-10-13T04:04:51.622734+0000","last_undegraded":"2019-10-13T04:04:51.622734+0000","last_fullsized":"2019-10-13T04:04:51.622734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432219+0000","last_change":"2019-10-13T04:04:48.430799+0000","last_active":"2019-10-13T04:04:48.432219+0000","last_peered":"2019-10-13T04:04:48.432219+0000","last_clean":"2019-10-13T04:04:48.432219+0000","last_became_active":"2019-10-13T04:04:48.430286+0000","last_became_peered":"2019-10-13T04:04:48.430286+0000","last_unstale":"2019-10-13T04:04:48.432219+0000","last_undegraded":"2019-10-13T04:04:48.432219+0000","last_fullsized":"2019-10-13T04:04:48.432219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.420869+0000","last_change":"2019-10-13T04:04:42.383533+0000","last_active":"2019-10-13T04:04:48.420869+0000","last_peered":"2019-10-13T04:04:48.420869+0000","last_clean":"2019-10-13T04:04:48.420869+0000","last_became_active":"2019-10-13T04:04:42.383201+0000","last_became_peered":"2019-10-13T04:04:42.383201+0000","last_unstale":"2019-10-13T04:04:48.420869+0000","last_undegraded":"2019-10-13T04:04:48.420869+0000","last_fullsized":"2019-10-13T04:04:48.420869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'7","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.624734+0000","last_change":"2019-10-13T04:04:45.398996+0000","last_active":"2019-10-13T04:04:51.624734+0000","last_peered":"2019-10-13T04:04:51.624734+0000","last_clean":"2019-10-13T04:04:51.624734+0000","last_became_active":"2019-10-13T04:04:45.398640+0000","last_became_peered":"2019-10-13T04:04:45.398640+0000","last_unstale":"2019-10-13T04:04:51.624734+0000","last_undegraded":"2019-10-13T04:04:51.624734+0000","last_fullsized":"2019-10-13T04:04:51.624734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.431909+0000","last_change":"2019-10-13T04:04:48.431132+0000","last_active":"2019-10-13T04:04:48.431909+0000","last_peered":"2019-10-13T04:04:48.431909+0000","last_clean":"2019-10-13T04:04:48.431909+0000","last_became_active":"2019-10-13T04:04:48.430617+0000","last_became_peered":"2019-10-13T04:04:48.430617+0000","last_unstale":"2019-10-13T04:04:48.431909+0000","last_undegraded":"2019-10-13T04:04:48.431909+0000","last_fullsized":"2019-10-13T04:04:48.431909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.421933+0000","last_change":"2019-10-13T04:04:42.384347+0000","last_active":"2019-10-13T04:04:48.421933+0000","last_peered":"2019-10-13T04:04:48.421933+0000","last_clean":"2019-10-13T04:04:48.421933+0000","last_became_active":"2019-10-13T04:04:42.383983+0000","last_became_peered":"2019-10-13T04:04:42.383983+0000","last_unstale":"2019-10-13T04:04:48.421933+0000","last_undegraded":"2019-10-13T04:04:48.421933+0000","last_fullsized":"2019-10-13T04:04:48.421933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.625379+0000","last_change":"2019-10-13T04:04:45.398273+0000","last_active":"2019-10-13T04:04:51.625379+0000","last_peered":"2019-10-13T04:04:51.625379+0000","last_clean":"2019-10-13T04:04:51.625379+0000","last_became_active":"2019-10-13T04:04:45.397947+0000","last_became_peered":"2019-10-13T04:04:45.397947+0000","last_unstale":"2019-10-13T04:04:51.625379+0000","last_undegraded":"2019-10-13T04:04:51.625379+0000","last_fullsized":"2019-10-13T04:04:51.625379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.563853+0000","last_change":"2019-10-13T04:04:48.563287+0000","last_active":"2019-10-13T04:04:48.563853+0000","last_peered":"2019-10-13T04:04:48.563853+0000","last_clean":"2019-10-13T04:04:48.563853+0000","last_became_active":"2019-10-13T04:04:48.562589+0000","last_became_peered":"2019-10-13T04:04:48.562589+0000","last_unstale":"2019-10-13T04:04:48.563853+0000","last_undegraded":"2019-10-13T04:04:48.563853+0000","last_fullsized":"2019-10-13T04:04:48.563853+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.419827+0000","last_change":"2019-10-13T04:04:42.483969+0000","last_active":"2019-10-13T04:04:48.419827+0000","last_peered":"2019-10-13T04:04:48.419827+0000","last_clean":"2019-10-13T04:04:48.419827+0000","last_became_active":"2019-10-13T04:04:42.483339+0000","last_became_peered":"2019-10-13T04:04:42.483339+0000","last_unstale":"2019-10-13T04:04:48.419827+0000","last_undegraded":"2019-10-13T04:04:48.419827+0000","last_fullsized":"2019-10-13T04:04:48.419827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.417706+0000","last_change":"2019-10-13T04:04:45.488583+0000","last_active":"2019-10-13T04:04:48.417706+0000","last_peered":"2019-10-13T04:04:48.417706+0000","last_clean":"2019-10-13T04:04:48.417706+0000","last_became_active":"2019-10-13T04:04:45.488145+0000","last_became_peered":"2019-10-13T04:04:45.488145+0000","last_unstale":"2019-10-13T04:04:48.417706+0000","last_undegraded":"2019-10-13T04:04:48.417706+0000","last_fullsized":"2019-10-13T04:04:48.417706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.564756+0000","last_change":"2019-10-13T04:04:48.564321+0000","last_active":"2019-10-13T04:04:48.564756+0000","last_peered":"2019-10-13T04:04:48.564756+0000","last_clean":"2019-10-13T04:04:48.564756+0000","last_became_active":"2019-10-13T04:04:48.562968+0000","last_became_peered":"2019-10-13T04:04:48.562968+0000","last_unstale":"2019-10-13T04:04:48.564756+0000","last_undegraded":"2019-10-13T04:04:48.564756+0000","last_fullsized":"2019-10-13T04:04:48.564756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.419991+0000","last_change":"2019-10-13T04:04:42.482494+0000","last_active":"2019-10-13T04:04:48.419991+0000","last_peered":"2019-10-13T04:04:48.419991+0000","last_clean":"2019-10-13T04:04:48.419991+0000","last_became_active":"2019-10-13T04:04:42.481467+0000","last_became_peered":"2019-10-13T04:04:42.481467+0000","last_unstale":"2019-10-13T04:04:48.419991+0000","last_undegraded":"2019-10-13T04:04:48.419991+0000","last_fullsized":"2019-10-13T04:04:48.419991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'5","reported_seq":"17","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.625409+0000","last_change":"2019-10-13T04:04:45.404217+0000","last_active":"2019-10-13T04:04:51.625409+0000","last_peered":"2019-10-13T04:04:51.625409+0000","last_clean":"2019-10-13T04:04:51.625409+0000","last_became_active":"2019-10-13T04:04:45.403737+0000","last_became_peered":"2019-10-13T04:04:45.403737+0000","last_unstale":"2019-10-13T04:04:51.625409+0000","last_undegraded":"2019-10-13T04:04:51.625409+0000","last_fullsized":"2019-10-13T04:04:51.625409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.561909+0000","last_change":"2019-10-13T04:04:48.560674+0000","last_active":"2019-10-13T04:04:48.561909+0000","last_peered":"2019-10-13T04:04:48.561909+0000","last_clean":"2019-10-13T04:04:48.561909+0000","last_became_active":"2019-10-13T04:04:48.560121+0000","last_became_peered":"2019-10-13T04:04:48.560121+0000","last_unstale":"2019-10-13T04:04:48.561909+0000","last_undegraded":"2019-10-13T04:04:48.561909+0000","last_fullsized":"2019-10-13T04:04:48.561909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.424872+0000","last_change":"2019-10-13T04:04:42.482487+0000","last_active":"2019-10-13T04:04:48.424872+0000","last_peered":"2019-10-13T04:04:48.424872+0000","last_clean":"2019-10-13T04:04:48.424872+0000","last_became_active":"2019-10-13T04:04:42.481484+0000","last_became_peered":"2019-10-13T04:04:42.481484+0000","last_unstale":"2019-10-13T04:04:48.424872+0000","last_undegraded":"2019-10-13T04:04:48.424872+0000","last_fullsized":"2019-10-13T04:04:48.424872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:48.721512+0000","last_change":"2019-10-13T04:04:45.404034+0000","last_active":"2019-10-13T04:04:48.721512+0000","last_peered":"2019-10-13T04:04:48.721512+0000","last_clean":"2019-10-13T04:04:48.721512+0000","last_became_active":"2019-10-13T04:04:45.403650+0000","last_became_peered":"2019-10-13T04:04:45.403650+0000","last_unstale":"2019-10-13T04:04:48.721512+0000","last_undegraded":"2019-10-13T04:04:48.721512+0000","last_fullsized":"2019-10-13T04:04:48.721512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.432181+0000","last_change":"2019-10-13T04:04:48.431396+0000","last_active":"2019-10-13T04:04:48.432181+0000","last_peered":"2019-10-13T04:04:48.432181+0000","last_clean":"2019-10-13T04:04:48.432181+0000","last_became_active":"2019-10-13T04:04:48.430913+0000","last_became_peered":"2019-10-13T04:04:48.430913+0000","last_unstale":"2019-10-13T04:04:48.432181+0000","last_undegraded":"2019-10-13T04:04:48.432181+0000","last_fullsized":"2019-10-13T04:04:48.432181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.558186+0000","last_change":"2019-10-13T04:04:42.484231+0000","last_active":"2019-10-13T04:04:48.558186+0000","last_peered":"2019-10-13T04:04:48.558186+0000","last_clean":"2019-10-13T04:04:48.558186+0000","last_became_active":"2019-10-13T04:04:42.483381+0000","last_became_peered":"2019-10-13T04:04:42.483381+0000","last_unstale":"2019-10-13T04:04:48.558186+0000","last_undegraded":"2019-10-13T04:04:48.558186+0000","last_fullsized":"2019-10-13T04:04:48.558186+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"23'6","reported_seq":"18","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.626065+0000","last_change":"2019-10-13T04:04:45.485691+0000","last_active":"2019-10-13T04:04:51.626065+0000","last_peered":"2019-10-13T04:04:51.626065+0000","last_clean":"2019-10-13T04:04:51.626065+0000","last_became_active":"2019-10-13T04:04:45.485029+0000","last_became_peered":"2019-10-13T04:04:45.485029+0000","last_unstale":"2019-10-13T04:04:51.626065+0000","last_undegraded":"2019-10-13T04:04:51.626065+0000","last_fullsized":"2019-10-13T04:04:51.626065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.563919+0000","last_change":"2019-10-13T04:04:48.561119+0000","last_active":"2019-10-13T04:04:48.563919+0000","last_peered":"2019-10-13T04:04:48.563919+0000","last_clean":"2019-10-13T04:04:48.563919+0000","last_became_active":"2019-10-13T04:04:48.560473+0000","last_became_peered":"2019-10-13T04:04:48.560473+0000","last_unstale":"2019-10-13T04:04:48.563919+0000","last_undegraded":"2019-10-13T04:04:48.563919+0000","last_fullsized":"2019-10-13T04:04:48.563919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:47.406960+0000","last_clean_scrub_stamp":"2019-10-13T04:04:47.406960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2019-10-13T04:04:48.558728+0000","last_change":"2019-10-13T04:04:42.381380+0000","last_active":"2019-10-13T04:04:48.558728+0000","last_peered":"2019-10-13T04:04:48.558728+0000","last_clean":"2019-10-13T04:04:48.558728+0000","last_became_active":"2019-10-13T04:04:42.381046+0000","last_became_peered":"2019-10-13T04:04:42.381046+0000","last_unstale":"2019-10-13T04:04:48.558728+0000","last_undegraded":"2019-10-13T04:04:48.558728+0000","last_fullsized":"2019-10-13T04:04:48.558728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:41.350847+0000","last_clean_scrub_stamp":"2019-10-13T04:04:41.350847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"23'8","reported_seq":"20","reported_epoch":"23","state":"active+clean","last_fresh":"2019-10-13T04:04:51.626109+0000","last_change":"2019-10-13T04:04:45.488504+0000","last_active":"2019-10-13T04:04:51.626109+0000","last_peered":"2019-10-13T04:04:51.626109+0000","last_clean":"2019-10-13T04:04:51.626109+0000","last_became_active":"2019-10-13T04:04:45.488054+0000","last_became_peered":"2019-10-13T04:04:45.488054+0000","last_unstale":"2019-10-13T04:04:51.626109+0000","last_undegraded":"2019-10-13T04:04:51.626109+0000","last_fullsized":"2019-10-13T04:04:51.626109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-13T04:04:44.384183+0000","last_clean_scrub_stamp":"2019-10-13T04:04:44.384183+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049424,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322416,"statfs":{"total":96636764160,"available":95562153984,"internally_reserved":1073741824,"allocated":851968,"data_stored":90093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.198}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.154}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.283}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.008}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.107}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.059}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.239}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049392,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322448,"statfs":{"total":96636764160,"available":95562186752,"internally_reserved":1073741824,"allocated":819200,"data_stored":89933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.035}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.633}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.569}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.320}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.259}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.897}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049408,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322432,"statfs":{"total":96636764160,"available":95562170368,"internally_reserved":1073741824,"allocated":835584,"data_stored":90893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.593}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.684}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.564}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.750}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.988}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.818}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.643}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049440,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322400,"statfs":{"total":96636764160,"available":95562137600,"internally_reserved":1073741824,"allocated":868352,"data_stored":91053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.131}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.976}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.937}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.205}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.279}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.525}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.883}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.162}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.204}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.125}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.366}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.429}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.802}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.862}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.718}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049360,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322480,"statfs":{"total":96636764160,"available":95562219520,"internally_reserved":1073741824,"allocated":786432,"data_stored":89367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.068}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.131}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.534}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.274}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.477}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1049376,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322464,"statfs":{"total":96636764160,"available":95562203136,"internally_reserved":1073741824,"allocated":802816,"data_stored":89401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.813}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.661}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.733}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.138}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.094}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.356}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.042}]}]}]}} 2019-10-13T04:04:59.641 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-10-13T04:04:59.641 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-10-13T04:04:59.641 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:59.642 INFO:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2019-10-13T04:04:59.728 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 Processor -- start 2019-10-13T04:04:59.729 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 -- start start 2019-10-13T04:04:59.729 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 0x7f82e40a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.729 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82e40a8c10 0x7f82e40a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:04:59.729 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40a77f0 con 0x7f82e40a6410 2019-10-13T04:04:59.729 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40a20e0 con 0x7f82e40a8c10 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82eab02700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40a2380 con 0x7f82e4065060 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82e40a8c10 0x7f82e40a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e909f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82e40a6410 0x7f82e4144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45884/0 (socket says 172.21.15.110:45884) 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e909f700 1 -- 172.21.15.110:0/381346116 learned_addr learned my addr 172.21.15.110:0/381346116 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e889e700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 0x7f82e40a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 -- 172.21.15.110:0/381346116 >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 msgr2=0x7f82e40a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:04:59.730 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 0x7f82e40a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 -- 172.21.15.110:0/381346116 >> v1:172.21.15.110:6789/0 conn(0x7f82e40a6410 legacy=0x7f82e4144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 -- 172.21.15.110:0/381346116 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e414f9f0 con 0x7f82e40a8c10 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82d7fff700 1 -- 172.21.15.110:0/381346116 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491670659 0 0) 0x7f82e40a77f0 con 0x7f82e40a6410 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e889e700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 0x7f82e40a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.110:3300/0 conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.110:3300/0 conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.731 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).ready entity=mon.0 client_cookie=3bd6dbc13e904b1d server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82d7fff700 1 -- 172.21.15.110:0/381346116 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82dc008ba0 con 0x7f82e40a8c10 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82e98a0700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.721+0000 7f82d7fff700 1 -- 172.21.15.110:0/381346116 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc0087c0 con 0x7f82e40a8c10 2019-10-13T04:04:59.732 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82d7fff700 1 -- 172.21.15.110:0/381346116 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82dc011700 con 0x7f82e40a8c10 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 msgr2=0x7f82e40a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f82e40a15e0 tx=0x7f82d8007a90).stop 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- 172.21.15.110:0/381346116 shutdown_connections 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 --2- 172.21.15.110:0/381346116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e40a8c10 0x7f82e40a9030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 --2- 172.21.15.110:0/381346116 >> v2:172.21.15.204:3300/0 conn(0x7f82e4065060 0x7f82e40a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- 172.21.15.110:0/381346116 shutdown_connections 2019-10-13T04:04:59.733 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- 172.21.15.110:0/381346116 wait complete. 2019-10-13T04:04:59.734 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- 172.21.15.110:0/381346116 >> 172.21.15.110:0/381346116 conn(0x7f82e405afb0 msgr2=0x7f82e409c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.736 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 Processor -- start 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- start start 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82e407cb40 0x7f82e40a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f82e4064440 0x7f82e40a9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40a9970 con 0x7f82e4062e50 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e407bce0 con 0x7f82e407cb40 2019-10-13T04:04:59.737 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82eab02700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e407bf80 con 0x7f82e4064440 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82e909f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82e407cb40 0x7f82e40a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82e909f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82e407cb40 0x7f82e40a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43178/0 (socket says 172.21.15.110:43178) 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.725+0000 7f82e909f700 1 -- 172.21.15.110:0/2822593629 learned_addr learned my addr 172.21.15.110:0/2822593629 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e889e700 1 --2- 172.21.15.110:0/2822593629 >> v2:172.21.15.204:3300/0 conn(0x7f82e4064440 0x7f82e40a9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 -- 172.21.15.110:0/2822593629 >> v2:172.21.15.204:3300/0 conn(0x7f82e4064440 msgr2=0x7f82e40a9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.738 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> v2:172.21.15.204:3300/0 conn(0x7f82e4064440 0x7f82e40a9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 -- 172.21.15.110:0/2822593629 >> v1:172.21.15.204:6789/0 conn(0x7f82e4062e50 legacy=0x7f82e4063270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 -- 172.21.15.110:0/2822593629 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e407a2e0 con 0x7f82e407cb40 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> v2:172.21.15.110:3300/0 conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> v2:172.21.15.110:3300/0 conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.739 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82e0011590 con 0x7f82e407cb40 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00111b0 con 0x7f82e407cb40 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82e0011b70 con 0x7f82e407cb40 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e4063a70 con 0x7f82e407cb40 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e4132a70 con 0x7f82e407cb40 2019-10-13T04:04:59.740 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f82e000b330 con 0x7f82e407cb40 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 0x7f82c4029d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f82e0034d60 con 0x7f82e407cb40 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e889e700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 0x7f82c4029d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:04:59.741 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.729+0000 7f82e889e700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 0x7f82c4029d60 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:04:59.742 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.737+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e40555c0 con 0x7f82e407cb40 2019-10-13T04:04:59.749 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.745+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.749 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.745+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f82e00260f0 con 0x7f82e407cb40 2019-10-13T04:04:59.938 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.933+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f82e4066520 con 0x7f82e407cb40 2019-10-13T04:04:59.939 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.933+0000 7f82e909f700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:04:59.939 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.933+0000 7f82d5ffb700 1 -- 172.21.15.110:0/2822593629 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7f82e000b620 con 0x7f82e407cb40 2019-10-13T04:04:59.939 INFO:teuthology.misc.health.smithi110.stdout:HEALTH_OK 2019-10-13T04:04:59.941 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 msgr2=0x7f82c4029d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 0x7f82c4029d60 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 msgr2=0x7f82e40a8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f82e407bea0 tx=0x7f82d8007a70).stop 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 shutdown_connections 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f82c4027860 0x7f82c4029d60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 --2- 172.21.15.110:0/2822593629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82e407cb40 0x7f82e40a8ed0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.942 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 --2- 172.21.15.110:0/2822593629 >> v2:172.21.15.204:3300/0 conn(0x7f82e4064440 0x7f82e40a9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:04:59.943 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 shutdown_connections 2019-10-13T04:04:59.943 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 wait complete. 2019-10-13T04:04:59.943 INFO:teuthology.misc.health.smithi110.stderr:2019-10-13T04:04:59.937+0000 7f82eab02700 1 -- 172.21.15.110:0/2822593629 >> 172.21.15.110:0/2822593629 conn(0x7f82e40a6cf0 msgr2=0x7f82e407dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:04:59.950 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2019-10-13T04:04:59.950 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:04:59.950 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:05:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.037+0000 7fa4562b7700 1 Processor -- start 2019-10-13T04:05:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.037+0000 7fa4562b7700 1 -- start start 2019-10-13T04:05:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4500a6410 0x7fa4500a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa4500a8c10 0x7fa4500a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4500a77f0 con 0x7fa450065060 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4500a20e0 con 0x7fa4500a8c10 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500a2380 con 0x7fa4500a6410 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa450065060 0x7fa450144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45898/0 (socket says 172.21.15.110:45898) 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44ffff700 1 -- 172.21.15.110:0/1581531288 learned_addr learned my addr 172.21.15.110:0/1581531288 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa454854700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4500a6410 0x7fa4500a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.049 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> v2:172.21.15.204:3300/0 conn(0x7fa4500a8c10 0x7fa4500a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547228911 0 0) 0x7fa4500a77f0 con 0x7fa450065060 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43c0030a0 con 0x7fa450065060 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1872431374 0 0) 0x7fa43c0030a0 con 0x7fa450065060 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4500a6410 msgr2=0x7fa4500a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4500a6410 0x7fa4500a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 >> v2:172.21.15.204:3300/0 conn(0x7fa4500a8c10 msgr2=0x7fa4500a9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.050 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 --2- 172.21.15.110:0/1581531288 >> v2:172.21.15.204:3300/0 conn(0x7fa4500a8c10 0x7fa4500a9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45014f9f0 con 0x7fa450065060 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7fa438001750 con 0x7fa450065060 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 >> v1:172.21.15.110:6789/0 conn(0x7fa450065060 legacy=0x7fa450144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c003700 0x7fa43c003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004410 0x7fa43c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500a20e0 con 0x7fa43c004d80 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500a2380 con 0x7fa43c003700 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c0051a0 con 0x7fa43c004410 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa454854700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c003700 0x7fa43c003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44ffff700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004410 0x7fa43c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 -- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004410 msgr2=0x7fa43c004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004410 0x7fa43c004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 -- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c003700 msgr2=0x7fa43c003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c003700 0x7fa43c003b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 -- 172.21.15.110:0/1581531288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c04fc20 con 0x7fa43c004d80 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa448002f30 tx=0x7fa440007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa448002f30 tx=0x7fa440007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.053 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fa448002f30 tx=0x7fa440007cb0).ready entity=mon.0 client_cookie=ae26730303fa87b1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fa448002f30 tx=0x7fa440007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa448008bc0 con 0x7fa43c004d80 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fa448002f30 tx=0x7fa440007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fa448002f30 tx=0x7fa440007cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4480087e0 con 0x7fa43c004d80 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa44f7fe700 1 -- 172.21.15.110:0/1581531288 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa448010900 con 0x7fa43c004d80 2019-10-13T04:05:00.054 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 msgr2=0x7fa43c04f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 --2- 172.21.15.110:0/1581531288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa43c004d80 0x7fa43c04f670 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fa448002f30 tx=0x7fa440007cb0).stop 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa455055700 1 -- 172.21.15.110:0/1581531288 reap_dead start 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- 172.21.15.110:0/1581531288 shutdown_connections 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- 172.21.15.110:0/1581531288 shutdown_connections 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- 172.21.15.110:0/1581531288 wait complete. 2019-10-13T04:05:00.055 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.041+0000 7fa4562b7700 1 -- 172.21.15.110:0/1581531288 >> 172.21.15.110:0/1581531288 conn(0x7fa45005afb0 msgr2=0x7fa45009c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 Processor -- start 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 -- start start 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa450119200 0x7fa450114900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 0x7fa450130470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa450114e50 con 0x7fa4500a2960 2019-10-13T04:05:00.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa450117f40 con 0x7fa45012e040 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa4562b7700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa450118cc0 con 0x7fa450119200 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa454854700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa4500a2960 0x7fa45014f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45908/0 (socket says 172.21.15.110:45908) 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa454854700 1 -- 172.21.15.110:0/303930436 learned_addr learned my addr 172.21.15.110:0/303930436 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa455055700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 0x7fa450130470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.204:3300/0 conn(0x7fa450119200 0x7fa450114900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038373096 0 0) 0x7fa450114e50 con 0x7fa4500a2960 2019-10-13T04:05:00.060 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.045+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4340030a0 con 0x7fa4500a2960 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 -- 172.21.15.110:0/303930436 >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 msgr2=0x7fa450130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 0x7fa450130470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 -- 172.21.15.110:0/303930436 >> v1:172.21.15.110:6789/0 conn(0x7fa4500a2960 legacy=0x7fa45014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 -- 172.21.15.110:0/303930436 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500a8700 con 0x7fa450119200 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa455055700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 0x7fa450130470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.204:3300/0 conn(0x7fa450119200 0x7fa450114900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.061 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.204:3300/0 conn(0x7fa450119200 0x7fa450114900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa43800ddd0 con 0x7fa450119200 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43800d9f0 con 0x7fa450119200 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.062 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa450116620 con 0x7fa450119200 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa438011c10 con 0x7fa450119200 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa450116000 con 0x7fa450119200 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fa43800e070 con 0x7fa450119200 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.063 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 0x7fa43402a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa438034760 con 0x7fa450119200 2019-10-13T04:05:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa454854700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 0x7fa43402a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.049+0000 7fa454854700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 0x7fa43402a260 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.064 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.053+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4500555c0 con 0x7fa450119200 2019-10-13T04:05:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.061+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.066 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.061+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fa43800e360 con 0x7fa450119200 2019-10-13T04:05:00.226 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.221+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa450118a90 con 0x7fa450119200 2019-10-13T04:05:00.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.225+0000 7fa44ffff700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.225+0000 7fa44d7fa700 1 -- 172.21.15.110:0/303930436 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7fa43800e580 con 0x7fa450119200 2019-10-13T04:05:00.230 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:05:00.230 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:05:00.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 msgr2=0x7fa43402a260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 0x7fa43402a260 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 msgr2=0x7fa450114900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7fa438001950 tx=0x7fa448007ac0).stop 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 shutdown_connections 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fa434027d60 0x7fa43402a260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 --2- 172.21.15.110:0/303930436 >> v2:172.21.15.110:3300/0 conn(0x7fa45012e040 0x7fa450130470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 --2- 172.21.15.110:0/303930436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa450119200 0x7fa450114900 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 shutdown_connections 2019-10-13T04:05:00.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 wait complete. 2019-10-13T04:05:00.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.229+0000 7fa4562b7700 1 -- 172.21.15.110:0/303930436 >> 172.21.15.110:0/303930436 conn(0x7fa45007df80 msgr2=0x7fa4500a7520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.242 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:40.397189+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:43.569877+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:46.432662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T04:05:00.243 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:00.243 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T04:05:00.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.333+0000 7fd1ee807700 1 Processor -- start 2019-10-13T04:05:00.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.333+0000 7fd1ee807700 1 -- start start 2019-10-13T04:05:00.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1e80a88e0 0x7fd1e80a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e80a17e0 0x7fd1e814e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80a0fc0 con 0x7fd1e80aa2b0 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e8059e10 con 0x7fd1e80a88e0 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e805a0b0 con 0x7fd1e80a17e0 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1e80a88e0 0x7fd1e80a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1e80a88e0 0x7fd1e80a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43206/0 (socket says 172.21.15.110:43206) 2019-10-13T04:05:00.344 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 -- 172.21.15.110:0/3836936196 learned_addr learned my addr 172.21.15.110:0/3836936196 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e80a17e0 0x7fd1e814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 -- 172.21.15.110:0/3836936196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e80a17e0 msgr2=0x7fd1e814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e80a17e0 0x7fd1e814e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 -- 172.21.15.110:0/3836936196 >> v1:172.21.15.204:6789/0 conn(0x7fd1e80aa2b0 legacy=0x7fd1e807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 -- 172.21.15.110:0/3836936196 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80a75f0 con 0x7fd1e80a88e0 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> v2:172.21.15.110:3300/0 conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd1e4001110 tx=0x7fd1e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.345 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> v2:172.21.15.110:3300/0 conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd1e4001110 tx=0x7fd1e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7fd1e4001110 tx=0x7fd1e4001150).ready entity=mon.0 client_cookie=12021d11c97cf154 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fd1e4001110 tx=0x7fd1e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fd1e4001110 tx=0x7fd1e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1df7fe700 1 -- 172.21.15.110:0/3836936196 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1d8008c80 con 0x7fd1e80a88e0 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1dffff700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fd1e4001110 tx=0x7fd1e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1df7fe700 1 -- 172.21.15.110:0/3836936196 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d80088a0 con 0x7fd1e80a88e0 2019-10-13T04:05:00.346 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1df7fe700 1 -- 172.21.15.110:0/3836936196 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1d8011800 con 0x7fd1e80a88e0 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 msgr2=0x7fd1e80a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7fd1e4001110 tx=0x7fd1e4001150).stop 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- 172.21.15.110:0/3836936196 shutdown_connections 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e80a17e0 0x7fd1e814e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 --2- 172.21.15.110:0/3836936196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1e80a88e0 0x7fd1e80a1290 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.347 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- 172.21.15.110:0/3836936196 shutdown_connections 2019-10-13T04:05:00.348 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- 172.21.15.110:0/3836936196 wait complete. 2019-10-13T04:05:00.348 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.337+0000 7fd1ee807700 1 -- 172.21.15.110:0/3836936196 >> 172.21.15.110:0/3836936196 conn(0x7fd1e805e980 msgr2=0x7fd1e80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.351 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 Processor -- start 2019-10-13T04:05:00.351 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- start start 2019-10-13T04:05:00.351 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 0x7fd1e811e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.351 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8064fa0 con 0x7fd1e80640e0 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e811fc20 con 0x7fd1e8127420 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80a3ea0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ecda4700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 0x7fd1e811e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.352 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ecda4700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 0x7fd1e811e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.110:52622/0 (socket says 172.21.15.110:52622) 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ecda4700 1 -- 172.21.15.110:0/1187775999 learned_addr learned my addr 172.21.15.110:0/1187775999 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.110:60074/0 (socket says 172.21.15.110:60074) 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 -- 172.21.15.110:0/1187775999 >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 msgr2=0x7fd1e811e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 0x7fd1e811e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 -- 172.21.15.110:0/1187775999 >> v1:172.21.15.204:6789/0 conn(0x7fd1e80640e0 legacy=0x7fd1e8136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.353 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 -- 172.21.15.110:0/1187775999 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e81366c0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1e0011df0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e811f860 con 0x7fd1e811e6c0 2019-10-13T04:05:00.354 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0011a10 con 0x7fd1e811e6c0 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1e0005c90 con 0x7fd1e811e6c0 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.341+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e813e060 con 0x7fd1e811e6c0 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fd1e0007140 con 0x7fd1e811e6c0 2019-10-13T04:05:00.355 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1dd7fa700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 0x7fd1c8029e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd1e00349c0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1ecda4700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 0x7fd1c8029e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.345+0000 7fd1ecda4700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 0x7fd1c8029e30 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.349+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e80555c0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.357+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.363 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.357+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fd1e0007430 con 0x7fd1e811e6c0 2019-10-13T04:05:00.518 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.513+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd1e80a79c0 con 0x7fd1e811e6c0 2019-10-13T04:05:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.517+0000 7fd1ed5a5700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.519 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.517+0000 7fd1dd7fa700 1 -- 172.21.15.110:0/1187775999 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd1e000e100 con 0x7fd1e811e6c0 2019-10-13T04:05:00.521 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:00.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 msgr2=0x7fd1c8029e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 0x7fd1c8029e30 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.524 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 msgr2=0x7fd1e813db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fd1e00032b0 tx=0x7fd1e40012f0).stop 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 shutdown_connections 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fd1c8027930 0x7fd1c8029e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 --2- 172.21.15.110:0/1187775999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1e811e6c0 0x7fd1e813db10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 --2- 172.21.15.110:0/1187775999 >> v2:172.21.15.204:3300/0 conn(0x7fd1e8127420 0x7fd1e811e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 shutdown_connections 2019-10-13T04:05:00.525 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 wait complete. 2019-10-13T04:05:00.526 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.521+0000 7fd1ee807700 1 -- 172.21.15.110:0/1187775999 >> 172.21.15.110:0/1187775999 conn(0x7fd1e80a6750 msgr2=0x7fd1e80a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.535 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:00.535 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 Processor -- start 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 -- start start 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f51e40aa2b0 0x7f51e407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 0x7f51e40a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40a0fc0 con 0x7f51e40a17e0 2019-10-13T04:05:00.639 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e4059e10 con 0x7f51e407ebf0 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51ea011700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e405a0b0 con 0x7f51e40aa2b0 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.625+0000 7f51e37fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 0x7f51e40a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e37fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 0x7f51e40a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43220/0 (socket says 172.21.15.110:43220) 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e37fe700 1 -- 172.21.15.110:0/3418668430 learned_addr learned my addr 172.21.15.110:0/3418668430 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> v2:172.21.15.204:3300/0 conn(0x7f51e40aa2b0 0x7f51e407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.640 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 -- 172.21.15.110:0/3418668430 >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 msgr2=0x7f51e40a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 0x7f51e40a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 -- 172.21.15.110:0/3418668430 >> v1:172.21.15.204:6789/0 conn(0x7f51e40a17e0 legacy=0x7f51e4147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 -- 172.21.15.110:0/3418668430 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e40a75f0 con 0x7f51e40aa2b0 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> v2:172.21.15.204:3300/0 conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f51cc002f30 tx=0x7f51dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> v2:172.21.15.204:3300/0 conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f51cc002f30 tx=0x7f51dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.641 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f51cc002f30 tx=0x7f51dc007710).ready entity=mon.1 client_cookie=9f9e5d582d8db45b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f51cc002f30 tx=0x7f51dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e2ffd700 1 -- 172.21.15.110:0/3418668430 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc008ba0 con 0x7f51e40aa2b0 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f51cc002f30 tx=0x7f51dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e3fff700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f51cc002f30 tx=0x7f51dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e2ffd700 1 -- 172.21.15.110:0/3418668430 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc0087c0 con 0x7f51e40aa2b0 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51e2ffd700 1 -- 172.21.15.110:0/3418668430 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51cc011700 con 0x7f51e40aa2b0 2019-10-13T04:05:00.642 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 msgr2=0x7f51e407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f51cc002f30 tx=0x7f51dc007710).stop 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- 172.21.15.110:0/3418668430 shutdown_connections 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 --2- 172.21.15.110:0/3418668430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51e40aa2b0 0x7f51e407e6a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 --2- 172.21.15.110:0/3418668430 >> v2:172.21.15.110:3300/0 conn(0x7f51e407ebf0 0x7f51e40a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- 172.21.15.110:0/3418668430 shutdown_connections 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- 172.21.15.110:0/3418668430 wait complete. 2019-10-13T04:05:00.643 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- 172.21.15.110:0/3418668430 >> 172.21.15.110:0/3418668430 conn(0x7f51e405e980 msgr2=0x7f51e40a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 Processor -- start 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.629+0000 7f51ea011700 1 -- start start 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40a3ea0 con 0x7f51e407ab20 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40a4e30 con 0x7f51e40a3020 2019-10-13T04:05:00.644 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40a2840 con 0x7f51e4122820 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e8daf700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f51e407ab20 0x7f51e40a4630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45940/0 (socket says 172.21.15.110:45940) 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e8daf700 1 -- 172.21.15.110:0/2708669307 learned_addr learned my addr 172.21.15.110:0/2708669307 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e3fff700 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f51e40a3020 0x7f51e40a1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.110:53756/0 (socket says 172.21.15.110:53756) 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421982852 0 0) 0x7f51e40a4e30 con 0x7f51e40a3020 2019-10-13T04:05:00.645 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51bc0030a0 con 0x7f51e40a3020 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128625849 0 0) 0x7f51e40a3ea0 con 0x7f51e407ab20 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e40a4e30 con 0x7f51e407ab20 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 -- 172.21.15.110:0/2708669307 >> v1:172.21.15.204:6789/0 conn(0x7f51e40a3020 legacy=0x7f51e40a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 -- 172.21.15.110:0/2708669307 >> v1:172.21.15.110:6789/0 conn(0x7f51e407ab20 legacy=0x7f51e40a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 -- 172.21.15.110:0/2708669307 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4140a20 con 0x7f51e4122820 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.646 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51dc008c80 con 0x7f51e4122820 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc0088a0 con 0x7f51e4122820 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.647 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e413ea00 con 0x7f51e4122820 2019-10-13T04:05:00.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51dc012c50 con 0x7f51e4122820 2019-10-13T04:05:00.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e413ede0 con 0x7f51e4122820 2019-10-13T04:05:00.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f51dc00b250 con 0x7f51e4122820 2019-10-13T04:05:00.648 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 0x7f51bc02a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.633+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f51dc034d30 con 0x7f51e4122820 2019-10-13T04:05:00.649 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.637+0000 7f51e3fff700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 0x7f51bc02a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.650 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.637+0000 7f51e3fff700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 0x7f51bc02a460 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.650 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.641+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e40555c0 con 0x7f51e4122820 2019-10-13T04:05:00.655 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.653+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.656 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.653+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f51dc00b540 con 0x7f51e4122820 2019-10-13T04:05:00.813 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.809+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f51e40a79c0 con 0x7f51e4122820 2019-10-13T04:05:00.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.809+0000 7f51e37fe700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.814 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.809+0000 7f51e0ff9700 1 -- 172.21.15.110:0/2708669307 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f51dc007110 con 0x7f51e4122820 2019-10-13T04:05:00.816 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:00.820 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 msgr2=0x7f51bc02a460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.820 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 0x7f51bc02a460 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.820 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 msgr2=0x7f51e407a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.820 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f51dc003070 tx=0x7f51dc010ef0).stop 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 shutdown_connections 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f51bc027f60 0x7f51bc02a460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 --2- 172.21.15.110:0/2708669307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51e4122820 0x7f51e407a5d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 shutdown_connections 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 wait complete. 2019-10-13T04:05:00.821 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.817+0000 7f51ea011700 1 -- 172.21.15.110:0/2708669307 >> 172.21.15.110:0/2708669307 conn(0x7f51e407ce20 msgr2=0x7f51e40a11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.834 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:00.834 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T04:05:00.939 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 Processor -- start 2019-10-13T04:05:00.939 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 -- start start 2019-10-13T04:05:00.939 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35ec07ebf0 0x7f35ec0a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0a0fc0 con 0x7f35ec0a17e0 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec059e10 con 0x7f35ec0aa2b0 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f429f700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec05a0b0 con 0x7f35ec07ebf0 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f283c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.940 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f283c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43232/0 (socket says 172.21.15.110:43232) 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f303d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35ec0a17e0 0x7f35ec147510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45948/0 (socket says 172.21.15.110:45948) 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f283c700 1 -- 172.21.15.110:0/9986449 learned_addr learned my addr 172.21.15.110:0/9986449 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.929+0000 7f35f203b700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35ec07ebf0 0x7f35ec0a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 -- 172.21.15.110:0/9986449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35ec07ebf0 msgr2=0x7f35ec0a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35ec07ebf0 0x7f35ec0a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 -- 172.21.15.110:0/9986449 >> v1:172.21.15.110:6789/0 conn(0x7f35ec0a17e0 legacy=0x7f35ec147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.941 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f183a700 1 -- 172.21.15.110:0/9986449 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847374421 0 0) 0x7f35ec0a0fc0 con 0x7f35ec0a17e0 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 -- 172.21.15.110:0/9986449 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0a75f0 con 0x7f35ec0aa2b0 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> v2:172.21.15.110:3300/0 conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> v2:172.21.15.110:3300/0 conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).ready entity=mon.0 client_cookie=3e5d67db53261b38 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.942 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f183a700 1 -- 172.21.15.110:0/9986449 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e8008c80 con 0x7f35ec0aa2b0 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f283c700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f183a700 1 -- 172.21.15.110:0/9986449 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e80088a0 con 0x7f35ec0aa2b0 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f183a700 1 -- 172.21.15.110:0/9986449 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e80119f0 con 0x7f35ec0aa2b0 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 -- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 msgr2=0x7f35ec07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.943 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7f35ec05ccf0 tx=0x7f35dc007710).stop 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 -- 172.21.15.110:0/9986449 shutdown_connections 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35ec0aa2b0 0x7f35ec07e6a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 --2- 172.21.15.110:0/9986449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35ec07ebf0 0x7f35ec0a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 -- 172.21.15.110:0/9986449 shutdown_connections 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 -- 172.21.15.110:0/9986449 wait complete. 2019-10-13T04:05:00.944 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.933+0000 7f35f429f700 1 -- 172.21.15.110:0/9986449 >> 172.21.15.110:0/9986449 conn(0x7f35ec05e980 msgr2=0x7f35ec0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:00.948 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 Processor -- start 2019-10-13T04:05:00.948 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 -- start start 2019-10-13T04:05:00.948 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f35ec121270 0x7f35ec13c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec11d0c0 con 0x7f35ec07c500 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec11c4f0 con 0x7f35ec1260a0 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f429f700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec11c720 con 0x7f35ec121270 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f203b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35ec07c500 0x7f35ec120a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45952/0 (socket says 172.21.15.110:45952) 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f203b700 1 -- 172.21.15.110:0/459092000 learned_addr learned my addr 172.21.15.110:0/459092000 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:00.949 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.937+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> v2:172.21.15.204:3300/0 conn(0x7f35ec121270 0x7f35ec13c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137075954 0 0) 0x7f35ec11d0c0 con 0x7f35ec07c500 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d40030a0 con 0x7f35ec07c500 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 -- 172.21.15.110:0/459092000 >> v1:172.21.15.204:6789/0 conn(0x7f35ec1260a0 legacy=0x7f35ec11bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 -- 172.21.15.110:0/459092000 >> v1:172.21.15.110:6789/0 conn(0x7f35ec07c500 legacy=0x7f35ec120a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 -- 172.21.15.110:0/459092000 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec11d5f0 con 0x7f35ec121270 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> v2:172.21.15.204:3300/0 conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.950 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> v2:172.21.15.204:3300/0 conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e4004ac0 con 0x7f35ec121270 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.951 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e40046e0 con 0x7f35ec121270 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e4012770 con 0x7f35ec121270 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec11d8c0 con 0x7f35ec121270 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec13cb00 con 0x7f35ec121270 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f35e4012a80 con 0x7f35ec121270 2019-10-13T04:05:00.952 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 0x7f35d402a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f35e4034dd0 con 0x7f35ec121270 2019-10-13T04:05:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.941+0000 7f35f283c700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 0x7f35d402a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.945+0000 7f35f283c700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 0x7f35d402a2e0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:00.953 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.945+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec042510 con 0x7f35ec121270 2019-10-13T04:05:00.958 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.953+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:00.958 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:00.953+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f35e4026160 con 0x7f35ec121270 2019-10-13T04:05:01.109 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.105+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f35ec0555c0 con 0x7f35ec121270 2019-10-13T04:05:01.110 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.105+0000 7f35f303d700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.110 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.105+0000 7f35e37fe700 1 -- 172.21.15.110:0/459092000 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f35e400a5b0 con 0x7f35ec121270 2019-10-13T04:05:01.112 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:01.114 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 msgr2=0x7f35d402a2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.114 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 0x7f35d402a2e0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 msgr2=0x7f35ec13c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f35e4000f00 tx=0x7f35e4010ad0).stop 2019-10-13T04:05:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 shutdown_connections 2019-10-13T04:05:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f35d4027de0 0x7f35d402a2e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 --2- 172.21.15.110:0/459092000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35ec121270 0x7f35ec13c5b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 shutdown_connections 2019-10-13T04:05:01.116 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 wait complete. 2019-10-13T04:05:01.117 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.109+0000 7f35f429f700 1 -- 172.21.15.110:0/459092000 >> 172.21.15.110:0/459092000 conn(0x7f35ec05e8c0 msgr2=0x7f35ec0a7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:01.124 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:01.125 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T04:05:01.227 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.217+0000 7f55b01e7700 1 Processor -- start 2019-10-13T04:05:01.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.217+0000 7f55b01e7700 1 -- start start 2019-10-13T04:05:01.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f55a8065060 0x7f55a80a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:01.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a80a8c10 0x7f55a80a9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:01.228 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a80a77f0 con 0x7f55a80a6410 2019-10-13T04:05:01.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a80a20e0 con 0x7f55a80a8c10 2019-10-13T04:05:01.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a80a2380 con 0x7f55a8065060 2019-10-13T04:05:01.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a80a8c10 0x7f55a80a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.229 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ae784700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f55a80a6410 0x7f55a8144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45958/0 (socket says 172.21.15.110:45958) 2019-10-13T04:05:01.230 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a80a8c10 0x7f55a80a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43250/0 (socket says 172.21.15.110:43250) 2019-10-13T04:05:01.230 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ae784700 1 -- 172.21.15.110:0/3032103705 learned_addr learned my addr 172.21.15.110:0/3032103705 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:01.230 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55adf83700 1 --2- 172.21.15.110:0/3032103705 >> v2:172.21.15.204:3300/0 conn(0x7f55a8065060 0x7f55a80a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 -- 172.21.15.110:0/3032103705 >> v2:172.21.15.204:3300/0 conn(0x7f55a8065060 msgr2=0x7f55a80a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> v2:172.21.15.204:3300/0 conn(0x7f55a8065060 0x7f55a80a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 -- 172.21.15.110:0/3032103705 >> v1:172.21.15.110:6789/0 conn(0x7f55a80a6410 legacy=0x7f55a8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.231 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 -- 172.21.15.110:0/3032103705 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a814f9f0 con 0x7f55a80a8c10 2019-10-13T04:05:01.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ad782700 1 -- 172.21.15.110:0/3032103705 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157830218 0 0) 0x7f55a80a77f0 con 0x7f55a80a6410 2019-10-13T04:05:01.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> v2:172.21.15.110:3300/0 conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f55a80a15e0 tx=0x7f5598007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> v2:172.21.15.110:3300/0 conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f55a80a15e0 tx=0x7f5598007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.232 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f55a80a15e0 tx=0x7f5598007710).ready entity=mon.0 client_cookie=bfb55e8939d08215 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f55a80a15e0 tx=0x7f5598007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ad782700 1 -- 172.21.15.110:0/3032103705 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55a00117a0 con 0x7f55a80a8c10 2019-10-13T04:05:01.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f55a80a15e0 tx=0x7f5598007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.233 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55aef85700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f55a80a15e0 tx=0x7f5598007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ad782700 1 -- 172.21.15.110:0/3032103705 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a0008b40 con 0x7f55a80a8c10 2019-10-13T04:05:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55ad782700 1 -- 172.21.15.110:0/3032103705 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55a0011d80 con 0x7f55a80a8c10 2019-10-13T04:05:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 msgr2=0x7f55a80a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f55a80a15e0 tx=0x7f5598007710).stop 2019-10-13T04:05:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- 172.21.15.110:0/3032103705 shutdown_connections 2019-10-13T04:05:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 --2- 172.21.15.110:0/3032103705 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a80a8c10 0x7f55a80a9030 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 --2- 172.21.15.110:0/3032103705 >> v2:172.21.15.204:3300/0 conn(0x7f55a8065060 0x7f55a80a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- 172.21.15.110:0/3032103705 shutdown_connections 2019-10-13T04:05:01.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- 172.21.15.110:0/3032103705 wait complete. 2019-10-13T04:05:01.236 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.221+0000 7f55b01e7700 1 -- 172.21.15.110:0/3032103705 >> 172.21.15.110:0/3032103705 conn(0x7f55a805afb0 msgr2=0x7f55a809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:01.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 Processor -- start 2019-10-13T04:05:01.241 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- start start 2019-10-13T04:05:01.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55a80635a0 0x7f55a80a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:01.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a811ab50 0x7f55a8115020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:01.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a8115570 con 0x7f55a80a5b80 2019-10-13T04:05:01.242 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a80a44e0 con 0x7f55a811ab50 2019-10-13T04:05:01.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a8115ff0 con 0x7f55a80635a0 2019-10-13T04:05:01.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a811ab50 0x7f55a8115020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55a811ab50 0x7f55a8115020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43252/0 (socket says 172.21.15.110:43252) 2019-10-13T04:05:01.243 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 -- 172.21.15.110:0/1456988230 learned_addr learned my addr 172.21.15.110:0/1456988230 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:01.244 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55ae784700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55a80635a0 0x7f55a80a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.244 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 -- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55a80635a0 msgr2=0x7f55a80a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.244 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55a80635a0 0x7f55a80a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.244 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 -- 172.21.15.110:0/1456988230 >> v1:172.21.15.204:6789/0 conn(0x7f55a80a5b80 legacy=0x7f55a80a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.253 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 -- 172.21.15.110:0/1456988230 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a80a2f50 con 0x7f55a811ab50 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> v2:172.21.15.110:3300/0 conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> v2:172.21.15.110:3300/0 conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5598012c90 con 0x7f55a811ab50 2019-10-13T04:05:01.254 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55980128b0 con 0x7f55a811ab50 2019-10-13T04:05:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a80a3220 con 0x7f55a811ab50 2019-10-13T04:05:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5598005910 con 0x7f55a811ab50 2019-10-13T04:05:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.225+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a8132c70 con 0x7f55a811ab50 2019-10-13T04:05:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f5598007180 con 0x7f55a811ab50 2019-10-13T04:05:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f559effd700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 0x7f5590029d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5598034f10 con 0x7f55a811ab50 2019-10-13T04:05:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f55ae784700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 0x7f5590029d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.229+0000 7f55ae784700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 0x7f5590029d20 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.233+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a80555c0 con 0x7f55a811ab50 2019-10-13T04:05:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.241+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.260 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.241+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f5598005c20 con 0x7f55a811ab50 2019-10-13T04:05:01.397 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.393+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55a807e800 con 0x7f55a811ab50 2019-10-13T04:05:01.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.393+0000 7f55adf83700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.398 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.393+0000 7f559effd700 1 -- 172.21.15.110:0/1456988230 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f55980247b0 con 0x7f55a811ab50 2019-10-13T04:05:01.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:05:01.401 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.110:6833/2227215932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2227215932},{"type":"v1","addr":"172.21.15.110:6833","nonce":2227215932}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":6}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T04:04:48.713885+0000","modified":"2019-10-13T04:04:52.629609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4321},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4321":{"gid":4321,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.204:6835/41460875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":41460875},{"type":"v1","addr":"172.21.15.204:6835","nonce":41460875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T04:05:01.401 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 6 2019-10-13T04:05:01.404 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 msgr2=0x7f5590029d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.404 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 0x7f5590029d20 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.405 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 msgr2=0x7f55a8115020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.405 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f55a4000fc0).stop 2019-10-13T04:05:01.405 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 shutdown_connections 2019-10-13T04:05:01.405 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55a80635a0 0x7f55a80a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f5590027820 0x7f5590029d20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.397+0000 7f55b01e7700 1 --2- 172.21.15.110:0/1456988230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55a811ab50 0x7f55a8115020 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.401+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 shutdown_connections 2019-10-13T04:05:01.406 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.401+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 wait complete. 2019-10-13T04:05:01.407 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.401+0000 7f55b01e7700 1 -- 172.21.15.110:0/1456988230 >> 172.21.15.110:0/1456988230 conn(0x7f55a80a8c10 msgr2=0x7f55a8116a80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:01.617 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4321}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 5, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4321': {u'export_targets': [], u'name': u'b', u'incarnation': 4, u'state_seq': 2, u'state': u'up:active', u'gid': 4321, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 41460875, u'type': u'v2', u'addr': u'172.21.15.204:6834'}, {u'nonce': 41460875, u'type': u'v1', u'addr': u'172.21.15.204:6835'}]}, u'addr': u'172.21.15.204:6835/41460875'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T04:04:48.713885+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T04:04:52.629609+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T04:05:01.617 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T04:05:01.617 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:05:01.618 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T04:05:01.723 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2019-10-13T04:05:01.723 INFO:teuthology.orchestra.run.smithi204.stdout: "cluster_fsid": "7c497bad-e78c-4b28-a8fa-4ca4e4cb6052", 2019-10-13T04:05:01.723 INFO:teuthology.orchestra.run.smithi204.stdout: "whoami": 0, 2019-10-13T04:05:01.723 INFO:teuthology.orchestra.run.smithi204.stdout: "id": 4321, 2019-10-13T04:05:01.723 INFO:teuthology.orchestra.run.smithi204.stdout: "want_state": "up:active", 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "state": "up:active", 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "rank_uptime": 10.11367257, 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "mdsmap_epoch": 5, 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_epoch": 23, 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_epoch_barrier": 23, 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout: "uptime": 10.421194494 2019-10-13T04:05:01.724 INFO:teuthology.orchestra.run.smithi204.stdout:} 2019-10-13T04:05:01.730 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "7c497bad-e78c-4b28-a8fa-4ca4e4cb6052", "whoami": 0, "id": 4321, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.11367257, "mdsmap_epoch": 5, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "uptime": 10.421194494 } 2019-10-13T04:05:01.730 INFO:teuthology.run_tasks:Running task ceph-fuse... 2019-10-13T04:05:01.764 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2019-10-13T04:05:01.764 INFO:tasks.ceph_fuse:config is None 2019-10-13T04:05:01.764 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2019-10-13T04:05:01.765 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2019-10-13T04:05:01.765 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:01.765 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-13T04:05:01.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 Processor -- start 2019-10-13T04:05:01.869 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 -- start start 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9d80a1970 0x7fc9d814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80a9410 con 0x7fc9d80a6410 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80a1d90 con 0x7fc9d8065060 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dfcdb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80a20e0 con 0x7fc9d80a1970 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9dea79700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9d80a1970 0x7fc9d814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.870 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9de278700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc9d80a6410 0x7fc9d8144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45974/0 (socket says 172.21.15.110:45974) 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.861+0000 7fc9de278700 1 -- 172.21.15.110:0/2300941262 learned_addr learned my addr 172.21.15.110:0/2300941262 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 -- 172.21.15.110:0/2300941262 >> v1:172.21.15.204:6789/0 conn(0x7fc9d8065060 legacy=0x7fc9d80a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 -- 172.21.15.110:0/2300941262 >> v1:172.21.15.110:6789/0 conn(0x7fc9d80a6410 legacy=0x7fc9d8144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 -- 172.21.15.110:0/2300941262 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80a77f0 con 0x7fc9d80a1970 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> v2:172.21.15.110:3300/0 conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.871 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> v2:172.21.15.110:3300/0 conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).ready entity=mon.0 client_cookie=fc659cf3f4b40f90 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dd276700 1 -- 172.21.15.110:0/2300941262 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d00119c0 con 0x7fc9d80a1970 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dea79700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dd276700 1 -- 172.21.15.110:0/2300941262 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d00115e0 con 0x7fc9d80a1970 2019-10-13T04:05:01.872 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dd276700 1 -- 172.21.15.110:0/2300941262 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d000a160 con 0x7fc9d80a1970 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 msgr2=0x7fc9d814f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7fc9d0008b00 tx=0x7fc9d000ff60).stop 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/2300941262 shutdown_connections 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/2300941262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d80a1970 0x7fc9d814f9f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/2300941262 shutdown_connections 2019-10-13T04:05:01.873 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/2300941262 wait complete. 2019-10-13T04:05:01.874 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.865+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/2300941262 >> 172.21.15.110:0/2300941262 conn(0x7fc9d805afb0 msgr2=0x7fc9d809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:01.876 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 Processor -- start 2019-10-13T04:05:01.876 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- start start 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9d805e8d0 0x7fc9d814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80a0570 con 0x7fc9d8060ae0 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d805fc00 con 0x7fc9d807b5e0 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d8061010 con 0x7fc9d805e8d0 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9d805e8d0 0x7fc9d814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.877 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9d805e8d0 0x7fc9d814f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43268/0 (socket says 172.21.15.110:43268) 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 -- 172.21.15.110:0/10970533 learned_addr learned my addr 172.21.15.110:0/10970533 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 -- 172.21.15.110:0/10970533 >> v1:172.21.15.204:6789/0 conn(0x7fc9d807b5e0 legacy=0x7fc9d812f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 -- 172.21.15.110:0/10970533 >> v1:172.21.15.110:6789/0 conn(0x7fc9d8060ae0 legacy=0x7fc9d805f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 -- 172.21.15.110:0/10970533 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d805fe30 con 0x7fc9d805e8d0 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> v2:172.21.15.110:3300/0 conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.878 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> v2:172.21.15.110:3300/0 conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d4011620 con 0x7fc9d805e8d0 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d4011240 con 0x7fc9d805e8d0 2019-10-13T04:05:01.879 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d814f6f0 con 0x7fc9d805e8d0 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d4011c00 con 0x7fc9d805e8d0 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.869+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d8130ed0 con 0x7fc9d805e8d0 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7fc9d4007180 con 0x7fc9d805e8d0 2019-10-13T04:05:01.880 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9ceffd700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 0x7fc9c0029d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:01.881 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc9d4034620 con 0x7fc9d805e8d0 2019-10-13T04:05:01.881 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9dda77700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 0x7fc9c0029d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:01.881 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.873+0000 7fc9dda77700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 0x7fc9c0029d40 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:01.882 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.877+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d80555c0 con 0x7fc9d805e8d0 2019-10-13T04:05:01.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.885+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:01.888 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:01.885+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7fc9d4007ce0 con 0x7fc9d805e8d0 2019-10-13T04:05:02.051 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.049+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d807e800 con 0x7fc9d805e8d0 2019-10-13T04:05:02.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.049+0000 7fc9de278700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.052 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.049+0000 7fc9ceffd700 1 -- 172.21.15.110:0/10970533 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14981 (secure 0 0 0) 0x7fc9d4025100 con 0x7fc9d805e8d0 2019-10-13T04:05:02.054 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:05:02.055 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":23,"fsid":"7c497bad-e78c-4b28-a8fa-4ca4e4cb6052","created":"2019-10-13T04:04:26.653820+0000","modified":"2019-10-13T04:04:48.713516+0000","last_up_change":"2019-10-13T04:04:36.316968+0000","last_in_change":"2019-10-13T04:04:36.316968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13T04:04:40.397189+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13T04:04:43.569877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13T04:04:46.432662+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"91e5f49d-d1d1-4381-9d98-0003d595cfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":14117},{"type":"v1","addr":"172.21.15.110:6817","nonce":14117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":14117},{"type":"v1","addr":"172.21.15.110:6819","nonce":14117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":14117},{"type":"v1","addr":"172.21.15.110:6823","nonce":14117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":14117},{"type":"v1","addr":"172.21.15.110:6821","nonce":14117}]},"public_addr":"172.21.15.110:6817/14117","cluster_addr":"172.21.15.110:6819/14117","heartbeat_back_addr":"172.21.15.110:6823/14117","heartbeat_front_addr":"172.21.15.110:6821/14117","state":["exists","up"]},{"osd":1,"uuid":"1ae65af5-8cf4-42f4-bbed-edc4a2b31b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":14116},{"type":"v1","addr":"172.21.15.110:6809","nonce":14116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":14116},{"type":"v1","addr":"172.21.15.110:6811","nonce":14116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":14116},{"type":"v1","addr":"172.21.15.110:6815","nonce":14116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":14116},{"type":"v1","addr":"172.21.15.110:6813","nonce":14116}]},"public_addr":"172.21.15.110:6809/14116","cluster_addr":"172.21.15.110:6811/14116","heartbeat_back_addr":"172.21.15.110:6815/14116","heartbeat_front_addr":"172.21.15.110:6813/14116","state":["exists","up"]},{"osd":2,"uuid":"2b38a9b6-4e8f-4957-8b7e-031ba1c8daf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":14118},{"type":"v1","addr":"172.21.15.110:6801","nonce":14118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":14118},{"type":"v1","addr":"172.21.15.110:6803","nonce":14118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":14118},{"type":"v1","addr":"172.21.15.110:6807","nonce":14118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":14118},{"type":"v1","addr":"172.21.15.110:6805","nonce":14118}]},"public_addr":"172.21.15.110:6801/14118","cluster_addr":"172.21.15.110:6803/14118","heartbeat_back_addr":"172.21.15.110:6807/14118","heartbeat_front_addr":"172.21.15.110:6805/14118","state":["exists","up"]},{"osd":3,"uuid":"f38f8d49-36c3-4dfa-8bb5-057288982141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":14157},{"type":"v1","addr":"172.21.15.110:6825","nonce":14157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":14157},{"type":"v1","addr":"172.21.15.110:6827","nonce":14157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":14157},{"type":"v1","addr":"172.21.15.110:6831","nonce":14157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":14157},{"type":"v1","addr":"172.21.15.110:6829","nonce":14157}]},"public_addr":"172.21.15.110:6825/14157","cluster_addr":"172.21.15.110:6827/14157","heartbeat_back_addr":"172.21.15.110:6831/14157","heartbeat_front_addr":"172.21.15.110:6829/14157","state":["exists","up"]},{"osd":4,"uuid":"1f731a59-e70a-4f9d-8114-c14e183b8b11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13912},{"type":"v1","addr":"172.21.15.204:6827","nonce":13912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13912},{"type":"v1","addr":"172.21.15.204:6829","nonce":13912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":13912},{"type":"v1","addr":"172.21.15.204:6833","nonce":13912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13912},{"type":"v1","addr":"172.21.15.204:6831","nonce":13912}]},"public_addr":"172.21.15.204:6827/13912","cluster_addr":"172.21.15.204:6829/13912","heartbeat_back_addr":"172.21.15.204:6833/13912","heartbeat_front_addr":"172.21.15.204:6831/13912","state":["exists","up"]},{"osd":5,"uuid":"566b8f0d-601e-451d-901c-3b5aa1f50b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13910},{"type":"v1","addr":"172.21.15.204:6811","nonce":13910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13910},{"type":"v1","addr":"172.21.15.204:6813","nonce":13910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":13910},{"type":"v1","addr":"172.21.15.204:6817","nonce":13910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13910},{"type":"v1","addr":"172.21.15.204:6815","nonce":13910}]},"public_addr":"172.21.15.204:6811/13910","cluster_addr":"172.21.15.204:6813/13910","heartbeat_back_addr":"172.21.15.204:6817/13910","heartbeat_front_addr":"172.21.15.204:6815/13910","state":["exists","up"]},{"osd":6,"uuid":"8cc0914d-13a9-4b3b-9df1-ba5519ef1a42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13911},{"type":"v1","addr":"172.21.15.204:6819","nonce":13911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13911},{"type":"v1","addr":"172.21.15.204:6821","nonce":13911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":13911},{"type":"v1","addr":"172.21.15.204:6825","nonce":13911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13911},{"type":"v1","addr":"172.21.15.204:6823","nonce":13911}]},"public_addr":"172.21.15.204:6819/13911","cluster_addr":"172.21.15.204:6821/13911","heartbeat_back_addr":"172.21.15.204:6825/13911","heartbeat_front_addr":"172.21.15.204:6823/13911","state":["exists","up"]},{"osd":7,"uuid":"e50eac16-d20a-4e5d-bbc2-f5eee81c0eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13913},{"type":"v1","addr":"172.21.15.204:6803","nonce":13913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13913},{"type":"v1","addr":"172.21.15.204:6805","nonce":13913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13913},{"type":"v1","addr":"172.21.15.204:6809","nonce":13913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13913},{"type":"v1","addr":"172.21.15.204:6807","nonce":13913}]},"public_addr":"172.21.15.204:6803/13913","cluster_addr":"172.21.15.204:6805/13913","heartbeat_back_addr":"172.21.15.204:6809/13913","heartbeat_front_addr":"172.21.15.204:6807/13913","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.691062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.645937+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.561496+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.628268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.686364+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.638122+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.684518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-13T04:04:34.369632+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-13T04:05:02.058 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 msgr2=0x7fc9c0029d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.058 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 0x7fc9c0029d40 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.058 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 msgr2=0x7fc9d814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.058 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7fc9d40013d0 tx=0x7fc9d0010470).stop 2019-10-13T04:05:02.058 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 shutdown_connections 2019-10-13T04:05:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7fc9c0027840 0x7fc9c0029d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 --2- 172.21.15.110:0/10970533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9d805e8d0 0x7fc9d814f1a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 shutdown_connections 2019-10-13T04:05:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 wait complete. 2019-10-13T04:05:02.059 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.053+0000 7fc9dfcdb700 1 -- 172.21.15.110:0/10970533 >> 172.21.15.110:0/10970533 conn(0x7fc9d80a6410 msgr2=0x7fc9d809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.071 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:40.397189+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:43.569877+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-13T04:04:46.432662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-13T04:05:02.072 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:02.072 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-13T04:05:02.169 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 Processor -- start 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 -- start start 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ff9e80a88e0 0x7ff9e80a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9e80a17e0 0x7ff9e814e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80a0fc0 con 0x7ff9e80aa2b0 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e8059e10 con 0x7ff9e80a17e0 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9edef9700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e805a0b0 con 0x7ff9e80a88e0 2019-10-13T04:05:02.170 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9ecc97700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9e80a17e0 0x7ff9e814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.161+0000 7ff9e7fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff9e80aa2b0 0x7ff9e807e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:45988/0 (socket says 172.21.15.110:45988) 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e7fff700 1 -- 172.21.15.110:0/727055698 learned_addr learned my addr 172.21.15.110:0/727055698 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/727055698 >> v2:172.21.15.204:3300/0 conn(0x7ff9e80a88e0 0x7ff9e80a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 -- 172.21.15.110:0/727055698 >> v2:172.21.15.204:3300/0 conn(0x7ff9e80a88e0 msgr2=0x7ff9e80a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> v2:172.21.15.204:3300/0 conn(0x7ff9e80a88e0 0x7ff9e80a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 -- 172.21.15.110:0/727055698 >> v1:172.21.15.110:6789/0 conn(0x7ff9e80aa2b0 legacy=0x7ff9e807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.171 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 -- 172.21.15.110:0/727055698 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80a75f0 con 0x7ff9e80a17e0 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e6ffd700 1 -- 172.21.15.110:0/727055698 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447944120 0 0) 0x7ff9e80a0fc0 con 0x7ff9e80aa2b0 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> v2:172.21.15.110:3300/0 conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> v2:172.21.15.110:3300/0 conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).ready entity=mon.0 client_cookie=1fceedcd1581482f server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.172 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e6ffd700 1 -- 172.21.15.110:0/727055698 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0008ba0 con 0x7ff9e80a17e0 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9ecc97700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e6ffd700 1 -- 172.21.15.110:0/727055698 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00087c0 con 0x7ff9e80a17e0 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9e6ffd700 1 -- 172.21.15.110:0/727055698 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0011610 con 0x7ff9e80a17e0 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 msgr2=0x7ff9e814e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7ff9e8059fd0 tx=0x7ff9d0007710).stop 2019-10-13T04:05:02.173 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- 172.21.15.110:0/727055698 shutdown_connections 2019-10-13T04:05:02.174 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 --2- 172.21.15.110:0/727055698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e80a17e0 0x7ff9e814e900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.174 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 --2- 172.21.15.110:0/727055698 >> v2:172.21.15.204:3300/0 conn(0x7ff9e80a88e0 0x7ff9e80a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.174 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- 172.21.15.110:0/727055698 shutdown_connections 2019-10-13T04:05:02.174 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- 172.21.15.110:0/727055698 wait complete. 2019-10-13T04:05:02.174 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- 172.21.15.110:0/727055698 >> 172.21.15.110:0/727055698 conn(0x7ff9e805e980 msgr2=0x7ff9e80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.177 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 Processor -- start 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.165+0000 7ff9edef9700 1 -- start start 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9e8068bd0 0x7ff9e81342b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9e807b910 0x7ff9e809d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e807c6f0 con 0x7ff9e80789c0 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e809d7a0 con 0x7ff9e807b910 2019-10-13T04:05:02.178 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e807be90 con 0x7ff9e8068bd0 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9e807b910 0x7ff9e809d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9e807b910 0x7ff9e809d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43282/0 (socket says 172.21.15.110:43282) 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 -- 172.21.15.110:0/2545668900 learned_addr learned my addr 172.21.15.110:0/2545668900 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e7fff700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9e8068bd0 0x7ff9e81342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 -- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9e8068bd0 msgr2=0x7ff9e81342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9e8068bd0 0x7ff9e81342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.179 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 -- 172.21.15.110:0/2545668900 >> v1:172.21.15.204:6789/0 conn(0x7ff9e80789c0 legacy=0x7ff9e80a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 -- 172.21.15.110:0/2545668900 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e807b520 con 0x7ff9e807b910 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> v2:172.21.15.110:3300/0 conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> v2:172.21.15.110:3300/0 conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9d0012b60 con 0x7ff9e807b910 2019-10-13T04:05:02.180 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8078de0 con 0x7ff9e807b910 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d0012780 con 0x7ff9e807b910 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9d0009390 con 0x7ff9e807b910 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e8079310 con 0x7ff9e807b910 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7ff9d0007170 con 0x7ff9e807b910 2019-10-13T04:05:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 0x7ff9c0029ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff9d0034fa0 con 0x7ff9e807b910 2019-10-13T04:05:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e7fff700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 0x7ff9c0029ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.182 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.169+0000 7ff9e7fff700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 0x7ff9c0029ce0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.183 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.177+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80555c0 con 0x7ff9e807b910 2019-10-13T04:05:02.187 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.185+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.187 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.185+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7ff9d0026330 con 0x7ff9e807b910 2019-10-13T04:05:02.337 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.333+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff9e80a79c0 con 0x7ff9e807b910 2019-10-13T04:05:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.333+0000 7ff9e77fe700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.338 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.333+0000 7ff9e4ff9700 1 -- 172.21.15.110:0/2545668900 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff9d00096b0 con 0x7ff9e807b910 2019-10-13T04:05:02.338 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 msgr2=0x7ff9c0029ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.341 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 0x7ff9c0029ce0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 msgr2=0x7ff9e809d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7ff9e807c050 tx=0x7ff9d8001340).stop 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 shutdown_connections 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9e8068bd0 0x7ff9e81342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7ff9c00277e0 0x7ff9c0029ce0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.342 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 --2- 172.21.15.110:0/2545668900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9e807b910 0x7ff9e809d250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 shutdown_connections 2019-10-13T04:05:02.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 wait complete. 2019-10-13T04:05:02.343 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.337+0000 7ff9edef9700 1 -- 172.21.15.110:0/2545668900 >> 172.21.15.110:0/2545668900 conn(0x7ff9e80a89a0 msgr2=0x7ff9e811edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.351 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:02.351 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-10-13T04:05:02.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 Processor -- start 2019-10-13T04:05:02.454 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 -- start start 2019-10-13T04:05:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f07e814d900 0x7f07e814fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80a0fc0 con 0x7f07e80a88e0 2019-10-13T04:05:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80a75f0 con 0x7f07e80aa2b0 2019-10-13T04:05:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07f057a700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e80a1a90 con 0x7f07e814d900 2019-10-13T04:05:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ee316700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f07e80a88e0 0x7f07e80a1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46002/0 (socket says 172.21.15.110:46002) 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ee316700 1 -- 172.21.15.110:0/2904423747 learned_addr learned my addr 172.21.15.110:0/2904423747 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ef318700 1 --2- 172.21.15.110:0/2904423747 >> v2:172.21.15.204:3300/0 conn(0x7f07e814d900 0x7f07e814fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949498796 0 0) 0x7f07e80a0fc0 con 0x7f07e80a88e0 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07d40030a0 con 0x7f07e80a88e0 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 779117610 0 0) 0x7f07d40030a0 con 0x7f07e80a88e0 2019-10-13T04:05:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 >> v2:172.21.15.204:3300/0 conn(0x7f07e814d900 msgr2=0x7f07e814fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 --2- 172.21.15.110:0/2904423747 >> v2:172.21.15.204:3300/0 conn(0x7f07e814d900 0x7f07e814fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 >> v1:172.21.15.204:6789/0 conn(0x7f07e80aa2b0 legacy=0x7f07e807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8059a20 con 0x7f07e80a88e0 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f07e4001750 con 0x7f07e80a88e0 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 >> v1:172.21.15.110:6789/0 conn(0x7f07e80a88e0 legacy=0x7f07e80a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07d4004410 0x7f07d4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07d4004d80 0x7f07d400b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80a1a90 con 0x7f07d4003720 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d404fc20 con 0x7f07d4004410 2019-10-13T04:05:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d404fe60 con 0x7f07d4004d80 2019-10-13T04:05:02.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ee316700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07d4004410 0x7f07d4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ef318700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07d4004d80 0x7f07d400b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 -- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07d4004d80 msgr2=0x7f07d400b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:02.459 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07d4004d80 0x7f07d400b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 -- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07d4004410 msgr2=0x7f07d4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:02.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07d4004410 0x7f07d4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 -- 172.21.15.110:0/2904423747 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4050160 con 0x7f07d4003720 2019-10-13T04:05:02.460 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07ef318700 1 -- 172.21.15.110:0/2904423747 reap_dead start 2019-10-13T04:05:02.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f07d8001000 tx=0x7f07e4001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f07d8001000 tx=0x7f07e4001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=READY pgs=190 cs=0 l=1 rx=0x7f07d8001000 tx=0x7f07e4001bd0).ready entity=mon.0 client_cookie=d5c131e626648a3b server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.461 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f07d8001000 tx=0x7f07e4001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d8011490 con 0x7f07d4003720 2019-10-13T04:05:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f07d8001000 tx=0x7f07e4001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07eeb17700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f07d8001000 tx=0x7f07e4001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d80110b0 con 0x7f07d4003720 2019-10-13T04:05:02.462 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.445+0000 7f07edb15700 1 -- 172.21.15.110:0/2904423747 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d8011a90 con 0x7f07d4003720 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 msgr2=0x7f07d4003b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 secure :-1 s=READY pgs=190 cs=0 l=1 rx=0x7f07d8001000 tx=0x7f07e4001bd0).stop 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- 172.21.15.110:0/2904423747 shutdown_connections 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 --2- 172.21.15.110:0/2904423747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07d4003720 0x7f07d4003b40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- 172.21.15.110:0/2904423747 shutdown_connections 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- 172.21.15.110:0/2904423747 wait complete. 2019-10-13T04:05:02.463 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- 172.21.15.110:0/2904423747 >> 172.21.15.110:0/2904423747 conn(0x7f07e805e980 msgr2=0x7f07e80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 Processor -- start 2019-10-13T04:05:02.464 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.449+0000 7f07f057a700 1 -- start start 2019-10-13T04:05:02.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f07e80a0540 0x7f07e8135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.468 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07e8136330 0x7f07e8136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e8133020 con 0x7f07e809eb30 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e8136ca0 con 0x7f07e8136330 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e809f690 con 0x7f07e80a0540 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07e8136330 0x7f07e8136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07e8136330 0x7f07e8136750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43306/0 (socket says 172.21.15.110:43306) 2019-10-13T04:05:02.469 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 -- 172.21.15.110:0/1881248736 learned_addr learned my addr 172.21.15.110:0/1881248736 (peer_addr_for_me v2:172.21.15.110:0/0) 2019-10-13T04:05:02.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ee316700 1 --2- 172.21.15.110:0/1881248736 >> v2:172.21.15.204:3300/0 conn(0x7f07e80a0540 0x7f07e8135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 -- 172.21.15.110:0/1881248736 >> v2:172.21.15.204:3300/0 conn(0x7f07e80a0540 msgr2=0x7f07e8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> v2:172.21.15.204:3300/0 conn(0x7f07e80a0540 0x7f07e8135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.470 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 -- 172.21.15.110:0/1881248736 >> v1:172.21.15.204:6789/0 conn(0x7f07e809eb30 legacy=0x7f07e8132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.471 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 -- 172.21.15.110:0/1881248736 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80a0960 con 0x7f07e8136330 2019-10-13T04:05:02.471 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> v2:172.21.15.110:3300/0 conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.471 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> v2:172.21.15.110:3300/0 conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.471 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.471 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07e0005c90 con 0x7f07e8136330 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e00058b0 con 0x7f07e8136330 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e809dfd0 con 0x7f07e8136330 2019-10-13T04:05:02.472 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07e0008890 con 0x7f07e8136330 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e809e370 con 0x7f07e8136330 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f07e0008a90 con 0x7f07e8136330 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 0x7f07d0029d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.473 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.453+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f07e0034460 con 0x7f07e8136330 2019-10-13T04:05:02.474 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.457+0000 7f07eeb17700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 0x7f07d0029d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.474 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.457+0000 7f07eeb17700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 0x7f07d0029d60 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.474 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.461+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e80555c0 con 0x7f07e8136330 2019-10-13T04:05:02.474 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.465+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.474 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.465+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f07e0008d80 con 0x7f07e8136330 2019-10-13T04:05:02.614 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.609+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f07e80a79c0 con 0x7f07e8136330 2019-10-13T04:05:02.615 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07ef318700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.615 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07df7fe700 1 -- 172.21.15.110:0/1881248736 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f07e0007460 con 0x7f07e8136330 2019-10-13T04:05:02.616 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:02.618 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 msgr2=0x7f07d0029d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.618 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 0x7f07d0029d60 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 msgr2=0x7f07e8136750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7f07d8000f10 tx=0x7f07e4001d90).stop 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 shutdown_connections 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f07d0027860 0x7f07d0029d60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 --2- 172.21.15.110:0/1881248736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07e8136330 0x7f07e8136750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 --2- 172.21.15.110:0/1881248736 >> v2:172.21.15.204:3300/0 conn(0x7f07e80a0540 0x7f07e8135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.619 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 shutdown_connections 2019-10-13T04:05:02.620 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 wait complete. 2019-10-13T04:05:02.620 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.613+0000 7f07f057a700 1 -- 172.21.15.110:0/1881248736 >> 172.21.15.110:0/1881248736 conn(0x7f07e80a96e0 msgr2=0x7f07e807d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.628 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:02.628 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-10-13T04:05:02.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 Processor -- start 2019-10-13T04:05:02.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 -- start start 2019-10-13T04:05:02.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e180aa2b0 0x7f9e1807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e1807ebf0 0x7f9e180a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:02.731 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180a0fc0 con 0x7f9e180a17e0 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e18059e10 con 0x7f9e180aa2b0 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1f258700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1805a0b0 con 0x7f9e1807ebf0 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1d7f5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e180aa2b0 0x7f9e1807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1dff6700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9e180a17e0 0x7f9e18147510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46026/0 (socket says 172.21.15.110:46026) 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1dff6700 1 -- 172.21.15.110:0/1617196661 learned_addr learned my addr 172.21.15.110:0/1617196661 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:02.732 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e1cff4700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e1807ebf0 0x7f9e180a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e0ffff700 1 -- 172.21.15.110:0/1617196661 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551728605 0 0) 0x7f9e180a0fc0 con 0x7f9e180a17e0 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.721+0000 7f9e0ffff700 1 -- 172.21.15.110:0/1617196661 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e040030a0 con 0x7f9e180a17e0 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/1617196661 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e1807ebf0 msgr2=0x7f9e180a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e1807ebf0 0x7f9e180a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/1617196661 >> v1:172.21.15.110:6789/0 conn(0x7f9e180a17e0 legacy=0x7f9e18147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/1617196661 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e180a75f0 con 0x7f9e180aa2b0 2019-10-13T04:05:02.733 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> v2:172.21.15.110:3300/0 conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> v2:172.21.15.110:3300/0 conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).ready entity=mon.0 client_cookie=8f5ac09060c7f9dc server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e0ffff700 1 -- 172.21.15.110:0/1617196661 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1400a160 con 0x7f9e180aa2b0 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.734 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e0ffff700 1 -- 172.21.15.110:0/1617196661 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14008bb0 con 0x7f9e180aa2b0 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e0ffff700 1 -- 172.21.15.110:0/1617196661 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1400a740 con 0x7f9e180aa2b0 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 -- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 msgr2=0x7f9e1807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f9e1805ccf0 tx=0x7f9e08002ff0).stop 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 -- 172.21.15.110:0/1617196661 shutdown_connections 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e180aa2b0 0x7f9e1807e6a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.735 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 --2- 172.21.15.110:0/1617196661 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e1807ebf0 0x7f9e180a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:02.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 -- 172.21.15.110:0/1617196661 shutdown_connections 2019-10-13T04:05:02.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 -- 172.21.15.110:0/1617196661 wait complete. 2019-10-13T04:05:02.736 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.725+0000 7f9e1f258700 1 -- 172.21.15.110:0/1617196661 >> 172.21.15.110:0/1617196661 conn(0x7f9e1805e980 msgr2=0x7f9e180a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.740 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 Processor -- start 2019-10-13T04:05:02.740 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- start start 2019-10-13T04:05:02.740 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e18126370 0x7f9e180a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18120c10 con 0x7f9e1807bd20 2019-10-13T04:05:02.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18120880 con 0x7f9e180a6080 2019-10-13T04:05:02.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e181200e0 con 0x7f9e18126370 2019-10-13T04:05:02.741 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1dff6700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9e1807bd20 0x7f9e1813a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46030/0 (socket says 172.21.15.110:46030) 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1dff6700 1 -- 172.21.15.110:0/2812366742 learned_addr learned my addr 172.21.15.110:0/2812366742 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> v2:172.21.15.110:3300/0 conn(0x7f9e18126370 0x7f9e180a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/2812366742 >> v1:172.21.15.204:6789/0 conn(0x7f9e180a6080 legacy=0x7f9e1807b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/2812366742 >> v1:172.21.15.110:6789/0 conn(0x7f9e1807bd20 legacy=0x7f9e1813a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 -- 172.21.15.110:0/2812366742 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1811e370 con 0x7f9e18126370 2019-10-13T04:05:02.742 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> v2:172.21.15.110:3300/0 conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> v2:172.21.15.110:3300/0 conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e14007340 con 0x7f9e18126370 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.743 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400a360 con 0x7f9e18126370 2019-10-13T04:05:02.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1811ebc0 con 0x7f9e18126370 2019-10-13T04:05:02.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e14005910 con 0x7f9e18126370 2019-10-13T04:05:02.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1813bcd0 con 0x7f9e18126370 2019-10-13T04:05:02.744 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.729+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f9e14005bb0 con 0x7f9e18126370 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e0dffb700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 0x7f9df8029cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9e14035240 con 0x7f9e18126370 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e1cff4700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 0x7f9df8029cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:02.745 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.733+0000 7f9e1cff4700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 0x7f9df8029cf0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:02.746 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.737+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e180555c0 con 0x7f9e18126370 2019-10-13T04:05:02.749 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.745+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.749 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.745+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f9e140130c0 con 0x7f9e18126370 2019-10-13T04:05:02.901 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.897+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9e180a7970 con 0x7f9e18126370 2019-10-13T04:05:02.902 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.897+0000 7f9e1d7f5700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:02.902 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.897+0000 7f9e0dffb700 1 -- 172.21.15.110:0/2812366742 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9e1401de40 con 0x7f9e18126370 2019-10-13T04:05:02.902 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2019-10-13T04:05:02.904 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 msgr2=0x7f9df8029cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.904 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 0x7f9df8029cf0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.904 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 msgr2=0x7f9e180a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f9e14007d70 tx=0x7f9e140085e0).stop 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 shutdown_connections 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f9df80277f0 0x7f9df8029cf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 --2- 172.21.15.110:0/2812366742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e18126370 0x7f9e180a5b30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 shutdown_connections 2019-10-13T04:05:02.905 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 wait complete. 2019-10-13T04:05:02.906 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:02.901+0000 7f9e1f258700 1 -- 172.21.15.110:0/2812366742 >> 172.21.15.110:0/2812366742 conn(0x7f9e180a16c0 msgr2=0x7f9e1807d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:02.917 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2019-10-13T04:05:02.917 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:02.917 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.005+0000 7f90f2d7e700 1 Processor -- start 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.005+0000 7f90f2d7e700 1 -- start start 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a1970 0x7f90ec14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0a9410 con 0x7f90ec065060 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0a1d90 con 0x7f90ec0a6410 2019-10-13T04:05:03.018 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0a20e0 con 0x7f90ec0a1970 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f0b1a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f90ec065060 0x7f90ec0a8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46036/0 (socket says 172.21.15.110:46036) 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f0b1a700 1 -- 172.21.15.110:0/390958430 learned_addr learned my addr 172.21.15.110:0/390958430 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a1970 0x7f90ec14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466419573 0 0) 0x7f90ec0a9410 con 0x7f90ec065060 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d80030a0 con 0x7f90ec065060 2019-10-13T04:05:03.019 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190243009 0 0) 0x7f90ec0a1d90 con 0x7f90ec0a6410 2019-10-13T04:05:03.020 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ec0a9410 con 0x7f90ec0a6410 2019-10-13T04:05:03.020 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2307033478 0 0) 0x7f90d80030a0 con 0x7f90ec065060 2019-10-13T04:05:03.020 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a1970 msgr2=0x7f90ec14f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:03.020 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a1970 0x7f90ec14f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:03.022 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 >> v1:172.21.15.204:6789/0 conn(0x7f90ec0a6410 legacy=0x7f90ec144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0a77f0 con 0x7f90ec065060 2019-10-13T04:05:03.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2578934224 0 0) 0x7f90e8001750 con 0x7f90ec065060 2019-10-13T04:05:03.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 >> v1:172.21.15.110:6789/0 conn(0x7f90ec065060 legacy=0x7f90ec0a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.023 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90d8003930 0x7f90d8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:03.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90d8004410 0x7f90d8004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:03.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-10-13T04:05:03.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0a20e0 con 0x7f90d8004e20 2019-10-13T04:05:03.024 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d804fbc0 con 0x7f90d8004410 2019-10-13T04:05:03.026 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d804fe00 con 0x7f90d8003930 2019-10-13T04:05:03.026 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f0b1a700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90d8004410 0x7f90d8004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.026 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.026 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f131b700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90d8003930 0x7f90d8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.026 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 -- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90d8003930 msgr2=0x7f90d8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90d8003930 0x7f90d8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 -- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90d8004410 msgr2=0x7f90d8004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90d8004410 0x7f90d8004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 -- 172.21.15.110:0/390958430 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d8050100 con 0x7f90d8004e20 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.027 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).ready entity=mon.0 client_cookie=e1a63302c628bfd1 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e4007160 con 0x7f90d8004e20 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4005bb0 con 0x7f90d8004e20 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90e3fff700 1 -- 172.21.15.110:0/390958430 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e4007740 con 0x7f90d8004e20 2019-10-13T04:05:03.028 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 msgr2=0x7f90d804f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/390958430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90d8004e20 0x7f90d804f670 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f90e40032b0 tx=0x7f90e8001ba0).stop 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f1b1c700 1 -- 172.21.15.110:0/390958430 reap_dead start 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- 172.21.15.110:0/390958430 shutdown_connections 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.009+0000 7f90f2d7e700 1 -- 172.21.15.110:0/390958430 shutdown_connections 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.013+0000 7f90f2d7e700 1 -- 172.21.15.110:0/390958430 wait complete. 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.013+0000 7f90f2d7e700 1 -- 172.21.15.110:0/390958430 >> 172.21.15.110:0/390958430 conn(0x7f90ec05afb0 msgr2=0x7f90ec09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:03.029 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.013+0000 7f90f2d7e700 1 Processor -- start 2019-10-13T04:05:03.030 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.013+0000 7f90f2d7e700 1 -- start start 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f90ec059e10 0x7f90ec124780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a51d0 0x7f90ec11f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0a44d0 con 0x7f90ec11f090 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ec0a8700 con 0x7f90ec059e10 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec123350 con 0x7f90ec0a51d0 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f0b1a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f90ec11f090 0x7f90ec0a49d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:46048/0 (socket says 172.21.15.110:46048) 2019-10-13T04:05:03.034 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f0b1a700 1 -- 172.21.15.110:0/4099462808 learned_addr learned my addr 172.21.15.110:0/4099462808 (peer_addr_for_me v1:172.21.15.110:0/0) 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> v2:172.21.15.204:3300/0 conn(0x7f90ec059e10 0x7f90ec124780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f1b1c700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a51d0 0x7f90ec11f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913785438 0 0) 0x7f90ec0a44d0 con 0x7f90ec11f090 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc0030a0 con 0x7f90ec11f090 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 -- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a51d0 msgr2=0x7f90ec11f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a51d0 0x7f90ec11f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 -- 172.21.15.110:0/4099462808 >> v1:172.21.15.110:6789/0 conn(0x7f90ec11f090 legacy=0x7f90ec0a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 -- 172.21.15.110:0/4099462808 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec13b3d0 con 0x7f90ec059e10 2019-10-13T04:05:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> v2:172.21.15.204:3300/0 conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> v2:172.21.15.204:3300/0 conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90dc012bc0 con 0x7f90ec059e10 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc0127e0 con 0x7f90ec059e10 2019-10-13T04:05:03.037 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec13a3c0 con 0x7f90ec059e10 2019-10-13T04:05:03.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90dc00e750 con 0x7f90ec059e10 2019-10-13T04:05:03.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec13a8c0 con 0x7f90ec059e10 2019-10-13T04:05:03.038 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 3) v1 ==== 32579+0+0 (secure 0 0 0) 0x7f90dc00e950 con 0x7f90ec059e10 2019-10-13T04:05:03.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.017+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.021+0000 7f90e1ffb700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 0x7f90cc02a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-10-13T04:05:03.039 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.021+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f90dc034ff0 con 0x7f90ec059e10 2019-10-13T04:05:03.040 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.021+0000 7f90f0b1a700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 0x7f90cc02a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-10-13T04:05:03.040 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.021+0000 7f90f0b1a700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 0x7f90cc02a2a0 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-10-13T04:05:03.040 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.021+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec0555c0 con 0x7f90ec059e10 2019-10-13T04:05:03.040 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.029+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.041 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.029+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+107904 (secure 0 0 0) 0x7f90dc00ec40 con 0x7f90ec059e10 2019-10-13T04:05:03.198 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.193+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90ec066520 con 0x7f90ec059e10 2019-10-13T04:05:03.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.197+0000 7f90f131b700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-10-13T04:05:03.199 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.197+0000 7f90e1ffb700 1 -- 172.21.15.110:0/4099462808 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f90dc003940 con 0x7f90ec059e10 2019-10-13T04:05:03.201 INFO:teuthology.orchestra.run.smithi110.stdout: 2019-10-13T04:05:03.201 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4377,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.110:6833/2227215932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2227215932},{"type":"v1","addr":"172.21.15.110:6833","nonce":2227215932}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":6}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-10-13T04:04:48.713885+0000","modified":"2019-10-13T04:04:52.629609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4321},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4321":{"gid":4321,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.204:6835/41460875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":41460875},{"type":"v1","addr":"172.21.15.204:6835","nonce":41460875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-10-13T04:05:03.201 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 6 2019-10-13T04:05:03.203 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 msgr2=0x7f90cc02a2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 0x7f90cc02a2a0 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 msgr2=0x7f90ec124780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f90ec120ff0 tx=0x7f90e800b540).stop 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 shutdown_connections 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:6800/13789,v1:172.21.15.204:6801/13789] conn(0x7f90cc027da0 0x7f90cc02a2a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:03.204 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90ec0a51d0 0x7f90ec11f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:03.205 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 --2- 172.21.15.110:0/4099462808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90ec059e10 0x7f90ec124780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-10-13T04:05:03.205 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 shutdown_connections 2019-10-13T04:05:03.205 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 wait complete. 2019-10-13T04:05:03.205 INFO:teuthology.orchestra.run.smithi110.stderr:2019-10-13T04:05:03.201+0000 7f90f2d7e700 1 -- 172.21.15.110:0/4099462808 >> 172.21.15.110:0/4099462808 conn(0x7f90ec0a36f0 msgr2=0x7f90ec125430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-10-13T04:05:03.415 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4321}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 5, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4321': {u'export_targets': [], u'name': u'b', u'incarnation': 4, u'state_seq': 2, u'state': u'up:active', u'gid': 4321, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 41460875, u'type': u'v2', u'addr': u'172.21.15.204:6834'}, {u'nonce': 41460875, u'type': u'v1', u'addr': u'172.21.15.204:6835'}]}, u'addr': u'172.21.15.204:6835/41460875'}}, u'fs_name': u'cephfs', u'created': u'2019-10-13T04:04:48.713885+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-10-13T04:04:52.629609+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-10-13T04:05:03.415 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-10-13T04:05:03.416 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:05:03.416 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-10-13T04:05:03.508 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2019-10-13T04:05:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "cluster_fsid": "7c497bad-e78c-4b28-a8fa-4ca4e4cb6052", 2019-10-13T04:05:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "whoami": 0, 2019-10-13T04:05:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "id": 4321, 2019-10-13T04:05:03.509 INFO:teuthology.orchestra.run.smithi204.stdout: "want_state": "up:active", 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "state": "up:active", 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "rank_uptime": 11.899244961999999, 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "mdsmap_epoch": 5, 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_epoch": 23, 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_epoch_barrier": 23, 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout: "uptime": 12.206768368000001 2019-10-13T04:05:03.510 INFO:teuthology.orchestra.run.smithi204.stdout:} 2019-10-13T04:05:03.515 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "7c497bad-e78c-4b28-a8fa-4ca4e4cb6052", "whoami": 0, "id": 4321, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.899244961999999, "mdsmap_epoch": 5, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "uptime": 12.206768368000001 } 2019-10-13T04:05:03.515 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2019-10-13T04:05:03.515 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2019-10-13T04:05:03.515 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi110.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2019-10-13T04:05:03.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.516 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.516 INFO:teuthology.orchestra.run.smithi110:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:03.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.528 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.528 INFO:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T04:05:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T04:05:03.611 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-10-13T04:05:03.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.612 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.612 INFO:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2019-10-13T04:05:03.691 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2019-10-13T04:05:03.691 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.691 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:03.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.761 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.761 INFO:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-10-13T04:05:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-10-13T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-10-13T04:05:03.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.781 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.781 INFO:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2019-10-13T04:05:03.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:2019-10-13T04:05:03.813+0000 7fe2cc7acf40 -1 init, newargv = 0x561ebe199660 newargc=7ceph-fuse[ 2019-10-13T04:05:03.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:16257]: starting ceph client 2019-10-13T04:05:03.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[16257]: starting fuse 2019-10-13T04:05:03.855 INFO:teuthology.orchestra.run.smithi110.stdout:49 2019-10-13T04:05:03.855 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2019-10-13T04:05:03.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> sudo python2 -c ' 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> import glob 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> import re 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> import os 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> import subprocess 2019-10-13T04:05:03.856 INFO:teuthology.orchestra.run.smithi110:> 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> def find_socket(client_name): 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2019-10-13T04:05:03.857 INFO:teuthology.orchestra.run.smithi110:> return files[0] 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> for f in files: 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> return f 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> 2019-10-13T04:05:03.858 INFO:teuthology.orchestra.run.smithi110:> print find_socket("client.0") 2019-10-13T04:05:03.859 INFO:teuthology.orchestra.run.smithi110:> ' 2019-10-13T04:05:03.959 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.0.16257.asok 2019-10-13T04:05:03.959 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.16257.asok 2019-10-13T04:05:03.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:03.960 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:03.960 INFO:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.16257.asok status 2019-10-13T04:05:04.143 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2019-10-13T04:05:04.143 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2019-10-13T04:05:04.143 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "a8ee8ac95ca822967c9dffe2a705f1d5502812bd", 2019-10-13T04:05:04.143 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev)", 2019-10-13T04:05:04.143 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "0", 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "16257", 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T04:05:04.144 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4662, 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2019-10-13T04:05:04.145 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4662 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:0", 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1065854802 2019-10-13T04:05:04.146 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:0", 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1065854802 2019-10-13T04:05:04.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.4662 172.21.15.110:0/1065854802", 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "172.21.15.110:0/1065854802", 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 5, 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 23, 2019-10-13T04:05:04.148 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2019-10-13T04:05:04.149 INFO:teuthology.orchestra.run.smithi110.stdout: "blacklisted": false 2019-10-13T04:05:04.149 INFO:teuthology.orchestra.run.smithi110.stdout:} 2019-10-13T04:05:04.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:04.149 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:04.150 INFO:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2019-10-13T04:05:04.150 INFO:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.161 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2019-10-13T04:05:04.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:05:04.162 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:04.162 INFO:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.245 INFO:teuthology.run_tasks:Running task check-counter... 2019-10-13T04:05:04.258 INFO:teuthology.run_tasks:Running task workunit... 2019-10-13T04:05:04.271 INFO:tasks.workunit:Pulling workunits from ref a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:05:04.272 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-10-13T04:05:04.272 INFO:tasks.workunit:timeout=3h 2019-10-13T04:05:04.272 INFO:tasks.workunit:cleanup=True 2019-10-13T04:05:04.272 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:04.272 INFO:teuthology.orchestra.run.smithi110:> stat -- /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.298 INFO:teuthology.orchestra.run.smithi110.stdout: File: /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.298 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2019-10-13T04:05:04.298 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 31h/49d Inode: 1 Links: 2 2019-10-13T04:05:04.298 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-13T04:05:04.299 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2019-10-13T04:05:04.299 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2019-10-13 04:04:51.609710159 +0000 2019-10-13T04:05:04.299 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2019-10-13 04:05:04.239241283 +0000 2019-10-13T04:05:04.299 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2019-10-13T04:05:04.299 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2019-10-13T04:05:04.299 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:04.300 INFO:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2019-10-13T04:05:04.396 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:04.396 INFO:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a8ee8ac95ca822967c9dffe2a705f1d5502812bd 2019-10-13T04:05:04.511 INFO:tasks.workunit.client.0.smithi110.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-10-13T04:05:25.992 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:25.992 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:25.996 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:05:25.996 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:52.146 INFO:tasks.workunit.client.0.smithi110.stderr:Note: checking out 'a8ee8ac95ca822967c9dffe2a705f1d5502812bd'. 2019-10-13T04:05:52.147 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:52.147 INFO:tasks.workunit.client.0.smithi110.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-10-13T04:05:52.147 INFO:tasks.workunit.client.0.smithi110.stderr:changes and commit them, and you can discard any commits you make in this 2019-10-13T04:05:52.147 INFO:tasks.workunit.client.0.smithi110.stderr:state without impacting any branches by performing another checkout. 2019-10-13T04:05:52.148 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:52.148 INFO:tasks.workunit.client.0.smithi110.stderr:If you want to create a new branch to retain commits you create, you may 2019-10-13T04:05:52.148 INFO:tasks.workunit.client.0.smithi110.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-10-13T04:05:52.149 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:52.149 INFO:tasks.workunit.client.0.smithi110.stderr: git checkout -b 2019-10-13T04:05:52.149 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:52.149 INFO:tasks.workunit.client.0.smithi110.stderr:HEAD is now at a8ee8ac95c Merge PR #29626 into wip-pdonnell-testing-20191012.025926 2019-10-13T04:05:52.151 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:52.151 INFO:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-10-13T04:05:52.160 INFO:tasks.workunit.client.0.smithi110.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-10-13T04:05:52.164 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T04:05:52.164 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-10-13T04:05:52.851 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-10-13T04:05:52.967 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-10-13T04:05:53.025 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-10-13T04:05:53.027 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T04:05:53.027 INFO:tasks.workunit.client.0.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-10-13T04:05:53.079 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-10-13T04:05:53.158 DEBUG:teuthology.orchestra.remote:smithi110:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-10-13T04:05:53.171 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2019-10-13T04:05:53.171 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2019-10-13T04:05:53.172 INFO:teuthology.orchestra.run.smithi110:Running (workunit test suites/ffsb.sh): 2019-10-13T04:05:53.172 INFO:teuthology.orchestra.run.smithi110:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2019-10-13T04:05:53.226 INFO:tasks.workunit.client.0.smithi110.stderr:--2019-10-13 04:05:53-- http://download.ceph.com/qa/ffsb.tar.bz2 2019-10-13T04:05:53.228 INFO:tasks.workunit.client.0.smithi110.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2019-10-13T04:05:53.248 INFO:tasks.workunit.client.0.smithi110.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2019-10-13T04:05:53.268 INFO:tasks.workunit.client.0.smithi110.stderr:HTTP request sent, awaiting response... 200 OK 2019-10-13T04:05:53.269 INFO:tasks.workunit.client.0.smithi110.stderr:Length: 128931 (126K) [application/octet-stream] 2019-10-13T04:05:53.272 INFO:tasks.workunit.client.0.smithi110.stderr:Saving to: 'ffsb.tar.bz2' 2019-10-13T04:05:53.278 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:53.309 INFO:tasks.workunit.client.0.smithi110.stderr: 0K .......... .......... .......... .......... .......... 39% 1.34M 0s 2019-10-13T04:05:53.329 INFO:tasks.workunit.client.0.smithi110.stderr: 50K .......... .......... .......... .......... .......... 79% 2.50M 0s 2019-10-13T04:05:53.331 INFO:tasks.workunit.client.0.smithi110.stderr: 100K .......... .......... ..... 100% 10.4M=0.06s 2019-10-13T04:05:53.332 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:53.332 INFO:tasks.workunit.client.0.smithi110.stderr:2019-10-13 04:05:53 (2.11 MB/s) - 'ffsb.tar.bz2' saved [128931/128931] 2019-10-13T04:05:53.332 INFO:tasks.workunit.client.0.smithi110.stderr: 2019-10-13T04:05:53.369 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ 2019-10-13T04:05:53.373 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_fs.c 2019-10-13T04:05:53.377 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/INSTALL 2019-10-13T04:05:53.382 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/fileops.c 2019-10-13T04:05:53.387 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/missing 2019-10-13T04:05:53.392 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/config.h.in 2019-10-13T04:05:53.395 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/config.guess 2019-10-13T04:05:53.401 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/fh.h 2019-10-13T04:05:53.405 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/COPYING 2019-10-13T04:05:53.410 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/util.h 2019-10-13T04:05:53.414 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/main.c 2019-10-13T04:05:53.417 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ChangeLog 2019-10-13T04:05:53.421 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rand.c 2019-10-13T04:05:53.424 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_fs.h 2019-10-13T04:05:53.428 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/install-sh 2019-10-13T04:05:53.434 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rwlock.h 2019-10-13T04:05:53.436 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/parser.h 2019-10-13T04:05:53.439 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/filelist.h 2019-10-13T04:05:53.443 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/LICENSE 2019-10-13T04:05:53.447 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/Makefile.in 2019-10-13T04:05:53.451 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/filelist.c 2019-10-13T04:05:53.454 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/configure 2019-10-13T04:05:53.465 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/README 2019-10-13T04:05:53.470 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/cirlist.c 2019-10-13T04:05:53.474 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/metaops.c 2019-10-13T04:05:53.477 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/registration-description 2019-10-13T04:05:53.481 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/list.c 2019-10-13T04:05:53.485 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/list.h 2019-10-13T04:05:53.489 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rand.h 2019-10-13T04:05:53.495 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/parser.c 2019-10-13T04:05:53.505 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_tg.c 2019-10-13T04:05:53.505 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/cirlist.h 2019-10-13T04:05:53.506 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/AUTHORS 2019-10-13T04:05:53.510 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/aclocal.m4 2019-10-13T04:05:53.514 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/configure.in 2019-10-13T04:05:53.517 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/examples/ 2019-10-13T04:05:53.519 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/examples/profile_everything 2019-10-13T04:05:53.522 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/depcomp 2019-10-13T04:05:53.526 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_stats.c 2019-10-13T04:05:53.531 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/util.c 2019-10-13T04:05:53.535 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/Makefile.am 2019-10-13T04:05:53.538 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_thread.c 2019-10-13T04:05:53.542 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rbt.c 2019-10-13T04:05:53.546 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rbt.h 2019-10-13T04:05:53.550 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_tg.h 2019-10-13T04:05:53.554 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/stamp-h.in 2019-10-13T04:05:53.558 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/config.sub 2019-10-13T04:05:53.562 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_thread.h 2019-10-13T04:05:53.566 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/fileops.h 2019-10-13T04:05:53.569 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_op.h 2019-10-13T04:05:53.573 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/public-description 2019-10-13T04:05:53.576 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/mkinstalldirs 2019-10-13T04:05:53.580 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb.h 2019-10-13T04:05:53.583 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/metaops.h 2019-10-13T04:05:53.586 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_op.c 2019-10-13T04:05:53.590 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/fh.c 2019-10-13T04:05:53.593 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/NEWS 2019-10-13T04:05:53.597 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/rwlock.c 2019-10-13T04:05:53.600 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_stats.h 2019-10-13T04:05:53.603 INFO:tasks.workunit.client.0.smithi110.stdout:ffsb-6.0-rc2/ffsb_fc.c 2019-10-13T04:05:53.628 INFO:tasks.workunit.client.0.smithi110.stdout:patching file parser.c 2019-10-13T04:05:54.023 INFO:tasks.workunit.client.0.smithi110.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2019-10-13T04:05:55.046 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether build environment is sane... yes 2019-10-13T04:05:55.075 INFO:tasks.workunit.client.0.smithi110.stdout:checking for a thread-safe mkdir -p... /bin/mkdir -p 2019-10-13T04:05:55.077 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gawk... gawk 2019-10-13T04:05:55.121 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether make sets $(MAKE)... yes 2019-10-13T04:05:55.225 INFO:tasks.workunit.client.0.smithi110.stdout:checking build system type... x86_64-unknown-linux-gnu 2019-10-13T04:05:55.226 INFO:tasks.workunit.client.0.smithi110.stdout:checking host system type... x86_64-unknown-linux-gnu 2019-10-13T04:05:55.271 INFO:tasks.workunit.client.0.smithi110.stdout:checking for style of include used by make... GNU 2019-10-13T04:05:55.281 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gcc... gcc 2019-10-13T04:05:55.436 INFO:tasks.workunit.client.0.smithi110.stdout:checking for C compiler default output file name... a.out 2019-10-13T04:05:55.443 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether the C compiler works... yes 2019-10-13T04:05:55.454 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether we are cross compiling... no 2019-10-13T04:05:55.569 INFO:tasks.workunit.client.0.smithi110.stdout:checking for suffix of executables... 2019-10-13T04:05:55.652 INFO:tasks.workunit.client.0.smithi110.stdout:checking for suffix of object files... o 2019-10-13T04:05:55.735 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether we are using the GNU C compiler... yes 2019-10-13T04:05:55.806 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether gcc accepts -g... yes 2019-10-13T04:05:56.032 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:05:56.033 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:56.033 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gcc option to accept ISO C89... none needed 2019-10-13T04:05:56.036 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:05:56.036 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:05:56.345 INFO:tasks.workunit.client.0.smithi110.stdout:checking dependency style of gcc... gcc3 2019-10-13T04:05:56.454 INFO:tasks.workunit.client.0.smithi110.stdout:checking how to run the C preprocessor... gcc -E 2019-10-13T04:05:56.546 INFO:tasks.workunit.client.0.smithi110.stdout:checking for grep that handles long lines and -e... /bin/grep 2019-10-13T04:05:56.549 INFO:tasks.workunit.client.0.smithi110.stdout:checking for egrep... /bin/grep -E 2019-10-13T04:05:56.569 INFO:tasks.workunit.client.0.smithi110.stdout:checking for AIX... no 2019-10-13T04:05:56.607 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gcc... (cached) gcc 2019-10-13T04:05:56.635 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether we are using the GNU C compiler... (cached) yes 2019-10-13T04:05:56.636 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether gcc accepts -g... (cached) yes 2019-10-13T04:05:56.637 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2019-10-13T04:05:56.639 INFO:tasks.workunit.client.0.smithi110.stdout:checking dependency style of gcc... (cached) gcc3 2019-10-13T04:05:56.790 INFO:tasks.workunit.client.0.smithi110.stdout:checking for main in -lm... yes 2019-10-13T04:05:56.941 INFO:tasks.workunit.client.0.smithi110.stdout:checking for main in -lpthread... yes 2019-10-13T04:05:57.261 INFO:tasks.workunit.client.0.smithi110.stdout:checking for ANSI C header files... yes 2019-10-13T04:05:57.390 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2019-10-13T04:05:57.501 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/types.h... yes 2019-10-13T04:05:57.622 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/stat.h... yes 2019-10-13T04:05:57.737 INFO:tasks.workunit.client.0.smithi110.stdout:checking for stdlib.h... yes 2019-10-13T04:05:57.849 INFO:tasks.workunit.client.0.smithi110.stdout:checking for string.h... yes 2019-10-13T04:05:57.967 INFO:tasks.workunit.client.0.smithi110.stdout:checking for memory.h... yes 2019-10-13T04:05:58.088 INFO:tasks.workunit.client.0.smithi110.stdout:checking for strings.h... yes 2019-10-13T04:05:58.222 INFO:tasks.workunit.client.0.smithi110.stdout:checking for inttypes.h... yes 2019-10-13T04:05:58.364 INFO:tasks.workunit.client.0.smithi110.stdout:checking for stdint.h... yes 2019-10-13T04:05:58.494 INFO:tasks.workunit.client.0.smithi110.stdout:checking for unistd.h... yes 2019-10-13T04:05:58.627 INFO:tasks.workunit.client.0.smithi110.stdout:checking pthread.h usability... yes 2019-10-13T04:05:58.672 INFO:tasks.workunit.client.0.smithi110.stdout:checking pthread.h presence... yes 2019-10-13T04:05:58.674 INFO:tasks.workunit.client.0.smithi110.stdout:checking for pthread.h... yes 2019-10-13T04:05:58.795 INFO:tasks.workunit.client.0.smithi110.stdout:checking fcntl.h usability... yes 2019-10-13T04:05:58.896 INFO:tasks.workunit.client.0.smithi110.stdout:checking fcntl.h presence... yes 2019-10-13T04:05:58.898 INFO:tasks.workunit.client.0.smithi110.stdout:checking for fcntl.h... yes 2019-10-13T04:05:59.019 INFO:tasks.workunit.client.0.smithi110.stdout:checking limits.h usability... yes 2019-10-13T04:05:59.067 INFO:tasks.workunit.client.0.smithi110.stdout:checking limits.h presence... yes 2019-10-13T04:05:59.069 INFO:tasks.workunit.client.0.smithi110.stdout:checking for limits.h... yes 2019-10-13T04:05:59.104 INFO:tasks.workunit.client.0.smithi110.stdout:checking for stdint.h... (cached) yes 2019-10-13T04:05:59.199 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/time.h usability... yes 2019-10-13T04:05:59.252 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/time.h presence... yes 2019-10-13T04:05:59.253 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/time.h... yes 2019-10-13T04:05:59.291 INFO:tasks.workunit.client.0.smithi110.stdout:checking for unistd.h... (cached) yes 2019-10-13T04:05:59.391 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/vfs.h usability... yes 2019-10-13T04:05:59.437 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/vfs.h presence... yes 2019-10-13T04:05:59.438 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/vfs.h... yes 2019-10-13T04:05:59.580 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/limits.h usability... no 2019-10-13T04:05:59.624 INFO:tasks.workunit.client.0.smithi110.stdout:checking sys/limits.h presence... no 2019-10-13T04:05:59.625 INFO:tasks.workunit.client.0.smithi110.stdout:checking for sys/limits.h... no 2019-10-13T04:05:59.695 INFO:tasks.workunit.client.0.smithi110.stdout:checking for an ANSI C-conforming const... yes 2019-10-13T04:05:59.768 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether time.h and sys/time.h may both be included... yes 2019-10-13T04:05:59.803 INFO:tasks.workunit.client.0.smithi110.stdout:checking for function prototypes... yes 2019-10-13T04:05:59.877 INFO:tasks.workunit.client.0.smithi110.stdout:checking whether setvbuf arguments are reversed... no 2019-10-13T04:06:00.041 INFO:tasks.workunit.client.0.smithi110.stdout:checking for system... yes 2019-10-13T04:06:00.211 INFO:tasks.workunit.client.0.smithi110.stdout:checking for gettimeofday... yes 2019-10-13T04:06:00.378 INFO:tasks.workunit.client.0.smithi110.stdout:checking for mkdir... yes 2019-10-13T04:06:00.551 INFO:tasks.workunit.client.0.smithi110.stdout:checking for strerror... yes 2019-10-13T04:06:00.714 INFO:tasks.workunit.client.0.smithi110.stdout:checking for open64... yes 2019-10-13T04:06:00.884 INFO:tasks.workunit.client.0.smithi110.stdout:checking for stat64... yes 2019-10-13T04:06:01.052 INFO:tasks.workunit.client.0.smithi110.stdout:checking for fseeko64... yes 2019-10-13T04:06:01.220 INFO:tasks.workunit.client.0.smithi110.stdout:checking for lrand48_r... yes 2019-10-13T04:06:01.378 INFO:tasks.workunit.client.0.smithi110.stdout:checking for srand48_r... yes 2019-10-13T04:06:01.445 INFO:tasks.workunit.client.0.smithi110.stdout:configure: creating ./config.status 2019-10-13T04:06:01.715 INFO:tasks.workunit.client.0.smithi110.stdout:config.status: creating Makefile 2019-10-13T04:06:01.752 INFO:tasks.workunit.client.0.smithi110.stdout:config.status: creating config.h 2019-10-13T04:06:01.809 INFO:tasks.workunit.client.0.smithi110.stdout:config.status: executing depfiles commands 2019-10-13T04:06:01.976 INFO:tasks.workunit.client.0.smithi110.stdout:config.status: executing default commands 2019-10-13T04:06:02.127 INFO:tasks.workunit.client.0.smithi110.stdout:make all-am 2019-10-13T04:06:02.177 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T04:06:02.177 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2019-10-13T04:06:02.288 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c: In function 'ffsb_readfile': 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:138:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr: printf("Error: read size %llu bytes too big " 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:138:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 5 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr: printf("Error: read size %llu bytes too big " 2019-10-13T04:06:02.289 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:140:29: note: format string is defined here 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr: " for file of size %llu bytes\n" 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:144:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr: printf("minimum file size must be at least " 2019-10-13T04:06:02.290 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:02.291 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:145:12: note: format string is defined here 2019-10-13T04:06:02.291 INFO:tasks.workunit.client.0.smithi110.stderr: " %llu bytes\n", minfilesize); 2019-10-13T04:06:02.291 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:02.291 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:02.291 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c: In function 'ffsb_stat': 2019-10-13T04:06:02.292 INFO:tasks.workunit.client.0.smithi110.stderr:fileops.c:542:2: warning: implicit declaration of function 'fhstat'; did you mean 'fstat'? [-Wimplicit-function-declaration] 2019-10-13T04:06:02.292 INFO:tasks.workunit.client.0.smithi110.stderr: fhstat(curfile->name, ft, fs); 2019-10-13T04:06:02.292 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~ 2019-10-13T04:06:02.292 INFO:tasks.workunit.client.0.smithi110.stderr: fstat 2019-10-13T04:06:02.424 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2019-10-13T04:06:02.458 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2019-10-13T04:06:02.753 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2019-10-13T04:06:02.779 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2019-10-13T04:06:03.126 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/main.Tpo .deps/main.Po 2019-10-13T04:06:03.141 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2019-10-13T04:06:03.353 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c: In function 'fhwrite': 2019-10-13T04:06:03.353 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c:180:18: warning: format '%d' expects argument of type 'int', but argument 2 has type 'ssize_t {aka long int}' [-Wformat=] 2019-10-13T04:06:03.353 INFO:tasks.workunit.client.0.smithi110.stderr: printf("Wrote %d instead of %d bytes.\n" 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr: ~^ 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr: %ld 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c: In function 'fhseek': 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c:212:41: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr: fprintf(stderr, "tried to seek to %lld\n", offset); 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:03.354 INFO:tasks.workunit.client.0.smithi110.stderr: %ld 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c:214:20: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr: fprintf(stderr, "tried to seek from current " 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr:fh.c:215:21: note: format string is defined here 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr: "position to %lld\n", offset); 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:03.355 INFO:tasks.workunit.client.0.smithi110.stderr: %ld 2019-10-13T04:06:03.456 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2019-10-13T04:06:03.486 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2019-10-13T04:06:03.800 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2019-10-13T04:06:03.813 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2019-10-13T04:06:04.074 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2019-10-13T04:06:04.086 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2019-10-13T04:06:04.268 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2019-10-13T04:06:04.300 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2019-10-13T04:06:04.517 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2019-10-13T04:06:04.531 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2019-10-13T04:06:04.854 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2019-10-13T04:06:04.880 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2019-10-13T04:06:05.079 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_tg.c: In function 'tg_print_config_helper': 2019-10-13T04:06:05.080 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_tg.c:274:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.080 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t read_size = %llu\t(%s)\n", tg->read_size, 2019-10-13T04:06:05.080 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ ~~~~~~~~~~~~~ 2019-10-13T04:06:05.080 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:05.080 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_tg.c:283:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.081 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t write_size = %llu\t(%s)\n", tg->write_size, 2019-10-13T04:06:05.081 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ ~~~~~~~~~~~~~~ 2019-10-13T04:06:05.081 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:05.205 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2019-10-13T04:06:05.222 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2019-10-13T04:06:05.449 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c: In function 'verify_file': 2019-10-13T04:06:05.449 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:217:19: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.449 INFO:tasks.workunit.client.0.smithi110.stderr: printf("size %llu bytes for file %s is invalid\n", 2019-10-13T04:06:05.449 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:05.449 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:05.451 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c: In function 'fs_print_config': 2019-10-13T04:06:05.451 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:595:22: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.451 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2019-10-13T04:06:05.451 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~~~^ 2019-10-13T04:06:05.451 INFO:tasks.workunit.client.0.smithi110.stderr: %12lu 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: fs->size_weights[i].size, 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~~~~~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:595:43: warning: unknown conversion type character ')' in format [-Wformat=] 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:603:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t min file size = %llu\t(%s)\n", fs->minfilesize, 2019-10-13T04:06:05.452 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2019-10-13T04:06:05.453 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:05.453 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:605:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:05.453 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t max file size = %llu\t(%s)\n", fs->maxfilesize, 2019-10-13T04:06:05.453 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2019-10-13T04:06:05.453 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:616:41: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t current utilization = %.2f\%\n", getfsutil(fs->basedir)*100); 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr: ^~ 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:618:42: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr: printf("\t desired utilization = %.2lf%\n", fs->desired_fsutil * 100); 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr: ^~ 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c: In function 'fs_needs_stats': 2019-10-13T04:06:05.454 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_fs.c:627:24: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:05.455 INFO:tasks.workunit.client.0.smithi110.stderr: return (fs != NULL) ? (int)fs->fsd.config : 0; 2019-10-13T04:06:05.455 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:05.602 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2019-10-13T04:06:05.617 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2019-10-13T04:06:05.846 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2019-10-13T04:06:05.860 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2019-10-13T04:06:06.057 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c: In function 'print_op_results': 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c:83:23: warning: embedded '\0' in format [-Wformat-contains-nul] 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr: sprintf(buf, "%s/sec\0", buf); 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr: ^~ 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c:86:19: warning: embedded '\0' in format [-Wformat-contains-nul] 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr: sprintf(buf, "NA\0"); 2019-10-13T04:06:06.058 INFO:tasks.workunit.client.0.smithi110.stderr: ^~ 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c: In function 'print_results': 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c:112:61: warning: unknown conversion type character 'T' in format [-Wformat=] 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr:ffsb_op.c:112:74: warning: unknown conversion type character 'O' in format [-Wformat=] 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2019-10-13T04:06:06.059 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.130 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2019-10-13T04:06:06.144 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2019-10-13T04:06:06.401 INFO:tasks.workunit.client.0.smithi110.stderr:util.c: In function 'ffsb_bench_gettimeofday': 2019-10-13T04:06:06.401 INFO:tasks.workunit.client.0.smithi110.stderr:util.c:333:46: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:06.401 INFO:tasks.workunit.client.0.smithi110.stderr: printf("average time for gettimeofday(): %llu nsec\n", average); 2019-10-13T04:06:06.401 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr:util.c: In function 'ffsb_bench_getpid': 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr:util.c:350:40: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr: printf("average time for getpid(): %llu nsec\n", average); 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr: ~~~^ 2019-10-13T04:06:06.402 INFO:tasks.workunit.client.0.smithi110.stderr: %lu 2019-10-13T04:06:06.508 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/util.Tpo .deps/util.Po 2019-10-13T04:06:06.523 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2019-10-13T04:06:06.716 INFO:tasks.workunit.client.0.smithi110.stderr:In file included from parser.h:22:0, 2019-10-13T04:06:06.716 INFO:tasks.workunit.client.0.smithi110.stderr: from parser.c:25: 2019-10-13T04:06:06.716 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c: In function 'init_filesys': 2019-10-13T04:06:06.718 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.718 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.718 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.718 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.718 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T04:06:06.719 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.720 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.721 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.721 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.721 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.721 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.721 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.722 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.723 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T04:06:06.724 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.725 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.726 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2019-10-13T04:06:06.729 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.729 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c: In function 'init_tg_stats': 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.730 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.731 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.731 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.732 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.732 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.733 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.734 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.735 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.736 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, 2019-10-13T04:06:06.737 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.738 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2019-10-13T04:06:06.741 INFO:tasks.workunit.client.0.smithi110.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2019-10-13T04:06:06.741 INFO:tasks.workunit.client.0.smithi110.stderr: ^ 2019-10-13T04:06:06.741 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2019-10-13T04:06:06.741 INFO:tasks.workunit.client.0.smithi110.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2019-10-13T04:06:06.741 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~ 2019-10-13T04:06:06.742 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2019-10-13T04:06:06.742 INFO:tasks.workunit.client.0.smithi110.stderr: container_of(ptr, type, member) 2019-10-13T04:06:06.742 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~ 2019-10-13T04:06:06.742 INFO:tasks.workunit.client.0.smithi110.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2019-10-13T04:06:06.742 INFO:tasks.workunit.client.0.smithi110.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2019-10-13T04:06:06.743 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~ 2019-10-13T04:06:06.743 INFO:tasks.workunit.client.0.smithi110.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2019-10-13T04:06:06.743 INFO:tasks.workunit.client.0.smithi110.stderr: list_for_each_entry(tmp_list, 2019-10-13T04:06:06.744 INFO:tasks.workunit.client.0.smithi110.stderr: ^~~~~~~~~~~~~~~~~~~ 2019-10-13T04:06:06.991 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2019-10-13T04:06:07.013 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2019-10-13T04:06:07.263 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2019-10-13T04:06:07.279 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2019-10-13T04:06:07.547 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2019-10-13T04:06:07.562 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2019-10-13T04:06:07.693 INFO:tasks.workunit.client.0.smithi110.stdout:mv -f .deps/list.Tpo .deps/list.Po 2019-10-13T04:06:07.718 INFO:tasks.workunit.client.0.smithi110.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2019-10-13T04:06:07.979 INFO:tasks.workunit.client.0.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2019-10-13T04:06:08.009 INFO:tasks.workunit.client.0.smithi110.stdout:FFSB version 6.0-RC2 started 2019-10-13T04:06:08.009 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.014 INFO:tasks.workunit.client.0.smithi110.stdout:benchmark time = 300 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout:ThreadGroup 0 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout:================ 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout: num_threads = 32 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout: read_random = off 2019-10-13T04:06:08.015 INFO:tasks.workunit.client.0.smithi110.stdout: read_size = 0 (0B) 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: read_blocksize = 0 (0B) 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: read_skip = off 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: read_skipsize = 0 (0B) 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: write_random = on 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: write_size = 5242880 (5MB) 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: fsync_file = 0 2019-10-13T04:06:08.016 INFO:tasks.workunit.client.0.smithi110.stdout: write_blocksize = 4096 (4KB) 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: wait time = 0 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: op weights 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: read = 0 (0.00%) 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: readall = 0 (0.00%) 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: write = 1 (100.00%) 2019-10-13T04:06:08.017 INFO:tasks.workunit.client.0.smithi110.stdout: create = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: append = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: delete = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: metaop = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: createdir = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: stat = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: writeall = 0 (0.00%) 2019-10-13T04:06:08.018 INFO:tasks.workunit.client.0.smithi110.stdout: writeall_fsync = 0 (0.00%) 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: open_close = 0 (0.00%) 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: write_fsync = 0 (0.00%) 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: create_fsync = 0 (0.00%) 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: append_fsync = 0 (0.00%) 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout:FileSystem . 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout:========== 2019-10-13T04:06:08.019 INFO:tasks.workunit.client.0.smithi110.stdout: num_dirs = 0 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: starting files = 128 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: min file size = 104857600 (100MB) 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: max file size = 104857600 (100MB) 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: directio = off 2019-10-13T04:06:08.020 INFO:tasks.workunit.client.0.smithi110.stdout: alignedio = on 2019-10-13T04:06:08.021 INFO:tasks.workunit.client.0.smithi110.stdout: bufferedio = off 2019-10-13T04:06:08.021 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.021 INFO:tasks.workunit.client.0.smithi110.stdout: aging is off 2019-10-13T04:06:08.022 INFO:tasks.workunit.client.0.smithi110.stdout: current utilization = 0.00% 2019-10-13T04:06:08.022 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:06:08.022 INFO:tasks.workunit.client.0.smithi110.stdout:checking existing fs: . 2019-10-13T04:06:08.022 INFO:tasks.workunit.client.0.smithi110.stdout:recreating new fileset 2019-10-13T04:06:08.022 INFO:tasks.workunit.client.0.smithi110.stderr:opendir: No such file or directory 2019-10-13T04:06:26.071 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:06:26.071 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:26.074 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:06:26.074 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:56.267 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:06:56.268 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:06:56.271 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:06:56.271 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:26.382 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:07:26.382 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:26.386 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:07:26.386 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:56.496 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:07:56.497 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:07:56.501 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:07:56.501 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:26.623 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:08:26.623 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:26.626 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:08:26.626 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:56.672 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:08:56.672 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:08:56.676 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:08:56.676 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:26.761 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:09:26.761 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:26.765 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:09:26.765 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:57.059 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:09:57.059 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:09:57.064 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:09:57.064 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:29.015 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:10:29.015 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:29.019 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:10:29.019 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:59.137 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:10:59.137 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:10:59.140 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:10:59.140 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:29.185 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:11:29.185 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:29.190 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:11:29.190 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:59.307 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:11:59.307 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:11:59.310 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:11:59.311 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:29.347 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:12:29.348 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:29.360 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:12:29.360 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:59.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:12:59.493 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:12:59.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:12:59.497 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:29.627 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:13:29.627 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:29.630 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:13:29.631 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:59.784 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:13:59.784 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:13:59.788 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:13:59.788 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:29.877 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:14:29.877 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:29.880 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:14:29.881 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:59.974 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:14:59.974 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:14:59.977 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:14:59.978 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:30.185 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:15:30.185 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:15:30.189 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:15:30.189 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:00.221 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:16:00.221 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:00.225 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:16:00.225 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:31.665 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:16:31.665 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:16:31.669 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:16:31.669 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:03.454 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:17:03.454 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:03.458 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:17:03.458 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:23.340 INFO:tasks.workunit.client.0.smithi110.stdout:fs setup took 675 secs 2019-10-13T04:17:25.292 INFO:tasks.workunit.client.0.smithi110.stdout:Syncing()...1 sec 2019-10-13T04:17:25.293 INFO:tasks.workunit.client.0.smithi110.stdout:Starting Actual Benchmark At: Sun Oct 13 04:17:25 2019 2019-10-13T04:17:25.293 INFO:tasks.workunit.client.0.smithi110.stdout: 2019-10-13T04:17:25.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:17:25.631+0000 7f2929706700 -1 osd.3 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.7s0 23/21 ECSubRead(tid=1864, to_read={3:ef7c1133:::100000001c8.00000014:head=942080,4096,0,987136,4096,0}, subchunks={3:ef7c1133:::100000001c8.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:25.847 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:25.841+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=1034, attrs_read=0)) 2019-10-13T04:17:25.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:25.953+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 5 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=1976, to_read={3:a38d5582:::1000000015a.00000014:head=733184,4096,0,745472,4096,0,1773568,4096,0,1847296,4096,0,2015232,4096,0}, subchunks={3:a38d5582:::1000000015a.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:26.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:17:26.035+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 22 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=2873, attrs_read=0)) 2019-10-13T04:17:26.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:17:26.095+0000 7f89fae01700 -1 osd.0 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 23/21 ECSubReadReply(tid=1965, attrs_read=0)) 2019-10-13T04:17:26.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:17:26.643+0000 7f2929706700 -1 osd.3 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.7s0 23/21 ECSubReadReply(tid=2066, attrs_read=0)) 2019-10-13T04:17:26.819 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:26.813+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 7 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=1252, attrs_read=0)) 2019-10-13T04:17:26.919 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:26.913+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 12 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=2219, to_read={3:b927bb15:::100000001c1.00000009:head=180224,4096,0,1744896,4096,0,1925120,8192,0}, subchunks={3:b927bb15:::100000001c1.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:27.854 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:27.849+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 6 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=1540, to_read={3:d55466b4:::100000001a5.00000000:head=1437696,4096,0,1642496,4096,0}, subchunks={3:d55466b4:::100000001a5.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:27.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:27.869+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=2405, to_read={3:baa1bdca:::10000000180.00000004:head=622592,4096,0,1064960,4096,0,1298432,4096,0,1306624,4096,0,1359872,4096,0}, subchunks={3:baa1bdca:::10000000180.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:28.891 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:28.885+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=1708, attrs_read=0)) 2019-10-13T04:17:29.893 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:29.885+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 8 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=2064, to_read={3:cccf71ee:::1000000019d.00000001:head=40960,4096,0,344064,4096,0,425984,4096,0,675840,4096,0,692224,4096,0,704512,4096,0,745472,4096,0,1097728,4096,0,1769472,4096,0,1802240,4096,0}, subchunks={3:cccf71ee:::1000000019d.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:30.864 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:30.857+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 11 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=2154, attrs_read=0)) 2019-10-13T04:17:31.845 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:31.837+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=2386, attrs_read=0)) 2019-10-13T04:17:33.575 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:17:33.575 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:33.578 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:17:33.578 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:17:33.923 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:33.913+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=2978, to_read={3:c6b6f42a:::10000000158.00000004:head=1892352,4096,0,1966080,4096,0}, subchunks={3:c6b6f42a:::10000000158.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:37.964 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:37.957+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=3699, attrs_read=0)) 2019-10-13T04:17:38.936 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:38.929+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=4840, to_read={3:ae59df06:::1000000015e.00000016:head=1093632,4096,0,1802240,4096,0,2019328,4096,0}, subchunks={3:ae59df06:::1000000015e.00000016:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:39.924 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:39.917+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=4031, attrs_read=0)) 2019-10-13T04:17:39.925 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:39.917+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 5 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=4988, attrs_read=0)) 2019-10-13T04:17:40.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:40.941+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=5328, attrs_read=0)) 2019-10-13T04:17:41.953 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:41.945+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=4358, to_read={3:d76597ca:::100000001b3.00000004:head=1437696,4096,0,1773568,4096,0,1916928,4096,0}, subchunks={3:d76597ca:::100000001b3.00000004:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:42.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:42.933+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=5629, attrs_read=0)) 2019-10-13T04:17:43.926 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:43.916+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=4659, attrs_read=0)) 2019-10-13T04:17:44.972 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:44.964+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=4859, attrs_read=0)) 2019-10-13T04:17:47.962 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:47.952+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=5344, to_read={3:c65c2e75:::10000000179.00000008:head=630784,4096,0,675840,4096,0,684032,4096,0,1306624,4096,0}, subchunks={3:c65c2e75:::10000000179.00000008:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:48.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:17:48.908+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=6450, attrs_read=0)) 2019-10-13T04:17:52.918 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:52.912+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=6119, attrs_read=0)) 2019-10-13T04:17:55.006 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:55.000+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=6341, to_read={3:d5c11ca7:::10000000187.00000003:head=1036288,4096,0,1384448,4096,0,1605632,4096,0}, subchunks={3:d5c11ca7:::10000000187.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:17:56.995 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:17:56.988+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=6714, to_read={3:d82f51b6:::10000000177.00000014:head=327680,4096,0,425984,4096,0,1777664,4096,0}, subchunks={3:d82f51b6:::10000000177.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:01.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:01.900+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=8287, to_read={3:a3d282e9:::10000000180.00000015:head=360448,4096,0}, subchunks={3:a3d282e9:::10000000180.00000015:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:02.073 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:02.064+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=7381, attrs_read=0)) 2019-10-13T04:18:03.625 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:18:03.625 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:03.628 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:18:03.628 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:04.110 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:04.104+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=7675, attrs_read=0)) 2019-10-13T04:18:08.856 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:08.848+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=9332, to_read={3:b718b40e:::100000001a5.0000000f:head=475136,4096,0,516096,4096,0,1437696,4096,0,1695744,4096,0}, subchunks={3:b718b40e:::100000001a5.0000000f:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:10.072 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:10.064+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=8474, attrs_read=0)) 2019-10-13T04:18:13.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:13.900+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=10130, to_read={3:bf733376:::10000000165.00000005:head=487424,4096,0,1236992,4096,0,1609728,4096,0}, subchunks={3:bf733376:::10000000165.00000005:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:19.823 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:19.816+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=10946, to_read={3:b04c766e:::100000001d1.00000007:head=290816,4096,0,401408,4096,0,413696,4096,0,491520,4096,0}, subchunks={3:b04c766e:::100000001d1.00000007:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:21.084 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:21.076+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=10011, to_read={3:cd2ba39f:::10000000193.0000000f:head=356352,4096,0,462848,4096,0,1867776,4096,0}, subchunks={3:cd2ba39f:::10000000193.0000000f:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:21.914 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:21.908+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=11271, attrs_read=0)) 2019-10-13T04:18:24.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:24.900+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=11746, to_read={3:b70118f3:::1000000015f.00000003:head=1110016,4096,0,1667072,4096,0}, subchunks={3:b70118f3:::1000000015f.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:25.898 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:25.892+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=11908, to_read={3:b84e77bb:::1000000016b.00000010:head=303104,4096,0,503808,4096,0,684032,4096,0}, subchunks={3:b84e77bb:::1000000016b.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:27.212 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:27.204+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=10775, to_read={3:ccfc0630:::1000000018f.0000000a:head=299008,4096,0,327680,4096,0,815104,4096,0}, subchunks={3:ccfc0630:::1000000018f.0000000a:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:28.242 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:28.236+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=10881, to_read={3:d86863c0:::10000000183.00000009:head=188416,4096,0,225280,4096,0}, subchunks={3:d86863c0:::10000000183.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:30.005 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:29.996+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=12465, attrs_read=0)) 2019-10-13T04:18:32.015 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:32.004+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=12763, attrs_read=0)) 2019-10-13T04:18:34.786 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:34.780+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=11591, attrs_read=0)) 2019-10-13T04:18:37.653 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:18:37.654 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:37.657 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:18:37.657 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:18:39.063 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:39.056+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=13738, to_read={3:ac363a09:::100000001d6.00000011:head=978944,4096,0,987136,4096,0}, subchunks={3:ac363a09:::100000001d6.00000011:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:39.709 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:39.700+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=12192, to_read={3:c2bb8eef:::100000001c5.00000012:head=454656,4096,0,1605632,4096,0,1757184,4096,0}, subchunks={3:c2bb8eef:::100000001c5.00000012:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:41.025 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:18:41.016+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=14041, attrs_read=0)) 2019-10-13T04:18:41.713 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:41.704+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=12418, to_read={3:db23a558:::100000001b3.00000003:head=1445888,4096,0,1822720,4096,0,1908736,4096,0}, subchunks={3:db23a558:::100000001b3.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:18:47.923 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:47.920+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=13046, attrs_read=0)) 2019-10-13T04:18:50.866 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:18:50.859+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=13315, attrs_read=0)) 2019-10-13T04:19:02.002 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:01.999+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=16824, to_read={3:ab5bd07c:::10000000192.00000011:head=282624,4096,0,344064,4096,0,761856,4096,0}, subchunks={3:ab5bd07c:::10000000192.00000011:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:06.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:06.027+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=17330, attrs_read=0)) 2019-10-13T04:19:09.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:09.071+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=17719, to_read={3:bf733376:::10000000165.00000005:head=970752,4096,0,1560576,4096,0,1884160,4096,0}, subchunks={3:bf733376:::10000000165.00000005:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:10.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:10.027+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=17856, to_read={3:bfbb0d0b:::10000000165.00000003:head=258048,4096,0,270336,4096,0,1257472,4096,0,1478656,4096,0}, subchunks={3:bfbb0d0b:::10000000165.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:10.880 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:10.875+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=15529, attrs_read=0)) 2019-10-13T04:19:11.048 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:19:11.049 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:11.052 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:19:11.052 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:20.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:20.119+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=19024, to_read={3:a2633a40:::100000001c0.00000003:head=770048,8192,0,856064,4096,0,1875968,4096,0}, subchunks={3:a2633a40:::100000001c0.00000003:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:27.990 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:27.983+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=17550, to_read={3:d52001ba:::1000000019e.0000000d:head=151552,4096,0,2056192,4096,0}, subchunks={3:d52001ba:::1000000019e.0000000d:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:34.881 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:34.875+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=18425, to_read={3:c289d928:::100000001c2.0000000a:head=69632,4096,0,135168,4096,0,524288,4096,0}, subchunks={3:c289d928:::100000001c2.0000000a:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:38.957 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:38.951+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=18978, attrs_read=0)) 2019-10-13T04:19:42.198 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:42.191+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=21498, attrs_read=0)) 2019-10-13T04:19:43.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:43.545+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=35693, to_read={3:30435717:::10000000184.0000000b:head=872448,4096,0,1732608,4096,0,1900544,4096,0}, subchunks={3:30435717:::10000000184.0000000b:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:44.275 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:44.271+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 5 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=21740, to_read={3:bb4c8e42:::1000000017d.00000010:head=397312,4096,0,950272,4096,0,1814528,4096,0}, subchunks={3:bb4c8e42:::1000000017d.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:44.399 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:19:44.399 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:44.402 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:19:44.402 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:19:44.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:44.529+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 6 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=36058, to_read={3:2d1b80ca:::100000001bd.00000007:head=20480,4096,0,28672,4096,0,311296,4096,0,344064,4096,0,647168,4096,0}, subchunks={3:2d1b80ca:::100000001bd.00000007:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:45.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:45.553+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 9 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=36330, to_read={3:228ded39:::100000001aa.00000007:head=1163264,4096,0,1351680,4096,0,1445888,4096,0}, subchunks={3:228ded39:::100000001aa.00000007:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:46.521 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:46.517+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 11 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=36751, to_read={3:3f3bd72a:::10000000160.00000002:head=528384,4096,0,544768,4096,0,663552,4096,0,684032,4096,0}, subchunks={3:3f3bd72a:::10000000160.00000002:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:46.857 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:46.851+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=19814, to_read={3:cad1ad50:::10000000192.00000016:head=12288,4096,0,323584,4096,0,618496,4096,0,1138688,4096,0}, subchunks={3:cad1ad50:::10000000192.00000016:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:47.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:47.473+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 17 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=36979, attrs_read=0)) 2019-10-13T04:19:51.854 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:19:51.847+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=20342, to_read={3:caba0c50:::100000001aa.00000000:head=1511424,4096,0,1626112,4096,0,1744896,4096,0}, subchunks={3:caba0c50:::100000001aa.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:52.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:52.609+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=38253, attrs_read=0)) 2019-10-13T04:19:53.663 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:53.657+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=38472, attrs_read=0)) 2019-10-13T04:19:55.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:55.705+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 7 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=38786, attrs_read=0)) 2019-10-13T04:19:56.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:19:56.226+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=23175, attrs_read=0)) 2019-10-13T04:19:56.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:56.665+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=39114, attrs_read=0)) 2019-10-13T04:19:57.704 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:57.701+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=39343, to_read={3:127d974b:::100000001d6.0000000a:head=520192,4096,0,1769472,4096,0,1810432,4096,0,2084864,4096,0}, subchunks={3:127d974b:::100000001d6.0000000a:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:19:58.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:19:58.673+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=39548, to_read={3:2b82cc20:::100000001cb.0000000c:head=303104,4096,0,712704,4096,0,1105920,4096,0,1597440,4096,0,1814528,4096,0}, subchunks={3:2b82cc20:::100000001cb.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:04.883 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:04.877+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=40996, attrs_read=0)) 2019-10-13T04:20:05.072 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:05.066+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=21563, attrs_read=0)) 2019-10-13T04:20:07.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:07.856+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=41678, attrs_read=0)) 2019-10-13T04:20:08.880 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:08.876+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=41937, to_read={3:30435717:::10000000184.0000000b:head=253952,4096,0,1146880,4096,0,1187840,4096,0,1454080,4096,0}, subchunks={3:30435717:::10000000184.0000000b:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:17.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:17.020+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=43843, to_read={3:1105820a:::100000001a6.0000000f:head=638976,4096,0,856064,4096,0,1179648,4096,0,1409024,4096,0,1527808,4096,0}, subchunks={3:1105820a:::100000001a6.0000000f:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:17.631 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:20:17.631 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:17.634 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:20:17.634 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:18.089 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:18.082+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=22895, attrs_read=0)) 2019-10-13T04:20:19.280 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:19.274+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=25750, to_read={3:a62acb92:::10000000181.00000012:head=692224,4096,0,913408,4096,0,1396736,4096,0,2039808,4096,0}, subchunks={3:a62acb92:::10000000181.00000012:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:21.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:21.024+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=44845, to_read={3:12d75f6c:::100000001c9.00000000:head=483328,4096,0,798720,4096,0,851968,4096,0,901120,4096,0}, subchunks={3:12d75f6c:::100000001c9.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:23.033 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:23.026+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=23473, attrs_read=0)) 2019-10-13T04:20:25.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:25.004+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 3 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=45885, to_read={3:2c259eaf:::1000000017e.00000007:head=1736704,4096,0,1851392,4096,0}, subchunks={3:2c259eaf:::1000000017e.00000007:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:26.260 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:26.254+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=26497, attrs_read=0)) 2019-10-13T04:20:28.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:28.964+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=46832, attrs_read=0)) 2019-10-13T04:20:29.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:29.952+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=47062, attrs_read=0)) 2019-10-13T04:20:31.248 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:31.242+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=27021, to_read={3:bda26aa4:::10000000173.00000005:head=335872,4096,0,724992,4096,0,995328,4096,0,1179648,4096,0}, subchunks={3:bda26aa4:::10000000173.00000005:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:31.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:31.920+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=47592, attrs_read=0)) 2019-10-13T04:20:33.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:33.920+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=47863, attrs_read=0)) 2019-10-13T04:20:36.156 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:36.150+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=24918, to_read={3:c1b175eb:::100000001c6.0000000e:head=479232,4096,0,663552,4096,0,921600,4096,0,1355776,4096,0,1724416,4096,0}, subchunks={3:c1b175eb:::100000001c6.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:37.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:20:37.440+0000 7fe528211700 -1 log_channel(cluster) log [ERR] : Health check failed: mon b is very low on available space (MON_DISK_CRIT) 2019-10-13T04:20:38.186 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:38.178+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=25116, to_read={3:df9dd252:::10000000162.0000000d:head=319488,4096,0,1396736,4096,0,1925120,4096,0}, subchunks={3:df9dd252:::10000000162.0000000d:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:42.203 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:42.198+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=25537, to_read={3:d78cbf22:::100000001a7.0000000c:head=40960,4096,0,278528,4096,0,368640,4096,0,835584,4096,0,1036288,4096,0,1265664,4096,0,2080768,4096,0}, subchunks={3:d78cbf22:::100000001a7.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:42.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:20:42.440+0000 7fe528211700 -1 log_channel(cluster) log [ERR] : Health check update: mons b,c are very low on available space (MON_DISK_CRIT) 2019-10-13T04:20:42.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:42.618+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=28156, to_read={3:b9c448ca:::10000000193.00000000:head=86016,4096,0,299008,4096,0,1708032,4096,0}, subchunks={3:b9c448ca:::10000000193.00000000:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:43.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:43.662+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=28288, to_read={3:a46fa481:::100000001a6.00000014:head=1126400,4096,0,1392640,4096,0,1732608,4096,0}, subchunks={3:a46fa481:::100000001a6.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:45.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:45.080+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=50570, attrs_read=0)) 2019-10-13T04:20:45.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:45.678+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=28538, to_read={3:ac73c8bc:::100000001c4.0000000c:head=28672,4096,0,65536,4096,0,188416,4096,0,1142784,4096,0,1155072,4096,0,1314816,4096,0,1359872,4096,0}, subchunks={3:ac73c8bc:::100000001c4.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:49.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:49.120+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=51352, to_read={3:2a781fb2:::1000000017e.00000002:head=1662976,4096,0,1781760,4096,0,1859584,4096,0}, subchunks={3:2a781fb2:::1000000017e.00000002:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:49.566 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:20:49.566 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:49.577 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:20:49.577 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:20:49.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T04:20:49.708+0000 7f14077c5700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 20493) UID: 0 2019-10-13T04:20:49.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:49.708+0000 7f6c8e916700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 20493) UID: 0 2019-10-13T04:20:49.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:20:49.708+0000 7f292b77e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 20493) UID: 0 2019-10-13T04:20:49.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:20:49.708+0000 7f89fce79700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 20493) UID: 0 2019-10-13T04:20:51.094 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:20:51.086+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.3s0 23/21 ECSubReadReply(tid=26529, attrs_read=0)) 2019-10-13T04:20:53.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:53.144+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.4s0 23/21 ECSubRead(tid=52175, to_read={3:3b57bc37:::100000001c9.00000014:head=286720,4096,0,770048,4096,0,1560576,4096,0}, subchunks={3:3b57bc37:::100000001c9.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:20:53.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:20:53.778+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=29508, attrs_read=0)) 2019-10-13T04:20:57.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:20:57.152+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=53011, to_read={3:1a4db5fd:::1000000016f.00000010:head=372736,4096,0,634880,4096,0,991232,4096,0,1290240,4096,0,1507328,4096,0}, subchunks={3:1a4db5fd:::1000000016f.00000010:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:01.093 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:21:01.085+0000 7fa157a32700 -1 osd.4 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.3s0 23/21 ECSubRead(tid=27605, to_read={3:d76ad95e:::1000000019c.00000009:head=1740800,4096,0,1777664,4096,0,2088960,4096,0}, subchunks={3:d76ad95e:::1000000019c.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:01.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:01.112+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=53765, attrs_read=0)) 2019-10-13T04:21:02.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:02.092+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=53991, attrs_read=0)) 2019-10-13T04:21:05.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:05.055+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=54590, attrs_read=0)) 2019-10-13T04:21:06.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:06.059+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 23/21 ECSubReadReply(tid=54818, attrs_read=0)) 2019-10-13T04:21:12.934 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:12.925+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=31962, attrs_read=0)) 2019-10-13T04:21:14.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:14.969+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.5s0 23/21 ECSubReadReply(tid=32202, attrs_read=0)) 2019-10-13T04:21:16.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:16.003+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=56715, to_read={3:079d7c4d:::100000001a8.0000000e:head=1720320,4096,0,1859584,4096,0,2080768,4096,0}, subchunks={3:079d7c4d:::100000001a8.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:19.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:19.079+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 23/21 ECSubReadReply(tid=57310, attrs_read=0)) 2019-10-13T04:21:21.006 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:21.001+0000 7f9e802c3700 -1 osd.5 23 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.5s0 23/21 ECSubRead(tid=32842, to_read={3:bf8acf2c:::10000000199.0000000e:head=626688,4096,0,774144,4096,0,1597440,4096,0,1650688,4096,0}, subchunks={3:bf8acf2c:::10000000199.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:22.022 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:22.015+0000 7f6c8c89e700 -1 osd.1 23 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 23/21 ECSubRead(tid=57925, to_read={3:09f3ade1:::100000001a8.00000014:head=131072,4096,0,688128,4096,0}, subchunks={3:09f3ade1:::100000001a8.00000014:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:22.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:22.777+0000 7f9e7dabe700 -1 bdev(0x55f60cd3a000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.787 INFO:tasks.ceph.osd.5.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9e7dabe700 time 2019-10-13T04:21:22.782667+0000 2019-10-13T04:21:22.787 INFO:tasks.ceph.osd.5.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.787 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.787 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55f601db102c] 2019-10-13T04:21:22.787 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9e7dabe700 time 2019-10-13T04:21:22.782667+0000 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.788 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55f601db102c] 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.789 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.5.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.5.smithi204.stderr: in thread 7f9e7dabe700 thread_name:bstore_aio 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.4.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa15522d700 time 2019-10-13T04:21:22.782883+0000 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.4.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x56267eaf402c] 2019-10-13T04:21:22.790 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.791 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.791 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.791 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.791 INFO:tasks.ceph.osd.4.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:22.791 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7fa15522d700 thread_name:bstore_aio 2019-10-13T04:21:22.793 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:21:22.781+0000 7fac1de38700 -1 bdev(0x559e4f124000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.793 INFO:tasks.ceph.osd.6.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fac1de38700 time 2019-10-13T04:21:22.785384+0000 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x559e42a5d02c] 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.794 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:21:22.781+0000 7fac1de38700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fac1de38700 time 2019-10-13T04:21:22.785384+0000 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.795 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x559e42a5d02c] 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:22.796 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7fac1de38700 thread_name:bstore_aio 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7fac28352890] 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fac27004e97] 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (abort()+0x141) [0x7fac27006801] 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x559e42a5d0fd] 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.797 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-13T04:21:22.785+0000 7fac1de38700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7fac1de38700 thread_name:bstore_aio 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.798 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7fac28352890] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fac27004e97] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (abort()+0x141) [0x7fac27006801] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x559e42a5d0fd] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.799 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.7.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8b1f6d5700 time 2019-10-13T04:21:22.784654+0000 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.7.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a10064302c] 2019-10-13T04:21:22.800 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr: in thread 7f8b1f6d5700 thread_name:bstore_aio 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 bdev(0x55a10cf72000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8b1f6d5700 time 2019-10-13T04:21:22.784654+0000 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a10064302c] 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.802 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (()+0x12890) [0x7f8b29bef890] 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f8b288a1e97] 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (abort()+0x141) [0x7f8b288a3801] 2019-10-13T04:21:22.803 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1006430fd] 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: 7: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: 8: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-13T04:21:22.785+0000 7f8b1f6d5700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: in thread 7f8b1f6d5700 thread_name:bstore_aio 2019-10-13T04:21:22.804 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (()+0x12890) [0x7f8b29bef890] 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f8b288a1e97] 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (abort()+0x141) [0x7f8b288a3801] 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1006430fd] 2019-10-13T04:21:22.805 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.7.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.7.smithi204.stderr: 7: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.7.smithi204.stderr: 8: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.7.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.806 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (()+0x12890) [0x7f9e87fd8890] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f9e86c8ae97] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (abort()+0x141) [0x7f9e86c8c801] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55f601db10fd] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 7: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr: 8: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.807 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: in thread 7f9e7dabe700 thread_name:bstore_aio 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (()+0x12890) [0x7f9e87fd8890] 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f9e86c8ae97] 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (abort()+0x141) [0x7f9e86c8c801] 2019-10-13T04:21:22.808 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55f601db10fd] 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: 7: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: 8: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.809 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7fa15f747890] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fa15e3f9e97] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (abort()+0x141) [0x7fa15e3fb801] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x56267eaf40fd] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.810 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:21:22.777+0000 7fa15522d700 -1 bdev(0x56268afbae00 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:21:22.781+0000 7fa15522d700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa15522d700 time 2019-10-13T04:21:22.782883+0000 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.811 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x56267eaf402c] 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.812 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-13T04:21:22.781+0000 7fa15522d700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7fa15522d700 thread_name:bstore_aio 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7fa15f747890] 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fa15e3f9e97] 2019-10-13T04:21:22.813 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (abort()+0x141) [0x7fa15e3fb801] 2019-10-13T04:21:22.814 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x56267eaf40fd] 2019-10-13T04:21:22.814 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.814 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.814 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.814 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.816 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.817 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.819 INFO:tasks.ceph.osd.6.smithi204.stderr: -12> 2019-10-13T04:21:22.781+0000 7fac1de38700 -1 bdev(0x559e4f124000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.819 INFO:tasks.ceph.osd.6.smithi204.stderr: -8> 2019-10-13T04:21:22.781+0000 7fac1de38700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fac1de38700 time 2019-10-13T04:21:22.785384+0000 2019-10-13T04:21:22.819 INFO:tasks.ceph.osd.6.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.819 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.819 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x559e42a5d02c] 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.820 INFO:tasks.ceph.osd.6.smithi204.stderr: 0> 2019-10-13T04:21:22.785+0000 7fac1de38700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7fac1de38700 thread_name:bstore_aio 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7fac28352890] 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fac27004e97] 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (abort()+0x141) [0x7fac27006801] 2019-10-13T04:21:22.821 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x559e42a5d0fd] 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.822 INFO:tasks.ceph.osd.6.smithi204.stderr: -9999> 2019-10-13T04:21:22.781+0000 7fac1de38700 -1 bdev(0x559e4f124000 /var/lib/ceph/osd/ceph-6/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: -9998> 2019-10-13T04:21:22.781+0000 7fac1de38700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fac1de38700 time 2019-10-13T04:21:22.785384+0000 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x559e42a5d02c] 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.823 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: -9997> 2019-10-13T04:21:22.785+0000 7fac1de38700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7fac1de38700 thread_name:bstore_aio 2019-10-13T04:21:22.824 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7fac28352890] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fac27004e97] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (abort()+0x141) [0x7fac27006801] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x559e42a5d0fd] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x559e430c62f9] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x559e430c824d] 2019-10-13T04:21:22.825 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (()+0x76db) [0x7fac283476db] 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (clone()+0x3f) [0x7fac270e788f] 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.4.smithi204.stderr: -103> 2019-10-13T04:21:22.777+0000 7fa15522d700 -1 bdev(0x56268afbae00 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.4.smithi204.stderr: -99> 2019-10-13T04:21:22.781+0000 7fa15522d700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa15522d700 time 2019-10-13T04:21:22.782883+0000 2019-10-13T04:21:22.826 INFO:tasks.ceph.osd.4.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x56267eaf402c] 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.827 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.828 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.828 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.828 INFO:tasks.ceph.osd.4.smithi204.stderr: 0> 2019-10-13T04:21:22.781+0000 7fa15522d700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.828 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7fa15522d700 thread_name:bstore_aio 2019-10-13T04:21:22.828 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7fa15f747890] 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fa15e3f9e97] 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (abort()+0x141) [0x7fa15e3fb801] 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x56267eaf40fd] 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.829 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: -9999> 2019-10-13T04:21:22.777+0000 7fa15522d700 -1 bdev(0x56268afbae00 /var/lib/ceph/osd/ceph-4/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr: -9998> 2019-10-13T04:21:22.781+0000 7fa15522d700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fa15522d700 time 2019-10-13T04:21:22.782883+0000 2019-10-13T04:21:22.830 INFO:tasks.ceph.osd.4.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x56267eaf402c] 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.831 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: -9997> 2019-10-13T04:21:22.781+0000 7fa15522d700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7fa15522d700 thread_name:bstore_aio 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7fa15f747890] 2019-10-13T04:21:22.832 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fa15e3f9e97] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (abort()+0x141) [0x7fa15e3fb801] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x56267eaf40fd] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x56267f15d2f9] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x56267f15f24d] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (()+0x76db) [0x7fa15f73c6db] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (clone()+0x3f) [0x7fa15e4dc88f] 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.833 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr: -287> 2019-10-13T04:21:22.777+0000 7f9e7dabe700 -1 bdev(0x55f60cd3a000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr: -144> 2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9e7dabe700 time 2019-10-13T04:21:22.782667+0000 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.834 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55f601db102c] 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 0> 2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: in thread 7f9e7dabe700 thread_name:bstore_aio 2019-10-13T04:21:22.835 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (()+0x12890) [0x7f9e87fd8890] 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f9e86c8ae97] 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (abort()+0x141) [0x7f9e86c8c801] 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55f601db10fd] 2019-10-13T04:21:22.836 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: 7: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: 8: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: -9999> 2019-10-13T04:21:22.777+0000 7f9e7dabe700 -1 bdev(0x55f60cd3a000 /var/lib/ceph/osd/ceph-5/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.837 INFO:tasks.ceph.osd.5.smithi204.stderr: -9998> 2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9e7dabe700 time 2019-10-13T04:21:22.782667+0000 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55f601db102c] 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.838 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: -9997> 2019-10-13T04:21:22.781+0000 7f9e7dabe700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: in thread 7f9e7dabe700 thread_name:bstore_aio 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.839 INFO:tasks.ceph.osd.5.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 1: (()+0x12890) [0x7f9e87fd8890] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f9e86c8ae97] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 3: (abort()+0x141) [0x7f9e86c8c801] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55f601db10fd] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55f60241a2f9] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55f60241c24d] 2019-10-13T04:21:22.840 INFO:tasks.ceph.osd.5.smithi204.stderr: 7: (()+0x76db) [0x7f9e87fcd6db] 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.5.smithi204.stderr: 8: (clone()+0x3f) [0x7f9e86d6d88f] 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.5.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.5.smithi204.stderr: 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.7.smithi204.stderr: -5> 2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 bdev(0x55a10cf72000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.7.smithi204.stderr: -1> 2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8b1f6d5700 time 2019-10-13T04:21:22.784654+0000 2019-10-13T04:21:22.841 INFO:tasks.ceph.osd.7.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a10064302c] 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.842 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 0> 2019-10-13T04:21:22.785+0000 7f8b1f6d5700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: in thread 7f8b1f6d5700 thread_name:bstore_aio 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (()+0x12890) [0x7f8b29bef890] 2019-10-13T04:21:22.843 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f8b288a1e97] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (abort()+0x141) [0x7f8b288a3801] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1006430fd] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 7: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: 8: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.844 INFO:tasks.ceph.osd.7.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.845 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.845 INFO:tasks.ceph.osd.7.smithi204.stderr: -9999> 2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 bdev(0x55a10cf72000 /var/lib/ceph/osd/ceph-7/block) _aio_thread got r=-28 ((28) No space left on device) 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: -9998> 2019-10-13T04:21:22.781+0000 7f8b1f6d5700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f8b1f6d5700 time 2019-10-13T04:21:22.784654+0000 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/os/bluestore/KernelDevice.cc: 550: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x55a10064302c] 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.846 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: -9997> 2019-10-13T04:21:22.785+0000 7f8b1f6d5700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: in thread 7f8b1f6d5700 thread_name:bstore_aio 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.847 INFO:tasks.ceph.osd.7.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 1: (()+0x12890) [0x7f8b29bef890] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f8b288a1e97] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 3: (abort()+0x141) [0x7f8b288a3801] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55a1006430fd] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 5: (KernelDevice::_aio_thread()+0xcd9) [0x55a100cac2f9] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55a100cae24d] 2019-10-13T04:21:22.848 INFO:tasks.ceph.osd.7.smithi204.stderr: 7: (()+0x76db) [0x7f8b29be46db] 2019-10-13T04:21:22.849 INFO:tasks.ceph.osd.7.smithi204.stderr: 8: (clone()+0x3f) [0x7f8b2898488f] 2019-10-13T04:21:22.849 INFO:tasks.ceph.osd.7.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:22.849 INFO:tasks.ceph.osd.7.smithi204.stderr: 2019-10-13T04:21:22.899 INFO:tasks.ceph.osd.7.smithi204.stderr:daemon-helper: command failed with exit status 5 2019-10-13T04:21:22.908 INFO:tasks.ceph.osd.6.smithi204.stderr:daemon-helper: command failed with exit status 5 2019-10-13T04:21:23.103 INFO:tasks.ceph.osd.5.smithi204.stderr:daemon-helper: command failed with exit status 5 2019-10-13T04:21:23.103 INFO:tasks.ceph.osd.4.smithi204.stderr:daemon-helper: command failed with exit status 5 2019-10-13T04:21:26.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:26.079+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 25/24 ECSubReadReply(tid=58413, attrs_read=0)) 2019-10-13T04:21:27.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:27.083+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 25/24 ECSubReadReply(tid=58665, attrs_read=0)) 2019-10-13T04:21:27.478 INFO:tasks.ceph.mon.c.smithi204.stderr:2019-10-13T04:21:27.473+0000 7fcaed4e1700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.479 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.479 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.479 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.480 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-13T04:21:27.473+0000 7f2f21409700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.481 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.481 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.481 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.483 INFO:tasks.ceph.mon.c.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fcaed4e1700 time 2019-10-13T04:21:27.476706+0000 2019-10-13T04:21:27.483 INFO:tasks.ceph.mon.c.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.483 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fcaf9cbe008] 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.484 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.485 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.485 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.485 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.485 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.485 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr: in thread 7fcaed4e1700 thread_name:ms_dispatch 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr:2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fcaed4e1700 time 2019-10-13T04:21:27.476706+0000 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.486 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fcaf9cbe008] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.487 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.488 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.488 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.488 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.488 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.488 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.489 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.489 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (()+0x12890) [0x7fcaf8b2b890] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fcaf7c23e97] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (abort()+0x141) [0x7fcaf7c25801] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7fcaf9cbe0d9] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.490 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.491 INFO:tasks.ceph.mon.c.smithi204.stderr: 13: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.492 INFO:tasks.ceph.mon.c.smithi204.stderr: 14: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.492 INFO:tasks.ceph.mon.c.smithi204.stderr:2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.492 INFO:tasks.ceph.mon.c.smithi204.stderr: in thread 7fcaed4e1700 thread_name:ms_dispatch 2019-10-13T04:21:27.492 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.494 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.494 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (()+0x12890) [0x7fcaf8b2b890] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fcaf7c23e97] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (abort()+0x141) [0x7fcaf7c25801] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7fcaf9cbe0d9] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.495 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.496 INFO:tasks.ceph.mon.c.smithi204.stderr: 13: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.c.smithi204.stderr: 14: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.c.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.b.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2f21409700 time 2019-10-13T04:21:27.478945+0000 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.b.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.497 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2f2dbe6008] 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.498 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.499 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.499 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.499 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.499 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.499 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.500 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-13T04:21:27.481+0000 7f2f21409700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2f21409700 time 2019-10-13T04:21:27.478945+0000 2019-10-13T04:21:27.500 INFO:tasks.ceph.mon.b.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.500 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.500 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.500 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2f2dbe6008] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.501 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.502 INFO:tasks.ceph.mon.b.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-13T04:21:27.503 INFO:tasks.ceph.mon.b.smithi204.stderr: in thread 7f2f21409700 thread_name:ms_dispatch 2019-10-13T04:21:27.503 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.504 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (()+0x12890) [0x7f2f2ca53890] 2019-10-13T04:21:27.504 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f2f2bb4be97] 2019-10-13T04:21:27.504 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (abort()+0x141) [0x7f2f2bb4d801] 2019-10-13T04:21:27.504 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2f2dbe60d9] 2019-10-13T04:21:27.504 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.505 INFO:tasks.ceph.mon.b.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr: 13: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr: 14: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-13T04:21:27.485+0000 7f2f21409700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr: in thread 7f2f21409700 thread_name:ms_dispatch 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.506 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.507 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (()+0x12890) [0x7f2f2ca53890] 2019-10-13T04:21:27.507 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f2f2bb4be97] 2019-10-13T04:21:27.507 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (abort()+0x141) [0x7f2f2bb4d801] 2019-10-13T04:21:27.507 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2f2dbe60d9] 2019-10-13T04:21:27.507 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.508 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.b.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.b.smithi204.stderr: 13: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.b.smithi204.stderr: 14: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.b.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.509 INFO:tasks.ceph.mon.c.smithi204.stderr: -3> 2019-10-13T04:21:27.473+0000 7fcaed4e1700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr: -1> 2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fcaed4e1700 time 2019-10-13T04:21:27.476706+0000 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.510 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fcaf9cbe008] 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.511 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.512 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.512 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.512 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.512 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.512 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: 0> 2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: in thread 7fcaed4e1700 thread_name:ms_dispatch 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (()+0x12890) [0x7fcaf8b2b890] 2019-10-13T04:21:27.513 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fcaf7c23e97] 2019-10-13T04:21:27.514 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (abort()+0x141) [0x7fcaf7c25801] 2019-10-13T04:21:27.514 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7fcaf9cbe0d9] 2019-10-13T04:21:27.514 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.514 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.514 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.515 INFO:tasks.ceph.mon.c.smithi204.stderr: 13: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.516 INFO:tasks.ceph.mon.c.smithi204.stderr: 14: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.516 INFO:tasks.ceph.mon.c.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.516 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.517 INFO:tasks.ceph.mon.c.smithi204.stderr: -9999> 2019-10-13T04:21:27.473+0000 7fcaed4e1700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-c/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.517 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.517 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.517 INFO:tasks.ceph.mon.c.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.517 INFO:tasks.ceph.mon.c.smithi204.stderr: -9998> 2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fcaed4e1700 time 2019-10-13T04:21:27.476706+0000 2019-10-13T04:21:27.518 INFO:tasks.ceph.mon.c.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.519 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.519 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.520 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fcaf9cbe008] 2019-10-13T04:21:27.520 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.520 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.520 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.520 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.521 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: -9997> 2019-10-13T04:21:27.477+0000 7fcaed4e1700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: in thread 7fcaed4e1700 thread_name:ms_dispatch 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: 1: (()+0x12890) [0x7fcaf8b2b890] 2019-10-13T04:21:27.522 INFO:tasks.ceph.mon.c.smithi204.stderr: 2: (gsignal()+0xc7) [0x7fcaf7c23e97] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 3: (abort()+0x141) [0x7fcaf7c25801] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7fcaf9cbe0d9] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x560b1f66a760] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x560b1f783e69] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x560b1f78989b] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x560b1f6abb1d] 2019-10-13T04:21:27.523 INFO:tasks.ceph.mon.c.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x560b1f6ac06a] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x560b1f6db8c8] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7fcaf9ea0caa] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fcaf9f3d3cd] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: 13: (()+0x76db) [0x7fcaf8b206db] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: 14: (clone()+0x3f) [0x7fcaf7d0688f] 2019-10-13T04:21:27.524 INFO:tasks.ceph.mon.c.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.c.smithi204.stderr: 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.b.smithi204.stderr: -3> 2019-10-13T04:21:27.473+0000 7f2f21409700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.525 INFO:tasks.ceph.mon.b.smithi204.stderr: -1> 2019-10-13T04:21:27.481+0000 7f2f21409700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2f21409700 time 2019-10-13T04:21:27.478945+0000 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2f2dbe6008] 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.526 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.527 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.528 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: 0> 2019-10-13T04:21:27.485+0000 7f2f21409700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: in thread 7f2f21409700 thread_name:ms_dispatch 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (()+0x12890) [0x7f2f2ca53890] 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f2f2bb4be97] 2019-10-13T04:21:27.529 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (abort()+0x141) [0x7f2f2bb4d801] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2f2dbe60d9] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.530 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: 13: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: 14: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.531 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr: -9999> 2019-10-13T04:21:27.473+0000 7f2f21409700 -1 rocksdb: submit_common error: IO error: No space left on deviceWhile open a file for appending: /var/lib/ceph/mon/ceph-b/store.db/000018.log: No space left on device code = 5 Rocksdb transaction: 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x00313132'8' Value size = 636) 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_v' Value size = 8) 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr:Put( Prefix = p key = 'xos'0x0070656e'ding_pn' Value size = 8) 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr: -9998> 2019-10-13T04:21:27.481+0000 7f2f21409700 -1 /build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f2f21409700 time 2019-10-13T04:21:27.478945+0000 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr:/build/ceph-15.0.0-6132-ga8ee8ac/src/mon/MonitorDBStore.h: 346: ceph_abort_msg("failed to write to db") 2019-10-13T04:21:27.532 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7f2f2dbe6008] 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.533 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.534 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: -9997> 2019-10-13T04:21:27.485+0000 7f2f21409700 -1 *** Caught signal (Aborted) ** 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: in thread 7f2f21409700 thread_name:ms_dispatch 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: ceph version 15.0.0-6132-ga8ee8ac (a8ee8ac95ca822967c9dffe2a705f1d5502812bd) octopus (dev) 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: 1: (()+0x12890) [0x7f2f2ca53890] 2019-10-13T04:21:27.535 INFO:tasks.ceph.mon.b.smithi204.stderr: 2: (gsignal()+0xc7) [0x7f2f2bb4be97] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 3: (abort()+0x141) [0x7f2f2bb4d801] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x7f2f2dbe60d9] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 5: (MonitorDBStore::apply_transaction(std::shared_ptr)+0xce0) [0x5617db55d760] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x399) [0x5617db676e69] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3fb) [0x5617db67c89b] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x172d) [0x5617db59eb1d] 2019-10-13T04:21:27.536 INFO:tasks.ceph.mon.b.smithi204.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x4da) [0x5617db59f06a] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x5617db5ce8c8] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: 11: (DispatchQueue::entry()+0x11ea) [0x7f2f2ddc8caa] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2f2de653cd] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: 13: (()+0x76db) [0x7f2f2ca486db] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: 14: (clone()+0x3f) [0x7f2f2bc2e88f] 2019-10-13T04:21:27.537 INFO:tasks.ceph.mon.b.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-13T04:21:27.538 INFO:tasks.ceph.mon.b.smithi204.stderr: 2019-10-13T04:21:27.709 INFO:tasks.ceph.mon.c.smithi204.stderr:daemon-helper: command crashed with signal 6 2019-10-13T04:21:28.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:28.115+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 25/24 ECSubReadReply(tid=58898, attrs_read=0)) 2019-10-13T04:21:28.504 INFO:tasks.ceph.mon.b.smithi204.stderr:daemon-helper: command crashed with signal 6 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2019-10-13T04:21:29.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2019-10-13T04:21:30.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:30.135+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 25/24 ECSubReadReply(tid=59278, attrs_read=0)) 2019-10-13T04:21:32.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:32.139+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 2 slow ops, oldest is MOSDECSubOpRead(3.0s0 25/24 ECSubRead(tid=59694, to_read={3:1538bfb8:::1000000019d.00000015:head=843776,4096,0,1998848,4096,0}, subchunks={3:1538bfb8:::1000000019d.00000015:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:33.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:33.179+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 25/24 ECSubRead(tid=59895, to_read={3:3fcb495d:::100000001cb.00000001:head=1761280,4096,0,1851392,4096,0,2027520,4096,0}, subchunks={3:3fcb495d:::100000001cb.00000001:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:34.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:34.147+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 25/24 ECSubReadReply(tid=60079, attrs_read=0)) 2019-10-13T04:21:34.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2019-10-13T04:21:34.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2019-10-13T04:21:34.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2019-10-13T04:21:34.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2019-10-13T04:21:34.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2019-10-13T04:21:34.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2019-10-13T04:21:37.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:37.159+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 25/24 ECSubRead(tid=60637, to_read={3:24fb9b69:::100000001cd.0000000c:head=851968,4096,0,974848,4096,0,1200128,4096,0,1208320,4096,0,1511424,4096,0,1765376,4096,0}, subchunks={3:24fb9b69:::100000001cd.0000000c:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~12s 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~12s 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~12s 2019-10-13T04:21:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~12s 2019-10-13T04:21:42.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:42.227+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 25/24 ECSubRead(tid=61560, to_read={3:18a645a9:::10000000185.00000009:head=376832,4096,0,749568,4096,0,831488,4096,0}, subchunks={3:18a645a9:::10000000185.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:43.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:43.295+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 25/24 ECSubRead(tid=61722, to_read={3:0863371a:::100000001b3.00000006:head=864256,4096,0,978944,4096,0,1617920,4096,0}, subchunks={3:0863371a:::100000001b3.00000006:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:21:46.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~18s 2019-10-13T04:21:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~18s 2019-10-13T04:21:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~18s 2019-10-13T04:21:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~18s 2019-10-13T04:21:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~18s 2019-10-13T04:21:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~18s 2019-10-13T04:21:48.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:21:48.367+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 25/24 ECSubReadReply(tid=62712, attrs_read=0)) 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~24s 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~24s 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~24s 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~24s 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~24s 2019-10-13T04:21:52.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~24s 2019-10-13T04:21:58.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2019-10-13T04:21:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2019-10-13T04:21:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2019-10-13T04:21:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2019-10-13T04:21:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2019-10-13T04:21:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2019-10-13T04:22:00.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:00.267+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 25/24 ECSubRead(tid=64910, to_read={3:2828d498:::1000000019d.0000000a:head=630784,4096,0,872448,4096,0,983040,4096,0}, subchunks={3:2828d498:::1000000019d.0000000a:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:22:04.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:04.226+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.0s0 25/24 ECSubReadReply(tid=65675, attrs_read=0)) 2019-10-13T04:22:04.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2019-10-13T04:22:04.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2019-10-13T04:22:04.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2019-10-13T04:22:04.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2019-10-13T04:22:04.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2019-10-13T04:22:04.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2019-10-13T04:22:05.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:05.186+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 25/24 ECSubRead(tid=65834, to_read={3:130ad609:::100000001c9.0000000e:head=741376,4096,0,1097728,4096,0,1155072,4096,0,1191936,4096,0}, subchunks={3:130ad609:::100000001c9.0000000e:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:22:06.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:06.146+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpReadReply(3.4s0 25/24 ECSubReadReply(tid=65954, attrs_read=0)) 2019-10-13T04:22:07.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:07.098+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.0s0 25/24 ECSubRead(tid=66122, to_read={3:1db6847b:::1000000019d.00000009:head=1036288,4096,0,1298432,4096,0}, subchunks={3:1db6847b:::1000000019d.00000009:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:22:08.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T04:22:08.366+0000 7f6c8c89e700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is MOSDECSubOpRead(3.4s0 25/24 ECSubRead(tid=66273, to_read={3:2619bf16:::100000001c9.00000011:head=815104,4096,0,942080,4096,0,1052672,4096,0,1073152,4096,0}, subchunks={3:2619bf16:::100000001c9.00000011:head=[0,1]}, attrs_to_read=)) 2019-10-13T04:22:10.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~41s 2019-10-13T04:22:10.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~41s 2019-10-13T04:22:10.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~41s 2019-10-13T04:22:10.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~41s 2019-10-13T04:22:10.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~41s 2019-10-13T04:22:10.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~41s 2019-10-13T04:22:16.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~47s 2019-10-13T04:22:16.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~47s 2019-10-13T04:22:16.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~47s 2019-10-13T04:22:16.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~47s 2019-10-13T04:22:16.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~47s 2019-10-13T04:22:16.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~47s 2019-10-13T04:22:21.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T04:22:21.482+0000 7f2929706700 -1 osd.3 25 get_health_metrics reporting 10 slow ops, oldest is MOSDECSubOpReadReply(3.7s0 25/24 ECSubReadReply(tid=66547, attrs_read=0)) 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~53s 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~53s 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~53s 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~53s 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~53s 2019-10-13T04:22:22.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~53s 2019-10-13T04:22:28.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~59s 2019-10-13T04:22:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~59s 2019-10-13T04:22:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~59s 2019-10-13T04:22:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~59s 2019-10-13T04:22:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~59s 2019-10-13T04:22:28.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~59s 2019-10-13T04:22:34.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2019-10-13T04:22:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2019-10-13T04:22:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2019-10-13T04:22:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~65s 2019-10-13T04:22:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2019-10-13T04:22:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2019-10-13T04:22:39.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~71s 2019-10-13T04:22:39.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~71s 2019-10-13T04:22:39.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~71s 2019-10-13T04:22:39.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~71s 2019-10-13T04:22:39.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~71s 2019-10-13T04:22:39.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~71s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~77s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~77s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~77s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~77s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~77s 2019-10-13T04:22:45.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~77s 2019-10-13T04:22:48.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T04:22:48.054+0000 7f89fae01700 -1 osd.0 25 get_health_metrics reporting 4 slow ops, oldest is MOSDECSubOpReadReply(3.1s0 25/24 ECSubReadReply(tid=66924, attrs_read=0)) 2019-10-13T04:22:51.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~83s 2019-10-13T04:22:51.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~83s 2019-10-13T04:22:51.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~83s 2019-10-13T04:22:51.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~83s 2019-10-13T04:22:51.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~83s 2019-10-13T04:22:51.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~83s 2019-10-13T04:22:57.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~89s 2019-10-13T04:22:57.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~89s 2019-10-13T04:22:57.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~89s 2019-10-13T04:22:57.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~89s 2019-10-13T04:22:57.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~89s 2019-10-13T04:22:57.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~89s 2019-10-13T04:23:03.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2019-10-13T04:23:03.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2019-10-13T04:23:03.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2019-10-13T04:23:03.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2019-10-13T04:23:03.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2019-10-13T04:23:03.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2019-10-13T04:23:09.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~100s 2019-10-13T04:23:09.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~100s 2019-10-13T04:23:09.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~100s 2019-10-13T04:23:09.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~100s 2019-10-13T04:23:09.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~100s 2019-10-13T04:23:09.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~100s 2019-10-13T04:23:15.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~106s 2019-10-13T04:23:15.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~106s 2019-10-13T04:23:15.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~106s 2019-10-13T04:23:15.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~106s 2019-10-13T04:23:15.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~106s 2019-10-13T04:23:15.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~106s 2019-10-13T04:23:21.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~112s 2019-10-13T04:23:21.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~112s 2019-10-13T04:23:21.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~112s 2019-10-13T04:23:21.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~112s 2019-10-13T04:23:21.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~112s 2019-10-13T04:23:21.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~112s 2019-10-13T04:23:27.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~118s 2019-10-13T04:23:27.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~118s 2019-10-13T04:23:27.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~118s 2019-10-13T04:23:27.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~118s 2019-10-13T04:23:27.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~118s 2019-10-13T04:23:27.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~118s 2019-10-13T04:23:32.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:32.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~124s 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~124s 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~124s 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~124s 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~124s 2019-10-13T04:23:33.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~124s 2019-10-13T04:23:37.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:37.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:23:39.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2019-10-13T04:23:39.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2019-10-13T04:23:39.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2019-10-13T04:23:39.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2019-10-13T04:23:39.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2019-10-13T04:23:39.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2019-10-13T04:23:42.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:42.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:23:44.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~136s 2019-10-13T04:23:44.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~136s 2019-10-13T04:23:44.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~136s 2019-10-13T04:23:44.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~136s 2019-10-13T04:23:44.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~136s 2019-10-13T04:23:44.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~136s 2019-10-13T04:23:47.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:47.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:23:50.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~142s 2019-10-13T04:23:50.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~142s 2019-10-13T04:23:50.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~142s 2019-10-13T04:23:50.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~142s 2019-10-13T04:23:50.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~142s 2019-10-13T04:23:50.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~142s 2019-10-13T04:23:52.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:52.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:23:56.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~148s 2019-10-13T04:23:56.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~148s 2019-10-13T04:23:56.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~148s 2019-10-13T04:23:56.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~148s 2019-10-13T04:23:56.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~148s 2019-10-13T04:23:56.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~148s 2019-10-13T04:23:57.482 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:23:57.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:02.482 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:02.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~154s 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~154s 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~154s 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~154s 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~154s 2019-10-13T04:24:02.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~154s 2019-10-13T04:24:07.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:07.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:08.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2019-10-13T04:24:08.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2019-10-13T04:24:08.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2019-10-13T04:24:08.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2019-10-13T04:24:08.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2019-10-13T04:24:08.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2019-10-13T04:24:12.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:12.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:14.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~166s 2019-10-13T04:24:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~166s 2019-10-13T04:24:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~166s 2019-10-13T04:24:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~166s 2019-10-13T04:24:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~166s 2019-10-13T04:24:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~166s 2019-10-13T04:24:17.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:17.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:20.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~171s 2019-10-13T04:24:20.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~171s 2019-10-13T04:24:20.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~171s 2019-10-13T04:24:20.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~171s 2019-10-13T04:24:20.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~171s 2019-10-13T04:24:20.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~171s 2019-10-13T04:24:22.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:22.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:26.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~177s 2019-10-13T04:24:26.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~177s 2019-10-13T04:24:26.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~177s 2019-10-13T04:24:26.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~177s 2019-10-13T04:24:26.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~177s 2019-10-13T04:24:26.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~177s 2019-10-13T04:24:27.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:27.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~183s 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~183s 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~183s 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~183s 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~183s 2019-10-13T04:24:32.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~183s 2019-10-13T04:24:32.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:32.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:37.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:37.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~189s 2019-10-13T04:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~189s 2019-10-13T04:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~189s 2019-10-13T04:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~189s 2019-10-13T04:24:38.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~189s 2019-10-13T04:24:38.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~189s 2019-10-13T04:24:42.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:42.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:44.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2019-10-13T04:24:44.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2019-10-13T04:24:44.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2019-10-13T04:24:44.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~195s 2019-10-13T04:24:44.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~195s 2019-10-13T04:24:44.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~195s 2019-10-13T04:24:47.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:47.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:50.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~201s 2019-10-13T04:24:50.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~201s 2019-10-13T04:24:50.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~201s 2019-10-13T04:24:50.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~201s 2019-10-13T04:24:50.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~201s 2019-10-13T04:24:50.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~201s 2019-10-13T04:24:52.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:52.484+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~207s 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~207s 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~207s 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~207s 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~207s 2019-10-13T04:24:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~207s 2019-10-13T04:24:57.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:24:57.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:01.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~213s 2019-10-13T04:25:01.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~213s 2019-10-13T04:25:01.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~213s 2019-10-13T04:25:01.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~213s 2019-10-13T04:25:01.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~213s 2019-10-13T04:25:01.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~213s 2019-10-13T04:25:02.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:02.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:07.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:07.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:07.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~219s 2019-10-13T04:25:07.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~219s 2019-10-13T04:25:07.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~219s 2019-10-13T04:25:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~219s 2019-10-13T04:25:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~219s 2019-10-13T04:25:07.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~219s 2019-10-13T04:25:12.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:12.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~225s 2019-10-13T04:25:13.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~225s 2019-10-13T04:25:17.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:17.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:19.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~231s 2019-10-13T04:25:19.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~231s 2019-10-13T04:25:19.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~231s 2019-10-13T04:25:19.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~231s 2019-10-13T04:25:19.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~231s 2019-10-13T04:25:19.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~231s 2019-10-13T04:25:22.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:22.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:25.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~236s 2019-10-13T04:25:25.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~236s 2019-10-13T04:25:25.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~236s 2019-10-13T04:25:25.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~236s 2019-10-13T04:25:25.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~236s 2019-10-13T04:25:25.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~236s 2019-10-13T04:25:27.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:27.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~242s 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~242s 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~242s 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~242s 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~242s 2019-10-13T04:25:31.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~242s 2019-10-13T04:25:32.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:32.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:37.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~248s 2019-10-13T04:25:37.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~248s 2019-10-13T04:25:37.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~248s 2019-10-13T04:25:37.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~248s 2019-10-13T04:25:37.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~248s 2019-10-13T04:25:37.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~248s 2019-10-13T04:25:37.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:37.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:42.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:42.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:43.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~254s 2019-10-13T04:25:43.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~254s 2019-10-13T04:25:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~254s 2019-10-13T04:25:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~254s 2019-10-13T04:25:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~254s 2019-10-13T04:25:43.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~254s 2019-10-13T04:25:47.497 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:47.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:49.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~260s 2019-10-13T04:25:49.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~260s 2019-10-13T04:25:49.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~260s 2019-10-13T04:25:49.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~260s 2019-10-13T04:25:49.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~260s 2019-10-13T04:25:49.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~260s 2019-10-13T04:25:52.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:52.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:25:55.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~266s 2019-10-13T04:25:55.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~266s 2019-10-13T04:25:55.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~266s 2019-10-13T04:25:55.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~266s 2019-10-13T04:25:55.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~266s 2019-10-13T04:25:55.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~266s 2019-10-13T04:25:57.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:25:57.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 491 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:00.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~272s 2019-10-13T04:26:00.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~272s 2019-10-13T04:26:00.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~272s 2019-10-13T04:26:00.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~272s 2019-10-13T04:26:00.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~272s 2019-10-13T04:26:00.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~272s 2019-10-13T04:26:02.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:02.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~278s 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~278s 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~278s 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~278s 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~278s 2019-10-13T04:26:06.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~278s 2019-10-13T04:26:07.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:07.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 527 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:12.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:12.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:12.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~284s 2019-10-13T04:26:12.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~284s 2019-10-13T04:26:12.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~284s 2019-10-13T04:26:12.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~284s 2019-10-13T04:26:12.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~284s 2019-10-13T04:26:12.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~284s 2019-10-13T04:26:17.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:17.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:18.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2019-10-13T04:26:18.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2019-10-13T04:26:18.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2019-10-13T04:26:18.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2019-10-13T04:26:18.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2019-10-13T04:26:18.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2019-10-13T04:26:22.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:22.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~296s 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~296s 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~296s 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~296s 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~296s 2019-10-13T04:26:24.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~296s 2019-10-13T04:26:27.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:27.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:30.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~301s 2019-10-13T04:26:30.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~301s 2019-10-13T04:26:30.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~301s 2019-10-13T04:26:30.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~301s 2019-10-13T04:26:30.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~301s 2019-10-13T04:26:30.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~301s 2019-10-13T04:26:30.496 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-10-13T04:26:30.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T04:26:30.496 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:26:30.496 INFO:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2019-10-13T04:26:30.496 INFO:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T04:26:32.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:32.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:37.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:37.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:42.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:42.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:47.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:47.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:52.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:52.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:26:57.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:26:57.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:02.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:02.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:07.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:07.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:12.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:12.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:17.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:17.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:22.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:22.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:27.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:27.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:32.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:32.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:37.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:37.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:42.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:42.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:47.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:47.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:52.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:52.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:27:57.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:27:57.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:02.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:02.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:07.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:07.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:12.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:12.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:17.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:17.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:22.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:22.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:27.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:27.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:32.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:32.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:37.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:37.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:42.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:42.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:47.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:47.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:52.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:52.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:28:57.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:28:57.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:02.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:02.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:05.454 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:29:05.454 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:05.458 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:29:05.458 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:07.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:07.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:12.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:12.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:17.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:17.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1200 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:22.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:22.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:27.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:27.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:32.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:32.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:35.557 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:29:35.557 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:35.562 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:29:35.562 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:29:37.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:37.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:42.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:42.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:47.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:47.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:52.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:52.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:29:57.532 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:29:57.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1348 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:02.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:02.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1366 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:05.690 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:30:05.690 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:05.693 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:30:05.694 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:07.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:07.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:12.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:12.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:17.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:17.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:22.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:22.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1440 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:27.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:27.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:32.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:32.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:35.835 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:30:35.835 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:35.839 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:30:35.839 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:30:37.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:37.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:42.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:42.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:47.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:47.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:52.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:52.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:30:57.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:30:57.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:02.541 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:02.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1587 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:05.952 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:31:05.952 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:05.957 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:31:05.957 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:07.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:07.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:12.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:12.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:17.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:17.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1643 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:22.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:22.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:27.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:27.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:32.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:32.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1699 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:35.996 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:31:35.996 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:36.009 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:31:36.010 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:31:37.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:37.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:42.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:42.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:47.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:47.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:52.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:52.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:31:57.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:31:57.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:02.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:02.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:06.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:32:06.118 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:06.122 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:32:06.122 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:07.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:07.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:12.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:12.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:17.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:17.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:22.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:22.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:27.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:27.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:32.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:32.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:36.241 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:32:36.242 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:36.246 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:32:36.246 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:32:37.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:37.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:42.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:42.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:47.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:47.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:52.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:52.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:32:57.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:32:57.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:02.562 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:02.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:06.357 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:33:06.357 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:06.360 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:33:06.361 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:07.562 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:07.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:12.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:12.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2068 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:17.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:17.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:22.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:22.559+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:27.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:27.559+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:32.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:32.559+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:36.430 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:33:36.430 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:36.434 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:33:36.434 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:33:37.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:37.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:42.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:42.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:47.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:47.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:52.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:52.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:33:57.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:33:57.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:02.572 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:02.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:06.539 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:34:06.539 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:06.543 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:34:06.543 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:07.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:07.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:12.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:12.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:17.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:17.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:22.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:22.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:27.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:27.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:32.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:32.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:36.680 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:34:36.680 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:36.684 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:34:36.684 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:34:37.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:37.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:42.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:42.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2399 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:47.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:47.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:52.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:52.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:34:57.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:34:57.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:02.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:02.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:06.791 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:35:06.791 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:06.794 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:35:06.794 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:07.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:07.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:12.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:12.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:17.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:17.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:22.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:22.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:27.587 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:27.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:32.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:32.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:36.857 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:35:36.857 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:36.861 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:35:36.861 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:35:37.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:37.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:42.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:42.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:47.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:47.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:52.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:52.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:35:57.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:35:57.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:02.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:02.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:06.987 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:36:06.987 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:06.991 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:36:06.992 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:07.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:07.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:12.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:12.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:17.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:17.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2750 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:22.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:22.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:27.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:27.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:32.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:32.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:37.101 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:36:37.101 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:37.104 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:36:37.105 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:36:37.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:37.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:42.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:42.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2842 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:47.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:47.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:52.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:36:57.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:36:57.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:02.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:02.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:07.214 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:37:07.214 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:07.218 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:37:07.218 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:07.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:07.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:12.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:12.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:17.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:17.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:22.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:22.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:27.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:27.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:32.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:32.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:37.337 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:37:37.337 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:37.345 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:37:37.345 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:37:37.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:37.602+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:42.611 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:42.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:47.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:47.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:52.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:52.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:37:57.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:37:57.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:02.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:02.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3137 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:07.433 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:38:07.433 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:07.437 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:38:07.437 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:07.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:07.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:12.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:12.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:17.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:17.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:22.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:22.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:27.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:27.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:32.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:32.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:37.479 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:38:37.480 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:37.484 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:38:37.484 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:38:37.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:37.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:42.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:42.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:47.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:47.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:52.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:52.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:38:57.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:38:57.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:02.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:02.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:07.608 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:39:07.609 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:07.612 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:39:07.613 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:07.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:07.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:12.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:12.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:17.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:17.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:22.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:22.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:27.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:27.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:32.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:32.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:37.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:37.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:37.733 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:39:37.733 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:37.737 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:39:37.737 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:39:42.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:42.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:47.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:47.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:52.635 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:52.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:39:57.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:39:57.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:02.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:02.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:07.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:07.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:07.771 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:40:07.771 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:07.775 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:40:07.775 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:12.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:12.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:17.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:17.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:22.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:22.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:27.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:27.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:32.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:32.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:37.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:37.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:37.835 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:40:37.835 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:37.839 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:40:37.839 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:40:42.645 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:42.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3728 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:47.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:47.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3748 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:52.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:52.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:40:57.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:40:57.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:02.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:02.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:07.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:07.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:07.976 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:41:07.976 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:07.980 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:41:07.981 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:12.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:12.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:17.653 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:17.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:22.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:22.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3875 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:27.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:27.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:30.502 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T04:41:30.504 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T04:41:32.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:32.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:37.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:37.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:38.279 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:41:38.280 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:38.283 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:41:38.283 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:41:42.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:42.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:47.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:47.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:52.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:52.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:41:57.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:41:57.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4005 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:02.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:02.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4023 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:07.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:07.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:08.385 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:42:08.385 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:08.389 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:42:08.389 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:12.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:12.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:17.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:17.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:22.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:22.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:27.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:27.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:32.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:32.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:37.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:37.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:38.461 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:42:38.461 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:38.464 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:42:38.465 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:42:42.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:42.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:47.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:47.665+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:52.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:52.665+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:42:57.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:42:57.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:02.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:02.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4244 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:07.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:07.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:08.570 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:43:08.570 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:08.573 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:43:08.574 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:12.675 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:12.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:17.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:17.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:22.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:22.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:27.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:27.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:32.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:32.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:37.681 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:37.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:38.701 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:43:38.701 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:38.704 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:43:38.704 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:43:42.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:42.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:47.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:47.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:52.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:52.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:43:57.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:43:57.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:02.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:02.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:07.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:07.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:08.832 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:44:08.832 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:08.835 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:44:08.835 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:12.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:12.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:17.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:17.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4521 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:22.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:22.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:27.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:27.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:32.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:32.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:37.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:37.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:38.945 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:44:38.945 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:38.949 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:44:38.949 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:44:42.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:42.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:47.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:47.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:52.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:52.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:44:57.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:44:57.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4669 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:02.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:02.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:07.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:07.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:09.060 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:45:09.060 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:09.063 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:45:09.064 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:12.700 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:12.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:17.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:17.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:22.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:22.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4761 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:27.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:27.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4781 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:32.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:32.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:37.705 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:37.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:39.324 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:45:39.325 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:39.329 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:45:39.329 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:45:42.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:42.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:47.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:47.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4855 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:52.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:52.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:45:57.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:45:57.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:02.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:02.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:07.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:07.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:09.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:46:09.432 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:09.436 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:46:09.436 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:12.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:12.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:17.714 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:17.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:22.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:22.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:27.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:27.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:32.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:32.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:37.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:37.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:39.558 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:46:39.558 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:39.562 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:46:39.562 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:46:42.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:42.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:47.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:47.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:52.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:52.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:46:57.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:46:57.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:02.725 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:02.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:07.726 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:07.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:09.652 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:47:09.652 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:09.656 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:47:09.656 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:12.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:12.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:17.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:17.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:22.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:22.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:27.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:27.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5224 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:32.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:32.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:37.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:37.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:39.769 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:47:39.769 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:39.773 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:47:39.773 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:47:42.734 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:42.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:47.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:47.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:52.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:52.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:47:57.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:47:57.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5333 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:02.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:02.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:07.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:07.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5371 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:09.899 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:48:09.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:09.903 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:48:09.903 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:12.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:12.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:17.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:17.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:22.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:22.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:27.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:27.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:32.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:32.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5463 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:37.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:37.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:39.988 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:48:39.989 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:39.992 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:48:39.992 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:48:42.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:42.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:47.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:47.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:52.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:52.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:48:57.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:48:57.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:02.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:02.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:07.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:07.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:10.110 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:49:10.110 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:10.115 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:49:10.115 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:12.754 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:12.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:17.754 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:17.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:22.756 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:22.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5647 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:27.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:27.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:32.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:32.750+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:37.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:37.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:40.151 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:49:40.151 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:40.155 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:49:40.155 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:49:42.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:42.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:47.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:47.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5741 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:52.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:52.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:49:57.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:49:57.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:02.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:02.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:07.766 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:07.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:10.278 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:50:10.278 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:10.283 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:50:10.283 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:12.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:12.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:17.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:17.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:22.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:22.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:27.771 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:27.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:32.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:32.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:37.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:37.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5924 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:40.589 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:50:40.589 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:40.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:50:40.593 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:50:42.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:42.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:47.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:47.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:52.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:52.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:50:57.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:50:57.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5998 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:02.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:02.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:07.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:07.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:10.632 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:51:10.633 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:10.636 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:51:10.636 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:12.781 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:12.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:17.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:17.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:22.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:22.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6090 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:27.785 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:27.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:32.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:32.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:37.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:37.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:40.745 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:51:40.746 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:40.752 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:51:40.752 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:51:42.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:42.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6163 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:47.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:47.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:52.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:52.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:51:57.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:51:57.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:02.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:02.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:07.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:07.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:10.867 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:52:10.868 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:10.871 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:52:10.872 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:12.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:12.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6275 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:17.798 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:17.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:22.798 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:22.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:27.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:27.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:32.801 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:32.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:37.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:37.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:40.988 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:52:40.988 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:40.992 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:52:40.992 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:52:42.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:42.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:47.806 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:47.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:52.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:52.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:52:57.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:52:57.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:02.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:02.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:07.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:07.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6479 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:11.128 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:53:11.128 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:11.132 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:53:11.132 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:12.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:12.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:17.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:17.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:22.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:22.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:27.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:27.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:32.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:32.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:37.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:37.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:41.241 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:53:41.241 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:41.245 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:53:41.245 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:53:42.818 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:42.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6606 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:47.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:47.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:52.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:52.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:53:57.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:53:57.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:02.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:02.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:07.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:07.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:11.348 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:54:11.348 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:11.400 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:54:11.400 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:12.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:12.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:17.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:17.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:22.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:22.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:27.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:27.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:32.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:32.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:37.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:37.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:41.540 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:54:41.540 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:41.544 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:54:41.544 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:54:42.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:42.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:47.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:47.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:52.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:52.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:54:57.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:54:57.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:02.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:02.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:07.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:07.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:11.686 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:55:11.686 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:11.691 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:55:11.691 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:12.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:12.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:17.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:17.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:22.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:22.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6975 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:27.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:27.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:32.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:32.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:37.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:37.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:41.913 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:55:41.914 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:41.917 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:55:41.918 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:55:42.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:42.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:47.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:47.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:52.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:52.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:55:57.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:55:57.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:02.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:02.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:07.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:07.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:12.027 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:56:12.027 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:12.031 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:56:12.031 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:12.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:12.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:17.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:17.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:22.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:22.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:27.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:27.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:32.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:32.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:37.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:37.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7253 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:42.164 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:56:42.164 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:42.168 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:56:42.168 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:56:42.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:42.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:47.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:47.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:52.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:52.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7308 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:56:57.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:56:57.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:02.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:02.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:07.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:07.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7364 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:12.280 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:57:12.280 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:12.284 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:57:12.285 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:12.872 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:12.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:17.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:17.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:22.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:22.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:27.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:27.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:32.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:32.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7456 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:37.878 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:37.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:42.377 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:57:42.378 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:42.381 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:57:42.381 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:57:42.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:42.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:47.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:47.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:52.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:52.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:57:57.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:57:57.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:02.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:02.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:07.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:07.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:12.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:58:12.493 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:12.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:58:12.497 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:12.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:12.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7604 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:17.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:17.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:22.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:22.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:27.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:27.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:32.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:32.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:37.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:37.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:42.626 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:58:42.626 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:42.630 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:58:42.630 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:58:42.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:42.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:47.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:47.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7733 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:52.897 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:52.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:58:57.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:58:57.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:02.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:02.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7787 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:07.901 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:07.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:12.722 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:59:12.723 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:12.728 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:59:12.729 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:12.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:12.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:17.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:17.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:22.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:22.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:27.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:27.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7881 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:32.907 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:32.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7899 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:37.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:37.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:42.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:42.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:43.038 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T04:59:43.038 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:43.042 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T04:59:43.042 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T04:59:47.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:47.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:52.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:52.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T04:59:57.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T04:59:57.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:02.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:02.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:07.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:07.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8029 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:12.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:12.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:13.103 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:00:13.103 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:13.106 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:00:13.107 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:17.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:17.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:22.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:22.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:27.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:27.915+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:32.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:32.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:37.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:37.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:42.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:42.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:43.385 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:00:43.385 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:43.389 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:00:43.389 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:00:47.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:47.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:52.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:52.922+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:00:57.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:00:57.922+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:02.931 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:02.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:07.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:07.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:12.934 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:12.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8268 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:13.507 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:01:13.507 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:13.512 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:01:13.512 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:17.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:17.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:22.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:22.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:27.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:27.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8324 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:32.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:32.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:37.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:37.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:42.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:42.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:43.623 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:01:43.623 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:43.627 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:01:43.627 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:01:47.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:47.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8397 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:52.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:52.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:01:57.946 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:01:57.941+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:02.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:02.941+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:07.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:07.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:12.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:12.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8489 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:13.749 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:02:13.749 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:13.753 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:02:13.753 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:17.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:17.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:22.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:22.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:27.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:27.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:32.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:32.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:37.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:37.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:42.960 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:42.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:43.864 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:02:43.864 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:43.868 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:02:43.868 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:02:47.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:47.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:52.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:52.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8637 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:02:57.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:02:57.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:02.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:02.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:07.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:07.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:12.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:12.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:13.984 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:03:13.985 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:13.988 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:03:13.988 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:17.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:17.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:22.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:22.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:27.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:27.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:32.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:32.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:37.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:37.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:42.976 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:42.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8820 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:44.058 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:03:44.058 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:44.062 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:03:44.062 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:03:47.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:47.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:52.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:52.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:03:57.980 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:03:57.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:02.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:02.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:07.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:07.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:12.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:12.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:14.400 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:04:14.400 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:14.406 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:04:14.406 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:17.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:17.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:22.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:22.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8968 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:27.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:27.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:32.990 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:32.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:37.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:37.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:42.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:42.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:44.439 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:04:44.439 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:44.443 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:04:44.443 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:04:47.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:47.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:52.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:52.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:04:57.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:04:57.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:02.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:02.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:08.000 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:07.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:13.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:12.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:14.545 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:05:14.545 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:14.548 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:05:14.549 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:18.003 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:17.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:23.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:22.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:28.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:28.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:33.008 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:33.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:38.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:38.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:43.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:43.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:44.678 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:05:44.679 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:44.682 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:05:44.682 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:05:48.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:48.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9283 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:53.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:53.009+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:05:58.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:05:58.009+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:03.016 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:03.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9337 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:08.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:08.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9357 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:13.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:13.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:14.792 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:06:14.792 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:14.796 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:06:14.796 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:18.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:18.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:23.022 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:23.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:28.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:28.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:33.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:33.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:38.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:38.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:43.028 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:43.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:45.066 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:06:45.066 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:45.070 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:06:45.070 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:06:48.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:48.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:53.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:53.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:06:58.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:06:58.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:03.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:03.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:08.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:08.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:13.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:13.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:15.188 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:07:15.188 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:15.192 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:07:15.192 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:18.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:18.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:23.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:23.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:28.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:28.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:33.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:33.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:38.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:38.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:43.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:43.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:45.285 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:07:45.286 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:45.290 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:07:45.290 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:07:48.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:48.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9726 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:53.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:53.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:07:58.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:07:58.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:03.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:03.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9780 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:08.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:08.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:13.055 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:13.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9818 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:15.411 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:08:15.411 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:15.414 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:08:15.415 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:18.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:18.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:23.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:23.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9854 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:28.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:28.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:33.061 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:33.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:38.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:38.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:43.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:43.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:45.530 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:08:45.530 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:45.535 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:08:45.535 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:08:48.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:48.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:53.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:53.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:08:58.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:08:58.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:03.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:03.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:08.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:08.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:13.074 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:13.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:15.660 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:09:15.660 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:15.665 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:09:15.665 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:18.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:18.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:23.077 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:23.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:28.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:28.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10095 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:33.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:33.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10113 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:38.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:38.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:43.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:43.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:45.765 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:09:45.765 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:45.769 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:09:45.769 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:09:48.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:48.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10169 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:53.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:53.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:09:58.087 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:09:58.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:03.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:03.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:08.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:08.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:13.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:13.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:15.896 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:10:15.896 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:15.903 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:10:15.903 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:18.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:18.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:23.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:23.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:28.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:28.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:33.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:33.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:38.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:38.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:43.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:43.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:46.136 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:10:46.136 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:46.164 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:10:46.164 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:10:48.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:48.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10390 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:53.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:53.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:10:58.105 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:10:58.099+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:03.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:03.099+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:08.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:08.099+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:13.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:13.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:16.276 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:11:16.277 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:16.281 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:11:16.281 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:18.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:18.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:23.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:23.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:28.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:28.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:33.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:33.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:38.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:38.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:43.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:43.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10592 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:46.331 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:11:46.331 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:46.335 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:11:46.335 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:11:48.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:48.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10612 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:53.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:53.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:11:58.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:11:58.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10647 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:03.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:03.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10666 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:08.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:08.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:13.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:13.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:16.438 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:12:16.438 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:16.441 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:12:16.442 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:18.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:18.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:23.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:23.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:28.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:28.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10759 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:33.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:33.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:38.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:38.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:43.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:43.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:46.561 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:12:46.561 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:46.564 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:12:46.565 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:12:48.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:48.134+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:53.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:53.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10851 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:12:58.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:12:58.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:03.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:03.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10887 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:08.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:08.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10907 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:13.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:13.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:16.603 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:13:16.603 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:16.640 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:13:16.640 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:18.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:18.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10943 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:23.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:23.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:28.151 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:28.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:33.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:33.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:38.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:38.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:43.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:43.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:46.830 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:13:46.830 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:46.834 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:13:46.834 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:13:48.160 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:48.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11055 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:53.167 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:53.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:13:58.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:13:58.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:03.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:03.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:08.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:08.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:13.167 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:13.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:16.955 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:14:16.955 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:16.963 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:14:16.964 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:18.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:18.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11164 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:23.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:23.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:28.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:28.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:33.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:33.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:38.175 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:38.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:43.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:43.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:47.094 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:14:47.094 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:47.103 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:14:47.103 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:14:48.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:48.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:53.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:53.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:14:58.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:14:58.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:03.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:03.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:08.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:08.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:13.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:13.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:17.225 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:15:17.226 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:17.229 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:15:17.229 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:18.188 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:18.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:23.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:23.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:28.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:28.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:33.193 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:33.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:38.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:38.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:43.196 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:43.186+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:47.572 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:15:47.572 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:47.576 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:15:47.576 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:15:48.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:48.190+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11497 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:53.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:53.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:15:58.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:15:58.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:03.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:03.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:08.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:08.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:13.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:13.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11589 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:17.619 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:16:17.619 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:17.623 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:16:17.623 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:18.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:18.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:23.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:23.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:28.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:28.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11645 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:33.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:33.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:38.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:38.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:43.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:43.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11699 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:47.723 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:16:47.723 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:47.726 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:16:47.726 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:16:48.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:48.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:53.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:53.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:16:58.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:16:58.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:03.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:03.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:08.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:08.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:13.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:13.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:17.862 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:17:17.863 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:17.866 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:17:17.866 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:18.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:18.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:23.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:23.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:28.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:28.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:33.232 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:33.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:38.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:38.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:43.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:43.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:47.916 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:17:47.916 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:47.920 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:17:47.920 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:17:48.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:48.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:53.238 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:53.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11958 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:17:58.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:17:58.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:03.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:03.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:08.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:08.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:13.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:13.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:18.042 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:18:18.042 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:18.046 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:18:18.046 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:18.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:18.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:23.248 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:23.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12068 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:28.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:28.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:33.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:33.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:38.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:38.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:43.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:43.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:48.154 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:18:48.154 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:48.158 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:18:48.158 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:18:48.256 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:48.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:53.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:53.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12180 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:18:58.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:18:58.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12197 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:03.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:03.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:08.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:08.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:13.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:13.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:18.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:18.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:18.460 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:19:18.461 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:18.464 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:19:18.465 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:23.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:23.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:28.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:28.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:33.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:33.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:38.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:38.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:43.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:43.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:48.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:48.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:48.508 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:19:48.508 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:48.511 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:19:48.512 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:19:53.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:53.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:19:58.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:19:58.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:03.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:03.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:08.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:08.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:13.285 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:13.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:18.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:18.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:18.651 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:20:18.651 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:18.656 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:20:18.657 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:23.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:23.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:28.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:28.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:33.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:33.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:38.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:38.289+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:43.297 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:43.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:48.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:48.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:48.901 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:20:48.901 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:48.904 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:20:48.905 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:20:53.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:53.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:20:58.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:20:58.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:03.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:03.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:08.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:08.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12678 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:13.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:13.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:18.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:18.305+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:19.005 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:21:19.005 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:19.009 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:21:19.009 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:23.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:23.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:28.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:28.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:33.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:33.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:38.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:38.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:43.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:43.312+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:48.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:48.312+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:49.120 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:21:49.121 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:49.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:21:49.125 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:21:53.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:53.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:21:58.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:21:58.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:03.325 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:03.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:08.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:08.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:13.328 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:13.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:18.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:18.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:19.226 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:22:19.226 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:19.230 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:22:19.230 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:23.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:23.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:28.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:28.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:33.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:33.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:38.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:38.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:43.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:43.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:48.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:48.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:49.333 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:22:49.334 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:49.337 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:22:49.338 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:22:53.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:53.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:22:58.343 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:22:58.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:03.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:03.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:08.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:08.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:13.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:13.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:18.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:18.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:19.456 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:23:19.456 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:19.460 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:23:19.460 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:23.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:23.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:28.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:28.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:33.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:33.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13213 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:38.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:38.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:43.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:43.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:48.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:48.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:49.584 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:23:49.584 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:49.589 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:23:49.589 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:23:53.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:53.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:23:58.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:23:58.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:03.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:03.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:08.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:08.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:13.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:13.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:18.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:18.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:19.698 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:24:19.698 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:19.701 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:24:19.702 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:23.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:23.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:28.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:28.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:33.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:33.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:38.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:38.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:43.382 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:43.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:48.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:48.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13490 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:49.801 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:24:49.801 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:49.804 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:24:49.805 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:24:53.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:53.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:24:58.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:24:58.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:03.390 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:03.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:08.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:08.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:13.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:13.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13582 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:18.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:18.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:19.918 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:25:19.918 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:19.922 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:25:19.922 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:23.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:23.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:28.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:28.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:33.401 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:33.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:38.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:38.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:43.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:43.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:48.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:48.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:50.222 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:25:50.223 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:50.226 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:25:50.227 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:25:53.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:53.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:25:58.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:25:58.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:03.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:03.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:08.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:08.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:13.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:13.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:18.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:18.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:20.336 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:26:20.336 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:20.343 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:26:20.343 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:23.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:23.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:28.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:28.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:33.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:33.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:38.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:38.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:43.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:43.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:48.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:48.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:50.408 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:26:50.408 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:50.412 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:26:50.413 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:26:53.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:53.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:26:58.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:26:58.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:03.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:03.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:08.436 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:08.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14007 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:13.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:13.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:18.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:18.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:20.576 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:27:20.576 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:20.584 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:27:20.584 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:23.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:23.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:28.444 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:28.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:33.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:33.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14099 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:38.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:38.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:43.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:43.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:48.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:48.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:50.710 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:27:50.710 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:50.714 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:27:50.714 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:27:53.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:53.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14172 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:27:58.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:27:58.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:03.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:03.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:08.459 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:08.453+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:13.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:13.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:18.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:18.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:20.833 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:28:20.834 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:20.837 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:28:20.837 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:23.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:23.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:28.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:28.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:33.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:33.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:38.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:38.465+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:43.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:43.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:48.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:48.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:50.952 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:28:50.952 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:50.958 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:28:50.959 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:28:53.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:53.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:28:58.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:28:58.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:03.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:03.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:08.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:08.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:13.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:13.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:18.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:18.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:21.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:29:21.061 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:21.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:29:21.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:23.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:23.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:28.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:28.484+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:33.499 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:33.484+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14542 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:38.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:38.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:43.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:43.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:48.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:48.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:51.173 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:29:51.174 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:51.178 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:29:51.178 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:29:53.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:53.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:29:58.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:29:58.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:03.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:03.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:08.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:08.499+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:13.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:13.499+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:18.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:18.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:21.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:30:21.318 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:21.322 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:30:21.322 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:23.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:23.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:28.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:28.507+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:33.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:33.507+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14763 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:38.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:38.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14781 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:43.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:43.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:48.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:48.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:51.385 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:30:51.385 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:51.389 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:30:51.389 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:30:53.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:53.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14837 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:30:58.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:30:58.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14855 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:03.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:03.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:08.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:08.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:13.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:13.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:18.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:18.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:21.426 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:31:21.426 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:21.431 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:31:21.431 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:23.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:23.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:28.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:28.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:33.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:33.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:38.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:38.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:43.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:43.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:48.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:48.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:51.484 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:31:51.485 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:51.488 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:31:51.488 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:31:53.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:53.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:31:58.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:31:58.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:03.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:03.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:08.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:08.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:13.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:13.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:18.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:18.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15148 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:21.607 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:32:21.607 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:21.610 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:32:21.611 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:23.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:23.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:28.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:28.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:33.562 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:33.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:38.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:38.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:43.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:43.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:48.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:48.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:51.715 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:32:51.715 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:51.719 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:32:51.719 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:32:53.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:53.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:32:58.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:32:58.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:03.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:03.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:08.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:08.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15333 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:13.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:13.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:18.581 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:18.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:21.847 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:33:21.847 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:21.851 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:33:21.852 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:23.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:23.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:28.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:28.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:33.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:33.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:38.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:38.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:43.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:43.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:48.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:48.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:52.133 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:33:52.134 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:52.137 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:33:52.137 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:33:53.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:53.587+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:33:58.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:33:58.587+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:03.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:03.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:08.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:08.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:13.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:13.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:18.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:18.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:22.272 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:34:22.272 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:22.276 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:34:22.276 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:23.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:23.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:28.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:28.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:33.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:33.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15647 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:38.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:38.602+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:43.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:43.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:48.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:48.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:52.490 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:34:52.490 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:52.494 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:34:52.494 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:34:53.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:53.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:34:58.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:34:58.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:03.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:03.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:08.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:08.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:13.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:13.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:18.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:18.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:22.572 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:35:22.572 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:22.576 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:35:22.576 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:23.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:23.622+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:28.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:28.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:33.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:33.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:38.635 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:38.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:43.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:43.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:48.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:48.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15924 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:52.653 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:35:52.654 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:52.657 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:35:52.657 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:35:53.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:53.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:35:58.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:35:58.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:03.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:03.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:08.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:08.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15998 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:13.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:13.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:18.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:18.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:22.753 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:36:22.753 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:22.757 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:36:22.757 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:23.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:23.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:28.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:28.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:33.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:33.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16090 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:38.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:38.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:43.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:43.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:48.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:48.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:52.866 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:36:52.866 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:52.870 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:36:52.870 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:36:53.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:53.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16163 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:36:58.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:36:58.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:03.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:03.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:08.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:08.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:13.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:13.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:18.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:18.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:22.914 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:37:22.914 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:22.918 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:37:22.919 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:23.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:23.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16275 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:28.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:28.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:33.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:33.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:38.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:38.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:43.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:43.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:48.686 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:48.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:53.035 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:37:53.036 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:53.040 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:37:53.040 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:37:53.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:53.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:37:58.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:37:58.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:03.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:03.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:08.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:08.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:13.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:13.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:18.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:18.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:23.104 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:38:23.104 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:23.108 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:38:23.108 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:23.700 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:23.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:28.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:28.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:33.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:33.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:38.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:38.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:43.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:43.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:48.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:48.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:53.219 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:38:53.221 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:53.224 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:38:53.225 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:38:53.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:53.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16606 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:38:58.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:38:58.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:03.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:03.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:08.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:08.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:13.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:13.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:18.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:18.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:23.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:39:23.318 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:23.322 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:39:23.322 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:23.725 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:23.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:28.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:28.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:33.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:33.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:38.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:38.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:43.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:43.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:48.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:48.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:53.449 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:39:53.449 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:53.453 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:39:53.453 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:39:53.738 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:53.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:39:58.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:39:58.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:03.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:03.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:08.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:08.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:13.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:13.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:18.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:18.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:23.555 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:40:23.555 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:23.559 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:40:23.559 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:23.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:23.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:28.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:28.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:33.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:33.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16975 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:38.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:38.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:43.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:43.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:48.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:48.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:53.620 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:40:53.621 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:53.624 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:40:53.624 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:40:53.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:53.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:40:58.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:40:58.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:03.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:03.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:08.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:08.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:13.771 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:13.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:18.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:18.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:23.665 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:41:23.665 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:23.669 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:41:23.669 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:23.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:23.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:28.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:28.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:33.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:33.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:38.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:38.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:43.783 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:43.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:48.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:48.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17253 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:53.779 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:41:53.779 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:53.782 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:41:53.783 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:41:53.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:53.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:41:58.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:41:58.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:03.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:03.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:08.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:08.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:13.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:13.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:18.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:18.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:23.801 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:23.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:23.892 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:42:23.893 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:23.896 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:42:23.896 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:28.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:28.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:33.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:33.798+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:38.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:38.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:43.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:43.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:48.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:48.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:53.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:53.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17491 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:42:54.057 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:42:54.057 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:54.061 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:42:54.061 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:42:58.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:42:58.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:03.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:03.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:08.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:08.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:13.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:13.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:18.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:18.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:23.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:23.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:24.170 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:43:24.171 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:24.174 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:43:24.175 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:28.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:28.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:33.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:33.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:38.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:38.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:43.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:43.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:48.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:48.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:53.840 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:53.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:43:54.267 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:43:54.267 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:54.270 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:43:54.271 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:43:58.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:43:58.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17731 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:03.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:03.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17750 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:08.846 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:08.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:13.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:13.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:18.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:18.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:23.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:23.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:24.313 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:44:24.313 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:24.317 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:44:24.317 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:28.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:28.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:33.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:33.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:38.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:38.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:43.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:43.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:48.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:48.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:53.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:53.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:44:54.444 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:44:54.444 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:54.448 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:44:54.448 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:44:58.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:44:58.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17953 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:03.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:03.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:08.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:08.867+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:13.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:13.867+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18007 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:18.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:18.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:23.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:23.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:24.548 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:45:24.549 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:24.552 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:45:24.553 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:28.881 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:28.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:33.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:33.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:38.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:38.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:43.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:43.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:48.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:48.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:53.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:53.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:45:54.614 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:45:54.614 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:54.618 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:45:54.618 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:45:58.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:45:58.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:03.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:03.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:08.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:08.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:13.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:13.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:18.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:18.894+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:23.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:23.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:24.744 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:46:24.745 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:24.748 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:46:24.749 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:28.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:28.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:33.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:33.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:38.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:38.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:43.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:43.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:48.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:48.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:53.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:53.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:46:54.858 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:46:54.858 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:54.862 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:46:54.862 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:46:58.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:46:58.909+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18395 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:03.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:03.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:08.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:08.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:13.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:13.921+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:18.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:18.921+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:23.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:23.925+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:25.022 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:47:25.022 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:25.026 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:47:25.026 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:28.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:28.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:33.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:33.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:38.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:38.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:43.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:43.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:48.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:48.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:53.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:53.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:47:55.127 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:47:55.127 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:55.131 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:47:55.131 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:47:58.946 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:47:58.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18617 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:03.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:03.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:08.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:08.944+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:13.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:13.944+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:18.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:18.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18691 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:23.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:23.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:25.383 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:48:25.383 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:25.386 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:48:25.387 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:28.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:28.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:33.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:33.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:38.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:38.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:43.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:43.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:48.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:48.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:53.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:53.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:48:55.438 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:48:55.438 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:55.442 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:48:55.442 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:48:58.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:48:58.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:03.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:03.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:08.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:08.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:13.980 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:13.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:18.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:18.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:23.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:23.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:25.559 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:49:25.560 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:25.563 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:49:25.563 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:28.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:28.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:33.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:33.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:38.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:38.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:43.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:43.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19005 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:48.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:48.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:53.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:53.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:49:55.699 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:49:55.699 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:55.706 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:49:55.707 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:49:58.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:49:58.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:04.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:03.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:09.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:08.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:14.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:13.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:19.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:19.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:24.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:24.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:25.839 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:50:25.840 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:25.844 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:50:25.844 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:29.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:29.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:34.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:34.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:39.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:39.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:44.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:44.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:49.022 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:49.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:54.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:54.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:50:55.913 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:50:55.913 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:55.917 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:50:55.917 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:50:59.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:50:59.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:04.029 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:04.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:09.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:09.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:14.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:14.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:19.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:19.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:24.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:24.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:25.962 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:51:25.962 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:25.965 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:51:25.966 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:29.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:29.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:34.044 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:34.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19410 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:39.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:39.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:44.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:44.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:49.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:49.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:54.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:54.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19484 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:51:56.064 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:51:56.064 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:56.068 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:51:56.068 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:51:59.055 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:51:59.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:04.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:04.052+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:09.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:09.052+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:14.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:14.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:19.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:19.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19577 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:24.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:24.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:26.119 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:52:26.119 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:26.123 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:52:26.123 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:29.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:29.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:34.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:34.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:39.074 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:39.068+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:44.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:44.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19669 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:49.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:49.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:54.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:54.075+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:52:56.399 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:52:56.399 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:56.403 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:52:56.403 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:52:59.083 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:52:59.075+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:04.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:04.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:09.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:09.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:14.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:14.083+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:19.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:19.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:24.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:24.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:26.447 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:53:26.447 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:26.451 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:53:26.451 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:29.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:29.091+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:34.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:34.091+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:39.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:39.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:44.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:44.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:49.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:49.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:54.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:54.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:53:56.569 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:53:56.569 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:56.573 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:53:56.573 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:53:59.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:53:59.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:04.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:04.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:09.116 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:09.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:14.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:14.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:19.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:19.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:24.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:24.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:26.744 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:54:26.745 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:26.748 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:54:26.748 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:29.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:29.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:34.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:34.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:39.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:39.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:44.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:44.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:49.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:49.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:54.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:54.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20148 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:54:56.884 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:54:56.885 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:56.889 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:54:56.889 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:54:59.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:54:59.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:04.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:04.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:09.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:09.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:14.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:14.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:19.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:19.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:24.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:24.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:27.116 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:55:27.117 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:27.120 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:55:27.121 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:29.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:29.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:34.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:34.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:39.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:39.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:44.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:44.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20333 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:49.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:49.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:54.166 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:54.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:55:57.349 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:55:57.349 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:57.353 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:55:57.353 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:55:59.169 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:55:59.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:04.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:04.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:09.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:09.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:14.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:14.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:19.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:19.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:24.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:24.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:27.479 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:56:27.479 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:27.485 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:56:27.485 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:29.182 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:29.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:34.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:34.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:39.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:39.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:44.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:44.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:49.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:49.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:54.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:54.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:56:57.528 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:56:57.528 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:57.533 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:56:57.533 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:56:59.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:56:59.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:04.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:04.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:09.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:09.195+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:14.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:14.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:19.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:19.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:24.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:24.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:27.659 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:57:27.660 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:27.663 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:57:27.663 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:29.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:29.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:34.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:34.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:39.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:39.210+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20757 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:44.218 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:44.210+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:49.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:49.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:54.223 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:54.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:57:57.732 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:57:57.733 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:57.736 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:57:57.736 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:57:59.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:57:59.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:04.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:04.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:09.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:09.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:14.232 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:14.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:19.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:19.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:24.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:24.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20924 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:27.776 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:58:27.776 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:27.804 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:58:27.804 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:29.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:29.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20943 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:34.242 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:34.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:39.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:39.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:44.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:44.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20998 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:49.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:49.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:54.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:54.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:58:57.951 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:58:57.951 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:57.955 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:58:57.955 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:58:59.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:58:59.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:04.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:04.253+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:09.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:09.253+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:14.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:14.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:19.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:19.261+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:24.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:24.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:28.078 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:59:28.078 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:28.082 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:59:28.082 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:29.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:29.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:34.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:34.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:39.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:39.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:44.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:44.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:49.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:49.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:54.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:54.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T05:59:58.417 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T05:59:58.417 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:58.421 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T05:59:58.421 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T05:59:59.285 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T05:59:59.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:04.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:04.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:09.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:09.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:14.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:14.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:19.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:19.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21348 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:24.297 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:24.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:28.531 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:00:28.531 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:28.536 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:00:28.536 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:29.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:29.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:34.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:34.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:39.305 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:39.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:44.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:44.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:49.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:49.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:54.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:54.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:00:58.678 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:00:58.678 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:58.682 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:00:58.682 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:00:59.315 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:00:59.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:04.318 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:04.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:09.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:09.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:14.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:14.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:19.328 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:19.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:24.327 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:24.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21589 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:28.809 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:01:28.809 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:28.813 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:01:28.813 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:29.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:29.322+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:34.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:34.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:39.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:39.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21643 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:44.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:44.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:49.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:49.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:54.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:54.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:01:58.922 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:01:58.923 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:58.927 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:01:58.927 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:01:59.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:01:59.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:04.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:04.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:09.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:09.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:14.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:14.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:19.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:19.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:24.357 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:24.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:29.057 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:02:29.057 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:29.061 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:02:29.061 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:29.359 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:29.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:34.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:34.357+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:39.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:39.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:44.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:44.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:49.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:49.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:54.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:54.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:02:59.174 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:02:59.174 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:59.178 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:02:59.178 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:02:59.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:02:59.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:04.376 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:04.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:09.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:09.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:14.382 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:14.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:19.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:19.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:24.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:24.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:29.308 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:03:29.309 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:29.313 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:03:29.313 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:29.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:29.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:34.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:34.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:39.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:39.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:44.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:44.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:49.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:49.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:54.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:54.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:59.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:03:59.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:03:59.455 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:03:59.456 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:03:59.460 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:03:59.460 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:04.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:04.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:09.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:09.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:14.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:14.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:19.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:19.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:24.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:24.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22253 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:29.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:29.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:29.560 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:04:29.561 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:29.564 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:04:29.564 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:34.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:34.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:39.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:39.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:44.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:44.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:49.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:49.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:54.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:54.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:59.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:04:59.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:04:59.825 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:04:59.826 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:04:59.829 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:04:59.830 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:04.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:04.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:09.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:09.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:14.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:14.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:19.444 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:19.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:24.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:24.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:29.450 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:29.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:29.941 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:05:29.941 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:29.945 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:05:29.945 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:05:34.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:34.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:39.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:39.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:44.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:44.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:49.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:49.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:54.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:54.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:05:59.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:05:59.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:00.308 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:06:00.308 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:00.312 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:06:00.312 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:04.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:04.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:09.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:09.465+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:14.474 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:14.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:19.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:19.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:24.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:24.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:29.482 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:29.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:30.421 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:06:30.421 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:30.425 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:06:30.425 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:06:34.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:34.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22731 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:39.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:39.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:44.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:44.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:49.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:49.489+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:54.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:54.489+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:06:59.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:06:59.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:00.552 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:07:00.552 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:00.556 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:07:00.556 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:04.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:04.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:09.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:09.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:14.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:14.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:19.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:19.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:24.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:24.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:29.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:29.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:30.664 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:07:30.665 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:30.668 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:07:30.669 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:07:34.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:34.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:39.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:39.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:44.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:44.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:49.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:49.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:54.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:54.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:07:59.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:07:59.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23200 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:00.777 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:08:00.778 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:00.781 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:08:00.781 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:04.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:04.527+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:09.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:09.527+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:14.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:14.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:19.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:19.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:24.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:24.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:29.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:29.539+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:30.824 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:08:30.825 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:30.837 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:08:30.837 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:08:34.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:34.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:39.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:39.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:44.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:44.547+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:49.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:49.546+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:54.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:54.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:08:59.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:08:59.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:00.939 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:09:00.939 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:00.943 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:09:00.943 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:04.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:04.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:09.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:09.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:14.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:14.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:19.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:19.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:24.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:24.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:29.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:29.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23748 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:31.080 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:09:31.081 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:31.086 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:09:31.086 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:09:34.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:34.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:39.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:39.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:44.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:44.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:49.587 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:49.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:54.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:54.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:09:59.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:09:59.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:01.126 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:10:01.127 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:01.133 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:10:01.133 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:04.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:04.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:09.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:09.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:15.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:14.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:19.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:19.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:24.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:24.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:29.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:29.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:31.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:10:31.319 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:31.322 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:10:31.323 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:10:34.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:34.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:39.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:39.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:44.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:44.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:49.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:49.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:54.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:54.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:10:59.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:10:59.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:01.633 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:11:01.633 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:01.637 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:11:01.637 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:04.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:04.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:09.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:09.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:14.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:14.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:19.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:19.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24410 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:24.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:24.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:29.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:29.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:31.759 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:11:31.759 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:31.763 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:11:31.763 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:11:34.645 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:34.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:39.647 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:39.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:44.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:44.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:49.653 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:49.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:54.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:54.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:11:59.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:11:59.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:01.864 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:12:01.864 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:01.868 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:12:01.868 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:04.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:04.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:09.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:09.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:14.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:14.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24741 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:19.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:19.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:24.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:24.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:29.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:29.667+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:31.994 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:12:31.994 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:31.998 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:12:31.998 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:12:34.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:34.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:39.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:39.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:44.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:44.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:49.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:49.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:54.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:54.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:12:59.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:12:59.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:02.115 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:13:02.115 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:02.121 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:13:02.121 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:04.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:04.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:09.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:09.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:14.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:14.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:19.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:19.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:24.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:24.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:29.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:29.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:32.214 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:13:32.215 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:32.218 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:13:32.219 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:13:34.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:34.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:39.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:39.705+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:44.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:44.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:49.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:49.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:54.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:54.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:13:59.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:13:59.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25366 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:02.327 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:14:02.327 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:02.330 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:14:02.330 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:04.726 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:04.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:09.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:09.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:14.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:14.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:19.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:19.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:24.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:24.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:29.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:29.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:32.473 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:14:32.474 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:32.477 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:14:32.477 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:14:34.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:34.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:39.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:39.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:44.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:44.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:49.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:49.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:54.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:54.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:14:59.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:14:59.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:02.524 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:15:02.524 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:02.528 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:15:02.528 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:04.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:04.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:09.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:09.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:14.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:14.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:19.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:19.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:24.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:24.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:29.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:29.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:32.642 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:15:32.642 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:32.646 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:15:32.646 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:15:34.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:34.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:39.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:39.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:44.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:44.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:49.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:49.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:54.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:54.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:15:59.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:15:59.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:03.006 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:16:03.007 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:03.010 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:16:03.010 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:04.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:04.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:09.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:09.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:14.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:14.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:19.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:19.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:24.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:24.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:29.806 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:29.798+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:33.062 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:16:33.062 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:33.066 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:16:33.066 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:16:34.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:34.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:39.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:39.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:44.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:44.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26308 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:49.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:49.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:54.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:54.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26364 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:16:59.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:16:59.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:03.192 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:17:03.192 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:03.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:17:03.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:04.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:04.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26421 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:09.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:09.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:14.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:14.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:19.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:19.826+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:24.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:24.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:29.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:29.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:33.248 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:17:33.248 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:33.252 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:17:33.252 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:17:34.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:34.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:39.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:39.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:44.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:44.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:49.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:49.845+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:54.854 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:54.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:17:59.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:17:59.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:03.587 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:18:03.587 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:03.591 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:18:03.591 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:04.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:04.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26763 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:09.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:09.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:14.865 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:14.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:19.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:19.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:24.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:24.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:29.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:29.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:33.701 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:18:33.701 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:33.705 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:18:33.705 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:18:34.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:34.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:39.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:39.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:44.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:44.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:49.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:49.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:54.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:54.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:18:59.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:18:59.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:03.781 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:19:03.781 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:03.785 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:19:03.785 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:04.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:04.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:09.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:09.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:14.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:14.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:19.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:19.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:24.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:24.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:29.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:29.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:33.928 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:19:33.928 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:33.932 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:19:33.932 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:19:34.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:34.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:39.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:39.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:44.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:44.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:49.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:49.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:54.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:54.915+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:19:59.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:19:59.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:04.041 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:20:04.041 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:04.045 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:20:04.045 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:04.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:04.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:09.931 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:09.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:14.934 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:14.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:19.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:19.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:24.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:24.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:29.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:29.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:34.172 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:20:34.172 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:34.175 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:20:34.176 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:20:34.946 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:34.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:39.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:39.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:44.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:44.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:49.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:49.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:54.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:54.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:20:59.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:20:59.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27759 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:04.312 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:21:04.313 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:04.317 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:21:04.317 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:04.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:04.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27787 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:09.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:09.957+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27815 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:14.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:14.961+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:19.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:19.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:24.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:24.969+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:29.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:29.973+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:34.425 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:21:34.425 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:34.429 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:21:34.430 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:21:34.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:34.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:39.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:39.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:44.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:44.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28015 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:49.990 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:49.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:54.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:54.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:21:59.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:21:59.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:04.540 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:22:04.540 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:04.544 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:22:04.544 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:04.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:04.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28129 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:10.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:09.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:15.005 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:15.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:20.008 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:20.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:25.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:25.004+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:30.014 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:30.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:34.662 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:22:34.662 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:34.666 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:22:34.666 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:22:35.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:35.012+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:40.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:40.012+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:45.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:45.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28357 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:50.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:50.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:22:55.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:22:55.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:00.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:00.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:04.871 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:23:04.871 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:04.875 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:23:04.875 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:05.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:05.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:10.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:10.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:15.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:15.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:20.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:20.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:25.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:25.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:30.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:30.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28612 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:34.912 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:23:34.912 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:34.916 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:23:34.916 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:23:35.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:35.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:40.055 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:40.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:45.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:45.051+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:50.061 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:50.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28726 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:23:55.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:23:55.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:00.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:00.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:05.034 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:24:05.035 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:05.039 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:24:05.039 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:05.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:05.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:10.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:10.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:15.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:15.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:20.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:20.070+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:25.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:25.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:30.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:30.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:35.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:35.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:35.139 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:24:35.139 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:35.143 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:24:35.143 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:24:40.091 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:40.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:45.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:45.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:50.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:50.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:24:55.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:24:55.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29095 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:00.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:00.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:05.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:05.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:05.239 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:25:05.239 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:05.249 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:25:05.250 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:10.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:10.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:15.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:15.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:20.115 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:20.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:25.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:25.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:30.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:30.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:35.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:35.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:35.305 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:25:35.305 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:35.309 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:25:35.309 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:25:40.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:40.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:45.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:45.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:50.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:50.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29409 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:25:55.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:25:55.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:00.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:00.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:05.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:05.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:05.369 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:26:05.370 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:05.373 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:26:05.373 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:10.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:10.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:15.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:15.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:20.151 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:20.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:25.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:25.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:30.158 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:30.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:35.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:35.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29664 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:35.512 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:26:35.513 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:35.516 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:26:35.516 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:26:40.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:40.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:45.167 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:45.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29722 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:50.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:50.163+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29750 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:26:55.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:26:55.167+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:00.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:00.167+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:05.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:05.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:05.622 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:27:05.622 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:05.626 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:27:05.626 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:10.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:10.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:15.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:15.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:20.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:20.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:25.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:25.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:30.196 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:30.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:35.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:35.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:35.744 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:27:35.744 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:35.748 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:27:35.748 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:27:40.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:40.195+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:45.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:45.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:50.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:50.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:27:55.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:27:55.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:00.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:00.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:05.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:05.210+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:05.866 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:28:05.866 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:05.870 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:28:05.870 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:10.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:10.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:15.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:15.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:20.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:20.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:25.232 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:25.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:30.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:30.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:35.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:35.230+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:35.989 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:28:35.989 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:35.993 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:28:35.993 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:28:40.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:40.234+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:45.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:45.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:50.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:50.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:28:55.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:28:55.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:00.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:00.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30491 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:05.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:05.253+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:06.109 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:29:06.109 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:06.113 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:29:06.113 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:10.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:10.253+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:15.263 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:15.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30575 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:20.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:20.261+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:25.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:25.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:30.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:30.269+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:35.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:35.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:36.239 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:29:36.239 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:36.243 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:29:36.243 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:29:40.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:40.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:45.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:45.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:50.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:50.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:29:55.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:29:55.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:00.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:00.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:05.297 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:05.292+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:06.354 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:30:06.355 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:06.359 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:30:06.359 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:10.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:10.292+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:15.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:15.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:20.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:20.300+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:25.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:25.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30974 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:30.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:30.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:35.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:35.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:36.486 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:30:36.487 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:36.490 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:30:36.490 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:30:40.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:40.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:45.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:45.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:50.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:50.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:30:55.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:30:55.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:00.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:00.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:05.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:05.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:06.557 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:31:06.557 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:06.560 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:31:06.561 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:10.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:10.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:15.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:15.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:20.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:20.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:25.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:25.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:30.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:30.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:35.355 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:35.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31371 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:36.679 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:31:36.679 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:36.683 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:31:36.683 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:31:40.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:40.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:45.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:45.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:50.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:50.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:31:55.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:31:55.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:00.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:00.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:05.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:05.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:06.843 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:32:06.843 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:06.847 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:32:06.847 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:10.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:10.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:15.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:15.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:20.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:20.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:25.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:25.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:30.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:30.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:35.394 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:35.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:36.970 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:32:36.970 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:36.974 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:32:36.974 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:32:40.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:40.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:45.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:45.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31771 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:50.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:50.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:32:55.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:32:55.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31827 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:00.410 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:00.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:05.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:05.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31885 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:07.094 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:33:07.095 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:07.098 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:33:07.099 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:10.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:10.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:15.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:15.413+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:20.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:20.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:25.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:25.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:30.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:30.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32026 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:35.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:35.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:37.217 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:33:37.217 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:37.222 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:33:37.222 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:33:40.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:40.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:45.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:45.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:50.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:50.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:33:55.445 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:33:55.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:00.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:00.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:05.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:05.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32226 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:07.324 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:34:07.324 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:07.337 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:34:07.342 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:10.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:10.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:15.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:15.452+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:20.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:20.456+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:25.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:25.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:30.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:30.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:35.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:35.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:37.461 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:34:37.462 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:37.465 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:34:37.466 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:34:40.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:40.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:45.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:45.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:50.482 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:50.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:34:55.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:34:55.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:00.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:00.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:05.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:05.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:07.580 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:35:07.580 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:07.584 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:35:07.584 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:10.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:10.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:15.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:15.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:20.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:20.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:25.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:25.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:30.508 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:30.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:35.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:35.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:37.702 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:35:37.703 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:37.706 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:35:37.706 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:35:40.514 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:40.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:45.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:45.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:50.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:50.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:35:55.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:35:55.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:00.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:00.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:05.532 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:05.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:07.776 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:36:07.777 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:07.781 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:36:07.781 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:10.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:10.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:15.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:15.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:20.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:20.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:25.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:25.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:30.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:30.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:35.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:35.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:37.900 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:36:37.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:37.904 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:36:37.904 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:36:40.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:40.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:45.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:45.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:50.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:50.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:36:55.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:36:55.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:00.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:00.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:05.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:05.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:08.022 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:37:08.022 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:08.029 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:37:08.029 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:10.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:10.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:15.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:15.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:20.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:20.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:25.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:25.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:30.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:30.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:35.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:35.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:38.140 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:37:38.140 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:38.143 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:37:38.144 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:37:40.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:40.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:45.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:45.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:50.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:50.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33489 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:37:55.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:37:55.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:00.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:00.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:05.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:05.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:08.253 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:38:08.253 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:08.257 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:38:08.257 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:10.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:10.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:15.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:15.611+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:20.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:20.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:25.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:25.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:30.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:30.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33709 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:35.632 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:35.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:38.368 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:38:38.368 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:38.372 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:38:38.372 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:38:40.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:40.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:45.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:45.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33790 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:50.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:50.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:38:55.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:38:55.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:00.649 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:00.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:05.653 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:05.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:08.474 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:39:08.474 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:08.478 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:39:08.478 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:10.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:10.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:15.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:15.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:20.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:20.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:25.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:25.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:30.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:30.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:35.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:35.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:38.598 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:39:38.598 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:38.602 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:39:38.603 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:39:40.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:40.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:45.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:45.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:50.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:50.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:39:55.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:39:55.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:00.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:00.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34210 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:05.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:05.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:08.695 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:40:08.696 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:08.699 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:40:08.699 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:10.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:10.689+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:15.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:15.693+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:20.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:20.697+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:25.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:25.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34348 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:30.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:30.705+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:35.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:35.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:38.810 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:40:38.810 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:38.813 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:40:38.814 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:40:40.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:40.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:45.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:45.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:50.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:50.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:40:55.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:40:55.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34513 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:00.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:00.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:05.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:05.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:08.884 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:41:08.884 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:08.887 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:41:08.888 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:10.738 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:10.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:15.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:15.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:20.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:20.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:25.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:25.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:30.752 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:30.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:35.756 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:35.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:39.015 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:41:39.015 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:39.019 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:41:39.019 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:41:40.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:40.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:45.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:45.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:50.766 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:50.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:41:55.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:41:55.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34851 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:00.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:00.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:05.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:05.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:09.096 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:42:09.096 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:09.101 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:42:09.101 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:10.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:10.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:15.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:15.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:20.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:20.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:25.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:25.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:30.798 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:30.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:35.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:35.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:39.211 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:42:39.211 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:39.215 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:42:39.215 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:42:40.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:40.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:45.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:45.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:50.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:50.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:42:55.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:42:55.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35203 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:00.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:00.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:05.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:05.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:09.300 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:43:09.300 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:09.304 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:43:09.304 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:10.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:10.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:15.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:15.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:20.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:20.826+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:25.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:25.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:30.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:30.834+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:35.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:35.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:39.415 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:43:39.415 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:39.419 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:43:39.419 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:43:40.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:40.842+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:45.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:45.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35497 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:50.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:50.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:43:55.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:43:55.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:00.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:00.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:05.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:05.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:09.531 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:44:09.531 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:09.535 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:44:09.535 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:10.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:10.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:15.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:15.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:20.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:20.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:25.881 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:25.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35731 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:30.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:30.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35763 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:35.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:35.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:39.604 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:44:39.604 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:39.608 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:44:39.608 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:44:40.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:40.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35823 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:45.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:45.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:50.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:50.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35881 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:44:55.901 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:44:55.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:00.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:00.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:05.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:05.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:09.734 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:45:09.734 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:09.738 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:45:09.738 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:10.912 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:10.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:15.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:15.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:20.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:20.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:25.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:25.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:30.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:30.920+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:35.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:35.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:39.863 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:45:39.863 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:39.867 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:45:39.867 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:45:40.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:40.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:45.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:45.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:50.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:50.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:45:55.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:45:55.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:00.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:00.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:05.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:05.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:09.939 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:46:09.939 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:09.945 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:46:09.945 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:10.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:10.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:15.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:15.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:20.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:20.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:25.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:25.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:30.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:30.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:35.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:35.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:40.079 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:46:40.079 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:40.083 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:46:40.083 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:46:40.976 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:40.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:45.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:45.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:50.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:50.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:46:55.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:46:55.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:00.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:00.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:05.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:05.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:10.162 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:47:10.162 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:10.166 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:47:10.166 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:10.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:10.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:16.003 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:15.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:21.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:20.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:26.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:26.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:31.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:31.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:36.019 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:36.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:40.286 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:47:40.286 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:40.290 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:47:40.291 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:47:41.023 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:41.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:46.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:46.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:51.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:51.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:47:56.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:47:56.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:01.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:01.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:06.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:06.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:10.334 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:48:10.334 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:10.338 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:48:10.338 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:11.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:11.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:16.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:16.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:21.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:21.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:26.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:26.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:31.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:31.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:36.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:36.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:40.466 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:48:40.466 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:40.470 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:48:40.470 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:48:41.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:41.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:46.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:46.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:51.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:51.068+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:48:56.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:48:56.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:01.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:01.076+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:06.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:06.080+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:10.573 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:49:10.573 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:10.576 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:49:10.577 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:11.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:11.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:16.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:16.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:21.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:21.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:26.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:26.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:31.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:31.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:36.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:36.100+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:40.687 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:49:40.687 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:40.691 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:49:40.691 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:49:41.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:41.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:46.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:46.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:51.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:51.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:49:56.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:49:56.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37676 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:01.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:01.119+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:06.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:06.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:10.794 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:50:10.794 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:10.799 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:50:10.799 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:11.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:11.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:16.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:16.127+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:21.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:21.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:26.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:26.135+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:31.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:31.139+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37881 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:36.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:36.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:40.916 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:50:40.916 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:40.920 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:50:40.920 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:50:41.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:41.146+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:46.158 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:46.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:51.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:51.154+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:50:56.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:50:56.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:01.169 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:01.162+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:06.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:06.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:10.990 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:51:10.990 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:10.994 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:51:10.994 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:11.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:11.170+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38113 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:16.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:16.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:21.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:21.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:26.188 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:26.178+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:31.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:31.182+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:36.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:36.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:41.130 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:51:41.130 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:41.134 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:51:41.134 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:51:41.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:41.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:46.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:46.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:51.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:51.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:51:56.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:51:56.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:01.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:01.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:06.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:06.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:11.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:11.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:11.244 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:52:11.244 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:11.248 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:52:11.248 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:16.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:16.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:21.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:21.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:26.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:26.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:31.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:31.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:36.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:36.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38604 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:41.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:41.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:41.349 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:52:41.350 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:41.353 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:52:41.354 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:52:46.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:46.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:51.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:51.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:52:56.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:52:56.252+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:01.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:01.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:06.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:06.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:11.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:11.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:11.481 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:53:11.481 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:11.485 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:53:11.485 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:16.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:16.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:21.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:21.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:26.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:26.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:31.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:31.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:36.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:36.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:41.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:41.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:41.589 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:53:41.589 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:41.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:53:41.593 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:53:46.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:46.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:51.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:51.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:53:56.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:53:56.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:01.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:01.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:06.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:06.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:11.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:11.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:11.705 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:54:11.705 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:11.708 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:54:11.709 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:16.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:16.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:21.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:21.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:26.325 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:26.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:31.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:31.322+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:36.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:36.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:41.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:41.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:41.817 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:54:41.817 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:41.821 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:54:41.821 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:54:46.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:46.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:51.343 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:51.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:54:56.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:54:56.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:01.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:01.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:06.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:06.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39472 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:11.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:11.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:11.955 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:55:11.955 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:11.963 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:55:11.964 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:16.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:16.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:21.366 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:21.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:26.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:26.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39587 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:31.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:31.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:36.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:36.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:41.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:41.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:42.076 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:55:42.077 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:42.081 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:55:42.081 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:55:46.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:46.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:51.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:51.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:55:56.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:55:56.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:01.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:01.393+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:06.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:06.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:11.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:11.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:12.147 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:56:12.147 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:12.151 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:56:12.151 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:16.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:16.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39878 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:21.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:21.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:26.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:26.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:31.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:31.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:36.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:36.416+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:41.426 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:41.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:42.277 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:56:42.278 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:42.281 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:56:42.281 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:56:46.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:46.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:51.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:51.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:56:56.439 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:56:56.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:01.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:01.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:06.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:06.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:11.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:11.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40197 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:12.385 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:57:12.385 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:12.389 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:57:12.389 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:16.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:16.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:21.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:21.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:26.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:26.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:31.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:31.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:36.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:36.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:41.473 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:41.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:42.507 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:57:42.507 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:42.511 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:57:42.511 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:57:46.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:46.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:51.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:51.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:57:56.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:57:56.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:01.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:01.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:06.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:06.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:11.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:11.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:12.621 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:58:12.621 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:12.643 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:58:12.643 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:16.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:16.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:21.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:21.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40604 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:26.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:26.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:31.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:31.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:36.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:36.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:41.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:41.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:42.754 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:58:42.754 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:42.759 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:58:42.759 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:58:46.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:46.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:51.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:51.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:58:56.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:58:56.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:01.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:01.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:06.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:06.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:11.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:11.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:12.857 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:59:12.857 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:12.860 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:59:12.861 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:16.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:16.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:21.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:21.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:26.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:26.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:31.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:31.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:36.562 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:36.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:41.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:41.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:42.991 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T06:59:42.991 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:42.995 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T06:59:42.995 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T06:59:46.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:46.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:51.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:51.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T06:59:56.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T06:59:56.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41148 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:01.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:01.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:06.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:06.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41203 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:11.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:11.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:13.106 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:00:13.106 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:13.110 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:00:13.110 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:16.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:16.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:21.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:21.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:26.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:26.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:31.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:31.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:36.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:36.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:41.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:41.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41409 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:43.437 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:00:43.437 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:43.440 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:00:43.441 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:00:46.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:46.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:51.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:51.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41465 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:00:56.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:00:56.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:01.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:01.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:06.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:06.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:11.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:11.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:13.794 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:01:13.794 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:13.798 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:01:13.798 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:16.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:16.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41609 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:21.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:21.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:26.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:26.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41666 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:31.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:31.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:36.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:36.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:41.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:41.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:43.876 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:01:43.876 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:43.881 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:01:43.881 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:01:46.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:46.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:51.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:51.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:01:56.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:01:56.667+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:01.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:01.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:06.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:06.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:11.686 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:11.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:13.998 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:02:13.998 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:14.002 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:02:14.002 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:16.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:16.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:21.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:21.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:26.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:26.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:31.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:31.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:36.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:36.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:41.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:41.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42103 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:44.138 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:02:44.138 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:44.145 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:02:44.146 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:02:46.714 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:46.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42129 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:51.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:51.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:02:56.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:02:56.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:01.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:01.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:06.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:06.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:11.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:11.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:14.261 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:03:14.262 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:14.273 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:03:14.273 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:16.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:16.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:21.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:21.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:26.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:26.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:31.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:31.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:36.756 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:36.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42420 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:41.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:41.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:44.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:03:44.318 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:44.322 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:03:44.323 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:03:46.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:46.761+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:51.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:51.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:03:56.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:03:56.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:01.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:01.772+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:06.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:06.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42592 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:11.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:11.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:14.375 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:04:14.375 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:14.379 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:04:14.379 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:16.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:16.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:21.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:21.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:26.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:26.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.793168+0000) 2019-10-13T07:04:28.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:26.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:28.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:26.788+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.793666+0000) 2019-10-13T07:04:28.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:26.792+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.793993+0000) 2019-10-13T07:04:28.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:26.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.796483+0000) 2019-10-13T07:04:28.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:26.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.802988+0000) 2019-10-13T07:04:28.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:26.792+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.794926+0000) 2019-10-13T07:04:28.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:27.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:27.057957+0000) 2019-10-13T07:04:28.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:26.792+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.795183+0000) 2019-10-13T07:04:28.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:26.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:26.796042+0000) 2019-10-13T07:04:28.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:27.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:27.471415+0000) 2019-10-13T07:04:28.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:27.472+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:27.474280+0000) 2019-10-13T07:04:30.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:30.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:30.600100+0000) 2019-10-13T07:04:30.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:30.696+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:30.699338+0000) 2019-10-13T07:04:31.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:31.492+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.496506+0000) 2019-10-13T07:04:31.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:31.493+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.496356+0000) 2019-10-13T07:04:31.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:31.553+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.557317+0000) 2019-10-13T07:04:31.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:31.792+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.796688+0000) 2019-10-13T07:04:31.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:31.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:31.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:31.792+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.797535+0000) 2019-10-13T07:04:31.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:31.796+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.797772+0000) 2019-10-13T07:04:31.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:31.796+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.798235+0000) 2019-10-13T07:04:31.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:31.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.799827+0000) 2019-10-13T07:04:31.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:31.796+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.801141+0000) 2019-10-13T07:04:31.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:31.796+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.799204+0000) 2019-10-13T07:04:31.807 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:31.801+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:31.804805+0000) 2019-10-13T07:04:36.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:36.797+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.801204+0000) 2019-10-13T07:04:36.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:36.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:36.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:36.800+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.802309+0000) 2019-10-13T07:04:36.805 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:36.800+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.803167+0000) 2019-10-13T07:04:36.805 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:36.800+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.802949+0000) 2019-10-13T07:04:36.806 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:36.800+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.803483+0000) 2019-10-13T07:04:36.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:36.800+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.803784+0000) 2019-10-13T07:04:36.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:36.800+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.804322+0000) 2019-10-13T07:04:36.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:36.805+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:36.809822+0000) 2019-10-13T07:04:37.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:37.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:37.058135+0000) 2019-10-13T07:04:37.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:37.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:37.471589+0000) 2019-10-13T07:04:37.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:37.472+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:37.474454+0000) 2019-10-13T07:04:40.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:40.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:40.600298+0000) 2019-10-13T07:04:40.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:40.696+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:40.699556+0000) 2019-10-13T07:04:41.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:41.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.496706+0000) 2019-10-13T07:04:41.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:41.493+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.496558+0000) 2019-10-13T07:04:41.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:41.561+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.563695+0000) 2019-10-13T07:04:41.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:41.800+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.805038+0000) 2019-10-13T07:04:41.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:41.800+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:41.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:41.801+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.805354+0000) 2019-10-13T07:04:41.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:41.804+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.806620+0000) 2019-10-13T07:04:41.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:41.804+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.806749+0000) 2019-10-13T07:04:41.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:41.804+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.807052+0000) 2019-10-13T07:04:41.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:41.804+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.807866+0000) 2019-10-13T07:04:41.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:41.804+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.807648+0000) 2019-10-13T07:04:41.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:41.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:41.814149+0000) 2019-10-13T07:04:44.430 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:04:44.431 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:44.434 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:04:44.434 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:04:46.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:46.804+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.809098+0000) 2019-10-13T07:04:46.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:46.804+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.809495+0000) 2019-10-13T07:04:46.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:46.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:46.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:46.808+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.810410+0000) 2019-10-13T07:04:46.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:46.805+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.810280+0000) 2019-10-13T07:04:46.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:46.808+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.811254+0000) 2019-10-13T07:04:46.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:46.808+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.811490+0000) 2019-10-13T07:04:46.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:46.808+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.812388+0000) 2019-10-13T07:04:46.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:46.813+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:46.817472+0000) 2019-10-13T07:04:47.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:47.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:47.058351+0000) 2019-10-13T07:04:47.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:47.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:47.471766+0000) 2019-10-13T07:04:47.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:47.472+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:47.474647+0000) 2019-10-13T07:04:50.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:50.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:50.600531+0000) 2019-10-13T07:04:50.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:50.696+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:50.699754+0000) 2019-10-13T07:04:51.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:51.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.496903+0000) 2019-10-13T07:04:51.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:51.493+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.496784+0000) 2019-10-13T07:04:51.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:51.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.570067+0000) 2019-10-13T07:04:51.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:51.808+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.813299+0000) 2019-10-13T07:04:51.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:51.808+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42855 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:51.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:51.812+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.814165+0000) 2019-10-13T07:04:51.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:51.812+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.814772+0000) 2019-10-13T07:04:51.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:51.809+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.814977+0000) 2019-10-13T07:04:51.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:51.812+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.815060+0000) 2019-10-13T07:04:51.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:51.812+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.815949+0000) 2019-10-13T07:04:51.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:51.812+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.816836+0000) 2019-10-13T07:04:51.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:51.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:51.822093+0000) 2019-10-13T07:04:56.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:04:56.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:04:56.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:04:56.812+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.817411+0000) 2019-10-13T07:04:56.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:56.815+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.818360+0000) 2019-10-13T07:04:56.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:04:56.815+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.818748+0000) 2019-10-13T07:04:56.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:04:56.815+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.818933+0000) 2019-10-13T07:04:56.822 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:56.815+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.819513+0000) 2019-10-13T07:04:56.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:04:56.815+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.820009+0000) 2019-10-13T07:04:56.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:56.815+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.819611+0000) 2019-10-13T07:04:56.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:04:56.824+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:56.828486+0000) 2019-10-13T07:04:57.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:04:57.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:57.058544+0000) 2019-10-13T07:04:57.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:04:57.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:57.471985+0000) 2019-10-13T07:05:00.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:04:57.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:04:57.474826+0000) 2019-10-13T07:05:00.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:00.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.600723+0000) 2019-10-13T07:05:00.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:00.695+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:00.699959+0000) 2019-10-13T07:05:01.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:01.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.497107+0000) 2019-10-13T07:05:01.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:01.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.497015+0000) 2019-10-13T07:05:01.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:01.572+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.576458+0000) 2019-10-13T07:05:01.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:01.815+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.821143+0000) 2019-10-13T07:05:01.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:01.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:01.824 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:01.819+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.822689+0000) 2019-10-13T07:05:01.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:01.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.823261+0000) 2019-10-13T07:05:01.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:01.819+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.823462+0000) 2019-10-13T07:05:01.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:01.816+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.823312+0000) 2019-10-13T07:05:01.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:01.819+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.824833+0000) 2019-10-13T07:05:01.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:01.823+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.827596+0000) 2019-10-13T07:05:01.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:01.824+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:01.831375+0000) 2019-10-13T07:05:06.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:06.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:06.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:06.823+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.826534+0000) 2019-10-13T07:05:06.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:06.823+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.827160+0000) 2019-10-13T07:05:06.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:06.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.827595+0000) 2019-10-13T07:05:06.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:06.824+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.828371+0000) 2019-10-13T07:05:06.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:06.823+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.828679+0000) 2019-10-13T07:05:06.841 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:06.823+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.829565+0000) 2019-10-13T07:05:06.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:06.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.830164+0000) 2019-10-13T07:05:06.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:06.832+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:06.836832+0000) 2019-10-13T07:05:07.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:07.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:07.058757+0000) 2019-10-13T07:05:07.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:07.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:07.472176+0000) 2019-10-13T07:05:07.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:07.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:07.475023+0000) 2019-10-13T07:05:10.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:10.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.600914+0000) 2019-10-13T07:05:10.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:10.695+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:10.700151+0000) 2019-10-13T07:05:11.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:11.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.497329+0000) 2019-10-13T07:05:11.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:11.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.497214+0000) 2019-10-13T07:05:11.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:11.576+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.583578+0000) 2019-10-13T07:05:11.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:11.827+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.829917+0000) 2019-10-13T07:05:11.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:11.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:11.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:11.827+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.830730+0000) 2019-10-13T07:05:11.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:11.827+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.831177+0000) 2019-10-13T07:05:11.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:11.827+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.831806+0000) 2019-10-13T07:05:11.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:11.827+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.832472+0000) 2019-10-13T07:05:11.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:11.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.832781+0000) 2019-10-13T07:05:11.836 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:11.831+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.834887+0000) 2019-10-13T07:05:11.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:11.832+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:11.838445+0000) 2019-10-13T07:05:14.545 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:05:14.545 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:14.549 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:05:14.549 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:16.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:16.831+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.834291+0000) 2019-10-13T07:05:16.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:16.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:16.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:16.831+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.835123+0000) 2019-10-13T07:05:16.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:16.831+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.835602+0000) 2019-10-13T07:05:16.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:16.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.836261+0000) 2019-10-13T07:05:16.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:16.831+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.836856+0000) 2019-10-13T07:05:16.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:16.832+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.837162+0000) 2019-10-13T07:05:16.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:16.835+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.838346+0000) 2019-10-13T07:05:16.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:16.836+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:16.843238+0000) 2019-10-13T07:05:17.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:17.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:17.058943+0000) 2019-10-13T07:05:17.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:17.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:17.472352+0000) 2019-10-13T07:05:17.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:17.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:17.475227+0000) 2019-10-13T07:05:20.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:20.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.601144+0000) 2019-10-13T07:05:20.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:20.699+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:20.700349+0000) 2019-10-13T07:05:21.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:21.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.497556+0000) 2019-10-13T07:05:21.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:21.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.497445+0000) 2019-10-13T07:05:21.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:21.588+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.590021+0000) 2019-10-13T07:05:21.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:21.832+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.837134+0000) 2019-10-13T07:05:21.840 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:21.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:21.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:21.835+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.839044+0000) 2019-10-13T07:05:21.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:21.835+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.840074+0000) 2019-10-13T07:05:21.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:21.835+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.839622+0000) 2019-10-13T07:05:21.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:21.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.840637+0000) 2019-10-13T07:05:21.843 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:21.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.841031+0000) 2019-10-13T07:05:21.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:21.835+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.841415+0000) 2019-10-13T07:05:21.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:21.840+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:21.844204+0000) 2019-10-13T07:05:26.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:26.839+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.842362+0000) 2019-10-13T07:05:26.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:26.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:26.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:26.839+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.842951+0000) 2019-10-13T07:05:26.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:26.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.843343+0000) 2019-10-13T07:05:26.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:26.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.844878+0000) 2019-10-13T07:05:26.847 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:26.839+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.845206+0000) 2019-10-13T07:05:26.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:26.843+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.845918+0000) 2019-10-13T07:05:26.848 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:26.843+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.846848+0000) 2019-10-13T07:05:26.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:26.848+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:26.852443+0000) 2019-10-13T07:05:27.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:27.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:27.059146+0000) 2019-10-13T07:05:27.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:27.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:27.472532+0000) 2019-10-13T07:05:27.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:27.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:27.475399+0000) 2019-10-13T07:05:30.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:30.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.601357+0000) 2019-10-13T07:05:30.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:30.699+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:30.700500+0000) 2019-10-13T07:05:31.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:31.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.497791+0000) 2019-10-13T07:05:31.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:31.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.497643+0000) 2019-10-13T07:05:31.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:31.596+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.596510+0000) 2019-10-13T07:05:31.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:31.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.845936+0000) 2019-10-13T07:05:31.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:31.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:31.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:31.843+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.847095+0000) 2019-10-13T07:05:31.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:31.843+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.848238+0000) 2019-10-13T07:05:31.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:31.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.848020+0000) 2019-10-13T07:05:31.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:31.843+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.848922+0000) 2019-10-13T07:05:31.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:31.843+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.849219+0000) 2019-10-13T07:05:31.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:31.847+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.850073+0000) 2019-10-13T07:05:31.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:31.848+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:31.853684+0000) 2019-10-13T07:05:36.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:36.843+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.849565+0000) 2019-10-13T07:05:36.852 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:36.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.849824+0000) 2019-10-13T07:05:36.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:36.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:36.853 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:36.847+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.851439+0000) 2019-10-13T07:05:36.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:36.847+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.851947+0000) 2019-10-13T07:05:36.854 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:36.847+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.851660+0000) 2019-10-13T07:05:36.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:36.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.852181+0000) 2019-10-13T07:05:36.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:36.847+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.853057+0000) 2019-10-13T07:05:36.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:36.852+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:36.856924+0000) 2019-10-13T07:05:37.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:37.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:37.059332+0000) 2019-10-13T07:05:37.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:37.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:37.472715+0000) 2019-10-13T07:05:37.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:37.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:37.475601+0000) 2019-10-13T07:05:40.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:40.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.601588+0000) 2019-10-13T07:05:40.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:40.699+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:40.700706+0000) 2019-10-13T07:05:41.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:41.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.497987+0000) 2019-10-13T07:05:41.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:41.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.497873+0000) 2019-10-13T07:05:41.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:41.600+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.602913+0000) 2019-10-13T07:05:41.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:41.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.853972+0000) 2019-10-13T07:05:41.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:41.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:41.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:41.851+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.855497+0000) 2019-10-13T07:05:41.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:41.851+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.855851+0000) 2019-10-13T07:05:41.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:41.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.856177+0000) 2019-10-13T07:05:41.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:41.852+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.856188+0000) 2019-10-13T07:05:41.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:41.851+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.857003+0000) 2019-10-13T07:05:41.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:41.851+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.857266+0000) 2019-10-13T07:05:41.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:41.856+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:41.862026+0000) 2019-10-13T07:05:44.609 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:05:44.609 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:44.613 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:05:44.614 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:05:46.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:46.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43172 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:46.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:46.855+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.859928+0000) 2019-10-13T07:05:46.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:46.855+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.860310+0000) 2019-10-13T07:05:46.862 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:46.855+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.860778+0000) 2019-10-13T07:05:46.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:46.855+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.861616+0000) 2019-10-13T07:05:46.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:46.855+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.861447+0000) 2019-10-13T07:05:46.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:46.856+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.861912+0000) 2019-10-13T07:05:46.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:46.859+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.863888+0000) 2019-10-13T07:05:46.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:46.860+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:46.865279+0000) 2019-10-13T07:05:47.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:47.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:47.059522+0000) 2019-10-13T07:05:47.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:47.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:47.472899+0000) 2019-10-13T07:05:47.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:47.471+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:47.475808+0000) 2019-10-13T07:05:50.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:50.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.601788+0000) 2019-10-13T07:05:51.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:50.699+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:50.700951+0000) 2019-10-13T07:05:51.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:51.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.498215+0000) 2019-10-13T07:05:51.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:51.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.498065+0000) 2019-10-13T07:05:51.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:51.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.609452+0000) 2019-10-13T07:05:51.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:51.858+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.862668+0000) 2019-10-13T07:05:51.865 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:51.858+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43203 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:51.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:51.858+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.864294+0000) 2019-10-13T07:05:51.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:51.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.863935+0000) 2019-10-13T07:05:51.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:51.858+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.865217+0000) 2019-10-13T07:05:51.867 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:51.858+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.865370+0000) 2019-10-13T07:05:51.869 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:51.862+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.866749+0000) 2019-10-13T07:05:51.869 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:51.862+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.867340+0000) 2019-10-13T07:05:51.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:51.872+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:51.873450+0000) 2019-10-13T07:05:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 124 2019-10-13T07:05:56.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:05:56.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.866423+0000) 2019-10-13T07:05:56.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:56.862+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.867173+0000) 2019-10-13T07:05:56.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:05:56.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:05:56.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:56.862+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.867591+0000) 2019-10-13T07:05:56.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:05:56.862+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.867850+0000) 2019-10-13T07:05:56.871 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:05:56.862+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.868419+0000) 2019-10-13T07:05:56.872 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:56.866+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.870143+0000) 2019-10-13T07:05:56.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:05:56.866+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.871035+0000) 2019-10-13T07:05:56.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:05:56.876+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:56.877818+0000) 2019-10-13T07:05:57.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:05:57.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:57.059707+0000) 2019-10-13T07:05:57.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:05:57.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:57.473100+0000) 2019-10-13T07:05:57.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:05:57.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:05:57.475987+0000) 2019-10-13T07:06:00.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:00.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.602018+0000) 2019-10-13T07:06:00.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:00.698+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:00.701154+0000) 2019-10-13T07:06:01.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:01.494+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.498457+0000) 2019-10-13T07:06:01.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:01.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.498271+0000) 2019-10-13T07:06:01.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:01.611+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.615911+0000) 2019-10-13T07:06:01.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:01.866+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.870899+0000) 2019-10-13T07:06:01.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:01.867+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.871194+0000) 2019-10-13T07:06:01.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:01.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:01.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:01.866+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.872169+0000) 2019-10-13T07:06:01.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:01.866+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.872153+0000) 2019-10-13T07:06:01.876 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:01.866+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.872477+0000) 2019-10-13T07:06:01.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:01.866+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.873185+0000) 2019-10-13T07:06:01.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:01.870+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.876521+0000) 2019-10-13T07:06:01.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:01.879+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:01.883251+0000) 2019-10-13T07:06:06.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:06.870+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.875333+0000) 2019-10-13T07:06:06.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:06.870+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.875703+0000) 2019-10-13T07:06:06.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:06.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:06.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:06.870+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.877251+0000) 2019-10-13T07:06:06.880 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:06.874+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.877978+0000) 2019-10-13T07:06:06.885 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:06.874+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.880372+0000) 2019-10-13T07:06:06.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:06.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.880733+0000) 2019-10-13T07:06:06.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:06.875+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.879253+0000) 2019-10-13T07:06:06.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:06.879+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:06.882910+0000) 2019-10-13T07:06:07.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:07.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:07.059898+0000) 2019-10-13T07:06:07.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:07.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:07.473283+0000) 2019-10-13T07:06:07.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:07.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:07.476190+0000) 2019-10-13T07:06:10.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:10.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.602244+0000) 2019-10-13T07:06:10.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:10.698+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:10.701371+0000) 2019-10-13T07:06:11.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:11.494+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.498670+0000) 2019-10-13T07:06:11.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:11.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.498462+0000) 2019-10-13T07:06:11.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:11.619+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.622526+0000) 2019-10-13T07:06:11.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:11.874+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.879440+0000) 2019-10-13T07:06:11.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:11.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:11.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:11.874+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.880831+0000) 2019-10-13T07:06:11.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:11.874+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.881272+0000) 2019-10-13T07:06:11.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:11.874+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.881372+0000) 2019-10-13T07:06:11.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:11.879+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.881350+0000) 2019-10-13T07:06:11.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:11.878+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.882754+0000) 2019-10-13T07:06:11.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:11.878+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.883074+0000) 2019-10-13T07:06:11.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:11.883+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:11.887737+0000) 2019-10-13T07:06:14.931 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:06:14.931 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:14.936 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:06:14.936 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:16.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:16.879+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.883356+0000) 2019-10-13T07:06:16.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:16.878+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.883495+0000) 2019-10-13T07:06:16.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:16.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43346 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:16.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:16.878+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.884419+0000) 2019-10-13T07:06:16.890 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:16.878+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.885578+0000) 2019-10-13T07:06:16.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:16.882+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.886099+0000) 2019-10-13T07:06:16.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:16.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.886616+0000) 2019-10-13T07:06:16.891 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:16.882+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.887061+0000) 2019-10-13T07:06:16.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:16.887+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:16.890730+0000) 2019-10-13T07:06:17.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:17.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:17.060108+0000) 2019-10-13T07:06:17.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:17.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:17.473462+0000) 2019-10-13T07:06:17.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:17.470+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:17.476361+0000) 2019-10-13T07:06:20.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:20.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.602473+0000) 2019-10-13T07:06:20.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:20.698+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:20.701524+0000) 2019-10-13T07:06:21.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:21.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.498867+0000) 2019-10-13T07:06:21.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:21.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.498655+0000) 2019-10-13T07:06:21.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:21.627+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.629052+0000) 2019-10-13T07:06:21.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:21.882+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.887855+0000) 2019-10-13T07:06:21.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:21.882+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:21.891 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:21.882+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.888860+0000) 2019-10-13T07:06:21.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:21.882+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.889320+0000) 2019-10-13T07:06:21.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:21.887+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.889300+0000) 2019-10-13T07:06:21.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:21.886+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.890598+0000) 2019-10-13T07:06:21.893 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:21.886+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.890468+0000) 2019-10-13T07:06:21.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:21.886+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.890864+0000) 2019-10-13T07:06:21.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:21.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:21.897975+0000) 2019-10-13T07:06:26.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:26.887+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.890768+0000) 2019-10-13T07:06:26.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:26.886+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.891927+0000) 2019-10-13T07:06:26.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:26.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:26.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:26.886+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.892605+0000) 2019-10-13T07:06:26.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:26.886+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.893212+0000) 2019-10-13T07:06:26.895 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:26.886+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.893537+0000) 2019-10-13T07:06:26.896 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:26.890+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.894190+0000) 2019-10-13T07:06:26.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:26.890+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.894600+0000) 2019-10-13T07:06:26.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:26.899+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:26.902429+0000) 2019-10-13T07:06:27.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:27.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:27.060290+0000) 2019-10-13T07:06:27.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:27.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:27.473657+0000) 2019-10-13T07:06:27.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:27.470+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:27.476528+0000) 2019-10-13T07:06:30.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:30.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.602704+0000) 2019-10-13T07:06:30.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:30.698+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:30.701776+0000) 2019-10-13T07:06:31.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:31.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.499059+0000) 2019-10-13T07:06:31.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:31.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.498847+0000) 2019-10-13T07:06:31.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:31.631+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.635533+0000) 2019-10-13T07:06:31.898 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:31.890+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.895888+0000) 2019-10-13T07:06:31.898 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:31.890+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.896499+0000) 2019-10-13T07:06:31.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:31.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:31.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:31.895+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.897572+0000) 2019-10-13T07:06:31.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:31.894+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.898605+0000) 2019-10-13T07:06:31.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:31.894+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.898896+0000) 2019-10-13T07:06:31.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:31.894+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.899683+0000) 2019-10-13T07:06:31.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:31.894+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.898350+0000) 2019-10-13T07:06:31.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:31.899+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:31.903592+0000) 2019-10-13T07:06:36.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:36.894+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.901132+0000) 2019-10-13T07:06:36.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:36.894+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:36.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:36.898+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.901880+0000) 2019-10-13T07:06:36.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:36.898+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.902422+0000) 2019-10-13T07:06:36.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:36.898+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.902571+0000) 2019-10-13T07:06:36.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:36.898+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.903628+0000) 2019-10-13T07:06:36.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:36.903+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.904363+0000) 2019-10-13T07:06:36.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:36.898+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.903955+0000) 2019-10-13T07:06:36.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:36.907+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:36.909821+0000) 2019-10-13T07:06:37.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:37.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:37.060471+0000) 2019-10-13T07:06:37.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:37.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:37.473841+0000) 2019-10-13T07:06:37.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:37.470+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:37.476705+0000) 2019-10-13T07:06:40.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:40.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.602919+0000) 2019-10-13T07:06:40.704 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:40.698+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:40.701981+0000) 2019-10-13T07:06:41.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:41.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.499265+0000) 2019-10-13T07:06:41.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:41.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.499048+0000) 2019-10-13T07:06:41.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:41.639+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.642003+0000) 2019-10-13T07:06:41.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:41.898+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.904154+0000) 2019-10-13T07:06:41.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:41.898+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.904791+0000) 2019-10-13T07:06:41.907 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:41.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:41.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:41.902+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.906018+0000) 2019-10-13T07:06:41.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:41.902+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.906377+0000) 2019-10-13T07:06:41.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:41.903+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.906202+0000) 2019-10-13T07:06:41.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:41.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.907985+0000) 2019-10-13T07:06:41.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:41.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.909532+0000) 2019-10-13T07:06:41.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:41.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:41.913982+0000) 2019-10-13T07:06:45.053 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:06:45.054 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:45.059 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:06:45.060 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:06:46.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:46.902+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.908272+0000) 2019-10-13T07:06:46.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:46.902+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.909037+0000) 2019-10-13T07:06:46.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:46.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:46.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:46.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.909275+0000) 2019-10-13T07:06:46.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:46.906+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.910232+0000) 2019-10-13T07:06:46.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:46.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.910591+0000) 2019-10-13T07:06:46.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:46.906+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.911281+0000) 2019-10-13T07:06:46.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:46.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.912102+0000) 2019-10-13T07:06:46.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:46.915+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:46.916663+0000) 2019-10-13T07:06:47.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:47.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:47.060662+0000) 2019-10-13T07:06:47.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:47.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:47.474052+0000) 2019-10-13T07:06:47.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:47.470+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:47.476912+0000) 2019-10-13T07:06:50.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:50.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.603119+0000) 2019-10-13T07:06:50.704 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:50.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:50.702177+0000) 2019-10-13T07:06:51.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:51.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.499462+0000) 2019-10-13T07:06:51.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:51.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.499242+0000) 2019-10-13T07:06:51.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:51.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.648536+0000) 2019-10-13T07:06:51.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:51.905+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.913091+0000) 2019-10-13T07:06:51.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:51.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.913061+0000) 2019-10-13T07:06:51.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:51.905+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:51.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:51.909+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.913822+0000) 2019-10-13T07:06:51.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:51.909+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.914689+0000) 2019-10-13T07:06:51.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:51.909+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.914785+0000) 2019-10-13T07:06:51.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:51.909+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.916950+0000) 2019-10-13T07:06:51.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:51.913+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.917704+0000) 2019-10-13T07:06:51.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:51.915+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:51.919722+0000) 2019-10-13T07:06:53.203 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T07:06:56.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:56.909+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.916761+0000) 2019-10-13T07:06:56.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:06:56.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:06:56.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:06:56.913+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.918564+0000) 2019-10-13T07:06:56.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:06:56.915+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.918468+0000) 2019-10-13T07:06:56.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:56.913+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.919591+0000) 2019-10-13T07:06:56.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:06:56.913+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.919922+0000) 2019-10-13T07:06:56.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:56.913+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.920194+0000) 2019-10-13T07:06:56.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:06:56.913+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.921666+0000) 2019-10-13T07:06:56.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:06:56.923+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:56.926181+0000) 2019-10-13T07:06:57.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:06:57.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:57.060875+0000) 2019-10-13T07:06:57.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:06:57.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:57.474256+0000) 2019-10-13T07:06:57.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:06:57.469+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:06:57.477089+0000) 2019-10-13T07:07:00.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:00.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.603320+0000) 2019-10-13T07:07:00.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:00.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:00.702416+0000) 2019-10-13T07:07:01.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:01.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.499649+0000) 2019-10-13T07:07:01.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:01.495+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.499444+0000) 2019-10-13T07:07:01.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:01.651+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.655070+0000) 2019-10-13T07:07:01.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:01.917+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.921560+0000) 2019-10-13T07:07:01.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:01.917+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.921890+0000) 2019-10-13T07:07:01.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:01.917+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.922514+0000) 2019-10-13T07:07:01.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:01.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43609 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:01.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:01.919+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.923109+0000) 2019-10-13T07:07:01.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:01.917+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.923247+0000) 2019-10-13T07:07:01.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:01.917+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.924877+0000) 2019-10-13T07:07:01.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:01.917+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.925635+0000) 2019-10-13T07:07:01.932 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:01.927+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:01.929813+0000) 2019-10-13T07:07:06.928 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:06.925+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.926151+0000) 2019-10-13T07:07:06.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:06.925+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.926747+0000) 2019-10-13T07:07:06.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:06.925+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:06.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:06.925+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.927050+0000) 2019-10-13T07:07:06.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:06.925+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.928881+0000) 2019-10-13T07:07:06.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:06.926+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.928593+0000) 2019-10-13T07:07:06.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:06.925+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.929573+0000) 2019-10-13T07:07:06.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:06.929+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.930358+0000) 2019-10-13T07:07:06.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:06.930+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:06.934875+0000) 2019-10-13T07:07:07.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:07.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:07.061058+0000) 2019-10-13T07:07:07.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:07.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:07.474403+0000) 2019-10-13T07:07:07.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:07.473+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:07.477265+0000) 2019-10-13T07:07:10.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:10.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.603521+0000) 2019-10-13T07:07:10.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:10.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:10.702648+0000) 2019-10-13T07:07:11.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:11.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.499851+0000) 2019-10-13T07:07:11.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:11.494+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.499635+0000) 2019-10-13T07:07:11.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:11.658+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.661226+0000) 2019-10-13T07:07:11.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:11.929+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.930083+0000) 2019-10-13T07:07:11.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:11.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:11.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:11.926+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.930819+0000) 2019-10-13T07:07:11.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:11.929+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.931748+0000) 2019-10-13T07:07:11.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:11.929+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.932215+0000) 2019-10-13T07:07:11.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:11.929+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.932410+0000) 2019-10-13T07:07:11.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:11.929+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.932950+0000) 2019-10-13T07:07:11.935 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:11.929+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.933295+0000) 2019-10-13T07:07:11.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:11.934+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:11.939806+0000) 2019-10-13T07:07:15.184 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:07:15.184 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:15.188 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:07:15.188 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:16.937 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:16.933+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.935021+0000) 2019-10-13T07:07:16.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:16.933+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.935380+0000) 2019-10-13T07:07:16.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:16.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:16.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:16.933+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.936208+0000) 2019-10-13T07:07:16.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:16.933+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.936281+0000) 2019-10-13T07:07:16.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:16.933+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.937068+0000) 2019-10-13T07:07:16.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:16.934+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.937005+0000) 2019-10-13T07:07:16.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:16.937+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.938347+0000) 2019-10-13T07:07:16.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:16.938+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:16.941271+0000) 2019-10-13T07:07:17.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:17.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:17.061242+0000) 2019-10-13T07:07:17.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:17.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:17.474623+0000) 2019-10-13T07:07:17.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:17.473+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:17.477412+0000) 2019-10-13T07:07:20.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:20.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.603677+0000) 2019-10-13T07:07:20.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:20.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:20.702877+0000) 2019-10-13T07:07:21.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:21.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.500068+0000) 2019-10-13T07:07:21.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:21.494+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.499832+0000) 2019-10-13T07:07:21.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:21.666+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.667911+0000) 2019-10-13T07:07:21.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:21.937+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.938667+0000) 2019-10-13T07:07:21.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:21.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:21.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:21.937+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.939752+0000) 2019-10-13T07:07:21.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:21.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.940300+0000) 2019-10-13T07:07:21.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:21.937+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.940629+0000) 2019-10-13T07:07:21.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:21.941+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.941844+0000) 2019-10-13T07:07:21.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:21.937+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.941369+0000) 2019-10-13T07:07:21.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:21.942+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.945435+0000) 2019-10-13T07:07:21.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:21.946+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:21.950891+0000) 2019-10-13T07:07:26.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:26.941+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.942840+0000) 2019-10-13T07:07:26.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:26.941+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:26.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:26.938+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.942968+0000) 2019-10-13T07:07:26.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:26.941+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.943877+0000) 2019-10-13T07:07:26.948 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:26.941+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.944794+0000) 2019-10-13T07:07:26.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:26.945+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.945788+0000) 2019-10-13T07:07:26.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:26.945+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.946682+0000) 2019-10-13T07:07:26.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:26.945+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.947163+0000) 2019-10-13T07:07:26.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:26.946+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:26.950407+0000) 2019-10-13T07:07:27.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:27.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:27.061456+0000) 2019-10-13T07:07:27.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:27.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:27.474849+0000) 2019-10-13T07:07:27.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:27.473+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:27.477591+0000) 2019-10-13T07:07:30.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:30.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.603917+0000) 2019-10-13T07:07:30.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:30.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:30.703109+0000) 2019-10-13T07:07:31.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:31.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.500224+0000) 2019-10-13T07:07:31.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:31.494+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.500057+0000) 2019-10-13T07:07:31.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:31.670+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.674391+0000) 2019-10-13T07:07:31.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:31.945+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.946879+0000) 2019-10-13T07:07:31.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:31.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:31.950 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:31.945+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.948299+0000) 2019-10-13T07:07:31.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:31.945+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.948884+0000) 2019-10-13T07:07:31.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:31.945+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.949682+0000) 2019-10-13T07:07:31.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:31.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.949735+0000) 2019-10-13T07:07:31.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:31.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.951164+0000) 2019-10-13T07:07:31.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:31.949+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.952570+0000) 2019-10-13T07:07:31.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:31.954+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:31.956151+0000) 2019-10-13T07:07:36.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:36.949+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.951442+0000) 2019-10-13T07:07:36.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:36.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:36.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:36.946+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.951600+0000) 2019-10-13T07:07:36.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:36.949+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.952346+0000) 2019-10-13T07:07:36.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:36.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.952725+0000) 2019-10-13T07:07:36.957 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:36.949+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.953114+0000) 2019-10-13T07:07:36.957 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:36.953+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.955031+0000) 2019-10-13T07:07:36.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:36.953+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.956170+0000) 2019-10-13T07:07:36.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:36.958+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:36.960379+0000) 2019-10-13T07:07:37.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:37.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:37.061670+0000) 2019-10-13T07:07:37.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:37.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:37.475042+0000) 2019-10-13T07:07:37.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:37.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:37.477795+0000) 2019-10-13T07:07:40.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:40.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.604146+0000) 2019-10-13T07:07:40.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:40.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:40.703312+0000) 2019-10-13T07:07:41.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:41.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.500374+0000) 2019-10-13T07:07:41.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:41.498+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.500244+0000) 2019-10-13T07:07:41.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:41.678+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.680784+0000) 2019-10-13T07:07:41.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:41.953+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.955327+0000) 2019-10-13T07:07:41.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:41.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:41.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:41.950+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.955914+0000) 2019-10-13T07:07:41.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:41.953+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.956753+0000) 2019-10-13T07:07:41.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:41.953+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.957080+0000) 2019-10-13T07:07:41.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:41.957+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.958574+0000) 2019-10-13T07:07:41.961 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:41.957+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.957811+0000) 2019-10-13T07:07:41.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:41.957+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.959726+0000) 2019-10-13T07:07:41.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:41.958+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:41.963918+0000) 2019-10-13T07:07:45.339 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:07:45.339 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:45.342 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:07:45.343 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:07:46.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:46.956+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.959665+0000) 2019-10-13T07:07:46.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:46.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:46.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:46.956+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.960605+0000) 2019-10-13T07:07:46.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:46.956+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.960992+0000) 2019-10-13T07:07:46.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:46.958+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.960938+0000) 2019-10-13T07:07:46.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:46.960+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.962015+0000) 2019-10-13T07:07:46.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:46.960+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.962302+0000) 2019-10-13T07:07:46.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:46.960+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.963712+0000) 2019-10-13T07:07:46.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:46.966+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:46.968977+0000) 2019-10-13T07:07:47.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:47.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:47.061876+0000) 2019-10-13T07:07:47.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:47.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:47.475220+0000) 2019-10-13T07:07:47.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:47.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:47.477952+0000) 2019-10-13T07:07:50.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:50.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.604345+0000) 2019-10-13T07:07:50.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:50.700+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:50.703512+0000) 2019-10-13T07:07:51.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:51.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.500616+0000) 2019-10-13T07:07:51.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:51.498+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.500442+0000) 2019-10-13T07:07:51.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:51.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.687099+0000) 2019-10-13T07:07:51.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:51.960+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.963187+0000) 2019-10-13T07:07:51.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:51.958+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.963595+0000) 2019-10-13T07:07:51.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:51.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:51.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:51.960+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.964904+0000) 2019-10-13T07:07:51.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:51.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.965448+0000) 2019-10-13T07:07:51.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:51.960+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.965608+0000) 2019-10-13T07:07:51.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:51.964+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.966300+0000) 2019-10-13T07:07:51.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:51.964+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.966838+0000) 2019-10-13T07:07:51.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:51.970+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:51.974283+0000) 2019-10-13T07:07:53.204 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2019-10-13T07:07:53.204 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2019-10-13T07:07:53.205 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:07:53.205 INFO:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-10-13T07:07:54.072 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi110 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a8ee8ac95ca822967c9dffe2a705f1d5502812bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2019-10-13T07:07:54.074 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2019-10-13T07:07:54.081 DEBUG:tasks.check_counter:Getting stats from a 2019-10-13T07:07:54.082 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:07:54.082 INFO:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump 2019-10-13T07:07:54.786 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2019-10-13T07:07:54.786 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T07:07:54.786 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 1975, 2019-10-13T07:07:54.786 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 3812, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 21837811, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 641388, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 1996, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 18446744073709551614, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 1.741456698, 2019-10-13T07:07:54.787 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.112941473, 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 1.866947710, 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2477, 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 4.148605770, 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001674850 2019-10-13T07:07:54.788 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.789 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 8875, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 17288, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 22989948, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 411757604, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 8590, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 18446744073709551615, 2019-10-13T07:07:54.790 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 6.334801816, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.847064238, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 5.407355579, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.007778383, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 15967, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 11.589743425, 2019-10-13T07:07:54.791 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000725856 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T07:07:54.792 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 5575, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 11186, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 22246180, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 224549976, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 5584, 2019-10-13T07:07:54.793 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 0, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 4.171087344, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.505820967, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 3.730738934, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9856, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 8.042343153, 2019-10-13T07:07:54.794 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000815984 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T07:07:54.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 0, 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2019-10-13T07:07:54.796 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T07:07:54.797 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T07:07:54.798 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 78671, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 44, 2019-10-13T07:07:54.799 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2019-10-13T07:07:54.800 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 0, 2019-10-13T07:07:54.801 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 0, 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T07:07:54.802 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 16422, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 65857719, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 16422, 2019-10-13T07:07:54.803 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 16422, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 65857719, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2019-10-13T07:07:54.804 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2019-10-13T07:07:54.805 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T07:07:54.805 INFO:teuthology.orchestra.run.smithi110.stdout: } 2019-10-13T07:07:54.805 INFO:teuthology.orchestra.run.smithi110.stdout:} 2019-10-13T07:07:54.806 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2019-10-13T07:07:54.806 DEBUG:tasks.check_counter:Getting stats from b 2019-10-13T07:07:54.807 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:07:54.807 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok perf dump 2019-10-13T07:07:54.994 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2019-10-13T07:07:54.994 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2019-10-13T07:07:54.994 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 644, 2019-10-13T07:07:54.994 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 1958, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 6290393, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 3171453, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 581, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.547740769, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.102198150, 2019-10-13T07:07:54.995 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.451001407, 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.002057383, 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 1594, 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 1.284839025, 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000806047 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:54.996 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:54.997 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2019-10-13T07:07:54.998 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 10867, 2019-10-13T07:07:54.998 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 16896, 2019-10-13T07:07:54.998 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 31486290, 2019-10-13T07:07:54.998 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 15081536, 2019-10-13T07:07:54.998 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2738, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 3.145462287, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.656289800, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 2.693285860, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.042557507, 2019-10-13T07:07:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:55.000 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 15063, 2019-10-13T07:07:55.001 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 6.648876698, 2019-10-13T07:07:55.001 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000441404 2019-10-13T07:07:55.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.001 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:55.001 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 16180, 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.004781751, 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000295 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 4227, 2019-10-13T07:07:55.002 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 6701, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 30997983, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 5385481, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2737, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 2.257984158, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.217558390, 2019-10-13T07:07:55.003 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 2.141271876, 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.058067400, 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 4910, 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 6.055847056, 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.001233370 2019-10-13T07:07:55.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.005 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2019-10-13T07:07:55.005 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.005 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.005 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.006 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.006 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2019-10-13T07:07:55.007 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 1, 2019-10-13T07:07:55.007 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2019-10-13T07:07:55.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.007 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-MDSRank": { 2019-10-13T07:07:55.007 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2019-10-13T07:07:55.008 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2019-10-13T07:07:55.008 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 3370, 2019-10-13T07:07:55.008 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 2.313498567, 2019-10-13T07:07:55.008 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000686498 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-PurgeQueue": { 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 344, 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.049535383, 2019-10-13T07:07:55.009 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000143998 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: "mds": { 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: "request": 4174, 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: "reply": 4174, 2019-10-13T07:07:55.010 INFO:teuthology.orchestra.run.smithi204.stdout: "reply_latency": { 2019-10-13T07:07:55.011 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 4174, 2019-10-13T07:07:55.011 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 3.817322015, 2019-10-13T07:07:55.011 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000914547 2019-10-13T07:07:55.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.011 INFO:teuthology.orchestra.run.smithi204.stdout: "forward": 0, 2019-10-13T07:07:55.012 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch": 0, 2019-10-13T07:07:55.012 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_commit": 21, 2019-10-13T07:07:55.012 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_split": 1, 2019-10-13T07:07:55.012 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_merge": 0, 2019-10-13T07:07:55.013 INFO:teuthology.orchestra.run.smithi204.stdout: "inode_max": 2147483647, 2019-10-13T07:07:55.013 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes": 248, 2019-10-13T07:07:55.013 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_top": 0, 2019-10-13T07:07:55.013 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_bottom": 248, 2019-10-13T07:07:55.013 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pin_tail": 0, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pinned": 248, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_expired": 216, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_with_caps": 239, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "caps": 239, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "subtrees": 2, 2019-10-13T07:07:55.014 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse": 4686, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_hit": 1447, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_forward": 0, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_discover": 0, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_dir_fetch": 0, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_remote_ino": 0, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_lock": 9, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "load_cent": 1265, 2019-10-13T07:07:55.015 INFO:teuthology.orchestra.run.smithi204.stdout: "q": 0, 2019-10-13T07:07:55.016 INFO:teuthology.orchestra.run.smithi204.stdout: "exported": 0, 2019-10-13T07:07:55.016 INFO:teuthology.orchestra.run.smithi204.stdout: "exported_inodes": 0, 2019-10-13T07:07:55.016 INFO:teuthology.orchestra.run.smithi204.stdout: "imported": 0, 2019-10-13T07:07:55.016 INFO:teuthology.orchestra.run.smithi204.stdout: "imported_inodes": 0, 2019-10-13T07:07:55.016 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_dir_fetch": 0, 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_backtrace_fetch": 0, 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_peer_discover": 0 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_cache": { 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays": 0, 2019-10-13T07:07:55.017 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_delayed": 0, 2019-10-13T07:07:55.018 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_enqueuing": 0, 2019-10-13T07:07:55.018 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_created": 233, 2019-10-13T07:07:55.018 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_enqueued": 233, 2019-10-13T07:07:55.018 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_reintegrated": 0, 2019-10-13T07:07:55.018 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_migrated": 0, 2019-10-13T07:07:55.019 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_processing": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_enqueued": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_prioritized": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_started": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_completed": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_enqueue_scrub": 0, 2019-10-13T07:07:55.020 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_exportdir": 0, 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_flush": 0, 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragmentdir": 1, 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragstats": 0, 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_inodestats": 0 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_log": { 2019-10-13T07:07:55.021 INFO:teuthology.orchestra.run.smithi204.stdout: "evadd": 3446, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "evex": 0, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "evtrm": 0, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "ev": 3446, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "evexg": 0, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "evexd": 0, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "segadd": 4, 2019-10-13T07:07:55.022 INFO:teuthology.orchestra.run.smithi204.stdout: "segex": 0, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "segtrm": 0, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "seg": 4, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "segexg": 0, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "segexd": 0, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "expos": 4194304, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "wrpos": 12404723, 2019-10-13T07:07:55.023 INFO:teuthology.orchestra.run.smithi204.stdout: "rdpos": 0, 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: "jlat": { 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 1613, 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 6.779351503, 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.004202945 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: "replayed": 0 2019-10-13T07:07:55.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_mem": { 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "ino": 251, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "ino+": 484, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "ino-": 233, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "dir": 27, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "dir+": 34, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "dir-": 7, 2019-10-13T07:07:55.025 INFO:teuthology.orchestra.run.smithi204.stdout: "dn": 294, 2019-10-13T07:07:55.027 INFO:teuthology.orchestra.run.smithi204.stdout: "dn+": 770, 2019-10-13T07:07:55.027 INFO:teuthology.orchestra.run.smithi204.stdout: "dn-": 476, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "cap": 239, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "cap+": 472, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "cap-": 233, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "rss": 51748, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "heap": 314020 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.028 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_server": { 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_client_request": 4261, 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_server_request": 0, 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_request": 4174, 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_session": 53, 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_slave_request": 0, 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "req_create_latency": { 2019-10-13T07:07:55.029 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 453, 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 1.022409247, 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002256974 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getattr_latency": { 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 9, 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.013635160, 2019-10-13T07:07:55.030 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.001515017 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getfilelock_latency": { 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.031 INFO:teuthology.orchestra.run.smithi204.stdout: "req_link_latency": { 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookup_latency": { 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 3149, 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.884799636, 2019-10-13T07:07:55.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000280977 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookuphash_latency": { 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.033 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupino_latency": { 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupname_latency": { 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.034 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupparent_latency": { 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupsnap_latency": { 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lssnap_latency": { 2019-10-13T07:07:55.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mkdir_latency": { 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 14, 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.035288130, 2019-10-13T07:07:55.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002520580 2019-10-13T07:07:55.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.038 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mknod_latency": { 2019-10-13T07:07:55.038 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.038 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.038 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mksnap_latency": { 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.039 INFO:teuthology.orchestra.run.smithi204.stdout: "req_open_latency": { 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 265, 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.691900537, 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002610945 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "req_readdir_latency": { 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 43, 2019-10-13T07:07:55.040 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.479988944, 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.011162533 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rename_latency": { 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 22, 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.061027358, 2019-10-13T07:07:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002773970 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: "req_renamesnap_latency": { 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.042 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmdir_latency": { 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 6, 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.128475387, 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.021412564 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmsnap_latency": { 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.043 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmxattr_latency": { 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.044 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setattr_latency": { 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 1, 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.001044278, 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.001044278 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.045 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setdirlayout_latency": { 2019-10-13T07:07:55.046 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setfilelock_latency": { 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.047 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setlayout_latency": { 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.048 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setxattr_latency": { 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: "req_symlink_latency": { 2019-10-13T07:07:55.049 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 4, 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.008698001, 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002174500 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "req_unlink_latency": { 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 208, 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.490055337, 2019-10-13T07:07:55.050 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.002356035 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_revoke_eviction": 0 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_sessions": { 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "session_count": 1, 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "session_add": 1, 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "session_remove": 0, 2019-10-13T07:07:55.051 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_open": 1, 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_stale": 0, 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: "total_load": 52, 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: "average_load": 52, 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: "avg_session_uptime": 1002 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.052 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 120, 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 120, 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2019-10-13T07:07:55.053 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2019-10-13T07:07:55.054 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2019-10-13T07:07:55.054 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2019-10-13T07:07:55.054 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2019-10-13T07:07:55.054 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2019-10-13T07:07:55.054 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2019-10-13T07:07:55.055 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 148834, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 667, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2019-10-13T07:07:55.056 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 5160, 2019-10-13T07:07:55.057 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 76, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 789371, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 5416, 2019-10-13T07:07:55.058 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "objecter": { 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "op_active": 0, 2019-10-13T07:07:55.059 INFO:teuthology.orchestra.run.smithi204.stdout: "op_laggy": 0, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send": 2199, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send_bytes": 8389860, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op_resend": 31, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op_reply": 2199, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op": 2199, 2019-10-13T07:07:55.060 INFO:teuthology.orchestra.run.smithi204.stdout: "op_r": 28, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "op_w": 2171, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "op_rmw": 0, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "op_pg": 0, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_stat": 0, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_create": 11, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_read": 28, 2019-10-13T07:07:55.061 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_write": 1641, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writefull": 84, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writesame": 0, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_append": 0, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_zero": 0, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_truncate": 1, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_delete": 246, 2019-10-13T07:07:55.062 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_mapext": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_sparse_read": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_clonerange": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_getxattr": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_setxattr": 22, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_cmpxattr": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_rmxattr": 0, 2019-10-13T07:07:55.063 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_resetxattrs": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_call": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_watch": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_notify": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_src_cmpxattr": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls_filter": 0, 2019-10-13T07:07:55.064 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_other": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_active": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_send": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_resend": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_ping": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_active": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_send": 0, 2019-10-13T07:07:55.065 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_resend": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_active": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_send": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_resend": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_active": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_send": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_resend": 0, 2019-10-13T07:07:55.066 INFO:teuthology.orchestra.run.smithi204.stdout: "command_active": 0, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "command_send": 0, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "command_resend": 0, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "map_epoch": 25, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "map_full": 0, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "map_inc": 2, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_sessions": 4, 2019-10-13T07:07:55.067 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_open": 54, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_close": 50, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_laggy": 0, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_wr": 360, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_rd": 0, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_del": 134 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.068 INFO:teuthology.orchestra.run.smithi204.stdout: "purge_queue": { 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops": 0, 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing": 0, 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed": 233, 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_item_in_journal": 0 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2019-10-13T07:07:55.069 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2019-10-13T07:07:55.070 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2019-10-13T07:07:55.070 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2019-10-13T07:07:55.070 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 15735, 2019-10-13T07:07:55.070 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 67609284, 2019-10-13T07:07:55.070 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 15735, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 15735, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 67609284, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.071 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_bytes": { 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2019-10-13T07:07:55.072 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 2199, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 8413393, 2019-10-13T07:07:55.073 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 1952, 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 8413393, 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_ops": { 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 1024, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2019-10-13T07:07:55.075 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 2199, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 2199, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 2199, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 2199, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.076 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle": { 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2019-10-13T07:07:55.077 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 233, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 23533, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 233, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2019-10-13T07:07:55.078 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 28, 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 23533, 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle-0x56221081f7c0": { 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 3446, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 8210419, 2019-10-13T07:07:55.080 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 3446, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 1613, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 8210419, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2019-10-13T07:07:55.081 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2019-10-13T07:07:55.082 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2019-10-13T07:07:55.082 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.082 INFO:teuthology.orchestra.run.smithi204.stdout: } 2019-10-13T07:07:55.082 INFO:teuthology.orchestra.run.smithi204.stdout:} 2019-10-13T07:07:55.084 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2019-10-13T07:07:55.084 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2019-10-13T07:07:55.100 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2019-10-13T07:07:55.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-10-13T07:07:55.101 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:07:55.101 INFO:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2019-10-13T07:07:55.101 INFO:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2019-10-13T07:07:56.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:07:56.964+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.968169+0000) 2019-10-13T07:07:56.971 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:56.964+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.968555+0000) 2019-10-13T07:07:56.971 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:07:56.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:07:56.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:07:56.966+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.968872+0000) 2019-10-13T07:07:56.972 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:07:56.968+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.969917+0000) 2019-10-13T07:07:56.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:56.968+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.970003+0000) 2019-10-13T07:07:56.973 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:07:56.968+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.970354+0000) 2019-10-13T07:07:56.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:56.972+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.974755+0000) 2019-10-13T07:07:56.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:07:56.970+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:56.975664+0000) 2019-10-13T07:07:57.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:07:57.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:57.062062+0000) 2019-10-13T07:07:57.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:07:57.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:57.475367+0000) 2019-10-13T07:07:57.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:07:57.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:07:57.478127+0000) 2019-10-13T07:08:00.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:00.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.604497+0000) 2019-10-13T07:08:00.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:00.700+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:00.703714+0000) 2019-10-13T07:08:01.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:01.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.500829+0000) 2019-10-13T07:08:01.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:01.498+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.500650+0000) 2019-10-13T07:08:01.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:01.690+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.693595+0000) 2019-10-13T07:08:05.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:01.968+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.971512+0000) 2019-10-13T07:08:05.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:01.968+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.972314+0000) 2019-10-13T07:08:05.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:01.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:05.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:01.968+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.973695+0000) 2019-10-13T07:08:05.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:01.972+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.973907+0000) 2019-10-13T07:08:05.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:01.972+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.974633+0000) 2019-10-13T07:08:05.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:01.972+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.975420+0000) 2019-10-13T07:08:05.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:01.970+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.972620+0000) 2019-10-13T07:08:05.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:01.978+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:01.980225+0000) 2019-10-13T07:08:06.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:06.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:06.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:06.972+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.977487+0000) 2019-10-13T07:08:06.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:06.976+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.977884+0000) 2019-10-13T07:08:06.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:06.976+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.978534+0000) 2019-10-13T07:08:06.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:06.974+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.978626+0000) 2019-10-13T07:08:06.981 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:06.976+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.980063+0000) 2019-10-13T07:08:06.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:06.976+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.980658+0000) 2019-10-13T07:08:06.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:06.976+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.981269+0000) 2019-10-13T07:08:06.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:06.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:06.984639+0000) 2019-10-13T07:08:07.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:07.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:07.062243+0000) 2019-10-13T07:08:07.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:07.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:07.475543+0000) 2019-10-13T07:08:07.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:07.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:07.478261+0000) 2019-10-13T07:08:10.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:10.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.604720+0000) 2019-10-13T07:08:10.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:10.700+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:10.703946+0000) 2019-10-13T07:08:11.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:11.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.501065+0000) 2019-10-13T07:08:11.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:11.498+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.500862+0000) 2019-10-13T07:08:11.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:11.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.700174+0000) 2019-10-13T07:08:11.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:11.976+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44015 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:11.983 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:11.976+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.981353+0000) 2019-10-13T07:08:11.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:11.980+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.981990+0000) 2019-10-13T07:08:11.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:11.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.982231+0000) 2019-10-13T07:08:11.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:11.980+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.982451+0000) 2019-10-13T07:08:11.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:11.978+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.982880+0000) 2019-10-13T07:08:11.986 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:11.980+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.983779+0000) 2019-10-13T07:08:11.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:11.980+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.984081+0000) 2019-10-13T07:08:11.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:11.986+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:11.989003+0000) 2019-10-13T07:08:15.436 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:08:15.437 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:15.440 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:08:15.441 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:16.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:16.980+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.984526+0000) 2019-10-13T07:08:16.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:16.980+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.985376+0000) 2019-10-13T07:08:16.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:16.981+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.985762+0000) 2019-10-13T07:08:16.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:16.984+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:16.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:16.984+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.986558+0000) 2019-10-13T07:08:16.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:16.984+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.986873+0000) 2019-10-13T07:08:16.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:16.984+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.987426+0000) 2019-10-13T07:08:16.991 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:16.984+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.988933+0000) 2019-10-13T07:08:16.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:16.989+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:16.993581+0000) 2019-10-13T07:08:17.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:17.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:17.062412+0000) 2019-10-13T07:08:17.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:17.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:17.475677+0000) 2019-10-13T07:08:17.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:17.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:17.478449+0000) 2019-10-13T07:08:20.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:20.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.604932+0000) 2019-10-13T07:08:20.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:20.700+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:20.704172+0000) 2019-10-13T07:08:21.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:21.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.501259+0000) 2019-10-13T07:08:21.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:21.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.501068+0000) 2019-10-13T07:08:21.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:21.705+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.706709+0000) 2019-10-13T07:08:21.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:21.988+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.990506+0000) 2019-10-13T07:08:21.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:21.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:21.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:21.988+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.990800+0000) 2019-10-13T07:08:21.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:21.985+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.991111+0000) 2019-10-13T07:08:21.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:21.988+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.991380+0000) 2019-10-13T07:08:21.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:21.988+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.991798+0000) 2019-10-13T07:08:21.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:21.988+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.992651+0000) 2019-10-13T07:08:21.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:21.988+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.993121+0000) 2019-10-13T07:08:22.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:21.993+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:21.998782+0000) 2019-10-13T07:08:26.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:26.992+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.993725+0000) 2019-10-13T07:08:26.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:26.992+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.994161+0000) 2019-10-13T07:08:26.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:26.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:26.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:26.989+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.994738+0000) 2019-10-13T07:08:26.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:26.992+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.996003+0000) 2019-10-13T07:08:26.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:26.992+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.996152+0000) 2019-10-13T07:08:26.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:26.996+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.997800+0000) 2019-10-13T07:08:27.000 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:26.996+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:26.998683+0000) 2019-10-13T07:08:27.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:26.997+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:27.003120+0000) 2019-10-13T07:08:27.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:27.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:27.062632+0000) 2019-10-13T07:08:27.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:27.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:27.475855+0000) 2019-10-13T07:08:27.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:27.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:27.478629+0000) 2019-10-13T07:08:30.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:30.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.605124+0000) 2019-10-13T07:08:30.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:30.700+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:30.704374+0000) 2019-10-13T07:08:31.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:31.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.501454+0000) 2019-10-13T07:08:31.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:31.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.501259+0000) 2019-10-13T07:08:31.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:31.709+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.713218+0000) 2019-10-13T07:08:32.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:31.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:32.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:31.996+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.999022+0000) 2019-10-13T07:08:32.002 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:31.996+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:31.999547+0000) 2019-10-13T07:08:32.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:31.996+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.000030+0000) 2019-10-13T07:08:32.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:31.996+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.000443+0000) 2019-10-13T07:08:32.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:31.997+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.002812+0000) 2019-10-13T07:08:32.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:32.000+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.003649+0000) 2019-10-13T07:08:32.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:32.000+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.005142+0000) 2019-10-13T07:08:32.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:32.001+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:32.007998+0000) 2019-10-13T07:08:37.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:37.000+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.002780+0000) 2019-10-13T07:08:37.005 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:37.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:37.006 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:37.000+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.003569+0000) 2019-10-13T07:08:37.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:36.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.003764+0000) 2019-10-13T07:08:37.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:37.000+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.004585+0000) 2019-10-13T07:08:37.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:37.000+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.004952+0000) 2019-10-13T07:08:37.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:37.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.005230+0000) 2019-10-13T07:08:37.009 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:37.004+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.007588+0000) 2019-10-13T07:08:37.014 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:37.009+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.012252+0000) 2019-10-13T07:08:37.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:37.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.062807+0000) 2019-10-13T07:08:37.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:37.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.476038+0000) 2019-10-13T07:08:37.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:37.476+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:37.478810+0000) 2019-10-13T07:08:40.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:40.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.605329+0000) 2019-10-13T07:08:40.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:40.704+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:40.704534+0000) 2019-10-13T07:08:41.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:41.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.501651+0000) 2019-10-13T07:08:41.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:41.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.501462+0000) 2019-10-13T07:08:41.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:41.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:41.719581+0000) 2019-10-13T07:08:42.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:42.004+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.006640+0000) 2019-10-13T07:08:42.010 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:42.004+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.006802+0000) 2019-10-13T07:08:42.010 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:42.004+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:42.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:42.004+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.007161+0000) 2019-10-13T07:08:42.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:42.001+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.007471+0000) 2019-10-13T07:08:42.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:42.004+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.009054+0000) 2019-10-13T07:08:42.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:42.008+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.010755+0000) 2019-10-13T07:08:42.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:42.008+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.010464+0000) 2019-10-13T07:08:42.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:42.009+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:42.015538+0000) 2019-10-13T07:08:45.542 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:08:45.542 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:45.546 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:08:45.546 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:08:47.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:47.007+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.010197+0000) 2019-10-13T07:08:47.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:47.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:47.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:47.007+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.011954+0000) 2019-10-13T07:08:47.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:47.007+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.011709+0000) 2019-10-13T07:08:47.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:47.007+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.012544+0000) 2019-10-13T07:08:47.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:47.007+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.012837+0000) 2019-10-13T07:08:47.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:47.011+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.014869+0000) 2019-10-13T07:08:47.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:47.009+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.015635+0000) 2019-10-13T07:08:47.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:47.013+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.016905+0000) 2019-10-13T07:08:47.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:47.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.062987+0000) 2019-10-13T07:08:47.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:47.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.476215+0000) 2019-10-13T07:08:47.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:47.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:47.478957+0000) 2019-10-13T07:08:50.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:50.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.605565+0000) 2019-10-13T07:08:50.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:50.699+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:50.704757+0000) 2019-10-13T07:08:51.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:51.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.501836+0000) 2019-10-13T07:08:51.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:51.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.501657+0000) 2019-10-13T07:08:51.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:51.721+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:51.726048+0000) 2019-10-13T07:08:52.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:52.011+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.014016+0000) 2019-10-13T07:08:52.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:52.009+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.014573+0000) 2019-10-13T07:08:52.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:52.011+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44247 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:52.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:52.011+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.015582+0000) 2019-10-13T07:08:52.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:52.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.016340+0000) 2019-10-13T07:08:52.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:52.011+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.016586+0000) 2019-10-13T07:08:52.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:52.011+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.017054+0000) 2019-10-13T07:08:52.019 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:52.011+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.017324+0000) 2019-10-13T07:08:52.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:52.021+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:52.024693+0000) 2019-10-13T07:08:57.020 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:08:57.015+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.018423+0000) 2019-10-13T07:08:57.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:08:57.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44273 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:08:57.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:57.015+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.019376+0000) 2019-10-13T07:08:57.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:08:57.015+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.020119+0000) 2019-10-13T07:08:57.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:08:57.015+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.020609+0000) 2019-10-13T07:08:57.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:08:57.017+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.021496+0000) 2019-10-13T07:08:57.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:57.019+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.022421+0000) 2019-10-13T07:08:57.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:57.019+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.022904+0000) 2019-10-13T07:08:57.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:08:57.021+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.025005+0000) 2019-10-13T07:08:57.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:08:57.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.063200+0000) 2019-10-13T07:08:57.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:08:57.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.476413+0000) 2019-10-13T07:08:57.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:08:57.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:08:57.479141+0000) 2019-10-13T07:09:00.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:00.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.605805+0000) 2019-10-13T07:09:00.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:00.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:00.704983+0000) 2019-10-13T07:09:01.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:01.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.502031+0000) 2019-10-13T07:09:01.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:01.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.501856+0000) 2019-10-13T07:09:01.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:01.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:01.732471+0000) 2019-10-13T07:09:02.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:02.019+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.022843+0000) 2019-10-13T07:09:02.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:02.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:02.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:02.019+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.023318+0000) 2019-10-13T07:09:02.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:02.019+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.023921+0000) 2019-10-13T07:09:02.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:02.019+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.024010+0000) 2019-10-13T07:09:02.027 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:02.019+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.024328+0000) 2019-10-13T07:09:02.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:02.019+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.024744+0000) 2019-10-13T07:09:02.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:02.021+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.024758+0000) 2019-10-13T07:09:02.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:02.025+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:02.032062+0000) 2019-10-13T07:09:07.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:07.023+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.026511+0000) 2019-10-13T07:09:07.029 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:07.023+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.027138+0000) 2019-10-13T07:09:07.029 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:07.023+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:07.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:07.023+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.028347+0000) 2019-10-13T07:09:07.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:07.025+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.028511+0000) 2019-10-13T07:09:07.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:07.027+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.029760+0000) 2019-10-13T07:09:07.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:07.027+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.030094+0000) 2019-10-13T07:09:07.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:07.027+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.030619+0000) 2019-10-13T07:09:07.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:07.029+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.033431+0000) 2019-10-13T07:09:07.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:07.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.063380+0000) 2019-10-13T07:09:07.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:07.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.476595+0000) 2019-10-13T07:09:07.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:07.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:07.479307+0000) 2019-10-13T07:09:10.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:10.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.606008+0000) 2019-10-13T07:09:10.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:10.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:10.705177+0000) 2019-10-13T07:09:11.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:11.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.502220+0000) 2019-10-13T07:09:11.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:11.497+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.502058+0000) 2019-10-13T07:09:11.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:11.737+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:11.738917+0000) 2019-10-13T07:09:12.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:12.027+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.031213+0000) 2019-10-13T07:09:12.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:12.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:12.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:12.027+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.031535+0000) 2019-10-13T07:09:12.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:12.027+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.031890+0000) 2019-10-13T07:09:12.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:12.029+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.032353+0000) 2019-10-13T07:09:12.036 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:12.031+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.033778+0000) 2019-10-13T07:09:12.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:12.027+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.033155+0000) 2019-10-13T07:09:12.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:12.031+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.034506+0000) 2019-10-13T07:09:12.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:12.037+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:12.040986+0000) 2019-10-13T07:09:15.647 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:09:15.647 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:15.651 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:09:15.652 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:17.037 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:17.031+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.035095+0000) 2019-10-13T07:09:17.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:17.031+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.035468+0000) 2019-10-13T07:09:17.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:17.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:17.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:17.031+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.037075+0000) 2019-10-13T07:09:17.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:17.031+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.037214+0000) 2019-10-13T07:09:17.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:17.033+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.037774+0000) 2019-10-13T07:09:17.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:17.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.039542+0000) 2019-10-13T07:09:17.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:17.035+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.040115+0000) 2019-10-13T07:09:17.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:17.037+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.042473+0000) 2019-10-13T07:09:17.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:17.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.063533+0000) 2019-10-13T07:09:17.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:17.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.476729+0000) 2019-10-13T07:09:17.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:17.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:17.479492+0000) 2019-10-13T07:09:20.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:20.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.606239+0000) 2019-10-13T07:09:20.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:20.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:20.705379+0000) 2019-10-13T07:09:21.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:21.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.502412+0000) 2019-10-13T07:09:21.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:21.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.502260+0000) 2019-10-13T07:09:21.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:21.740+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:21.745497+0000) 2019-10-13T07:09:22.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:22.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44420 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:22.042 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:22.035+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.040851+0000) 2019-10-13T07:09:22.043 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:22.036+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.040585+0000) 2019-10-13T07:09:22.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:22.035+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.041247+0000) 2019-10-13T07:09:22.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:22.035+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.041525+0000) 2019-10-13T07:09:22.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:22.039+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.041934+0000) 2019-10-13T07:09:22.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:22.039+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.043157+0000) 2019-10-13T07:09:22.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:22.039+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.043802+0000) 2019-10-13T07:09:22.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:22.040+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:22.045678+0000) 2019-10-13T07:09:27.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:27.039+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.043003+0000) 2019-10-13T07:09:27.045 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:27.036+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.042970+0000) 2019-10-13T07:09:27.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:27.039+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.044100+0000) 2019-10-13T07:09:27.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:27.039+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.044139+0000) 2019-10-13T07:09:27.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:27.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:27.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:27.039+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.045124+0000) 2019-10-13T07:09:27.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:27.039+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.045435+0000) 2019-10-13T07:09:27.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:27.039+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.045607+0000) 2019-10-13T07:09:27.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:27.048+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.052123+0000) 2019-10-13T07:09:27.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:27.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.063650+0000) 2019-10-13T07:09:27.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:27.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.476900+0000) 2019-10-13T07:09:27.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:27.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:27.479625+0000) 2019-10-13T07:09:30.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:30.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.606437+0000) 2019-10-13T07:09:30.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:30.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:30.705582+0000) 2019-10-13T07:09:31.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:31.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.502636+0000) 2019-10-13T07:09:31.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:31.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.502461+0000) 2019-10-13T07:09:31.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:31.748+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:31.751941+0000) 2019-10-13T07:09:32.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:32.040+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.047864+0000) 2019-10-13T07:09:32.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:32.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:32.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:32.043+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.048990+0000) 2019-10-13T07:09:32.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:32.043+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.049228+0000) 2019-10-13T07:09:32.052 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:32.047+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.049736+0000) 2019-10-13T07:09:32.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:32.047+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.050037+0000) 2019-10-13T07:09:32.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:32.047+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.050925+0000) 2019-10-13T07:09:32.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:32.047+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.051039+0000) 2019-10-13T07:09:32.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:32.052+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:32.056501+0000) 2019-10-13T07:09:37.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:37.047+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.052214+0000) 2019-10-13T07:09:37.055 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:37.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:37.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:37.051+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.053965+0000) 2019-10-13T07:09:37.057 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:37.048+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.054552+0000) 2019-10-13T07:09:37.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:37.051+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.055482+0000) 2019-10-13T07:09:37.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:37.051+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.055840+0000) 2019-10-13T07:09:37.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:37.051+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.056314+0000) 2019-10-13T07:09:37.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:37.051+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.055621+0000) 2019-10-13T07:09:37.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:37.056+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.060173+0000) 2019-10-13T07:09:37.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:37.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.063794+0000) 2019-10-13T07:09:37.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:37.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.477088+0000) 2019-10-13T07:09:37.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:37.475+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:37.479813+0000) 2019-10-13T07:09:40.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:40.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.606656+0000) 2019-10-13T07:09:40.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:40.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:40.705783+0000) 2019-10-13T07:09:41.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:41.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.502826+0000) 2019-10-13T07:09:41.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:41.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.502675+0000) 2019-10-13T07:09:41.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:41.756+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:41.758305+0000) 2019-10-13T07:09:42.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:42.050+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.056303+0000) 2019-10-13T07:09:42.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:42.050+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.057521+0000) 2019-10-13T07:09:42.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:42.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:42.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:42.054+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.057915+0000) 2019-10-13T07:09:42.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:42.054+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.059137+0000) 2019-10-13T07:09:42.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:42.054+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.059001+0000) 2019-10-13T07:09:42.062 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:42.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.059944+0000) 2019-10-13T07:09:42.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:42.054+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.060985+0000) 2019-10-13T07:09:42.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:42.060+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:42.064478+0000) 2019-10-13T07:09:45.769 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:09:46.913 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:46.917 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:09:46.917 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:09:47.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:47.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.061438+0000) 2019-10-13T07:09:47.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:47.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:47.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:47.058+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.062130+0000) 2019-10-13T07:09:47.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:47.058+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.062561+0000) 2019-10-13T07:09:47.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:47.056+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.062350+0000) 2019-10-13T07:09:47.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:47.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.063283+0000) 2019-10-13T07:09:47.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:47.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.063931+0000) 2019-10-13T07:09:47.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:47.062+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.065819+0000) 2019-10-13T07:09:47.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:47.062+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.066294+0000) 2019-10-13T07:09:47.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:47.064+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.070650+0000) 2019-10-13T07:09:47.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:47.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.477297+0000) 2019-10-13T07:09:47.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:47.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:47.479977+0000) 2019-10-13T07:09:50.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:50.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.606856+0000) 2019-10-13T07:09:50.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:50.702+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:50.705981+0000) 2019-10-13T07:09:51.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:51.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.503026+0000) 2019-10-13T07:09:51.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:51.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.502874+0000) 2019-10-13T07:09:51.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:51.764+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:51.764761+0000) 2019-10-13T07:09:52.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:52.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.065281+0000) 2019-10-13T07:09:52.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:52.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:52.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:52.062+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.066560+0000) 2019-10-13T07:09:52.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:52.062+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.067018+0000) 2019-10-13T07:09:52.070 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:52.060+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.067674+0000) 2019-10-13T07:09:52.072 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:52.062+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.068696+0000) 2019-10-13T07:09:52.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:52.062+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.069273+0000) 2019-10-13T07:09:52.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:52.062+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.069626+0000) 2019-10-13T07:09:52.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:52.068+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:52.076064+0000) 2019-10-13T07:09:57.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:57.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.064154+0000) 2019-10-13T07:09:57.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:09:57.062+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.069314+0000) 2019-10-13T07:09:57.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:09:57.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:09:57.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:09:57.066+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.070256+0000) 2019-10-13T07:09:57.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:09:57.064+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.071772+0000) 2019-10-13T07:09:57.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:09:57.066+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.072556+0000) 2019-10-13T07:09:57.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:57.066+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.072673+0000) 2019-10-13T07:09:57.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:09:57.066+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.073116+0000) 2019-10-13T07:09:57.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:57.070+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.074797+0000) 2019-10-13T07:09:57.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:09:57.076+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.080186+0000) 2019-10-13T07:09:57.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:09:57.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.477478+0000) 2019-10-13T07:09:57.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:09:57.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:09:57.480157+0000) 2019-10-13T07:10:00.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:00.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.607053+0000) 2019-10-13T07:10:00.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:00.702+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:00.706184+0000) 2019-10-13T07:10:01.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:01.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.503222+0000) 2019-10-13T07:10:01.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:01.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.503080+0000) 2019-10-13T07:10:01.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:01.768+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:01.774814+0000) 2019-10-13T07:10:02.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:02.070+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.073900+0000) 2019-10-13T07:10:02.076 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:02.070+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.074320+0000) 2019-10-13T07:10:02.077 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:02.070+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:02.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:02.070+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.075000+0000) 2019-10-13T07:10:02.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:02.070+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.075576+0000) 2019-10-13T07:10:02.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:02.068+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.075325+0000) 2019-10-13T07:10:02.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:02.070+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.076106+0000) 2019-10-13T07:10:02.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:02.070+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.076358+0000) 2019-10-13T07:10:02.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:02.076+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:02.081869+0000) 2019-10-13T07:10:07.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:07.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.064386+0000) 2019-10-13T07:10:07.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:07.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:07.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:07.074+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.079463+0000) 2019-10-13T07:10:07.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:07.074+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.080681+0000) 2019-10-13T07:10:07.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:07.074+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.080980+0000) 2019-10-13T07:10:07.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:07.074+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.081374+0000) 2019-10-13T07:10:07.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:07.074+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.081608+0000) 2019-10-13T07:10:07.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:07.076+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.081510+0000) 2019-10-13T07:10:07.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:07.078+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.082292+0000) 2019-10-13T07:10:07.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:07.080+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.086166+0000) 2019-10-13T07:10:07.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:07.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.477599+0000) 2019-10-13T07:10:07.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:07.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:07.480328+0000) 2019-10-13T07:10:10.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:10.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.607258+0000) 2019-10-13T07:10:10.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:10.702+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:10.706401+0000) 2019-10-13T07:10:11.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:11.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.503450+0000) 2019-10-13T07:10:11.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:11.496+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.503272+0000) 2019-10-13T07:10:11.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:11.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:11.781393+0000) 2019-10-13T07:10:12.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:12.076+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.082848+0000) 2019-10-13T07:10:12.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:12.078+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.082138+0000) 2019-10-13T07:10:12.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:12.078+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.082359+0000) 2019-10-13T07:10:12.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:12.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:12.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:12.078+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.083953+0000) 2019-10-13T07:10:12.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:12.078+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.084434+0000) 2019-10-13T07:10:12.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:12.078+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.085445+0000) 2019-10-13T07:10:12.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:12.082+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.085806+0000) 2019-10-13T07:10:12.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:12.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:12.092188+0000) 2019-10-13T07:10:17.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:17.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.064586+0000) 2019-10-13T07:10:17.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:17.080+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.085894+0000) 2019-10-13T07:10:17.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:17.082+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.086819+0000) 2019-10-13T07:10:17.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:17.082+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.087022+0000) 2019-10-13T07:10:17.089 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:17.082+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.087417+0000) 2019-10-13T07:10:17.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:17.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:17.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:17.082+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.088304+0000) 2019-10-13T07:10:17.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:17.082+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.089047+0000) 2019-10-13T07:10:17.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:17.086+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.089804+0000) 2019-10-13T07:10:17.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:17.092+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.096491+0000) 2019-10-13T07:10:17.153 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:10:17.153 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:17.157 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:10:17.158 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:17.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:17.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.477798+0000) 2019-10-13T07:10:17.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:17.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:17.480509+0000) 2019-10-13T07:10:20.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:20.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.607451+0000) 2019-10-13T07:10:21.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:20.706+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:20.706596+0000) 2019-10-13T07:10:21.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:21.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.503663+0000) 2019-10-13T07:10:21.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:21.500+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.503505+0000) 2019-10-13T07:10:21.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:21.784+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:21.787852+0000) 2019-10-13T07:10:22.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:22.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:22.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:22.086+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.092431+0000) 2019-10-13T07:10:22.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:22.086+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.092513+0000) 2019-10-13T07:10:22.101 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:22.092+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.093868+0000) 2019-10-13T07:10:22.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:22.086+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.093559+0000) 2019-10-13T07:10:22.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:22.090+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.093989+0000) 2019-10-13T07:10:22.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:22.090+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.094922+0000) 2019-10-13T07:10:22.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:22.090+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.095800+0000) 2019-10-13T07:10:22.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:22.100+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:22.101742+0000) 2019-10-13T07:10:27.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:27.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.064786+0000) 2019-10-13T07:10:27.105 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:27.095+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.098222+0000) 2019-10-13T07:10:27.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:27.090+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.094874+0000) 2019-10-13T07:10:27.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:27.090+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.095508+0000) 2019-10-13T07:10:27.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:27.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:27.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:27.090+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.096361+0000) 2019-10-13T07:10:27.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:27.090+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.096757+0000) 2019-10-13T07:10:27.106 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:27.090+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.097516+0000) 2019-10-13T07:10:27.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:27.094+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.099982+0000) 2019-10-13T07:10:27.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:27.103+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.105109+0000) 2019-10-13T07:10:27.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:27.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.477975+0000) 2019-10-13T07:10:27.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:27.474+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:27.480683+0000) 2019-10-13T07:10:30.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:30.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.607661+0000) 2019-10-13T07:10:30.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:30.706+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:30.706839+0000) 2019-10-13T07:10:31.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:31.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.503871+0000) 2019-10-13T07:10:31.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:31.499+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.503704+0000) 2019-10-13T07:10:31.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:31.791+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:31.794348+0000) 2019-10-13T07:10:32.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:32.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:32.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:32.099+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.101172+0000) 2019-10-13T07:10:32.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:32.098+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.102176+0000) 2019-10-13T07:10:32.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:32.098+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.102709+0000) 2019-10-13T07:10:32.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:32.098+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.102829+0000) 2019-10-13T07:10:32.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:32.098+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.103302+0000) 2019-10-13T07:10:32.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:32.098+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.103828+0000) 2019-10-13T07:10:32.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:32.098+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.104982+0000) 2019-10-13T07:10:32.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:32.111+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:32.112344+0000) 2019-10-13T07:10:37.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:37.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.064995+0000) 2019-10-13T07:10:37.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:37.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:37.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:37.098+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.105087+0000) 2019-10-13T07:10:37.108 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:37.102+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.105789+0000) 2019-10-13T07:10:37.108 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:37.103+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.106304+0000) 2019-10-13T07:10:37.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:37.102+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.107292+0000) 2019-10-13T07:10:37.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:37.102+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.107714+0000) 2019-10-13T07:10:37.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:37.102+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.108217+0000) 2019-10-13T07:10:37.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:37.102+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.108549+0000) 2019-10-13T07:10:37.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:37.111+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.112320+0000) 2019-10-13T07:10:37.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:37.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.478165+0000) 2019-10-13T07:10:37.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:37.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:37.480845+0000) 2019-10-13T07:10:40.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:40.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.607855+0000) 2019-10-13T07:10:40.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:40.701+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:40.707044+0000) 2019-10-13T07:10:41.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:41.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.504102+0000) 2019-10-13T07:10:41.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:41.499+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.503937+0000) 2019-10-13T07:10:41.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:41.803+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:41.805919+0000) 2019-10-13T07:10:42.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:42.101+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.108346+0000) 2019-10-13T07:10:42.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:42.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:42.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:42.101+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.109046+0000) 2019-10-13T07:10:42.113 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:42.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.109288+0000) 2019-10-13T07:10:42.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:42.105+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.110502+0000) 2019-10-13T07:10:42.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:42.107+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.111280+0000) 2019-10-13T07:10:42.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:42.105+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.111007+0000) 2019-10-13T07:10:42.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:42.105+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.111484+0000) 2019-10-13T07:10:42.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:42.115+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:42.118423+0000) 2019-10-13T07:10:47.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:47.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.065288+0000) 2019-10-13T07:10:47.114 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:47.105+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.112215+0000) 2019-10-13T07:10:47.115 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:47.107+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.111952+0000) 2019-10-13T07:10:47.115 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:47.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:47.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:47.105+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.113150+0000) 2019-10-13T07:10:47.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:47.105+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.113604+0000) 2019-10-13T07:10:47.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:47.109+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.114484+0000) 2019-10-13T07:10:47.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:47.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.114980+0000) 2019-10-13T07:10:47.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:47.109+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.116254+0000) 2019-10-13T07:10:47.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:47.123+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.124282+0000) 2019-10-13T07:10:47.201 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:10:47.201 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:47.205 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:10:47.206 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:10:47.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:47.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.478344+0000) 2019-10-13T07:10:47.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:47.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:47.481015+0000) 2019-10-13T07:10:50.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:50.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.608061+0000) 2019-10-13T07:10:52.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:51.503+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.504136+0000) 2019-10-13T07:10:52.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:52.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.117342+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:51.811+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.812303+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:52.123+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.124770+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:50.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:50.707222+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:52.109+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.117155+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:51.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:51.504297+0000) 2019-10-13T07:10:52.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:52.113+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.118208+0000) 2019-10-13T07:10:52.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:52.109+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.116608+0000) 2019-10-13T07:10:52.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:52.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:52.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:52.109+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.117172+0000) 2019-10-13T07:10:52.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:52.109+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.117609+0000) 2019-10-13T07:10:52.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:52.113+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:52.119752+0000) 2019-10-13T07:10:57.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:57.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.065549+0000) 2019-10-13T07:10:57.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:10:57.117+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.120429+0000) 2019-10-13T07:10:57.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:10:57.117+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.119941+0000) 2019-10-13T07:10:57.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:57.117+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.120916+0000) 2019-10-13T07:10:57.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:10:57.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:10:57.124 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:10:57.121+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.122164+0000) 2019-10-13T07:10:57.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:10:57.119+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.122569+0000) 2019-10-13T07:10:57.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:57.121+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.122615+0000) 2019-10-13T07:10:57.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:10:57.121+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.123303+0000) 2019-10-13T07:10:57.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:10:57.127+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.128249+0000) 2019-10-13T07:10:57.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:10:57.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.478527+0000) 2019-10-13T07:10:57.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:10:57.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:10:57.481194+0000) 2019-10-13T07:11:00.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:00.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.608277+0000) 2019-10-13T07:11:00.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:00.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:00.707422+0000) 2019-10-13T07:11:01.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:01.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.504498+0000) 2019-10-13T07:11:01.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:01.503+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.504360+0000) 2019-10-13T07:11:01.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:01.815+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:01.818897+0000) 2019-10-13T07:11:02.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:02.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.124793+0000) 2019-10-13T07:11:02.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:02.123+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.124839+0000) 2019-10-13T07:11:02.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:02.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:02.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:02.125+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.125880+0000) 2019-10-13T07:11:02.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:02.125+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.126734+0000) 2019-10-13T07:11:02.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:02.125+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.128537+0000) 2019-10-13T07:11:02.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:02.125+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.128882+0000) 2019-10-13T07:11:02.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:02.125+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.129456+0000) 2019-10-13T07:11:02.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:02.131+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:02.133215+0000) 2019-10-13T07:11:07.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:07.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.065769+0000) 2019-10-13T07:11:07.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:07.125+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.129212+0000) 2019-10-13T07:11:07.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:07.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:07.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:07.125+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.129588+0000) 2019-10-13T07:11:07.132 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:07.129+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.129935+0000) 2019-10-13T07:11:07.132 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:07.129+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.130606+0000) 2019-10-13T07:11:07.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:07.127+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.130507+0000) 2019-10-13T07:11:07.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:07.129+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.133606+0000) 2019-10-13T07:11:07.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:07.133+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.135426+0000) 2019-10-13T07:11:07.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:07.135+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.136360+0000) 2019-10-13T07:11:07.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:07.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.478707+0000) 2019-10-13T07:11:07.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:07.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:07.481397+0000) 2019-10-13T07:11:10.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:10.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.608509+0000) 2019-10-13T07:11:10.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:10.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:10.707599+0000) 2019-10-13T07:11:11.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:11.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.504710+0000) 2019-10-13T07:11:11.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:11.503+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.504556+0000) 2019-10-13T07:11:11.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:11.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:11.825070+0000) 2019-10-13T07:11:12.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:12.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:12.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:12.133+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.134295+0000) 2019-10-13T07:11:12.136 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:12.133+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.134861+0000) 2019-10-13T07:11:12.137 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:12.131+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.134891+0000) 2019-10-13T07:11:12.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:12.133+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.135241+0000) 2019-10-13T07:11:12.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:12.133+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.135771+0000) 2019-10-13T07:11:12.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:12.133+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.136041+0000) 2019-10-13T07:11:12.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:12.133+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.137042+0000) 2019-10-13T07:11:12.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:12.139+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:12.143388+0000) 2019-10-13T07:11:17.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:17.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.065948+0000) 2019-10-13T07:11:17.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:17.135+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.136632+0000) 2019-10-13T07:11:17.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:17.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:17.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:17.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.138042+0000) 2019-10-13T07:11:17.140 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:17.137+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.138805+0000) 2019-10-13T07:11:17.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:17.137+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.138960+0000) 2019-10-13T07:11:17.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:17.137+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.139510+0000) 2019-10-13T07:11:17.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:17.137+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.140168+0000) 2019-10-13T07:11:17.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:17.137+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.140736+0000) 2019-10-13T07:11:17.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:17.143+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.147364+0000) 2019-10-13T07:11:17.325 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:11:17.325 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:17.329 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:11:17.329 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:17.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:17.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.478906+0000) 2019-10-13T07:11:17.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:17.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:17.481571+0000) 2019-10-13T07:11:20.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:20.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.608719+0000) 2019-10-13T07:11:20.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:20.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:20.707775+0000) 2019-10-13T07:11:21.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:21.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.504908+0000) 2019-10-13T07:11:21.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:21.503+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.504779+0000) 2019-10-13T07:11:21.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:21.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:21.836416+0000) 2019-10-13T07:11:22.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:22.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.140900+0000) 2019-10-13T07:11:22.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:22.137+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.141392+0000) 2019-10-13T07:11:22.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:22.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:22.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:22.141+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.141994+0000) 2019-10-13T07:11:22.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:22.141+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.142695+0000) 2019-10-13T07:11:22.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:22.139+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.142333+0000) 2019-10-13T07:11:22.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:22.141+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.143738+0000) 2019-10-13T07:11:22.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:22.141+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.144062+0000) 2019-10-13T07:11:22.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:22.147+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:22.148536+0000) 2019-10-13T07:11:27.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:27.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.066163+0000) 2019-10-13T07:11:27.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:27.141+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.145001+0000) 2019-10-13T07:11:27.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:27.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:27.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:27.145+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.146021+0000) 2019-10-13T07:11:27.148 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:27.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.146377+0000) 2019-10-13T07:11:27.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:27.145+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.146666+0000) 2019-10-13T07:11:27.149 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:27.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.146548+0000) 2019-10-13T07:11:27.150 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:27.145+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.147654+0000) 2019-10-13T07:11:27.151 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:27.145+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.149322+0000) 2019-10-13T07:11:27.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:27.155+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.156862+0000) 2019-10-13T07:11:27.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:27.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.479089+0000) 2019-10-13T07:11:27.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:27.481+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:27.481768+0000) 2019-10-13T07:11:30.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:30.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.608949+0000) 2019-10-13T07:11:30.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:30.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:30.707966+0000) 2019-10-13T07:11:31.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:31.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.505110+0000) 2019-10-13T07:11:31.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:31.503+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.504975+0000) 2019-10-13T07:11:31.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:31.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:31.847778+0000) 2019-10-13T07:11:32.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:32.146+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.151844+0000) 2019-10-13T07:11:32.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:32.145+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.149525+0000) 2019-10-13T07:11:32.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:32.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:32.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:32.149+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.150050+0000) 2019-10-13T07:11:32.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:32.149+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.150383+0000) 2019-10-13T07:11:32.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:32.149+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.150687+0000) 2019-10-13T07:11:32.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:32.149+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.151363+0000) 2019-10-13T07:11:32.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:32.149+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.153548+0000) 2019-10-13T07:11:32.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:32.158+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:32.163408+0000) 2019-10-13T07:11:37.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:37.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.066367+0000) 2019-10-13T07:11:37.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:37.148+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.153519+0000) 2019-10-13T07:11:37.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:37.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:37.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:37.152+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.155121+0000) 2019-10-13T07:11:37.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:37.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.155305+0000) 2019-10-13T07:11:37.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:37.152+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.155686+0000) 2019-10-13T07:11:37.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:37.154+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.156504+0000) 2019-10-13T07:11:37.161 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:37.156+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.159007+0000) 2019-10-13T07:11:37.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:37.156+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.159771+0000) 2019-10-13T07:11:37.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:37.162+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.164590+0000) 2019-10-13T07:11:37.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:37.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.479300+0000) 2019-10-13T07:11:37.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:37.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:37.481944+0000) 2019-10-13T07:11:40.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:40.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.609155+0000) 2019-10-13T07:11:40.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:40.704+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:40.708160+0000) 2019-10-13T07:11:41.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:41.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.505299+0000) 2019-10-13T07:11:41.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:41.502+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.505203+0000) 2019-10-13T07:11:41.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:41.850+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:41.854181+0000) 2019-10-13T07:11:42.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:42.152+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.157314+0000) 2019-10-13T07:11:42.160 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:42.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:42.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:42.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.158871+0000) 2019-10-13T07:11:42.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:42.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.159114+0000) 2019-10-13T07:11:42.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:42.154+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.159629+0000) 2019-10-13T07:11:42.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:42.156+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.161505+0000) 2019-10-13T07:11:42.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:42.160+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.162373+0000) 2019-10-13T07:11:42.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:42.160+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.163123+0000) 2019-10-13T07:11:42.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:42.162+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:42.165260+0000) 2019-10-13T07:11:47.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:47.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.066580+0000) 2019-10-13T07:11:47.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:47.158+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.161666+0000) 2019-10-13T07:11:47.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:47.160+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.162624+0000) 2019-10-13T07:11:47.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:47.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:47.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:47.160+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.162976+0000) 2019-10-13T07:11:47.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:47.160+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.164701+0000) 2019-10-13T07:11:47.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:47.160+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.165670+0000) 2019-10-13T07:11:47.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:47.160+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.163969+0000) 2019-10-13T07:11:47.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:47.160+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.165201+0000) 2019-10-13T07:11:47.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:47.170+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.172219+0000) 2019-10-13T07:11:47.451 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:11:47.452 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:47.456 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:11:47.456 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:11:47.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:47.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.479490+0000) 2019-10-13T07:11:47.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:47.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:47.482074+0000) 2019-10-13T07:11:50.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:50.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.609359+0000) 2019-10-13T07:11:50.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:50.704+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:50.708361+0000) 2019-10-13T07:11:51.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:51.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.505495+0000) 2019-10-13T07:11:51.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:51.502+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.505398+0000) 2019-10-13T07:11:51.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:51.862+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:51.865778+0000) 2019-10-13T07:11:52.169 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:52.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45314 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:52.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:52.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.167347+0000) 2019-10-13T07:11:52.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:52.164+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.167962+0000) 2019-10-13T07:11:52.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:52.166+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.168244+0000) 2019-10-13T07:11:52.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:52.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.169033+0000) 2019-10-13T07:11:52.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:52.164+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.169612+0000) 2019-10-13T07:11:52.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:52.164+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.169572+0000) 2019-10-13T07:11:52.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:52.168+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.171434+0000) 2019-10-13T07:11:52.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:52.174+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:52.177792+0000) 2019-10-13T07:11:57.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:57.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.066774+0000) 2019-10-13T07:11:57.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:11:57.166+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.169748+0000) 2019-10-13T07:11:57.172 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:57.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.170858+0000) 2019-10-13T07:11:57.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:11:57.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:11:57.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:57.168+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.171465+0000) 2019-10-13T07:11:57.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:11:57.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.172100+0000) 2019-10-13T07:11:57.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:11:57.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.172617+0000) 2019-10-13T07:11:57.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:11:57.168+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.172877+0000) 2019-10-13T07:11:57.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:11:57.172+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.174048+0000) 2019-10-13T07:11:57.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:11:57.174+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.176839+0000) 2019-10-13T07:11:57.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:11:57.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.479667+0000) 2019-10-13T07:11:57.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:11:57.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:11:57.482266+0000) 2019-10-13T07:12:00.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:00.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.609563+0000) 2019-10-13T07:12:00.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:00.704+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:00.708518+0000) 2019-10-13T07:12:01.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:01.502+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.505683+0000) 2019-10-13T07:12:01.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:01.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.505718+0000) 2019-10-13T07:12:01.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:01.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:01.872192+0000) 2019-10-13T07:12:02.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:02.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.174755+0000) 2019-10-13T07:12:02.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:02.172+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.174682+0000) 2019-10-13T07:12:02.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:02.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:02.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:02.172+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.174956+0000) 2019-10-13T07:12:02.178 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:02.172+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.175720+0000) 2019-10-13T07:12:02.181 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:02.172+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.176837+0000) 2019-10-13T07:12:02.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:02.172+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.177706+0000) 2019-10-13T07:12:02.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:02.172+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.177275+0000) 2019-10-13T07:12:02.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:02.178+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:02.181092+0000) 2019-10-13T07:12:07.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:07.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.066941+0000) 2019-10-13T07:12:07.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:07.174+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.178266+0000) 2019-10-13T07:12:07.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:07.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:07.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:07.176+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.179042+0000) 2019-10-13T07:12:07.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:07.176+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.179475+0000) 2019-10-13T07:12:07.182 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:07.176+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.180313+0000) 2019-10-13T07:12:07.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:07.176+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.181010+0000) 2019-10-13T07:12:07.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:07.176+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.181564+0000) 2019-10-13T07:12:07.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:07.180+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.182659+0000) 2019-10-13T07:12:07.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:07.182+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.187577+0000) 2019-10-13T07:12:07.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:07.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.479878+0000) 2019-10-13T07:12:07.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:07.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:07.482450+0000) 2019-10-13T07:12:10.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:10.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.609789+0000) 2019-10-13T07:12:10.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:10.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:10.708686+0000) 2019-10-13T07:12:11.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:11.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.505951+0000) 2019-10-13T07:12:11.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:11.502+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.505895+0000) 2019-10-13T07:12:11.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:11.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:11.883837+0000) 2019-10-13T07:12:12.183 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:12.178+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.181228+0000) 2019-10-13T07:12:12.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:12.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.182606+0000) 2019-10-13T07:12:12.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:12.180+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.183298+0000) 2019-10-13T07:12:12.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:12.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:12.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:12.180+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.183743+0000) 2019-10-13T07:12:12.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:12.180+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.184719+0000) 2019-10-13T07:12:12.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:12.180+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.185585+0000) 2019-10-13T07:12:12.188 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:12.184+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.186338+0000) 2019-10-13T07:12:12.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:12.186+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:12.191071+0000) 2019-10-13T07:12:17.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:17.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.067163+0000) 2019-10-13T07:12:17.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:17.182+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.186216+0000) 2019-10-13T07:12:17.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:17.184+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.187135+0000) 2019-10-13T07:12:17.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:17.184+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.187502+0000) 2019-10-13T07:12:17.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:17.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:17.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:17.184+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.187929+0000) 2019-10-13T07:12:17.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:17.184+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.188520+0000) 2019-10-13T07:12:17.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:17.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.189502+0000) 2019-10-13T07:12:17.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:17.188+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.190285+0000) 2019-10-13T07:12:17.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:17.190+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.196042+0000) 2019-10-13T07:12:17.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:17.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.480077+0000) 2019-10-13T07:12:17.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:17.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:17.482632+0000) 2019-10-13T07:12:17.496 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:12:17.496 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:17.500 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:12:17.500 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:20.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:20.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.610008+0000) 2019-10-13T07:12:20.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:20.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:20.708871+0000) 2019-10-13T07:12:21.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:21.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.506144+0000) 2019-10-13T07:12:21.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:21.502+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.506091+0000) 2019-10-13T07:12:21.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:21.886+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:21.890262+0000) 2019-10-13T07:12:22.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:22.188+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.190451+0000) 2019-10-13T07:12:22.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:22.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:22.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:22.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.192617+0000) 2019-10-13T07:12:22.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:22.190+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.192610+0000) 2019-10-13T07:12:22.195 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:22.188+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.193485+0000) 2019-10-13T07:12:22.195 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:22.192+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.193881+0000) 2019-10-13T07:12:22.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:22.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.194583+0000) 2019-10-13T07:12:22.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:22.192+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.196225+0000) 2019-10-13T07:12:22.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:22.198+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:22.200314+0000) 2019-10-13T07:12:27.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:27.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.067363+0000) 2019-10-13T07:12:27.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:27.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:27.199 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:27.190+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.196013+0000) 2019-10-13T07:12:27.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:27.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.197095+0000) 2019-10-13T07:12:27.199 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:27.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.197600+0000) 2019-10-13T07:12:27.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:27.196+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.198120+0000) 2019-10-13T07:12:27.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:27.196+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.198378+0000) 2019-10-13T07:12:27.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:27.196+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.199297+0000) 2019-10-13T07:12:27.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:27.196+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.200908+0000) 2019-10-13T07:12:27.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:27.198+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.203868+0000) 2019-10-13T07:12:27.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:27.480+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.480260+0000) 2019-10-13T07:12:27.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:27.480+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:27.482807+0000) 2019-10-13T07:12:30.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:30.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.610209+0000) 2019-10-13T07:12:30.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:30.704+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:30.709070+0000) 2019-10-13T07:12:31.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:31.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.506337+0000) 2019-10-13T07:12:31.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:31.506+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.506321+0000) 2019-10-13T07:12:31.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:31.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:31.896620+0000) 2019-10-13T07:12:32.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:32.198+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.201520+0000) 2019-10-13T07:12:32.208 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:32.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.199610+0000) 2019-10-13T07:12:32.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:32.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:32.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:32.196+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.200573+0000) 2019-10-13T07:12:32.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:32.196+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.200956+0000) 2019-10-13T07:12:32.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:32.200+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.201800+0000) 2019-10-13T07:12:32.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:32.200+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.202837+0000) 2019-10-13T07:12:32.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:32.200+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.203827+0000) 2019-10-13T07:12:32.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:32.206+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:32.210356+0000) 2019-10-13T07:12:37.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:37.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.067593+0000) 2019-10-13T07:12:37.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:37.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:37.207 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:37.202+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.204982+0000) 2019-10-13T07:12:37.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:37.199+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.205496+0000) 2019-10-13T07:12:37.208 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:37.203+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.206425+0000) 2019-10-13T07:12:37.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:37.203+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.206644+0000) 2019-10-13T07:12:37.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:37.203+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.207029+0000) 2019-10-13T07:12:37.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:37.203+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.208003+0000) 2019-10-13T07:12:37.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:37.203+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.208228+0000) 2019-10-13T07:12:37.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:37.206+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.212070+0000) 2019-10-13T07:12:37.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:37.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.480470+0000) 2019-10-13T07:12:37.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:37.479+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:37.483017+0000) 2019-10-13T07:12:40.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:40.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.610453+0000) 2019-10-13T07:12:40.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:40.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:40.709250+0000) 2019-10-13T07:12:41.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:41.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.506567+0000) 2019-10-13T07:12:41.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:41.505+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.506522+0000) 2019-10-13T07:12:41.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:41.897+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:41.903014+0000) 2019-10-13T07:12:42.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:42.203+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.207904+0000) 2019-10-13T07:12:42.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:42.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:42.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:42.203+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.209521+0000) 2019-10-13T07:12:42.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:42.207+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.209923+0000) 2019-10-13T07:12:42.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:42.205+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.211981+0000) 2019-10-13T07:12:42.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:42.207+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.213114+0000) 2019-10-13T07:12:42.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:42.207+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.213615+0000) 2019-10-13T07:12:42.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:42.211+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.214595+0000) 2019-10-13T07:12:42.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:42.217+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:42.220514+0000) 2019-10-13T07:12:47.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:47.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.067816+0000) 2019-10-13T07:12:47.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:47.207+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:47.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:47.207+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.213016+0000) 2019-10-13T07:12:47.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:47.207+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.213396+0000) 2019-10-13T07:12:47.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:47.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.213723+0000) 2019-10-13T07:12:47.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:47.211+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.213809+0000) 2019-10-13T07:12:47.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:47.211+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.214618+0000) 2019-10-13T07:12:47.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:47.211+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.214949+0000) 2019-10-13T07:12:47.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:47.211+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.215993+0000) 2019-10-13T07:12:47.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:47.213+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.218321+0000) 2019-10-13T07:12:47.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:47.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.480645+0000) 2019-10-13T07:12:47.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:47.479+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:47.483172+0000) 2019-10-13T07:12:47.643 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:12:47.644 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:47.647 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:12:47.647 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:12:50.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:50.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.610676+0000) 2019-10-13T07:12:50.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:50.703+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:50.709439+0000) 2019-10-13T07:12:51.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:51.505+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.506722+0000) 2019-10-13T07:12:51.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:51.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.506762+0000) 2019-10-13T07:12:51.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:51.905+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:51.909124+0000) 2019-10-13T07:12:52.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:52.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.215974+0000) 2019-10-13T07:12:52.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:52.211+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:52.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:52.211+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.217143+0000) 2019-10-13T07:12:52.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:52.211+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.217576+0000) 2019-10-13T07:12:52.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:52.215+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.217922+0000) 2019-10-13T07:12:52.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:52.215+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.218517+0000) 2019-10-13T07:12:52.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:52.215+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.219316+0000) 2019-10-13T07:12:52.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:52.215+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.219867+0000) 2019-10-13T07:12:52.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:52.221+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:52.224388+0000) 2019-10-13T07:12:57.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:57.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.068022+0000) 2019-10-13T07:12:57.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:57.215+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.220324+0000) 2019-10-13T07:12:57.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:12:57.215+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.221115+0000) 2019-10-13T07:12:57.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:57.215+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.221131+0000) 2019-10-13T07:12:57.223 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:12:57.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:12:57.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:12:57.215+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.221553+0000) 2019-10-13T07:12:57.224 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:12:57.219+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.222144+0000) 2019-10-13T07:12:57.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:12:57.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.222500+0000) 2019-10-13T07:12:57.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:12:57.217+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.222562+0000) 2019-10-13T07:12:57.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:12:57.225+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.229100+0000) 2019-10-13T07:12:57.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:12:57.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.480824+0000) 2019-10-13T07:12:57.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:12:57.479+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:12:57.483352+0000) 2019-10-13T07:13:00.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:00.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.610882+0000) 2019-10-13T07:13:00.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:00.707+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:00.709672+0000) 2019-10-13T07:13:01.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:01.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.506954+0000) 2019-10-13T07:13:01.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:01.505+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.506942+0000) 2019-10-13T07:13:01.924 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:01.917+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:01.920733+0000) 2019-10-13T07:13:02.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:02.219+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.224651+0000) 2019-10-13T07:13:02.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:02.219+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.224997+0000) 2019-10-13T07:13:02.227 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:02.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.225309+0000) 2019-10-13T07:13:02.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:02.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:02.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:02.223+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.226459+0000) 2019-10-13T07:13:02.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:02.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.226739+0000) 2019-10-13T07:13:02.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:02.223+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.227841+0000) 2019-10-13T07:13:02.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:02.225+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.229512+0000) 2019-10-13T07:13:02.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:02.229+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:02.232895+0000) 2019-10-13T07:13:07.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:07.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.068227+0000) 2019-10-13T07:13:07.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:07.223+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.228708+0000) 2019-10-13T07:13:07.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:07.223+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.229139+0000) 2019-10-13T07:13:07.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:07.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.229443+0000) 2019-10-13T07:13:07.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:07.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:07.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:07.227+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.230604+0000) 2019-10-13T07:13:07.233 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:07.227+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.231359+0000) 2019-10-13T07:13:07.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:07.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.233190+0000) 2019-10-13T07:13:07.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:07.227+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.232865+0000) 2019-10-13T07:13:07.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:07.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.237208+0000) 2019-10-13T07:13:07.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:07.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.481004+0000) 2019-10-13T07:13:07.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:07.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:07.483550+0000) 2019-10-13T07:13:10.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:10.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.611097+0000) 2019-10-13T07:13:10.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:10.707+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:10.709898+0000) 2019-10-13T07:13:11.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:11.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.507174+0000) 2019-10-13T07:13:11.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:11.501+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.507132+0000) 2019-10-13T07:13:11.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:11.925+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:11.927127+0000) 2019-10-13T07:13:12.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:12.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:12.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:12.231+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.234576+0000) 2019-10-13T07:13:12.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:12.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.234458+0000) 2019-10-13T07:13:12.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:12.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.235380+0000) 2019-10-13T07:13:12.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:12.231+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.236033+0000) 2019-10-13T07:13:12.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:12.231+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.236215+0000) 2019-10-13T07:13:12.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:12.231+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.236881+0000) 2019-10-13T07:13:12.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:12.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.235939+0000) 2019-10-13T07:13:12.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:12.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:12.239659+0000) 2019-10-13T07:13:17.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:17.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.068493+0000) 2019-10-13T07:13:17.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:17.231+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.237622+0000) 2019-10-13T07:13:17.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:17.235+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.238117+0000) 2019-10-13T07:13:17.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:17.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.238542+0000) 2019-10-13T07:13:17.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:17.235+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:17.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:17.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.239561+0000) 2019-10-13T07:13:17.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:17.233+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.239510+0000) 2019-10-13T07:13:17.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:17.239+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.242201+0000) 2019-10-13T07:13:17.247 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:17.241+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.244788+0000) 2019-10-13T07:13:17.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:17.239+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.242012+0000) 2019-10-13T07:13:17.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:17.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.481185+0000) 2019-10-13T07:13:17.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:17.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:17.483732+0000) 2019-10-13T07:13:17.867 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:13:17.867 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:17.870 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:13:17.871 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:20.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:20.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.611314+0000) 2019-10-13T07:13:20.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:20.707+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:20.710106+0000) 2019-10-13T07:13:21.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:21.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.507398+0000) 2019-10-13T07:13:21.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:21.501+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.507325+0000) 2019-10-13T07:13:21.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:21.929+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:21.933806+0000) 2019-10-13T07:13:22.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:22.239+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.242408+0000) 2019-10-13T07:13:22.245 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:22.239+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.242896+0000) 2019-10-13T07:13:22.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:22.239+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.243281+0000) 2019-10-13T07:13:22.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:22.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:22.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:22.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.243563+0000) 2019-10-13T07:13:22.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:22.239+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.244282+0000) 2019-10-13T07:13:22.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:22.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.244900+0000) 2019-10-13T07:13:22.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:22.243+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.245890+0000) 2019-10-13T07:13:22.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:22.249+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:22.253392+0000) 2019-10-13T07:13:27.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:27.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.068699+0000) 2019-10-13T07:13:27.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:27.243+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.246573+0000) 2019-10-13T07:13:27.250 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:27.241+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.246436+0000) 2019-10-13T07:13:27.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:27.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45890 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:27.251 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:27.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.247297+0000) 2019-10-13T07:13:27.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:27.243+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.248302+0000) 2019-10-13T07:13:27.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:27.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.249665+0000) 2019-10-13T07:13:27.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:27.247+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.252179+0000) 2019-10-13T07:13:27.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:27.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.253055+0000) 2019-10-13T07:13:27.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:27.249+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.254527+0000) 2019-10-13T07:13:27.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:27.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.481360+0000) 2019-10-13T07:13:27.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:27.479+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:27.483906+0000) 2019-10-13T07:13:30.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:30.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.611519+0000) 2019-10-13T07:13:30.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:30.706+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:30.710367+0000) 2019-10-13T07:13:31.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:31.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.507547+0000) 2019-10-13T07:13:31.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:31.501+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.507518+0000) 2019-10-13T07:13:31.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:31.937+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:31.940268+0000) 2019-10-13T07:13:32.252 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:32.246+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.250658+0000) 2019-10-13T07:13:32.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:32.246+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.251025+0000) 2019-10-13T07:13:32.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:32.246+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.251654+0000) 2019-10-13T07:13:32.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:32.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.253103+0000) 2019-10-13T07:13:32.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:32.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:32.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:32.246+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.253262+0000) 2019-10-13T07:13:32.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:32.250+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.254371+0000) 2019-10-13T07:13:32.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:32.250+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.255700+0000) 2019-10-13T07:13:32.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:32.257+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:32.261272+0000) 2019-10-13T07:13:37.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:37.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.068904+0000) 2019-10-13T07:13:37.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:37.250+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.256005+0000) 2019-10-13T07:13:37.261 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:37.253+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.257191+0000) 2019-10-13T07:13:37.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:37.250+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.257088+0000) 2019-10-13T07:13:37.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:37.250+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.257360+0000) 2019-10-13T07:13:37.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:37.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:37.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:37.254+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.257993+0000) 2019-10-13T07:13:37.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:37.254+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.258705+0000) 2019-10-13T07:13:37.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:37.254+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.260393+0000) 2019-10-13T07:13:37.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:37.257+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.263275+0000) 2019-10-13T07:13:37.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:37.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.481535+0000) 2019-10-13T07:13:37.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:37.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:37.484082+0000) 2019-10-13T07:13:40.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:40.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.611756+0000) 2019-10-13T07:13:40.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:40.706+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:40.710539+0000) 2019-10-13T07:13:41.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:41.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.507738+0000) 2019-10-13T07:13:41.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:41.505+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.507740+0000) 2019-10-13T07:13:41.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:41.941+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:41.946649+0000) 2019-10-13T07:13:42.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:42.254+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.260596+0000) 2019-10-13T07:13:42.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:42.257+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.262922+0000) 2019-10-13T07:13:42.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:42.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.261977+0000) 2019-10-13T07:13:42.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:42.258+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.262387+0000) 2019-10-13T07:13:42.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:42.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:42.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:42.258+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.264354+0000) 2019-10-13T07:13:42.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:42.258+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.265257+0000) 2019-10-13T07:13:42.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:42.258+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.264943+0000) 2019-10-13T07:13:42.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:42.265+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:42.271548+0000) 2019-10-13T07:13:47.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:47.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.069108+0000) 2019-10-13T07:13:47.268 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:47.262+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.266438+0000) 2019-10-13T07:13:47.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:47.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:47.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:47.262+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.267448+0000) 2019-10-13T07:13:47.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:47.262+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.268586+0000) 2019-10-13T07:13:47.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:47.262+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.269523+0000) 2019-10-13T07:13:47.272 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:47.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.269472+0000) 2019-10-13T07:13:47.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:47.266+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.270267+0000) 2019-10-13T07:13:47.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:47.266+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.270871+0000) 2019-10-13T07:13:47.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:47.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.274831+0000) 2019-10-13T07:13:47.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:47.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.481712+0000) 2019-10-13T07:13:47.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:47.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:47.484265+0000) 2019-10-13T07:13:47.923 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:13:47.923 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:47.927 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:13:47.927 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:13:50.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:50.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.611957+0000) 2019-10-13T07:13:52.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:51.504+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.507931+0000) 2019-10-13T07:13:52.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:52.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.272185+0000) 2019-10-13T07:13:52.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:51.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.953093+0000) 2019-10-13T07:13:52.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:52.272+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.276705+0000) 2019-10-13T07:13:52.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:50.706+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:50.710758+0000) 2019-10-13T07:13:52.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:52.270+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.274464+0000) 2019-10-13T07:13:52.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:51.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:51.507948+0000) 2019-10-13T07:13:52.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:52.266+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.272336+0000) 2019-10-13T07:13:52.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:52.266+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.270712+0000) 2019-10-13T07:13:52.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:52.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.271680+0000) 2019-10-13T07:13:52.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:52.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:52.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:52.266+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.273029+0000) 2019-10-13T07:13:52.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:52.266+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:52.273279+0000) 2019-10-13T07:13:57.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:57.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.069307+0000) 2019-10-13T07:13:57.277 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:57.270+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.275179+0000) 2019-10-13T07:13:57.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:13:57.270+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.275734+0000) 2019-10-13T07:13:57.278 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:13:57.270+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.275985+0000) 2019-10-13T07:13:57.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:13:57.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:13:57.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:13:57.272+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.276418+0000) 2019-10-13T07:13:57.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:57.270+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.277297+0000) 2019-10-13T07:13:57.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:13:57.274+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.278991+0000) 2019-10-13T07:13:57.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:13:57.274+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.278232+0000) 2019-10-13T07:13:57.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:13:57.280+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.284575+0000) 2019-10-13T07:13:57.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:13:57.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.481874+0000) 2019-10-13T07:13:57.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:13:57.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:13:57.484485+0000) 2019-10-13T07:14:00.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:00.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.612194+0000) 2019-10-13T07:14:00.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:00.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:00.710987+0000) 2019-10-13T07:14:01.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:01.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.508150+0000) 2019-10-13T07:14:01.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:01.504+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.508133+0000) 2019-10-13T07:14:01.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:01.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:01.959457+0000) 2019-10-13T07:14:02.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:02.274+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.279904+0000) 2019-10-13T07:14:02.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:02.274+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.280742+0000) 2019-10-13T07:14:02.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:02.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:02.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:02.274+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.281499+0000) 2019-10-13T07:14:02.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:02.278+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.281878+0000) 2019-10-13T07:14:02.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:02.278+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.282321+0000) 2019-10-13T07:14:02.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:02.278+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.282838+0000) 2019-10-13T07:14:02.285 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:02.276+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.282869+0000) 2019-10-13T07:14:02.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:02.284+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:02.289242+0000) 2019-10-13T07:14:07.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:07.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.069517+0000) 2019-10-13T07:14:07.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:07.278+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.284173+0000) 2019-10-13T07:14:07.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:07.278+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.284867+0000) 2019-10-13T07:14:07.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:07.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:07.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:07.278+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.285087+0000) 2019-10-13T07:14:07.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:07.278+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.285416+0000) 2019-10-13T07:14:07.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:07.282+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.286099+0000) 2019-10-13T07:14:07.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:07.282+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.286485+0000) 2019-10-13T07:14:07.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:07.280+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.286287+0000) 2019-10-13T07:14:07.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:07.288+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.292887+0000) 2019-10-13T07:14:07.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:07.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.482074+0000) 2019-10-13T07:14:07.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:07.478+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:07.484671+0000) 2019-10-13T07:14:10.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:10.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.612415+0000) 2019-10-13T07:14:10.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:10.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:10.711186+0000) 2019-10-13T07:14:11.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:11.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.508389+0000) 2019-10-13T07:14:11.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:11.504+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.508328+0000) 2019-10-13T07:14:11.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:11.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:11.965893+0000) 2019-10-13T07:14:12.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:12.282+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.287853+0000) 2019-10-13T07:14:12.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:12.282+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.288932+0000) 2019-10-13T07:14:12.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:12.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:12.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:12.282+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.289333+0000) 2019-10-13T07:14:12.292 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:12.284+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.289841+0000) 2019-10-13T07:14:12.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:12.286+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.290206+0000) 2019-10-13T07:14:12.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:12.286+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.291097+0000) 2019-10-13T07:14:12.295 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:12.286+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.292133+0000) 2019-10-13T07:14:12.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:12.292+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:12.297263+0000) 2019-10-13T07:14:17.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:17.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.069712+0000) 2019-10-13T07:14:17.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:17.286+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.292444+0000) 2019-10-13T07:14:17.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:17.286+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.292741+0000) 2019-10-13T07:14:17.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:17.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:17.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:17.290+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.294092+0000) 2019-10-13T07:14:17.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:17.288+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.293871+0000) 2019-10-13T07:14:17.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:17.290+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.294721+0000) 2019-10-13T07:14:17.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:17.290+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.295249+0000) 2019-10-13T07:14:17.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:17.290+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.295518+0000) 2019-10-13T07:14:17.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:17.296+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.300158+0000) 2019-10-13T07:14:17.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:17.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.482274+0000) 2019-10-13T07:14:17.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:17.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:17.484845+0000) 2019-10-13T07:14:18.183 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:14:18.184 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:18.187 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:14:18.187 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:20.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:20.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.612616+0000) 2019-10-13T07:14:20.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:20.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:20.711392+0000) 2019-10-13T07:14:21.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:21.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.508619+0000) 2019-10-13T07:14:21.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:21.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.508518+0000) 2019-10-13T07:14:21.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:21.968+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:21.972295+0000) 2019-10-13T07:14:22.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:22.290+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.296759+0000) 2019-10-13T07:14:22.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:22.290+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.297355+0000) 2019-10-13T07:14:22.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:22.290+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:22.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:22.294+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.298072+0000) 2019-10-13T07:14:22.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:22.294+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.298582+0000) 2019-10-13T07:14:22.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:22.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.298553+0000) 2019-10-13T07:14:22.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:22.294+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.300053+0000) 2019-10-13T07:14:22.304 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:22.298+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.302031+0000) 2019-10-13T07:14:22.309 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:22.300+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:22.306930+0000) 2019-10-13T07:14:27.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:27.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.069910+0000) 2019-10-13T07:14:27.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:27.294+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:27.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:27.298+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.302157+0000) 2019-10-13T07:14:27.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:27.296+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.302779+0000) 2019-10-13T07:14:27.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:27.298+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.303945+0000) 2019-10-13T07:14:27.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:27.298+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.304608+0000) 2019-10-13T07:14:27.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:27.298+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.305031+0000) 2019-10-13T07:14:27.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:27.302+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.306181+0000) 2019-10-13T07:14:27.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:27.298+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.305411+0000) 2019-10-13T07:14:27.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:27.304+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.309539+0000) 2019-10-13T07:14:27.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:27.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.482449+0000) 2019-10-13T07:14:27.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:27.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:27.485019+0000) 2019-10-13T07:14:30.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:30.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.612815+0000) 2019-10-13T07:14:30.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:30.705+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:30.711607+0000) 2019-10-13T07:14:31.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:31.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.508815+0000) 2019-10-13T07:14:31.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:31.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.508730+0000) 2019-10-13T07:14:31.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:31.972+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:31.978710+0000) 2019-10-13T07:14:32.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:32.300+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.304722+0000) 2019-10-13T07:14:32.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:32.301+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.305999+0000) 2019-10-13T07:14:32.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:32.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:32.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:32.301+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.306489+0000) 2019-10-13T07:14:32.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:32.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.307005+0000) 2019-10-13T07:14:32.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:32.301+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.307455+0000) 2019-10-13T07:14:32.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:32.301+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.307816+0000) 2019-10-13T07:14:32.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:32.301+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.308299+0000) 2019-10-13T07:14:32.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:32.308+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:32.314920+0000) 2019-10-13T07:14:37.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:37.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.070129+0000) 2019-10-13T07:14:37.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:37.305+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.309187+0000) 2019-10-13T07:14:37.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:37.305+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.310520+0000) 2019-10-13T07:14:37.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:37.305+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:37.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:37.304+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.310506+0000) 2019-10-13T07:14:37.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:37.305+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.311607+0000) 2019-10-13T07:14:37.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:37.305+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.311809+0000) 2019-10-13T07:14:37.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:37.305+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.312193+0000) 2019-10-13T07:14:37.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:37.305+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.312630+0000) 2019-10-13T07:14:37.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:37.312+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.318701+0000) 2019-10-13T07:14:37.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:37.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.482629+0000) 2019-10-13T07:14:37.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:37.477+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:37.485191+0000) 2019-10-13T07:14:40.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:40.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.613019+0000) 2019-10-13T07:14:40.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:40.709+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:40.711804+0000) 2019-10-13T07:14:41.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:41.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.509040+0000) 2019-10-13T07:14:41.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:41.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.508931+0000) 2019-10-13T07:14:41.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:41.984+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:41.985132+0000) 2019-10-13T07:14:42.316 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:42.309+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.314117+0000) 2019-10-13T07:14:42.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:42.309+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.314338+0000) 2019-10-13T07:14:42.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:42.309+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.315443+0000) 2019-10-13T07:14:42.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:42.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:42.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:42.308+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.315105+0000) 2019-10-13T07:14:42.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:42.309+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.316423+0000) 2019-10-13T07:14:42.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:42.309+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.316790+0000) 2019-10-13T07:14:42.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:42.313+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.318126+0000) 2019-10-13T07:14:42.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:42.320+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:42.325198+0000) 2019-10-13T07:14:47.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:47.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.070337+0000) 2019-10-13T07:14:47.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:47.317+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.318378+0000) 2019-10-13T07:14:47.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:47.317+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.319163+0000) 2019-10-13T07:14:47.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:47.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:47.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:47.317+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.320343+0000) 2019-10-13T07:14:47.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:47.317+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.321069+0000) 2019-10-13T07:14:47.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:47.321+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.322037+0000) 2019-10-13T07:14:47.324 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:47.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.322257+0000) 2019-10-13T07:14:47.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:47.320+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.323413+0000) 2019-10-13T07:14:47.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:47.324+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.326627+0000) 2019-10-13T07:14:47.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:47.481+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.482809+0000) 2019-10-13T07:14:47.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:47.481+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:47.485398+0000) 2019-10-13T07:14:48.255 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:14:48.256 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:48.259 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:14:48.259 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:14:50.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:50.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.613217+0000) 2019-10-13T07:14:50.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:50.709+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:50.712034+0000) 2019-10-13T07:14:51.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:51.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.509238+0000) 2019-10-13T07:14:51.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:51.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.509153+0000) 2019-10-13T07:14:51.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:51.987+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:51.991472+0000) 2019-10-13T07:14:52.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:52.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:52.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:52.321+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.324470+0000) 2019-10-13T07:14:52.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:52.321+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.325010+0000) 2019-10-13T07:14:52.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:52.321+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.325610+0000) 2019-10-13T07:14:52.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:52.325+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.326179+0000) 2019-10-13T07:14:52.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:52.323+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.326506+0000) 2019-10-13T07:14:52.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:52.325+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.326544+0000) 2019-10-13T07:14:52.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:52.325+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.327768+0000) 2019-10-13T07:14:52.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:52.331+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:52.333077+0000) 2019-10-13T07:14:57.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:57.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.070555+0000) 2019-10-13T07:14:57.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:14:57.325+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:14:57.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:14:57.325+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.328714+0000) 2019-10-13T07:14:57.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:57.325+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.329094+0000) 2019-10-13T07:14:57.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:57.325+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.329449+0000) 2019-10-13T07:14:57.332 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:14:57.327+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.329488+0000) 2019-10-13T07:14:57.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:14:57.329+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.330503+0000) 2019-10-13T07:14:57.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:14:57.329+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.331761+0000) 2019-10-13T07:14:57.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:14:57.329+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.332484+0000) 2019-10-13T07:14:57.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:14:57.335+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.339208+0000) 2019-10-13T07:14:57.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:14:57.481+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.483027+0000) 2019-10-13T07:14:57.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:14:57.481+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:14:57.485574+0000) 2019-10-13T07:15:00.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:00.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.613417+0000) 2019-10-13T07:15:00.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:00.709+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:00.712236+0000) 2019-10-13T07:15:01.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:01.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.509436+0000) 2019-10-13T07:15:01.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:01.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.509354+0000) 2019-10-13T07:15:02.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:01.995+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:01.997938+0000) 2019-10-13T07:15:02.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:02.329+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.332029+0000) 2019-10-13T07:15:02.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:02.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:02.335 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:02.329+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.332573+0000) 2019-10-13T07:15:02.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:02.329+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.333211+0000) 2019-10-13T07:15:02.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:02.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.332917+0000) 2019-10-13T07:15:02.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:02.329+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.333586+0000) 2019-10-13T07:15:02.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:02.333+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.334013+0000) 2019-10-13T07:15:02.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:02.333+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.335190+0000) 2019-10-13T07:15:02.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:02.339+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:02.340450+0000) 2019-10-13T07:15:07.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:07.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.070775+0000) 2019-10-13T07:15:07.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:07.333+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.335968+0000) 2019-10-13T07:15:07.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:07.333+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.336594+0000) 2019-10-13T07:15:07.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:07.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:07.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:07.333+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.337342+0000) 2019-10-13T07:15:07.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:07.333+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.337678+0000) 2019-10-13T07:15:07.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:07.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.338284+0000) 2019-10-13T07:15:07.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:07.337+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.339622+0000) 2019-10-13T07:15:07.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:07.337+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.339914+0000) 2019-10-13T07:15:07.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:07.339+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.342920+0000) 2019-10-13T07:15:07.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:07.481+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.483219+0000) 2019-10-13T07:15:07.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:07.485+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:07.485777+0000) 2019-10-13T07:15:10.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:10.613+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.613604+0000) 2019-10-13T07:15:10.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:10.709+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:10.712430+0000) 2019-10-13T07:15:11.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:11.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:11.509657+0000) 2019-10-13T07:15:11.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:11.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:11.509581+0000) 2019-10-13T07:15:12.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:12.003+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.004344+0000) 2019-10-13T07:15:12.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:12.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:12.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:12.337+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.341264+0000) 2019-10-13T07:15:12.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:12.341+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.342157+0000) 2019-10-13T07:15:12.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:12.339+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.341923+0000) 2019-10-13T07:15:12.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:12.341+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.343250+0000) 2019-10-13T07:15:12.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:12.341+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.343744+0000) 2019-10-13T07:15:12.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:12.341+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.344068+0000) 2019-10-13T07:15:12.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:12.341+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.344306+0000) 2019-10-13T07:15:12.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:12.347+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:12.348866+0000) 2019-10-13T07:15:17.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:17.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.070939+0000) 2019-10-13T07:15:17.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:17.341+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.344711+0000) 2019-10-13T07:15:17.347 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:17.341+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.344313+0000) 2019-10-13T07:15:17.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:17.341+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:17.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:17.341+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.345630+0000) 2019-10-13T07:15:17.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:17.345+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.346149+0000) 2019-10-13T07:15:17.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:17.345+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.347191+0000) 2019-10-13T07:15:17.349 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:17.345+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.347494+0000) 2019-10-13T07:15:17.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:17.343+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.347889+0000) 2019-10-13T07:15:17.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:17.351+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.355058+0000) 2019-10-13T07:15:17.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:17.481+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.483427+0000) 2019-10-13T07:15:17.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:17.485+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:17.485982+0000) 2019-10-13T07:15:18.376 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:15:18.376 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:18.380 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:15:18.381 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:20.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:20.613+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.613808+0000) 2019-10-13T07:15:20.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:20.709+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:20.712623+0000) 2019-10-13T07:15:21.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:21.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:21.509852+0000) 2019-10-13T07:15:21.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:21.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:21.509784+0000) 2019-10-13T07:15:22.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:22.007+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.010657+0000) 2019-10-13T07:15:22.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:22.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:22.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:22.345+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.349299+0000) 2019-10-13T07:15:22.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:22.345+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.349614+0000) 2019-10-13T07:15:22.352 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:22.347+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.350088+0000) 2019-10-13T07:15:22.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:22.349+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.350170+0000) 2019-10-13T07:15:22.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:22.349+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.351208+0000) 2019-10-13T07:15:22.353 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:22.349+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.351613+0000) 2019-10-13T07:15:22.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:22.349+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.351905+0000) 2019-10-13T07:15:22.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:22.355+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:22.359359+0000) 2019-10-13T07:15:27.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:27.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.071177+0000) 2019-10-13T07:15:27.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:27.348+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.352934+0000) 2019-10-13T07:15:27.355 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:27.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:27.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:27.352+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.353767+0000) 2019-10-13T07:15:27.356 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:27.352+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.354545+0000) 2019-10-13T07:15:27.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:27.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.354506+0000) 2019-10-13T07:15:27.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:27.352+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.355287+0000) 2019-10-13T07:15:27.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:27.352+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.355771+0000) 2019-10-13T07:15:27.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:27.352+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.356799+0000) 2019-10-13T07:15:27.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:27.359+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.361034+0000) 2019-10-13T07:15:27.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:27.480+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.483656+0000) 2019-10-13T07:15:27.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:27.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:27.486169+0000) 2019-10-13T07:15:30.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:30.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.614011+0000) 2019-10-13T07:15:30.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:30.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:30.712862+0000) 2019-10-13T07:15:31.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:31.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:31.510053+0000) 2019-10-13T07:15:31.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:31.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:31.509988+0000) 2019-10-13T07:15:32.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:32.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.017206+0000) 2019-10-13T07:15:32.359 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:32.352+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:32.360 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:32.356+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.358251+0000) 2019-10-13T07:15:32.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:32.356+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.358768+0000) 2019-10-13T07:15:32.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:32.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.358947+0000) 2019-10-13T07:15:32.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:32.356+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.359650+0000) 2019-10-13T07:15:32.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:32.356+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.359908+0000) 2019-10-13T07:15:32.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:32.359+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.360220+0000) 2019-10-13T07:15:32.363 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:32.356+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.360564+0000) 2019-10-13T07:15:32.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:32.363+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:32.365029+0000) 2019-10-13T07:15:37.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:37.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.071346+0000) 2019-10-13T07:15:37.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:37.356+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.360677+0000) 2019-10-13T07:15:37.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:37.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.361337+0000) 2019-10-13T07:15:37.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:37.356+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.361690+0000) 2019-10-13T07:15:37.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:37.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:37.366 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:37.360+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.362522+0000) 2019-10-13T07:15:37.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:37.360+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.362997+0000) 2019-10-13T07:15:37.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:37.360+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.363968+0000) 2019-10-13T07:15:37.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:37.363+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.364439+0000) 2019-10-13T07:15:37.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:37.367+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.370554+0000) 2019-10-13T07:15:37.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:37.480+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.483863+0000) 2019-10-13T07:15:37.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:37.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:37.486350+0000) 2019-10-13T07:15:40.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:40.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.614208+0000) 2019-10-13T07:15:40.715 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:40.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:40.713068+0000) 2019-10-13T07:15:41.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:41.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:41.510271+0000) 2019-10-13T07:15:41.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:41.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:41.510194+0000) 2019-10-13T07:15:42.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:42.019+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.023734+0000) 2019-10-13T07:15:42.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:42.364+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:42.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:42.364+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.366941+0000) 2019-10-13T07:15:42.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:42.364+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.367294+0000) 2019-10-13T07:15:42.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:42.364+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.367979+0000) 2019-10-13T07:15:42.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:42.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.368401+0000) 2019-10-13T07:15:42.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:42.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.368687+0000) 2019-10-13T07:15:42.371 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:42.364+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.369479+0000) 2019-10-13T07:15:42.372 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:42.368+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.371001+0000) 2019-10-13T07:15:42.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:42.371+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:42.375276+0000) 2019-10-13T07:15:47.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:47.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.071560+0000) 2019-10-13T07:15:47.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:47.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.369069+0000) 2019-10-13T07:15:47.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:47.368+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.371007+0000) 2019-10-13T07:15:47.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:47.368+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.371257+0000) 2019-10-13T07:15:47.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:47.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:47.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:47.368+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.372167+0000) 2019-10-13T07:15:47.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:47.368+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.372841+0000) 2019-10-13T07:15:47.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:47.368+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.373488+0000) 2019-10-13T07:15:47.376 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:47.372+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.373918+0000) 2019-10-13T07:15:47.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:47.375+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.376406+0000) 2019-10-13T07:15:47.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:47.480+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.484043+0000) 2019-10-13T07:15:47.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:47.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:47.486535+0000) 2019-10-13T07:15:48.507 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:15:48.507 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:48.511 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:15:48.511 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:15:50.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:50.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.614414+0000) 2019-10-13T07:15:50.715 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:50.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:50.713264+0000) 2019-10-13T07:15:51.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:51.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:51.510496+0000) 2019-10-13T07:15:51.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:51.507+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:51.510382+0000) 2019-10-13T07:15:52.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:52.027+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.030235+0000) 2019-10-13T07:15:52.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:52.372+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.374723+0000) 2019-10-13T07:15:52.377 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:52.372+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.375484+0000) 2019-10-13T07:15:52.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:52.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:52.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:52.375+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.376300+0000) 2019-10-13T07:15:52.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:52.372+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.377203+0000) 2019-10-13T07:15:52.379 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:52.372+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.377516+0000) 2019-10-13T07:15:52.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:52.376+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.378281+0000) 2019-10-13T07:15:52.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:52.376+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.378781+0000) 2019-10-13T07:15:52.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:52.379+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:52.381272+0000) 2019-10-13T07:15:57.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:57.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.071765+0000) 2019-10-13T07:15:57.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:57.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.380059+0000) 2019-10-13T07:15:57.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:57.376+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.380677+0000) 2019-10-13T07:15:57.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:15:57.376+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.381043+0000) 2019-10-13T07:15:57.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:15:57.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:15:57.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:15:57.380+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.382301+0000) 2019-10-13T07:15:57.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:15:57.379+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.382269+0000) 2019-10-13T07:15:57.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:15:57.380+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.383624+0000) 2019-10-13T07:15:57.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:15:57.380+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.384142+0000) 2019-10-13T07:15:57.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:15:57.387+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.388936+0000) 2019-10-13T07:15:57.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:15:57.480+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.484217+0000) 2019-10-13T07:15:57.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:15:57.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:15:57.486712+0000) 2019-10-13T07:16:00.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:00.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.614648+0000) 2019-10-13T07:16:00.715 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:00.712+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:00.713472+0000) 2019-10-13T07:16:01.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:01.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:01.510694+0000) 2019-10-13T07:16:01.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:01.506+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:01.510597+0000) 2019-10-13T07:16:02.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:02.034+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.036808+0000) 2019-10-13T07:16:02.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:02.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.384929+0000) 2019-10-13T07:16:02.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:02.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:02.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:02.380+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.385357+0000) 2019-10-13T07:16:02.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:02.384+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.385953+0000) 2019-10-13T07:16:02.388 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:02.382+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.385930+0000) 2019-10-13T07:16:02.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:02.384+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.386744+0000) 2019-10-13T07:16:02.389 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:02.384+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.387157+0000) 2019-10-13T07:16:02.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:02.384+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.389022+0000) 2019-10-13T07:16:02.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:02.390+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:02.395089+0000) 2019-10-13T07:16:07.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:07.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.071996+0000) 2019-10-13T07:16:07.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:07.384+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.388966+0000) 2019-10-13T07:16:07.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:07.384+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.389468+0000) 2019-10-13T07:16:07.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:07.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:07.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:07.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.390327+0000) 2019-10-13T07:16:07.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:07.386+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.390325+0000) 2019-10-13T07:16:07.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:07.388+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.391094+0000) 2019-10-13T07:16:07.394 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:07.388+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.392072+0000) 2019-10-13T07:16:07.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:07.388+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.392329+0000) 2019-10-13T07:16:07.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:07.394+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.396511+0000) 2019-10-13T07:16:07.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:07.484+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.484404+0000) 2019-10-13T07:16:07.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:07.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:07.486925+0000) 2019-10-13T07:16:10.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:10.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.614847+0000) 2019-10-13T07:16:10.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:10.708+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:10.713664+0000) 2019-10-13T07:16:11.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:11.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:11.510882+0000) 2019-10-13T07:16:11.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:11.506+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:11.510785+0000) 2019-10-13T07:16:12.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:12.038+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.043269+0000) 2019-10-13T07:16:12.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:12.388+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.393415+0000) 2019-10-13T07:16:12.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:12.388+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.393554+0000) 2019-10-13T07:16:12.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:12.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:12.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:12.392+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.395219+0000) 2019-10-13T07:16:12.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:12.392+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.395659+0000) 2019-10-13T07:16:12.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:12.392+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.396166+0000) 2019-10-13T07:16:12.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:12.390+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.396005+0000) 2019-10-13T07:16:12.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:12.392+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.396905+0000) 2019-10-13T07:16:12.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:12.398+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:12.400188+0000) 2019-10-13T07:16:17.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:17.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.072229+0000) 2019-10-13T07:16:17.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:17.392+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.397496+0000) 2019-10-13T07:16:17.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:17.396+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.398098+0000) 2019-10-13T07:16:17.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:17.396+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.398468+0000) 2019-10-13T07:16:17.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:17.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:17.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:17.396+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.398763+0000) 2019-10-13T07:16:17.401 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:17.396+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.399863+0000) 2019-10-13T07:16:17.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:17.396+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.401011+0000) 2019-10-13T07:16:17.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:17.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.401268+0000) 2019-10-13T07:16:17.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:17.402+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.406970+0000) 2019-10-13T07:16:17.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:17.484+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.484609+0000) 2019-10-13T07:16:17.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:17.484+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:17.487127+0000) 2019-10-13T07:16:18.864 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:16:20.530 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:20.534 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:16:20.534 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:20.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:20.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.615049+0000) 2019-10-13T07:16:20.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:20.712+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:20.713861+0000) 2019-10-13T07:16:21.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:21.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:21.511033+0000) 2019-10-13T07:16:21.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:21.510+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:21.510980+0000) 2019-10-13T07:16:22.052 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:22.046+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.049773+0000) 2019-10-13T07:16:22.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:22.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.401590+0000) 2019-10-13T07:16:22.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:22.399+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.402798+0000) 2019-10-13T07:16:22.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:22.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46958 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:22.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:22.399+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.402994+0000) 2019-10-13T07:16:22.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:22.399+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.403447+0000) 2019-10-13T07:16:22.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:22.399+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.403829+0000) 2019-10-13T07:16:22.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:22.399+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.404387+0000) 2019-10-13T07:16:22.407 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:22.399+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.404662+0000) 2019-10-13T07:16:22.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:22.406+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:22.411175+0000) 2019-10-13T07:16:27.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:27.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.072436+0000) 2019-10-13T07:16:27.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:27.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:27.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:27.403+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.407497+0000) 2019-10-13T07:16:27.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:27.406+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.408598+0000) 2019-10-13T07:16:27.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:27.410+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.413841+0000) 2019-10-13T07:16:27.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:27.403+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.407969+0000) 2019-10-13T07:16:27.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:27.403+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.409354+0000) 2019-10-13T07:16:27.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:27.407+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.409954+0000) 2019-10-13T07:16:27.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:27.407+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.410209+0000) 2019-10-13T07:16:27.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:27.407+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.410928+0000) 2019-10-13T07:16:27.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:27.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.484824+0000) 2019-10-13T07:16:27.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:27.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:27.487374+0000) 2019-10-13T07:16:30.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:30.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.615267+0000) 2019-10-13T07:16:30.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:30.711+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:30.714062+0000) 2019-10-13T07:16:31.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:31.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:31.511235+0000) 2019-10-13T07:16:31.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:31.510+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:31.511187+0000) 2019-10-13T07:16:32.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:32.054+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.056185+0000) 2019-10-13T07:16:32.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:32.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.411341+0000) 2019-10-13T07:16:32.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:32.407+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.410453+0000) 2019-10-13T07:16:32.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:32.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:32.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:32.407+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.411545+0000) 2019-10-13T07:16:32.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:32.407+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.412379+0000) 2019-10-13T07:16:32.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:32.407+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.412782+0000) 2019-10-13T07:16:32.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:32.411+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.413995+0000) 2019-10-13T07:16:32.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:32.411+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.414504+0000) 2019-10-13T07:16:32.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:32.414+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:32.419337+0000) 2019-10-13T07:16:37.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:37.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.072654+0000) 2019-10-13T07:16:37.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:37.411+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.415000+0000) 2019-10-13T07:16:37.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:37.411+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.415143+0000) 2019-10-13T07:16:37.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:37.410+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.415378+0000) 2019-10-13T07:16:37.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:37.411+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.415648+0000) 2019-10-13T07:16:37.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:37.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:37.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:37.411+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.416283+0000) 2019-10-13T07:16:37.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:37.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.417534+0000) 2019-10-13T07:16:37.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:37.415+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.418271+0000) 2019-10-13T07:16:37.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:37.418+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.421860+0000) 2019-10-13T07:16:37.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:37.483+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.484998+0000) 2019-10-13T07:16:37.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:37.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:37.487549+0000) 2019-10-13T07:16:40.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:40.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.615477+0000) 2019-10-13T07:16:40.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:40.711+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:40.714278+0000) 2019-10-13T07:16:41.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:41.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:41.511431+0000) 2019-10-13T07:16:41.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:41.506+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:41.511406+0000) 2019-10-13T07:16:42.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:42.062+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.062658+0000) 2019-10-13T07:16:42.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:42.415+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.419413+0000) 2019-10-13T07:16:42.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:42.414+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.419798+0000) 2019-10-13T07:16:42.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:42.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:42.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:42.415+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.420144+0000) 2019-10-13T07:16:42.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:42.415+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.420598+0000) 2019-10-13T07:16:42.423 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:42.415+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.420957+0000) 2019-10-13T07:16:42.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:42.419+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.422857+0000) 2019-10-13T07:16:42.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:42.419+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.423698+0000) 2019-10-13T07:16:42.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:42.422+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:42.425424+0000) 2019-10-13T07:16:47.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:47.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.072860+0000) 2019-10-13T07:16:47.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:47.419+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.423510+0000) 2019-10-13T07:16:47.426 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:47.419+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.423854+0000) 2019-10-13T07:16:47.426 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:47.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:47.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:47.419+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.424149+0000) 2019-10-13T07:16:47.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:47.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.425231+0000) 2019-10-13T07:16:47.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:47.422+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.425065+0000) 2019-10-13T07:16:47.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:47.423+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.426265+0000) 2019-10-13T07:16:47.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:47.423+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.426696+0000) 2019-10-13T07:16:47.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:47.426+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.430491+0000) 2019-10-13T07:16:47.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:47.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.485192+0000) 2019-10-13T07:16:47.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:47.487+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:47.487745+0000) 2019-10-13T07:16:50.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:50.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.615721+0000) 2019-10-13T07:16:51.639 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:16:51.639 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:51.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:51.506+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:51.511606+0000) 2019-10-13T07:16:51.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:50.711+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:50.714493+0000) 2019-10-13T07:16:51.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:51.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:51.511668+0000) 2019-10-13T07:16:51.643 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:16:51.644 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:16:52.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:52.070+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.073923+0000) 2019-10-13T07:16:52.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:52.423+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.427594+0000) 2019-10-13T07:16:52.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:52.423+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.427956+0000) 2019-10-13T07:16:52.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:52.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:52.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:52.423+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.428823+0000) 2019-10-13T07:16:52.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:52.423+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.429325+0000) 2019-10-13T07:16:52.431 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:52.423+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.429192+0000) 2019-10-13T07:16:52.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:52.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.429174+0000) 2019-10-13T07:16:52.432 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:52.427+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.430633+0000) 2019-10-13T07:16:52.438 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:52.434+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:52.436216+0000) 2019-10-13T07:16:57.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:57.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.073067+0000) 2019-10-13T07:16:57.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:16:57.427+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.431923+0000) 2019-10-13T07:16:57.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:16:57.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:16:57.434 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:16:57.427+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.432380+0000) 2019-10-13T07:16:57.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:16:57.427+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.432906+0000) 2019-10-13T07:16:57.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:57.427+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.433518+0000) 2019-10-13T07:16:57.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:57.427+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.433201+0000) 2019-10-13T07:16:57.436 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:16:57.430+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.433681+0000) 2019-10-13T07:16:57.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:16:57.431+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.434986+0000) 2019-10-13T07:16:57.442 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:16:57.438+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.440380+0000) 2019-10-13T07:16:57.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:16:57.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.485389+0000) 2019-10-13T07:16:57.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:16:57.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:16:57.487915+0000) 2019-10-13T07:17:00.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:00.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.615923+0000) 2019-10-13T07:17:00.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:00.711+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:00.714692+0000) 2019-10-13T07:17:01.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:01.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:01.511853+0000) 2019-10-13T07:17:01.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:01.510+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:01.511804+0000) 2019-10-13T07:17:02.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:02.078+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.080422+0000) 2019-10-13T07:17:02.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:02.431+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.435800+0000) 2019-10-13T07:17:02.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:02.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.436490+0000) 2019-10-13T07:17:02.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:02.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:02.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:02.431+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.437385+0000) 2019-10-13T07:17:02.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:02.431+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.437588+0000) 2019-10-13T07:17:02.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:02.435+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.437924+0000) 2019-10-13T07:17:02.440 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:02.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.438369+0000) 2019-10-13T07:17:02.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:02.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.439431+0000) 2019-10-13T07:17:02.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:02.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:02.442199+0000) 2019-10-13T07:17:07.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:07.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.073292+0000) 2019-10-13T07:17:07.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:07.435+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.439841+0000) 2019-10-13T07:17:07.442 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:07.433+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.439964+0000) 2019-10-13T07:17:07.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:07.435+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.440858+0000) 2019-10-13T07:17:07.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:07.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:07.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:07.439+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.441949+0000) 2019-10-13T07:17:07.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:07.439+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.442189+0000) 2019-10-13T07:17:07.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:07.439+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.443780+0000) 2019-10-13T07:17:07.446 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:07.439+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.444330+0000) 2019-10-13T07:17:07.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:07.441+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.446578+0000) 2019-10-13T07:17:07.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:07.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.485554+0000) 2019-10-13T07:17:07.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:07.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:07.488119+0000) 2019-10-13T07:17:10.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:10.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.616124+0000) 2019-10-13T07:17:10.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:10.711+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:10.714849+0000) 2019-10-13T07:17:11.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:11.505+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:11.512030+0000) 2019-10-13T07:17:11.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:11.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:11.512063+0000) 2019-10-13T07:17:12.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:12.089+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.092034+0000) 2019-10-13T07:17:12.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:12.439+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.444351+0000) 2019-10-13T07:17:12.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:12.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:12.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:12.439+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.445375+0000) 2019-10-13T07:17:12.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:12.441+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.445142+0000) 2019-10-13T07:17:12.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:12.443+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.446018+0000) 2019-10-13T07:17:12.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:12.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.446392+0000) 2019-10-13T07:17:12.450 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:12.443+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.447221+0000) 2019-10-13T07:17:12.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:12.443+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.448157+0000) 2019-10-13T07:17:12.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:12.449+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:12.455375+0000) 2019-10-13T07:17:17.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:17.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.073491+0000) 2019-10-13T07:17:17.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:17.443+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.448053+0000) 2019-10-13T07:17:17.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:17.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.448609+0000) 2019-10-13T07:17:17.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:17.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:17.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:17.443+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.449265+0000) 2019-10-13T07:17:17.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:17.447+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.449858+0000) 2019-10-13T07:17:17.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:17.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.450703+0000) 2019-10-13T07:17:17.454 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:17.447+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.451520+0000) 2019-10-13T07:17:17.454 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:17.447+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.450929+0000) 2019-10-13T07:17:17.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:17.453+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.458212+0000) 2019-10-13T07:17:17.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:17.483+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.485784+0000) 2019-10-13T07:17:17.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:17.483+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:17.488301+0000) 2019-10-13T07:17:20.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:20.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.616344+0000) 2019-10-13T07:17:20.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:20.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:20.715053+0000) 2019-10-13T07:17:21.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:21.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:21.512279+0000) 2019-10-13T07:17:21.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:21.509+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:21.512224+0000) 2019-10-13T07:17:21.723 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:17:21.723 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:21.729 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:17:21.729 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:22.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:22.097+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.098602+0000) 2019-10-13T07:17:22.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:22.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:22.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:22.450+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.453778+0000) 2019-10-13T07:17:22.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:22.450+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.454419+0000) 2019-10-13T07:17:22.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:22.449+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.455243+0000) 2019-10-13T07:17:22.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:22.450+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.456000+0000) 2019-10-13T07:17:22.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:22.450+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.455002+0000) 2019-10-13T07:17:22.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:22.450+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.456172+0000) 2019-10-13T07:17:22.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:22.450+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.457303+0000) 2019-10-13T07:17:22.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:22.453+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:22.459019+0000) 2019-10-13T07:17:27.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:27.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.073727+0000) 2019-10-13T07:17:27.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:27.453+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.456307+0000) 2019-10-13T07:17:27.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:27.450+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.457295+0000) 2019-10-13T07:17:27.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:27.454+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.457906+0000) 2019-10-13T07:17:27.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:27.454+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.458274+0000) 2019-10-13T07:17:27.460 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:27.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:27.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:27.454+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.459001+0000) 2019-10-13T07:17:27.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:27.454+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.459755+0000) 2019-10-13T07:17:27.463 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:27.454+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.461533+0000) 2019-10-13T07:17:27.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:27.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.462633+0000) 2019-10-13T07:17:27.497 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:27.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.486008+0000) 2019-10-13T07:17:27.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:27.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:27.488464+0000) 2019-10-13T07:17:30.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:30.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.616577+0000) 2019-10-13T07:17:30.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:30.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:30.715269+0000) 2019-10-13T07:17:31.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:31.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:31.512500+0000) 2019-10-13T07:17:31.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:31.509+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:31.512451+0000) 2019-10-13T07:17:32.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:32.101+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.105028+0000) 2019-10-13T07:17:32.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:32.458+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.462113+0000) 2019-10-13T07:17:32.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:32.458+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.462519+0000) 2019-10-13T07:17:32.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:32.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:32.465 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:32.458+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.463597+0000) 2019-10-13T07:17:32.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:32.457+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.463896+0000) 2019-10-13T07:17:32.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:32.458+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.464092+0000) 2019-10-13T07:17:32.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:32.458+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.464808+0000) 2019-10-13T07:17:32.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:32.458+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.465707+0000) 2019-10-13T07:17:32.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:32.465+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:32.469063+0000) 2019-10-13T07:17:37.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:37.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.073956+0000) 2019-10-13T07:17:37.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:37.462+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.468026+0000) 2019-10-13T07:17:37.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:37.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:37.471 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:37.462+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.468439+0000) 2019-10-13T07:17:37.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:37.462+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.468730+0000) 2019-10-13T07:17:37.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:37.462+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.469038+0000) 2019-10-13T07:17:37.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:37.462+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.469509+0000) 2019-10-13T07:17:37.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:37.462+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.469681+0000) 2019-10-13T07:17:37.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:37.465+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.469852+0000) 2019-10-13T07:17:37.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:37.473+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.479025+0000) 2019-10-13T07:17:37.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:37.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.486133+0000) 2019-10-13T07:17:37.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:37.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:37.488594+0000) 2019-10-13T07:17:40.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:40.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:40.616824+0000) 2019-10-13T07:17:40.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:40.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:40.715480+0000) 2019-10-13T07:17:41.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:41.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.512670+0000) 2019-10-13T07:17:41.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:41.509+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:41.512649+0000) 2019-10-13T07:17:42.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:42.105+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.111562+0000) 2019-10-13T07:17:42.473 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:42.466+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.471524+0000) 2019-10-13T07:17:42.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:42.466+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.472077+0000) 2019-10-13T07:17:42.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:42.466+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.472669+0000) 2019-10-13T07:17:42.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:42.466+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.472983+0000) 2019-10-13T07:17:42.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:42.466+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:42.476 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:42.470+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.474243+0000) 2019-10-13T07:17:42.478 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:42.469+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.475551+0000) 2019-10-13T07:17:42.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:42.470+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.475551+0000) 2019-10-13T07:17:42.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:42.477+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:42.480242+0000) 2019-10-13T07:17:47.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:47.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.074168+0000) 2019-10-13T07:17:47.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:47.470+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.476003+0000) 2019-10-13T07:17:47.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:47.474+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.477900+0000) 2019-10-13T07:17:47.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:47.474+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:47.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:47.474+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.479180+0000) 2019-10-13T07:17:47.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:47.474+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.479846+0000) 2019-10-13T07:17:47.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:47.474+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.480078+0000) 2019-10-13T07:17:47.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:47.474+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.480799+0000) 2019-10-13T07:17:47.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:47.477+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.480884+0000) 2019-10-13T07:17:47.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:47.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.486254+0000) 2019-10-13T07:17:47.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:47.481+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.486934+0000) 2019-10-13T07:17:47.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:47.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:47.488748+0000) 2019-10-13T07:17:50.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:50.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:50.617019+0000) 2019-10-13T07:17:50.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:50.710+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:50.715682+0000) 2019-10-13T07:17:51.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:51.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.512872+0000) 2019-10-13T07:17:51.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:51.509+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:51.512848+0000) 2019-10-13T07:17:51.886 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:17:51.887 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:51.890 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:17:51.890 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:17:52.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:52.117+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.118097+0000) 2019-10-13T07:17:52.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:52.474+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.480655+0000) 2019-10-13T07:17:52.483 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:52.477+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.481319+0000) 2019-10-13T07:17:52.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:52.478+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.482417+0000) 2019-10-13T07:17:52.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:52.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:52.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:52.478+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.483558+0000) 2019-10-13T07:17:52.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:52.478+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.483899+0000) 2019-10-13T07:17:52.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:52.478+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.484516+0000) 2019-10-13T07:17:52.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:52.478+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.484952+0000) 2019-10-13T07:17:52.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:52.485+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:52.488575+0000) 2019-10-13T07:17:57.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:57.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.074457+0000) 2019-10-13T07:17:57.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:17:57.482+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.486629+0000) 2019-10-13T07:17:57.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:17:57.482+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.487319+0000) 2019-10-13T07:17:57.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:17:57.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:17:57.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:57.482+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.487766+0000) 2019-10-13T07:17:57.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:17:57.482+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.488138+0000) 2019-10-13T07:17:57.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:17:57.485+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.488349+0000) 2019-10-13T07:17:57.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:17:57.482+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.488912+0000) 2019-10-13T07:17:57.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:17:57.486+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.490018+0000) 2019-10-13T07:17:57.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:17:57.486+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.492081+0000) 2019-10-13T07:17:57.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:17:57.493+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:17:57.496937+0000) 2019-10-13T07:18:00.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:00.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:00.617180+0000) 2019-10-13T07:18:00.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:00.714+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:00.715884+0000) 2019-10-13T07:18:04.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:01.509+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.513048+0000) 2019-10-13T07:18:04.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:02.485+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.490015+0000) 2019-10-13T07:18:04.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:02.121+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.124615+0000) 2019-10-13T07:18:04.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:02.493+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.500070+0000) 2019-10-13T07:18:04.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:01.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:01.513065+0000) 2019-10-13T07:18:04.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:02.486+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.493313+0000) 2019-10-13T07:18:04.715 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:02.486+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.491816+0000) 2019-10-13T07:18:04.715 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:04.234+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:04.236856+0000) 2019-10-13T07:18:04.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:02.486+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:04.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:02.486+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.492558+0000) 2019-10-13T07:18:04.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:02.486+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.492927+0000) 2019-10-13T07:18:04.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:02.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.493590+0000) 2019-10-13T07:18:04.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:02.490+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:02.494695+0000) 2019-10-13T07:18:07.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:07.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.074658+0000) 2019-10-13T07:18:07.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:07.486+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.489104+0000) 2019-10-13T07:18:07.497 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:07.490+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.495824+0000) 2019-10-13T07:18:07.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:07.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:07.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:07.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.497552+0000) 2019-10-13T07:18:07.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:07.494+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.497785+0000) 2019-10-13T07:18:07.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:07.493+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.497901+0000) 2019-10-13T07:18:07.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:07.494+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.498257+0000) 2019-10-13T07:18:07.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:07.494+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.498979+0000) 2019-10-13T07:18:07.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:07.494+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.499867+0000) 2019-10-13T07:18:07.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:07.501+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:07.505376+0000) 2019-10-13T07:18:10.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:10.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:10.617378+0000) 2019-10-13T07:18:10.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:10.714+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:10.716084+0000) 2019-10-13T07:18:11.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:11.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.513279+0000) 2019-10-13T07:18:11.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:11.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:11.513260+0000) 2019-10-13T07:18:12.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:12.124+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.131061+0000) 2019-10-13T07:18:12.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:12.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.499427+0000) 2019-10-13T07:18:12.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:12.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:12.503 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:12.494+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.501002+0000) 2019-10-13T07:18:12.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:12.494+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.501352+0000) 2019-10-13T07:18:12.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:12.498+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.501733+0000) 2019-10-13T07:18:12.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:12.498+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.502596+0000) 2019-10-13T07:18:12.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:12.496+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.502894+0000) 2019-10-13T07:18:12.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:12.498+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.503938+0000) 2019-10-13T07:18:12.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:12.500+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:12.506657+0000) 2019-10-13T07:18:14.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:14.230+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:14.237024+0000) 2019-10-13T07:18:17.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:17.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.074874+0000) 2019-10-13T07:18:17.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:17.485+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.489335+0000) 2019-10-13T07:18:17.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:17.500+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.504942+0000) 2019-10-13T07:18:17.508 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:17.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:17.508 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:17.501+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.506081+0000) 2019-10-13T07:18:17.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:17.501+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.506969+0000) 2019-10-13T07:18:17.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:17.501+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.507639+0000) 2019-10-13T07:18:17.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:17.501+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.508647+0000) 2019-10-13T07:18:17.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:17.501+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.508972+0000) 2019-10-13T07:18:17.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:17.501+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.509627+0000) 2019-10-13T07:18:17.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:17.508+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:17.512757+0000) 2019-10-13T07:18:20.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:20.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:20.617582+0000) 2019-10-13T07:18:20.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:20.713+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:20.716287+0000) 2019-10-13T07:18:21.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:21.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.513473+0000) 2019-10-13T07:18:21.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:21.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:21.513458+0000) 2019-10-13T07:18:22.018 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:18:22.018 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:22.021 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:18:22.021 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:22.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:22.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.140353+0000) 2019-10-13T07:18:22.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:22.504+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.509156+0000) 2019-10-13T07:18:22.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:22.505+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.510167+0000) 2019-10-13T07:18:22.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:22.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:22.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:22.505+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.510882+0000) 2019-10-13T07:18:22.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:22.505+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.511135+0000) 2019-10-13T07:18:22.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:22.505+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.511760+0000) 2019-10-13T07:18:22.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:22.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.511464+0000) 2019-10-13T07:18:22.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:22.505+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.512270+0000) 2019-10-13T07:18:22.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:22.512+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:22.519066+0000) 2019-10-13T07:18:24.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:24.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:24.237231+0000) 2019-10-13T07:18:27.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:27.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.075071+0000) 2019-10-13T07:18:27.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:27.485+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.489569+0000) 2019-10-13T07:18:27.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:27.509+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.514328+0000) 2019-10-13T07:18:27.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:27.509+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.514648+0000) 2019-10-13T07:18:27.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:27.509+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.514978+0000) 2019-10-13T07:18:27.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:27.509+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:27.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:27.509+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.515969+0000) 2019-10-13T07:18:27.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:27.508+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.515859+0000) 2019-10-13T07:18:27.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:27.509+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.516694+0000) 2019-10-13T07:18:27.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:27.509+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.517198+0000) 2019-10-13T07:18:27.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:27.516+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:27.521987+0000) 2019-10-13T07:18:30.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:30.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:30.617806+0000) 2019-10-13T07:18:30.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:30.713+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:30.716510+0000) 2019-10-13T07:18:31.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:31.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.513702+0000) 2019-10-13T07:18:31.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:31.508+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:31.513660+0000) 2019-10-13T07:18:32.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:32.144+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.146822+0000) 2019-10-13T07:18:32.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:32.517+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.518383+0000) 2019-10-13T07:18:32.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:32.517+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.518965+0000) 2019-10-13T07:18:32.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:32.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:32.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:32.517+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.519438+0000) 2019-10-13T07:18:32.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:32.512+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.519838+0000) 2019-10-13T07:18:32.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:32.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.519883+0000) 2019-10-13T07:18:32.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:32.517+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.520092+0000) 2019-10-13T07:18:32.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:32.521+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.522505+0000) 2019-10-13T07:18:32.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:32.524+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:32.528378+0000) 2019-10-13T07:18:34.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:34.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:34.237433+0000) 2019-10-13T07:18:37.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:37.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.075287+0000) 2019-10-13T07:18:37.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:37.489+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.489769+0000) 2019-10-13T07:18:37.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:37.521+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.523095+0000) 2019-10-13T07:18:37.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:37.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:37.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:37.521+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.523594+0000) 2019-10-13T07:18:37.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:37.521+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.524355+0000) 2019-10-13T07:18:37.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:37.521+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.524889+0000) 2019-10-13T07:18:37.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:37.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.525348+0000) 2019-10-13T07:18:37.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:37.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.526273+0000) 2019-10-13T07:18:37.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:37.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.526176+0000) 2019-10-13T07:18:37.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:37.524+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:37.528593+0000) 2019-10-13T07:18:40.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:40.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:40.618012+0000) 2019-10-13T07:18:40.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:40.713+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:40.716671+0000) 2019-10-13T07:18:41.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:41.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.513927+0000) 2019-10-13T07:18:41.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:41.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:41.513890+0000) 2019-10-13T07:18:42.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:42.152+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.153288+0000) 2019-10-13T07:18:42.528 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:42.525+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.526722+0000) 2019-10-13T07:18:42.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:42.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.527731+0000) 2019-10-13T07:18:42.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:42.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47811 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:42.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:42.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.529090+0000) 2019-10-13T07:18:42.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:42.525+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.529282+0000) 2019-10-13T07:18:42.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:42.529+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.530070+0000) 2019-10-13T07:18:42.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:42.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.530205+0000) 2019-10-13T07:18:42.533 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:42.529+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.531098+0000) 2019-10-13T07:18:42.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:42.532+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:42.537062+0000) 2019-10-13T07:18:44.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:44.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:44.237636+0000) 2019-10-13T07:18:47.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:47.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.075464+0000) 2019-10-13T07:18:47.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:47.489+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.489970+0000) 2019-10-13T07:18:47.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:47.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.532476+0000) 2019-10-13T07:18:47.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:47.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:47.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:47.529+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.533014+0000) 2019-10-13T07:18:47.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:47.529+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.533401+0000) 2019-10-13T07:18:47.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:47.528+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.533301+0000) 2019-10-13T07:18:47.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:47.533+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.534188+0000) 2019-10-13T07:18:47.536 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:47.533+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.535055+0000) 2019-10-13T07:18:47.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:47.533+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.535538+0000) 2019-10-13T07:18:47.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:47.532+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:47.538814+0000) 2019-10-13T07:18:50.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:50.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:50.618227+0000) 2019-10-13T07:18:50.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:50.713+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:50.716875+0000) 2019-10-13T07:18:51.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:51.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.514128+0000) 2019-10-13T07:18:51.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:51.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:51.514131+0000) 2019-10-13T07:18:52.133 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:18:52.133 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:52.137 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:18:52.137 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:18:52.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:52.152+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.159687+0000) 2019-10-13T07:18:52.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:52.533+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.536090+0000) 2019-10-13T07:18:52.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:52.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:52.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:52.533+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.537350+0000) 2019-10-13T07:18:52.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:52.537+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.538041+0000) 2019-10-13T07:18:52.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:52.532+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.537904+0000) 2019-10-13T07:18:52.540 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:52.537+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.539033+0000) 2019-10-13T07:18:52.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:52.537+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.539827+0000) 2019-10-13T07:18:52.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:52.541+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.542887+0000) 2019-10-13T07:18:52.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:52.544+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:52.548924+0000) 2019-10-13T07:18:54.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:54.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:54.237869+0000) 2019-10-13T07:18:57.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:57.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.075620+0000) 2019-10-13T07:18:57.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:57.489+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.490187+0000) 2019-10-13T07:18:57.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:18:57.537+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.540419+0000) 2019-10-13T07:18:57.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:18:57.537+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.540800+0000) 2019-10-13T07:18:57.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:18:57.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:18:57.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:18:57.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.540701+0000) 2019-10-13T07:18:57.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:18:57.541+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.541795+0000) 2019-10-13T07:18:57.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:18:57.541+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.541829+0000) 2019-10-13T07:18:57.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:18:57.541+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.543150+0000) 2019-10-13T07:18:57.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:18:57.541+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.542725+0000) 2019-10-13T07:18:57.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:18:57.544+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:18:57.549460+0000) 2019-10-13T07:19:00.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:00.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:00.618456+0000) 2019-10-13T07:19:00.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:00.713+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:00.717076+0000) 2019-10-13T07:19:01.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:01.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.514385+0000) 2019-10-13T07:19:01.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:01.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:01.514372+0000) 2019-10-13T07:19:02.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:02.164+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.165484+0000) 2019-10-13T07:19:08.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:02.541+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.545119+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:04.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:04.238071+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:07.545+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.548739+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:02.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:07.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:02.545+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.545772+0000) 2019-10-13T07:19:08.810 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:07.549+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.550199+0000) 2019-10-13T07:19:08.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:02.545+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.546144+0000) 2019-10-13T07:19:08.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:07.549+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.549922+0000) 2019-10-13T07:19:08.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:02.545+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.546764+0000) 2019-10-13T07:19:08.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:07.489+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.490417+0000) 2019-10-13T07:19:08.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:07.549+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.550523+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:02.545+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.548120+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:07.549+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.550944+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:02.545+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.548958+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:07.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.075821+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:07.549+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.552492+0000) 2019-10-13T07:19:08.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:02.540+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.546738+0000) 2019-10-13T07:19:08.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:07.544+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.550691+0000) 2019-10-13T07:19:08.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:02.544+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:02.549349+0000) 2019-10-13T07:19:08.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:07.548+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:07.555311+0000) 2019-10-13T07:19:10.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:10.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:10.618655+0000) 2019-10-13T07:19:10.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:10.717+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:10.717290+0000) 2019-10-13T07:19:11.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:11.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.514575+0000) 2019-10-13T07:19:11.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:11.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:11.514569+0000) 2019-10-13T07:19:12.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:12.168+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.172068+0000) 2019-10-13T07:19:12.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:12.549+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.552900+0000) 2019-10-13T07:19:12.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:12.549+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.553326+0000) 2019-10-13T07:19:12.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:12.553+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.553790+0000) 2019-10-13T07:19:12.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:12.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:12.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:12.553+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.554058+0000) 2019-10-13T07:19:12.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:12.552+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.553943+0000) 2019-10-13T07:19:12.557 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:12.553+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.555756+0000) 2019-10-13T07:19:12.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:12.557+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.558818+0000) 2019-10-13T07:19:12.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:12.560+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:12.561469+0000) 2019-10-13T07:19:14.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:14.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:14.238272+0000) 2019-10-13T07:19:17.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:17.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.076034+0000) 2019-10-13T07:19:17.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:17.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.490623+0000) 2019-10-13T07:19:17.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:17.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.557295+0000) 2019-10-13T07:19:17.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:17.556+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.557232+0000) 2019-10-13T07:19:17.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:17.556+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.558151+0000) 2019-10-13T07:19:17.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:17.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:17.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:17.556+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.558863+0000) 2019-10-13T07:19:17.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:17.556+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.558498+0000) 2019-10-13T07:19:17.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:17.556+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.560297+0000) 2019-10-13T07:19:17.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:17.556+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.559796+0000) 2019-10-13T07:19:17.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:17.564+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:17.564851+0000) 2019-10-13T07:19:20.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:20.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:20.618853+0000) 2019-10-13T07:19:20.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:20.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:20.717521+0000) 2019-10-13T07:19:21.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:21.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.514776+0000) 2019-10-13T07:19:21.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:21.511+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:21.514770+0000) 2019-10-13T07:19:22.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:22.175+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.178494+0000) 2019-10-13T07:19:22.270 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:19:22.270 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:22.274 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:19:22.274 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:22.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:22.560+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.562188+0000) 2019-10-13T07:19:22.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:22.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:22.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:22.560+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.563272+0000) 2019-10-13T07:19:22.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:22.560+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.563650+0000) 2019-10-13T07:19:22.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:22.560+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.563931+0000) 2019-10-13T07:19:22.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:22.559+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.563825+0000) 2019-10-13T07:19:22.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:22.560+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.564706+0000) 2019-10-13T07:19:22.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:22.564+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.565808+0000) 2019-10-13T07:19:22.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:22.567+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:22.570338+0000) 2019-10-13T07:19:24.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:24.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:24.238494+0000) 2019-10-13T07:19:27.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:27.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.076272+0000) 2019-10-13T07:19:27.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:27.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.490824+0000) 2019-10-13T07:19:27.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:27.564+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.566444+0000) 2019-10-13T07:19:27.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:27.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:27.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:27.564+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.568303+0000) 2019-10-13T07:19:27.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:27.567+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.568358+0000) 2019-10-13T07:19:27.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:27.564+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.569222+0000) 2019-10-13T07:19:27.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:27.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.569500+0000) 2019-10-13T07:19:27.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:27.568+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.569790+0000) 2019-10-13T07:19:27.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:27.568+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.570501+0000) 2019-10-13T07:19:27.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:27.571+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:27.574919+0000) 2019-10-13T07:19:30.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:30.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:30.619053+0000) 2019-10-13T07:19:30.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:30.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:30.717720+0000) 2019-10-13T07:19:31.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:31.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.514968+0000) 2019-10-13T07:19:31.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:31.511+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:31.514964+0000) 2019-10-13T07:19:32.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:32.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.185098+0000) 2019-10-13T07:19:32.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:32.568+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.571251+0000) 2019-10-13T07:19:32.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:32.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:32.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:32.568+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.572084+0000) 2019-10-13T07:19:32.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:32.567+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.571958+0000) 2019-10-13T07:19:32.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:32.568+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.573343+0000) 2019-10-13T07:19:32.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:32.572+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.573918+0000) 2019-10-13T07:19:32.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:32.572+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.574886+0000) 2019-10-13T07:19:32.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:32.572+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.574890+0000) 2019-10-13T07:19:32.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:32.575+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:32.577330+0000) 2019-10-13T07:19:34.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:34.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:34.238722+0000) 2019-10-13T07:19:37.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:37.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.076476+0000) 2019-10-13T07:19:37.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:37.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.491033+0000) 2019-10-13T07:19:37.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:37.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:37.579 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:37.572+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.576806+0000) 2019-10-13T07:19:37.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:37.572+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.577313+0000) 2019-10-13T07:19:37.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:37.576+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.577928+0000) 2019-10-13T07:19:37.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:37.576+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.578278+0000) 2019-10-13T07:19:37.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:37.575+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.578497+0000) 2019-10-13T07:19:37.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:37.576+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.579742+0000) 2019-10-13T07:19:37.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:37.576+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.580237+0000) 2019-10-13T07:19:37.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:37.583+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:37.586035+0000) 2019-10-13T07:19:40.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:40.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:40.619255+0000) 2019-10-13T07:19:40.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:40.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:40.717950+0000) 2019-10-13T07:19:41.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:41.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.515166+0000) 2019-10-13T07:19:41.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:41.511+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:41.515162+0000) 2019-10-13T07:19:42.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:42.187+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.191515+0000) 2019-10-13T07:19:42.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:42.576+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.580609+0000) 2019-10-13T07:19:42.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:42.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:42.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:42.576+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.581488+0000) 2019-10-13T07:19:42.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:42.580+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.581858+0000) 2019-10-13T07:19:42.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:42.580+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.582809+0000) 2019-10-13T07:19:42.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:42.579+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.582973+0000) 2019-10-13T07:19:42.586 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:42.580+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.583529+0000) 2019-10-13T07:19:42.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:42.580+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.584534+0000) 2019-10-13T07:19:42.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:42.587+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:42.589577+0000) 2019-10-13T07:19:44.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:44.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:44.238919+0000) 2019-10-13T07:19:47.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:47.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.076701+0000) 2019-10-13T07:19:47.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:47.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.491248+0000) 2019-10-13T07:19:47.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:47.579+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.583961+0000) 2019-10-13T07:19:47.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:47.580+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.584823+0000) 2019-10-13T07:19:47.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:47.584+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.585992+0000) 2019-10-13T07:19:47.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:47.584+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.586062+0000) 2019-10-13T07:19:47.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:47.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:47.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:47.584+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.586989+0000) 2019-10-13T07:19:47.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:47.584+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.586687+0000) 2019-10-13T07:19:47.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:47.584+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.588123+0000) 2019-10-13T07:19:47.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:47.591+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:47.592284+0000) 2019-10-13T07:19:50.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:50.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:50.619466+0000) 2019-10-13T07:19:50.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:50.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:50.718150+0000) 2019-10-13T07:19:51.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:51.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.515389+0000) 2019-10-13T07:19:51.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:51.511+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:51.515373+0000) 2019-10-13T07:19:52.200 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:52.195+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.198102+0000) 2019-10-13T07:19:52.409 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:19:52.410 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:52.413 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:19:52.413 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:19:52.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:52.588+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.590022+0000) 2019-10-13T07:19:52.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:52.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:52.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:52.588+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.590035+0000) 2019-10-13T07:19:52.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:52.588+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.590405+0000) 2019-10-13T07:19:52.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:52.588+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.590738+0000) 2019-10-13T07:19:52.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:52.587+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.591959+0000) 2019-10-13T07:19:52.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:52.588+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.592800+0000) 2019-10-13T07:19:52.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:52.588+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.593239+0000) 2019-10-13T07:19:52.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:52.599+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:52.600580+0000) 2019-10-13T07:19:54.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:54.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:54.239127+0000) 2019-10-13T07:19:57.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:57.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.076891+0000) 2019-10-13T07:19:57.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:57.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.491460+0000) 2019-10-13T07:19:57.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:19:57.588+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.593325+0000) 2019-10-13T07:19:57.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:19:57.591+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.593963+0000) 2019-10-13T07:19:57.597 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:19:57.592+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.594707+0000) 2019-10-13T07:19:57.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:19:57.592+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.594865+0000) 2019-10-13T07:19:57.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:19:57.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:19:57.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:19:57.592+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.595597+0000) 2019-10-13T07:19:57.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:19:57.592+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.595791+0000) 2019-10-13T07:19:57.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:19:57.592+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.597428+0000) 2019-10-13T07:19:57.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:19:57.603+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:19:57.605384+0000) 2019-10-13T07:20:00.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:00.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:00.619668+0000) 2019-10-13T07:20:00.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:00.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:00.718347+0000) 2019-10-13T07:20:01.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:01.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.515585+0000) 2019-10-13T07:20:01.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:01.511+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:01.515567+0000) 2019-10-13T07:20:02.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:02.203+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.204449+0000) 2019-10-13T07:20:02.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:02.596+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.598732+0000) 2019-10-13T07:20:02.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:02.596+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.599130+0000) 2019-10-13T07:20:02.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:02.596+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.599971+0000) 2019-10-13T07:20:02.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:02.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48298 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:02.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:02.595+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.599663+0000) 2019-10-13T07:20:02.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:02.596+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.600560+0000) 2019-10-13T07:20:02.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:02.596+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.600894+0000) 2019-10-13T07:20:02.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:02.600+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.602628+0000) 2019-10-13T07:20:02.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:02.607+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:02.610609+0000) 2019-10-13T07:20:04.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:04.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:04.239333+0000) 2019-10-13T07:20:07.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:07.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.077123+0000) 2019-10-13T07:20:07.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:07.488+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.491693+0000) 2019-10-13T07:20:07.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:07.600+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.603470+0000) 2019-10-13T07:20:07.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:07.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:07.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:07.600+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.605118+0000) 2019-10-13T07:20:07.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:07.604+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.605981+0000) 2019-10-13T07:20:07.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:07.604+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.606160+0000) 2019-10-13T07:20:07.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:07.604+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.606508+0000) 2019-10-13T07:20:07.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:07.603+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.606625+0000) 2019-10-13T07:20:07.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:07.604+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.607622+0000) 2019-10-13T07:20:07.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:07.607+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:07.611097+0000) 2019-10-13T07:20:10.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:10.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:10.619875+0000) 2019-10-13T07:20:10.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:10.716+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:10.718564+0000) 2019-10-13T07:20:11.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:11.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.515806+0000) 2019-10-13T07:20:11.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:11.515+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:11.515765+0000) 2019-10-13T07:20:12.213 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:12.207+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.210815+0000) 2019-10-13T07:20:12.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:12.603+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.608325+0000) 2019-10-13T07:20:12.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:12.603+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.608321+0000) 2019-10-13T07:20:12.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:12.603+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.609051+0000) 2019-10-13T07:20:12.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:12.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.609183+0000) 2019-10-13T07:20:12.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:12.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:12.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:12.603+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.609689+0000) 2019-10-13T07:20:12.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:12.607+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.610637+0000) 2019-10-13T07:20:12.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:12.607+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.611566+0000) 2019-10-13T07:20:12.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:12.611+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:12.614827+0000) 2019-10-13T07:20:14.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:14.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:14.239534+0000) 2019-10-13T07:20:17.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:17.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.077335+0000) 2019-10-13T07:20:17.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:17.487+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.491895+0000) 2019-10-13T07:20:17.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:17.607+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.613426+0000) 2019-10-13T07:20:17.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:17.611+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:17.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:17.611+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.614176+0000) 2019-10-13T07:20:17.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:17.611+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.614640+0000) 2019-10-13T07:20:17.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:17.611+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.614807+0000) 2019-10-13T07:20:17.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:17.611+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.614722+0000) 2019-10-13T07:20:17.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:17.611+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.615139+0000) 2019-10-13T07:20:17.618 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:17.611+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.616232+0000) 2019-10-13T07:20:17.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:17.619+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:17.622261+0000) 2019-10-13T07:20:20.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:20.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:20.620084+0000) 2019-10-13T07:20:20.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:20.715+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:20.718797+0000) 2019-10-13T07:20:21.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:21.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.516005+0000) 2019-10-13T07:20:21.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:21.515+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:21.515987+0000) 2019-10-13T07:20:22.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:22.215+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.217335+0000) 2019-10-13T07:20:22.478 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:20:22.478 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:22.482 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:20:22.482 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:22.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:22.611+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.617168+0000) 2019-10-13T07:20:22.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:22.611+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.617471+0000) 2019-10-13T07:20:22.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:22.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:22.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:22.615+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.618022+0000) 2019-10-13T07:20:22.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:22.615+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.619660+0000) 2019-10-13T07:20:22.622 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:22.615+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.620339+0000) 2019-10-13T07:20:22.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:22.615+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.620617+0000) 2019-10-13T07:20:22.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:22.615+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.621043+0000) 2019-10-13T07:20:22.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:22.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:22.625111+0000) 2019-10-13T07:20:24.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:24.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:24.239733+0000) 2019-10-13T07:20:27.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:27.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.077565+0000) 2019-10-13T07:20:27.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:27.491+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.492112+0000) 2019-10-13T07:20:27.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:27.615+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.621631+0000) 2019-10-13T07:20:27.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:27.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:27.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:27.618+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.622667+0000) 2019-10-13T07:20:27.625 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:27.619+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.623503+0000) 2019-10-13T07:20:27.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:27.619+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.624201+0000) 2019-10-13T07:20:27.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:27.619+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.624816+0000) 2019-10-13T07:20:27.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:27.623+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.626705+0000) 2019-10-13T07:20:27.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:27.619+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.625230+0000) 2019-10-13T07:20:27.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:27.626+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:27.630074+0000) 2019-10-13T07:20:30.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:30.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:30.620299+0000) 2019-10-13T07:20:30.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:30.715+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:30.719028+0000) 2019-10-13T07:20:31.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:31.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.516230+0000) 2019-10-13T07:20:31.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:31.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:31.516183+0000) 2019-10-13T07:20:32.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:32.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.228621+0000) 2019-10-13T07:20:32.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:32.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:32.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:32.623+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.628005+0000) 2019-10-13T07:20:32.630 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:32.623+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.628473+0000) 2019-10-13T07:20:32.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:32.623+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.628657+0000) 2019-10-13T07:20:32.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:32.626+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.628933+0000) 2019-10-13T07:20:32.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:32.627+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.630094+0000) 2019-10-13T07:20:32.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:32.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.629801+0000) 2019-10-13T07:20:32.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:32.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.630592+0000) 2019-10-13T07:20:32.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:32.630+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:32.634538+0000) 2019-10-13T07:20:34.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:34.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:34.239975+0000) 2019-10-13T07:20:37.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:37.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.077781+0000) 2019-10-13T07:20:37.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:37.487+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.492333+0000) 2019-10-13T07:20:37.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:37.627+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.630389+0000) 2019-10-13T07:20:37.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:37.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:37.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:37.627+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.631620+0000) 2019-10-13T07:20:37.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:37.627+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.631932+0000) 2019-10-13T07:20:37.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:37.630+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.632176+0000) 2019-10-13T07:20:37.635 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:37.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.633026+0000) 2019-10-13T07:20:37.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:37.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.633618+0000) 2019-10-13T07:20:37.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:37.631+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.634011+0000) 2019-10-13T07:20:37.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:37.634+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:37.636974+0000) 2019-10-13T07:20:40.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:40.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:40.620512+0000) 2019-10-13T07:20:40.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:40.715+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:40.719228+0000) 2019-10-13T07:20:41.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:41.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.516428+0000) 2019-10-13T07:20:41.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:41.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:41.516378+0000) 2019-10-13T07:20:42.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:42.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.235156+0000) 2019-10-13T07:20:42.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:42.631+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.634733+0000) 2019-10-13T07:20:42.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:42.631+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.635381+0000) 2019-10-13T07:20:42.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:42.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:42.638 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:42.631+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.636089+0000) 2019-10-13T07:20:42.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:42.630+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.635880+0000) 2019-10-13T07:20:42.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:42.631+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.636972+0000) 2019-10-13T07:20:42.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:42.631+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.637472+0000) 2019-10-13T07:20:42.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:42.631+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.637665+0000) 2019-10-13T07:20:42.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:42.642+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:42.644501+0000) 2019-10-13T07:20:44.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:44.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:44.240198+0000) 2019-10-13T07:20:47.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:47.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.077983+0000) 2019-10-13T07:20:47.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:47.491+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.492565+0000) 2019-10-13T07:20:47.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:47.634+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.638366+0000) 2019-10-13T07:20:47.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:47.635+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.639739+0000) 2019-10-13T07:20:47.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:47.635+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.640189+0000) 2019-10-13T07:20:47.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:47.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:47.643 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:47.635+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.641512+0000) 2019-10-13T07:20:47.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:47.639+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.641924+0000) 2019-10-13T07:20:47.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:47.639+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.642588+0000) 2019-10-13T07:20:47.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:47.639+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.643751+0000) 2019-10-13T07:20:47.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:47.642+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:47.646278+0000) 2019-10-13T07:20:50.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:50.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:50.620745+0000) 2019-10-13T07:20:50.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:50.715+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:50.719425+0000) 2019-10-13T07:20:51.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:51.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.516627+0000) 2019-10-13T07:20:51.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:51.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:51.516572+0000) 2019-10-13T07:20:52.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:52.242+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.246592+0000) 2019-10-13T07:20:52.602 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:20:52.602 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:52.606 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:20:52.606 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:20:52.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:52.638+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.642796+0000) 2019-10-13T07:20:52.646 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:52.639+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.644739+0000) 2019-10-13T07:20:52.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:52.639+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.644707+0000) 2019-10-13T07:20:52.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:52.639+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.645401+0000) 2019-10-13T07:20:52.647 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:52.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:52.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:52.643+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.646278+0000) 2019-10-13T07:20:52.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:52.643+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.646668+0000) 2019-10-13T07:20:52.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:52.643+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.647053+0000) 2019-10-13T07:20:52.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:52.646+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:52.651699+0000) 2019-10-13T07:20:54.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:54.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:54.240412+0000) 2019-10-13T07:20:57.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:57.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.078187+0000) 2019-10-13T07:20:57.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:57.487+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.492794+0000) 2019-10-13T07:20:57.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:20:57.643+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.648889+0000) 2019-10-13T07:20:57.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:20:57.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.649036+0000) 2019-10-13T07:20:57.652 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:20:57.647+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.650052+0000) 2019-10-13T07:20:57.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:20:57.647+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.650321+0000) 2019-10-13T07:20:57.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:20:57.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:20:57.653 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:20:57.647+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.650821+0000) 2019-10-13T07:20:57.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:20:57.647+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.651088+0000) 2019-10-13T07:20:57.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:20:57.647+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.652341+0000) 2019-10-13T07:20:57.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:20:57.654+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:20:57.656635+0000) 2019-10-13T07:21:00.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:00.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:00.620950+0000) 2019-10-13T07:21:00.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:00.719+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:00.719620+0000) 2019-10-13T07:21:01.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:01.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.516826+0000) 2019-10-13T07:21:01.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:01.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:01.516768+0000) 2019-10-13T07:21:02.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:02.250+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.253051+0000) 2019-10-13T07:21:02.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:02.650+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.652782+0000) 2019-10-13T07:21:02.656 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:02.647+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.653291+0000) 2019-10-13T07:21:02.656 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:02.647+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.653532+0000) 2019-10-13T07:21:02.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:02.647+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.653570+0000) 2019-10-13T07:21:02.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:02.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:02.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:02.651+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.655623+0000) 2019-10-13T07:21:02.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:02.651+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.656080+0000) 2019-10-13T07:21:02.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:02.651+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.655616+0000) 2019-10-13T07:21:02.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:02.658+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:02.660846+0000) 2019-10-13T07:21:04.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:04.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:04.240596+0000) 2019-10-13T07:21:07.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:07.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.078432+0000) 2019-10-13T07:21:07.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:07.491+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.492996+0000) 2019-10-13T07:21:07.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:07.655+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.658381+0000) 2019-10-13T07:21:07.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:07.655+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.658361+0000) 2019-10-13T07:21:07.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:07.655+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.658408+0000) 2019-10-13T07:21:07.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:07.654+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.658592+0000) 2019-10-13T07:21:07.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:07.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:07.662 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:07.655+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.659161+0000) 2019-10-13T07:21:07.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:07.655+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.659364+0000) 2019-10-13T07:21:07.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:07.655+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.659657+0000) 2019-10-13T07:21:07.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:07.662+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:07.665281+0000) 2019-10-13T07:21:10.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:10.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:10.621155+0000) 2019-10-13T07:21:10.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:10.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:10.719823+0000) 2019-10-13T07:21:11.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:11.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.517058+0000) 2019-10-13T07:21:11.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:11.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:11.517001+0000) 2019-10-13T07:21:12.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:12.258+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.259540+0000) 2019-10-13T07:21:12.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:12.658+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.662789+0000) 2019-10-13T07:21:12.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:12.658+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.663383+0000) 2019-10-13T07:21:12.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:12.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:12.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:12.658+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.664785+0000) 2019-10-13T07:21:12.667 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:12.658+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.665253+0000) 2019-10-13T07:21:12.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:12.662+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.665901+0000) 2019-10-13T07:21:12.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:12.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.666497+0000) 2019-10-13T07:21:12.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:12.662+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.666228+0000) 2019-10-13T07:21:12.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:12.666+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:12.668854+0000) 2019-10-13T07:21:14.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:14.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:14.240820+0000) 2019-10-13T07:21:17.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:17.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.078632+0000) 2019-10-13T07:21:17.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:17.486+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.493217+0000) 2019-10-13T07:21:17.669 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:17.662+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.667280+0000) 2019-10-13T07:21:17.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:17.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:17.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:17.662+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.668363+0000) 2019-10-13T07:21:17.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:17.662+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.668698+0000) 2019-10-13T07:21:17.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:17.666+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.668379+0000) 2019-10-13T07:21:17.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:17.662+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.669442+0000) 2019-10-13T07:21:17.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:17.662+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.669650+0000) 2019-10-13T07:21:17.672 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:17.666+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.670137+0000) 2019-10-13T07:21:17.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:17.670+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:17.674643+0000) 2019-10-13T07:21:20.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:20.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:20.621396+0000) 2019-10-13T07:21:21.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:21.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.517206+0000) 2019-10-13T07:21:21.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:20.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:20.720025+0000) 2019-10-13T07:21:21.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:21.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:21.517253+0000) 2019-10-13T07:21:22.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:22.262+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.265979+0000) 2019-10-13T07:21:22.673 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:22.666+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.671452+0000) 2019-10-13T07:21:22.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:22.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:22.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:22.666+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.672155+0000) 2019-10-13T07:21:22.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:22.666+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.672319+0000) 2019-10-13T07:21:22.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:22.666+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.672576+0000) 2019-10-13T07:21:22.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:22.666+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.672808+0000) 2019-10-13T07:21:22.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:22.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.673001+0000) 2019-10-13T07:21:22.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:22.670+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.674759+0000) 2019-10-13T07:21:22.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:22.678+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:22.681231+0000) 2019-10-13T07:21:22.750 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:21:22.750 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:22.754 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:21:22.754 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:24.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:24.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:24.241022+0000) 2019-10-13T07:21:27.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:27.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.078798+0000) 2019-10-13T07:21:27.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:27.490+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.493426+0000) 2019-10-13T07:21:27.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:27.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.675685+0000) 2019-10-13T07:21:27.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:27.670+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.676396+0000) 2019-10-13T07:21:27.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:27.670+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.676405+0000) 2019-10-13T07:21:27.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:27.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:27.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:27.670+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.676927+0000) 2019-10-13T07:21:27.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:27.674+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.677349+0000) 2019-10-13T07:21:27.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:27.674+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.678350+0000) 2019-10-13T07:21:27.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:27.674+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.679765+0000) 2019-10-13T07:21:27.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:27.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:27.683295+0000) 2019-10-13T07:21:30.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:30.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:30.621627+0000) 2019-10-13T07:21:30.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:30.714+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:30.720230+0000) 2019-10-13T07:21:31.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:31.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.517513+0000) 2019-10-13T07:21:31.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:31.514+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:31.517400+0000) 2019-10-13T07:21:32.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:32.269+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.272446+0000) 2019-10-13T07:21:32.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:32.674+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.679522+0000) 2019-10-13T07:21:32.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:32.674+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.680284+0000) 2019-10-13T07:21:32.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:32.674+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.680544+0000) 2019-10-13T07:21:32.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:32.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:32.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:32.678+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.681881+0000) 2019-10-13T07:21:32.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:32.677+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.681927+0000) 2019-10-13T07:21:32.684 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:32.678+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.682629+0000) 2019-10-13T07:21:32.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:32.678+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.683058+0000) 2019-10-13T07:21:32.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:32.681+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:32.687345+0000) 2019-10-13T07:21:34.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:34.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:34.241224+0000) 2019-10-13T07:21:37.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:37.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.079029+0000) 2019-10-13T07:21:37.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:37.486+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.493626+0000) 2019-10-13T07:21:37.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:37.678+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.684445+0000) 2019-10-13T07:21:37.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:37.681+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.684456+0000) 2019-10-13T07:21:37.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:37.678+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.684630+0000) 2019-10-13T07:21:37.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:37.678+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.685502+0000) 2019-10-13T07:21:37.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:37.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:37.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:37.682+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.685829+0000) 2019-10-13T07:21:37.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:37.682+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.686267+0000) 2019-10-13T07:21:37.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:37.686+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.690608+0000) 2019-10-13T07:21:37.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:37.693+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:37.697799+0000) 2019-10-13T07:21:40.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:40.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:40.621836+0000) 2019-10-13T07:21:40.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:40.714+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:40.720429+0000) 2019-10-13T07:21:41.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:41.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.517736+0000) 2019-10-13T07:21:41.520 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:41.513+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:41.517630+0000) 2019-10-13T07:21:42.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:42.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.279043+0000) 2019-10-13T07:21:42.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:42.686+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.690497+0000) 2019-10-13T07:21:42.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:42.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:42.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:42.685+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.690286+0000) 2019-10-13T07:21:42.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:42.686+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.691992+0000) 2019-10-13T07:21:42.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:42.686+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.692557+0000) 2019-10-13T07:21:42.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:42.686+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.692779+0000) 2019-10-13T07:21:42.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:42.686+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.693350+0000) 2019-10-13T07:21:42.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:42.690+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.693730+0000) 2019-10-13T07:21:42.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:42.697+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:42.700297+0000) 2019-10-13T07:21:44.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:44.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:44.241460+0000) 2019-10-13T07:21:47.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:47.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.079229+0000) 2019-10-13T07:21:47.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:47.490+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.493854+0000) 2019-10-13T07:21:47.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:47.690+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.694049+0000) 2019-10-13T07:21:47.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:47.690+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.695056+0000) 2019-10-13T07:21:47.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:47.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:47.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:47.690+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.695712+0000) 2019-10-13T07:21:47.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:47.689+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.695393+0000) 2019-10-13T07:21:47.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:47.690+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.696383+0000) 2019-10-13T07:21:47.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:47.690+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.696782+0000) 2019-10-13T07:21:47.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:47.694+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.698254+0000) 2019-10-13T07:21:47.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:47.697+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:47.700405+0000) 2019-10-13T07:21:50.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:50.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:50.622049+0000) 2019-10-13T07:21:50.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:50.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:50.720649+0000) 2019-10-13T07:21:51.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:51.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.517971+0000) 2019-10-13T07:21:51.520 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:51.517+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:51.517826+0000) 2019-10-13T07:21:52.288 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:52.281+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.285528+0000) 2019-10-13T07:21:52.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:52.694+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.698922+0000) 2019-10-13T07:21:52.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:52.694+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.699405+0000) 2019-10-13T07:21:52.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:52.694+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.699851+0000) 2019-10-13T07:21:52.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:52.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:52.702 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:52.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.700414+0000) 2019-10-13T07:21:52.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:52.697+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.700160+0000) 2019-10-13T07:21:52.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:52.694+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.700816+0000) 2019-10-13T07:21:52.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:52.694+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.701494+0000) 2019-10-13T07:21:52.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:52.701+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:52.706391+0000) 2019-10-13T07:21:52.805 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:21:52.805 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:52.809 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:21:52.809 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:21:54.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:54.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:54.241670+0000) 2019-10-13T07:21:57.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:57.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.079455+0000) 2019-10-13T07:21:57.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:57.490+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.494085+0000) 2019-10-13T07:21:57.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:21:57.698+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.703549+0000) 2019-10-13T07:21:57.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:21:57.698+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.703998+0000) 2019-10-13T07:21:57.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:21:57.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:21:57.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:21:57.698+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.704961+0000) 2019-10-13T07:21:57.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:21:57.701+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.704965+0000) 2019-10-13T07:21:57.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:21:57.702+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.706317+0000) 2019-10-13T07:21:57.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:21:57.702+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.707458+0000) 2019-10-13T07:21:57.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:21:57.702+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.708637+0000) 2019-10-13T07:21:57.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:21:57.709+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:21:57.714059+0000) 2019-10-13T07:22:00.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:00.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:00.622282+0000) 2019-10-13T07:22:00.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:00.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:00.720851+0000) 2019-10-13T07:22:01.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:01.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.518202+0000) 2019-10-13T07:22:01.520 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:01.517+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:01.518021+0000) 2019-10-13T07:22:02.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:02.289+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.291971+0000) 2019-10-13T07:22:02.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:02.702+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.708204+0000) 2019-10-13T07:22:02.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:02.702+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.708673+0000) 2019-10-13T07:22:02.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:02.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:02.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:02.702+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.709012+0000) 2019-10-13T07:22:02.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:02.705+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.709088+0000) 2019-10-13T07:22:02.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:02.706+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.711696+0000) 2019-10-13T07:22:02.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:02.706+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.712587+0000) 2019-10-13T07:22:02.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:02.706+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.712943+0000) 2019-10-13T07:22:02.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:02.713+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:02.718390+0000) 2019-10-13T07:22:04.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:04.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:04.241907+0000) 2019-10-13T07:22:07.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:07.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.079700+0000) 2019-10-13T07:22:07.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:07.489+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.494306+0000) 2019-10-13T07:22:07.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:07.705+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.712490+0000) 2019-10-13T07:22:07.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:07.705+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:07.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:07.709+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.713316+0000) 2019-10-13T07:22:07.716 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:07.709+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.714150+0000) 2019-10-13T07:22:07.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:07.709+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.714472+0000) 2019-10-13T07:22:07.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:07.709+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.714780+0000) 2019-10-13T07:22:07.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:07.709+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.715436+0000) 2019-10-13T07:22:07.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:07.709+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.715554+0000) 2019-10-13T07:22:07.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:07.713+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:07.718502+0000) 2019-10-13T07:22:10.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:10.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:10.622517+0000) 2019-10-13T07:22:10.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:10.717+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:10.721085+0000) 2019-10-13T07:22:11.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:11.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.518393+0000) 2019-10-13T07:22:11.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:11.517+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:11.518247+0000) 2019-10-13T07:22:12.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:12.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.298451+0000) 2019-10-13T07:22:12.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:12.709+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.716554+0000) 2019-10-13T07:22:12.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:12.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:12.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:12.713+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.717906+0000) 2019-10-13T07:22:12.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:12.713+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.718528+0000) 2019-10-13T07:22:12.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:12.713+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.718420+0000) 2019-10-13T07:22:12.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:12.713+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.719294+0000) 2019-10-13T07:22:12.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:12.713+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.719959+0000) 2019-10-13T07:22:12.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:12.713+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.720982+0000) 2019-10-13T07:22:12.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:12.721+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:12.725355+0000) 2019-10-13T07:22:14.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:14.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:14.242112+0000) 2019-10-13T07:22:17.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:17.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.079896+0000) 2019-10-13T07:22:17.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:17.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.494569+0000) 2019-10-13T07:22:17.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:17.713+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.720936+0000) 2019-10-13T07:22:17.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:17.713+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.721536+0000) 2019-10-13T07:22:17.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:17.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49147 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:17.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:17.717+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.721976+0000) 2019-10-13T07:22:17.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:17.717+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.722870+0000) 2019-10-13T07:22:17.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:17.717+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.722760+0000) 2019-10-13T07:22:17.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:17.717+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.723700+0000) 2019-10-13T07:22:17.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:17.717+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.724821+0000) 2019-10-13T07:22:17.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:17.721+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:17.726783+0000) 2019-10-13T07:22:20.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:20.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:20.622739+0000) 2019-10-13T07:22:20.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:20.717+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:20.721304+0000) 2019-10-13T07:22:21.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:21.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.518607+0000) 2019-10-13T07:22:21.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:21.513+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:21.518446+0000) 2019-10-13T07:22:22.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:22.301+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.304717+0000) 2019-10-13T07:22:22.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:22.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:22.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:22.725+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.727009+0000) 2019-10-13T07:22:22.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:22.725+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.727686+0000) 2019-10-13T07:22:22.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:22.725+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.728745+0000) 2019-10-13T07:22:22.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:22.725+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.729243+0000) 2019-10-13T07:22:22.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:22.729+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.729805+0000) 2019-10-13T07:22:22.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:22.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.729705+0000) 2019-10-13T07:22:22.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:22.729+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.730732+0000) 2019-10-13T07:22:22.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:22.729+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:22.732384+0000) 2019-10-13T07:22:22.936 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:22:22.936 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:22.940 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:22:22.940 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:24.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:24.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:24.242326+0000) 2019-10-13T07:22:27.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:27.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.080118+0000) 2019-10-13T07:22:27.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:27.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.494786+0000) 2019-10-13T07:22:27.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:27.729+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.729955+0000) 2019-10-13T07:22:27.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:27.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:27.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:27.729+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.730783+0000) 2019-10-13T07:22:27.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:27.729+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.731299+0000) 2019-10-13T07:22:27.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:27.729+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.732489+0000) 2019-10-13T07:22:27.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:27.729+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.732657+0000) 2019-10-13T07:22:27.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:27.729+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.733085+0000) 2019-10-13T07:22:27.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:27.733+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.734747+0000) 2019-10-13T07:22:27.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:27.733+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:27.736280+0000) 2019-10-13T07:22:30.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:30.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:30.622959+0000) 2019-10-13T07:22:30.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:30.717+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:30.721505+0000) 2019-10-13T07:22:31.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:31.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.518805+0000) 2019-10-13T07:22:31.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:31.513+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:31.518673+0000) 2019-10-13T07:22:32.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:32.309+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.311311+0000) 2019-10-13T07:22:32.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:32.733+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.734473+0000) 2019-10-13T07:22:32.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:32.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:32.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:32.733+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.735489+0000) 2019-10-13T07:22:32.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:32.733+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.735883+0000) 2019-10-13T07:22:32.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:32.729+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.735471+0000) 2019-10-13T07:22:32.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:32.733+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.736447+0000) 2019-10-13T07:22:32.740 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:32.733+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.737597+0000) 2019-10-13T07:22:32.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:32.737+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.738435+0000) 2019-10-13T07:22:32.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:32.737+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:32.741698+0000) 2019-10-13T07:22:34.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:34.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:34.242545+0000) 2019-10-13T07:22:37.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:37.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.080341+0000) 2019-10-13T07:22:37.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:37.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.495012+0000) 2019-10-13T07:22:37.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:37.737+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.738282+0000) 2019-10-13T07:22:37.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:37.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:37.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:37.737+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.739776+0000) 2019-10-13T07:22:37.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:37.737+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.740281+0000) 2019-10-13T07:22:37.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:37.733+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.740031+0000) 2019-10-13T07:22:37.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:37.741+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.742827+0000) 2019-10-13T07:22:37.745 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:37.741+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.743363+0000) 2019-10-13T07:22:37.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:37.741+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.743917+0000) 2019-10-13T07:22:37.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:37.741+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:37.747942+0000) 2019-10-13T07:22:40.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:40.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:40.623164+0000) 2019-10-13T07:22:40.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:40.717+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:40.721704+0000) 2019-10-13T07:22:41.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:41.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.518998+0000) 2019-10-13T07:22:41.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:41.512+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:41.518869+0000) 2019-10-13T07:22:42.319 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:42.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.317289+0000) 2019-10-13T07:22:42.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:42.741+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.743138+0000) 2019-10-13T07:22:42.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:42.741+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.742831+0000) 2019-10-13T07:22:42.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:42.736+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.743908+0000) 2019-10-13T07:22:42.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:42.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:42.747 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:42.741+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.744088+0000) 2019-10-13T07:22:42.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:42.741+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.745256+0000) 2019-10-13T07:22:42.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:42.745+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.745798+0000) 2019-10-13T07:22:42.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:42.745+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.747657+0000) 2019-10-13T07:22:42.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:42.744+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:42.751746+0000) 2019-10-13T07:22:44.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:44.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:44.242756+0000) 2019-10-13T07:22:47.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:47.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.080537+0000) 2019-10-13T07:22:47.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:47.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.495214+0000) 2019-10-13T07:22:47.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:47.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:47.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:47.745+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.748251+0000) 2019-10-13T07:22:47.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:47.745+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.748571+0000) 2019-10-13T07:22:47.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:47.745+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.749219+0000) 2019-10-13T07:22:47.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:47.749+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.750348+0000) 2019-10-13T07:22:47.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:47.749+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.750559+0000) 2019-10-13T07:22:47.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:47.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.750561+0000) 2019-10-13T07:22:47.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:47.749+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.751825+0000) 2019-10-13T07:22:47.758 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:47.748+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:47.756040+0000) 2019-10-13T07:22:50.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:50.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:50.623366+0000) 2019-10-13T07:22:50.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:50.721+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:50.721956+0000) 2019-10-13T07:22:51.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:51.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.519198+0000) 2019-10-13T07:22:51.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:51.516+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:51.519073+0000) 2019-10-13T07:22:52.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:52.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.323888+0000) 2019-10-13T07:22:52.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:52.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.751471+0000) 2019-10-13T07:22:52.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:52.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.751542+0000) 2019-10-13T07:22:52.754 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:52.749+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:52.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:52.749+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.752861+0000) 2019-10-13T07:22:52.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:52.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.753719+0000) 2019-10-13T07:22:52.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:52.753+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.754123+0000) 2019-10-13T07:22:52.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:52.753+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.754473+0000) 2019-10-13T07:22:52.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:52.753+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.755019+0000) 2019-10-13T07:22:52.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:52.752+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:52.759877+0000) 2019-10-13T07:22:53.008 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:22:53.008 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:53.012 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:22:53.012 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:22:54.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:54.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:54.242984+0000) 2019-10-13T07:22:55.106 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1355, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2019-10-13T07:22:55.108 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run result = self._run(*self.args, **self.kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 307, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1343, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2019-10-13T07:22:57.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:57.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.080691+0000) 2019-10-13T07:22:57.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:57.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.495414+0000) 2019-10-13T07:22:57.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:22:57.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:22:57.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:22:57.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.757261+0000) 2019-10-13T07:22:57.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:22:57.753+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.757174+0000) 2019-10-13T07:22:57.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:22:57.753+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.757460+0000) 2019-10-13T07:22:57.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:22:57.752+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.757564+0000) 2019-10-13T07:22:57.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:22:57.757+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.758911+0000) 2019-10-13T07:22:57.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:22:57.757+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.759413+0000) 2019-10-13T07:22:57.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:22:57.757+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.760011+0000) 2019-10-13T07:22:57.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:22:57.756+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:22:57.763924+0000) 2019-10-13T07:23:00.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:00.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:00.623574+0000) 2019-10-13T07:23:00.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:00.721+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:00.722112+0000) 2019-10-13T07:23:01.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:01.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.519399+0000) 2019-10-13T07:23:01.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:01.516+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:01.519301+0000) 2019-10-13T07:23:02.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:02.324+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.330418+0000) 2019-10-13T07:23:02.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:02.757+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.760696+0000) 2019-10-13T07:23:02.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:02.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:02.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:02.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.761580+0000) 2019-10-13T07:23:02.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:02.761+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.762673+0000) 2019-10-13T07:23:02.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:02.761+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.762411+0000) 2019-10-13T07:23:02.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:02.761+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.763345+0000) 2019-10-13T07:23:02.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:02.761+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.764084+0000) 2019-10-13T07:23:02.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:02.761+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.764388+0000) 2019-10-13T07:23:02.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:02.764+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:02.770990+0000) 2019-10-13T07:23:04.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:04.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:04.243245+0000) 2019-10-13T07:23:07.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:07.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.080904+0000) 2019-10-13T07:23:07.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:07.492+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.495608+0000) 2019-10-13T07:23:07.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:07.760+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.764501+0000) 2019-10-13T07:23:07.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:07.760+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.765635+0000) 2019-10-13T07:23:07.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:07.764+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.766027+0000) 2019-10-13T07:23:07.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:07.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:07.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:07.764+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.766519+0000) 2019-10-13T07:23:07.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:07.764+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.767019+0000) 2019-10-13T07:23:07.771 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:07.764+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.767827+0000) 2019-10-13T07:23:07.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:07.764+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.768145+0000) 2019-10-13T07:23:07.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:07.768+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:07.775094+0000) 2019-10-13T07:23:10.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:10.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:10.623776+0000) 2019-10-13T07:23:10.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:10.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:10.722342+0000) 2019-10-13T07:23:11.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:11.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.519590+0000) 2019-10-13T07:23:11.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:11.516+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:11.519494+0000) 2019-10-13T07:23:12.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:12.336+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.336854+0000) 2019-10-13T07:23:12.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:12.764+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.769418+0000) 2019-10-13T07:23:12.772 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:12.768+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.770442+0000) 2019-10-13T07:23:12.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:12.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.770158+0000) 2019-10-13T07:23:12.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:12.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:12.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:12.768+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.770923+0000) 2019-10-13T07:23:12.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:12.768+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.771830+0000) 2019-10-13T07:23:12.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:12.768+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.772327+0000) 2019-10-13T07:23:12.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:12.768+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.772775+0000) 2019-10-13T07:23:12.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:12.772+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:12.778756+0000) 2019-10-13T07:23:14.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:14.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:14.243451+0000) 2019-10-13T07:23:17.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:17.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.081140+0000) 2019-10-13T07:23:17.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:17.492+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.495817+0000) 2019-10-13T07:23:17.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:17.772+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.774066+0000) 2019-10-13T07:23:17.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:17.772+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:17.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:17.772+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.775567+0000) 2019-10-13T07:23:17.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:17.772+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.776001+0000) 2019-10-13T07:23:17.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:17.772+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.776405+0000) 2019-10-13T07:23:17.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:17.772+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.776531+0000) 2019-10-13T07:23:17.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:17.776+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.778691+0000) 2019-10-13T07:23:17.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:17.776+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.778151+0000) 2019-10-13T07:23:17.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:17.776+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:17.782029+0000) 2019-10-13T07:23:20.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:20.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:20.623972+0000) 2019-10-13T07:23:20.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:20.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:20.722586+0000) 2019-10-13T07:23:21.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:21.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.519856+0000) 2019-10-13T07:23:21.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:21.516+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:21.519695+0000) 2019-10-13T07:23:22.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:22.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.343294+0000) 2019-10-13T07:23:22.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:22.776+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.778808+0000) 2019-10-13T07:23:22.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:22.772+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.778860+0000) 2019-10-13T07:23:22.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:22.776+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.779251+0000) 2019-10-13T07:23:22.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:22.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:22.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:22.776+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.780204+0000) 2019-10-13T07:23:22.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:22.776+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.781024+0000) 2019-10-13T07:23:22.783 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:22.780+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.781818+0000) 2019-10-13T07:23:22.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:22.780+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.781827+0000) 2019-10-13T07:23:22.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:22.780+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:22.787222+0000) 2019-10-13T07:23:23.113 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:23:23.113 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:23.116 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:23:23.117 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:24.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:24.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:24.243669+0000) 2019-10-13T07:23:27.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:27.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.081365+0000) 2019-10-13T07:23:27.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:27.492+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.496054+0000) 2019-10-13T07:23:27.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:27.776+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.782832+0000) 2019-10-13T07:23:27.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:27.780+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.783853+0000) 2019-10-13T07:23:27.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:27.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:27.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:27.780+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.784479+0000) 2019-10-13T07:23:27.786 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:27.780+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.784610+0000) 2019-10-13T07:23:27.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:27.780+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.785198+0000) 2019-10-13T07:23:27.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:27.780+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.785579+0000) 2019-10-13T07:23:27.788 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:27.784+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.785807+0000) 2019-10-13T07:23:27.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:27.784+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:27.791820+0000) 2019-10-13T07:23:30.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:30.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:30.624177+0000) 2019-10-13T07:23:30.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:30.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:30.722788+0000) 2019-10-13T07:23:31.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:31.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.520048+0000) 2019-10-13T07:23:31.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:31.516+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:31.519900+0000) 2019-10-13T07:23:32.352 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:32.348+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.349638+0000) 2019-10-13T07:23:32.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:32.784+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.787738+0000) 2019-10-13T07:23:32.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:32.784+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.788388+0000) 2019-10-13T07:23:32.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:32.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:32.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:32.784+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.789031+0000) 2019-10-13T07:23:32.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:32.784+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.789412+0000) 2019-10-13T07:23:32.791 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:32.784+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.789700+0000) 2019-10-13T07:23:32.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:32.784+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.789772+0000) 2019-10-13T07:23:32.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:32.788+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.792059+0000) 2019-10-13T07:23:32.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:32.792+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:32.796710+0000) 2019-10-13T07:23:34.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:34.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:34.243873+0000) 2019-10-13T07:23:37.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:37.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.081569+0000) 2019-10-13T07:23:37.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:37.492+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.496255+0000) 2019-10-13T07:23:37.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:37.788+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.791943+0000) 2019-10-13T07:23:37.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:37.788+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.792368+0000) 2019-10-13T07:23:37.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:37.788+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.793160+0000) 2019-10-13T07:23:37.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:37.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.793138+0000) 2019-10-13T07:23:37.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:37.792+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.793909+0000) 2019-10-13T07:23:37.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:37.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:37.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:37.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.795134+0000) 2019-10-13T07:23:37.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:37.792+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.796060+0000) 2019-10-13T07:23:37.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:37.796+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:37.800618+0000) 2019-10-13T07:23:40.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:40.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:40.624406+0000) 2019-10-13T07:23:40.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:40.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:40.722991+0000) 2019-10-13T07:23:41.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:41.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.520238+0000) 2019-10-13T07:23:41.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:41.520+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:41.520118+0000) 2019-10-13T07:23:42.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:42.356+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.356588+0000) 2019-10-13T07:23:42.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:42.792+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.797306+0000) 2019-10-13T07:23:42.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:42.796+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.797036+0000) 2019-10-13T07:23:42.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:42.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:42.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:42.796+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.799014+0000) 2019-10-13T07:23:42.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:42.796+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.799400+0000) 2019-10-13T07:23:42.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:42.796+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.799713+0000) 2019-10-13T07:23:42.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:42.796+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.800291+0000) 2019-10-13T07:23:42.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:42.796+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.801273+0000) 2019-10-13T07:23:42.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:42.804+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:42.805925+0000) 2019-10-13T07:23:44.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:44.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:44.244107+0000) 2019-10-13T07:23:47.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:47.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.081824+0000) 2019-10-13T07:23:47.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:47.492+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.496464+0000) 2019-10-13T07:23:47.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:47.799+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.800946+0000) 2019-10-13T07:23:47.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:47.800+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.802633+0000) 2019-10-13T07:23:47.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:47.800+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:47.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:47.800+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.803836+0000) 2019-10-13T07:23:47.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:47.800+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.805038+0000) 2019-10-13T07:23:47.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:47.800+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.805075+0000) 2019-10-13T07:23:47.807 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:47.800+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.805364+0000) 2019-10-13T07:23:47.807 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:47.804+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.805772+0000) 2019-10-13T07:23:47.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:47.807+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:47.810901+0000) 2019-10-13T07:23:50.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:50.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:50.624590+0000) 2019-10-13T07:23:50.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:50.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:50.723189+0000) 2019-10-13T07:23:51.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:51.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.520434+0000) 2019-10-13T07:23:51.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:51.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:51.520322+0000) 2019-10-13T07:23:52.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:52.359+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.363195+0000) 2019-10-13T07:23:52.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:52.804+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.806979+0000) 2019-10-13T07:23:52.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:52.804+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.807441+0000) 2019-10-13T07:23:52.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:52.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:52.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:52.804+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.808194+0000) 2019-10-13T07:23:52.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:52.807+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.808278+0000) 2019-10-13T07:23:52.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:52.808+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.812521+0000) 2019-10-13T07:23:52.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:52.808+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.811451+0000) 2019-10-13T07:23:52.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:52.808+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.811961+0000) 2019-10-13T07:23:52.819 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:52.815+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:52.816791+0000) 2019-10-13T07:23:53.247 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:23:53.247 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:53.251 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:23:53.251 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:23:54.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:54.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:54.244315+0000) 2019-10-13T07:23:57.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:57.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.082080+0000) 2019-10-13T07:23:57.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:57.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.496662+0000) 2019-10-13T07:23:57.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:23:57.807+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.810906+0000) 2019-10-13T07:23:57.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:23:57.808+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.812045+0000) 2019-10-13T07:23:57.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:23:57.808+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:23:57.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:23:57.808+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.812598+0000) 2019-10-13T07:23:57.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:23:57.808+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.812924+0000) 2019-10-13T07:23:57.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:23:57.808+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.813476+0000) 2019-10-13T07:23:57.816 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:23:57.812+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.813931+0000) 2019-10-13T07:23:57.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:23:57.812+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.816128+0000) 2019-10-13T07:23:57.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:23:57.819+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:23:57.820817+0000) 2019-10-13T07:24:00.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:00.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:00.624782+0000) 2019-10-13T07:24:00.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:00.720+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:00.723419+0000) 2019-10-13T07:24:01.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:01.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.520649+0000) 2019-10-13T07:24:01.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:01.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:01.520519+0000) 2019-10-13T07:24:02.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:02.367+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.369575+0000) 2019-10-13T07:24:02.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:02.811+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.816003+0000) 2019-10-13T07:24:02.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:02.811+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.816567+0000) 2019-10-13T07:24:02.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:02.811+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.816836+0000) 2019-10-13T07:24:02.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:02.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:02.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:02.815+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.817129+0000) 2019-10-13T07:24:02.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:02.811+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.817153+0000) 2019-10-13T07:24:02.822 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:02.815+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.818584+0000) 2019-10-13T07:24:02.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:02.815+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.818790+0000) 2019-10-13T07:24:02.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:02.819+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:02.821539+0000) 2019-10-13T07:24:04.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:04.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:04.244516+0000) 2019-10-13T07:24:07.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:07.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.082305+0000) 2019-10-13T07:24:07.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:07.495+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.496860+0000) 2019-10-13T07:24:07.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:07.815+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.820132+0000) 2019-10-13T07:24:07.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:07.815+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.820786+0000) 2019-10-13T07:24:07.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:07.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:07.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:07.819+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.821176+0000) 2019-10-13T07:24:07.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:07.819+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.822183+0000) 2019-10-13T07:24:07.826 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:07.819+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.824378+0000) 2019-10-13T07:24:07.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:07.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.823758+0000) 2019-10-13T07:24:07.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:07.819+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.825182+0000) 2019-10-13T07:24:07.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:07.827+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:07.828609+0000) 2019-10-13T07:24:10.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:10.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:10.625016+0000) 2019-10-13T07:24:10.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:10.719+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:10.723638+0000) 2019-10-13T07:24:11.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:11.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.520797+0000) 2019-10-13T07:24:11.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:11.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:11.520756+0000) 2019-10-13T07:24:12.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:12.375+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.376277+0000) 2019-10-13T07:24:12.826 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:12.819+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.824171+0000) 2019-10-13T07:24:12.826 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:12.819+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.825014+0000) 2019-10-13T07:24:12.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:12.819+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.825642+0000) 2019-10-13T07:24:12.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:12.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.826132+0000) 2019-10-13T07:24:12.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:12.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:12.828 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:12.823+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.826860+0000) 2019-10-13T07:24:12.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:12.823+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.826612+0000) 2019-10-13T07:24:12.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:12.823+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.828146+0000) 2019-10-13T07:24:12.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:12.831+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:12.834036+0000) 2019-10-13T07:24:14.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:14.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:14.244714+0000) 2019-10-13T07:24:17.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:17.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.082517+0000) 2019-10-13T07:24:17.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:17.495+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.497095+0000) 2019-10-13T07:24:17.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:17.827+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.831041+0000) 2019-10-13T07:24:17.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:17.827+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.830047+0000) 2019-10-13T07:24:17.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:17.827+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.830517+0000) 2019-10-13T07:24:17.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:17.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:17.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:17.827+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.832254+0000) 2019-10-13T07:24:17.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:17.827+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.832603+0000) 2019-10-13T07:24:17.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:17.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.833131+0000) 2019-10-13T07:24:17.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:17.831+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.834572+0000) 2019-10-13T07:24:17.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:17.835+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:17.837735+0000) 2019-10-13T07:24:20.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:20.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:20.625221+0000) 2019-10-13T07:24:20.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:20.719+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:20.723863+0000) 2019-10-13T07:24:21.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:21.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.520987+0000) 2019-10-13T07:24:21.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:21.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:21.520985+0000) 2019-10-13T07:24:22.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:22.387+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.387498+0000) 2019-10-13T07:24:22.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:22.831+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.835178+0000) 2019-10-13T07:24:22.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:22.831+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.835547+0000) 2019-10-13T07:24:22.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:22.831+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.836058+0000) 2019-10-13T07:24:22.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:22.831+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.836407+0000) 2019-10-13T07:24:22.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:22.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:22.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:22.831+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.837080+0000) 2019-10-13T07:24:22.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:22.835+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.838032+0000) 2019-10-13T07:24:22.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:22.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.838577+0000) 2019-10-13T07:24:22.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:22.843+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:22.845838+0000) 2019-10-13T07:24:23.363 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:24:23.363 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:23.367 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:24:23.367 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:24.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:24.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:24.244914+0000) 2019-10-13T07:24:27.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:27.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.082723+0000) 2019-10-13T07:24:27.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:27.495+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.497291+0000) 2019-10-13T07:24:27.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:27.835+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.839397+0000) 2019-10-13T07:24:27.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:27.835+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.840212+0000) 2019-10-13T07:24:27.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:27.835+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.840938+0000) 2019-10-13T07:24:27.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:27.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:27.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:27.835+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.841607+0000) 2019-10-13T07:24:27.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:27.839+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.841878+0000) 2019-10-13T07:24:27.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:27.839+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.843603+0000) 2019-10-13T07:24:27.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:27.839+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.843356+0000) 2019-10-13T07:24:27.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:27.847+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:27.848653+0000) 2019-10-13T07:24:30.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:30.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:30.625420+0000) 2019-10-13T07:24:30.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:30.719+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:30.724068+0000) 2019-10-13T07:24:31.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:31.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.521185+0000) 2019-10-13T07:24:31.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:31.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:31.521180+0000) 2019-10-13T07:24:32.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:32.395+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.397758+0000) 2019-10-13T07:24:32.846 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:32.839+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.844959+0000) 2019-10-13T07:24:32.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:32.839+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.845433+0000) 2019-10-13T07:24:32.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:32.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:32.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:32.843+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.845837+0000) 2019-10-13T07:24:32.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:32.843+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.846125+0000) 2019-10-13T07:24:32.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:32.843+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.846288+0000) 2019-10-13T07:24:32.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:32.843+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.846427+0000) 2019-10-13T07:24:32.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:32.843+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.846949+0000) 2019-10-13T07:24:32.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:32.851+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:32.854084+0000) 2019-10-13T07:24:34.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:34.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:34.245123+0000) 2019-10-13T07:24:37.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:37.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.082984+0000) 2019-10-13T07:24:37.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:37.491+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.497498+0000) 2019-10-13T07:24:37.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:37.843+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.848710+0000) 2019-10-13T07:24:37.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:37.843+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.848994+0000) 2019-10-13T07:24:37.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:37.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:37.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:37.847+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.850817+0000) 2019-10-13T07:24:37.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:37.847+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.852568+0000) 2019-10-13T07:24:37.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:37.847+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.852261+0000) 2019-10-13T07:24:37.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:37.847+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.853431+0000) 2019-10-13T07:24:37.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:37.851+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.853771+0000) 2019-10-13T07:24:37.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:37.851+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:37.855485+0000) 2019-10-13T07:24:40.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:40.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:40.625656+0000) 2019-10-13T07:24:40.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:40.723+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:40.724303+0000) 2019-10-13T07:24:41.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:41.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.521381+0000) 2019-10-13T07:24:41.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:41.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:41.521375+0000) 2019-10-13T07:24:42.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:42.403+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.404335+0000) 2019-10-13T07:24:42.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:42.847+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.853692+0000) 2019-10-13T07:24:42.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:42.851+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.854188+0000) 2019-10-13T07:24:42.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:42.851+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.854605+0000) 2019-10-13T07:24:42.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:42.851+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.854968+0000) 2019-10-13T07:24:42.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:42.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.855304+0000) 2019-10-13T07:24:42.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:42.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:42.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:42.851+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.856059+0000) 2019-10-13T07:24:42.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:42.851+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.856873+0000) 2019-10-13T07:24:42.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:42.859+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:42.862599+0000) 2019-10-13T07:24:44.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:44.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:44.245348+0000) 2019-10-13T07:24:47.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:47.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.083196+0000) 2019-10-13T07:24:47.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:47.495+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.497691+0000) 2019-10-13T07:24:47.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:47.855+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.859225+0000) 2019-10-13T07:24:47.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:47.855+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.860224+0000) 2019-10-13T07:24:47.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:47.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:47.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:47.855+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.861607+0000) 2019-10-13T07:24:47.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:47.859+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.862149+0000) 2019-10-13T07:24:47.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:47.859+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.862728+0000) 2019-10-13T07:24:47.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:47.859+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.863599+0000) 2019-10-13T07:24:47.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:47.859+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.863683+0000) 2019-10-13T07:24:47.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:47.863+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:47.865317+0000) 2019-10-13T07:24:50.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:50.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:50.625861+0000) 2019-10-13T07:24:50.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:50.719+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:50.724528+0000) 2019-10-13T07:24:51.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:51.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.521603+0000) 2019-10-13T07:24:51.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:51.519+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:51.521572+0000) 2019-10-13T07:24:52.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:52.406+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.410762+0000) 2019-10-13T07:24:52.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:52.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.864253+0000) 2019-10-13T07:24:52.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:52.862+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.864393+0000) 2019-10-13T07:24:52.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:52.859+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.864936+0000) 2019-10-13T07:24:52.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:52.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:52.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:52.859+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.865503+0000) 2019-10-13T07:24:52.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:52.863+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.866736+0000) 2019-10-13T07:24:52.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:52.863+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.867538+0000) 2019-10-13T07:24:52.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:52.863+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.868402+0000) 2019-10-13T07:24:52.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:52.866+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:52.870860+0000) 2019-10-13T07:24:53.471 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:24:53.471 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:53.475 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:24:53.475 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:24:54.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:54.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:54.245552+0000) 2019-10-13T07:24:57.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:57.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.083420+0000) 2019-10-13T07:24:57.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:57.495+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.497929+0000) 2019-10-13T07:24:57.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:24:57.862+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.867393+0000) 2019-10-13T07:24:57.870 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:24:57.863+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.868724+0000) 2019-10-13T07:24:57.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:24:57.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:24:57.872 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:24:57.863+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.869621+0000) 2019-10-13T07:24:57.872 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:24:57.867+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.869776+0000) 2019-10-13T07:24:57.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:24:57.867+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.870116+0000) 2019-10-13T07:24:57.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:24:57.867+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.871121+0000) 2019-10-13T07:24:57.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:24:57.867+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.873625+0000) 2019-10-13T07:24:57.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:24:57.874+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:24:57.876908+0000) 2019-10-13T07:25:00.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:00.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:00.626091+0000) 2019-10-13T07:25:00.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:00.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:00.724731+0000) 2019-10-13T07:25:01.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:01.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.521806+0000) 2019-10-13T07:25:01.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:01.518+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:01.521766+0000) 2019-10-13T07:25:02.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:02.414+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.417129+0000) 2019-10-13T07:25:02.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:02.866+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.872810+0000) 2019-10-13T07:25:02.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:02.870+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.872659+0000) 2019-10-13T07:25:02.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:02.870+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.873968+0000) 2019-10-13T07:25:02.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:02.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:02.876 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:02.870+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.874879+0000) 2019-10-13T07:25:02.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:02.870+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.875543+0000) 2019-10-13T07:25:02.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:02.870+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.877058+0000) 2019-10-13T07:25:02.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:02.874+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.878190+0000) 2019-10-13T07:25:02.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:02.878+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:02.881682+0000) 2019-10-13T07:25:04.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:04.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:04.245788+0000) 2019-10-13T07:25:07.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:07.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.083648+0000) 2019-10-13T07:25:07.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:07.494+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.498142+0000) 2019-10-13T07:25:07.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:07.874+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.878009+0000) 2019-10-13T07:25:07.880 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:07.874+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.878485+0000) 2019-10-13T07:25:07.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:07.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:07.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:07.874+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.879439+0000) 2019-10-13T07:25:07.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:07.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.880047+0000) 2019-10-13T07:25:07.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:07.874+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.879769+0000) 2019-10-13T07:25:07.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:07.874+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.880661+0000) 2019-10-13T07:25:07.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:07.882+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.885592+0000) 2019-10-13T07:25:07.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:07.874+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:07.881476+0000) 2019-10-13T07:25:10.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:10.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:10.626320+0000) 2019-10-13T07:25:10.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:10.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:10.724924+0000) 2019-10-13T07:25:11.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:11.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.521999+0000) 2019-10-13T07:25:11.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:11.518+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:11.521969+0000) 2019-10-13T07:25:12.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:12.426+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.428762+0000) 2019-10-13T07:25:12.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:12.878+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.882724+0000) 2019-10-13T07:25:12.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:12.878+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.882784+0000) 2019-10-13T07:25:12.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:12.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:12.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:12.878+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.883388+0000) 2019-10-13T07:25:12.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:12.878+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.884459+0000) 2019-10-13T07:25:12.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:12.878+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.885156+0000) 2019-10-13T07:25:12.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:12.878+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.884888+0000) 2019-10-13T07:25:12.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:12.882+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.886120+0000) 2019-10-13T07:25:12.891 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:12.886+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:12.889415+0000) 2019-10-13T07:25:14.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:14.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:14.245984+0000) 2019-10-13T07:25:17.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:17.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.083870+0000) 2019-10-13T07:25:17.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:17.494+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.498341+0000) 2019-10-13T07:25:17.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:17.882+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.886968+0000) 2019-10-13T07:25:17.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:17.882+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:17.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:17.882+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.888062+0000) 2019-10-13T07:25:17.890 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:17.882+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.888332+0000) 2019-10-13T07:25:17.891 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:17.882+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.889447+0000) 2019-10-13T07:25:17.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:17.886+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.890551+0000) 2019-10-13T07:25:17.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:17.886+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.890785+0000) 2019-10-13T07:25:17.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:17.886+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.891389+0000) 2019-10-13T07:25:17.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:17.890+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:17.895229+0000) 2019-10-13T07:25:20.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:20.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:20.626536+0000) 2019-10-13T07:25:20.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:20.718+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:20.725119+0000) 2019-10-13T07:25:21.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:21.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.522198+0000) 2019-10-13T07:25:21.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:21.518+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:21.522164+0000) 2019-10-13T07:25:22.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:22.430+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.435216+0000) 2019-10-13T07:25:22.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:22.886+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.891121+0000) 2019-10-13T07:25:22.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:22.886+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.891530+0000) 2019-10-13T07:25:22.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:22.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:22.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:22.886+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.892564+0000) 2019-10-13T07:25:22.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:22.890+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.892398+0000) 2019-10-13T07:25:22.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:22.886+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.893633+0000) 2019-10-13T07:25:22.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:22.890+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.893869+0000) 2019-10-13T07:25:22.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:22.890+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.894463+0000) 2019-10-13T07:25:22.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:22.894+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:22.897829+0000) 2019-10-13T07:25:23.723 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:25:23.723 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:23.727 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:25:23.727 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:24.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:24.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:24.246188+0000) 2019-10-13T07:25:27.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:27.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.084116+0000) 2019-10-13T07:25:27.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:27.494+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.498586+0000) 2019-10-13T07:25:27.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:27.890+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.896620+0000) 2019-10-13T07:25:27.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:27.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:27.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:27.890+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.897348+0000) 2019-10-13T07:25:27.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:27.894+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.897177+0000) 2019-10-13T07:25:27.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:27.894+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.898128+0000) 2019-10-13T07:25:27.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:27.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.898361+0000) 2019-10-13T07:25:27.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:27.894+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.898587+0000) 2019-10-13T07:25:27.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:27.894+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.899136+0000) 2019-10-13T07:25:27.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:27.902+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:27.904666+0000) 2019-10-13T07:25:30.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:30.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:30.626787+0000) 2019-10-13T07:25:30.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:30.722+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:30.725314+0000) 2019-10-13T07:25:31.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:31.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.522367+0000) 2019-10-13T07:25:31.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:31.518+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:31.522395+0000) 2019-10-13T07:25:32.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:32.438+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.441649+0000) 2019-10-13T07:25:32.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:32.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.901073+0000) 2019-10-13T07:25:32.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:32.894+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.901223+0000) 2019-10-13T07:25:32.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:32.898+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.901167+0000) 2019-10-13T07:25:32.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:32.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:32.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:32.898+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.902289+0000) 2019-10-13T07:25:32.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:32.898+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.903119+0000) 2019-10-13T07:25:32.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:32.898+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.903388+0000) 2019-10-13T07:25:32.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:32.898+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.903832+0000) 2019-10-13T07:25:32.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:32.906+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:32.910123+0000) 2019-10-13T07:25:34.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:34.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:34.246420+0000) 2019-10-13T07:25:37.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:37.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.084333+0000) 2019-10-13T07:25:37.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:37.498+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.498817+0000) 2019-10-13T07:25:37.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:37.898+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.905522+0000) 2019-10-13T07:25:37.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:37.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:37.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:37.902+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.907059+0000) 2019-10-13T07:25:37.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:37.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.907923+0000) 2019-10-13T07:25:37.911 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:37.902+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.909064+0000) 2019-10-13T07:25:37.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:37.902+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.907523+0000) 2019-10-13T07:25:37.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:37.906+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.909819+0000) 2019-10-13T07:25:37.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:37.906+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.909828+0000) 2019-10-13T07:25:37.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:37.914+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:37.917042+0000) 2019-10-13T07:25:40.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:40.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:40.626992+0000) 2019-10-13T07:25:40.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:40.722+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:40.725544+0000) 2019-10-13T07:25:41.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:41.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.522605+0000) 2019-10-13T07:25:41.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:41.522+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:41.522574+0000) 2019-10-13T07:25:42.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:42.446+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.448026+0000) 2019-10-13T07:25:42.912 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:42.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.910113+0000) 2019-10-13T07:25:42.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:42.906+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.910443+0000) 2019-10-13T07:25:42.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:42.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:42.914 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:42.906+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.911897+0000) 2019-10-13T07:25:42.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:42.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.912426+0000) 2019-10-13T07:25:42.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:42.906+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.912372+0000) 2019-10-13T07:25:42.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:42.910+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.912536+0000) 2019-10-13T07:25:42.916 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:42.906+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.913532+0000) 2019-10-13T07:25:42.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:42.914+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:42.917227+0000) 2019-10-13T07:25:44.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:44.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:44.246656+0000) 2019-10-13T07:25:47.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:47.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.084528+0000) 2019-10-13T07:25:47.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:47.498+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.499019+0000) 2019-10-13T07:25:47.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:47.910+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.914029+0000) 2019-10-13T07:25:47.917 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:47.910+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.915358+0000) 2019-10-13T07:25:47.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:47.910+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.915565+0000) 2019-10-13T07:25:47.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:47.910+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.915983+0000) 2019-10-13T07:25:47.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:47.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:47.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:47.910+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.916568+0000) 2019-10-13T07:25:47.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:47.910+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.916870+0000) 2019-10-13T07:25:47.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:47.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.919280+0000) 2019-10-13T07:25:47.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:47.918+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:47.921309+0000) 2019-10-13T07:25:50.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:50.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:50.627254+0000) 2019-10-13T07:25:50.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:50.722+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:50.725783+0000) 2019-10-13T07:25:51.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:51.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.522824+0000) 2019-10-13T07:25:51.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:51.522+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:51.522747+0000) 2019-10-13T07:25:52.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:52.454+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.454503+0000) 2019-10-13T07:25:52.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:52.914+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.918564+0000) 2019-10-13T07:25:52.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:52.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.920316+0000) 2019-10-13T07:25:52.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:52.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.920952+0000) 2019-10-13T07:25:52.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:52.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:52.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:52.918+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.921286+0000) 2019-10-13T07:25:52.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:52.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.922566+0000) 2019-10-13T07:25:52.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:52.918+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.922760+0000) 2019-10-13T07:25:52.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:52.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.924344+0000) 2019-10-13T07:25:52.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:52.926+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:52.929711+0000) 2019-10-13T07:25:53.783 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:25:53.783 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:53.787 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:25:53.787 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:25:54.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:54.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:54.246888+0000) 2019-10-13T07:25:57.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:57.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.084793+0000) 2019-10-13T07:25:57.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:57.493+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.499236+0000) 2019-10-13T07:25:57.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:25:57.917+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.923882+0000) 2019-10-13T07:25:57.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:25:57.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:25:57.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:25:57.917+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.925476+0000) 2019-10-13T07:25:57.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:25:57.921+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.925797+0000) 2019-10-13T07:25:57.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:25:57.921+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.926521+0000) 2019-10-13T07:25:57.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:25:57.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.927038+0000) 2019-10-13T07:25:57.930 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:25:57.921+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.925899+0000) 2019-10-13T07:25:57.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:25:57.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.927478+0000) 2019-10-13T07:25:57.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:25:57.926+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:25:57.931800+0000) 2019-10-13T07:26:00.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:00.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:00.627458+0000) 2019-10-13T07:26:00.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:00.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:00.725983+0000) 2019-10-13T07:26:01.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:01.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.523014+0000) 2019-10-13T07:26:01.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:01.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:01.522986+0000) 2019-10-13T07:26:02.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:02.457+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.460882+0000) 2019-10-13T07:26:02.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:02.925+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:02.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:02.925+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.930501+0000) 2019-10-13T07:26:02.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:02.925+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.931076+0000) 2019-10-13T07:26:02.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:02.925+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.931793+0000) 2019-10-13T07:26:02.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:02.925+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.932626+0000) 2019-10-13T07:26:02.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:02.925+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.933162+0000) 2019-10-13T07:26:02.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:02.929+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.933939+0000) 2019-10-13T07:26:02.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:02.929+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.934013+0000) 2019-10-13T07:26:02.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:02.933+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:02.938820+0000) 2019-10-13T07:26:04.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:04.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:04.247120+0000) 2019-10-13T07:26:07.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:07.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.085012+0000) 2019-10-13T07:26:07.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:07.497+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.499435+0000) 2019-10-13T07:26:07.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:07.929+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.933642+0000) 2019-10-13T07:26:07.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:07.929+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.934519+0000) 2019-10-13T07:26:07.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:07.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:07.937 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:07.929+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.934801+0000) 2019-10-13T07:26:07.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:07.929+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.935702+0000) 2019-10-13T07:26:07.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:07.929+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.936539+0000) 2019-10-13T07:26:07.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:07.929+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.935957+0000) 2019-10-13T07:26:07.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:07.933+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.936957+0000) 2019-10-13T07:26:07.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:07.941+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:07.944806+0000) 2019-10-13T07:26:10.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:10.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:10.627692+0000) 2019-10-13T07:26:10.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:10.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:10.726193+0000) 2019-10-13T07:26:11.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:11.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.523192+0000) 2019-10-13T07:26:11.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:11.517+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:11.523184+0000) 2019-10-13T07:26:12.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:12.461+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.467193+0000) 2019-10-13T07:26:12.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:12.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.938478+0000) 2019-10-13T07:26:12.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:12.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:12.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:12.937+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.939927+0000) 2019-10-13T07:26:12.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:12.937+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.940494+0000) 2019-10-13T07:26:12.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:12.937+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.940687+0000) 2019-10-13T07:26:12.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:12.941+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.941833+0000) 2019-10-13T07:26:12.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:12.941+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.941864+0000) 2019-10-13T07:26:12.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:12.941+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.942706+0000) 2019-10-13T07:26:12.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:12.941+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:12.946796+0000) 2019-10-13T07:26:14.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:14.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:14.247354+0000) 2019-10-13T07:26:17.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:17.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.085227+0000) 2019-10-13T07:26:17.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:17.497+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.499642+0000) 2019-10-13T07:26:17.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:17.941+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.942687+0000) 2019-10-13T07:26:17.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:17.937+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.943292+0000) 2019-10-13T07:26:17.946 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:17.941+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:17.946 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:17.941+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.944684+0000) 2019-10-13T07:26:17.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:17.941+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.945065+0000) 2019-10-13T07:26:17.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:17.941+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.945382+0000) 2019-10-13T07:26:17.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:17.945+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.946265+0000) 2019-10-13T07:26:17.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:17.945+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.946670+0000) 2019-10-13T07:26:17.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:17.945+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:17.950121+0000) 2019-10-13T07:26:20.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:20.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:20.627901+0000) 2019-10-13T07:26:20.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:20.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:20.726389+0000) 2019-10-13T07:26:21.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:21.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.523424+0000) 2019-10-13T07:26:21.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:21.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:21.523384+0000) 2019-10-13T07:26:22.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:22.469+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.473656+0000) 2019-10-13T07:26:22.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:22.941+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.946116+0000) 2019-10-13T07:26:22.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:22.945+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.947719+0000) 2019-10-13T07:26:22.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:22.945+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.948114+0000) 2019-10-13T07:26:22.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:22.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:22.950 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:22.945+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.948817+0000) 2019-10-13T07:26:22.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:22.945+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.949424+0000) 2019-10-13T07:26:22.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:22.949+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.950035+0000) 2019-10-13T07:26:22.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:22.949+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.950330+0000) 2019-10-13T07:26:22.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:22.949+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:22.955944+0000) 2019-10-13T07:26:23.844 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:26:23.845 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:23.848 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:26:23.848 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:24.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:24.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:24.247556+0000) 2019-10-13T07:26:27.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:27.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.085447+0000) 2019-10-13T07:26:27.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:27.497+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.499852+0000) 2019-10-13T07:26:27.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:27.949+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.952303+0000) 2019-10-13T07:26:27.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:27.949+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.952598+0000) 2019-10-13T07:26:27.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:27.949+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.953159+0000) 2019-10-13T07:26:27.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:27.949+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.953332+0000) 2019-10-13T07:26:27.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:27.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.953426+0000) 2019-10-13T07:26:27.956 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:27.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:27.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:27.953+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.954843+0000) 2019-10-13T07:26:27.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:27.953+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.955703+0000) 2019-10-13T07:26:27.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:27.957+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:27.960903+0000) 2019-10-13T07:26:30.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:30.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:30.628106+0000) 2019-10-13T07:26:30.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:30.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:30.726613+0000) 2019-10-13T07:26:31.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:31.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.523628+0000) 2019-10-13T07:26:31.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:31.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:31.523581+0000) 2019-10-13T07:26:32.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:32.477+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.480190+0000) 2019-10-13T07:26:32.960 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:32.957+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:33.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:32.957+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.958753+0000) 2019-10-13T07:26:33.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:32.957+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.960826+0000) 2019-10-13T07:26:33.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:32.961+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.967672+0000) 2019-10-13T07:26:33.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:32.957+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.959173+0000) 2019-10-13T07:26:33.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:32.957+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.959565+0000) 2019-10-13T07:26:33.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:32.957+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.959835+0000) 2019-10-13T07:26:33.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:32.957+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.960401+0000) 2019-10-13T07:26:33.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:32.961+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:32.962373+0000) 2019-10-13T07:26:34.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:34.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:34.247764+0000) 2019-10-13T07:26:37.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:37.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.085685+0000) 2019-10-13T07:26:37.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:37.497+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.500093+0000) 2019-10-13T07:26:37.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:37.957+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.961587+0000) 2019-10-13T07:26:37.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:37.961+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:37.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:37.961+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.963132+0000) 2019-10-13T07:26:37.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:37.961+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.963697+0000) 2019-10-13T07:26:37.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:37.961+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.964075+0000) 2019-10-13T07:26:37.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:37.961+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.964275+0000) 2019-10-13T07:26:37.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:37.961+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.965698+0000) 2019-10-13T07:26:37.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:37.961+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.965243+0000) 2019-10-13T07:26:37.973 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:37.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:37.970935+0000) 2019-10-13T07:26:40.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:40.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:40.628347+0000) 2019-10-13T07:26:40.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:40.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:40.726811+0000) 2019-10-13T07:26:41.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:41.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.523853+0000) 2019-10-13T07:26:41.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:41.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:41.523784+0000) 2019-10-13T07:26:42.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:42.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.486779+0000) 2019-10-13T07:26:42.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:42.961+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.965150+0000) 2019-10-13T07:26:42.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:42.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50815 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:42.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:42.965+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.967253+0000) 2019-10-13T07:26:42.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:42.965+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.966713+0000) 2019-10-13T07:26:42.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:42.965+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.967487+0000) 2019-10-13T07:26:42.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:42.965+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.968608+0000) 2019-10-13T07:26:42.971 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:42.965+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.968962+0000) 2019-10-13T07:26:42.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:42.969+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.971621+0000) 2019-10-13T07:26:42.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:42.969+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:42.974179+0000) 2019-10-13T07:26:44.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:44.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:44.247964+0000) 2019-10-13T07:26:47.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:47.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.085929+0000) 2019-10-13T07:26:47.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:47.497+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.500320+0000) 2019-10-13T07:26:47.972 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:47.969+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.970737+0000) 2019-10-13T07:26:47.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:47.965+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.970962+0000) 2019-10-13T07:26:47.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:47.969+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:47.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:47.969+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.971928+0000) 2019-10-13T07:26:47.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:47.969+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.972253+0000) 2019-10-13T07:26:47.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:47.969+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.972561+0000) 2019-10-13T07:26:47.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:47.969+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.972816+0000) 2019-10-13T07:26:47.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:47.969+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.973289+0000) 2019-10-13T07:26:47.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:47.973+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:47.979421+0000) 2019-10-13T07:26:50.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:50.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:50.628563+0000) 2019-10-13T07:26:50.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:50.725+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:50.727019+0000) 2019-10-13T07:26:51.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:51.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.524035+0000) 2019-10-13T07:26:51.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:51.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:51.523984+0000) 2019-10-13T07:26:52.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:52.489+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.493165+0000) 2019-10-13T07:26:52.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:52.972+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.974670+0000) 2019-10-13T07:26:52.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:52.969+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.974802+0000) 2019-10-13T07:26:52.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:52.972+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.976617+0000) 2019-10-13T07:26:52.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:52.972+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.977122+0000) 2019-10-13T07:26:52.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:52.972+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.977484+0000) 2019-10-13T07:26:52.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:52.976+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.978127+0000) 2019-10-13T07:26:52.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:52.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:52.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:52.976+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.978344+0000) 2019-10-13T07:26:52.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:52.977+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:52.982109+0000) 2019-10-13T07:26:53.975 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:26:53.975 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:53.980 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:26:53.980 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:26:54.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:54.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:54.248192+0000) 2019-10-13T07:26:57.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:57.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.086129+0000) 2019-10-13T07:26:57.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:57.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.500537+0000) 2019-10-13T07:26:57.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:26:57.980+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.981721+0000) 2019-10-13T07:26:57.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:26:57.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:26:57.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:26:57.977+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.982195+0000) 2019-10-13T07:26:57.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:26:57.980+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.983113+0000) 2019-10-13T07:26:57.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:26:57.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.983630+0000) 2019-10-13T07:26:57.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:26:57.980+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.983861+0000) 2019-10-13T07:26:57.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:26:57.980+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.984361+0000) 2019-10-13T07:26:57.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:26:57.980+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.985172+0000) 2019-10-13T07:26:57.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:26:57.989+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:26:57.992269+0000) 2019-10-13T07:27:00.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:00.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:00.628729+0000) 2019-10-13T07:27:00.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:00.724+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:00.727218+0000) 2019-10-13T07:27:01.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:01.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.524230+0000) 2019-10-13T07:27:01.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:01.521+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:01.524196+0000) 2019-10-13T07:27:02.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:02.493+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.499873+0000) 2019-10-13T07:27:02.987 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:02.980+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.985255+0000) 2019-10-13T07:27:02.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:02.981+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.985186+0000) 2019-10-13T07:27:02.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:02.984+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.986665+0000) 2019-10-13T07:27:02.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:02.984+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:02.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:02.984+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.987254+0000) 2019-10-13T07:27:02.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:02.984+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.987851+0000) 2019-10-13T07:27:02.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:02.984+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.988211+0000) 2019-10-13T07:27:02.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:02.984+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.988599+0000) 2019-10-13T07:27:02.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:02.989+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:02.995697+0000) 2019-10-13T07:27:04.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:04.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:04.248430+0000) 2019-10-13T07:27:07.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:07.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.086329+0000) 2019-10-13T07:27:07.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:07.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.500753+0000) 2019-10-13T07:27:07.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:07.988+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.989892+0000) 2019-10-13T07:27:07.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:07.984+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.990253+0000) 2019-10-13T07:27:07.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:07.988+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.991080+0000) 2019-10-13T07:27:07.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:07.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:07.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:07.988+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.991625+0000) 2019-10-13T07:27:07.994 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:07.988+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.991967+0000) 2019-10-13T07:27:07.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:07.988+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.992252+0000) 2019-10-13T07:27:07.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:07.988+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.992546+0000) 2019-10-13T07:27:08.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:07.992+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:07.999955+0000) 2019-10-13T07:27:10.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:10.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:10.628933+0000) 2019-10-13T07:27:10.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:10.724+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:10.727416+0000) 2019-10-13T07:27:11.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:11.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.524422+0000) 2019-10-13T07:27:11.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:11.520+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:11.524395+0000) 2019-10-13T07:27:12.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:12.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.506237+0000) 2019-10-13T07:27:12.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:12.992+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.993856+0000) 2019-10-13T07:27:12.996 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:12.992+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.995171+0000) 2019-10-13T07:27:12.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:12.988+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.995091+0000) 2019-10-13T07:27:12.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:12.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51005 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:12.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:12.992+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.996505+0000) 2019-10-13T07:27:12.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:12.992+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.996614+0000) 2019-10-13T07:27:12.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:12.992+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.997476+0000) 2019-10-13T07:27:13.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:12.996+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:12.998632+0000) 2019-10-13T07:27:13.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:12.996+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:13.003043+0000) 2019-10-13T07:27:14.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:14.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:14.248646+0000) 2019-10-13T07:27:17.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:17.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.086560+0000) 2019-10-13T07:27:17.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:17.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.500961+0000) 2019-10-13T07:27:18.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:17.996+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.999158+0000) 2019-10-13T07:27:18.001 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:17.996+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.999395+0000) 2019-10-13T07:27:18.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:17.996+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:17.999961+0000) 2019-10-13T07:27:18.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:17.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:18.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:17.996+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.000666+0000) 2019-10-13T07:27:18.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:17.996+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.001637+0000) 2019-10-13T07:27:18.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:18.000+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.002308+0000) 2019-10-13T07:27:18.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:18.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.003299+0000) 2019-10-13T07:27:18.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:18.000+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:18.007823+0000) 2019-10-13T07:27:20.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:20.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:20.629158+0000) 2019-10-13T07:27:20.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:20.724+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:20.727611+0000) 2019-10-13T07:27:21.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:21.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.524622+0000) 2019-10-13T07:27:21.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:21.520+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:21.524599+0000) 2019-10-13T07:27:22.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:22.508+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:22.512667+0000) 2019-10-13T07:27:23.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:23.000+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.003793+0000) 2019-10-13T07:27:23.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:23.000+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.004328+0000) 2019-10-13T07:27:23.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:23.000+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.004659+0000) 2019-10-13T07:27:23.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:23.000+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.004461+0000) 2019-10-13T07:27:23.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:23.004+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:23.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:23.004+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.005982+0000) 2019-10-13T07:27:23.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:23.004+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.006334+0000) 2019-10-13T07:27:23.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:23.004+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.006967+0000) 2019-10-13T07:27:23.014 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:23.008+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:23.012475+0000) 2019-10-13T07:27:24.045 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:27:24.045 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:24.049 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:27:24.049 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:24.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:24.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:24.248851+0000) 2019-10-13T07:27:27.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:27.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:27.086818+0000) 2019-10-13T07:27:27.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:27.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:27.501155+0000) 2019-10-13T07:27:28.012 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:28.008+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.010302+0000) 2019-10-13T07:27:28.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:28.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:28.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:28.004+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.010144+0000) 2019-10-13T07:27:28.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:28.008+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.011095+0000) 2019-10-13T07:27:28.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:28.008+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.011393+0000) 2019-10-13T07:27:28.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:28.008+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.011760+0000) 2019-10-13T07:27:28.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:28.008+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.012406+0000) 2019-10-13T07:27:28.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:28.008+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.012899+0000) 2019-10-13T07:27:28.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:28.012+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:28.017529+0000) 2019-10-13T07:27:30.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:30.628+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:30.629408+0000) 2019-10-13T07:27:30.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:30.724+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:30.727835+0000) 2019-10-13T07:27:31.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:31.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.524793+0000) 2019-10-13T07:27:31.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:31.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:31.524824+0000) 2019-10-13T07:27:32.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:32.520+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:32.523567+0000) 2019-10-13T07:27:33.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:33.008+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.013310+0000) 2019-10-13T07:27:33.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:33.012+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.014740+0000) 2019-10-13T07:27:33.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:33.012+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:33.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:33.012+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.015324+0000) 2019-10-13T07:27:33.017 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:33.012+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.015620+0000) 2019-10-13T07:27:33.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:33.012+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.016222+0000) 2019-10-13T07:27:33.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:33.012+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.017094+0000) 2019-10-13T07:27:33.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:33.016+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.018725+0000) 2019-10-13T07:27:33.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:33.020+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:33.024678+0000) 2019-10-13T07:27:34.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:34.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:34.249070+0000) 2019-10-13T07:27:37.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:37.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:37.087031+0000) 2019-10-13T07:27:37.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:37.496+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:37.501396+0000) 2019-10-13T07:27:38.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:38.016+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.018319+0000) 2019-10-13T07:27:38.020 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:38.016+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.019142+0000) 2019-10-13T07:27:38.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:38.016+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.019531+0000) 2019-10-13T07:27:38.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:38.016+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.019979+0000) 2019-10-13T07:27:38.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:38.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:38.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:38.012+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.019900+0000) 2019-10-13T07:27:38.022 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:38.016+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.021094+0000) 2019-10-13T07:27:38.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:38.020+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.022779+0000) 2019-10-13T07:27:38.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:38.024+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:38.028171+0000) 2019-10-13T07:27:40.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:40.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:40.629611+0000) 2019-10-13T07:27:40.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:40.724+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:40.728035+0000) 2019-10-13T07:27:41.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:41.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.525011+0000) 2019-10-13T07:27:41.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:41.520+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:41.525034+0000) 2019-10-13T07:27:42.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:42.528+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:42.530161+0000) 2019-10-13T07:27:43.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:43.020+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.022991+0000) 2019-10-13T07:27:43.025 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:43.020+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.023706+0000) 2019-10-13T07:27:43.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:43.020+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.024363+0000) 2019-10-13T07:27:43.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:43.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:43.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:43.016+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.024069+0000) 2019-10-13T07:27:43.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:43.020+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.025396+0000) 2019-10-13T07:27:43.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:43.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.026128+0000) 2019-10-13T07:27:43.031 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:43.028+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.030278+0000) 2019-10-13T07:27:43.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:43.024+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:43.031300+0000) 2019-10-13T07:27:44.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:44.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:44.249223+0000) 2019-10-13T07:27:47.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:47.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:47.087267+0000) 2019-10-13T07:27:47.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:47.500+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:47.501629+0000) 2019-10-13T07:27:48.029 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:48.024+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.027390+0000) 2019-10-13T07:27:48.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:48.024+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.028136+0000) 2019-10-13T07:27:48.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:48.024+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.028495+0000) 2019-10-13T07:27:48.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:48.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51224 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:48.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:48.024+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.029131+0000) 2019-10-13T07:27:48.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:48.024+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.029700+0000) 2019-10-13T07:27:48.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:48.024+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.029676+0000) 2019-10-13T07:27:48.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:48.028+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.031350+0000) 2019-10-13T07:27:48.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:48.032+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:48.037838+0000) 2019-10-13T07:27:50.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:50.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:50.629840+0000) 2019-10-13T07:27:50.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:50.723+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:50.728250+0000) 2019-10-13T07:27:51.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:51.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.525201+0000) 2019-10-13T07:27:51.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:51.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:51.525225+0000) 2019-10-13T07:27:52.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:52.540+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:52.540989+0000) 2019-10-13T07:27:53.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:53.027+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.031933+0000) 2019-10-13T07:27:53.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:53.027+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.032747+0000) 2019-10-13T07:27:53.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:53.027+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.033153+0000) 2019-10-13T07:27:53.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:53.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:53.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:53.027+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.033533+0000) 2019-10-13T07:27:53.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:53.028+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.033943+0000) 2019-10-13T07:27:53.036 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:53.031+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.035140+0000) 2019-10-13T07:27:53.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:53.031+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.035704+0000) 2019-10-13T07:27:53.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:53.036+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:53.043157+0000) 2019-10-13T07:27:54.169 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:27:54.169 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:54.173 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:27:54.173 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:27:54.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:54.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:54.249432+0000) 2019-10-13T07:27:57.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:57.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:57.087488+0000) 2019-10-13T07:27:57.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:57.499+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:57.501852+0000) 2019-10-13T07:27:58.038 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:27:58.031+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.036620+0000) 2019-10-13T07:27:58.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:27:58.031+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.037157+0000) 2019-10-13T07:27:58.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:27:58.035+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.037905+0000) 2019-10-13T07:27:58.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:27:58.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:27:58.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:27:58.035+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.039372+0000) 2019-10-13T07:27:58.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:27:58.032+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.039175+0000) 2019-10-13T07:27:58.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:27:58.035+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.040808+0000) 2019-10-13T07:27:58.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:27:58.035+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.040932+0000) 2019-10-13T07:27:58.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:27:58.040+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:27:58.046544+0000) 2019-10-13T07:28:00.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:00.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:00.630071+0000) 2019-10-13T07:28:00.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:00.723+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:00.728458+0000) 2019-10-13T07:28:01.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:01.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.525426+0000) 2019-10-13T07:28:01.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:01.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:01.525459+0000) 2019-10-13T07:28:02.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:02.540+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:02.547301+0000) 2019-10-13T07:28:03.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:03.036+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.040390+0000) 2019-10-13T07:28:03.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:03.039+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.041924+0000) 2019-10-13T07:28:03.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:03.039+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.042395+0000) 2019-10-13T07:28:03.044 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:03.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:03.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:03.039+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.042799+0000) 2019-10-13T07:28:03.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:03.039+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.043411+0000) 2019-10-13T07:28:03.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:03.039+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.043863+0000) 2019-10-13T07:28:03.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:03.039+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.044294+0000) 2019-10-13T07:28:03.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:03.044+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:03.051231+0000) 2019-10-13T07:28:04.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:04.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:04.249643+0000) 2019-10-13T07:28:07.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:07.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:07.087696+0000) 2019-10-13T07:28:07.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:07.499+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:07.502081+0000) 2019-10-13T07:28:08.049 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:08.043+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.047613+0000) 2019-10-13T07:28:08.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:08.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.047809+0000) 2019-10-13T07:28:08.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:08.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:08.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:08.043+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.048694+0000) 2019-10-13T07:28:08.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:08.043+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.049566+0000) 2019-10-13T07:28:08.051 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:08.047+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.050128+0000) 2019-10-13T07:28:08.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:08.047+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.050521+0000) 2019-10-13T07:28:08.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:08.047+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.051166+0000) 2019-10-13T07:28:08.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:08.056+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:08.057533+0000) 2019-10-13T07:28:10.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:10.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:10.630276+0000) 2019-10-13T07:28:10.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:10.727+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:10.728616+0000) 2019-10-13T07:28:11.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:11.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.525569+0000) 2019-10-13T07:28:11.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:11.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:11.525671+0000) 2019-10-13T07:28:12.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:12.552+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:12.553669+0000) 2019-10-13T07:28:13.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:13.047+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.052990+0000) 2019-10-13T07:28:13.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:13.047+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.052542+0000) 2019-10-13T07:28:13.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:13.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:13.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:13.047+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.053138+0000) 2019-10-13T07:28:13.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:13.051+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.053735+0000) 2019-10-13T07:28:13.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:13.051+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.053950+0000) 2019-10-13T07:28:13.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:13.051+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.054693+0000) 2019-10-13T07:28:13.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:13.055+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.056852+0000) 2019-10-13T07:28:13.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:13.059+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:13.060186+0000) 2019-10-13T07:28:14.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:14.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:14.249861+0000) 2019-10-13T07:28:17.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:17.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:17.087925+0000) 2019-10-13T07:28:17.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:17.499+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:17.502280+0000) 2019-10-13T07:28:18.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:18.051+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.056150+0000) 2019-10-13T07:28:18.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:18.051+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.056783+0000) 2019-10-13T07:28:18.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:18.051+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.057210+0000) 2019-10-13T07:28:18.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:18.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:18.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:18.055+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.058568+0000) 2019-10-13T07:28:18.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:18.055+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.058639+0000) 2019-10-13T07:28:18.062 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:18.055+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.060239+0000) 2019-10-13T07:28:18.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:18.055+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.058832+0000) 2019-10-13T07:28:18.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:18.063+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:18.065895+0000) 2019-10-13T07:28:20.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:20.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:20.630477+0000) 2019-10-13T07:28:20.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:20.727+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:20.728821+0000) 2019-10-13T07:28:21.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:21.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.525785+0000) 2019-10-13T07:28:21.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:21.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:21.525869+0000) 2019-10-13T07:28:22.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:22.559+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:22.560244+0000) 2019-10-13T07:28:23.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:23.055+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.061529+0000) 2019-10-13T07:28:23.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:23.059+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.062371+0000) 2019-10-13T07:28:23.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:23.059+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.062912+0000) 2019-10-13T07:28:23.065 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:23.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:23.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:23.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.064142+0000) 2019-10-13T07:28:23.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:23.059+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.064259+0000) 2019-10-13T07:28:23.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:23.059+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.064705+0000) 2019-10-13T07:28:23.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:23.063+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.065104+0000) 2019-10-13T07:28:23.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:23.071+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:23.072438+0000) 2019-10-13T07:28:24.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:24.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:24.250067+0000) 2019-10-13T07:28:24.317 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:28:24.318 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:24.321 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:28:24.321 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:27.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:27.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:27.088133+0000) 2019-10-13T07:28:27.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:27.499+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:27.502523+0000) 2019-10-13T07:28:28.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:28.063+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.066878+0000) 2019-10-13T07:28:28.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:28.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:28.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:28.063+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.068609+0000) 2019-10-13T07:28:28.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:28.063+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.069164+0000) 2019-10-13T07:28:28.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:28.067+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.069770+0000) 2019-10-13T07:28:28.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:28.067+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.069553+0000) 2019-10-13T07:28:28.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:28.067+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.070615+0000) 2019-10-13T07:28:28.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:28.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.071224+0000) 2019-10-13T07:28:28.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:28.075+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:28.078786+0000) 2019-10-13T07:28:30.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:30.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:30.630682+0000) 2019-10-13T07:28:30.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:30.727+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:30.729026+0000) 2019-10-13T07:28:31.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:31.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.525963+0000) 2019-10-13T07:28:31.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:31.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:31.526061+0000) 2019-10-13T07:28:32.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:32.563+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:32.566721+0000) 2019-10-13T07:28:33.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:33.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.071522+0000) 2019-10-13T07:28:33.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:33.067+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.072552+0000) 2019-10-13T07:28:33.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:33.067+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.072907+0000) 2019-10-13T07:28:33.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:33.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:33.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:33.071+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.073077+0000) 2019-10-13T07:28:33.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:33.071+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.074261+0000) 2019-10-13T07:28:33.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:33.071+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.075891+0000) 2019-10-13T07:28:33.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:33.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.076494+0000) 2019-10-13T07:28:33.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:33.075+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:33.077835+0000) 2019-10-13T07:28:34.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:34.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:34.250268+0000) 2019-10-13T07:28:37.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:37.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:37.088347+0000) 2019-10-13T07:28:37.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:37.499+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:37.502738+0000) 2019-10-13T07:28:38.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:38.071+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.076221+0000) 2019-10-13T07:28:38.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:38.075+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.076302+0000) 2019-10-13T07:28:38.079 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:38.071+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.077107+0000) 2019-10-13T07:28:38.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:38.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:38.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:38.075+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.078481+0000) 2019-10-13T07:28:38.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:38.075+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.079615+0000) 2019-10-13T07:28:38.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:38.075+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.080238+0000) 2019-10-13T07:28:38.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:38.075+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.080934+0000) 2019-10-13T07:28:38.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:38.083+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:38.084912+0000) 2019-10-13T07:28:40.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:40.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:40.630877+0000) 2019-10-13T07:28:40.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:40.727+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:40.729220+0000) 2019-10-13T07:28:41.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:41.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.526163+0000) 2019-10-13T07:28:41.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:41.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:41.526261+0000) 2019-10-13T07:28:42.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:42.571+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:42.573184+0000) 2019-10-13T07:28:43.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:43.075+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.081248+0000) 2019-10-13T07:28:43.083 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:43.075+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.081435+0000) 2019-10-13T07:28:43.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:43.079+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.082256+0000) 2019-10-13T07:28:43.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:43.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.082423+0000) 2019-10-13T07:28:43.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:43.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:43.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:43.079+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.082857+0000) 2019-10-13T07:28:43.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:43.079+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.084368+0000) 2019-10-13T07:28:43.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:43.079+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.084697+0000) 2019-10-13T07:28:43.093 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:43.087+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:43.090899+0000) 2019-10-13T07:28:44.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:44.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:44.250471+0000) 2019-10-13T07:28:47.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:47.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:47.088564+0000) 2019-10-13T07:28:47.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:47.498+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:47.502967+0000) 2019-10-13T07:28:48.087 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:48.082+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.085934+0000) 2019-10-13T07:28:48.088 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:48.082+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.086409+0000) 2019-10-13T07:28:48.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:48.082+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.087185+0000) 2019-10-13T07:28:48.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:48.082+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.087283+0000) 2019-10-13T07:28:48.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:48.082+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.087702+0000) 2019-10-13T07:28:48.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:48.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:48.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:48.087+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.088663+0000) 2019-10-13T07:28:48.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:48.086+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.089927+0000) 2019-10-13T07:28:48.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:48.095+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:48.096361+0000) 2019-10-13T07:28:50.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:50.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:50.631082+0000) 2019-10-13T07:28:50.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:50.726+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:50.729416+0000) 2019-10-13T07:28:51.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:51.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.526378+0000) 2019-10-13T07:28:51.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:51.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:51.526487+0000) 2019-10-13T07:28:52.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:52.579+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:52.579474+0000) 2019-10-13T07:28:53.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:53.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51634 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:53.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:53.086+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.093079+0000) 2019-10-13T07:28:53.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:53.086+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.093576+0000) 2019-10-13T07:28:53.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:53.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.093945+0000) 2019-10-13T07:28:53.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:53.090+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.094243+0000) 2019-10-13T07:28:53.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:53.090+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.094561+0000) 2019-10-13T07:28:53.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:53.090+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.095122+0000) 2019-10-13T07:28:53.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:53.091+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.095107+0000) 2019-10-13T07:28:53.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:53.099+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:53.101861+0000) 2019-10-13T07:28:54.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:54.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:54.250711+0000) 2019-10-13T07:28:54.439 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:28:54.439 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:54.443 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:28:54.443 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:28:57.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:57.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:57.088782+0000) 2019-10-13T07:28:57.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:57.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:57.503205+0000) 2019-10-13T07:28:58.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:28:58.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.095773+0000) 2019-10-13T07:28:58.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:28:58.090+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.096797+0000) 2019-10-13T07:28:58.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:28:58.095+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.096682+0000) 2019-10-13T07:28:58.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:28:58.094+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.097775+0000) 2019-10-13T07:28:58.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:28:58.094+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.098162+0000) 2019-10-13T07:28:58.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:28:58.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:28:58.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:28:58.094+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.098662+0000) 2019-10-13T07:28:58.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:28:58.094+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.099093+0000) 2019-10-13T07:28:58.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:28:58.103+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:28:58.104973+0000) 2019-10-13T07:29:00.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:00.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:00.631310+0000) 2019-10-13T07:29:00.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:00.726+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:00.729620+0000) 2019-10-13T07:29:01.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:01.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.526589+0000) 2019-10-13T07:29:01.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:01.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:01.526690+0000) 2019-10-13T07:29:02.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:02.583+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:02.586091+0000) 2019-10-13T07:29:03.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:03.098+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.102030+0000) 2019-10-13T07:29:03.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:03.099+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.102264+0000) 2019-10-13T07:29:03.105 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:03.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:03.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:03.098+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.103581+0000) 2019-10-13T07:29:03.106 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:03.098+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.104342+0000) 2019-10-13T07:29:03.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:03.098+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.103951+0000) 2019-10-13T07:29:03.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:03.098+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.105624+0000) 2019-10-13T07:29:03.108 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:03.102+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.106183+0000) 2019-10-13T07:29:03.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:03.107+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:03.110293+0000) 2019-10-13T07:29:04.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:04.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:04.250955+0000) 2019-10-13T07:29:07.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:07.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:07.088984+0000) 2019-10-13T07:29:07.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:07.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:07.503406+0000) 2019-10-13T07:29:08.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:08.102+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.107344+0000) 2019-10-13T07:29:08.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:08.102+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.107912+0000) 2019-10-13T07:29:08.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:08.102+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.108145+0000) 2019-10-13T07:29:08.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:08.102+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.108486+0000) 2019-10-13T07:29:08.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:08.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:08.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:08.102+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.109172+0000) 2019-10-13T07:29:08.111 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:08.102+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.109413+0000) 2019-10-13T07:29:08.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:08.107+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.109346+0000) 2019-10-13T07:29:08.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:08.111+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:08.115977+0000) 2019-10-13T07:29:10.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:10.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:10.631539+0000) 2019-10-13T07:29:10.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:10.726+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:10.729852+0000) 2019-10-13T07:29:11.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:11.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.526802+0000) 2019-10-13T07:29:11.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:11.523+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:11.526886+0000) 2019-10-13T07:29:12.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:12.591+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:12.592722+0000) 2019-10-13T07:29:13.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:13.106+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.111352+0000) 2019-10-13T07:29:13.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:13.106+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.112633+0000) 2019-10-13T07:29:13.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:13.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:13.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:13.106+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.113463+0000) 2019-10-13T07:29:13.115 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:13.110+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.113886+0000) 2019-10-13T07:29:13.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:13.111+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.113669+0000) 2019-10-13T07:29:13.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:13.110+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.116635+0000) 2019-10-13T07:29:13.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:13.110+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.117227+0000) 2019-10-13T07:29:13.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:13.115+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:13.119166+0000) 2019-10-13T07:29:14.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:14.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:14.251190+0000) 2019-10-13T07:29:17.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:17.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:17.089179+0000) 2019-10-13T07:29:17.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:17.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:17.503636+0000) 2019-10-13T07:29:18.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:18.110+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.116888+0000) 2019-10-13T07:29:18.119 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:18.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.116706+0000) 2019-10-13T07:29:18.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:18.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:18.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:18.114+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.117724+0000) 2019-10-13T07:29:18.120 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:18.114+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.118163+0000) 2019-10-13T07:29:18.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:18.114+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.118629+0000) 2019-10-13T07:29:18.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:18.114+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.118929+0000) 2019-10-13T07:29:18.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:18.114+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.120365+0000) 2019-10-13T07:29:18.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:18.123+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:18.125173+0000) 2019-10-13T07:29:20.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:20.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:20.631741+0000) 2019-10-13T07:29:20.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:20.726+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:20.730047+0000) 2019-10-13T07:29:21.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:21.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.526951+0000) 2019-10-13T07:29:21.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:21.522+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:21.527084+0000) 2019-10-13T07:29:22.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:22.594+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:22.599137+0000) 2019-10-13T07:29:23.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:23.114+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.121090+0000) 2019-10-13T07:29:23.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:23.118+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.122438+0000) 2019-10-13T07:29:23.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:23.118+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.122005+0000) 2019-10-13T07:29:23.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:23.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:23.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:23.118+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.123055+0000) 2019-10-13T07:29:23.126 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:23.118+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.123811+0000) 2019-10-13T07:29:23.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:23.118+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.124087+0000) 2019-10-13T07:29:23.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:23.118+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.124546+0000) 2019-10-13T07:29:23.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:23.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:23.129207+0000) 2019-10-13T07:29:24.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:24.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:24.251444+0000) 2019-10-13T07:29:24.513 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:29:24.515 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:24.519 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:29:24.519 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:27.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:27.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:27.089362+0000) 2019-10-13T07:29:27.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:27.498+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:27.503842+0000) 2019-10-13T07:29:28.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:28.122+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.125970+0000) 2019-10-13T07:29:28.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:28.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:28.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:28.122+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.127141+0000) 2019-10-13T07:29:28.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:28.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.127236+0000) 2019-10-13T07:29:28.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:28.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.126885+0000) 2019-10-13T07:29:28.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:28.122+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.128014+0000) 2019-10-13T07:29:28.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:28.122+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.128776+0000) 2019-10-13T07:29:28.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:28.122+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.129071+0000) 2019-10-13T07:29:28.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:28.134+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:28.138035+0000) 2019-10-13T07:29:30.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:30.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:30.631974+0000) 2019-10-13T07:29:30.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:30.730+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:30.730262+0000) 2019-10-13T07:29:31.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:31.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.527138+0000) 2019-10-13T07:29:31.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:31.522+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:31.527311+0000) 2019-10-13T07:29:32.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:32.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:32.605718+0000) 2019-10-13T07:29:33.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:33.126+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.131445+0000) 2019-10-13T07:29:33.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:33.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:33.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:33.126+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.132458+0000) 2019-10-13T07:29:33.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:33.126+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.132629+0000) 2019-10-13T07:29:33.135 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:33.130+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.132379+0000) 2019-10-13T07:29:33.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:33.126+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.133370+0000) 2019-10-13T07:29:33.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:33.130+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.133770+0000) 2019-10-13T07:29:33.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:33.130+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.135205+0000) 2019-10-13T07:29:33.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:33.134+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:33.137091+0000) 2019-10-13T07:29:34.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:34.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:34.251662+0000) 2019-10-13T07:29:37.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:37.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:37.089600+0000) 2019-10-13T07:29:37.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:37.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:37.504047+0000) 2019-10-13T07:29:38.136 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:38.130+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.134851+0000) 2019-10-13T07:29:38.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:38.130+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.135311+0000) 2019-10-13T07:29:38.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:38.130+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.136412+0000) 2019-10-13T07:29:38.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:38.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:38.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:38.130+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.137356+0000) 2019-10-13T07:29:38.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:38.130+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.137491+0000) 2019-10-13T07:29:38.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:38.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.137717+0000) 2019-10-13T07:29:38.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:38.134+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.139057+0000) 2019-10-13T07:29:38.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:38.138+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:38.142571+0000) 2019-10-13T07:29:40.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:40.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:40.632201+0000) 2019-10-13T07:29:40.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:40.730+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:40.730423+0000) 2019-10-13T07:29:41.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:41.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.527326+0000) 2019-10-13T07:29:41.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:41.522+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:41.527512+0000) 2019-10-13T07:29:42.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:42.610+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:42.612186+0000) 2019-10-13T07:29:43.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:43.133+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.141029+0000) 2019-10-13T07:29:43.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:43.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51948 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:43.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:43.137+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.142131+0000) 2019-10-13T07:29:43.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:43.137+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.142509+0000) 2019-10-13T07:29:43.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:43.137+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.142982+0000) 2019-10-13T07:29:43.145 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:43.137+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.143675+0000) 2019-10-13T07:29:43.146 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:43.142+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.144162+0000) 2019-10-13T07:29:43.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:43.137+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.144929+0000) 2019-10-13T07:29:43.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:43.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:43.148753+0000) 2019-10-13T07:29:44.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:44.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:44.251881+0000) 2019-10-13T07:29:47.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:47.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:47.089806+0000) 2019-10-13T07:29:47.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:47.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:47.504253+0000) 2019-10-13T07:29:48.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:48.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.144803+0000) 2019-10-13T07:29:48.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:48.137+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.145305+0000) 2019-10-13T07:29:48.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:48.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:48.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:48.141+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.146384+0000) 2019-10-13T07:29:48.148 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:48.142+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.146328+0000) 2019-10-13T07:29:48.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:48.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.147636+0000) 2019-10-13T07:29:48.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:48.150+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.152355+0000) 2019-10-13T07:29:48.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:48.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.148835+0000) 2019-10-13T07:29:48.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:48.145+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:48.149947+0000) 2019-10-13T07:29:50.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:50.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:50.632401+0000) 2019-10-13T07:29:50.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:50.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:50.730624+0000) 2019-10-13T07:29:51.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:51.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.527519+0000) 2019-10-13T07:29:51.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:51.526+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:51.527682+0000) 2019-10-13T07:29:52.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:52.614+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:52.618805+0000) 2019-10-13T07:29:53.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:53.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.148877+0000) 2019-10-13T07:29:53.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:53.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.149942+0000) 2019-10-13T07:29:53.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:53.145+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.150016+0000) 2019-10-13T07:29:53.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:53.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:53.153 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:53.146+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.150367+0000) 2019-10-13T07:29:53.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:53.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.151704+0000) 2019-10-13T07:29:53.154 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:53.145+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.151470+0000) 2019-10-13T07:29:53.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:53.149+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.154298+0000) 2019-10-13T07:29:53.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:53.154+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:53.159530+0000) 2019-10-13T07:29:54.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:54.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:54.252078+0000) 2019-10-13T07:29:54.643 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:29:54.644 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:54.648 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:29:54.648 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:29:57.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:57.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:57.090008+0000) 2019-10-13T07:29:57.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:57.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:57.504452+0000) 2019-10-13T07:29:58.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:29:58.153+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.154192+0000) 2019-10-13T07:29:58.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:29:58.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52040 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:29:58.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:29:58.153+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.155531+0000) 2019-10-13T07:29:58.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:29:58.150+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.155287+0000) 2019-10-13T07:29:58.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:29:58.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.156238+0000) 2019-10-13T07:29:58.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:29:58.153+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.156878+0000) 2019-10-13T07:29:58.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:29:58.153+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.157152+0000) 2019-10-13T07:29:58.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:29:58.157+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.157790+0000) 2019-10-13T07:29:58.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:29:58.158+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:29:58.160185+0000) 2019-10-13T07:30:00.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:00.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:00.632624+0000) 2019-10-13T07:30:00.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:00.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:00.730855+0000) 2019-10-13T07:30:01.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:01.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.527717+0000) 2019-10-13T07:30:01.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:01.526+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:01.527898+0000) 2019-10-13T07:30:02.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:02.622+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:02.625195+0000) 2019-10-13T07:30:03.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:03.157+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.158878+0000) 2019-10-13T07:30:03.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:03.157+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.159575+0000) 2019-10-13T07:30:03.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:03.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.159616+0000) 2019-10-13T07:30:03.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:03.157+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.159933+0000) 2019-10-13T07:30:03.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:03.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:03.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:03.157+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.160643+0000) 2019-10-13T07:30:03.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:03.157+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.161580+0000) 2019-10-13T07:30:03.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:03.161+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.161961+0000) 2019-10-13T07:30:03.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:03.162+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:03.165376+0000) 2019-10-13T07:30:04.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:04.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:04.252300+0000) 2019-10-13T07:30:07.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:07.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:07.090207+0000) 2019-10-13T07:30:07.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:07.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:07.504660+0000) 2019-10-13T07:30:08.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:08.161+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.163552+0000) 2019-10-13T07:30:08.166 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:08.161+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.165012+0000) 2019-10-13T07:30:08.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:08.161+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.165398+0000) 2019-10-13T07:30:08.167 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:08.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:08.168 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:08.165+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.166809+0000) 2019-10-13T07:30:08.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:08.165+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.168047+0000) 2019-10-13T07:30:08.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:08.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.167933+0000) 2019-10-13T07:30:08.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:08.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.169681+0000) 2019-10-13T07:30:08.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:08.170+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:08.174214+0000) 2019-10-13T07:30:10.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:10.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:10.632857+0000) 2019-10-13T07:30:10.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:10.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:10.731057+0000) 2019-10-13T07:30:11.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:11.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.527907+0000) 2019-10-13T07:30:11.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:11.526+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:11.528120+0000) 2019-10-13T07:30:12.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:12.630+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:12.631594+0000) 2019-10-13T07:30:13.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:13.165+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.168632+0000) 2019-10-13T07:30:13.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:13.165+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.169327+0000) 2019-10-13T07:30:13.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:13.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:13.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:13.169+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.170075+0000) 2019-10-13T07:30:13.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:13.169+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.170833+0000) 2019-10-13T07:30:13.173 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:13.166+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.170931+0000) 2019-10-13T07:30:13.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:13.169+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.172271+0000) 2019-10-13T07:30:13.174 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:13.169+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.172985+0000) 2019-10-13T07:30:13.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:13.174+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:13.179729+0000) 2019-10-13T07:30:14.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:14.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:14.252505+0000) 2019-10-13T07:30:17.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:17.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.090445+0000) 2019-10-13T07:30:17.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:17.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:17.504891+0000) 2019-10-13T07:30:18.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:18.170+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.172660+0000) 2019-10-13T07:30:18.175 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:18.173+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.173809+0000) 2019-10-13T07:30:18.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:18.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:18.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:18.173+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.174927+0000) 2019-10-13T07:30:18.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:18.173+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.175760+0000) 2019-10-13T07:30:18.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:18.173+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.176052+0000) 2019-10-13T07:30:18.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:18.173+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.176349+0000) 2019-10-13T07:30:18.178 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:18.173+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.176616+0000) 2019-10-13T07:30:18.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:18.178+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:18.183601+0000) 2019-10-13T07:30:20.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:20.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:20.633067+0000) 2019-10-13T07:30:20.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:20.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:20.731265+0000) 2019-10-13T07:30:21.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:21.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:21.528103+0000) 2019-10-13T07:30:21.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:21.526+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:21.528331+0000) 2019-10-13T07:30:22.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:22.634+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:22.638024+0000) 2019-10-13T07:30:23.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:23.177+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.178961+0000) 2019-10-13T07:30:23.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:23.174+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.178943+0000) 2019-10-13T07:30:23.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:23.177+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.179789+0000) 2019-10-13T07:30:23.182 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:23.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:23.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:23.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.180509+0000) 2019-10-13T07:30:23.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:23.177+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.180794+0000) 2019-10-13T07:30:23.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:23.177+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.181704+0000) 2019-10-13T07:30:23.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:23.181+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.182436+0000) 2019-10-13T07:30:23.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:23.186+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:23.188581+0000) 2019-10-13T07:30:24.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:24.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:24.252712+0000) 2019-10-13T07:30:24.762 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:30:24.762 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:24.766 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:30:24.766 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:27.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:27.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.090659+0000) 2019-10-13T07:30:27.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:27.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:27.505098+0000) 2019-10-13T07:30:28.185 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:28.181+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.183112+0000) 2019-10-13T07:30:28.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:28.181+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.183772+0000) 2019-10-13T07:30:28.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:28.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:28.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:28.181+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.184895+0000) 2019-10-13T07:30:28.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:28.181+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.184740+0000) 2019-10-13T07:30:28.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:28.185+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.186306+0000) 2019-10-13T07:30:28.188 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:28.185+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.186634+0000) 2019-10-13T07:30:28.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:28.185+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.188101+0000) 2019-10-13T07:30:28.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:28.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:28.194267+0000) 2019-10-13T07:30:30.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:30.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:30.633268+0000) 2019-10-13T07:30:30.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:30.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:30.731492+0000) 2019-10-13T07:30:31.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:31.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:31.528303+0000) 2019-10-13T07:30:31.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:31.525+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:31.528537+0000) 2019-10-13T07:30:32.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:32.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:32.644490+0000) 2019-10-13T07:30:33.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:33.185+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.188077+0000) 2019-10-13T07:30:33.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:33.185+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.188516+0000) 2019-10-13T07:30:33.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:33.185+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.188831+0000) 2019-10-13T07:30:33.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:33.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:33.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:33.185+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.188543+0000) 2019-10-13T07:30:33.192 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:33.189+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.190051+0000) 2019-10-13T07:30:33.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:33.189+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.190620+0000) 2019-10-13T07:30:33.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:33.189+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.192317+0000) 2019-10-13T07:30:33.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:33.193+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:33.197025+0000) 2019-10-13T07:30:34.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:34.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:34.252970+0000) 2019-10-13T07:30:37.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:37.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.090854+0000) 2019-10-13T07:30:37.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:37.501+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:37.505298+0000) 2019-10-13T07:30:38.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:38.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.192298+0000) 2019-10-13T07:30:38.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:38.189+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.193485+0000) 2019-10-13T07:30:38.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:38.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:38.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:38.193+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.194972+0000) 2019-10-13T07:30:38.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:38.189+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.194547+0000) 2019-10-13T07:30:38.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:38.193+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.195660+0000) 2019-10-13T07:30:38.197 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:38.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.196340+0000) 2019-10-13T07:30:38.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:38.197+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.197830+0000) 2019-10-13T07:30:38.200 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:38.193+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:38.197592+0000) 2019-10-13T07:30:40.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:40.628+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:40.633477+0000) 2019-10-13T07:30:40.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:40.728+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:40.731695+0000) 2019-10-13T07:30:41.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:41.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:41.528502+0000) 2019-10-13T07:30:41.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:41.525+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:41.528718+0000) 2019-10-13T07:30:42.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:42.649+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:42.650990+0000) 2019-10-13T07:30:43.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:43.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.197466+0000) 2019-10-13T07:30:43.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:43.196+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.197956+0000) 2019-10-13T07:30:43.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:43.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:43.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:43.196+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.198803+0000) 2019-10-13T07:30:43.200 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:43.193+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.198636+0000) 2019-10-13T07:30:43.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:43.196+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.199601+0000) 2019-10-13T07:30:43.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:43.200+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.201791+0000) 2019-10-13T07:30:43.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:43.200+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.201771+0000) 2019-10-13T07:30:43.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:43.201+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:43.207181+0000) 2019-10-13T07:30:44.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:44.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:44.253133+0000) 2019-10-13T07:30:47.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:47.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.091069+0000) 2019-10-13T07:30:47.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:47.500+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:47.505523+0000) 2019-10-13T07:30:48.203 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:48.196+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.201145+0000) 2019-10-13T07:30:48.204 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:48.200+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.202283+0000) 2019-10-13T07:30:48.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:48.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.201975+0000) 2019-10-13T07:30:48.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:48.200+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.202844+0000) 2019-10-13T07:30:48.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:48.200+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.203303+0000) 2019-10-13T07:30:48.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:48.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.203523+0000) 2019-10-13T07:30:48.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:48.200+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.203505+0000) 2019-10-13T07:30:48.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:48.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:48.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:48.205+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:48.210747+0000) 2019-10-13T07:30:50.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:50.628+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:50.633678+0000) 2019-10-13T07:30:50.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:50.728+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:50.731890+0000) 2019-10-13T07:30:51.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:51.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:51.528720+0000) 2019-10-13T07:30:51.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:51.525+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:51.528866+0000) 2019-10-13T07:30:52.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:52.653+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:52.657595+0000) 2019-10-13T07:30:53.210 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:53.204+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.207546+0000) 2019-10-13T07:30:53.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:53.204+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.207729+0000) 2019-10-13T07:30:53.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:53.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:53.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:53.204+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.208600+0000) 2019-10-13T07:30:53.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:53.204+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.208779+0000) 2019-10-13T07:30:53.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:53.205+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.209066+0000) 2019-10-13T07:30:53.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:53.204+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.209081+0000) 2019-10-13T07:30:53.212 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:53.204+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.209699+0000) 2019-10-13T07:30:53.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:53.213+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:53.216726+0000) 2019-10-13T07:30:54.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:54.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:54.253336+0000) 2019-10-13T07:30:54.882 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:30:54.882 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:54.886 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:30:54.886 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:30:57.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:57.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.091262+0000) 2019-10-13T07:30:57.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:57.504+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:57.505767+0000) 2019-10-13T07:30:58.213 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:30:58.205+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.211454+0000) 2019-10-13T07:30:58.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:30:58.208+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.213093+0000) 2019-10-13T07:30:58.215 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:30:58.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.213546+0000) 2019-10-13T07:30:58.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:30:58.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:30:58.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:30:58.212+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.214314+0000) 2019-10-13T07:30:58.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:30:58.212+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.214290+0000) 2019-10-13T07:30:58.216 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:30:58.212+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.215212+0000) 2019-10-13T07:30:58.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:30:58.212+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.215785+0000) 2019-10-13T07:30:58.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:30:58.213+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:30:58.219175+0000) 2019-10-13T07:31:00.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:00.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:00.633919+0000) 2019-10-13T07:31:00.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:00.728+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:00.732089+0000) 2019-10-13T07:31:01.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:01.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:01.528917+0000) 2019-10-13T07:31:01.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:01.525+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:01.529018+0000) 2019-10-13T07:31:02.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:02.661+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:02.664183+0000) 2019-10-13T07:31:03.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:03.212+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.217006+0000) 2019-10-13T07:31:03.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:03.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:03.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:03.213+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.218421+0000) 2019-10-13T07:31:03.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:03.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.219230+0000) 2019-10-13T07:31:03.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:03.216+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.219735+0000) 2019-10-13T07:31:03.221 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:03.216+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.220120+0000) 2019-10-13T07:31:03.222 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:03.216+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.220512+0000) 2019-10-13T07:31:03.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:03.216+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.221723+0000) 2019-10-13T07:31:03.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:03.221+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:03.227159+0000) 2019-10-13T07:31:04.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:04.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:04.253537+0000) 2019-10-13T07:31:07.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:07.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.091462+0000) 2019-10-13T07:31:07.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:07.504+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:07.505984+0000) 2019-10-13T07:31:08.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:08.220+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.221723+0000) 2019-10-13T07:31:08.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:08.220+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.222190+0000) 2019-10-13T07:31:08.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:08.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:08.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:08.217+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.222926+0000) 2019-10-13T07:31:08.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:08.220+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.223106+0000) 2019-10-13T07:31:08.226 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:08.220+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.223804+0000) 2019-10-13T07:31:08.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:08.220+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.224249+0000) 2019-10-13T07:31:08.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:08.220+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.225041+0000) 2019-10-13T07:31:08.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:08.225+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:08.232073+0000) 2019-10-13T07:31:10.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:10.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:10.634140+0000) 2019-10-13T07:31:10.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:10.728+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:10.732286+0000) 2019-10-13T07:31:11.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:11.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:11.529105+0000) 2019-10-13T07:31:11.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:11.525+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:11.529211+0000) 2019-10-13T07:31:12.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:12.669+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:12.670641+0000) 2019-10-13T07:31:13.228 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:13.224+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.226231+0000) 2019-10-13T07:31:13.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:13.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:13.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:13.224+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.227613+0000) 2019-10-13T07:31:13.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:13.224+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.227913+0000) 2019-10-13T07:31:13.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:13.224+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.228253+0000) 2019-10-13T07:31:13.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:13.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.228342+0000) 2019-10-13T07:31:13.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:13.228+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.230032+0000) 2019-10-13T07:31:13.233 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:13.224+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.229221+0000) 2019-10-13T07:31:13.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:13.229+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:13.233430+0000) 2019-10-13T07:31:14.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:14.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:14.253735+0000) 2019-10-13T07:31:17.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:17.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.091661+0000) 2019-10-13T07:31:17.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:17.504+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:17.506212+0000) 2019-10-13T07:31:18.233 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:18.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.230646+0000) 2019-10-13T07:31:18.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:18.228+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.230856+0000) 2019-10-13T07:31:18.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:18.228+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.231850+0000) 2019-10-13T07:31:18.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:18.228+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.232188+0000) 2019-10-13T07:31:18.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:18.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:18.234 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:18.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.232567+0000) 2019-10-13T07:31:18.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:18.228+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.233370+0000) 2019-10-13T07:31:18.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:18.232+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.234240+0000) 2019-10-13T07:31:18.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:18.237+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:18.240436+0000) 2019-10-13T07:31:20.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:20.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:20.634365+0000) 2019-10-13T07:31:20.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:20.732+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:20.732485+0000) 2019-10-13T07:31:21.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:21.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:21.529326+0000) 2019-10-13T07:31:21.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:21.529+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:21.529411+0000) 2019-10-13T07:31:22.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:22.673+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:22.677187+0000) 2019-10-13T07:31:23.238 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:23.232+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.235963+0000) 2019-10-13T07:31:23.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:23.232+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.236359+0000) 2019-10-13T07:31:23.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:23.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52577 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:23.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:23.232+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.237161+0000) 2019-10-13T07:31:23.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:23.236+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.237776+0000) 2019-10-13T07:31:23.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:23.233+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.237919+0000) 2019-10-13T07:31:23.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:23.236+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.239311+0000) 2019-10-13T07:31:23.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:23.236+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.238947+0000) 2019-10-13T07:31:23.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:23.241+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:23.244826+0000) 2019-10-13T07:31:24.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:24.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:24.253971+0000) 2019-10-13T07:31:24.955 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:31:24.956 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:24.960 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:31:24.960 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:27.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:27.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.091856+0000) 2019-10-13T07:31:27.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:27.504+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:27.506413+0000) 2019-10-13T07:31:28.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:28.236+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.239965+0000) 2019-10-13T07:31:28.244 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:28.236+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.241012+0000) 2019-10-13T07:31:28.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:28.240+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.242159+0000) 2019-10-13T07:31:28.245 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:28.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.240737+0000) 2019-10-13T07:31:28.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:28.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:28.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:28.240+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.242347+0000) 2019-10-13T07:31:28.245 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:28.240+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.243053+0000) 2019-10-13T07:31:28.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:28.240+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.243722+0000) 2019-10-13T07:31:28.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:28.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:28.247432+0000) 2019-10-13T07:31:30.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:30.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:30.634567+0000) 2019-10-13T07:31:30.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:30.728+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:30.732713+0000) 2019-10-13T07:31:31.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:31.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:31.529538+0000) 2019-10-13T07:31:31.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:31.529+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:31.529565+0000) 2019-10-13T07:31:32.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:32.680+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:32.683626+0000) 2019-10-13T07:31:33.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:33.240+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.244577+0000) 2019-10-13T07:31:33.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:33.244+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.245838+0000) 2019-10-13T07:31:33.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:33.244+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.246713+0000) 2019-10-13T07:31:33.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:33.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:33.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:33.244+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.247028+0000) 2019-10-13T07:31:33.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:33.244+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.247878+0000) 2019-10-13T07:31:33.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:33.244+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.247550+0000) 2019-10-13T07:31:33.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:33.244+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.248827+0000) 2019-10-13T07:31:33.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:33.248+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:33.253877+0000) 2019-10-13T07:31:34.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:34.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:34.254190+0000) 2019-10-13T07:31:37.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:37.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.092081+0000) 2019-10-13T07:31:37.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:37.503+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:37.506661+0000) 2019-10-13T07:31:38.251 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:38.247+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.249719+0000) 2019-10-13T07:31:38.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:38.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.251160+0000) 2019-10-13T07:31:38.253 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:38.244+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.250957+0000) 2019-10-13T07:31:38.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:38.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:38.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:38.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.252145+0000) 2019-10-13T07:31:38.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:38.247+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.252685+0000) 2019-10-13T07:31:38.254 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:38.247+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.252812+0000) 2019-10-13T07:31:38.255 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:38.251+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.254239+0000) 2019-10-13T07:31:38.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:38.252+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:38.256735+0000) 2019-10-13T07:31:40.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:40.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:40.634775+0000) 2019-10-13T07:31:41.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:40.731+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:40.732911+0000) 2019-10-13T07:31:41.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:41.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:41.529734+0000) 2019-10-13T07:31:41.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:41.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:41.529767+0000) 2019-10-13T07:31:42.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:42.688+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:42.690007+0000) 2019-10-13T07:31:43.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:43.248+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.255369+0000) 2019-10-13T07:31:43.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:43.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:43.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:43.251+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.257071+0000) 2019-10-13T07:31:43.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:43.251+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.257327+0000) 2019-10-13T07:31:43.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:43.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.258365+0000) 2019-10-13T07:31:43.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:43.255+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.257779+0000) 2019-10-13T07:31:43.260 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:43.255+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.258865+0000) 2019-10-13T07:31:43.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:43.255+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.259491+0000) 2019-10-13T07:31:43.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:43.256+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:43.261047+0000) 2019-10-13T07:31:44.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:44.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:44.254400+0000) 2019-10-13T07:31:47.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:47.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.092289+0000) 2019-10-13T07:31:47.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:47.503+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:47.506879+0000) 2019-10-13T07:31:48.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:48.255+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.260007+0000) 2019-10-13T07:31:48.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:48.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.260367+0000) 2019-10-13T07:31:48.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:48.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:48.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:48.255+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.261304+0000) 2019-10-13T07:31:48.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:48.256+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.262256+0000) 2019-10-13T07:31:48.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:48.259+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.263690+0000) 2019-10-13T07:31:48.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:48.259+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.263783+0000) 2019-10-13T07:31:48.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:48.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.264186+0000) 2019-10-13T07:31:48.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:48.264+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:48.269802+0000) 2019-10-13T07:31:50.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:50.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:50.634973+0000) 2019-10-13T07:31:50.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:50.731+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:50.733156+0000) 2019-10-13T07:31:51.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:51.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:51.529994+0000) 2019-10-13T07:31:51.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:51.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:51.529940+0000) 2019-10-13T07:31:52.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:52.692+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:52.696467+0000) 2019-10-13T07:31:53.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:53.259+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.265018+0000) 2019-10-13T07:31:53.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:53.263+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.266201+0000) 2019-10-13T07:31:53.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:53.263+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.266421+0000) 2019-10-13T07:31:53.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:53.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:53.268 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:53.263+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.266794+0000) 2019-10-13T07:31:53.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:53.263+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.267352+0000) 2019-10-13T07:31:53.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:53.260+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.267290+0000) 2019-10-13T07:31:53.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:53.263+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.268152+0000) 2019-10-13T07:31:53.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:53.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:53.272628+0000) 2019-10-13T07:31:54.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:54.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:54.254583+0000) 2019-10-13T07:31:55.079 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:31:55.079 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:55.083 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:31:55.083 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:31:57.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:57.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.092526+0000) 2019-10-13T07:31:57.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:57.503+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:57.507090+0000) 2019-10-13T07:31:58.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:31:58.267+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.270332+0000) 2019-10-13T07:31:58.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:31:58.267+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.270686+0000) 2019-10-13T07:31:58.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:31:58.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.270861+0000) 2019-10-13T07:31:58.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:31:58.267+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.270998+0000) 2019-10-13T07:31:58.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:31:58.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:31:58.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:31:58.267+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.272144+0000) 2019-10-13T07:31:58.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:31:58.267+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.272348+0000) 2019-10-13T07:31:58.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:31:58.271+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.273978+0000) 2019-10-13T07:31:58.280 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:31:58.272+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:31:58.278697+0000) 2019-10-13T07:32:00.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:00.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:00.635191+0000) 2019-10-13T07:32:00.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:00.727+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:00.733383+0000) 2019-10-13T07:32:01.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:01.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:01.530194+0000) 2019-10-13T07:32:01.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:01.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:01.530088+0000) 2019-10-13T07:32:02.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:02.696+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:02.703085+0000) 2019-10-13T07:32:03.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:03.271+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.274346+0000) 2019-10-13T07:32:03.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:03.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.274790+0000) 2019-10-13T07:32:03.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:03.271+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.275943+0000) 2019-10-13T07:32:03.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:03.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:03.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:03.271+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.276134+0000) 2019-10-13T07:32:03.278 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:03.271+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.276702+0000) 2019-10-13T07:32:03.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:03.271+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.277071+0000) 2019-10-13T07:32:03.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:03.275+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.277855+0000) 2019-10-13T07:32:03.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:03.280+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:03.284426+0000) 2019-10-13T07:32:04.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:04.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:04.254764+0000) 2019-10-13T07:32:07.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:07.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.092746+0000) 2019-10-13T07:32:07.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:07.503+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:07.507288+0000) 2019-10-13T07:32:08.281 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:08.272+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.279280+0000) 2019-10-13T07:32:08.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:08.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:08.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:08.275+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.281360+0000) 2019-10-13T07:32:08.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:08.279+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.281810+0000) 2019-10-13T07:32:08.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:08.279+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.282476+0000) 2019-10-13T07:32:08.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:08.279+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.282862+0000) 2019-10-13T07:32:08.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:08.279+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.283760+0000) 2019-10-13T07:32:08.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:08.279+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.284375+0000) 2019-10-13T07:32:08.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:08.284+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:08.289300+0000) 2019-10-13T07:32:10.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:10.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:10.635392+0000) 2019-10-13T07:32:10.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:10.731+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:10.733617+0000) 2019-10-13T07:32:11.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:11.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:11.530360+0000) 2019-10-13T07:32:11.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:11.524+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:11.530287+0000) 2019-10-13T07:32:12.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:12.708+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:12.709078+0000) 2019-10-13T07:32:13.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:13.279+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.284324+0000) 2019-10-13T07:32:13.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:13.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:13.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:13.283+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.286639+0000) 2019-10-13T07:32:13.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:13.283+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.287486+0000) 2019-10-13T07:32:13.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:13.280+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.287291+0000) 2019-10-13T07:32:13.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:13.283+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.287588+0000) 2019-10-13T07:32:13.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:13.283+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.288106+0000) 2019-10-13T07:32:13.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:13.283+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.289715+0000) 2019-10-13T07:32:13.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:13.288+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:13.294431+0000) 2019-10-13T07:32:14.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:14.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:14.254968+0000) 2019-10-13T07:32:17.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:17.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.092983+0000) 2019-10-13T07:32:17.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:17.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:17.507491+0000) 2019-10-13T07:32:18.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:18.283+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.289454+0000) 2019-10-13T07:32:18.291 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:18.287+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.289935+0000) 2019-10-13T07:32:18.292 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:18.287+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.290392+0000) 2019-10-13T07:32:18.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:18.287+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.291102+0000) 2019-10-13T07:32:18.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:18.287+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.291214+0000) 2019-10-13T07:32:18.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:18.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:18.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:18.284+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.291021+0000) 2019-10-13T07:32:18.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:18.287+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.292301+0000) 2019-10-13T07:32:18.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:18.292+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:18.298380+0000) 2019-10-13T07:32:20.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:20.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:20.635604+0000) 2019-10-13T07:32:20.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:20.731+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:20.733823+0000) 2019-10-13T07:32:21.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:21.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:21.530567+0000) 2019-10-13T07:32:21.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:21.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:21.530436+0000) 2019-10-13T07:32:22.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:22.712+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:22.715558+0000) 2019-10-13T07:32:23.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:23.291+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.294735+0000) 2019-10-13T07:32:23.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:23.288+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.294601+0000) 2019-10-13T07:32:23.297 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:23.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:23.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:23.291+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.295395+0000) 2019-10-13T07:32:23.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:23.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.295887+0000) 2019-10-13T07:32:23.298 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:23.291+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.296847+0000) 2019-10-13T07:32:23.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:23.291+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.297419+0000) 2019-10-13T07:32:23.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:23.295+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.298105+0000) 2019-10-13T07:32:23.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:23.300+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:23.304528+0000) 2019-10-13T07:32:24.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:24.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:24.255148+0000) 2019-10-13T07:32:25.202 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:32:25.203 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:25.206 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:32:25.207 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:27.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:27.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.093202+0000) 2019-10-13T07:32:27.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:27.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:27.507692+0000) 2019-10-13T07:32:28.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:28.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.298759+0000) 2019-10-13T07:32:28.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:28.295+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.299770+0000) 2019-10-13T07:32:28.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:28.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:28.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:28.295+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.300564+0000) 2019-10-13T07:32:28.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:28.295+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.301029+0000) 2019-10-13T07:32:28.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:28.295+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.301688+0000) 2019-10-13T07:32:28.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:28.299+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.302460+0000) 2019-10-13T07:32:28.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:28.299+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.303624+0000) 2019-10-13T07:32:28.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:28.300+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:28.305652+0000) 2019-10-13T07:32:30.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:30.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:30.635813+0000) 2019-10-13T07:32:30.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:30.731+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:30.734030+0000) 2019-10-13T07:32:31.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:31.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:31.530763+0000) 2019-10-13T07:32:31.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:31.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:31.530630+0000) 2019-10-13T07:32:32.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:32.720+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:32.722077+0000) 2019-10-13T07:32:33.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:33.299+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.303943+0000) 2019-10-13T07:32:33.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:33.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.304367+0000) 2019-10-13T07:32:33.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:33.299+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.304955+0000) 2019-10-13T07:32:33.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:33.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:33.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:33.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.305861+0000) 2019-10-13T07:32:33.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:33.304+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.305852+0000) 2019-10-13T07:32:33.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:33.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.307336+0000) 2019-10-13T07:32:33.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:33.303+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.306833+0000) 2019-10-13T07:32:33.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:33.312+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:33.313574+0000) 2019-10-13T07:32:34.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:34.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:34.255346+0000) 2019-10-13T07:32:37.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:37.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.093403+0000) 2019-10-13T07:32:37.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:37.506+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:37.507893+0000) 2019-10-13T07:32:38.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:38.302+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.308136+0000) 2019-10-13T07:32:38.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:38.302+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.308955+0000) 2019-10-13T07:32:38.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:38.306+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:38.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:38.306+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.310441+0000) 2019-10-13T07:32:38.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:38.306+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.311231+0000) 2019-10-13T07:32:38.314 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:38.308+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.311946+0000) 2019-10-13T07:32:38.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:38.306+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.312845+0000) 2019-10-13T07:32:38.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:38.310+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.313978+0000) 2019-10-13T07:32:38.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:38.316+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:38.316552+0000) 2019-10-13T07:32:40.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:40.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:40.635973+0000) 2019-10-13T07:32:40.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:40.730+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:40.734246+0000) 2019-10-13T07:32:41.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:41.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:41.530949+0000) 2019-10-13T07:32:41.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:41.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:41.530840+0000) 2019-10-13T07:32:42.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:42.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:42.732763+0000) 2019-10-13T07:32:43.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:43.306+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.313650+0000) 2019-10-13T07:32:43.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:43.310+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.314155+0000) 2019-10-13T07:32:43.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:43.310+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.314292+0000) 2019-10-13T07:32:43.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:43.310+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:43.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:43.310+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.314959+0000) 2019-10-13T07:32:43.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:43.310+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.316014+0000) 2019-10-13T07:32:43.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:43.315+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.316811+0000) 2019-10-13T07:32:43.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:43.310+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.316904+0000) 2019-10-13T07:32:43.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:43.319+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:43.321651+0000) 2019-10-13T07:32:44.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:44.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:44.255527+0000) 2019-10-13T07:32:47.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:47.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.093617+0000) 2019-10-13T07:32:47.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:47.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:47.508124+0000) 2019-10-13T07:32:48.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:48.314+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.319503+0000) 2019-10-13T07:32:48.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:48.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:48.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:48.314+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.319869+0000) 2019-10-13T07:32:48.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:48.314+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.320429+0000) 2019-10-13T07:32:48.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:48.314+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.321010+0000) 2019-10-13T07:32:48.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:48.314+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.321349+0000) 2019-10-13T07:32:48.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:48.318+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.322314+0000) 2019-10-13T07:32:48.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:48.319+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.322022+0000) 2019-10-13T07:32:48.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:48.323+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:48.327270+0000) 2019-10-13T07:32:50.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:50.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:50.636193+0000) 2019-10-13T07:32:50.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:50.730+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:50.734419+0000) 2019-10-13T07:32:51.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:51.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:51.531014+0000) 2019-10-13T07:32:51.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:51.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:51.531143+0000) 2019-10-13T07:32:52.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:52.739+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:52.742075+0000) 2019-10-13T07:32:53.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:53.318+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.323073+0000) 2019-10-13T07:32:53.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:53.318+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.323841+0000) 2019-10-13T07:32:53.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:53.318+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.324402+0000) 2019-10-13T07:32:53.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:53.323+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.324388+0000) 2019-10-13T07:32:53.327 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:53.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:53.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:53.318+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.325381+0000) 2019-10-13T07:32:53.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:53.318+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.325616+0000) 2019-10-13T07:32:53.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:53.322+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.326524+0000) 2019-10-13T07:32:53.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:53.327+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:53.328827+0000) 2019-10-13T07:32:54.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:54.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:54.255708+0000) 2019-10-13T07:32:55.468 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:32:55.468 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:55.472 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:32:55.472 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:32:57.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:57.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.093820+0000) 2019-10-13T07:32:57.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:57.502+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:57.508332+0000) 2019-10-13T07:32:58.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:32:58.322+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.328759+0000) 2019-10-13T07:32:58.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:32:58.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53173 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:32:58.332 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:32:58.327+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.329687+0000) 2019-10-13T07:32:58.332 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:32:58.326+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.330564+0000) 2019-10-13T07:32:58.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:32:58.326+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.330834+0000) 2019-10-13T07:32:58.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:32:58.326+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.331293+0000) 2019-10-13T07:32:58.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:32:58.326+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.331943+0000) 2019-10-13T07:32:58.334 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:32:58.326+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.332517+0000) 2019-10-13T07:32:58.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:32:58.335+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:32:58.339655+0000) 2019-10-13T07:33:00.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:00.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:00.636395+0000) 2019-10-13T07:33:00.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:00.734+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:00.734624+0000) 2019-10-13T07:33:01.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:01.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:01.531368+0000) 2019-10-13T07:33:01.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:01.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:01.531208+0000) 2019-10-13T07:33:02.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:02.747+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:02.748572+0000) 2019-10-13T07:33:03.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:03.326+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.332654+0000) 2019-10-13T07:33:03.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:03.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:03.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:03.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.335498+0000) 2019-10-13T07:33:03.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:03.330+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.335892+0000) 2019-10-13T07:33:03.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:03.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.336222+0000) 2019-10-13T07:33:03.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:03.334+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.337781+0000) 2019-10-13T07:33:03.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:03.330+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.337423+0000) 2019-10-13T07:33:03.341 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:03.334+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.338817+0000) 2019-10-13T07:33:03.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:03.343+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:03.345867+0000) 2019-10-13T07:33:04.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:04.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:04.255925+0000) 2019-10-13T07:33:07.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:07.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.094048+0000) 2019-10-13T07:33:07.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:07.506+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:07.508530+0000) 2019-10-13T07:33:08.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:08.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:08.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:08.334+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.341042+0000) 2019-10-13T07:33:08.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:08.334+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.341289+0000) 2019-10-13T07:33:08.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:08.338+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.341864+0000) 2019-10-13T07:33:08.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:08.338+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.342153+0000) 2019-10-13T07:33:08.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:08.339+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.341862+0000) 2019-10-13T07:33:08.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:08.338+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.343182+0000) 2019-10-13T07:33:08.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:08.338+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.343514+0000) 2019-10-13T07:33:08.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:08.343+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:08.346616+0000) 2019-10-13T07:33:10.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:10.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:10.636601+0000) 2019-10-13T07:33:10.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:10.734+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:10.734822+0000) 2019-10-13T07:33:11.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:11.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:11.531559+0000) 2019-10-13T07:33:11.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:11.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:11.531423+0000) 2019-10-13T07:33:12.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:12.759+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:12.760136+0000) 2019-10-13T07:33:13.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:13.338+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.343554+0000) 2019-10-13T07:33:13.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:13.338+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.344327+0000) 2019-10-13T07:33:13.346 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:13.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:13.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:13.343+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.345282+0000) 2019-10-13T07:33:13.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:13.342+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.346766+0000) 2019-10-13T07:33:13.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:13.342+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.347165+0000) 2019-10-13T07:33:13.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:13.342+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.348006+0000) 2019-10-13T07:33:13.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:13.342+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.347949+0000) 2019-10-13T07:33:13.356 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:13.351+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:13.353954+0000) 2019-10-13T07:33:14.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:14.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:14.256107+0000) 2019-10-13T07:33:17.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:17.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.094251+0000) 2019-10-13T07:33:17.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:17.506+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:17.508728+0000) 2019-10-13T07:33:18.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:18.342+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.349324+0000) 2019-10-13T07:33:18.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:18.346+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.349990+0000) 2019-10-13T07:33:18.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:18.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:18.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:18.346+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.350591+0000) 2019-10-13T07:33:18.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:18.347+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.351230+0000) 2019-10-13T07:33:18.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:18.346+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.352383+0000) 2019-10-13T07:33:18.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:18.346+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.352473+0000) 2019-10-13T07:33:18.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:18.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.353019+0000) 2019-10-13T07:33:18.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:18.355+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:18.356298+0000) 2019-10-13T07:33:20.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:20.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:20.636801+0000) 2019-10-13T07:33:20.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:20.730+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:20.735022+0000) 2019-10-13T07:33:21.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:21.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:21.531749+0000) 2019-10-13T07:33:21.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:21.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:21.531579+0000) 2019-10-13T07:33:22.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:22.763+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:22.766640+0000) 2019-10-13T07:33:23.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:23.350+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.353267+0000) 2019-10-13T07:33:23.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:23.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.353203+0000) 2019-10-13T07:33:23.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:23.350+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.354322+0000) 2019-10-13T07:33:23.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:23.350+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.354508+0000) 2019-10-13T07:33:23.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:23.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.355725+0000) 2019-10-13T07:33:23.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:23.350+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.356294+0000) 2019-10-13T07:33:23.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:23.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:23.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:23.350+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.356599+0000) 2019-10-13T07:33:23.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:23.359+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:23.361523+0000) 2019-10-13T07:33:24.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:24.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:24.256289+0000) 2019-10-13T07:33:25.525 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:33:25.525 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:25.529 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:33:25.529 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:27.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:27.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.094472+0000) 2019-10-13T07:33:27.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:27.506+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:27.508930+0000) 2019-10-13T07:33:28.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:28.354+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.360319+0000) 2019-10-13T07:33:28.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:28.354+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.361259+0000) 2019-10-13T07:33:28.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:28.359+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.361029+0000) 2019-10-13T07:33:28.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:28.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:28.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:28.358+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.363055+0000) 2019-10-13T07:33:28.365 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:28.358+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.363342+0000) 2019-10-13T07:33:28.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:28.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.363687+0000) 2019-10-13T07:33:28.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:28.358+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.364888+0000) 2019-10-13T07:33:28.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:28.371+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:28.373367+0000) 2019-10-13T07:33:30.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:30.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:30.637007+0000) 2019-10-13T07:33:30.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:30.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:30.735221+0000) 2019-10-13T07:33:31.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:31.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:31.531952+0000) 2019-10-13T07:33:31.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:31.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:31.531793+0000) 2019-10-13T07:33:32.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:32.771+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:32.773077+0000) 2019-10-13T07:33:33.366 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:33.361+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.364810+0000) 2019-10-13T07:33:33.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:33.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.368389+0000) 2019-10-13T07:33:33.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:33.375+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.376705+0000) 2019-10-13T07:33:33.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:33.361+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.366979+0000) 2019-10-13T07:33:33.896 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:33.361+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.367498+0000) 2019-10-13T07:33:33.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:33.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:33.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:33.361+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.368178+0000) 2019-10-13T07:33:33.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:33.361+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.368348+0000) 2019-10-13T07:33:33.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:33.361+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:33.369676+0000) 2019-10-13T07:33:34.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:34.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:34.256465+0000) 2019-10-13T07:33:37.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:37.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.094716+0000) 2019-10-13T07:33:37.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:37.505+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:37.509131+0000) 2019-10-13T07:33:38.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:38.367+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.371413+0000) 2019-10-13T07:33:38.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:38.365+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.372281+0000) 2019-10-13T07:33:38.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:38.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:38.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:38.365+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.373198+0000) 2019-10-13T07:33:38.375 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:38.369+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.373777+0000) 2019-10-13T07:33:38.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:38.369+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.374349+0000) 2019-10-13T07:33:38.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:38.369+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.374794+0000) 2019-10-13T07:33:38.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:38.369+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.374998+0000) 2019-10-13T07:33:38.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:38.379+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:38.381183+0000) 2019-10-13T07:33:40.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:40.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:40.637146+0000) 2019-10-13T07:33:40.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:40.729+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:40.735461+0000) 2019-10-13T07:33:41.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:41.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:41.532155+0000) 2019-10-13T07:33:41.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:41.527+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:41.531998+0000) 2019-10-13T07:33:42.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:42.775+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:42.779510+0000) 2019-10-13T07:33:43.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:43.369+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.376885+0000) 2019-10-13T07:33:43.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:43.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:43.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:43.373+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.378499+0000) 2019-10-13T07:33:43.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:43.375+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.378194+0000) 2019-10-13T07:33:43.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:43.373+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.379067+0000) 2019-10-13T07:33:43.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:43.373+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.379415+0000) 2019-10-13T07:33:43.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:43.373+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.380082+0000) 2019-10-13T07:33:43.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:43.373+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.380763+0000) 2019-10-13T07:33:43.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:43.383+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:43.386186+0000) 2019-10-13T07:33:44.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:44.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:44.256637+0000) 2019-10-13T07:33:47.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:47.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.094912+0000) 2019-10-13T07:33:47.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:47.505+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:47.509332+0000) 2019-10-13T07:33:48.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:48.377+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.381313+0000) 2019-10-13T07:33:48.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:48.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.381212+0000) 2019-10-13T07:33:48.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:48.381+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.382233+0000) 2019-10-13T07:33:48.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:48.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:48.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:48.381+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.382439+0000) 2019-10-13T07:33:48.385 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:48.381+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.383388+0000) 2019-10-13T07:33:48.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:48.381+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.384221+0000) 2019-10-13T07:33:48.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:48.381+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.384660+0000) 2019-10-13T07:33:48.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:48.386+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:48.391142+0000) 2019-10-13T07:33:50.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:50.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:50.637351+0000) 2019-10-13T07:33:50.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:50.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:50.735651+0000) 2019-10-13T07:33:51.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:51.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:51.532190+0000) 2019-10-13T07:33:51.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:51.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:51.532389+0000) 2019-10-13T07:33:52.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:52.782+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:52.785944+0000) 2019-10-13T07:33:53.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:53.385+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.385776+0000) 2019-10-13T07:33:53.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:53.385+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.387313+0000) 2019-10-13T07:33:53.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:53.385+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.387425+0000) 2019-10-13T07:33:53.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:53.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53521 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:53.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:53.385+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.388661+0000) 2019-10-13T07:33:53.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:53.385+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.388162+0000) 2019-10-13T07:33:53.391 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:53.386+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.388470+0000) 2019-10-13T07:33:53.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:53.385+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.389588+0000) 2019-10-13T07:33:53.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:53.394+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:53.396982+0000) 2019-10-13T07:33:54.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:54.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:54.256852+0000) 2019-10-13T07:33:55.734 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:33:55.734 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:55.738 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:33:55.738 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:33:57.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:57.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.095162+0000) 2019-10-13T07:33:57.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:57.505+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:57.509531+0000) 2019-10-13T07:33:58.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:33:58.389+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.390172+0000) 2019-10-13T07:33:58.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:33:58.390+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.392299+0000) 2019-10-13T07:33:58.394 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:33:58.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:33:58.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:33:58.393+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.393897+0000) 2019-10-13T07:33:58.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:33:58.393+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.394950+0000) 2019-10-13T07:33:58.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:33:58.393+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.395084+0000) 2019-10-13T07:33:58.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:33:58.393+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.395438+0000) 2019-10-13T07:33:58.397 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:33:58.393+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.395851+0000) 2019-10-13T07:33:58.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:33:58.394+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:33:58.399968+0000) 2019-10-13T07:34:00.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:00.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:00.637556+0000) 2019-10-13T07:34:00.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:00.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:00.735906+0000) 2019-10-13T07:34:01.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:01.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:01.532574+0000) 2019-10-13T07:34:01.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:01.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:01.532389+0000) 2019-10-13T07:34:02.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:02.790+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:02.792561+0000) 2019-10-13T07:34:03.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:03.393+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.396493+0000) 2019-10-13T07:34:03.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:03.393+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.397660+0000) 2019-10-13T07:34:03.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:03.394+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.397515+0000) 2019-10-13T07:34:03.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:03.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:03.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:03.397+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.398208+0000) 2019-10-13T07:34:03.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:03.397+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.398459+0000) 2019-10-13T07:34:03.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:03.397+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.399557+0000) 2019-10-13T07:34:03.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:03.397+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.400946+0000) 2019-10-13T07:34:03.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:03.402+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:03.404940+0000) 2019-10-13T07:34:04.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:04.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:04.257048+0000) 2019-10-13T07:34:07.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:07.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.095391+0000) 2019-10-13T07:34:07.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:07.509+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:07.509734+0000) 2019-10-13T07:34:08.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:08.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:08.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:08.401+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.403323+0000) 2019-10-13T07:34:08.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:08.401+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.403831+0000) 2019-10-13T07:34:08.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:08.401+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.404516+0000) 2019-10-13T07:34:08.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:08.402+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.404491+0000) 2019-10-13T07:34:08.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:08.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.405637+0000) 2019-10-13T07:34:08.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:08.405+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.406331+0000) 2019-10-13T07:34:08.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:08.405+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.406915+0000) 2019-10-13T07:34:08.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:08.406+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:08.409711+0000) 2019-10-13T07:34:10.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:10.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:10.637785+0000) 2019-10-13T07:34:10.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:10.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:10.736139+0000) 2019-10-13T07:34:11.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:11.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:11.532767+0000) 2019-10-13T07:34:11.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:11.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:11.532601+0000) 2019-10-13T07:34:12.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:12.798+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:12.798939+0000) 2019-10-13T07:34:13.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:13.405+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.406739+0000) 2019-10-13T07:34:13.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:13.405+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.407470+0000) 2019-10-13T07:34:13.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:13.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:13.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:13.405+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.408437+0000) 2019-10-13T07:34:13.410 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:13.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.408328+0000) 2019-10-13T07:34:13.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:13.405+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.409256+0000) 2019-10-13T07:34:13.411 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:13.405+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.409571+0000) 2019-10-13T07:34:13.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:13.409+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.409893+0000) 2019-10-13T07:34:13.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:13.410+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:13.414028+0000) 2019-10-13T07:34:14.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:14.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:14.257218+0000) 2019-10-13T07:34:17.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:17.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.095589+0000) 2019-10-13T07:34:17.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:17.509+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:17.509972+0000) 2019-10-13T07:34:18.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:18.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.411976+0000) 2019-10-13T07:34:18.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:18.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:18.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:18.413+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.414131+0000) 2019-10-13T07:34:18.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:18.413+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.414559+0000) 2019-10-13T07:34:18.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:18.413+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.414917+0000) 2019-10-13T07:34:18.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:18.413+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.415445+0000) 2019-10-13T07:34:18.417 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:18.413+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.416053+0000) 2019-10-13T07:34:18.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:18.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.416840+0000) 2019-10-13T07:34:18.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:18.414+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:18.419829+0000) 2019-10-13T07:34:20.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:20.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:20.637994+0000) 2019-10-13T07:34:20.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:20.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:20.736338+0000) 2019-10-13T07:34:21.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:21.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:21.532981+0000) 2019-10-13T07:34:21.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:21.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:21.532793+0000) 2019-10-13T07:34:22.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:22.806+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:22.810370+0000) 2019-10-13T07:34:23.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:23.413+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.416239+0000) 2019-10-13T07:34:23.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:23.413+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.417222+0000) 2019-10-13T07:34:23.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:23.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.417469+0000) 2019-10-13T07:34:23.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:23.414+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.417449+0000) 2019-10-13T07:34:23.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:23.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:23.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:23.417+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.418875+0000) 2019-10-13T07:34:23.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:23.417+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.419690+0000) 2019-10-13T07:34:23.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:23.417+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.421192+0000) 2019-10-13T07:34:23.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:23.422+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:23.425765+0000) 2019-10-13T07:34:24.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:24.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:24.257389+0000) 2019-10-13T07:34:25.788 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:34:25.788 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:25.792 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:34:25.792 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:27.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:27.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.095793+0000) 2019-10-13T07:34:27.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:27.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:27.510175+0000) 2019-10-13T07:34:28.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:28.416+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.421590+0000) 2019-10-13T07:34:28.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:28.420+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.422239+0000) 2019-10-13T07:34:28.424 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:28.420+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.421882+0000) 2019-10-13T07:34:28.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:28.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:28.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:28.420+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.422573+0000) 2019-10-13T07:34:28.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:28.420+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.423044+0000) 2019-10-13T07:34:28.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:28.418+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.423787+0000) 2019-10-13T07:34:28.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:28.420+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.423624+0000) 2019-10-13T07:34:28.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:28.430+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:28.433581+0000) 2019-10-13T07:34:30.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:30.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:30.638197+0000) 2019-10-13T07:34:30.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:30.732+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:30.736538+0000) 2019-10-13T07:34:31.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:31.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:31.533178+0000) 2019-10-13T07:34:31.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:31.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:31.533020+0000) 2019-10-13T07:34:32.819 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:32.814+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:32.816885+0000) 2019-10-13T07:34:33.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:33.424+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.426486+0000) 2019-10-13T07:34:33.428 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:33.424+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.426985+0000) 2019-10-13T07:34:33.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:33.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:33.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:33.424+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.427841+0000) 2019-10-13T07:34:33.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:33.422+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.427681+0000) 2019-10-13T07:34:33.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:33.424+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.428615+0000) 2019-10-13T07:34:33.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:33.424+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.428775+0000) 2019-10-13T07:34:33.432 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:33.424+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.429369+0000) 2019-10-13T07:34:33.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:33.434+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:33.436157+0000) 2019-10-13T07:34:34.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:34.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:34.257600+0000) 2019-10-13T07:34:37.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:37.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.095987+0000) 2019-10-13T07:34:37.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:37.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:37.510394+0000) 2019-10-13T07:34:38.432 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:38.428+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.430349+0000) 2019-10-13T07:34:38.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:38.428+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.431295+0000) 2019-10-13T07:34:38.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:38.428+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.431515+0000) 2019-10-13T07:34:38.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:38.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:38.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:38.428+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.432520+0000) 2019-10-13T07:34:38.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:38.428+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.432677+0000) 2019-10-13T07:34:38.435 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:38.430+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.433204+0000) 2019-10-13T07:34:38.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:38.432+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.434283+0000) 2019-10-13T07:34:38.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:38.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:38.441890+0000) 2019-10-13T07:34:40.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:40.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:40.638351+0000) 2019-10-13T07:34:40.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:40.732+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:40.736733+0000) 2019-10-13T07:34:41.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:41.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:41.533405+0000) 2019-10-13T07:34:41.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:41.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:41.533214+0000) 2019-10-13T07:34:42.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:42.822+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:42.823307+0000) 2019-10-13T07:34:43.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:43.432+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.435542+0000) 2019-10-13T07:34:43.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:43.432+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.436929+0000) 2019-10-13T07:34:43.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:43.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.436653+0000) 2019-10-13T07:34:43.439 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:43.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:43.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:43.436+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.438324+0000) 2019-10-13T07:34:43.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:43.436+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.438727+0000) 2019-10-13T07:34:43.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:43.436+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.440081+0000) 2019-10-13T07:34:43.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:43.436+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.441405+0000) 2019-10-13T07:34:43.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:43.442+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:43.446565+0000) 2019-10-13T07:34:44.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:44.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:44.257776+0000) 2019-10-13T07:34:47.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:47.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.096186+0000) 2019-10-13T07:34:47.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:47.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:47.510592+0000) 2019-10-13T07:34:48.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:48.436+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.440232+0000) 2019-10-13T07:34:48.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:48.436+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.440485+0000) 2019-10-13T07:34:48.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:48.436+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.441112+0000) 2019-10-13T07:34:48.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:48.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:48.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:48.438+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.442181+0000) 2019-10-13T07:34:48.444 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:48.440+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.443222+0000) 2019-10-13T07:34:48.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:48.440+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.442310+0000) 2019-10-13T07:34:48.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:48.444+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.446610+0000) 2019-10-13T07:34:48.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:48.446+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:48.450670+0000) 2019-10-13T07:34:50.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:50.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:50.638550+0000) 2019-10-13T07:34:50.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:50.732+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:50.736931+0000) 2019-10-13T07:34:51.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:51.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:51.533599+0000) 2019-10-13T07:34:51.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:51.530+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:51.533408+0000) 2019-10-13T07:34:52.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:52.825+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:52.829739+0000) 2019-10-13T07:34:53.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:53.440+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.444270+0000) 2019-10-13T07:34:53.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:53.444+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.445904+0000) 2019-10-13T07:34:53.447 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:53.444+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.446297+0000) 2019-10-13T07:34:53.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:53.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:53.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:53.441+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.446130+0000) 2019-10-13T07:34:53.449 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:53.444+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.447358+0000) 2019-10-13T07:34:53.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:53.444+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.447268+0000) 2019-10-13T07:34:53.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:53.444+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.447727+0000) 2019-10-13T07:34:53.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:53.449+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:53.455308+0000) 2019-10-13T07:34:54.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:54.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:54.257981+0000) 2019-10-13T07:34:55.928 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:34:55.928 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:55.936 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:34:55.936 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:34:57.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:57.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.096392+0000) 2019-10-13T07:34:57.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:57.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:57.510826+0000) 2019-10-13T07:34:58.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:34:58.444+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.449598+0000) 2019-10-13T07:34:58.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:34:58.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:34:58.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:34:58.448+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.451623+0000) 2019-10-13T07:34:58.453 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:34:58.448+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.452059+0000) 2019-10-13T07:34:58.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:34:58.449+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.452680+0000) 2019-10-13T07:34:58.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:34:58.452+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.454099+0000) 2019-10-13T07:34:58.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:34:58.452+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.456578+0000) 2019-10-13T07:34:58.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:34:58.452+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.457211+0000) 2019-10-13T07:34:58.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:34:58.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:34:58.462402+0000) 2019-10-13T07:35:00.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:00.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:00.638788+0000) 2019-10-13T07:35:00.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:00.736+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:00.737140+0000) 2019-10-13T07:35:01.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:01.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:01.533809+0000) 2019-10-13T07:35:01.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:01.529+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:01.533607+0000) 2019-10-13T07:35:02.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:02.833+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:02.836197+0000) 2019-10-13T07:35:03.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:03.452+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.455170+0000) 2019-10-13T07:35:03.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:03.452+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.456369+0000) 2019-10-13T07:35:03.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:03.452+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.456614+0000) 2019-10-13T07:35:03.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:03.449+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.453698+0000) 2019-10-13T07:35:03.460 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:03.452+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53953 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:03.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:03.452+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.457446+0000) 2019-10-13T07:35:03.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:03.456+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.457779+0000) 2019-10-13T07:35:03.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:03.456+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.458160+0000) 2019-10-13T07:35:03.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:03.461+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:03.464723+0000) 2019-10-13T07:35:04.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:04.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:04.258170+0000) 2019-10-13T07:35:07.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:07.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.096651+0000) 2019-10-13T07:35:07.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:07.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:07.511024+0000) 2019-10-13T07:35:08.462 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:08.456+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.460133+0000) 2019-10-13T07:35:08.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:08.456+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.460815+0000) 2019-10-13T07:35:08.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:08.456+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.461149+0000) 2019-10-13T07:35:08.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:08.460+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:08.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:08.457+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.462144+0000) 2019-10-13T07:35:08.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:08.460+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.463024+0000) 2019-10-13T07:35:08.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:08.460+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.463849+0000) 2019-10-13T07:35:08.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:08.460+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.464291+0000) 2019-10-13T07:35:08.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:08.461+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:08.466256+0000) 2019-10-13T07:35:10.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:10.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:10.639017+0000) 2019-10-13T07:35:11.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:11.529+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:11.533808+0000) 2019-10-13T07:35:11.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:10.736+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:10.737335+0000) 2019-10-13T07:35:11.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:11.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:11.534024+0000) 2019-10-13T07:35:12.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:12.837+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:12.842554+0000) 2019-10-13T07:35:13.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:13.460+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.464946+0000) 2019-10-13T07:35:13.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:13.464+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:13.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:13.464+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.467676+0000) 2019-10-13T07:35:13.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:13.464+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.468479+0000) 2019-10-13T07:35:13.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:13.464+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.468951+0000) 2019-10-13T07:35:13.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:13.465+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.468805+0000) 2019-10-13T07:35:13.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:13.464+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.469484+0000) 2019-10-13T07:35:13.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:13.468+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.470286+0000) 2019-10-13T07:35:13.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:13.473+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:13.476589+0000) 2019-10-13T07:35:14.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:14.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:14.258410+0000) 2019-10-13T07:35:17.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:17.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.096861+0000) 2019-10-13T07:35:17.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:17.508+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:17.511228+0000) 2019-10-13T07:35:18.472 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:18.465+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.470112+0000) 2019-10-13T07:35:18.474 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:18.468+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:18.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:18.472+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.473816+0000) 2019-10-13T07:35:18.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:18.472+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.474005+0000) 2019-10-13T07:35:18.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:18.472+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.474587+0000) 2019-10-13T07:35:18.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:18.472+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.474841+0000) 2019-10-13T07:35:18.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:18.472+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.474576+0000) 2019-10-13T07:35:18.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:18.472+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.475557+0000) 2019-10-13T07:35:18.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:18.473+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:18.478547+0000) 2019-10-13T07:35:20.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:20.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:20.639237+0000) 2019-10-13T07:35:20.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:20.732+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:20.737531+0000) 2019-10-13T07:35:21.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:21.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:21.534177+0000) 2019-10-13T07:35:21.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:21.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:21.534036+0000) 2019-10-13T07:35:22.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:22.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:22.848900+0000) 2019-10-13T07:35:23.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:23.472+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.476414+0000) 2019-10-13T07:35:23.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:23.472+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.476872+0000) 2019-10-13T07:35:23.479 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:23.473+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.477655+0000) 2019-10-13T07:35:23.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:23.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54077 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:23.480 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:23.476+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.478651+0000) 2019-10-13T07:35:23.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:23.476+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.478633+0000) 2019-10-13T07:35:23.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:23.476+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.479645+0000) 2019-10-13T07:35:23.483 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:23.476+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.481461+0000) 2019-10-13T07:35:23.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:23.485+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:23.488110+0000) 2019-10-13T07:35:24.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:24.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:24.258593+0000) 2019-10-13T07:35:26.050 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:35:26.050 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:26.054 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:35:26.055 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:27.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:27.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.097107+0000) 2019-10-13T07:35:27.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:27.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:27.511421+0000) 2019-10-13T07:35:28.483 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:28.477+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.480666+0000) 2019-10-13T07:35:28.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:28.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:28.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:28.479+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.483360+0000) 2019-10-13T07:35:28.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:28.479+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.484327+0000) 2019-10-13T07:35:28.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:28.479+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.484973+0000) 2019-10-13T07:35:28.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:28.479+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.485237+0000) 2019-10-13T07:35:28.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:28.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.485898+0000) 2019-10-13T07:35:28.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:28.483+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.486022+0000) 2019-10-13T07:35:28.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:28.485+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:28.489394+0000) 2019-10-13T07:35:30.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:30.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:30.639442+0000) 2019-10-13T07:35:30.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:30.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:30.737772+0000) 2019-10-13T07:35:31.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:31.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:31.534389+0000) 2019-10-13T07:35:31.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:31.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:31.534249+0000) 2019-10-13T07:35:32.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:32.853+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:32.855424+0000) 2019-10-13T07:35:33.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:33.483+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.487439+0000) 2019-10-13T07:35:33.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:33.483+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.486858+0000) 2019-10-13T07:35:33.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:33.483+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.487196+0000) 2019-10-13T07:35:33.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:33.483+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.488081+0000) 2019-10-13T07:35:33.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:33.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:33.493 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:33.483+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.489222+0000) 2019-10-13T07:35:33.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:33.487+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.489836+0000) 2019-10-13T07:35:33.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:33.481+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.485680+0000) 2019-10-13T07:35:33.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:33.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:33.495560+0000) 2019-10-13T07:35:34.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:34.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:34.258771+0000) 2019-10-13T07:35:37.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:37.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.097319+0000) 2019-10-13T07:35:37.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:37.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:37.511647+0000) 2019-10-13T07:35:38.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:38.487+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.492242+0000) 2019-10-13T07:35:38.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:38.487+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.492849+0000) 2019-10-13T07:35:38.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:38.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:38.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:38.491+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.494282+0000) 2019-10-13T07:35:38.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:38.491+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.494813+0000) 2019-10-13T07:35:38.497 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:38.489+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.495433+0000) 2019-10-13T07:35:38.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:38.491+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.495536+0000) 2019-10-13T07:35:38.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:38.491+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.496365+0000) 2019-10-13T07:35:38.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:38.497+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:38.503750+0000) 2019-10-13T07:35:40.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:40.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:40.639637+0000) 2019-10-13T07:35:40.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:40.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:40.738016+0000) 2019-10-13T07:35:41.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:41.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:41.534583+0000) 2019-10-13T07:35:41.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:41.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:41.534466+0000) 2019-10-13T07:35:42.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:42.861+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:42.861766+0000) 2019-10-13T07:35:43.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:43.491+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.497074+0000) 2019-10-13T07:35:43.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:43.495+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.498252+0000) 2019-10-13T07:35:43.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:43.495+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.498589+0000) 2019-10-13T07:35:43.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:43.493+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.498490+0000) 2019-10-13T07:35:43.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:43.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54200 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:43.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:43.495+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.499751+0000) 2019-10-13T07:35:43.502 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:43.495+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.500557+0000) 2019-10-13T07:35:43.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:43.499+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.502306+0000) 2019-10-13T07:35:43.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:43.501+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:43.506360+0000) 2019-10-13T07:35:44.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:44.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:44.259005+0000) 2019-10-13T07:35:47.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:47.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.097534+0000) 2019-10-13T07:35:47.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:47.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:47.511909+0000) 2019-10-13T07:35:48.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:48.499+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.502824+0000) 2019-10-13T07:35:48.505 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:48.499+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.503218+0000) 2019-10-13T07:35:48.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:48.497+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.503655+0000) 2019-10-13T07:35:48.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:48.499+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:48.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:48.499+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.503860+0000) 2019-10-13T07:35:48.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:48.499+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.505089+0000) 2019-10-13T07:35:48.507 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:48.499+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.504977+0000) 2019-10-13T07:35:48.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:48.503+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.507080+0000) 2019-10-13T07:35:48.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:48.505+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:48.508617+0000) 2019-10-13T07:35:50.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:50.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:50.639836+0000) 2019-10-13T07:35:50.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:50.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:50.738251+0000) 2019-10-13T07:35:51.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:51.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:51.534798+0000) 2019-10-13T07:35:51.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:51.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:51.534613+0000) 2019-10-13T07:35:52.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:52.865+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:52.868302+0000) 2019-10-13T07:35:53.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:53.501+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.506795+0000) 2019-10-13T07:35:53.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:53.503+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.507966+0000) 2019-10-13T07:35:53.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:53.503+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.508232+0000) 2019-10-13T07:35:53.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:53.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:53.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:53.503+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.509428+0000) 2019-10-13T07:35:53.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:53.503+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.509553+0000) 2019-10-13T07:35:53.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:53.507+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.510239+0000) 2019-10-13T07:35:53.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:53.507+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.511246+0000) 2019-10-13T07:35:53.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:53.513+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:53.516501+0000) 2019-10-13T07:35:54.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:54.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:54.259185+0000) 2019-10-13T07:35:56.172 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:35:56.173 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:56.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:35:56.177 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:35:57.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:57.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.097736+0000) 2019-10-13T07:35:57.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:57.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:57.512107+0000) 2019-10-13T07:35:58.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:35:58.507+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.512461+0000) 2019-10-13T07:35:58.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:35:58.507+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.512618+0000) 2019-10-13T07:35:58.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:35:58.508+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.513430+0000) 2019-10-13T07:35:58.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:35:58.507+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.513367+0000) 2019-10-13T07:35:58.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:35:58.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:35:58.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:35:58.511+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.514336+0000) 2019-10-13T07:35:58.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:35:58.511+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.515041+0000) 2019-10-13T07:35:58.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:35:58.511+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.515824+0000) 2019-10-13T07:35:58.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:35:58.516+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:35:58.521690+0000) 2019-10-13T07:36:00.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:00.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:00.640040+0000) 2019-10-13T07:36:00.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:00.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:00.738480+0000) 2019-10-13T07:36:01.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:01.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:01.534993+0000) 2019-10-13T07:36:01.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:01.532+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:01.534819+0000) 2019-10-13T07:36:02.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:02.868+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:02.874453+0000) 2019-10-13T07:36:03.519 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:03.511+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.517475+0000) 2019-10-13T07:36:03.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:03.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54324 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:03.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:03.515+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.519500+0000) 2019-10-13T07:36:03.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:03.515+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.520007+0000) 2019-10-13T07:36:03.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:03.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.520230+0000) 2019-10-13T07:36:03.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:03.515+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.520355+0000) 2019-10-13T07:36:03.523 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:03.515+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.521028+0000) 2019-10-13T07:36:03.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:03.519+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.522343+0000) 2019-10-13T07:36:03.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:03.524+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:03.528547+0000) 2019-10-13T07:36:04.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:04.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:04.259365+0000) 2019-10-13T07:36:07.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:07.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.098040+0000) 2019-10-13T07:36:07.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:07.507+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:07.512344+0000) 2019-10-13T07:36:08.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:08.516+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.522525+0000) 2019-10-13T07:36:08.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:08.519+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.523326+0000) 2019-10-13T07:36:08.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:08.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:08.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:08.519+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.524305+0000) 2019-10-13T07:36:08.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:08.519+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.524614+0000) 2019-10-13T07:36:08.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:08.519+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.524841+0000) 2019-10-13T07:36:08.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:08.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.525138+0000) 2019-10-13T07:36:08.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:08.523+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.525967+0000) 2019-10-13T07:36:08.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:08.524+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:08.531828+0000) 2019-10-13T07:36:10.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:10.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:10.640243+0000) 2019-10-13T07:36:10.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:10.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:10.738682+0000) 2019-10-13T07:36:11.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:11.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:11.535186+0000) 2019-10-13T07:36:11.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:11.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:11.535022+0000) 2019-10-13T07:36:12.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:12.880+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:12.880848+0000) 2019-10-13T07:36:13.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:13.523+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.526986+0000) 2019-10-13T07:36:13.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:13.523+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.527540+0000) 2019-10-13T07:36:13.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:13.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:13.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:13.524+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.528887+0000) 2019-10-13T07:36:13.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:13.527+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.530000+0000) 2019-10-13T07:36:13.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:13.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.530477+0000) 2019-10-13T07:36:13.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:13.527+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.531041+0000) 2019-10-13T07:36:13.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:13.527+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.531378+0000) 2019-10-13T07:36:13.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:13.532+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:13.539505+0000) 2019-10-13T07:36:14.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:14.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:14.259544+0000) 2019-10-13T07:36:17.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:17.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.098249+0000) 2019-10-13T07:36:17.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:17.511+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:17.512576+0000) 2019-10-13T07:36:18.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:18.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.531354+0000) 2019-10-13T07:36:18.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:18.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:18.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:18.531+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.534639+0000) 2019-10-13T07:36:18.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:18.531+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.534448+0000) 2019-10-13T07:36:18.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:18.531+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.535906+0000) 2019-10-13T07:36:18.537 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:18.531+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.535510+0000) 2019-10-13T07:36:18.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:18.531+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.536511+0000) 2019-10-13T07:36:18.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:18.531+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.537079+0000) 2019-10-13T07:36:18.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:18.536+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:18.540954+0000) 2019-10-13T07:36:20.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:20.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:20.640459+0000) 2019-10-13T07:36:20.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:20.735+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:20.738895+0000) 2019-10-13T07:36:21.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:21.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:21.535383+0000) 2019-10-13T07:36:21.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:21.532+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:21.535220+0000) 2019-10-13T07:36:22.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:22.880+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:22.887266+0000) 2019-10-13T07:36:23.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:23.532+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.537707+0000) 2019-10-13T07:36:23.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:23.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.538832+0000) 2019-10-13T07:36:23.541 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:23.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:23.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:23.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.539397+0000) 2019-10-13T07:36:23.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:23.534+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.539494+0000) 2019-10-13T07:36:23.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:23.534+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.539800+0000) 2019-10-13T07:36:23.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:23.534+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.540229+0000) 2019-10-13T07:36:23.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:23.538+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.542629+0000) 2019-10-13T07:36:23.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:23.544+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:23.548252+0000) 2019-10-13T07:36:24.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:24.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:24.259719+0000) 2019-10-13T07:36:26.411 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:36:26.411 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:26.415 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:36:26.415 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:27.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:27.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.098470+0000) 2019-10-13T07:36:27.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:27.510+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:27.512792+0000) 2019-10-13T07:36:28.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:28.536+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.542315+0000) 2019-10-13T07:36:28.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:28.538+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.543160+0000) 2019-10-13T07:36:28.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:28.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:28.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:28.538+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.544353+0000) 2019-10-13T07:36:28.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:28.538+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.544772+0000) 2019-10-13T07:36:28.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:28.538+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.545155+0000) 2019-10-13T07:36:28.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:28.542+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.545887+0000) 2019-10-13T07:36:28.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:28.542+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.546400+0000) 2019-10-13T07:36:28.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:28.544+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:28.549350+0000) 2019-10-13T07:36:30.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:30.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:30.640662+0000) 2019-10-13T07:36:30.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:30.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:30.739099+0000) 2019-10-13T07:36:31.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:31.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:31.535583+0000) 2019-10-13T07:36:31.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:31.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:31.535415+0000) 2019-10-13T07:36:32.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:32.892+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:32.893728+0000) 2019-10-13T07:36:33.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:33.542+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.547292+0000) 2019-10-13T07:36:33.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:33.542+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.548570+0000) 2019-10-13T07:36:33.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:33.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:33.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:33.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.549513+0000) 2019-10-13T07:36:33.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:33.546+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.549903+0000) 2019-10-13T07:36:33.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:33.544+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.549686+0000) 2019-10-13T07:36:33.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:33.546+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.550680+0000) 2019-10-13T07:36:33.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:33.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.551986+0000) 2019-10-13T07:36:33.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:33.552+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:33.556749+0000) 2019-10-13T07:36:34.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:34.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:34.259898+0000) 2019-10-13T07:36:37.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:37.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.098697+0000) 2019-10-13T07:36:37.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:37.510+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:37.512991+0000) 2019-10-13T07:36:38.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:38.546+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.552533+0000) 2019-10-13T07:36:38.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:38.546+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.552960+0000) 2019-10-13T07:36:38.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:38.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.553425+0000) 2019-10-13T07:36:38.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:38.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:38.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:38.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.553704+0000) 2019-10-13T07:36:38.557 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:38.550+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.554679+0000) 2019-10-13T07:36:38.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:38.550+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.554801+0000) 2019-10-13T07:36:38.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:38.550+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.555580+0000) 2019-10-13T07:36:38.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:38.556+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:38.562660+0000) 2019-10-13T07:36:40.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:40.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:40.640889+0000) 2019-10-13T07:36:40.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:40.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:40.739300+0000) 2019-10-13T07:36:41.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:41.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:41.535781+0000) 2019-10-13T07:36:41.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:41.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:41.535647+0000) 2019-10-13T07:36:42.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:42.896+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:42.900328+0000) 2019-10-13T07:36:43.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:43.554+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.557867+0000) 2019-10-13T07:36:43.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:43.550+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.557700+0000) 2019-10-13T07:36:43.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:43.550+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.557170+0000) 2019-10-13T07:36:43.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:43.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:43.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:43.554+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.559254+0000) 2019-10-13T07:36:43.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:43.554+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.558903+0000) 2019-10-13T07:36:43.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:43.552+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.559295+0000) 2019-10-13T07:36:43.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:43.554+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.561108+0000) 2019-10-13T07:36:43.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:43.560+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:43.565449+0000) 2019-10-13T07:36:44.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:44.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:44.260097+0000) 2019-10-13T07:36:47.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:47.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.098906+0000) 2019-10-13T07:36:47.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:47.510+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:47.513195+0000) 2019-10-13T07:36:48.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:48.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:48.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:48.558+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.563924+0000) 2019-10-13T07:36:48.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:48.558+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.564059+0000) 2019-10-13T07:36:48.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:48.558+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.564452+0000) 2019-10-13T07:36:48.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:48.558+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.565165+0000) 2019-10-13T07:36:48.567 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:48.558+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.564992+0000) 2019-10-13T07:36:48.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:48.560+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.565276+0000) 2019-10-13T07:36:48.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:48.562+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.566105+0000) 2019-10-13T07:36:48.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:48.568+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:48.574485+0000) 2019-10-13T07:36:50.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:50.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:50.641097+0000) 2019-10-13T07:36:50.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:50.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:50.739499+0000) 2019-10-13T07:36:51.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:51.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:51.535999+0000) 2019-10-13T07:36:51.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:51.528+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:51.535854+0000) 2019-10-13T07:36:52.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:52.904+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:52.906436+0000) 2019-10-13T07:36:53.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:53.562+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.567078+0000) 2019-10-13T07:36:53.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:53.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:53.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:53.562+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.568639+0000) 2019-10-13T07:36:53.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:53.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.569288+0000) 2019-10-13T07:36:53.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:53.566+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.572087+0000) 2019-10-13T07:36:53.574 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:53.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.572792+0000) 2019-10-13T07:36:53.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:53.566+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.573286+0000) 2019-10-13T07:36:53.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:53.566+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.573727+0000) 2019-10-13T07:36:53.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:53.572+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:53.577719+0000) 2019-10-13T07:36:54.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:54.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:54.260292+0000) 2019-10-13T07:36:56.524 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:36:56.525 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:56.528 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:36:56.529 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:36:57.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:57.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.099142+0000) 2019-10-13T07:36:57.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:57.506+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:57.513395+0000) 2019-10-13T07:36:58.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:36:58.566+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.571833+0000) 2019-10-13T07:36:58.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:36:58.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:36:58.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:36:58.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.573559+0000) 2019-10-13T07:36:58.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:36:58.570+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.573843+0000) 2019-10-13T07:36:58.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:36:58.572+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.573988+0000) 2019-10-13T07:36:58.577 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:36:58.570+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.576131+0000) 2019-10-13T07:36:58.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:36:58.570+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.576291+0000) 2019-10-13T07:36:58.578 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:36:58.570+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.577061+0000) 2019-10-13T07:36:58.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:36:58.580+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:36:58.580472+0000) 2019-10-13T07:37:00.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:00.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:00.641331+0000) 2019-10-13T07:37:00.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:00.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:00.739714+0000) 2019-10-13T07:37:01.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:01.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:01.536203+0000) 2019-10-13T07:37:01.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:01.536+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:01.536113+0000) 2019-10-13T07:37:02.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:02.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:02.912791+0000) 2019-10-13T07:37:03.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:03.570+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.576285+0000) 2019-10-13T07:37:03.578 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:03.570+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.576925+0000) 2019-10-13T07:37:03.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:03.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:03.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:03.574+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.578110+0000) 2019-10-13T07:37:03.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:03.574+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.578186+0000) 2019-10-13T07:37:03.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:03.574+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.578865+0000) 2019-10-13T07:37:03.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:03.576+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.579741+0000) 2019-10-13T07:37:03.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:03.574+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.580699+0000) 2019-10-13T07:37:03.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:03.584+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:03.587122+0000) 2019-10-13T07:37:04.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:04.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:04.260469+0000) 2019-10-13T07:37:07.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:07.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.099353+0000) 2019-10-13T07:37:07.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:07.510+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:07.513595+0000) 2019-10-13T07:37:08.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:08.574+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.581558+0000) 2019-10-13T07:37:08.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:08.578+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.582667+0000) 2019-10-13T07:37:08.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:08.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54726 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:08.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:08.578+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.583261+0000) 2019-10-13T07:37:08.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:08.583+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.584459+0000) 2019-10-13T07:37:08.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:08.578+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.583829+0000) 2019-10-13T07:37:08.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:08.578+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.584196+0000) 2019-10-13T07:37:08.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:08.582+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.585906+0000) 2019-10-13T07:37:08.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:08.587+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:08.591338+0000) 2019-10-13T07:37:10.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:10.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:10.641535+0000) 2019-10-13T07:37:10.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:10.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:10.739930+0000) 2019-10-13T07:37:11.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:11.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:11.536398+0000) 2019-10-13T07:37:11.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:11.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:11.536278+0000) 2019-10-13T07:37:12.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:12.915+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:12.919342+0000) 2019-10-13T07:37:13.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:13.582+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.586604+0000) 2019-10-13T07:37:13.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:13.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.586881+0000) 2019-10-13T07:37:13.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:13.582+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54759 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:13.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:13.582+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.588950+0000) 2019-10-13T07:37:13.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:13.582+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.589374+0000) 2019-10-13T07:37:13.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:13.586+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.590403+0000) 2019-10-13T07:37:13.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:13.582+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.589464+0000) 2019-10-13T07:37:13.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:13.586+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.591328+0000) 2019-10-13T07:37:13.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:13.591+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:13.592745+0000) 2019-10-13T07:37:14.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:14.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:14.260647+0000) 2019-10-13T07:37:17.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:17.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.099559+0000) 2019-10-13T07:37:17.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:17.510+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:17.513827+0000) 2019-10-13T07:37:18.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:18.585+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.591333+0000) 2019-10-13T07:37:18.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:18.585+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.591568+0000) 2019-10-13T07:37:18.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:18.587+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.592018+0000) 2019-10-13T07:37:18.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:18.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.592197+0000) 2019-10-13T07:37:18.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:18.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:18.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:18.585+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.593689+0000) 2019-10-13T07:37:18.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:18.589+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.594106+0000) 2019-10-13T07:37:18.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:18.589+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.594819+0000) 2019-10-13T07:37:18.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:18.599+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:18.601876+0000) 2019-10-13T07:37:20.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:20.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:20.641765+0000) 2019-10-13T07:37:20.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:20.733+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:20.740129+0000) 2019-10-13T07:37:21.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:21.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:21.536594+0000) 2019-10-13T07:37:21.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:21.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:21.536479+0000) 2019-10-13T07:37:22.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:22.923+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:22.925614+0000) 2019-10-13T07:37:23.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:23.589+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.596163+0000) 2019-10-13T07:37:23.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:23.589+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.597518+0000) 2019-10-13T07:37:23.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:23.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54820 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:23.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:23.593+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.598579+0000) 2019-10-13T07:37:23.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:23.593+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.598959+0000) 2019-10-13T07:37:23.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:23.593+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.599482+0000) 2019-10-13T07:37:23.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:23.595+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.600061+0000) 2019-10-13T07:37:23.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:23.593+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.600007+0000) 2019-10-13T07:37:23.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:23.603+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:23.606846+0000) 2019-10-13T07:37:24.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:24.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:24.260822+0000) 2019-10-13T07:37:26.605 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:37:26.605 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:26.609 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:37:26.609 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:27.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:27.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.099764+0000) 2019-10-13T07:37:27.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:27.509+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:27.514021+0000) 2019-10-13T07:37:28.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:28.597+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.602027+0000) 2019-10-13T07:37:28.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:28.597+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.602393+0000) 2019-10-13T07:37:28.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:28.597+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.603209+0000) 2019-10-13T07:37:28.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:28.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:28.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:28.597+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.604180+0000) 2019-10-13T07:37:28.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:28.597+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.604852+0000) 2019-10-13T07:37:28.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:28.603+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.604696+0000) 2019-10-13T07:37:28.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:28.601+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.605773+0000) 2019-10-13T07:37:28.611 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:28.607+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:28.609897+0000) 2019-10-13T07:37:30.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:30.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:30.641933+0000) 2019-10-13T07:37:30.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:30.737+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:30.740329+0000) 2019-10-13T07:37:31.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:31.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:31.536795+0000) 2019-10-13T07:37:31.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:31.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:31.536693+0000) 2019-10-13T07:37:32.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:32.927+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:32.932000+0000) 2019-10-13T07:37:33.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:33.605+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.606577+0000) 2019-10-13T07:37:33.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:33.605+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.607007+0000) 2019-10-13T07:37:33.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:33.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:33.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:33.605+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.608725+0000) 2019-10-13T07:37:33.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:33.607+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.609729+0000) 2019-10-13T07:37:33.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:33.609+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.610441+0000) 2019-10-13T07:37:33.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:33.609+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.610920+0000) 2019-10-13T07:37:33.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:33.609+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.612156+0000) 2019-10-13T07:37:33.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:33.615+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:33.616437+0000) 2019-10-13T07:37:34.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:34.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:34.260999+0000) 2019-10-13T07:37:37.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:37.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.099980+0000) 2019-10-13T07:37:37.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:37.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:37.514261+0000) 2019-10-13T07:37:38.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:38.609+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.611592+0000) 2019-10-13T07:37:38.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:38.609+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.612099+0000) 2019-10-13T07:37:38.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:38.609+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.612780+0000) 2019-10-13T07:37:38.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:38.611+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.612586+0000) 2019-10-13T07:37:38.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:38.609+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.613644+0000) 2019-10-13T07:37:38.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:38.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:38.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:38.613+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.614031+0000) 2019-10-13T07:37:38.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:38.613+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.615582+0000) 2019-10-13T07:37:38.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:38.619+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:38.622128+0000) 2019-10-13T07:37:40.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:40.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:40.642136+0000) 2019-10-13T07:37:40.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:40.737+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:40.740531+0000) 2019-10-13T07:37:41.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:41.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:41.536988+0000) 2019-10-13T07:37:41.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:41.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:41.536932+0000) 2019-10-13T07:37:42.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:42.935+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:42.938343+0000) 2019-10-13T07:37:43.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:43.613+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.616728+0000) 2019-10-13T07:37:43.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:43.613+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.617471+0000) 2019-10-13T07:37:43.619 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:43.617+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.618313+0000) 2019-10-13T07:37:43.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:43.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:43.620 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:43.617+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.618858+0000) 2019-10-13T07:37:43.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:43.617+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.618832+0000) 2019-10-13T07:37:43.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:43.617+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.620135+0000) 2019-10-13T07:37:43.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:43.619+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.620686+0000) 2019-10-13T07:37:43.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:43.627+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:43.629065+0000) 2019-10-13T07:37:44.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:44.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:44.261179+0000) 2019-10-13T07:37:47.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:47.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.100207+0000) 2019-10-13T07:37:47.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:47.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:47.514495+0000) 2019-10-13T07:37:48.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:48.621+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.622431+0000) 2019-10-13T07:37:48.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:48.621+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.622832+0000) 2019-10-13T07:37:48.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:48.621+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.623159+0000) 2019-10-13T07:37:48.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:48.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:48.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:48.621+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.624392+0000) 2019-10-13T07:37:48.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:48.623+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.624196+0000) 2019-10-13T07:37:48.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:48.625+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.625922+0000) 2019-10-13T07:37:48.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:48.625+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.626717+0000) 2019-10-13T07:37:48.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:48.631+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:48.634572+0000) 2019-10-13T07:37:50.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:50.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:50.642368+0000) 2019-10-13T07:37:50.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:50.737+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:50.740786+0000) 2019-10-13T07:37:51.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:51.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:51.537188+0000) 2019-10-13T07:37:51.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:51.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:51.537140+0000) 2019-10-13T07:37:52.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:52.943+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:52.945666+0000) 2019-10-13T07:37:53.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:53.625+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.626632+0000) 2019-10-13T07:37:53.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:53.625+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.628108+0000) 2019-10-13T07:37:53.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:53.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:53.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:53.627+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.629018+0000) 2019-10-13T07:37:53.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:53.629+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.630105+0000) 2019-10-13T07:37:53.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:53.629+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.630412+0000) 2019-10-13T07:37:53.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:53.629+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.631814+0000) 2019-10-13T07:37:53.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:53.629+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.632682+0000) 2019-10-13T07:37:53.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:53.631+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:53.635107+0000) 2019-10-13T07:37:54.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:54.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:54.261365+0000) 2019-10-13T07:37:56.744 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:37:56.744 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:56.748 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:37:56.748 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:37:57.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:57.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.100408+0000) 2019-10-13T07:37:57.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:57.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:57.514722+0000) 2019-10-13T07:37:58.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:37:58.629+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.631511+0000) 2019-10-13T07:37:58.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:37:58.629+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.632558+0000) 2019-10-13T07:37:58.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:37:58.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:37:58.635 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:37:58.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.633875+0000) 2019-10-13T07:37:58.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:37:58.633+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.634209+0000) 2019-10-13T07:37:58.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:37:58.631+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.634820+0000) 2019-10-13T07:37:58.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:37:58.633+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.635847+0000) 2019-10-13T07:37:58.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:37:58.633+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.636814+0000) 2019-10-13T07:37:58.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:37:58.639+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:37:58.641379+0000) 2019-10-13T07:38:00.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:00.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:00.642573+0000) 2019-10-13T07:38:00.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:00.737+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:00.741016+0000) 2019-10-13T07:38:01.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:01.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:01.537426+0000) 2019-10-13T07:38:01.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:01.535+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:01.537378+0000) 2019-10-13T07:38:02.954 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:02.947+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:02.952000+0000) 2019-10-13T07:38:03.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:03.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.636615+0000) 2019-10-13T07:38:03.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:03.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.636770+0000) 2019-10-13T07:38:03.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:03.637+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.638172+0000) 2019-10-13T07:38:03.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:03.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:03.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:03.637+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.639189+0000) 2019-10-13T07:38:03.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:03.637+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.639756+0000) 2019-10-13T07:38:03.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:03.637+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.640916+0000) 2019-10-13T07:38:03.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:03.641+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.641772+0000) 2019-10-13T07:38:03.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:03.643+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:03.647032+0000) 2019-10-13T07:38:04.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:04.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:04.261586+0000) 2019-10-13T07:38:07.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:07.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.100639+0000) 2019-10-13T07:38:07.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:07.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:07.514942+0000) 2019-10-13T07:38:08.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:08.641+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.642271+0000) 2019-10-13T07:38:08.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:08.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:08.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:08.641+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.643678+0000) 2019-10-13T07:38:08.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:08.641+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.643743+0000) 2019-10-13T07:38:08.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:08.641+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.644069+0000) 2019-10-13T07:38:08.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:08.639+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.643852+0000) 2019-10-13T07:38:08.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:08.641+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.644483+0000) 2019-10-13T07:38:08.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:08.641+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.644935+0000) 2019-10-13T07:38:08.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:08.647+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:08.650019+0000) 2019-10-13T07:38:10.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:10.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:10.642788+0000) 2019-10-13T07:38:10.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:10.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:10.741219+0000) 2019-10-13T07:38:11.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:11.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:11.537651+0000) 2019-10-13T07:38:11.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:11.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:11.537601+0000) 2019-10-13T07:38:12.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:12.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:12.963589+0000) 2019-10-13T07:38:13.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:13.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.646500+0000) 2019-10-13T07:38:13.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:13.645+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.647080+0000) 2019-10-13T07:38:13.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:13.642+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.647382+0000) 2019-10-13T07:38:13.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:13.645+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.648121+0000) 2019-10-13T07:38:13.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:13.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:13.650 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:13.645+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.648715+0000) 2019-10-13T07:38:13.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:13.645+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.649441+0000) 2019-10-13T07:38:13.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:13.649+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.650190+0000) 2019-10-13T07:38:13.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:13.650+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:13.654389+0000) 2019-10-13T07:38:14.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:14.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:14.261788+0000) 2019-10-13T07:38:17.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:17.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.100841+0000) 2019-10-13T07:38:17.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:17.512+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:17.515151+0000) 2019-10-13T07:38:18.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:18.646+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.651587+0000) 2019-10-13T07:38:18.654 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:18.648+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.652580+0000) 2019-10-13T07:38:18.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:18.648+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.652873+0000) 2019-10-13T07:38:18.655 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:18.648+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.653227+0000) 2019-10-13T07:38:18.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:18.648+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.653539+0000) 2019-10-13T07:38:18.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:18.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55158 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:18.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:18.652+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.654290+0000) 2019-10-13T07:38:18.656 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:18.652+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.654714+0000) 2019-10-13T07:38:18.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:18.654+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:18.658849+0000) 2019-10-13T07:38:20.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:20.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:20.642992+0000) 2019-10-13T07:38:20.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:20.736+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:20.741423+0000) 2019-10-13T07:38:21.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:21.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:21.537849+0000) 2019-10-13T07:38:21.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:21.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:21.537796+0000) 2019-10-13T07:38:22.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:22.966+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:22.970021+0000) 2019-10-13T07:38:23.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:23.654+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.656206+0000) 2019-10-13T07:38:23.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:23.652+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.657628+0000) 2019-10-13T07:38:23.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:23.656+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.659443+0000) 2019-10-13T07:38:23.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:23.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:23.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:23.656+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.659864+0000) 2019-10-13T07:38:23.662 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:23.656+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.660487+0000) 2019-10-13T07:38:23.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:23.656+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.661256+0000) 2019-10-13T07:38:23.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:23.660+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.661789+0000) 2019-10-13T07:38:23.669 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:23.662+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:23.667408+0000) 2019-10-13T07:38:24.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:24.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:24.261973+0000) 2019-10-13T07:38:26.815 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:38:26.816 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:26.819 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:38:26.819 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:27.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:27.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.101073+0000) 2019-10-13T07:38:27.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:27.512+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:27.515348+0000) 2019-10-13T07:38:28.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:28.660+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.662114+0000) 2019-10-13T07:38:28.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:28.658+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.663359+0000) 2019-10-13T07:38:28.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:28.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:28.666 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:28.660+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.664427+0000) 2019-10-13T07:38:28.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:28.660+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.664538+0000) 2019-10-13T07:38:28.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:28.660+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.664887+0000) 2019-10-13T07:38:28.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:28.660+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.665653+0000) 2019-10-13T07:38:28.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:28.664+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.666261+0000) 2019-10-13T07:38:28.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:28.666+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:28.669027+0000) 2019-10-13T07:38:30.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:30.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:30.643199+0000) 2019-10-13T07:38:30.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:30.736+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:30.741616+0000) 2019-10-13T07:38:31.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:31.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:31.538046+0000) 2019-10-13T07:38:31.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:31.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:31.538024+0000) 2019-10-13T07:38:32.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:32.974+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:32.976564+0000) 2019-10-13T07:38:33.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:33.666+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.668235+0000) 2019-10-13T07:38:33.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:33.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.669176+0000) 2019-10-13T07:38:33.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:33.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:33.671 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:33.664+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.669348+0000) 2019-10-13T07:38:33.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:33.668+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.669901+0000) 2019-10-13T07:38:33.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:33.668+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.670156+0000) 2019-10-13T07:38:33.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:33.668+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.670517+0000) 2019-10-13T07:38:33.673 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:33.668+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.670914+0000) 2019-10-13T07:38:33.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:33.674+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:33.678200+0000) 2019-10-13T07:38:34.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:34.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:34.262154+0000) 2019-10-13T07:38:37.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:37.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.101280+0000) 2019-10-13T07:38:37.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:37.512+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:37.515573+0000) 2019-10-13T07:38:38.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:38.668+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.672871+0000) 2019-10-13T07:38:38.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:38.668+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.673378+0000) 2019-10-13T07:38:38.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:38.670+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.673518+0000) 2019-10-13T07:38:38.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:38.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:38.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:38.672+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.675078+0000) 2019-10-13T07:38:38.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:38.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.675342+0000) 2019-10-13T07:38:38.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:38.672+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.675706+0000) 2019-10-13T07:38:38.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:38.672+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.677013+0000) 2019-10-13T07:38:38.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:38.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:38.683608+0000) 2019-10-13T07:38:40.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:40.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:40.643400+0000) 2019-10-13T07:38:40.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:40.740+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:40.741819+0000) 2019-10-13T07:38:41.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:41.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:41.538193+0000) 2019-10-13T07:38:41.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:41.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:41.538242+0000) 2019-10-13T07:38:42.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:42.982+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:42.983009+0000) 2019-10-13T07:38:43.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:43.676+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.678226+0000) 2019-10-13T07:38:43.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:43.676+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.678477+0000) 2019-10-13T07:38:43.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:43.674+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.678451+0000) 2019-10-13T07:38:43.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:43.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.679214+0000) 2019-10-13T07:38:43.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:43.676+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.679692+0000) 2019-10-13T07:38:43.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:43.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:43.682 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:43.676+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.680660+0000) 2019-10-13T07:38:43.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:43.676+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.680816+0000) 2019-10-13T07:38:43.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:43.682+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:43.684683+0000) 2019-10-13T07:38:44.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:44.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:44.262357+0000) 2019-10-13T07:38:47.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:47.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.101486+0000) 2019-10-13T07:38:47.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:47.512+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:47.515775+0000) 2019-10-13T07:38:48.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:48.680+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.683540+0000) 2019-10-13T07:38:48.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:48.680+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.684313+0000) 2019-10-13T07:38:48.686 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:48.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:48.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:48.684+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.685843+0000) 2019-10-13T07:38:48.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:48.684+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.686160+0000) 2019-10-13T07:38:48.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:48.682+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.686061+0000) 2019-10-13T07:38:48.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:48.684+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.686849+0000) 2019-10-13T07:38:48.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:48.684+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.687094+0000) 2019-10-13T07:38:48.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:48.690+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:48.692773+0000) 2019-10-13T07:38:50.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:50.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:50.643605+0000) 2019-10-13T07:38:50.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:50.740+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:50.742018+0000) 2019-10-13T07:38:51.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:51.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:51.538401+0000) 2019-10-13T07:38:51.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:51.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:51.538436+0000) 2019-10-13T07:38:52.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:52.986+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:52.989384+0000) 2019-10-13T07:38:53.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:53.684+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.688478+0000) 2019-10-13T07:38:53.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:53.684+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.689058+0000) 2019-10-13T07:38:53.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:53.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:53.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:53.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.691090+0000) 2019-10-13T07:38:53.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:53.686+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.690998+0000) 2019-10-13T07:38:53.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:53.688+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.692074+0000) 2019-10-13T07:38:53.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:53.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.691985+0000) 2019-10-13T07:38:53.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:53.688+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.692288+0000) 2019-10-13T07:38:53.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:53.690+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:53.695805+0000) 2019-10-13T07:38:54.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:54.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:54.262560+0000) 2019-10-13T07:38:56.981 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:38:56.981 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:56.985 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:38:56.985 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:38:57.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:57.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.101730+0000) 2019-10-13T07:38:57.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:57.512+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:57.516011+0000) 2019-10-13T07:38:58.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:38:58.692+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.694131+0000) 2019-10-13T07:38:58.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:38:58.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:38:58.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:38:58.692+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.695774+0000) 2019-10-13T07:38:58.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:38:58.692+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.696164+0000) 2019-10-13T07:38:58.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:38:58.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.695941+0000) 2019-10-13T07:38:58.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:38:58.692+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.697291+0000) 2019-10-13T07:38:58.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:38:58.692+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.697336+0000) 2019-10-13T07:38:58.699 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:38:58.696+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.698312+0000) 2019-10-13T07:38:58.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:38:58.698+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:38:58.703733+0000) 2019-10-13T07:39:00.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:00.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:00.643800+0000) 2019-10-13T07:39:00.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:00.740+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:00.742260+0000) 2019-10-13T07:39:01.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:01.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:01.538591+0000) 2019-10-13T07:39:01.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:01.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:01.538643+0000) 2019-10-13T07:39:02.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:02.990+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:02.995768+0000) 2019-10-13T07:39:03.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:03.696+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.698797+0000) 2019-10-13T07:39:03.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:03.696+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.699036+0000) 2019-10-13T07:39:03.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:03.696+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.699722+0000) 2019-10-13T07:39:03.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:03.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:03.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:03.698+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.700247+0000) 2019-10-13T07:39:03.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:03.696+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.701653+0000) 2019-10-13T07:39:03.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:03.700+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.701844+0000) 2019-10-13T07:39:03.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:03.700+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.702340+0000) 2019-10-13T07:39:03.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:03.702+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:03.706470+0000) 2019-10-13T07:39:04.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:04.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:04.262761+0000) 2019-10-13T07:39:07.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:07.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.101961+0000) 2019-10-13T07:39:07.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:07.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:07.516243+0000) 2019-10-13T07:39:08.705 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:08.700+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.703806+0000) 2019-10-13T07:39:08.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:08.700+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.704093+0000) 2019-10-13T07:39:08.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:08.700+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.704496+0000) 2019-10-13T07:39:08.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:08.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:08.707 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:08.700+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.705274+0000) 2019-10-13T07:39:08.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:08.700+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.705425+0000) 2019-10-13T07:39:08.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:08.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.705949+0000) 2019-10-13T07:39:08.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:08.702+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.707061+0000) 2019-10-13T07:39:08.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:08.710+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:08.715725+0000) 2019-10-13T07:39:10.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:10.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:10.644032+0000) 2019-10-13T07:39:10.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:10.740+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:10.742477+0000) 2019-10-13T07:39:11.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:11.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:11.538789+0000) 2019-10-13T07:39:11.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:11.534+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:11.538844+0000) 2019-10-13T07:39:13.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:12.998+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.002089+0000) 2019-10-13T07:39:13.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:13.703+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.708089+0000) 2019-10-13T07:39:13.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:13.703+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.708665+0000) 2019-10-13T07:39:13.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:13.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:13.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:13.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.710544+0000) 2019-10-13T07:39:13.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:13.706+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.710504+0000) 2019-10-13T07:39:13.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:13.707+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.711512+0000) 2019-10-13T07:39:13.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:13.707+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.711861+0000) 2019-10-13T07:39:13.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:13.707+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.713044+0000) 2019-10-13T07:39:13.723 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:13.718+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:13.721859+0000) 2019-10-13T07:39:14.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:14.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:14.262940+0000) 2019-10-13T07:39:17.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:17.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.102180+0000) 2019-10-13T07:39:17.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:17.515+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:17.516480+0000) 2019-10-13T07:39:18.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:18.711+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.714006+0000) 2019-10-13T07:39:18.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:18.711+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.714961+0000) 2019-10-13T07:39:18.716 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:18.711+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.715326+0000) 2019-10-13T07:39:18.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:18.711+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.715987+0000) 2019-10-13T07:39:18.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:18.713+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.716377+0000) 2019-10-13T07:39:18.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:18.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:18.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:18.711+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.716770+0000) 2019-10-13T07:39:18.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:18.711+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.717661+0000) 2019-10-13T07:39:18.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:18.717+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:18.724017+0000) 2019-10-13T07:39:20.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:20.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:20.644229+0000) 2019-10-13T07:39:20.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:20.739+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:20.742677+0000) 2019-10-13T07:39:21.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:21.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:21.539013+0000) 2019-10-13T07:39:21.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:21.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:21.539060+0000) 2019-10-13T07:39:23.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:23.005+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.008690+0000) 2019-10-13T07:39:23.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:23.715+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.720615+0000) 2019-10-13T07:39:23.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:23.715+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.721298+0000) 2019-10-13T07:39:23.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:23.719+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:23.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:23.719+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.722560+0000) 2019-10-13T07:39:23.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:23.719+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.722721+0000) 2019-10-13T07:39:23.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:23.719+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.723225+0000) 2019-10-13T07:39:23.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:23.721+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.725933+0000) 2019-10-13T07:39:23.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:23.721+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.726151+0000) 2019-10-13T07:39:23.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:23.719+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:23.725213+0000) 2019-10-13T07:39:24.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:24.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:24.263126+0000) 2019-10-13T07:39:27.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:27.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.102387+0000) 2019-10-13T07:39:27.109 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:39:27.109 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:27.113 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:39:27.113 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:27.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:27.515+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:27.516721+0000) 2019-10-13T07:39:28.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:28.719+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.724749+0000) 2019-10-13T07:39:28.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:28.723+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.726271+0000) 2019-10-13T07:39:28.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:28.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:28.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:28.723+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.727740+0000) 2019-10-13T07:39:28.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:28.723+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.728762+0000) 2019-10-13T07:39:28.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:28.723+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.729242+0000) 2019-10-13T07:39:28.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:28.727+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.729802+0000) 2019-10-13T07:39:28.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:28.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.729760+0000) 2019-10-13T07:39:28.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:28.729+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:28.732584+0000) 2019-10-13T07:39:30.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:30.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:30.644428+0000) 2019-10-13T07:39:30.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:30.739+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:30.742879+0000) 2019-10-13T07:39:31.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:31.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:31.539206+0000) 2019-10-13T07:39:31.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:31.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:31.539281+0000) 2019-10-13T07:39:33.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:33.013+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.015220+0000) 2019-10-13T07:39:33.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:33.727+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.730978+0000) 2019-10-13T07:39:33.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:33.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.731139+0000) 2019-10-13T07:39:33.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:33.727+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.731658+0000) 2019-10-13T07:39:33.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:33.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:33.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:33.727+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.732799+0000) 2019-10-13T07:39:33.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:33.731+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.733923+0000) 2019-10-13T07:39:33.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:33.731+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.734553+0000) 2019-10-13T07:39:33.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:33.727+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.733296+0000) 2019-10-13T07:39:33.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:33.733+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:33.737542+0000) 2019-10-13T07:39:34.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:34.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:34.263306+0000) 2019-10-13T07:39:37.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:37.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.102603+0000) 2019-10-13T07:39:37.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:37.515+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:37.516920+0000) 2019-10-13T07:39:38.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:38.731+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.735388+0000) 2019-10-13T07:39:38.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:38.731+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.736046+0000) 2019-10-13T07:39:38.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:38.731+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.736719+0000) 2019-10-13T07:39:38.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:38.731+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.736859+0000) 2019-10-13T07:39:38.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:38.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:38.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:38.735+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.737894+0000) 2019-10-13T07:39:38.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:38.733+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.738022+0000) 2019-10-13T07:39:38.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:38.735+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.740081+0000) 2019-10-13T07:39:38.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:38.741+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:38.745962+0000) 2019-10-13T07:39:40.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:40.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:40.644659+0000) 2019-10-13T07:39:40.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:40.739+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:40.743083+0000) 2019-10-13T07:39:41.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:41.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:41.539415+0000) 2019-10-13T07:39:41.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:41.537+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:41.539509+0000) 2019-10-13T07:39:43.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:43.017+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.021194+0000) 2019-10-13T07:39:43.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:43.735+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.740096+0000) 2019-10-13T07:39:43.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:43.735+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.741472+0000) 2019-10-13T07:39:43.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:43.739+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.742168+0000) 2019-10-13T07:39:43.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:43.739+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.742280+0000) 2019-10-13T07:39:43.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:43.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:43.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:43.737+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.742904+0000) 2019-10-13T07:39:43.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:43.739+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.744305+0000) 2019-10-13T07:39:43.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:43.739+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.744213+0000) 2019-10-13T07:39:43.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:43.745+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:43.748513+0000) 2019-10-13T07:39:44.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:44.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:44.263524+0000) 2019-10-13T07:39:47.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:47.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.102824+0000) 2019-10-13T07:39:47.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:47.511+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:47.517131+0000) 2019-10-13T07:39:48.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:48.739+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.745638+0000) 2019-10-13T07:39:48.748 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:48.743+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.746493+0000) 2019-10-13T07:39:48.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:48.743+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.746851+0000) 2019-10-13T07:39:48.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:48.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:48.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:48.741+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.747118+0000) 2019-10-13T07:39:48.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:48.743+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.747264+0000) 2019-10-13T07:39:48.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:48.743+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.747953+0000) 2019-10-13T07:39:48.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:48.743+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.749278+0000) 2019-10-13T07:39:48.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:48.749+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:48.755798+0000) 2019-10-13T07:39:50.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:50.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:50.644853+0000) 2019-10-13T07:39:50.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:50.739+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:50.743290+0000) 2019-10-13T07:39:51.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:51.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:51.539610+0000) 2019-10-13T07:39:51.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:51.533+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:51.539705+0000) 2019-10-13T07:39:53.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:53.025+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.027555+0000) 2019-10-13T07:39:53.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:53.747+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.750643+0000) 2019-10-13T07:39:53.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:53.747+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.751403+0000) 2019-10-13T07:39:53.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:53.747+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.751624+0000) 2019-10-13T07:39:53.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:53.745+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.751979+0000) 2019-10-13T07:39:53.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:53.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:53.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:53.747+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.752982+0000) 2019-10-13T07:39:53.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:53.747+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.753445+0000) 2019-10-13T07:39:53.759 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:53.751+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.754342+0000) 2019-10-13T07:39:53.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:53.753+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:53.758915+0000) 2019-10-13T07:39:54.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:54.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:54.263708+0000) 2019-10-13T07:39:57.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:57.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.103063+0000) 2019-10-13T07:39:57.235 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:39:57.235 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:57.238 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:39:57.239 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:39:57.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:57.511+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:57.517355+0000) 2019-10-13T07:39:58.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:39:58.751+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.756118+0000) 2019-10-13T07:39:58.758 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:39:58.751+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.756819+0000) 2019-10-13T07:39:58.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:39:58.753+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.756553+0000) 2019-10-13T07:39:58.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:39:58.755+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.758038+0000) 2019-10-13T07:39:58.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:39:58.755+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:39:58.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:39:58.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.758458+0000) 2019-10-13T07:39:58.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:39:58.755+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.759269+0000) 2019-10-13T07:39:58.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:39:58.755+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.760052+0000) 2019-10-13T07:39:58.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:39:58.757+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:39:58.763398+0000) 2019-10-13T07:40:00.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:00.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:00.645059+0000) 2019-10-13T07:40:00.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:00.739+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:00.743497+0000) 2019-10-13T07:40:01.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:01.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:01.539803+0000) 2019-10-13T07:40:01.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:01.537+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:01.539914+0000) 2019-10-13T07:40:03.036 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:03.033+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.034093+0000) 2019-10-13T07:40:03.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:03.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.761368+0000) 2019-10-13T07:40:03.763 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:03.759+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.762165+0000) 2019-10-13T07:40:03.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:03.757+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.762121+0000) 2019-10-13T07:40:03.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:03.759+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:03.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:03.759+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.764272+0000) 2019-10-13T07:40:03.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:03.759+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.764581+0000) 2019-10-13T07:40:03.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:03.759+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.764888+0000) 2019-10-13T07:40:03.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:03.759+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.765441+0000) 2019-10-13T07:40:03.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:03.765+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:03.771896+0000) 2019-10-13T07:40:04.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:04.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:04.263946+0000) 2019-10-13T07:40:07.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:07.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.103298+0000) 2019-10-13T07:40:07.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:07.511+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:07.517562+0000) 2019-10-13T07:40:08.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:08.763+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.766756+0000) 2019-10-13T07:40:08.769 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:08.763+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.767418+0000) 2019-10-13T07:40:08.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:08.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:08.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:08.765+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.768660+0000) 2019-10-13T07:40:08.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:08.763+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.769473+0000) 2019-10-13T07:40:08.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:08.767+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.770635+0000) 2019-10-13T07:40:08.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:08.767+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.773322+0000) 2019-10-13T07:40:08.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:08.767+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.772922+0000) 2019-10-13T07:40:08.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:08.773+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:08.779098+0000) 2019-10-13T07:40:10.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:10.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:10.645272+0000) 2019-10-13T07:40:10.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:10.742+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:10.743651+0000) 2019-10-13T07:40:11.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:11.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:11.540069+0000) 2019-10-13T07:40:11.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:11.537+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:11.540131+0000) 2019-10-13T07:40:13.042 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:13.037+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.040605+0000) 2019-10-13T07:40:13.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:13.766+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.773119+0000) 2019-10-13T07:40:13.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:13.766+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.773508+0000) 2019-10-13T07:40:13.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:13.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:13.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:13.769+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.774767+0000) 2019-10-13T07:40:13.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:13.770+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.775924+0000) 2019-10-13T07:40:13.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:13.770+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.776256+0000) 2019-10-13T07:40:13.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:13.770+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.777450+0000) 2019-10-13T07:40:13.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:13.774+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.778307+0000) 2019-10-13T07:40:13.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:13.777+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:13.781823+0000) 2019-10-13T07:40:14.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:14.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:14.264125+0000) 2019-10-13T07:40:17.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:17.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.103496+0000) 2019-10-13T07:40:17.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:17.514+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:17.517809+0000) 2019-10-13T07:40:18.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:18.774+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.778281+0000) 2019-10-13T07:40:18.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:18.774+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.778660+0000) 2019-10-13T07:40:18.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:18.774+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.779106+0000) 2019-10-13T07:40:18.781 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:18.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:18.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:18.774+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.780567+0000) 2019-10-13T07:40:18.783 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:18.774+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.781300+0000) 2019-10-13T07:40:18.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:18.777+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.781269+0000) 2019-10-13T07:40:18.785 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:18.778+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.783444+0000) 2019-10-13T07:40:18.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:18.781+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:18.785015+0000) 2019-10-13T07:40:20.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:20.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:20.645509+0000) 2019-10-13T07:40:20.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:20.742+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:20.743855+0000) 2019-10-13T07:40:21.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:21.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:21.540293+0000) 2019-10-13T07:40:21.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:21.537+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:21.540345+0000) 2019-10-13T07:40:23.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:23.041+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.047062+0000) 2019-10-13T07:40:23.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:23.776+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.783932+0000) 2019-10-13T07:40:23.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:23.778+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.783677+0000) 2019-10-13T07:40:23.789 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:23.778+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.784000+0000) 2019-10-13T07:40:23.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:23.778+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.784870+0000) 2019-10-13T07:40:23.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:23.778+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:23.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:23.782+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.786153+0000) 2019-10-13T07:40:23.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:23.782+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.786390+0000) 2019-10-13T07:40:23.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:23.782+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.787101+0000) 2019-10-13T07:40:23.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:23.784+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:23.791445+0000) 2019-10-13T07:40:24.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:24.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:24.264311+0000) 2019-10-13T07:40:27.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:27.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.103693+0000) 2019-10-13T07:40:27.353 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:40:27.354 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:27.358 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:40:27.358 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:27.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:27.514+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:27.518032+0000) 2019-10-13T07:40:28.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:28.782+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.788718+0000) 2019-10-13T07:40:28.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:28.784+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.789399+0000) 2019-10-13T07:40:28.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:28.786+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.790155+0000) 2019-10-13T07:40:28.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:28.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:28.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:28.786+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.791573+0000) 2019-10-13T07:40:28.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:28.786+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.792089+0000) 2019-10-13T07:40:28.794 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:28.786+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.792258+0000) 2019-10-13T07:40:28.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:28.786+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.792644+0000) 2019-10-13T07:40:28.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:28.792+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:28.797270+0000) 2019-10-13T07:40:30.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:30.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:30.645707+0000) 2019-10-13T07:40:30.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:30.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:30.744085+0000) 2019-10-13T07:40:31.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:31.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:31.540482+0000) 2019-10-13T07:40:31.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:31.536+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:31.540582+0000) 2019-10-13T07:40:33.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:33.048+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.053356+0000) 2019-10-13T07:40:33.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:33.790+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.794143+0000) 2019-10-13T07:40:33.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:33.790+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.795567+0000) 2019-10-13T07:40:33.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:33.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:33.798 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:33.790+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.796424+0000) 2019-10-13T07:40:33.798 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:33.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.796989+0000) 2019-10-13T07:40:33.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:33.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.799223+0000) 2019-10-13T07:40:33.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:33.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.800116+0000) 2019-10-13T07:40:33.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:33.794+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.800905+0000) 2019-10-13T07:40:33.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:33.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:33.803093+0000) 2019-10-13T07:40:34.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:34.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:34.264515+0000) 2019-10-13T07:40:37.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:37.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.103852+0000) 2019-10-13T07:40:37.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:37.514+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:37.518281+0000) 2019-10-13T07:40:38.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:38.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.799826+0000) 2019-10-13T07:40:38.802 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:38.794+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.800934+0000) 2019-10-13T07:40:38.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:38.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:38.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:38.794+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.801462+0000) 2019-10-13T07:40:38.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:38.798+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.801989+0000) 2019-10-13T07:40:38.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:38.798+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.802244+0000) 2019-10-13T07:40:38.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:38.798+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.802696+0000) 2019-10-13T07:40:38.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:38.798+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.803259+0000) 2019-10-13T07:40:38.807 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:38.800+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:38.805740+0000) 2019-10-13T07:40:40.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:40.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:40.645969+0000) 2019-10-13T07:40:40.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:40.742+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:40.744299+0000) 2019-10-13T07:40:41.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:41.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:41.540702+0000) 2019-10-13T07:40:41.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:41.536+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:41.540813+0000) 2019-10-13T07:40:43.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:43.056+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.059756+0000) 2019-10-13T07:40:43.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:43.800+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.804652+0000) 2019-10-13T07:40:43.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:43.802+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.805864+0000) 2019-10-13T07:40:43.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:43.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:43.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:43.802+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.806315+0000) 2019-10-13T07:40:43.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:43.802+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.806533+0000) 2019-10-13T07:40:43.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:43.802+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.807084+0000) 2019-10-13T07:40:43.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:43.802+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.807347+0000) 2019-10-13T07:40:43.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:43.802+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.808345+0000) 2019-10-13T07:40:43.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:43.804+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:43.809935+0000) 2019-10-13T07:40:44.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:44.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:44.264724+0000) 2019-10-13T07:40:47.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:47.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.104039+0000) 2019-10-13T07:40:47.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:47.514+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:47.518482+0000) 2019-10-13T07:40:48.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:48.802+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.809298+0000) 2019-10-13T07:40:48.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:48.806+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.810046+0000) 2019-10-13T07:40:48.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:48.806+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.811010+0000) 2019-10-13T07:40:48.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:48.806+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.811406+0000) 2019-10-13T07:40:48.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:48.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:48.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:48.804+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.811579+0000) 2019-10-13T07:40:48.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:48.806+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.812372+0000) 2019-10-13T07:40:48.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:48.806+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.812744+0000) 2019-10-13T07:40:48.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:48.812+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:48.816982+0000) 2019-10-13T07:40:50.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:50.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:50.646186+0000) 2019-10-13T07:40:50.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:50.742+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:50.744519+0000) 2019-10-13T07:40:51.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:51.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:51.540914+0000) 2019-10-13T07:40:51.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:51.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:51.541008+0000) 2019-10-13T07:40:53.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:53.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.066254+0000) 2019-10-13T07:40:53.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:53.808+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.814395+0000) 2019-10-13T07:40:53.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:53.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:53.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:53.810+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.816501+0000) 2019-10-13T07:40:53.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:53.810+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.817508+0000) 2019-10-13T07:40:53.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:53.814+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.818032+0000) 2019-10-13T07:40:53.820 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:53.814+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.819039+0000) 2019-10-13T07:40:53.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:53.814+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.819749+0000) 2019-10-13T07:40:53.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:53.814+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.820067+0000) 2019-10-13T07:40:53.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:53.816+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:53.822774+0000) 2019-10-13T07:40:54.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:54.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:54.264900+0000) 2019-10-13T07:40:57.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:57.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.104252+0000) 2019-10-13T07:40:57.484 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:40:57.484 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:57.488 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:40:57.488 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:40:57.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:57.514+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:57.518683+0000) 2019-10-13T07:40:58.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:40:58.814+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.819898+0000) 2019-10-13T07:40:58.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:40:58.814+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.820211+0000) 2019-10-13T07:40:58.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:40:58.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:40:58.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:40:58.814+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.820878+0000) 2019-10-13T07:40:58.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:40:58.818+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.821766+0000) 2019-10-13T07:40:58.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:40:58.816+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.822713+0000) 2019-10-13T07:40:58.825 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:40:58.818+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.823528+0000) 2019-10-13T07:40:58.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:40:58.818+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.824564+0000) 2019-10-13T07:40:58.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:40:58.824+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:40:58.829339+0000) 2019-10-13T07:41:00.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:00.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:00.646386+0000) 2019-10-13T07:41:00.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:00.738+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:00.744761+0000) 2019-10-13T07:41:01.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:01.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:01.541140+0000) 2019-10-13T07:41:01.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:01.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:01.541230+0000) 2019-10-13T07:41:03.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:03.072+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.072709+0000) 2019-10-13T07:41:03.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:03.818+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.824006+0000) 2019-10-13T07:41:03.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:03.820+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.826505+0000) 2019-10-13T07:41:03.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:03.824+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.831766+0000) 2019-10-13T07:41:03.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:03.818+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.824978+0000) 2019-10-13T07:41:03.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:03.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:03.838 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:03.822+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.826314+0000) 2019-10-13T07:41:03.838 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:03.822+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.827569+0000) 2019-10-13T07:41:03.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:03.822+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.829094+0000) 2019-10-13T07:41:03.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:03.822+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:03.828340+0000) 2019-10-13T07:41:04.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:04.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:04.265081+0000) 2019-10-13T07:41:07.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:07.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.104451+0000) 2019-10-13T07:41:07.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:07.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:07.518889+0000) 2019-10-13T07:41:08.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:08.824+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.828819+0000) 2019-10-13T07:41:08.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:08.825+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.830275+0000) 2019-10-13T07:41:08.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:08.825+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.830363+0000) 2019-10-13T07:41:08.832 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:08.825+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.830781+0000) 2019-10-13T07:41:08.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:08.825+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.831156+0000) 2019-10-13T07:41:08.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:08.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:08.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:08.825+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.831680+0000) 2019-10-13T07:41:08.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:08.825+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.832051+0000) 2019-10-13T07:41:08.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:08.832+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:08.837727+0000) 2019-10-13T07:41:10.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:10.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:10.646593+0000) 2019-10-13T07:41:10.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:10.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:10.744963+0000) 2019-10-13T07:41:11.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:11.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:11.541361+0000) 2019-10-13T07:41:11.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:11.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:11.541426+0000) 2019-10-13T07:41:13.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:13.076+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.079249+0000) 2019-10-13T07:41:13.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:13.829+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.834343+0000) 2019-10-13T07:41:13.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:13.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.834661+0000) 2019-10-13T07:41:13.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:13.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.834897+0000) 2019-10-13T07:41:13.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:13.829+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.835280+0000) 2019-10-13T07:41:13.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:13.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:13.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:13.829+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.837581+0000) 2019-10-13T07:41:13.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:13.833+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.838066+0000) 2019-10-13T07:41:13.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:13.833+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.840079+0000) 2019-10-13T07:41:13.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:13.836+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:13.843083+0000) 2019-10-13T07:41:14.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:14.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:14.265273+0000) 2019-10-13T07:41:17.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:17.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.104667+0000) 2019-10-13T07:41:17.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:17.513+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:17.519086+0000) 2019-10-13T07:41:18.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:18.833+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.839133+0000) 2019-10-13T07:41:18.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:18.833+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.839927+0000) 2019-10-13T07:41:18.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:18.833+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.839438+0000) 2019-10-13T07:41:18.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:18.833+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.840961+0000) 2019-10-13T07:41:18.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:18.836+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.840835+0000) 2019-10-13T07:41:18.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:18.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:18.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:18.837+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.842018+0000) 2019-10-13T07:41:18.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:18.837+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.842545+0000) 2019-10-13T07:41:18.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:18.840+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:18.845901+0000) 2019-10-13T07:41:20.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:20.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:20.646774+0000) 2019-10-13T07:41:20.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:20.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:20.745159+0000) 2019-10-13T07:41:21.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:21.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:21.541649+0000) 2019-10-13T07:41:21.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:21.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:21.541588+0000) 2019-10-13T07:41:23.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:23.084+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.085750+0000) 2019-10-13T07:41:23.846 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:23.841+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.844584+0000) 2019-10-13T07:41:23.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:23.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.844985+0000) 2019-10-13T07:41:23.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:23.845+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56306 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:23.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:23.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.846376+0000) 2019-10-13T07:41:23.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:23.845+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.847485+0000) 2019-10-13T07:41:23.850 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:23.845+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.848244+0000) 2019-10-13T07:41:23.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:23.845+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.849204+0000) 2019-10-13T07:41:23.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:23.845+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.849516+0000) 2019-10-13T07:41:23.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:23.844+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:23.850864+0000) 2019-10-13T07:41:24.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:24.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:24.265463+0000) 2019-10-13T07:41:27.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:27.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.104870+0000) 2019-10-13T07:41:27.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:27.517+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:27.519292+0000) 2019-10-13T07:41:27.545 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:41:27.545 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:27.552 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:41:27.552 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:28.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:28.845+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.849425+0000) 2019-10-13T07:41:28.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:28.849+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.850340+0000) 2019-10-13T07:41:28.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:28.848+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.850203+0000) 2019-10-13T07:41:28.853 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:28.849+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.851170+0000) 2019-10-13T07:41:28.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:28.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:28.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:28.849+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.851557+0000) 2019-10-13T07:41:28.854 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:28.849+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.852026+0000) 2019-10-13T07:41:28.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:28.849+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.852324+0000) 2019-10-13T07:41:28.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:28.856+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:28.859823+0000) 2019-10-13T07:41:30.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:30.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:30.646955+0000) 2019-10-13T07:41:30.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:30.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:30.745357+0000) 2019-10-13T07:41:31.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:31.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:31.541781+0000) 2019-10-13T07:41:31.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:31.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:31.541852+0000) 2019-10-13T07:41:33.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:33.091+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.092126+0000) 2019-10-13T07:41:33.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:33.851+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.854212+0000) 2019-10-13T07:41:33.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:33.853+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.855315+0000) 2019-10-13T07:41:33.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:33.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:33.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:33.857+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.858089+0000) 2019-10-13T07:41:33.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:33.857+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.858505+0000) 2019-10-13T07:41:33.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:33.857+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.859201+0000) 2019-10-13T07:41:33.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:33.857+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.859968+0000) 2019-10-13T07:41:33.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:33.857+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.859783+0000) 2019-10-13T07:41:33.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:33.863+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:33.866381+0000) 2019-10-13T07:41:34.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:34.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:34.265638+0000) 2019-10-13T07:41:37.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:37.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.105102+0000) 2019-10-13T07:41:37.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:37.517+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:37.519529+0000) 2019-10-13T07:41:38.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:38.855+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.859589+0000) 2019-10-13T07:41:38.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:38.857+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.861098+0000) 2019-10-13T07:41:38.862 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:38.857+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.861148+0000) 2019-10-13T07:41:38.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:38.857+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.861588+0000) 2019-10-13T07:41:38.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:38.861+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.862001+0000) 2019-10-13T07:41:38.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:38.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:38.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:38.861+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.863878+0000) 2019-10-13T07:41:38.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:38.861+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.864225+0000) 2019-10-13T07:41:38.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:38.867+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:38.869105+0000) 2019-10-13T07:41:40.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:40.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:40.647169+0000) 2019-10-13T07:41:40.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:40.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:40.745585+0000) 2019-10-13T07:41:41.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:41.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:41.541972+0000) 2019-10-13T07:41:41.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:41.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:41.542047+0000) 2019-10-13T07:41:43.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:43.095+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.098694+0000) 2019-10-13T07:41:43.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:43.865+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.866142+0000) 2019-10-13T07:41:43.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:43.865+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.867071+0000) 2019-10-13T07:41:43.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:43.863+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.866973+0000) 2019-10-13T07:41:43.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:43.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:43.870 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:43.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.868018+0000) 2019-10-13T07:41:43.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:43.865+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.868141+0000) 2019-10-13T07:41:43.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:43.865+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.868460+0000) 2019-10-13T07:41:43.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:43.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.868673+0000) 2019-10-13T07:41:43.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:43.871+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:43.875135+0000) 2019-10-13T07:41:44.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:44.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:44.265820+0000) 2019-10-13T07:41:47.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:47.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.105314+0000) 2019-10-13T07:41:47.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:47.517+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:47.519727+0000) 2019-10-13T07:41:48.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:48.869+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.871357+0000) 2019-10-13T07:41:48.873 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:48.869+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.871764+0000) 2019-10-13T07:41:48.874 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:48.869+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.872098+0000) 2019-10-13T07:41:48.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:48.869+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.872758+0000) 2019-10-13T07:41:48.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:48.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:48.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:48.869+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.873380+0000) 2019-10-13T07:41:48.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:48.871+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.873836+0000) 2019-10-13T07:41:48.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:48.873+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.874984+0000) 2019-10-13T07:41:48.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:48.875+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:48.879825+0000) 2019-10-13T07:41:50.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:50.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:50.647360+0000) 2019-10-13T07:41:50.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:50.745+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:50.745781+0000) 2019-10-13T07:41:51.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:51.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:51.542193+0000) 2019-10-13T07:41:51.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:51.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:51.542242+0000) 2019-10-13T07:41:53.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:53.103+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.105087+0000) 2019-10-13T07:41:53.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:53.873+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.875996+0000) 2019-10-13T07:41:53.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:53.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:53.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:53.877+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.878252+0000) 2019-10-13T07:41:53.880 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:53.877+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.878773+0000) 2019-10-13T07:41:53.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:53.877+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.879016+0000) 2019-10-13T07:41:53.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:53.877+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.879639+0000) 2019-10-13T07:41:53.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:53.875+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.879904+0000) 2019-10-13T07:41:53.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:53.877+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.880816+0000) 2019-10-13T07:41:53.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:53.883+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:53.884518+0000) 2019-10-13T07:41:54.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:54.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:54.265999+0000) 2019-10-13T07:41:57.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:57.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.105491+0000) 2019-10-13T07:41:57.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:57.517+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:57.519947+0000) 2019-10-13T07:41:57.660 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:41:57.660 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:57.664 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:41:57.664 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:41:58.883 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:41:58.877+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.881452+0000) 2019-10-13T07:41:58.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:41:58.881+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.882134+0000) 2019-10-13T07:41:58.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:41:58.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:41:58.885 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:41:58.881+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.883247+0000) 2019-10-13T07:41:58.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:41:58.881+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.883793+0000) 2019-10-13T07:41:58.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:41:58.879+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.883741+0000) 2019-10-13T07:41:58.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:41:58.881+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.884580+0000) 2019-10-13T07:41:58.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:41:58.885+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.885871+0000) 2019-10-13T07:41:58.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:41:58.887+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:41:58.891281+0000) 2019-10-13T07:42:00.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:00.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:00.647591+0000) 2019-10-13T07:42:00.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:00.745+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:00.745980+0000) 2019-10-13T07:42:01.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:01.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:01.542440+0000) 2019-10-13T07:42:01.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:01.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:01.542381+0000) 2019-10-13T07:42:03.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:03.111+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.111611+0000) 2019-10-13T07:42:03.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:03.884+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.885906+0000) 2019-10-13T07:42:03.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:03.884+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.887544+0000) 2019-10-13T07:42:03.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:03.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:03.889 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:03.884+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.887892+0000) 2019-10-13T07:42:03.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:03.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.888128+0000) 2019-10-13T07:42:03.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:03.887+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.888414+0000) 2019-10-13T07:42:03.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:03.884+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.888695+0000) 2019-10-13T07:42:03.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:03.884+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.889460+0000) 2019-10-13T07:42:03.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:03.891+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:03.892581+0000) 2019-10-13T07:42:04.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:04.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:04.266206+0000) 2019-10-13T07:42:07.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:07.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.105695+0000) 2019-10-13T07:42:07.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:07.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:07.520185+0000) 2019-10-13T07:42:08.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:08.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56583 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:08.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:08.888+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.892636+0000) 2019-10-13T07:42:08.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:08.891+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.892562+0000) 2019-10-13T07:42:08.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:08.888+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.893383+0000) 2019-10-13T07:42:08.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:08.892+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.893785+0000) 2019-10-13T07:42:08.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:08.892+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.894481+0000) 2019-10-13T07:42:08.897 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:08.892+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.895463+0000) 2019-10-13T07:42:08.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:08.892+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.895873+0000) 2019-10-13T07:42:08.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:08.899+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:08.903068+0000) 2019-10-13T07:42:10.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:10.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:10.647782+0000) 2019-10-13T07:42:10.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:10.744+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:10.746179+0000) 2019-10-13T07:42:11.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:11.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:11.542572+0000) 2019-10-13T07:42:11.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:11.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:11.542635+0000) 2019-10-13T07:42:13.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:13.115+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.118097+0000) 2019-10-13T07:42:13.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:13.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:13.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:13.892+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.897533+0000) 2019-10-13T07:42:13.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:13.895+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.897600+0000) 2019-10-13T07:42:13.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:13.896+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.898781+0000) 2019-10-13T07:42:13.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:13.896+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.899306+0000) 2019-10-13T07:42:13.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:13.896+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.899586+0000) 2019-10-13T07:42:13.903 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:13.896+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.899972+0000) 2019-10-13T07:42:13.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:13.896+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.900353+0000) 2019-10-13T07:42:13.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:13.903+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:13.904737+0000) 2019-10-13T07:42:14.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:14.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:14.266390+0000) 2019-10-13T07:42:17.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:17.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.105964+0000) 2019-10-13T07:42:17.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:17.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:17.520383+0000) 2019-10-13T07:42:18.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:18.900+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.902372+0000) 2019-10-13T07:42:18.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:18.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:18.904 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:18.900+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.902703+0000) 2019-10-13T07:42:18.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:18.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.903147+0000) 2019-10-13T07:42:18.905 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:18.900+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.903265+0000) 2019-10-13T07:42:18.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:18.899+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.904013+0000) 2019-10-13T07:42:18.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:18.904+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.907010+0000) 2019-10-13T07:42:18.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:18.900+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.904102+0000) 2019-10-13T07:42:18.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:18.907+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:18.910461+0000) 2019-10-13T07:42:20.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:20.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:20.647995+0000) 2019-10-13T07:42:20.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:20.744+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:20.746378+0000) 2019-10-13T07:42:21.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:21.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:21.542810+0000) 2019-10-13T07:42:21.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:21.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:21.542832+0000) 2019-10-13T07:42:23.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:23.123+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.124485+0000) 2019-10-13T07:42:23.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:23.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:23.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:23.904+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.907180+0000) 2019-10-13T07:42:23.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:23.904+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.908125+0000) 2019-10-13T07:42:23.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:23.903+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.907831+0000) 2019-10-13T07:42:23.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:23.904+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.908907+0000) 2019-10-13T07:42:23.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:23.904+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.909276+0000) 2019-10-13T07:42:23.911 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:23.904+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.909572+0000) 2019-10-13T07:42:23.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:23.908+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.910385+0000) 2019-10-13T07:42:23.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:23.915+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:23.916691+0000) 2019-10-13T07:42:24.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:24.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:24.266562+0000) 2019-10-13T07:42:27.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:27.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:27.106198+0000) 2019-10-13T07:42:27.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:27.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:27.520609+0000) 2019-10-13T07:42:27.782 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:42:27.783 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:27.787 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:42:27.787 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:28.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:28.908+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.910478+0000) 2019-10-13T07:42:28.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:28.908+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.911507+0000) 2019-10-13T07:42:28.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:28.908+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.911824+0000) 2019-10-13T07:42:28.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:28.908+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.912322+0000) 2019-10-13T07:42:28.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:28.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:28.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:28.908+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.912650+0000) 2019-10-13T07:42:28.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:28.908+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.913307+0000) 2019-10-13T07:42:28.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:28.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.913332+0000) 2019-10-13T07:42:28.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:28.919+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:28.921535+0000) 2019-10-13T07:42:30.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:30.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:30.648189+0000) 2019-10-13T07:42:30.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:30.744+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:30.746574+0000) 2019-10-13T07:42:31.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:31.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:31.543004+0000) 2019-10-13T07:42:31.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:31.539+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:31.543028+0000) 2019-10-13T07:42:33.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:33.127+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.130942+0000) 2019-10-13T07:42:33.917 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:33.912+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.916009+0000) 2019-10-13T07:42:33.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:33.911+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.915968+0000) 2019-10-13T07:42:33.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:33.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.917235+0000) 2019-10-13T07:42:33.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:33.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:33.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:33.916+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.917970+0000) 2019-10-13T07:42:33.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:33.916+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.918256+0000) 2019-10-13T07:42:33.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:33.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.919494+0000) 2019-10-13T07:42:33.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:33.912+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.917441+0000) 2019-10-13T07:42:33.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:33.919+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:33.923540+0000) 2019-10-13T07:42:34.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:34.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:34.266757+0000) 2019-10-13T07:42:37.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:37.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:37.106418+0000) 2019-10-13T07:42:37.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:37.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:37.520810+0000) 2019-10-13T07:42:38.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:38.916+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.920985+0000) 2019-10-13T07:42:38.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:38.918+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.921123+0000) 2019-10-13T07:42:38.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:38.920+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:38.924 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:38.920+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.922373+0000) 2019-10-13T07:42:38.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:38.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.922891+0000) 2019-10-13T07:42:38.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:38.920+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.923319+0000) 2019-10-13T07:42:38.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:38.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.923814+0000) 2019-10-13T07:42:38.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:38.920+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.924495+0000) 2019-10-13T07:42:38.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:38.926+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:38.931138+0000) 2019-10-13T07:42:40.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:40.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:40.648383+0000) 2019-10-13T07:42:40.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:40.744+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:40.746773+0000) 2019-10-13T07:42:41.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:41.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:41.543205+0000) 2019-10-13T07:42:41.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:41.538+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:41.543228+0000) 2019-10-13T07:42:43.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:43.134+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.137419+0000) 2019-10-13T07:42:43.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:43.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:43.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:43.924+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.926981+0000) 2019-10-13T07:42:43.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:43.924+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.927297+0000) 2019-10-13T07:42:43.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:43.924+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.927787+0000) 2019-10-13T07:42:43.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:43.924+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.928035+0000) 2019-10-13T07:42:43.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:43.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.927973+0000) 2019-10-13T07:42:43.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:43.924+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.929056+0000) 2019-10-13T07:42:43.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:43.928+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.930084+0000) 2019-10-13T07:42:43.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:43.930+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:43.935329+0000) 2019-10-13T07:42:44.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:44.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:44.266941+0000) 2019-10-13T07:42:47.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:47.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:47.106655+0000) 2019-10-13T07:42:47.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:47.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:47.521011+0000) 2019-10-13T07:42:48.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:48.926+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.930003+0000) 2019-10-13T07:42:48.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:48.928+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.931336+0000) 2019-10-13T07:42:48.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:48.928+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.931714+0000) 2019-10-13T07:42:48.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:48.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:48.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:48.928+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.932281+0000) 2019-10-13T07:42:48.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:48.928+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.933025+0000) 2019-10-13T07:42:48.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:48.928+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.932683+0000) 2019-10-13T07:42:48.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:48.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.933786+0000) 2019-10-13T07:42:48.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:48.934+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:48.937868+0000) 2019-10-13T07:42:50.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:50.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:50.648589+0000) 2019-10-13T07:42:50.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:50.744+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:50.746968+0000) 2019-10-13T07:42:51.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:51.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:51.543428+0000) 2019-10-13T07:42:51.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:51.538+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:51.543425+0000) 2019-10-13T07:42:53.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:53.142+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.143949+0000) 2019-10-13T07:42:53.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:53.930+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.934591+0000) 2019-10-13T07:42:53.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:53.932+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.935162+0000) 2019-10-13T07:42:53.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:53.932+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.935681+0000) 2019-10-13T07:42:53.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:53.932+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.936953+0000) 2019-10-13T07:42:53.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:53.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:53.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:53.936+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.937834+0000) 2019-10-13T07:42:53.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:53.936+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.938195+0000) 2019-10-13T07:42:53.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:53.936+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.938923+0000) 2019-10-13T07:42:53.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:53.938+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:53.943289+0000) 2019-10-13T07:42:54.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:54.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:54.267119+0000) 2019-10-13T07:42:57.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:57.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:57.106857+0000) 2019-10-13T07:42:57.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:57.516+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:57.521249+0000) 2019-10-13T07:42:57.889 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:42:57.889 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:57.892 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:42:57.893 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:42:58.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:42:58.936+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.940124+0000) 2019-10-13T07:42:58.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:42:58.936+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.941532+0000) 2019-10-13T07:42:58.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:42:58.936+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.941700+0000) 2019-10-13T07:42:58.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:42:58.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:42:58.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:42:58.940+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.942049+0000) 2019-10-13T07:42:58.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:42:58.938+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.942508+0000) 2019-10-13T07:42:58.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:42:58.940+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.943408+0000) 2019-10-13T07:42:58.946 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:42:58.940+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.943762+0000) 2019-10-13T07:42:58.951 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:42:58.946+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:42:58.949499+0000) 2019-10-13T07:43:00.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:00.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:00.648778+0000) 2019-10-13T07:43:00.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:00.743+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:00.747165+0000) 2019-10-13T07:43:01.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:01.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:01.543655+0000) 2019-10-13T07:43:01.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:01.538+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:01.543625+0000) 2019-10-13T07:43:03.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:03.146+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.150367+0000) 2019-10-13T07:43:03.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:03.943+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.945452+0000) 2019-10-13T07:43:03.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:03.943+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.946909+0000) 2019-10-13T07:43:03.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:03.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56924 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:03.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:03.943+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.947293+0000) 2019-10-13T07:43:03.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:03.943+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.947669+0000) 2019-10-13T07:43:03.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:03.943+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.948144+0000) 2019-10-13T07:43:03.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:03.946+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.948400+0000) 2019-10-13T07:43:03.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:03.943+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.949033+0000) 2019-10-13T07:43:03.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:03.950+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:03.955079+0000) 2019-10-13T07:43:04.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:04.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:04.267353+0000) 2019-10-13T07:43:07.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:07.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:07.107053+0000) 2019-10-13T07:43:07.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:07.515+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:07.521443+0000) 2019-10-13T07:43:08.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:08.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.950148+0000) 2019-10-13T07:43:08.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:08.947+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.951348+0000) 2019-10-13T07:43:08.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:08.947+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.951614+0000) 2019-10-13T07:43:08.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:08.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:08.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:08.947+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.951986+0000) 2019-10-13T07:43:08.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:08.947+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.952811+0000) 2019-10-13T07:43:08.955 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:08.947+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.953070+0000) 2019-10-13T07:43:08.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:08.947+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.953644+0000) 2019-10-13T07:43:08.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:08.954+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:08.959602+0000) 2019-10-13T07:43:10.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:10.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:10.648986+0000) 2019-10-13T07:43:10.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:10.743+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:10.747314+0000) 2019-10-13T07:43:11.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:11.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:11.543842+0000) 2019-10-13T07:43:11.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:11.538+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:11.543843+0000) 2019-10-13T07:43:13.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:13.154+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.157811+0000) 2019-10-13T07:43:13.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:13.951+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.955891+0000) 2019-10-13T07:43:13.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:13.951+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.956717+0000) 2019-10-13T07:43:13.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:13.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:13.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:13.954+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.956688+0000) 2019-10-13T07:43:13.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:13.951+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.957632+0000) 2019-10-13T07:43:13.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:13.955+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.958253+0000) 2019-10-13T07:43:13.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:13.955+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.958549+0000) 2019-10-13T07:43:13.961 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:13.955+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.959201+0000) 2019-10-13T07:43:13.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:13.958+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:13.963600+0000) 2019-10-13T07:43:14.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:14.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:14.267563+0000) 2019-10-13T07:43:17.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:17.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:17.107283+0000) 2019-10-13T07:43:17.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:17.515+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:17.521604+0000) 2019-10-13T07:43:18.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:18.955+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.960439+0000) 2019-10-13T07:43:18.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:18.955+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.960832+0000) 2019-10-13T07:43:18.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:18.955+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.961497+0000) 2019-10-13T07:43:18.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:18.958+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.961434+0000) 2019-10-13T07:43:18.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:18.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:18.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:18.959+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.962399+0000) 2019-10-13T07:43:18.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:18.959+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.962973+0000) 2019-10-13T07:43:18.965 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:18.959+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.963208+0000) 2019-10-13T07:43:18.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:18.966+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:18.970050+0000) 2019-10-13T07:43:20.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:20.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:20.649208+0000) 2019-10-13T07:43:20.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:20.743+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:20.747461+0000) 2019-10-13T07:43:21.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:21.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:21.544040+0000) 2019-10-13T07:43:21.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:21.538+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:21.544054+0000) 2019-10-13T07:43:23.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:23.162+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.164157+0000) 2019-10-13T07:43:23.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:23.959+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.965613+0000) 2019-10-13T07:43:23.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:23.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:23.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:23.963+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.967798+0000) 2019-10-13T07:43:23.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:23.963+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.968160+0000) 2019-10-13T07:43:23.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:23.966+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.968509+0000) 2019-10-13T07:43:23.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:23.963+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.968581+0000) 2019-10-13T07:43:23.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:23.967+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.969801+0000) 2019-10-13T07:43:23.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:23.967+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.970085+0000) 2019-10-13T07:43:23.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:23.970+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:23.973395+0000) 2019-10-13T07:43:24.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:24.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:24.267776+0000) 2019-10-13T07:43:27.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:27.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:27.107464+0000) 2019-10-13T07:43:27.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:27.519+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:27.521827+0000) 2019-10-13T07:43:28.045 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:43:28.046 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:28.049 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:43:28.050 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:28.972 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:28.967+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.970826+0000) 2019-10-13T07:43:28.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:28.966+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.970628+0000) 2019-10-13T07:43:28.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:28.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:28.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:28.967+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.972306+0000) 2019-10-13T07:43:28.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:28.967+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.972969+0000) 2019-10-13T07:43:28.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:28.967+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.973411+0000) 2019-10-13T07:43:28.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:28.967+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.972622+0000) 2019-10-13T07:43:28.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:28.971+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.974562+0000) 2019-10-13T07:43:28.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:28.978+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:28.980557+0000) 2019-10-13T07:43:30.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:30.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:30.649373+0000) 2019-10-13T07:43:30.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:30.743+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:30.747666+0000) 2019-10-13T07:43:31.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:31.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:31.544257+0000) 2019-10-13T07:43:31.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:31.542+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:31.544285+0000) 2019-10-13T07:43:33.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:33.166+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.170584+0000) 2019-10-13T07:43:33.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:33.971+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.975311+0000) 2019-10-13T07:43:33.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:33.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:33.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:33.971+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.977461+0000) 2019-10-13T07:43:33.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:33.975+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.977821+0000) 2019-10-13T07:43:33.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:33.974+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.977733+0000) 2019-10-13T07:43:33.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:33.975+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.978605+0000) 2019-10-13T07:43:33.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:33.975+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.978941+0000) 2019-10-13T07:43:33.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:33.975+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.980294+0000) 2019-10-13T07:43:33.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:33.978+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:33.981511+0000) 2019-10-13T07:43:34.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:34.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:34.267988+0000) 2019-10-13T07:43:37.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:37.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:37.107682+0000) 2019-10-13T07:43:37.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:37.519+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:37.522052+0000) 2019-10-13T07:43:38.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:38.978+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.980328+0000) 2019-10-13T07:43:38.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:38.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:38.984 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:38.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.982382+0000) 2019-10-13T07:43:38.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:38.979+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.982772+0000) 2019-10-13T07:43:38.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:38.979+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.983095+0000) 2019-10-13T07:43:38.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:38.979+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.983592+0000) 2019-10-13T07:43:38.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:38.979+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.983774+0000) 2019-10-13T07:43:38.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:38.979+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.984834+0000) 2019-10-13T07:43:38.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:38.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:38.987083+0000) 2019-10-13T07:43:40.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:40.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:40.649551+0000) 2019-10-13T07:43:40.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:40.747+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:40.747837+0000) 2019-10-13T07:43:41.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:41.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:41.544448+0000) 2019-10-13T07:43:41.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:41.542+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:41.544479+0000) 2019-10-13T07:43:43.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:43.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.177057+0000) 2019-10-13T07:43:43.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:43.983+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.985876+0000) 2019-10-13T07:43:43.987 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:43.983+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.986170+0000) 2019-10-13T07:43:43.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:43.983+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.986775+0000) 2019-10-13T07:43:43.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:43.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57173 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:43.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:43.983+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.987520+0000) 2019-10-13T07:43:43.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:43.981+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.987654+0000) 2019-10-13T07:43:43.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:43.983+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.988463+0000) 2019-10-13T07:43:43.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:43.987+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.990328+0000) 2019-10-13T07:43:43.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:43.989+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:43.992867+0000) 2019-10-13T07:43:44.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:44.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:44.268172+0000) 2019-10-13T07:43:47.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:47.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:47.107879+0000) 2019-10-13T07:43:47.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:47.519+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:47.522252+0000) 2019-10-13T07:43:48.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:48.987+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.990900+0000) 2019-10-13T07:43:48.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:48.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:48.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:48.989+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.992422+0000) 2019-10-13T07:43:48.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:48.987+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.993276+0000) 2019-10-13T07:43:48.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:48.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.994037+0000) 2019-10-13T07:43:48.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:48.991+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.994465+0000) 2019-10-13T07:43:48.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:48.991+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.995216+0000) 2019-10-13T07:43:48.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:48.987+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.993591+0000) 2019-10-13T07:43:49.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:48.993+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:48.998699+0000) 2019-10-13T07:43:50.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:50.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:50.649773+0000) 2019-10-13T07:43:50.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:50.743+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:50.748028+0000) 2019-10-13T07:43:51.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:51.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:51.544638+0000) 2019-10-13T07:43:51.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:51.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:51.544695+0000) 2019-10-13T07:43:53.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:53.177+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.183578+0000) 2019-10-13T07:43:53.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:53.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.995694+0000) 2019-10-13T07:43:53.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:53.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:53.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:53.991+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.997368+0000) 2019-10-13T07:43:53.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:53.991+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.997712+0000) 2019-10-13T07:43:54.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:53.995+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.999029+0000) 2019-10-13T07:43:54.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:53.993+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:53.999477+0000) 2019-10-13T07:43:54.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:53.995+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:54.000546+0000) 2019-10-13T07:43:54.002 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:53.995+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:54.000794+0000) 2019-10-13T07:43:54.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:54.001+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:54.005357+0000) 2019-10-13T07:43:54.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:54.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:54.268363+0000) 2019-10-13T07:43:57.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:57.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:57.108093+0000) 2019-10-13T07:43:57.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:57.518+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:57.522448+0000) 2019-10-13T07:43:58.185 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:43:58.185 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:58.189 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:43:58.189 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:43:59.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:43:58.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:43:59.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:43:58.998+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.002295+0000) 2019-10-13T07:43:59.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:43:58.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.003096+0000) 2019-10-13T07:43:59.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:43:58.998+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.003576+0000) 2019-10-13T07:43:59.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:43:58.998+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.004012+0000) 2019-10-13T07:43:59.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:43:59.001+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.004488+0000) 2019-10-13T07:43:59.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:43:58.998+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.004835+0000) 2019-10-13T07:43:59.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:43:58.998+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.005372+0000) 2019-10-13T07:43:59.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:43:59.005+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:43:59.008653+0000) 2019-10-13T07:44:00.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:00.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:00.649954+0000) 2019-10-13T07:44:00.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:00.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:00.748226+0000) 2019-10-13T07:44:01.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:01.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:01.544834+0000) 2019-10-13T07:44:01.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:01.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:01.544943+0000) 2019-10-13T07:44:03.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:03.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:03.189851+0000) 2019-10-13T07:44:04.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:03.998+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.005571+0000) 2019-10-13T07:44:04.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:04.002+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.005971+0000) 2019-10-13T07:44:04.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:04.002+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.006285+0000) 2019-10-13T07:44:04.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:04.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57298 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:04.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:04.002+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.007632+0000) 2019-10-13T07:44:04.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:04.001+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.006967+0000) 2019-10-13T07:44:04.010 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:04.002+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.008476+0000) 2019-10-13T07:44:04.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:04.002+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.008126+0000) 2019-10-13T07:44:04.014 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:04.009+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.013007+0000) 2019-10-13T07:44:04.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:04.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:04.268547+0000) 2019-10-13T07:44:07.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:07.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:07.108291+0000) 2019-10-13T07:44:07.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:07.518+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:07.522653+0000) 2019-10-13T07:44:09.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:09.006+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.010738+0000) 2019-10-13T07:44:09.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:09.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:09.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:09.009+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.012208+0000) 2019-10-13T07:44:09.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:09.006+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.012955+0000) 2019-10-13T07:44:09.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:09.010+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.014026+0000) 2019-10-13T07:44:09.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:09.010+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.014339+0000) 2019-10-13T07:44:09.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:09.010+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.014809+0000) 2019-10-13T07:44:09.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:09.010+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.014770+0000) 2019-10-13T07:44:09.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:09.013+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:09.019044+0000) 2019-10-13T07:44:10.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:10.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:10.650107+0000) 2019-10-13T07:44:10.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:10.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:10.748404+0000) 2019-10-13T07:44:11.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:11.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:11.545020+0000) 2019-10-13T07:44:11.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:11.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:11.545160+0000) 2019-10-13T07:44:13.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:13.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:13.196440+0000) 2019-10-13T07:44:14.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:14.009+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.015077+0000) 2019-10-13T07:44:14.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:14.010+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.016273+0000) 2019-10-13T07:44:14.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:14.010+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.016675+0000) 2019-10-13T07:44:14.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:14.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:14.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:14.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.017898+0000) 2019-10-13T07:44:14.020 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:14.014+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.018872+0000) 2019-10-13T07:44:14.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:14.014+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.019624+0000) 2019-10-13T07:44:14.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:14.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.020080+0000) 2019-10-13T07:44:14.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:14.021+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.024140+0000) 2019-10-13T07:44:14.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:14.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:14.268747+0000) 2019-10-13T07:44:17.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:17.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:17.108503+0000) 2019-10-13T07:44:17.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:17.518+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:17.522864+0000) 2019-10-13T07:44:19.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:19.013+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.019530+0000) 2019-10-13T07:44:19.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:19.014+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.021019+0000) 2019-10-13T07:44:19.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:19.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:19.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:19.018+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.024078+0000) 2019-10-13T07:44:19.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:19.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.024436+0000) 2019-10-13T07:44:19.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:19.018+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.024973+0000) 2019-10-13T07:44:19.027 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:19.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.024998+0000) 2019-10-13T07:44:19.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:19.018+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.025368+0000) 2019-10-13T07:44:19.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:19.025+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:19.029238+0000) 2019-10-13T07:44:20.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:20.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:20.650290+0000) 2019-10-13T07:44:20.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:20.742+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:20.748613+0000) 2019-10-13T07:44:21.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:21.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:21.545214+0000) 2019-10-13T07:44:21.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:21.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:21.545382+0000) 2019-10-13T07:44:23.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:23.197+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:23.202887+0000) 2019-10-13T07:44:24.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:24.022+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.026409+0000) 2019-10-13T07:44:24.028 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:24.022+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.026994+0000) 2019-10-13T07:44:24.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:24.022+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.027369+0000) 2019-10-13T07:44:24.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:24.022+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.028398+0000) 2019-10-13T07:44:24.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:24.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:24.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:24.022+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.029337+0000) 2019-10-13T07:44:24.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:24.025+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.029118+0000) 2019-10-13T07:44:24.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:24.026+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.030643+0000) 2019-10-13T07:44:24.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:24.029+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.035453+0000) 2019-10-13T07:44:24.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:24.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:24.268946+0000) 2019-10-13T07:44:27.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:27.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:27.108762+0000) 2019-10-13T07:44:27.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:27.522+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:27.523066+0000) 2019-10-13T07:44:28.298 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:44:28.298 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:28.302 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:44:28.302 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:29.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:29.026+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.031952+0000) 2019-10-13T07:44:29.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:29.025+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.032067+0000) 2019-10-13T07:44:29.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:29.026+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.033686+0000) 2019-10-13T07:44:29.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:29.026+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.033687+0000) 2019-10-13T07:44:29.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:29.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:29.036 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:29.030+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.034963+0000) 2019-10-13T07:44:29.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:29.030+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.034474+0000) 2019-10-13T07:44:29.038 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:29.030+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.035542+0000) 2019-10-13T07:44:29.042 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:29.037+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:29.040720+0000) 2019-10-13T07:44:30.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:30.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:30.650461+0000) 2019-10-13T07:44:30.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:30.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:30.748814+0000) 2019-10-13T07:44:31.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:31.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:31.545407+0000) 2019-10-13T07:44:31.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:31.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:31.545578+0000) 2019-10-13T07:44:33.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:33.209+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:33.209438+0000) 2019-10-13T07:44:34.039 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:34.030+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.037057+0000) 2019-10-13T07:44:34.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:34.030+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.037442+0000) 2019-10-13T07:44:34.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:34.034+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.038309+0000) 2019-10-13T07:44:34.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:34.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:34.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:34.034+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.039755+0000) 2019-10-13T07:44:34.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:34.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.039752+0000) 2019-10-13T07:44:34.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:34.037+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.040581+0000) 2019-10-13T07:44:34.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:34.038+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.042066+0000) 2019-10-13T07:44:34.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:34.041+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.044605+0000) 2019-10-13T07:44:34.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:34.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:34.269128+0000) 2019-10-13T07:44:37.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:37.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:37.108969+0000) 2019-10-13T07:44:37.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:37.522+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:37.523298+0000) 2019-10-13T07:44:39.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:39.038+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.042757+0000) 2019-10-13T07:44:39.045 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:39.038+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.043487+0000) 2019-10-13T07:44:39.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:39.038+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.044575+0000) 2019-10-13T07:44:39.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:39.038+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.044455+0000) 2019-10-13T07:44:39.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:39.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:39.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:39.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.044950+0000) 2019-10-13T07:44:39.048 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:39.041+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.045689+0000) 2019-10-13T07:44:39.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:39.038+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.045670+0000) 2019-10-13T07:44:39.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:39.045+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:39.049788+0000) 2019-10-13T07:44:40.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:40.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:40.650648+0000) 2019-10-13T07:44:40.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:40.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:40.749007+0000) 2019-10-13T07:44:41.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:41.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:41.545635+0000) 2019-10-13T07:44:41.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:41.541+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:41.545765+0000) 2019-10-13T07:44:43.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:43.213+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:43.215788+0000) 2019-10-13T07:44:44.050 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:44.042+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.048486+0000) 2019-10-13T07:44:44.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:44.042+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.048792+0000) 2019-10-13T07:44:44.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:44.042+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.049187+0000) 2019-10-13T07:44:44.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:44.042+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.049681+0000) 2019-10-13T07:44:44.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:44.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57546 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:44.052 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:44.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.050472+0000) 2019-10-13T07:44:44.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:44.046+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.050959+0000) 2019-10-13T07:44:44.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:44.045+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.051642+0000) 2019-10-13T07:44:44.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:44.049+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.056073+0000) 2019-10-13T07:44:44.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:44.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:44.269264+0000) 2019-10-13T07:44:47.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:47.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:47.109214+0000) 2019-10-13T07:44:47.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:47.522+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:47.523509+0000) 2019-10-13T07:44:49.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:49.046+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.053561+0000) 2019-10-13T07:44:49.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:49.050+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.054530+0000) 2019-10-13T07:44:49.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:49.050+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.054930+0000) 2019-10-13T07:44:49.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:49.049+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.054639+0000) 2019-10-13T07:44:49.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:49.050+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.055560+0000) 2019-10-13T07:44:49.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:49.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:49.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:49.050+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.056090+0000) 2019-10-13T07:44:49.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:49.050+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.057512+0000) 2019-10-13T07:44:49.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:49.057+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:49.062293+0000) 2019-10-13T07:44:50.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:50.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:50.650879+0000) 2019-10-13T07:44:50.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:50.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:50.749222+0000) 2019-10-13T07:44:51.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:51.542+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:51.545833+0000) 2019-10-13T07:44:51.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:51.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:51.545967+0000) 2019-10-13T07:44:53.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:53.220+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:53.222102+0000) 2019-10-13T07:44:54.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:54.054+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.059150+0000) 2019-10-13T07:44:54.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:54.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.059330+0000) 2019-10-13T07:44:54.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:54.054+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.060378+0000) 2019-10-13T07:44:54.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:54.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:54.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:54.054+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.061498+0000) 2019-10-13T07:44:54.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:54.058+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.062502+0000) 2019-10-13T07:44:54.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:54.058+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.062861+0000) 2019-10-13T07:44:54.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:54.058+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.062255+0000) 2019-10-13T07:44:54.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:54.060+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.065731+0000) 2019-10-13T07:44:54.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:54.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:54.269444+0000) 2019-10-13T07:44:57.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:57.105+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:57.109411+0000) 2019-10-13T07:44:57.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:57.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:57.523720+0000) 2019-10-13T07:44:58.422 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:44:58.422 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:58.426 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:44:58.426 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:44:59.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:44:59.057+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.064668+0000) 2019-10-13T07:44:59.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:44:59.057+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.065042+0000) 2019-10-13T07:44:59.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:44:59.060+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.064979+0000) 2019-10-13T07:44:59.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:44:59.061+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.065836+0000) 2019-10-13T07:44:59.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:44:59.061+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.066255+0000) 2019-10-13T07:44:59.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:44:59.061+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.066539+0000) 2019-10-13T07:44:59.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:44:59.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57634 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:44:59.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:44:59.061+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.067268+0000) 2019-10-13T07:44:59.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:44:59.068+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:44:59.074451+0000) 2019-10-13T07:45:00.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:00.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:00.651123+0000) 2019-10-13T07:45:00.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:00.741+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:00.749438+0000) 2019-10-13T07:45:01.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:01.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:01.546018+0000) 2019-10-13T07:45:01.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:01.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:01.546192+0000) 2019-10-13T07:45:03.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:03.224+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:03.228511+0000) 2019-10-13T07:45:04.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:04.064+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.071038+0000) 2019-10-13T07:45:04.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:04.065+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.070133+0000) 2019-10-13T07:45:04.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:04.065+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.070924+0000) 2019-10-13T07:45:04.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:04.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:04.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:04.065+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.071946+0000) 2019-10-13T07:45:04.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:04.065+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.072509+0000) 2019-10-13T07:45:04.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:04.065+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.073127+0000) 2019-10-13T07:45:04.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:04.065+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.073492+0000) 2019-10-13T07:45:04.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:04.072+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.079013+0000) 2019-10-13T07:45:04.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:04.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:04.269646+0000) 2019-10-13T07:45:07.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:07.105+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:07.109604+0000) 2019-10-13T07:45:07.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:07.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:07.523948+0000) 2019-10-13T07:45:09.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:09.073+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.075180+0000) 2019-10-13T07:45:09.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:09.073+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.076300+0000) 2019-10-13T07:45:09.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:09.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:09.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:09.072+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.076144+0000) 2019-10-13T07:45:09.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:09.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.077164+0000) 2019-10-13T07:45:09.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:09.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.077521+0000) 2019-10-13T07:45:09.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:09.077+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.078169+0000) 2019-10-13T07:45:09.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:09.077+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.078424+0000) 2019-10-13T07:45:09.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:09.076+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:09.084034+0000) 2019-10-13T07:45:10.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:10.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:10.651331+0000) 2019-10-13T07:45:10.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:10.745+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:10.749640+0000) 2019-10-13T07:45:11.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:11.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:11.546216+0000) 2019-10-13T07:45:11.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:11.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:11.546343+0000) 2019-10-13T07:45:13.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:13.232+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:13.234883+0000) 2019-10-13T07:45:14.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:14.077+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.079809+0000) 2019-10-13T07:45:14.081 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:14.077+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.080240+0000) 2019-10-13T07:45:14.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:14.077+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.081325+0000) 2019-10-13T07:45:14.083 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:14.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57731 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:14.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:14.077+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.081501+0000) 2019-10-13T07:45:14.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:14.081+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.082560+0000) 2019-10-13T07:45:14.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:14.076+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.082497+0000) 2019-10-13T07:45:14.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:14.081+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.083369+0000) 2019-10-13T07:45:14.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:14.080+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.087684+0000) 2019-10-13T07:45:14.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:14.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:14.269820+0000) 2019-10-13T07:45:17.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:17.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:17.109802+0000) 2019-10-13T07:45:17.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:17.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:17.524167+0000) 2019-10-13T07:45:19.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:19.081+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.084657+0000) 2019-10-13T07:45:19.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:19.081+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.085327+0000) 2019-10-13T07:45:19.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:19.085+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.085789+0000) 2019-10-13T07:45:19.087 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:19.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:19.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:19.080+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.085912+0000) 2019-10-13T07:45:19.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:19.085+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.086880+0000) 2019-10-13T07:45:19.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:19.085+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.087100+0000) 2019-10-13T07:45:19.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:19.085+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.089007+0000) 2019-10-13T07:45:19.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:19.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:19.093511+0000) 2019-10-13T07:45:20.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:20.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:20.651535+0000) 2019-10-13T07:45:20.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:20.749+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:20.749837+0000) 2019-10-13T07:45:21.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:21.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:21.546417+0000) 2019-10-13T07:45:21.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:21.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:21.546539+0000) 2019-10-13T07:45:23.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:23.240+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:23.241410+0000) 2019-10-13T07:45:24.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:24.085+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.089556+0000) 2019-10-13T07:45:24.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:24.084+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.089344+0000) 2019-10-13T07:45:24.092 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:24.089+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.090939+0000) 2019-10-13T07:45:24.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:24.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:24.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:24.089+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.091686+0000) 2019-10-13T07:45:24.093 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:24.089+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.092161+0000) 2019-10-13T07:45:24.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:24.089+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.092741+0000) 2019-10-13T07:45:24.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:24.093+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.093885+0000) 2019-10-13T07:45:24.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:24.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.096031+0000) 2019-10-13T07:45:24.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:24.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:24.270028+0000) 2019-10-13T07:45:27.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:27.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:27.109964+0000) 2019-10-13T07:45:27.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:27.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:27.524406+0000) 2019-10-13T07:45:28.562 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:45:28.562 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:28.566 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:45:28.566 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:29.096 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:29.088+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.094111+0000) 2019-10-13T07:45:29.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:29.093+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.095685+0000) 2019-10-13T07:45:29.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:29.093+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.095901+0000) 2019-10-13T07:45:29.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:29.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:29.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:29.093+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.096142+0000) 2019-10-13T07:45:29.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:29.093+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.096900+0000) 2019-10-13T07:45:29.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:29.093+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.097428+0000) 2019-10-13T07:45:29.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:29.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.098052+0000) 2019-10-13T07:45:29.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:29.100+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:29.106989+0000) 2019-10-13T07:45:30.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:30.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:30.651784+0000) 2019-10-13T07:45:30.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:30.749+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:30.750021+0000) 2019-10-13T07:45:31.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:31.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:31.546630+0000) 2019-10-13T07:45:31.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:31.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:31.546751+0000) 2019-10-13T07:45:33.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:33.244+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:33.247734+0000) 2019-10-13T07:45:34.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:34.097+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.099740+0000) 2019-10-13T07:45:34.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:34.096+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.100069+0000) 2019-10-13T07:45:34.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:34.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.100812+0000) 2019-10-13T07:45:34.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:34.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57854 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:34.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:34.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.101833+0000) 2019-10-13T07:45:34.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:34.101+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.102016+0000) 2019-10-13T07:45:34.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:34.101+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.103343+0000) 2019-10-13T07:45:34.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:34.101+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.103751+0000) 2019-10-13T07:45:34.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:34.100+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.107889+0000) 2019-10-13T07:45:34.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:34.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:34.270209+0000) 2019-10-13T07:45:37.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:37.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:37.110179+0000) 2019-10-13T07:45:37.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:37.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:37.524638+0000) 2019-10-13T07:45:39.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:39.101+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.104952+0000) 2019-10-13T07:45:39.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:39.101+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.105528+0000) 2019-10-13T07:45:39.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:39.105+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.106035+0000) 2019-10-13T07:45:39.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:39.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:39.108 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:39.100+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.106674+0000) 2019-10-13T07:45:39.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:39.105+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.107707+0000) 2019-10-13T07:45:39.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:39.105+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.108878+0000) 2019-10-13T07:45:39.110 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:39.105+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.108402+0000) 2019-10-13T07:45:39.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:39.108+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:39.112373+0000) 2019-10-13T07:45:40.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:40.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:40.651977+0000) 2019-10-13T07:45:40.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:40.749+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:40.750173+0000) 2019-10-13T07:45:41.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:41.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:41.546821+0000) 2019-10-13T07:45:41.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:41.540+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:41.546979+0000) 2019-10-13T07:45:43.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:43.252+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:43.254231+0000) 2019-10-13T07:45:44.112 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:44.109+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.110269+0000) 2019-10-13T07:45:44.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:44.109+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.110857+0000) 2019-10-13T07:45:44.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:44.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:44.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:44.109+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.113018+0000) 2019-10-13T07:45:44.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:44.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.113481+0000) 2019-10-13T07:45:44.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:44.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.113697+0000) 2019-10-13T07:45:44.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:44.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.114063+0000) 2019-10-13T07:45:44.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:44.113+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.115074+0000) 2019-10-13T07:45:44.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:44.116+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.120995+0000) 2019-10-13T07:45:44.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:44.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:44.270399+0000) 2019-10-13T07:45:47.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:47.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:47.110376+0000) 2019-10-13T07:45:47.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:47.521+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:47.524837+0000) 2019-10-13T07:45:49.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:49.113+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.115603+0000) 2019-10-13T07:45:49.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:49.113+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.116395+0000) 2019-10-13T07:45:49.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:49.117+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.118120+0000) 2019-10-13T07:45:49.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:49.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:49.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:49.117+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.119222+0000) 2019-10-13T07:45:49.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:49.117+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.120002+0000) 2019-10-13T07:45:49.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:49.121+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.122049+0000) 2019-10-13T07:45:49.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:49.120+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.121862+0000) 2019-10-13T07:45:49.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:49.120+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:49.124075+0000) 2019-10-13T07:45:50.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:50.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:50.652183+0000) 2019-10-13T07:45:50.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:50.749+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:50.750359+0000) 2019-10-13T07:45:51.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:51.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:51.547011+0000) 2019-10-13T07:45:51.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:51.544+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:51.547178+0000) 2019-10-13T07:45:53.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:53.260+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:53.260677+0000) 2019-10-13T07:45:54.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:54.120+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.122239+0000) 2019-10-13T07:45:54.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:54.120+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.122648+0000) 2019-10-13T07:45:54.125 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:54.120+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.123032+0000) 2019-10-13T07:45:54.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:54.120+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:54.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:54.120+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.123750+0000) 2019-10-13T07:45:54.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:54.120+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.124198+0000) 2019-10-13T07:45:54.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:54.120+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.125507+0000) 2019-10-13T07:45:54.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:54.124+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.125491+0000) 2019-10-13T07:45:54.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:54.128+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.131942+0000) 2019-10-13T07:45:54.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:54.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:54.270576+0000) 2019-10-13T07:45:57.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:57.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:57.110629+0000) 2019-10-13T07:45:57.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:57.520+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:57.525002+0000) 2019-10-13T07:45:58.639 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:45:58.640 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:58.643 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:45:58.644 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:45:59.128 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:45:59.124+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.126817+0000) 2019-10-13T07:45:59.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:45:59.123+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.126675+0000) 2019-10-13T07:45:59.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:45:59.124+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.127745+0000) 2019-10-13T07:45:59.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:45:59.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:45:59.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:45:59.128+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.129825+0000) 2019-10-13T07:45:59.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:45:59.128+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.130078+0000) 2019-10-13T07:45:59.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:45:59.128+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.131090+0000) 2019-10-13T07:45:59.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:45:59.128+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.131592+0000) 2019-10-13T07:45:59.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:45:59.135+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:45:59.138886+0000) 2019-10-13T07:46:00.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:00.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:00.652386+0000) 2019-10-13T07:46:00.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:00.748+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:00.750521+0000) 2019-10-13T07:46:01.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:01.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:01.547169+0000) 2019-10-13T07:46:01.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:01.543+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:01.547394+0000) 2019-10-13T07:46:03.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:03.263+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:03.267179+0000) 2019-10-13T07:46:04.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:04.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.132255+0000) 2019-10-13T07:46:04.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:04.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.132976+0000) 2019-10-13T07:46:04.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:04.128+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.133269+0000) 2019-10-13T07:46:04.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:04.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:04.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:04.131+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.134500+0000) 2019-10-13T07:46:04.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:04.132+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.135519+0000) 2019-10-13T07:46:04.137 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:04.132+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.135727+0000) 2019-10-13T07:46:04.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:04.132+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.136682+0000) 2019-10-13T07:46:04.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:04.139+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.141298+0000) 2019-10-13T07:46:04.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:04.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:04.270775+0000) 2019-10-13T07:46:07.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:07.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:07.110862+0000) 2019-10-13T07:46:07.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:07.520+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:07.525229+0000) 2019-10-13T07:46:09.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:09.136+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.138047+0000) 2019-10-13T07:46:09.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:09.136+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.138691+0000) 2019-10-13T07:46:09.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:09.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:09.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:09.136+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.140240+0000) 2019-10-13T07:46:09.142 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:09.139+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.140557+0000) 2019-10-13T07:46:09.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:09.140+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.141865+0000) 2019-10-13T07:46:09.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:09.140+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.142142+0000) 2019-10-13T07:46:09.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:09.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.143006+0000) 2019-10-13T07:46:09.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:09.139+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:09.143822+0000) 2019-10-13T07:46:10.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:10.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:10.652539+0000) 2019-10-13T07:46:10.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:10.748+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:10.750670+0000) 2019-10-13T07:46:11.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:11.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:11.547345+0000) 2019-10-13T07:46:11.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:11.543+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:11.547569+0000) 2019-10-13T07:46:13.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:13.271+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:13.273300+0000) 2019-10-13T07:46:14.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:14.140+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.143492+0000) 2019-10-13T07:46:14.145 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:14.140+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.143968+0000) 2019-10-13T07:46:14.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:14.140+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.144402+0000) 2019-10-13T07:46:14.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:14.140+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.144793+0000) 2019-10-13T07:46:14.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:14.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:14.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:14.144+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.145980+0000) 2019-10-13T07:46:14.148 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:14.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.145929+0000) 2019-10-13T07:46:14.148 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:14.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.147032+0000) 2019-10-13T07:46:14.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:14.147+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.151138+0000) 2019-10-13T07:46:14.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:14.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:14.270953+0000) 2019-10-13T07:46:17.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:17.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:17.111128+0000) 2019-10-13T07:46:17.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:17.520+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:17.525432+0000) 2019-10-13T07:46:19.151 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:19.148+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.149832+0000) 2019-10-13T07:46:19.151 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:19.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:19.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:19.148+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.150460+0000) 2019-10-13T07:46:19.152 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:19.148+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.150778+0000) 2019-10-13T07:46:19.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:19.148+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.151945+0000) 2019-10-13T07:46:19.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:19.148+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.152202+0000) 2019-10-13T07:46:19.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:19.151+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.152955+0000) 2019-10-13T07:46:19.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:19.148+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.152564+0000) 2019-10-13T07:46:19.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:19.155+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:19.157383+0000) 2019-10-13T07:46:20.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:20.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:20.652745+0000) 2019-10-13T07:46:20.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:20.748+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:20.750821+0000) 2019-10-13T07:46:21.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:21.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:21.547494+0000) 2019-10-13T07:46:21.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:21.547+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:21.547764+0000) 2019-10-13T07:46:23.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:23.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:23.279701+0000) 2019-10-13T07:46:24.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:24.152+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.154215+0000) 2019-10-13T07:46:24.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:24.152+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.154383+0000) 2019-10-13T07:46:24.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:24.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:24.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:24.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.155067+0000) 2019-10-13T07:46:24.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:24.152+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.156064+0000) 2019-10-13T07:46:24.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:24.155+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.156073+0000) 2019-10-13T07:46:24.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:24.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.158064+0000) 2019-10-13T07:46:24.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:24.152+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.157280+0000) 2019-10-13T07:46:24.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:24.163+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.164827+0000) 2019-10-13T07:46:24.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:24.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:24.271126+0000) 2019-10-13T07:46:27.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:27.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:27.111350+0000) 2019-10-13T07:46:27.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:27.520+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:27.525636+0000) 2019-10-13T07:46:28.711 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:46:28.712 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:28.715 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:46:28.716 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:29.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:29.156+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.158741+0000) 2019-10-13T07:46:29.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:29.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.159212+0000) 2019-10-13T07:46:29.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:29.159+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.160173+0000) 2019-10-13T07:46:29.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:29.156+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.160791+0000) 2019-10-13T07:46:29.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:29.156+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.161272+0000) 2019-10-13T07:46:29.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:29.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:29.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:29.160+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.162126+0000) 2019-10-13T07:46:29.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:29.160+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.164791+0000) 2019-10-13T07:46:29.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:29.167+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:29.169083+0000) 2019-10-13T07:46:30.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:30.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:30.652950+0000) 2019-10-13T07:46:30.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:30.748+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:30.750973+0000) 2019-10-13T07:46:31.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:31.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:31.547704+0000) 2019-10-13T07:46:31.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:31.547+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:31.547988+0000) 2019-10-13T07:46:33.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:33.287+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:33.290739+0000) 2019-10-13T07:46:34.168 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:34.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.166469+0000) 2019-10-13T07:46:34.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:34.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58226 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:34.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:34.164+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.166972+0000) 2019-10-13T07:46:34.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:34.164+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.168010+0000) 2019-10-13T07:46:34.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:34.164+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.168183+0000) 2019-10-13T07:46:34.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:34.167+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.168774+0000) 2019-10-13T07:46:34.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:34.168+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.169727+0000) 2019-10-13T07:46:34.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:34.171+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.173562+0000) 2019-10-13T07:46:34.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:34.168+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.170204+0000) 2019-10-13T07:46:34.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:34.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:34.271306+0000) 2019-10-13T07:46:37.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:37.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:37.111513+0000) 2019-10-13T07:46:37.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:37.524+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:37.525899+0000) 2019-10-13T07:46:39.172 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:39.168+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.170312+0000) 2019-10-13T07:46:39.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:39.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.170786+0000) 2019-10-13T07:46:39.173 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:39.167+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.171230+0000) 2019-10-13T07:46:39.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:39.168+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.171904+0000) 2019-10-13T07:46:39.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:39.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:39.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:39.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.173618+0000) 2019-10-13T07:46:39.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:39.168+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.173713+0000) 2019-10-13T07:46:39.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:39.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.174123+0000) 2019-10-13T07:46:39.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:39.179+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:39.181370+0000) 2019-10-13T07:46:40.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:40.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:40.653172+0000) 2019-10-13T07:46:40.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:40.748+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:40.751149+0000) 2019-10-13T07:46:41.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:41.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:41.547931+0000) 2019-10-13T07:46:41.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:41.547+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:41.548215+0000) 2019-10-13T07:46:43.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:43.295+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:43.297263+0000) 2019-10-13T07:46:44.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:44.172+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.175996+0000) 2019-10-13T07:46:44.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:44.172+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.176746+0000) 2019-10-13T07:46:44.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:44.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:44.179 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:44.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.178312+0000) 2019-10-13T07:46:44.180 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:44.176+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.178996+0000) 2019-10-13T07:46:44.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:44.175+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.179041+0000) 2019-10-13T07:46:44.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:44.176+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.179265+0000) 2019-10-13T07:46:44.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:44.176+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.179863+0000) 2019-10-13T07:46:44.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:44.183+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.184762+0000) 2019-10-13T07:46:44.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:44.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:44.271436+0000) 2019-10-13T07:46:47.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:47.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:47.111713+0000) 2019-10-13T07:46:47.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:47.524+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:47.526130+0000) 2019-10-13T07:46:49.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:49.175+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.181314+0000) 2019-10-13T07:46:49.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:49.175+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.181589+0000) 2019-10-13T07:46:49.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:49.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:49.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:49.179+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.183719+0000) 2019-10-13T07:46:49.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:49.179+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.184888+0000) 2019-10-13T07:46:49.187 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:49.183+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.185259+0000) 2019-10-13T07:46:49.188 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:49.183+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.186585+0000) 2019-10-13T07:46:49.188 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:49.183+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.187067+0000) 2019-10-13T07:46:49.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:49.191+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:49.192944+0000) 2019-10-13T07:46:50.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:50.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:50.653376+0000) 2019-10-13T07:46:50.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:50.747+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:50.751297+0000) 2019-10-13T07:46:51.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:51.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:51.548126+0000) 2019-10-13T07:46:51.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:51.547+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:51.548443+0000) 2019-10-13T07:46:53.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:53.303+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:53.303759+0000) 2019-10-13T07:46:54.188 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:54.183+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.186621+0000) 2019-10-13T07:46:54.188 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:54.183+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.187585+0000) 2019-10-13T07:46:54.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:54.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:54.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:54.187+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.189901+0000) 2019-10-13T07:46:54.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:54.187+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.190398+0000) 2019-10-13T07:46:54.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:54.187+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.191196+0000) 2019-10-13T07:46:54.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:54.187+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.192047+0000) 2019-10-13T07:46:54.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:54.187+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.192533+0000) 2019-10-13T07:46:54.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:54.191+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.194545+0000) 2019-10-13T07:46:54.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:54.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:54.271604+0000) 2019-10-13T07:46:57.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:57.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:57.111918+0000) 2019-10-13T07:46:57.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:57.523+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:57.526336+0000) 2019-10-13T07:46:58.773 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:46:58.773 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:58.777 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:46:58.777 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:46:59.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:46:59.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:46:59.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:46:59.191+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.194152+0000) 2019-10-13T07:46:59.196 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:46:59.191+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.194406+0000) 2019-10-13T07:46:59.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:46:59.191+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.194854+0000) 2019-10-13T07:46:59.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:46:59.191+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.195324+0000) 2019-10-13T07:46:59.197 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:46:59.191+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.195067+0000) 2019-10-13T07:46:59.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:46:59.191+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.195968+0000) 2019-10-13T07:46:59.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:46:59.195+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.198151+0000) 2019-10-13T07:46:59.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:46:59.199+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:46:59.201191+0000) 2019-10-13T07:47:00.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:00.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:00.653609+0000) 2019-10-13T07:47:00.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:00.747+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:00.751497+0000) 2019-10-13T07:47:01.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:01.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:01.548353+0000) 2019-10-13T07:47:01.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:01.547+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:01.548638+0000) 2019-10-13T07:47:03.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:03.306+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:03.310270+0000) 2019-10-13T07:47:04.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:04.194+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.196559+0000) 2019-10-13T07:47:04.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:04.195+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.198346+0000) 2019-10-13T07:47:04.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:04.195+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:04.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:04.195+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.199004+0000) 2019-10-13T07:47:04.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:04.195+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.199276+0000) 2019-10-13T07:47:04.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:04.195+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.199695+0000) 2019-10-13T07:47:04.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:04.195+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.200220+0000) 2019-10-13T07:47:04.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:04.195+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.201579+0000) 2019-10-13T07:47:04.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:04.202+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.207874+0000) 2019-10-13T07:47:04.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:04.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:04.271788+0000) 2019-10-13T07:47:07.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:07.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:07.112154+0000) 2019-10-13T07:47:07.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:07.523+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:07.526533+0000) 2019-10-13T07:47:09.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:09.199+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.203335+0000) 2019-10-13T07:47:09.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:09.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:09.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:09.202+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.204790+0000) 2019-10-13T07:47:09.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:09.203+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.205952+0000) 2019-10-13T07:47:09.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:09.203+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.206345+0000) 2019-10-13T07:47:09.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:09.203+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.207283+0000) 2019-10-13T07:47:09.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:09.203+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.207854+0000) 2019-10-13T07:47:09.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:09.203+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.208392+0000) 2019-10-13T07:47:09.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:09.206+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:09.211354+0000) 2019-10-13T07:47:10.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:10.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:10.653806+0000) 2019-10-13T07:47:10.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:10.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:10.751706+0000) 2019-10-13T07:47:11.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:11.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:11.548551+0000) 2019-10-13T07:47:11.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:11.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:11.548865+0000) 2019-10-13T07:47:13.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:13.314+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:13.316715+0000) 2019-10-13T07:47:14.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:14.203+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.207032+0000) 2019-10-13T07:47:14.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:14.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:14.210 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:14.203+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.209102+0000) 2019-10-13T07:47:14.211 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:14.206+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.208938+0000) 2019-10-13T07:47:14.211 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:14.207+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.209943+0000) 2019-10-13T07:47:14.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:14.207+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.210462+0000) 2019-10-13T07:47:14.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:14.207+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.211215+0000) 2019-10-13T07:47:14.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:14.207+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.212195+0000) 2019-10-13T07:47:14.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:14.214+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.218148+0000) 2019-10-13T07:47:14.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:14.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:14.271999+0000) 2019-10-13T07:47:17.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:17.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:17.112393+0000) 2019-10-13T07:47:17.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:17.523+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:17.526732+0000) 2019-10-13T07:47:19.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:19.207+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.212182+0000) 2019-10-13T07:47:19.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:19.207+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.212556+0000) 2019-10-13T07:47:19.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:19.207+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.213359+0000) 2019-10-13T07:47:19.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:19.207+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:19.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:19.210+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.212967+0000) 2019-10-13T07:47:19.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:19.211+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.214109+0000) 2019-10-13T07:47:19.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:19.211+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.214286+0000) 2019-10-13T07:47:19.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:19.211+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.215199+0000) 2019-10-13T07:47:19.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:19.218+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:19.221030+0000) 2019-10-13T07:47:20.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:20.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:20.654027+0000) 2019-10-13T07:47:20.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:20.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:20.751898+0000) 2019-10-13T07:47:21.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:21.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:21.548748+0000) 2019-10-13T07:47:21.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:21.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:21.549065+0000) 2019-10-13T07:47:23.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:23.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:23.323215+0000) 2019-10-13T07:47:24.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:24.211+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.217439+0000) 2019-10-13T07:47:24.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:24.211+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.217682+0000) 2019-10-13T07:47:24.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:24.215+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.218290+0000) 2019-10-13T07:47:24.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:24.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:24.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:24.214+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.216158+0000) 2019-10-13T07:47:24.221 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:24.215+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.219255+0000) 2019-10-13T07:47:24.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:24.215+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.219723+0000) 2019-10-13T07:47:24.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:24.215+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.221379+0000) 2019-10-13T07:47:24.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:24.222+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.224634+0000) 2019-10-13T07:47:24.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:24.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:24.272153+0000) 2019-10-13T07:47:27.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:27.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:27.112596+0000) 2019-10-13T07:47:27.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:27.523+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:27.526928+0000) 2019-10-13T07:47:28.881 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:47:28.881 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:28.884 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:47:28.884 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:29.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:29.218+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.222606+0000) 2019-10-13T07:47:29.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:29.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:29.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:29.219+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.223585+0000) 2019-10-13T07:47:29.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:29.219+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.224697+0000) 2019-10-13T07:47:29.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:29.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.225372+0000) 2019-10-13T07:47:29.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:29.223+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.226895+0000) 2019-10-13T07:47:29.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:29.223+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.226160+0000) 2019-10-13T07:47:29.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:29.223+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.227631+0000) 2019-10-13T07:47:29.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:29.230+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:29.232693+0000) 2019-10-13T07:47:30.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:30.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:30.654225+0000) 2019-10-13T07:47:30.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:30.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:30.752098+0000) 2019-10-13T07:47:31.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:31.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:31.548900+0000) 2019-10-13T07:47:31.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:31.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:31.549264+0000) 2019-10-13T07:47:33.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:33.326+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:33.329737+0000) 2019-10-13T07:47:34.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:34.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.227372+0000) 2019-10-13T07:47:34.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:34.223+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.227849+0000) 2019-10-13T07:47:34.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:34.223+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.228352+0000) 2019-10-13T07:47:34.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:34.223+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:34.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:34.226+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.228224+0000) 2019-10-13T07:47:34.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:34.223+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.229562+0000) 2019-10-13T07:47:34.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:34.227+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.230086+0000) 2019-10-13T07:47:34.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:34.227+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.231415+0000) 2019-10-13T07:47:34.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:34.234+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.236506+0000) 2019-10-13T07:47:34.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:34.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:34.272301+0000) 2019-10-13T07:47:37.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:37.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:37.112807+0000) 2019-10-13T07:47:37.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:37.523+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:37.527131+0000) 2019-10-13T07:47:39.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:39.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:39.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:39.231+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.233834+0000) 2019-10-13T07:47:39.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:39.231+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.233962+0000) 2019-10-13T07:47:39.236 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:39.230+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.233890+0000) 2019-10-13T07:47:39.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:39.231+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.235332+0000) 2019-10-13T07:47:39.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:39.231+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.235494+0000) 2019-10-13T07:47:39.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:39.231+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.236200+0000) 2019-10-13T07:47:39.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:39.231+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.237086+0000) 2019-10-13T07:47:39.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:39.234+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:39.239948+0000) 2019-10-13T07:47:40.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:40.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:40.654438+0000) 2019-10-13T07:47:40.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:40.747+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:40.752321+0000) 2019-10-13T07:47:41.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:41.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:41.549092+0000) 2019-10-13T07:47:41.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:41.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:41.549467+0000) 2019-10-13T07:47:43.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:43.334+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:43.336216+0000) 2019-10-13T07:47:44.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:44.231+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.236430+0000) 2019-10-13T07:47:44.238 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:44.231+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.237004+0000) 2019-10-13T07:47:44.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:44.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.237375+0000) 2019-10-13T07:47:44.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:44.231+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.237671+0000) 2019-10-13T07:47:44.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:44.235+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:44.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:44.234+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.238437+0000) 2019-10-13T07:47:44.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:44.235+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.239697+0000) 2019-10-13T07:47:44.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:44.235+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.240454+0000) 2019-10-13T07:47:44.247 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:44.242+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.245907+0000) 2019-10-13T07:47:44.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:44.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:44.272473+0000) 2019-10-13T07:47:47.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:47.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:47.113008+0000) 2019-10-13T07:47:47.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:47.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:47.527332+0000) 2019-10-13T07:47:49.244 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:49.238+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.242586+0000) 2019-10-13T07:47:49.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:49.238+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.243040+0000) 2019-10-13T07:47:49.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:49.238+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:49.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:49.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.244244+0000) 2019-10-13T07:47:49.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:49.238+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.245238+0000) 2019-10-13T07:47:49.247 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:49.238+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.245644+0000) 2019-10-13T07:47:49.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:49.242+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.246664+0000) 2019-10-13T07:47:49.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:49.242+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.247431+0000) 2019-10-13T07:47:49.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:49.250+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:49.252664+0000) 2019-10-13T07:47:50.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:50.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:50.654646+0000) 2019-10-13T07:47:50.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:50.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:50.752542+0000) 2019-10-13T07:47:51.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:51.542+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:51.549290+0000) 2019-10-13T07:47:51.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:51.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:51.549660+0000) 2019-10-13T07:47:53.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:53.342+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:53.342713+0000) 2019-10-13T07:47:54.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:54.242+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.247469+0000) 2019-10-13T07:47:54.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:54.242+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.248325+0000) 2019-10-13T07:47:54.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:54.242+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.249040+0000) 2019-10-13T07:47:54.250 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:54.242+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.249170+0000) 2019-10-13T07:47:54.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:54.242+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.249690+0000) 2019-10-13T07:47:54.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:54.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:54.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:54.246+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.249872+0000) 2019-10-13T07:47:54.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:54.246+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.250911+0000) 2019-10-13T07:47:54.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:54.250+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.254098+0000) 2019-10-13T07:47:54.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:54.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:54.272633+0000) 2019-10-13T07:47:57.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:57.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:57.113200+0000) 2019-10-13T07:47:57.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:57.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:57.527530+0000) 2019-10-13T07:47:58.996 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:47:58.996 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:59.000 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:47:59.000 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:47:59.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:47:59.246+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.253504+0000) 2019-10-13T07:47:59.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:47:59.250+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.254252+0000) 2019-10-13T07:47:59.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:47:59.250+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.254573+0000) 2019-10-13T07:47:59.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:47:59.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:47:59.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:47:59.250+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.255238+0000) 2019-10-13T07:47:59.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:47:59.250+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.255580+0000) 2019-10-13T07:47:59.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:47:59.250+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.255861+0000) 2019-10-13T07:47:59.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:47:59.250+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.255974+0000) 2019-10-13T07:47:59.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:47:59.258+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:47:59.262565+0000) 2019-10-13T07:48:00.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:00.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:00.654849+0000) 2019-10-13T07:48:00.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:00.750+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:00.752739+0000) 2019-10-13T07:48:01.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:01.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:01.549485+0000) 2019-10-13T07:48:01.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:01.546+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:01.549859+0000) 2019-10-13T07:48:03.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:03.346+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:03.349143+0000) 2019-10-13T07:48:04.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:04.254+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.258872+0000) 2019-10-13T07:48:04.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:04.254+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.259237+0000) 2019-10-13T07:48:04.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:04.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:04.261 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:04.254+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.259827+0000) 2019-10-13T07:48:04.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:04.254+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.260571+0000) 2019-10-13T07:48:04.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:04.258+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.260515+0000) 2019-10-13T07:48:04.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:04.254+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.261453+0000) 2019-10-13T07:48:04.264 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:04.258+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.262907+0000) 2019-10-13T07:48:04.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:04.266+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.270238+0000) 2019-10-13T07:48:04.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:04.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:04.272741+0000) 2019-10-13T07:48:07.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:07.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:07.113425+0000) 2019-10-13T07:48:07.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:07.522+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:07.527730+0000) 2019-10-13T07:48:09.265 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:09.258+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.263102+0000) 2019-10-13T07:48:09.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:09.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.263844+0000) 2019-10-13T07:48:09.266 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:09.258+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.264366+0000) 2019-10-13T07:48:09.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:09.258+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.264630+0000) 2019-10-13T07:48:09.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:09.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:09.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:09.262+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.266129+0000) 2019-10-13T07:48:09.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:09.262+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.266489+0000) 2019-10-13T07:48:09.269 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:09.261+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.267051+0000) 2019-10-13T07:48:09.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:09.269+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:09.274005+0000) 2019-10-13T07:48:10.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:10.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:10.655006+0000) 2019-10-13T07:48:10.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:10.750+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:10.752898+0000) 2019-10-13T07:48:11.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:11.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:11.549675+0000) 2019-10-13T07:48:11.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:11.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:11.550055+0000) 2019-10-13T07:48:13.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:13.353+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:13.355523+0000) 2019-10-13T07:48:14.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:14.262+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.268570+0000) 2019-10-13T07:48:14.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:14.262+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.269019+0000) 2019-10-13T07:48:14.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:14.266+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.270271+0000) 2019-10-13T07:48:14.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:14.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:14.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:14.266+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.270861+0000) 2019-10-13T07:48:14.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:14.265+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.270722+0000) 2019-10-13T07:48:14.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:14.266+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.271588+0000) 2019-10-13T07:48:14.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:14.266+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.272951+0000) 2019-10-13T07:48:14.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:14.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.273186+0000) 2019-10-13T07:48:14.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:14.273+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:14.276213+0000) 2019-10-13T07:48:17.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:17.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:17.113628+0000) 2019-10-13T07:48:17.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:17.522+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:17.527893+0000) 2019-10-13T07:48:19.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:19.266+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.273649+0000) 2019-10-13T07:48:19.275 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:19.270+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.274413+0000) 2019-10-13T07:48:19.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:19.270+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.274613+0000) 2019-10-13T07:48:19.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:19.269+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.274861+0000) 2019-10-13T07:48:19.277 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:19.270+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.276119+0000) 2019-10-13T07:48:19.278 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:19.270+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.276428+0000) 2019-10-13T07:48:19.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:19.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:19.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:19.270+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.276947+0000) 2019-10-13T07:48:19.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:19.277+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:19.281455+0000) 2019-10-13T07:48:20.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:20.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:20.655207+0000) 2019-10-13T07:48:20.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:20.746+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:20.753124+0000) 2019-10-13T07:48:21.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:21.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:21.549934+0000) 2019-10-13T07:48:21.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:21.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:21.550310+0000) 2019-10-13T07:48:23.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:23.357+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:23.362109+0000) 2019-10-13T07:48:24.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:24.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.273796+0000) 2019-10-13T07:48:24.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:24.274+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.280075+0000) 2019-10-13T07:48:24.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:24.274+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.280135+0000) 2019-10-13T07:48:24.282 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:24.274+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.281530+0000) 2019-10-13T07:48:24.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:24.278+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.281958+0000) 2019-10-13T07:48:24.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:24.278+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.282256+0000) 2019-10-13T07:48:24.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:24.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:24.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:24.278+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.283195+0000) 2019-10-13T07:48:24.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:24.277+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.283343+0000) 2019-10-13T07:48:24.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:24.285+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:24.291364+0000) 2019-10-13T07:48:27.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:27.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:27.113860+0000) 2019-10-13T07:48:27.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:27.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:27.528097+0000) 2019-10-13T07:48:29.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:48:29.119 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:29.122 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:48:29.122 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:29.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:29.282+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.286263+0000) 2019-10-13T07:48:29.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:29.282+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.287171+0000) 2019-10-13T07:48:29.290 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:29.281+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.287117+0000) 2019-10-13T07:48:29.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:29.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:29.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:29.282+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.288234+0000) 2019-10-13T07:48:29.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:29.282+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.288754+0000) 2019-10-13T07:48:29.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:29.282+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.289045+0000) 2019-10-13T07:48:29.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:29.282+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.289614+0000) 2019-10-13T07:48:29.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:29.293+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:29.296388+0000) 2019-10-13T07:48:30.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:30.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:30.655423+0000) 2019-10-13T07:48:30.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:30.750+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:30.753348+0000) 2019-10-13T07:48:31.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:31.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:31.550131+0000) 2019-10-13T07:48:31.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:31.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:31.550504+0000) 2019-10-13T07:48:33.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:33.365+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:33.368531+0000) 2019-10-13T07:48:34.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:34.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.273956+0000) 2019-10-13T07:48:34.292 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:34.286+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.291410+0000) 2019-10-13T07:48:34.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:34.285+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.291432+0000) 2019-10-13T07:48:34.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:34.286+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.292619+0000) 2019-10-13T07:48:34.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:34.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:34.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:34.290+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.294030+0000) 2019-10-13T07:48:34.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:34.290+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.294934+0000) 2019-10-13T07:48:34.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:34.290+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.295209+0000) 2019-10-13T07:48:34.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:34.290+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.294760+0000) 2019-10-13T07:48:34.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:34.297+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:34.300991+0000) 2019-10-13T07:48:37.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:37.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:37.114120+0000) 2019-10-13T07:48:37.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:37.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:37.528266+0000) 2019-10-13T07:48:39.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:39.290+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.296426+0000) 2019-10-13T07:48:39.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:39.293+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.296259+0000) 2019-10-13T07:48:39.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:39.294+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:39.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:39.294+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.298767+0000) 2019-10-13T07:48:39.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:39.294+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.298822+0000) 2019-10-13T07:48:39.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:39.294+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.299606+0000) 2019-10-13T07:48:39.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:39.294+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.299822+0000) 2019-10-13T07:48:39.301 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:39.294+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.299923+0000) 2019-10-13T07:48:39.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:39.301+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:39.305912+0000) 2019-10-13T07:48:40.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:40.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:40.655684+0000) 2019-10-13T07:48:40.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:40.750+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:40.753546+0000) 2019-10-13T07:48:41.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:41.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:41.550323+0000) 2019-10-13T07:48:41.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:41.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:41.550719+0000) 2019-10-13T07:48:43.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:43.373+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:43.374878+0000) 2019-10-13T07:48:44.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:44.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.274161+0000) 2019-10-13T07:48:44.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:44.297+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.301961+0000) 2019-10-13T07:48:44.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:44.297+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.303752+0000) 2019-10-13T07:48:44.305 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:44.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:44.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:44.297+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.304375+0000) 2019-10-13T07:48:44.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:44.297+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.304853+0000) 2019-10-13T07:48:44.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:44.301+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.304868+0000) 2019-10-13T07:48:44.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:44.301+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.305995+0000) 2019-10-13T07:48:44.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:44.301+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.307327+0000) 2019-10-13T07:48:44.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:44.309+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:44.313423+0000) 2019-10-13T07:48:47.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:47.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:47.114349+0000) 2019-10-13T07:48:47.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:47.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:47.528493+0000) 2019-10-13T07:48:49.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:49.301+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.307607+0000) 2019-10-13T07:48:49.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:49.301+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.308706+0000) 2019-10-13T07:48:49.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:49.301+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.309437+0000) 2019-10-13T07:48:49.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:49.305+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:49.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:49.305+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.309981+0000) 2019-10-13T07:48:49.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:49.305+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.310183+0000) 2019-10-13T07:48:49.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:49.305+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.311728+0000) 2019-10-13T07:48:49.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:49.305+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.310941+0000) 2019-10-13T07:48:49.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:49.309+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:49.314890+0000) 2019-10-13T07:48:50.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:50.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:50.655899+0000) 2019-10-13T07:48:50.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:50.749+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:50.753761+0000) 2019-10-13T07:48:51.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:51.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:51.550516+0000) 2019-10-13T07:48:51.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:51.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:51.550965+0000) 2019-10-13T07:48:53.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:53.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:53.380955+0000) 2019-10-13T07:48:54.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:54.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.274360+0000) 2019-10-13T07:48:54.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:54.309+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.313091+0000) 2019-10-13T07:48:54.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:54.309+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.313796+0000) 2019-10-13T07:48:54.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:54.309+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.314473+0000) 2019-10-13T07:48:54.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:54.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:54.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:54.309+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.314431+0000) 2019-10-13T07:48:54.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:54.309+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.315373+0000) 2019-10-13T07:48:54.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:54.309+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.316812+0000) 2019-10-13T07:48:54.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:54.309+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.317319+0000) 2019-10-13T07:48:54.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:54.317+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:54.320710+0000) 2019-10-13T07:48:57.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:57.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:57.114551+0000) 2019-10-13T07:48:57.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:57.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:57.528700+0000) 2019-10-13T07:48:59.247 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:48:59.247 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:59.251 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:48:59.251 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:48:59.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:48:59.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:48:59.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:48:59.317+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.319664+0000) 2019-10-13T07:48:59.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:48:59.317+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.320231+0000) 2019-10-13T07:48:59.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:48:59.317+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.320242+0000) 2019-10-13T07:48:59.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:48:59.317+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.321611+0000) 2019-10-13T07:48:59.324 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:48:59.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.322380+0000) 2019-10-13T07:48:59.324 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:48:59.321+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.323042+0000) 2019-10-13T07:48:59.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:48:59.317+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.322402+0000) 2019-10-13T07:48:59.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:48:59.321+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:48:59.325678+0000) 2019-10-13T07:49:00.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:00.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:00.656086+0000) 2019-10-13T07:49:00.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:00.753+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:00.753960+0000) 2019-10-13T07:49:01.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:01.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:01.550665+0000) 2019-10-13T07:49:01.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:01.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:01.551169+0000) 2019-10-13T07:49:03.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:03.385+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:03.387609+0000) 2019-10-13T07:49:04.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:04.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.274559+0000) 2019-10-13T07:49:04.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:04.321+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.323126+0000) 2019-10-13T07:49:04.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:04.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.323769+0000) 2019-10-13T07:49:04.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:04.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.323838+0000) 2019-10-13T07:49:04.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:04.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:04.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:04.321+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.325626+0000) 2019-10-13T07:49:04.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:04.325+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.327843+0000) 2019-10-13T07:49:04.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:04.325+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.328096+0000) 2019-10-13T07:49:04.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:04.325+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.328509+0000) 2019-10-13T07:49:04.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:04.329+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:04.334731+0000) 2019-10-13T07:49:07.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:07.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:07.114748+0000) 2019-10-13T07:49:07.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:07.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:07.528962+0000) 2019-10-13T07:49:09.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:09.325+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.328543+0000) 2019-10-13T07:49:09.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:09.325+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.329484+0000) 2019-10-13T07:49:09.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:09.329+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.330204+0000) 2019-10-13T07:49:09.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:09.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:09.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:09.329+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.330700+0000) 2019-10-13T07:49:09.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:09.329+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.331214+0000) 2019-10-13T07:49:09.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:09.329+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.331881+0000) 2019-10-13T07:49:09.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:09.329+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.332355+0000) 2019-10-13T07:49:09.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:09.333+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:09.337226+0000) 2019-10-13T07:49:10.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:10.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:10.656287+0000) 2019-10-13T07:49:10.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:10.753+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:10.754187+0000) 2019-10-13T07:49:11.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:11.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:11.550858+0000) 2019-10-13T07:49:11.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:11.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:11.551364+0000) 2019-10-13T07:49:13.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:13.393+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:13.394074+0000) 2019-10-13T07:49:14.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:14.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.274773+0000) 2019-10-13T07:49:14.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:14.333+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.336176+0000) 2019-10-13T07:49:14.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:14.333+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.336360+0000) 2019-10-13T07:49:14.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:14.333+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.337037+0000) 2019-10-13T07:49:14.345 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:14.333+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.337615+0000) 2019-10-13T07:49:14.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:14.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:14.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:14.329+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.332903+0000) 2019-10-13T07:49:14.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:14.333+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.333873+0000) 2019-10-13T07:49:14.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:14.337+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.337811+0000) 2019-10-13T07:49:14.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:14.337+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:14.342805+0000) 2019-10-13T07:49:17.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:17.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:17.114946+0000) 2019-10-13T07:49:17.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:17.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:17.529192+0000) 2019-10-13T07:49:19.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:19.337+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.338078+0000) 2019-10-13T07:49:19.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:19.337+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.338331+0000) 2019-10-13T07:49:19.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:19.337+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.338972+0000) 2019-10-13T07:49:19.340 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:19.337+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.339577+0000) 2019-10-13T07:49:19.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:19.337+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.339744+0000) 2019-10-13T07:49:19.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:19.332+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.339785+0000) 2019-10-13T07:49:19.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:19.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59244 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:19.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:19.337+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.340177+0000) 2019-10-13T07:49:19.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:19.340+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:19.345152+0000) 2019-10-13T07:49:20.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:20.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:20.656522+0000) 2019-10-13T07:49:20.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:20.753+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:20.754389+0000) 2019-10-13T07:49:21.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:21.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:21.551050+0000) 2019-10-13T07:49:21.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:21.544+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:21.551569+0000) 2019-10-13T07:49:23.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:23.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:23.400440+0000) 2019-10-13T07:49:24.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:24.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.275003+0000) 2019-10-13T07:49:24.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:24.341+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.343127+0000) 2019-10-13T07:49:24.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:24.341+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.343762+0000) 2019-10-13T07:49:24.346 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:24.341+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.345178+0000) 2019-10-13T07:49:24.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:24.341+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:24.347 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:24.341+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.345526+0000) 2019-10-13T07:49:24.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:24.340+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.345887+0000) 2019-10-13T07:49:24.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:24.345+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.347184+0000) 2019-10-13T07:49:24.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:24.345+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.348441+0000) 2019-10-13T07:49:24.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:24.348+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:24.352561+0000) 2019-10-13T07:49:27.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:27.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:27.115159+0000) 2019-10-13T07:49:27.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:27.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:27.529435+0000) 2019-10-13T07:49:29.349 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:29.345+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.348017+0000) 2019-10-13T07:49:29.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:29.345+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.348585+0000) 2019-10-13T07:49:29.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:29.345+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.349178+0000) 2019-10-13T07:49:29.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:29.345+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.349457+0000) 2019-10-13T07:49:29.351 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:29.344+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.349259+0000) 2019-10-13T07:49:29.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:29.349+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.350125+0000) 2019-10-13T07:49:29.352 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:29.349+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.350089+0000) 2019-10-13T07:49:29.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:29.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:29.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:29.344+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:29.351694+0000) 2019-10-13T07:49:29.368 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:49:29.369 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:29.372 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:49:29.372 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:30.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:30.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:30.656723+0000) 2019-10-13T07:49:30.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:30.753+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:30.754627+0000) 2019-10-13T07:49:31.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:31.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:31.551280+0000) 2019-10-13T07:49:31.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:31.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:31.551774+0000) 2019-10-13T07:49:33.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:33.404+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:33.406973+0000) 2019-10-13T07:49:34.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:34.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.275199+0000) 2019-10-13T07:49:34.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:34.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:34.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:34.353+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.357138+0000) 2019-10-13T07:49:34.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:34.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.357626+0000) 2019-10-13T07:49:34.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:34.357+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.358752+0000) 2019-10-13T07:49:34.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:34.352+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.357425+0000) 2019-10-13T07:49:34.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:34.357+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.359165+0000) 2019-10-13T07:49:34.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:34.357+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.359571+0000) 2019-10-13T07:49:34.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:34.357+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.359860+0000) 2019-10-13T07:49:34.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:34.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:34.365060+0000) 2019-10-13T07:49:37.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:37.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:37.115412+0000) 2019-10-13T07:49:37.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:37.525+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:37.529632+0000) 2019-10-13T07:49:39.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:39.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.360415+0000) 2019-10-13T07:49:39.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:39.360+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.361868+0000) 2019-10-13T07:49:39.363 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:39.360+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.362318+0000) 2019-10-13T07:49:39.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:39.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:39.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:39.360+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.362733+0000) 2019-10-13T07:49:39.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:39.356+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.362708+0000) 2019-10-13T07:49:39.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:39.360+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.364201+0000) 2019-10-13T07:49:39.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:39.360+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.364856+0000) 2019-10-13T07:49:39.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:39.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:39.366750+0000) 2019-10-13T07:49:40.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:40.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:40.656879+0000) 2019-10-13T07:49:40.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:40.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:40.754854+0000) 2019-10-13T07:49:41.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:41.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:41.551494+0000) 2019-10-13T07:49:41.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:41.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:41.551978+0000) 2019-10-13T07:49:43.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:43.412+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:43.413358+0000) 2019-10-13T07:49:44.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:44.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.275398+0000) 2019-10-13T07:49:44.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:44.360+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.365015+0000) 2019-10-13T07:49:44.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:44.364+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.365866+0000) 2019-10-13T07:49:44.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:44.364+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:44.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:44.364+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.368517+0000) 2019-10-13T07:49:44.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:44.364+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.368247+0000) 2019-10-13T07:49:44.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:44.368+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.370008+0000) 2019-10-13T07:49:44.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:44.368+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.370765+0000) 2019-10-13T07:49:44.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:44.368+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.371781+0000) 2019-10-13T07:49:44.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:44.372+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:44.376510+0000) 2019-10-13T07:49:47.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:47.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:47.115643+0000) 2019-10-13T07:49:47.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:47.528+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:47.529828+0000) 2019-10-13T07:49:49.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:49.368+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.371139+0000) 2019-10-13T07:49:49.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:49.364+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.371045+0000) 2019-10-13T07:49:49.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:49.368+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.372802+0000) 2019-10-13T07:49:49.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:49.368+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.373058+0000) 2019-10-13T07:49:49.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:49.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:49.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:49.368+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.373404+0000) 2019-10-13T07:49:49.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:49.372+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.373922+0000) 2019-10-13T07:49:49.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:49.372+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.374758+0000) 2019-10-13T07:49:49.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:49.376+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:49.380431+0000) 2019-10-13T07:49:50.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:50.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:50.657090+0000) 2019-10-13T07:49:50.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:50.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:50.755049+0000) 2019-10-13T07:49:51.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:51.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:51.551724+0000) 2019-10-13T07:49:51.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:51.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:51.552172+0000) 2019-10-13T07:49:53.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:53.416+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:53.419690+0000) 2019-10-13T07:49:54.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:54.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.275551+0000) 2019-10-13T07:49:54.381 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:54.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.375856+0000) 2019-10-13T07:49:54.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:54.372+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.377452+0000) 2019-10-13T07:49:54.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:54.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:54.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:54.376+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.379179+0000) 2019-10-13T07:49:54.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:54.376+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.379773+0000) 2019-10-13T07:49:54.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:54.376+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.380288+0000) 2019-10-13T07:49:54.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:54.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.380546+0000) 2019-10-13T07:49:54.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:54.376+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.381348+0000) 2019-10-13T07:49:54.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:54.376+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:54.383287+0000) 2019-10-13T07:49:57.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:57.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:57.115799+0000) 2019-10-13T07:49:57.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:57.528+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:57.530058+0000) 2019-10-13T07:49:59.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:49:59.376+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.381569+0000) 2019-10-13T07:49:59.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:49:59.380+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.382671+0000) 2019-10-13T07:49:59.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:49:59.376+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.382391+0000) 2019-10-13T07:49:59.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:49:59.380+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.383249+0000) 2019-10-13T07:49:59.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:49:59.380+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.384130+0000) 2019-10-13T07:49:59.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:49:59.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:49:59.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:49:59.380+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.385653+0000) 2019-10-13T07:49:59.388 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:49:59.380+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.384585+0000) 2019-10-13T07:49:59.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:49:59.384+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:49:59.390770+0000) 2019-10-13T07:49:59.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:49:59.432 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:49:59.435 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:49:59.435 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:00.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:00.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:00.657301+0000) 2019-10-13T07:50:00.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:00.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:00.755285+0000) 2019-10-13T07:50:01.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:01.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:01.551947+0000) 2019-10-13T07:50:01.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:01.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:01.552333+0000) 2019-10-13T07:50:03.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:03.420+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:03.426038+0000) 2019-10-13T07:50:04.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:04.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.275758+0000) 2019-10-13T07:50:04.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:04.384+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.387412+0000) 2019-10-13T07:50:04.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:04.384+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.388017+0000) 2019-10-13T07:50:04.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:04.384+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.388312+0000) 2019-10-13T07:50:04.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:04.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59527 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:04.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:04.388+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.390978+0000) 2019-10-13T07:50:04.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:04.388+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.391809+0000) 2019-10-13T07:50:04.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:04.388+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.392191+0000) 2019-10-13T07:50:04.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:04.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.393165+0000) 2019-10-13T07:50:04.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:04.388+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:04.394849+0000) 2019-10-13T07:50:07.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:07.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:07.115998+0000) 2019-10-13T07:50:07.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:07.528+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:07.530292+0000) 2019-10-13T07:50:09.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:09.392+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.394163+0000) 2019-10-13T07:50:09.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:09.392+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.394063+0000) 2019-10-13T07:50:09.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:09.392+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.394690+0000) 2019-10-13T07:50:09.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:09.388+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.394763+0000) 2019-10-13T07:50:09.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:09.392+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.395868+0000) 2019-10-13T07:50:09.397 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:09.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:09.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:09.392+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.397228+0000) 2019-10-13T07:50:09.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:09.392+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.397058+0000) 2019-10-13T07:50:09.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:09.396+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:09.402047+0000) 2019-10-13T07:50:10.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:10.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:10.657496+0000) 2019-10-13T07:50:10.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:10.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:10.755515+0000) 2019-10-13T07:50:11.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:11.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:11.552165+0000) 2019-10-13T07:50:11.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:11.552+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:11.552547+0000) 2019-10-13T07:50:13.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:13.432+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:13.432535+0000) 2019-10-13T07:50:14.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:14.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.275976+0000) 2019-10-13T07:50:14.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:14.396+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.399686+0000) 2019-10-13T07:50:14.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:14.396+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.400236+0000) 2019-10-13T07:50:14.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:14.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:14.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:14.400+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.401770+0000) 2019-10-13T07:50:14.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:14.400+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.401893+0000) 2019-10-13T07:50:14.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:14.396+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.402581+0000) 2019-10-13T07:50:14.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:14.400+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.402521+0000) 2019-10-13T07:50:14.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:14.400+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.404651+0000) 2019-10-13T07:50:14.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:14.404+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:14.408697+0000) 2019-10-13T07:50:17.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:17.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:17.116199+0000) 2019-10-13T07:50:17.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:17.528+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:17.530491+0000) 2019-10-13T07:50:19.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:19.400+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.404145+0000) 2019-10-13T07:50:19.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:19.404+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.405892+0000) 2019-10-13T07:50:19.408 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:19.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:19.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:19.404+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.406505+0000) 2019-10-13T07:50:19.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:19.404+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.406635+0000) 2019-10-13T07:50:19.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:19.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.406935+0000) 2019-10-13T07:50:19.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:19.404+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.407218+0000) 2019-10-13T07:50:19.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:19.404+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.407359+0000) 2019-10-13T07:50:19.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:19.412+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:19.413054+0000) 2019-10-13T07:50:20.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:20.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:20.657732+0000) 2019-10-13T07:50:20.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:20.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:20.755669+0000) 2019-10-13T07:50:21.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:21.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:21.552358+0000) 2019-10-13T07:50:21.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:21.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:21.552748+0000) 2019-10-13T07:50:23.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:23.435+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:23.439090+0000) 2019-10-13T07:50:24.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:24.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.276176+0000) 2019-10-13T07:50:24.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:24.408+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.410130+0000) 2019-10-13T07:50:24.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:24.408+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.410574+0000) 2019-10-13T07:50:24.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:24.408+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.411061+0000) 2019-10-13T07:50:24.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:24.408+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.411597+0000) 2019-10-13T07:50:24.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:24.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:24.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:24.408+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.412621+0000) 2019-10-13T07:50:24.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:24.411+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.412173+0000) 2019-10-13T07:50:24.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:24.408+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.413709+0000) 2019-10-13T07:50:24.420 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:24.415+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:24.418790+0000) 2019-10-13T07:50:27.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:27.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:27.116437+0000) 2019-10-13T07:50:27.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:27.528+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:27.530725+0000) 2019-10-13T07:50:29.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:29.412+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.414631+0000) 2019-10-13T07:50:29.417 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:29.412+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.416061+0000) 2019-10-13T07:50:29.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:29.412+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.416617+0000) 2019-10-13T07:50:29.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:29.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:29.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:29.412+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.417227+0000) 2019-10-13T07:50:29.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:29.416+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.417734+0000) 2019-10-13T07:50:29.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:29.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.417857+0000) 2019-10-13T07:50:29.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:29.416+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.418044+0000) 2019-10-13T07:50:29.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:29.423+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:29.424425+0000) 2019-10-13T07:50:29.562 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:50:29.562 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:29.565 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:50:29.566 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:30.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:30.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:30.657978+0000) 2019-10-13T07:50:30.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:30.752+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:30.755892+0000) 2019-10-13T07:50:31.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:31.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:31.552553+0000) 2019-10-13T07:50:31.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:31.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:31.552950+0000) 2019-10-13T07:50:33.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:33.443+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:33.445541+0000) 2019-10-13T07:50:34.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:34.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.276374+0000) 2019-10-13T07:50:34.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:34.416+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.420896+0000) 2019-10-13T07:50:34.422 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:34.416+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.421146+0000) 2019-10-13T07:50:34.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:34.420+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.422534+0000) 2019-10-13T07:50:34.424 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:34.419+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.422353+0000) 2019-10-13T07:50:34.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:34.420+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.423124+0000) 2019-10-13T07:50:34.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:34.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:34.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:34.420+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.423590+0000) 2019-10-13T07:50:34.425 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:34.420+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.424055+0000) 2019-10-13T07:50:34.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:34.427+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:34.428686+0000) 2019-10-13T07:50:37.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:37.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:37.116667+0000) 2019-10-13T07:50:37.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:37.527+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:37.530938+0000) 2019-10-13T07:50:39.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:39.423+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.427350+0000) 2019-10-13T07:50:39.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:39.423+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.427495+0000) 2019-10-13T07:50:39.430 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:39.423+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.427941+0000) 2019-10-13T07:50:39.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:39.423+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.428388+0000) 2019-10-13T07:50:39.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:39.423+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.428737+0000) 2019-10-13T07:50:39.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:39.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.429004+0000) 2019-10-13T07:50:39.431 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:39.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:39.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:39.427+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.429917+0000) 2019-10-13T07:50:39.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:39.431+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:39.432513+0000) 2019-10-13T07:50:40.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:40.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:40.658179+0000) 2019-10-13T07:50:40.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:40.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:40.756093+0000) 2019-10-13T07:50:41.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:41.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:41.552752+0000) 2019-10-13T07:50:41.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:41.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:41.553153+0000) 2019-10-13T07:50:43.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:43.447+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:43.451994+0000) 2019-10-13T07:50:44.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:44.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.276578+0000) 2019-10-13T07:50:44.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:44.427+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.432864+0000) 2019-10-13T07:50:44.435 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:44.431+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.432696+0000) 2019-10-13T07:50:44.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:44.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.433936+0000) 2019-10-13T07:50:44.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:44.431+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.434272+0000) 2019-10-13T07:50:44.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:44.431+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.434868+0000) 2019-10-13T07:50:44.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:44.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:44.438 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:44.431+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.435620+0000) 2019-10-13T07:50:44.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:44.431+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.436386+0000) 2019-10-13T07:50:44.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:44.435+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:44.439897+0000) 2019-10-13T07:50:47.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:47.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:47.116855+0000) 2019-10-13T07:50:47.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:47.527+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:47.531131+0000) 2019-10-13T07:50:49.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:49.435+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.437227+0000) 2019-10-13T07:50:49.439 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:49.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.438425+0000) 2019-10-13T07:50:49.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:49.435+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.439978+0000) 2019-10-13T07:50:49.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:49.435+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.440536+0000) 2019-10-13T07:50:49.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:49.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:49.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:49.435+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.440626+0000) 2019-10-13T07:50:49.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:49.435+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.441098+0000) 2019-10-13T07:50:49.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:49.439+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.442410+0000) 2019-10-13T07:50:49.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:49.447+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:49.448634+0000) 2019-10-13T07:50:50.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:50.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:50.658385+0000) 2019-10-13T07:50:50.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:50.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:50.756340+0000) 2019-10-13T07:50:51.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:51.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:51.552943+0000) 2019-10-13T07:50:51.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:51.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:51.553374+0000) 2019-10-13T07:50:53.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:53.455+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:53.458608+0000) 2019-10-13T07:50:54.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:54.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.276816+0000) 2019-10-13T07:50:54.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:54.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.443253+0000) 2019-10-13T07:50:54.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:54.439+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.444854+0000) 2019-10-13T07:50:54.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:54.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:54.447 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:54.439+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.445567+0000) 2019-10-13T07:50:54.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:54.443+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.445879+0000) 2019-10-13T07:50:54.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:54.443+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.446325+0000) 2019-10-13T07:50:54.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:54.443+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.446943+0000) 2019-10-13T07:50:54.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:54.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.447328+0000) 2019-10-13T07:50:54.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:54.447+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:54.450866+0000) 2019-10-13T07:50:57.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:57.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:57.117046+0000) 2019-10-13T07:50:57.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:57.527+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:57.531323+0000) 2019-10-13T07:50:59.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:50:59.447+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.448394+0000) 2019-10-13T07:50:59.450 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:50:59.443+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.449352+0000) 2019-10-13T07:50:59.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:50:59.447+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.450266+0000) 2019-10-13T07:50:59.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:50:59.447+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.450608+0000) 2019-10-13T07:50:59.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:50:59.447+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.451129+0000) 2019-10-13T07:50:59.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:50:59.447+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:50:59.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:50:59.447+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.451533+0000) 2019-10-13T07:50:59.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:50:59.447+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.451828+0000) 2019-10-13T07:50:59.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:50:59.455+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:50:59.458547+0000) 2019-10-13T07:50:59.654 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:50:59.654 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:50:59.658 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:50:59.658 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:00.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:00.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:00.658610+0000) 2019-10-13T07:51:00.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:00.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:00.756577+0000) 2019-10-13T07:51:01.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:01.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:01.553136+0000) 2019-10-13T07:51:01.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:01.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:01.553596+0000) 2019-10-13T07:51:03.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:03.463+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:03.464944+0000) 2019-10-13T07:51:04.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:04.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.277015+0000) 2019-10-13T07:51:04.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:04.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:04.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:04.451+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.455886+0000) 2019-10-13T07:51:04.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:04.451+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.456178+0000) 2019-10-13T07:51:04.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:04.451+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.457016+0000) 2019-10-13T07:51:04.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:04.455+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.457682+0000) 2019-10-13T07:51:04.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:04.451+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.457268+0000) 2019-10-13T07:51:04.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:04.455+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.458899+0000) 2019-10-13T07:51:04.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:04.455+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.459082+0000) 2019-10-13T07:51:04.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:04.459+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:04.463963+0000) 2019-10-13T07:51:07.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:07.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:07.117250+0000) 2019-10-13T07:51:07.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:07.527+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:07.531556+0000) 2019-10-13T07:51:09.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:09.455+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.459143+0000) 2019-10-13T07:51:09.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:09.455+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.459756+0000) 2019-10-13T07:51:09.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:09.455+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.460707+0000) 2019-10-13T07:51:09.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:09.455+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.461284+0000) 2019-10-13T07:51:09.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:09.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:09.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:09.455+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.461506+0000) 2019-10-13T07:51:09.463 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:09.459+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.462033+0000) 2019-10-13T07:51:09.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:09.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.462952+0000) 2019-10-13T07:51:09.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:09.463+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:09.466271+0000) 2019-10-13T07:51:10.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:10.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:10.658818+0000) 2019-10-13T07:51:10.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:10.751+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:10.756775+0000) 2019-10-13T07:51:11.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:11.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:11.553341+0000) 2019-10-13T07:51:11.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:11.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:11.553800+0000) 2019-10-13T07:51:13.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:13.471+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:13.471431+0000) 2019-10-13T07:51:14.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:14.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.277234+0000) 2019-10-13T07:51:14.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:14.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.465240+0000) 2019-10-13T07:51:14.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:14.463+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.466005+0000) 2019-10-13T07:51:14.467 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:14.463+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.466412+0000) 2019-10-13T07:51:14.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:14.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:14.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:14.463+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.466949+0000) 2019-10-13T07:51:14.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:14.463+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.468096+0000) 2019-10-13T07:51:14.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:14.463+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.469074+0000) 2019-10-13T07:51:14.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:14.467+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.469927+0000) 2019-10-13T07:51:14.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:14.467+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:14.471568+0000) 2019-10-13T07:51:17.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:17.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:17.117474+0000) 2019-10-13T07:51:17.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:17.527+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:17.531747+0000) 2019-10-13T07:51:19.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:19.467+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.469834+0000) 2019-10-13T07:51:19.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:19.467+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.472587+0000) 2019-10-13T07:51:19.474 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:19.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:19.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:19.471+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.473916+0000) 2019-10-13T07:51:19.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:19.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.474151+0000) 2019-10-13T07:51:19.475 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:19.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.474794+0000) 2019-10-13T07:51:19.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:19.471+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.475578+0000) 2019-10-13T07:51:19.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:19.471+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.475054+0000) 2019-10-13T07:51:19.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:19.475+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:19.478125+0000) 2019-10-13T07:51:20.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:20.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:20.659022+0000) 2019-10-13T07:51:20.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:20.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:20.756974+0000) 2019-10-13T07:51:21.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:21.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:21.553540+0000) 2019-10-13T07:51:21.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:21.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:21.553994+0000) 2019-10-13T07:51:23.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:23.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:23.477785+0000) 2019-10-13T07:51:24.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:24.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.277440+0000) 2019-10-13T07:51:24.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:24.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.476110+0000) 2019-10-13T07:51:24.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:24.471+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.476549+0000) 2019-10-13T07:51:24.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:24.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.477192+0000) 2019-10-13T07:51:24.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:24.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60029 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:24.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:24.475+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.479555+0000) 2019-10-13T07:51:24.481 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:24.475+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.480286+0000) 2019-10-13T07:51:24.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:24.475+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.480049+0000) 2019-10-13T07:51:24.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:24.475+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.481303+0000) 2019-10-13T07:51:24.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:24.483+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:24.487573+0000) 2019-10-13T07:51:27.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:27.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:27.117693+0000) 2019-10-13T07:51:27.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:27.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:27.531945+0000) 2019-10-13T07:51:29.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:29.479+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.481728+0000) 2019-10-13T07:51:29.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:29.479+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.482964+0000) 2019-10-13T07:51:29.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:29.478+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.483046+0000) 2019-10-13T07:51:29.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:29.479+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.483254+0000) 2019-10-13T07:51:29.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:29.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:29.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:29.479+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.484590+0000) 2019-10-13T07:51:29.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:29.479+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.485286+0000) 2019-10-13T07:51:29.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:29.479+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.485577+0000) 2019-10-13T07:51:29.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:29.486+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:29.491776+0000) 2019-10-13T07:51:29.724 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:51:29.725 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:29.729 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:51:29.729 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:30.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:30.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:30.659223+0000) 2019-10-13T07:51:31.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:31.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:31.554199+0000) 2019-10-13T07:51:31.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:30.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:30.757178+0000) 2019-10-13T07:51:31.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:31.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:31.553736+0000) 2019-10-13T07:51:33.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:33.482+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:33.484163+0000) 2019-10-13T07:51:34.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:34.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.277662+0000) 2019-10-13T07:51:34.488 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:34.482+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.486660+0000) 2019-10-13T07:51:34.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:34.482+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.487451+0000) 2019-10-13T07:51:34.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:34.482+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.488258+0000) 2019-10-13T07:51:34.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:34.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:34.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:34.482+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.489148+0000) 2019-10-13T07:51:34.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:34.482+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.489391+0000) 2019-10-13T07:51:34.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:34.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.490716+0000) 2019-10-13T07:51:34.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:34.486+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.491398+0000) 2019-10-13T07:51:34.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:34.494+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:34.496946+0000) 2019-10-13T07:51:37.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:37.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:37.117887+0000) 2019-10-13T07:51:37.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:37.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:37.532168+0000) 2019-10-13T07:51:39.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:39.486+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.491491+0000) 2019-10-13T07:51:39.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:39.486+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.492867+0000) 2019-10-13T07:51:39.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:39.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.493894+0000) 2019-10-13T07:51:39.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:39.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:39.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:39.490+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.494213+0000) 2019-10-13T07:51:39.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:39.490+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.494559+0000) 2019-10-13T07:51:39.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:39.490+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.495425+0000) 2019-10-13T07:51:39.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:39.494+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.498043+0000) 2019-10-13T07:51:39.501 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:39.490+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:39.496127+0000) 2019-10-13T07:51:40.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:40.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:40.659456+0000) 2019-10-13T07:51:40.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:40.754+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:40.757431+0000) 2019-10-13T07:51:41.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:41.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:41.553967+0000) 2019-10-13T07:51:41.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:41.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:41.554394+0000) 2019-10-13T07:51:43.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:43.490+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:43.493829+0000) 2019-10-13T07:51:44.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:44.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.277872+0000) 2019-10-13T07:51:44.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:44.494+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.497838+0000) 2019-10-13T07:51:44.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:44.494+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.498386+0000) 2019-10-13T07:51:44.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:44.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:44.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:44.494+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.500376+0000) 2019-10-13T07:51:44.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:44.498+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.500243+0000) 2019-10-13T07:51:44.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:44.494+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.501151+0000) 2019-10-13T07:51:44.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:44.494+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.501652+0000) 2019-10-13T07:51:44.503 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:44.498+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.501765+0000) 2019-10-13T07:51:44.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:44.498+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:44.503231+0000) 2019-10-13T07:51:47.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:47.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:47.118122+0000) 2019-10-13T07:51:47.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:47.530+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:47.532403+0000) 2019-10-13T07:51:49.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:49.498+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.503588+0000) 2019-10-13T07:51:49.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:49.498+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.503947+0000) 2019-10-13T07:51:49.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:49.498+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.504578+0000) 2019-10-13T07:51:49.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:49.498+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.505053+0000) 2019-10-13T07:51:49.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:49.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:49.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:49.502+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.505952+0000) 2019-10-13T07:51:49.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:49.502+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.505937+0000) 2019-10-13T07:51:49.508 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:49.502+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.506932+0000) 2019-10-13T07:51:49.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:49.506+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:49.512038+0000) 2019-10-13T07:51:50.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:50.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:50.659730+0000) 2019-10-13T07:51:50.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:50.750+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:50.757635+0000) 2019-10-13T07:51:51.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:51.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:51.554157+0000) 2019-10-13T07:51:51.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:51.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:51.554594+0000) 2019-10-13T07:51:53.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:53.498+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:53.500229+0000) 2019-10-13T07:51:54.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:54.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.278072+0000) 2019-10-13T07:51:54.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:54.502+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.508310+0000) 2019-10-13T07:51:54.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:54.506+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.510626+0000) 2019-10-13T07:51:54.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:54.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:54.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:54.506+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.511117+0000) 2019-10-13T07:51:54.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:54.506+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.512063+0000) 2019-10-13T07:51:54.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:54.506+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.512907+0000) 2019-10-13T07:51:54.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:54.510+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.512775+0000) 2019-10-13T07:51:54.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:54.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.514481+0000) 2019-10-13T07:51:54.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:54.510+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:54.515103+0000) 2019-10-13T07:51:57.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:57.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:57.118398+0000) 2019-10-13T07:51:57.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:57.530+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:57.532635+0000) 2019-10-13T07:51:59.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:51:59.510+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.514053+0000) 2019-10-13T07:51:59.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:51:59.510+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.514643+0000) 2019-10-13T07:51:59.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:51:59.510+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.514449+0000) 2019-10-13T07:51:59.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:51:59.510+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.515429+0000) 2019-10-13T07:51:59.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:51:59.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:51:59.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:51:59.510+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.516979+0000) 2019-10-13T07:51:59.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:51:59.514+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.518853+0000) 2019-10-13T07:51:59.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:51:59.514+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.517897+0000) 2019-10-13T07:51:59.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:51:59.518+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:51:59.524029+0000) 2019-10-13T07:51:59.876 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:51:59.876 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:51:59.879 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:51:59.879 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:00.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:00.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:00.659950+0000) 2019-10-13T07:52:00.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:00.754+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:00.757837+0000) 2019-10-13T07:52:01.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:01.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:01.554371+0000) 2019-10-13T07:52:02.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:01.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:01.554837+0000) 2019-10-13T07:52:03.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:03.506+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:03.510058+0000) 2019-10-13T07:52:04.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:04.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.278301+0000) 2019-10-13T07:52:04.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:04.514+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.519691+0000) 2019-10-13T07:52:04.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:04.514+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.520509+0000) 2019-10-13T07:52:04.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:04.514+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.520918+0000) 2019-10-13T07:52:04.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:04.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:04.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:04.518+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.522409+0000) 2019-10-13T07:52:04.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:04.518+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.523096+0000) 2019-10-13T07:52:04.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:04.518+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.523429+0000) 2019-10-13T07:52:04.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:04.518+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.523862+0000) 2019-10-13T07:52:04.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:04.522+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:04.526813+0000) 2019-10-13T07:52:07.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:07.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:07.118613+0000) 2019-10-13T07:52:07.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:07.530+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:07.532838+0000) 2019-10-13T07:52:09.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:09.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.524739+0000) 2019-10-13T07:52:09.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:09.522+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.527611+0000) 2019-10-13T07:52:09.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:09.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:09.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:09.522+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.527982+0000) 2019-10-13T07:52:09.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:09.522+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.528965+0000) 2019-10-13T07:52:09.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:09.526+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.529973+0000) 2019-10-13T07:52:09.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:09.526+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.530401+0000) 2019-10-13T07:52:09.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:09.526+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.529982+0000) 2019-10-13T07:52:09.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:09.534+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:09.536272+0000) 2019-10-13T07:52:10.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:10.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:10.660201+0000) 2019-10-13T07:52:10.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:10.754+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:10.758055+0000) 2019-10-13T07:52:11.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:11.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:11.554594+0000) 2019-10-13T07:52:11.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:11.554+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:11.555067+0000) 2019-10-13T07:52:13.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:13.514+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:13.516402+0000) 2019-10-13T07:52:14.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:14.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.278512+0000) 2019-10-13T07:52:14.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:14.526+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.530555+0000) 2019-10-13T07:52:14.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:14.526+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.531455+0000) 2019-10-13T07:52:14.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:14.526+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.531832+0000) 2019-10-13T07:52:14.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:14.526+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.532957+0000) 2019-10-13T07:52:14.534 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:14.526+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.532825+0000) 2019-10-13T07:52:14.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:14.526+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.533349+0000) 2019-10-13T07:52:14.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:14.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:14.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:14.530+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.534098+0000) 2019-10-13T07:52:14.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:14.538+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:14.540473+0000) 2019-10-13T07:52:17.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:17.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:17.118813+0000) 2019-10-13T07:52:17.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:17.530+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:17.533057+0000) 2019-10-13T07:52:19.538 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:19.530+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.537333+0000) 2019-10-13T07:52:19.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:19.534+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.537982+0000) 2019-10-13T07:52:19.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:19.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.538650+0000) 2019-10-13T07:52:19.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:19.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:19.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:19.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.539113+0000) 2019-10-13T07:52:19.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:19.534+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.539993+0000) 2019-10-13T07:52:19.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:19.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.540604+0000) 2019-10-13T07:52:19.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:19.534+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.541001+0000) 2019-10-13T07:52:19.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:19.542+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:19.544177+0000) 2019-10-13T07:52:20.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:20.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:20.660397+0000) 2019-10-13T07:52:20.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:20.758+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:20.758278+0000) 2019-10-13T07:52:21.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:21.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:21.554790+0000) 2019-10-13T07:52:21.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:21.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:21.555293+0000) 2019-10-13T07:52:23.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:23.522+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:23.522985+0000) 2019-10-13T07:52:24.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:24.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.278709+0000) 2019-10-13T07:52:24.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:24.538+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.543057+0000) 2019-10-13T07:52:24.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:24.538+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.543959+0000) 2019-10-13T07:52:24.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:24.538+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.544340+0000) 2019-10-13T07:52:24.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:24.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:24.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:24.538+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.544635+0000) 2019-10-13T07:52:24.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:24.538+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.545102+0000) 2019-10-13T07:52:24.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:24.542+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.544318+0000) 2019-10-13T07:52:24.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:24.542+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.545836+0000) 2019-10-13T07:52:24.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:24.546+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:24.551442+0000) 2019-10-13T07:52:27.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:27.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:27.119020+0000) 2019-10-13T07:52:27.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:27.526+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:27.533252+0000) 2019-10-13T07:52:29.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:29.542+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.548504+0000) 2019-10-13T07:52:29.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:29.542+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.549221+0000) 2019-10-13T07:52:29.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:29.546+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.549386+0000) 2019-10-13T07:52:29.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:29.542+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.549478+0000) 2019-10-13T07:52:29.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:29.546+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.550227+0000) 2019-10-13T07:52:29.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:29.546+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.550726+0000) 2019-10-13T07:52:29.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:29.546+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.550924+0000) 2019-10-13T07:52:29.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:29.546+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:29.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:29.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:29.557505+0000) 2019-10-13T07:52:29.993 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:52:29.993 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:29.996 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:52:29.997 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:52:30.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:30.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:30.660623+0000) 2019-10-13T07:52:30.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:30.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:30.758477+0000) 2019-10-13T07:52:31.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:31.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:31.555020+0000) 2019-10-13T07:52:31.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:31.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:31.555495+0000) 2019-10-13T07:52:33.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:33.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:33.529344+0000) 2019-10-13T07:52:34.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:34.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.278910+0000) 2019-10-13T07:52:34.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:34.549+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.554020+0000) 2019-10-13T07:52:34.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:34.549+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.553954+0000) 2019-10-13T07:52:34.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:34.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.555542+0000) 2019-10-13T07:52:34.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:34.549+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.555837+0000) 2019-10-13T07:52:34.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:34.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:34.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:34.549+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.556443+0000) 2019-10-13T07:52:34.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:34.549+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.556468+0000) 2019-10-13T07:52:34.559 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:34.549+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.556955+0000) 2019-10-13T07:52:34.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:34.561+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:34.565083+0000) 2019-10-13T07:52:37.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:37.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:37.119254+0000) 2019-10-13T07:52:37.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:37.529+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:37.533456+0000) 2019-10-13T07:52:39.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:39.553+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.560012+0000) 2019-10-13T07:52:39.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:39.553+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.560881+0000) 2019-10-13T07:52:39.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:39.557+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.560730+0000) 2019-10-13T07:52:39.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:39.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:39.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:39.557+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.562709+0000) 2019-10-13T07:52:39.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:39.557+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.563525+0000) 2019-10-13T07:52:39.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:39.557+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.563977+0000) 2019-10-13T07:52:39.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:39.557+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.564672+0000) 2019-10-13T07:52:39.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:39.565+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:39.571558+0000) 2019-10-13T07:52:40.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:40.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:40.660857+0000) 2019-10-13T07:52:40.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:40.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:40.758680+0000) 2019-10-13T07:52:41.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:41.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:41.555213+0000) 2019-10-13T07:52:41.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:41.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:41.555786+0000) 2019-10-13T07:52:43.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:43.529+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:43.535887+0000) 2019-10-13T07:52:44.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:44.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.279140+0000) 2019-10-13T07:52:44.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:44.557+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.564734+0000) 2019-10-13T07:52:44.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:44.557+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.565173+0000) 2019-10-13T07:52:44.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:44.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:44.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:44.561+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.567360+0000) 2019-10-13T07:52:44.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:44.565+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.568079+0000) 2019-10-13T07:52:44.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:44.561+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.568956+0000) 2019-10-13T07:52:44.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:44.561+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.569529+0000) 2019-10-13T07:52:44.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:44.561+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.569644+0000) 2019-10-13T07:52:44.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:44.569+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:44.574523+0000) 2019-10-13T07:52:47.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:47.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:47.119493+0000) 2019-10-13T07:52:47.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:47.529+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:47.533659+0000) 2019-10-13T07:52:49.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:49.569+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.571466+0000) 2019-10-13T07:52:49.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:49.569+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.571819+0000) 2019-10-13T07:52:49.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:49.569+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.572056+0000) 2019-10-13T07:52:49.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:49.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.572454+0000) 2019-10-13T07:52:49.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:49.569+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.572450+0000) 2019-10-13T07:52:49.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:49.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:49.576 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:49.569+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.573558+0000) 2019-10-13T07:52:49.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:49.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.575581+0000) 2019-10-13T07:52:49.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:49.573+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:49.578016+0000) 2019-10-13T07:52:50.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:50.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:50.661068+0000) 2019-10-13T07:52:50.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:50.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:50.758879+0000) 2019-10-13T07:52:51.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:51.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:51.555408+0000) 2019-10-13T07:52:51.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:51.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:51.556002+0000) 2019-10-13T07:52:53.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:53.541+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:53.545538+0000) 2019-10-13T07:52:54.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:54.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.279344+0000) 2019-10-13T07:52:54.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:54.573+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.576066+0000) 2019-10-13T07:52:54.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:54.573+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.576591+0000) 2019-10-13T07:52:54.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:54.573+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.577707+0000) 2019-10-13T07:52:54.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:54.577+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.578510+0000) 2019-10-13T07:52:54.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:54.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:54.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:54.577+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.578892+0000) 2019-10-13T07:52:54.580 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:54.577+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.579438+0000) 2019-10-13T07:52:54.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:54.577+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.579688+0000) 2019-10-13T07:52:54.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:54.577+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:54.583012+0000) 2019-10-13T07:52:57.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:57.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:57.119689+0000) 2019-10-13T07:52:57.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:57.533+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:57.533863+0000) 2019-10-13T07:52:59.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:52:59.581+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.582028+0000) 2019-10-13T07:52:59.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:52:59.581+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.582481+0000) 2019-10-13T07:52:59.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:52:59.581+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.582913+0000) 2019-10-13T07:52:59.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:52:59.581+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.583293+0000) 2019-10-13T07:52:59.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:52:59.577+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.583032+0000) 2019-10-13T07:52:59.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:52:59.581+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.584145+0000) 2019-10-13T07:52:59.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:52:59.581+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.584176+0000) 2019-10-13T07:52:59.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:52:59.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:52:59.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:52:59.585+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:52:59.591445+0000) 2019-10-13T07:53:00.147 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:53:00.147 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:00.151 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:53:00.151 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:00.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:00.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:00.661265+0000) 2019-10-13T07:53:00.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:00.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:00.759078+0000) 2019-10-13T07:53:01.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:01.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:01.555563+0000) 2019-10-13T07:53:01.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:01.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:01.556206+0000) 2019-10-13T07:53:03.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:03.549+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:03.552037+0000) 2019-10-13T07:53:04.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:04.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.279555+0000) 2019-10-13T07:53:04.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:04.581+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.586637+0000) 2019-10-13T07:53:04.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:04.585+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.589029+0000) 2019-10-13T07:53:04.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:04.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:04.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:04.585+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.589631+0000) 2019-10-13T07:53:04.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:04.589+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.589961+0000) 2019-10-13T07:53:04.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:04.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.590542+0000) 2019-10-13T07:53:04.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:04.589+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.591057+0000) 2019-10-13T07:53:04.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:04.589+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.591884+0000) 2019-10-13T07:53:04.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:04.589+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:04.596021+0000) 2019-10-13T07:53:07.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:07.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:07.119887+0000) 2019-10-13T07:53:07.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:07.533+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:07.534079+0000) 2019-10-13T07:53:09.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:09.589+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.593377+0000) 2019-10-13T07:53:12.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:09.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.596671+0000) 2019-10-13T07:53:12.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:11.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:11.556403+0000) 2019-10-13T07:53:12.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:09.601+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.604159+0000) 2019-10-13T07:53:12.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:09.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:12.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:09.593+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.595753+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:10.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:10.661468+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:09.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.595434+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:09.593+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.596626+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:11.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:11.555754+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:09.597+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.598487+0000) 2019-10-13T07:53:12.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:10.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:10.759316+0000) 2019-10-13T07:53:12.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:09.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:09.598468+0000) 2019-10-13T07:53:13.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:13.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:13.558451+0000) 2019-10-13T07:53:14.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:14.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.279711+0000) 2019-10-13T07:53:14.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:14.597+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.598955+0000) 2019-10-13T07:53:14.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:14.597+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.599353+0000) 2019-10-13T07:53:14.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:14.597+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.599630+0000) 2019-10-13T07:53:14.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:14.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:14.601 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:14.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.600308+0000) 2019-10-13T07:53:14.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:14.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.600630+0000) 2019-10-13T07:53:14.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:14.597+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.600749+0000) 2019-10-13T07:53:14.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:14.597+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.601432+0000) 2019-10-13T07:53:14.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:14.601+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:14.606652+0000) 2019-10-13T07:53:17.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:17.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:17.120086+0000) 2019-10-13T07:53:17.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:17.533+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:17.534315+0000) 2019-10-13T07:53:19.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:19.601+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.602764+0000) 2019-10-13T07:53:19.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:19.601+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.603470+0000) 2019-10-13T07:53:19.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:19.601+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.604126+0000) 2019-10-13T07:53:19.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:19.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60748 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:19.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:19.601+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.605674+0000) 2019-10-13T07:53:19.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:19.605+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.606641+0000) 2019-10-13T07:53:19.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:19.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.606512+0000) 2019-10-13T07:53:19.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:19.605+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.607315+0000) 2019-10-13T07:53:19.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:19.609+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:19.613114+0000) 2019-10-13T07:53:20.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:20.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:20.661705+0000) 2019-10-13T07:53:20.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:20.757+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:20.759518+0000) 2019-10-13T07:53:21.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:21.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:21.555963+0000) 2019-10-13T07:53:21.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:21.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:21.556629+0000) 2019-10-13T07:53:23.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:23.561+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:23.564748+0000) 2019-10-13T07:53:24.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:24.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.279924+0000) 2019-10-13T07:53:24.609 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:24.605+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.608237+0000) 2019-10-13T07:53:24.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:24.605+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.608608+0000) 2019-10-13T07:53:24.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:24.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:24.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:24.609+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.611464+0000) 2019-10-13T07:53:24.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:24.609+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.612219+0000) 2019-10-13T07:53:24.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:24.609+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.613685+0000) 2019-10-13T07:53:24.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:24.613+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.614243+0000) 2019-10-13T07:53:24.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:24.609+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.614477+0000) 2019-10-13T07:53:24.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:24.613+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:24.619866+0000) 2019-10-13T07:53:27.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:27.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:27.120300+0000) 2019-10-13T07:53:27.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:27.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:27.534530+0000) 2019-10-13T07:53:29.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:29.612+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.614142+0000) 2019-10-13T07:53:29.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:29.612+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.615513+0000) 2019-10-13T07:53:29.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:29.612+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.616466+0000) 2019-10-13T07:53:29.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:29.613+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.616146+0000) 2019-10-13T07:53:29.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:29.612+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.616080+0000) 2019-10-13T07:53:29.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:29.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:29.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:29.612+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.617305+0000) 2019-10-13T07:53:29.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:29.616+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.617924+0000) 2019-10-13T07:53:29.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:29.617+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:29.623260+0000) 2019-10-13T07:53:30.278 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:53:30.278 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:30.282 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:53:30.282 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:53:30.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:30.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:30.661966+0000) 2019-10-13T07:53:30.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:30.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:30.759723+0000) 2019-10-13T07:53:31.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:31.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:31.556156+0000) 2019-10-13T07:53:31.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:31.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:31.556863+0000) 2019-10-13T07:53:33.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:33.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:33.571221+0000) 2019-10-13T07:53:34.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:34.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.280159+0000) 2019-10-13T07:53:34.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:34.616+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.621026+0000) 2019-10-13T07:53:34.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:34.616+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.621365+0000) 2019-10-13T07:53:34.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:34.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:34.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:34.620+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.622653+0000) 2019-10-13T07:53:34.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:34.617+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.622274+0000) 2019-10-13T07:53:34.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:34.620+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.623310+0000) 2019-10-13T07:53:34.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:34.620+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.623971+0000) 2019-10-13T07:53:34.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:34.624+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.626496+0000) 2019-10-13T07:53:34.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:34.625+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:34.628617+0000) 2019-10-13T07:53:37.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:37.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:37.120514+0000) 2019-10-13T07:53:37.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:37.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:37.534766+0000) 2019-10-13T07:53:39.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:39.620+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.625398+0000) 2019-10-13T07:53:39.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:39.624+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.625899+0000) 2019-10-13T07:53:39.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:39.624+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.626036+0000) 2019-10-13T07:53:39.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:39.624+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.626835+0000) 2019-10-13T07:53:39.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:39.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:39.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:39.621+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.626920+0000) 2019-10-13T07:53:39.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:39.624+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.628126+0000) 2019-10-13T07:53:39.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:39.624+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.628285+0000) 2019-10-13T07:53:39.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:39.629+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:39.635403+0000) 2019-10-13T07:53:40.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:40.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:40.662174+0000) 2019-10-13T07:53:40.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:40.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:40.759924+0000) 2019-10-13T07:53:41.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:41.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:41.556378+0000) 2019-10-13T07:53:41.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:41.552+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:41.557088+0000) 2019-10-13T07:53:43.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:43.572+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:43.577665+0000) 2019-10-13T07:53:44.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:44.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.280359+0000) 2019-10-13T07:53:44.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:44.628+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.631424+0000) 2019-10-13T07:53:44.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:44.628+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.631214+0000) 2019-10-13T07:53:44.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:44.628+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.631853+0000) 2019-10-13T07:53:44.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:44.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:44.634 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:44.628+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.632702+0000) 2019-10-13T07:53:44.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:44.632+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.633883+0000) 2019-10-13T07:53:44.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:44.628+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.633138+0000) 2019-10-13T07:53:44.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:44.628+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.633950+0000) 2019-10-13T07:53:44.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:44.636+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:44.642044+0000) 2019-10-13T07:53:47.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:47.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:47.120716+0000) 2019-10-13T07:53:47.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:47.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:47.534968+0000) 2019-10-13T07:53:49.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:49.632+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.635840+0000) 2019-10-13T07:53:49.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:49.632+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.636765+0000) 2019-10-13T07:53:49.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:49.632+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.636839+0000) 2019-10-13T07:53:49.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:49.632+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.637172+0000) 2019-10-13T07:53:49.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:49.636+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.638355+0000) 2019-10-13T07:53:49.639 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:49.636+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.638638+0000) 2019-10-13T07:53:49.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:49.636+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.639919+0000) 2019-10-13T07:53:49.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:49.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:49.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:49.640+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:49.647361+0000) 2019-10-13T07:53:50.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:50.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:50.662419+0000) 2019-10-13T07:53:50.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:50.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:50.760126+0000) 2019-10-13T07:53:51.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:51.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:51.556574+0000) 2019-10-13T07:53:51.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:51.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:51.557307+0000) 2019-10-13T07:53:53.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:53.580+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:53.584012+0000) 2019-10-13T07:53:54.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:54.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.280583+0000) 2019-10-13T07:53:54.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:54.636+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.641052+0000) 2019-10-13T07:53:54.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:54.640+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.642294+0000) 2019-10-13T07:53:54.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:54.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:54.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:54.640+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.644254+0000) 2019-10-13T07:53:54.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:54.640+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.644730+0000) 2019-10-13T07:53:54.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:54.640+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.645053+0000) 2019-10-13T07:53:54.646 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:54.640+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.645693+0000) 2019-10-13T07:53:54.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:54.644+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.646073+0000) 2019-10-13T07:53:54.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:54.644+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:54.651126+0000) 2019-10-13T07:53:57.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:57.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:57.120913+0000) 2019-10-13T07:53:57.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:57.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:57.535171+0000) 2019-10-13T07:53:59.648 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:53:59.644+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.647466+0000) 2019-10-13T07:53:59.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:53:59.644+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.647806+0000) 2019-10-13T07:53:59.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:53:59.644+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.648346+0000) 2019-10-13T07:53:59.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:53:59.644+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.648323+0000) 2019-10-13T07:53:59.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:53:59.644+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.649116+0000) 2019-10-13T07:53:59.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:53:59.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:53:59.651 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:53:59.644+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.649662+0000) 2019-10-13T07:53:59.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:53:59.648+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.650877+0000) 2019-10-13T07:53:59.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:53:59.652+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:53:59.657550+0000) 2019-10-13T07:54:00.376 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:54:00.376 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:00.381 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:54:00.381 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:00.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:00.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:00.662575+0000) 2019-10-13T07:54:00.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:00.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:00.760331+0000) 2019-10-13T07:54:01.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:01.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:01.556795+0000) 2019-10-13T07:54:01.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:01.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:01.557509+0000) 2019-10-13T07:54:03.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:03.588+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:03.590378+0000) 2019-10-13T07:54:04.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:04.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.280783+0000) 2019-10-13T07:54:04.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:04.648+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.651930+0000) 2019-10-13T07:54:04.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:04.648+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.652790+0000) 2019-10-13T07:54:04.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:04.648+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.653442+0000) 2019-10-13T07:54:04.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:04.652+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.655560+0000) 2019-10-13T07:54:04.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:04.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:04.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:04.652+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.656164+0000) 2019-10-13T07:54:04.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:04.652+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.656869+0000) 2019-10-13T07:54:04.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:04.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.657289+0000) 2019-10-13T07:54:04.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:04.656+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:04.663439+0000) 2019-10-13T07:54:07.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:07.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:07.121115+0000) 2019-10-13T07:54:07.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:07.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:07.535435+0000) 2019-10-13T07:54:09.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:09.656+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.660979+0000) 2019-10-13T07:54:09.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:09.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:09.666 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:09.660+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.661954+0000) 2019-10-13T07:54:09.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:09.656+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.661119+0000) 2019-10-13T07:54:09.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:09.660+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.662545+0000) 2019-10-13T07:54:09.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:09.660+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.663134+0000) 2019-10-13T07:54:09.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:09.660+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.663469+0000) 2019-10-13T07:54:09.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:09.660+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.664079+0000) 2019-10-13T07:54:09.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:09.664+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:09.670557+0000) 2019-10-13T07:54:10.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:10.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:10.662776+0000) 2019-10-13T07:54:10.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:10.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:10.760522+0000) 2019-10-13T07:54:11.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:11.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:11.557010+0000) 2019-10-13T07:54:11.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:11.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:11.557712+0000) 2019-10-13T07:54:13.598 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:13.596+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:13.596746+0000) 2019-10-13T07:54:14.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:14.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.280940+0000) 2019-10-13T07:54:14.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:14.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.664288+0000) 2019-10-13T07:54:15.345 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:14.660+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.664521+0000) 2019-10-13T07:54:15.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:14.668+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.674908+0000) 2019-10-13T07:54:15.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:14.664+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.666592+0000) 2019-10-13T07:54:15.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:14.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:15.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:14.664+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.667013+0000) 2019-10-13T07:54:15.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:14.664+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.667599+0000) 2019-10-13T07:54:15.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:14.664+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.668025+0000) 2019-10-13T07:54:15.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:14.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:14.668736+0000) 2019-10-13T07:54:17.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:17.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:17.121349+0000) 2019-10-13T07:54:17.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:17.532+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:17.535631+0000) 2019-10-13T07:54:19.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:19.668+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.669977+0000) 2019-10-13T07:54:19.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:19.668+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.671227+0000) 2019-10-13T07:54:19.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:19.668+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.671580+0000) 2019-10-13T07:54:19.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:19.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:19.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:19.668+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.671931+0000) 2019-10-13T07:54:19.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:19.668+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.672889+0000) 2019-10-13T07:54:19.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:19.668+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.673058+0000) 2019-10-13T07:54:19.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:19.672+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.674274+0000) 2019-10-13T07:54:19.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:19.676+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:19.680547+0000) 2019-10-13T07:54:20.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:20.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:20.662957+0000) 2019-10-13T07:54:20.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:20.756+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:20.760727+0000) 2019-10-13T07:54:21.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:21.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:21.557225+0000) 2019-10-13T07:54:21.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:21.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:21.557907+0000) 2019-10-13T07:54:23.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:23.600+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:23.603296+0000) 2019-10-13T07:54:24.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:24.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.281137+0000) 2019-10-13T07:54:24.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:24.671+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.675161+0000) 2019-10-13T07:54:24.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:24.671+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.676018+0000) 2019-10-13T07:54:24.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:24.671+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.676231+0000) 2019-10-13T07:54:24.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:24.672+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.676364+0000) 2019-10-13T07:54:24.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:24.671+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.676862+0000) 2019-10-13T07:54:24.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:24.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:24.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:24.675+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.677955+0000) 2019-10-13T07:54:24.681 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:24.675+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.679152+0000) 2019-10-13T07:54:24.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:24.680+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:24.685843+0000) 2019-10-13T07:54:27.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:27.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:27.121560+0000) 2019-10-13T07:54:27.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:27.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:27.535836+0000) 2019-10-13T07:54:29.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:29.672+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.679792+0000) 2019-10-13T07:54:29.682 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:29.675+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.680605+0000) 2019-10-13T07:54:29.682 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:29.675+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.681597+0000) 2019-10-13T07:54:29.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:29.679+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.681925+0000) 2019-10-13T07:54:29.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:29.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:29.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:29.679+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.682415+0000) 2019-10-13T07:54:29.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:29.679+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.683004+0000) 2019-10-13T07:54:29.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:29.679+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.684265+0000) 2019-10-13T07:54:29.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:29.688+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:29.693061+0000) 2019-10-13T07:54:30.490 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:54:30.490 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:30.505 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:54:30.505 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:54:30.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:30.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:30.663171+0000) 2019-10-13T07:54:30.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:30.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:30.760931+0000) 2019-10-13T07:54:31.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:31.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:31.557436+0000) 2019-10-13T07:54:31.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:31.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:31.558110+0000) 2019-10-13T07:54:33.616 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:33.612+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:33.614831+0000) 2019-10-13T07:54:34.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:34.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.281337+0000) 2019-10-13T07:54:34.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:34.680+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.685642+0000) 2019-10-13T07:54:34.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:34.683+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.686644+0000) 2019-10-13T07:54:34.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:34.683+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.687021+0000) 2019-10-13T07:54:34.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:34.683+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.687382+0000) 2019-10-13T07:54:34.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:34.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:34.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:34.683+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.687984+0000) 2019-10-13T07:54:34.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:34.683+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.688451+0000) 2019-10-13T07:54:34.690 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:34.683+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.689036+0000) 2019-10-13T07:54:34.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:34.688+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:34.695633+0000) 2019-10-13T07:54:37.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:37.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:37.121725+0000) 2019-10-13T07:54:37.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:37.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:37.535995+0000) 2019-10-13T07:54:39.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:39.687+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.691590+0000) 2019-10-13T07:54:39.693 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:39.687+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.691875+0000) 2019-10-13T07:54:39.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:39.687+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.692611+0000) 2019-10-13T07:54:39.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:39.687+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.693271+0000) 2019-10-13T07:54:39.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:39.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:39.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:39.688+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.693828+0000) 2019-10-13T07:54:39.695 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:39.691+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.694973+0000) 2019-10-13T07:54:39.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:39.691+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.696706+0000) 2019-10-13T07:54:39.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:39.696+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:39.701441+0000) 2019-10-13T07:54:40.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:40.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:40.663344+0000) 2019-10-13T07:54:40.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:40.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:40.761090+0000) 2019-10-13T07:54:41.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:41.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:41.557632+0000) 2019-10-13T07:54:41.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:41.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:41.558331+0000) 2019-10-13T07:54:43.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:43.620+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:43.621141+0000) 2019-10-13T07:54:44.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:44.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.281537+0000) 2019-10-13T07:54:44.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:44.691+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.697525+0000) 2019-10-13T07:54:44.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:44.695+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.697878+0000) 2019-10-13T07:54:44.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:44.695+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.698672+0000) 2019-10-13T07:54:44.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:44.695+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.699160+0000) 2019-10-13T07:54:44.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:44.696+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.699252+0000) 2019-10-13T07:54:44.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:44.695+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.699995+0000) 2019-10-13T07:54:44.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:44.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:44.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:44.695+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.700789+0000) 2019-10-13T07:54:44.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:44.704+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:44.706468+0000) 2019-10-13T07:54:47.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:47.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:47.121948+0000) 2019-10-13T07:54:47.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:47.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:47.536187+0000) 2019-10-13T07:54:49.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:49.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:49.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:49.703+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.705990+0000) 2019-10-13T07:54:49.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:49.703+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.706713+0000) 2019-10-13T07:54:49.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:49.703+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.707375+0000) 2019-10-13T07:54:49.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:49.703+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.707776+0000) 2019-10-13T07:54:49.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:49.703+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.707975+0000) 2019-10-13T07:54:49.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:49.703+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.708294+0000) 2019-10-13T07:54:49.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:49.703+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.708785+0000) 2019-10-13T07:54:49.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:49.711+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:49.715593+0000) 2019-10-13T07:54:50.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:50.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:50.663534+0000) 2019-10-13T07:54:50.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:50.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:50.761271+0000) 2019-10-13T07:54:51.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:51.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:51.557799+0000) 2019-10-13T07:54:51.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:51.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:51.558546+0000) 2019-10-13T07:54:53.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:53.623+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:53.627694+0000) 2019-10-13T07:54:54.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:54.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.281769+0000) 2019-10-13T07:54:54.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:54.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.710418+0000) 2019-10-13T07:54:54.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:54.707+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.711002+0000) 2019-10-13T07:54:54.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:54.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:54.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:54.707+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.711490+0000) 2019-10-13T07:54:54.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:54.707+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.712343+0000) 2019-10-13T07:54:54.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:54.707+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.712888+0000) 2019-10-13T07:54:54.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:54.711+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.716507+0000) 2019-10-13T07:54:54.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:54.711+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.714958+0000) 2019-10-13T07:54:54.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:54.719+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:54.721193+0000) 2019-10-13T07:54:57.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:57.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:57.122164+0000) 2019-10-13T07:54:57.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:57.535+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:57.536400+0000) 2019-10-13T07:54:59.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:54:59.711+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.716204+0000) 2019-10-13T07:54:59.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:54:59.711+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.715968+0000) 2019-10-13T07:54:59.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:54:59.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:54:59.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:54:59.715+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.716181+0000) 2019-10-13T07:54:59.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:54:59.711+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.717519+0000) 2019-10-13T07:54:59.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:54:59.715+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.718009+0000) 2019-10-13T07:54:59.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:54:59.715+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.718966+0000) 2019-10-13T07:54:59.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:54:59.715+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.718348+0000) 2019-10-13T07:54:59.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:54:59.719+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:54:59.720818+0000) 2019-10-13T07:55:00.555 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:55:00.555 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:00.559 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:55:00.559 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:00.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:00.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:00.663669+0000) 2019-10-13T07:55:00.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:00.759+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:00.761473+0000) 2019-10-13T07:55:01.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:01.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:01.558034+0000) 2019-10-13T07:55:01.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:01.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:01.558760+0000) 2019-10-13T07:55:03.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:03.631+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:03.634077+0000) 2019-10-13T07:55:04.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:04.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.282005+0000) 2019-10-13T07:55:04.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:04.715+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.719892+0000) 2019-10-13T07:55:04.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:04.715+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.720738+0000) 2019-10-13T07:55:04.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:04.715+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.721362+0000) 2019-10-13T07:55:04.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:04.719+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.721626+0000) 2019-10-13T07:55:04.723 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:04.719+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.722493+0000) 2019-10-13T07:55:04.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:04.719+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:04.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:04.719+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.723173+0000) 2019-10-13T07:55:04.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:04.719+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.724212+0000) 2019-10-13T07:55:04.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:04.727+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:04.728285+0000) 2019-10-13T07:55:07.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:07.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:07.122351+0000) 2019-10-13T07:55:07.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:07.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:07.536612+0000) 2019-10-13T07:55:09.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:09.723+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.726430+0000) 2019-10-13T07:55:09.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:09.723+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.726927+0000) 2019-10-13T07:55:09.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:09.723+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.727172+0000) 2019-10-13T07:55:09.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:09.723+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.727586+0000) 2019-10-13T07:55:09.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:09.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:09.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:09.723+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.728810+0000) 2019-10-13T07:55:09.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:09.723+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.727994+0000) 2019-10-13T07:55:09.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:09.723+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.729222+0000) 2019-10-13T07:55:09.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:09.731+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:09.735747+0000) 2019-10-13T07:55:10.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:10.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:10.663874+0000) 2019-10-13T07:55:10.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:10.755+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:10.761714+0000) 2019-10-13T07:55:11.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:11.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:11.558249+0000) 2019-10-13T07:55:11.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:11.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:11.558990+0000) 2019-10-13T07:55:13.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:13.639+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:13.640431+0000) 2019-10-13T07:55:14.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:14.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.282226+0000) 2019-10-13T07:55:14.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:14.727+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.731545+0000) 2019-10-13T07:55:14.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:14.727+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.732114+0000) 2019-10-13T07:55:14.734 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:14.731+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.733059+0000) 2019-10-13T07:55:14.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:14.731+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.733883+0000) 2019-10-13T07:55:14.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:14.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:14.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:14.731+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.735601+0000) 2019-10-13T07:55:14.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:14.731+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.735902+0000) 2019-10-13T07:55:14.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:14.731+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.736948+0000) 2019-10-13T07:55:14.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:14.735+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:14.739113+0000) 2019-10-13T07:55:17.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:17.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:17.122547+0000) 2019-10-13T07:55:17.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:17.531+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:17.536872+0000) 2019-10-13T07:55:19.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:19.731+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.737387+0000) 2019-10-13T07:55:19.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:19.735+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.738159+0000) 2019-10-13T07:55:19.739 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:19.735+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.738344+0000) 2019-10-13T07:55:19.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:19.735+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.739148+0000) 2019-10-13T07:55:19.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:19.735+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.739093+0000) 2019-10-13T07:55:19.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:19.735+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:19.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:19.735+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.740081+0000) 2019-10-13T07:55:19.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:19.735+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.741220+0000) 2019-10-13T07:55:19.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:19.743+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:19.746404+0000) 2019-10-13T07:55:20.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:20.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:20.664081+0000) 2019-10-13T07:55:20.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:20.759+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:20.761950+0000) 2019-10-13T07:55:21.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:21.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:21.558486+0000) 2019-10-13T07:55:22.081 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:21.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:21.559178+0000) 2019-10-13T07:55:23.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:23.643+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:23.646969+0000) 2019-10-13T07:55:24.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:24.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.282425+0000) 2019-10-13T07:55:24.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:24.738+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.743619+0000) 2019-10-13T07:55:24.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:24.738+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.744325+0000) 2019-10-13T07:55:24.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:24.743+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.744154+0000) 2019-10-13T07:55:24.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:24.738+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.744920+0000) 2019-10-13T07:55:24.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:24.738+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.745482+0000) 2019-10-13T07:55:24.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:24.738+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.745612+0000) 2019-10-13T07:55:24.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:24.742+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.745944+0000) 2019-10-13T07:55:24.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:24.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:24.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:24.751+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:24.753328+0000) 2019-10-13T07:55:27.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:27.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:27.122716+0000) 2019-10-13T07:55:27.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:27.534+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:27.537077+0000) 2019-10-13T07:55:29.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:29.742+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.748698+0000) 2019-10-13T07:55:29.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:29.742+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.749466+0000) 2019-10-13T07:55:29.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:29.746+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.749713+0000) 2019-10-13T07:55:29.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:29.746+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.750235+0000) 2019-10-13T07:55:29.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:29.747+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.750816+0000) 2019-10-13T07:55:29.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:29.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:29.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:29.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.751269+0000) 2019-10-13T07:55:29.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:29.746+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.751691+0000) 2019-10-13T07:55:29.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:29.755+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:29.756450+0000) 2019-10-13T07:55:30.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:30.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:30.664259+0000) 2019-10-13T07:55:30.678 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:55:30.679 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:30.682 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:55:30.682 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:55:30.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:30.758+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:30.762105+0000) 2019-10-13T07:55:31.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:31.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:31.558717+0000) 2019-10-13T07:55:31.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:31.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:31.559392+0000) 2019-10-13T07:55:33.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:33.651+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:33.653383+0000) 2019-10-13T07:55:34.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:34.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.282625+0000) 2019-10-13T07:55:34.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:34.751+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.753664+0000) 2019-10-13T07:55:34.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:34.750+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:34.757 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:34.750+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.756883+0000) 2019-10-13T07:55:34.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:34.750+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.757497+0000) 2019-10-13T07:55:34.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:34.754+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.757815+0000) 2019-10-13T07:55:34.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:34.754+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.758154+0000) 2019-10-13T07:55:34.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:34.754+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.758390+0000) 2019-10-13T07:55:34.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:34.754+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.759363+0000) 2019-10-13T07:55:34.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:34.763+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:34.764509+0000) 2019-10-13T07:55:37.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:37.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:37.122971+0000) 2019-10-13T07:55:37.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:37.534+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:37.537305+0000) 2019-10-13T07:55:39.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:39.754+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.759871+0000) 2019-10-13T07:55:39.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:39.754+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.760581+0000) 2019-10-13T07:55:39.762 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:39.754+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.761204+0000) 2019-10-13T07:55:39.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:39.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:39.763 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:39.758+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.761884+0000) 2019-10-13T07:55:39.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:39.758+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.762582+0000) 2019-10-13T07:55:39.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:39.759+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.761750+0000) 2019-10-13T07:55:39.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:39.758+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.763012+0000) 2019-10-13T07:55:39.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:39.763+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:39.766713+0000) 2019-10-13T07:55:40.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:40.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:40.664436+0000) 2019-10-13T07:55:40.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:40.758+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:40.762305+0000) 2019-10-13T07:55:41.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:41.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:41.558935+0000) 2019-10-13T07:55:41.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:41.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:41.559612+0000) 2019-10-13T07:55:43.661 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:43.659+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:43.659737+0000) 2019-10-13T07:55:44.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:44.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.282859+0000) 2019-10-13T07:55:44.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:44.763+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.764573+0000) 2019-10-13T07:55:44.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:44.762+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.765975+0000) 2019-10-13T07:55:44.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:44.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:44.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:44.762+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.767221+0000) 2019-10-13T07:55:44.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:44.762+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.767516+0000) 2019-10-13T07:55:44.768 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:44.762+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.767954+0000) 2019-10-13T07:55:44.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:44.762+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.768602+0000) 2019-10-13T07:55:44.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:44.766+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.770239+0000) 2019-10-13T07:55:44.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:44.767+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:44.771205+0000) 2019-10-13T07:55:47.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:47.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:47.123200+0000) 2019-10-13T07:55:47.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:47.530+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:47.537499+0000) 2019-10-13T07:55:49.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:49.766+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.770342+0000) 2019-10-13T07:55:49.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:49.766+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.770935+0000) 2019-10-13T07:55:49.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:49.767+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.770896+0000) 2019-10-13T07:55:49.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:49.766+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.772277+0000) 2019-10-13T07:55:49.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:49.766+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:49.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:49.766+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.772847+0000) 2019-10-13T07:55:49.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:49.766+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.773422+0000) 2019-10-13T07:55:49.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:49.770+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.774217+0000) 2019-10-13T07:55:49.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:49.771+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:49.775986+0000) 2019-10-13T07:55:50.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:50.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:50.664614+0000) 2019-10-13T07:55:50.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:50.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:50.762539+0000) 2019-10-13T07:55:51.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:51.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:51.559123+0000) 2019-10-13T07:55:51.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:51.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:51.559760+0000) 2019-10-13T07:55:53.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:53.662+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:53.666162+0000) 2019-10-13T07:55:54.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:54.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.283027+0000) 2019-10-13T07:55:54.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:54.770+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.776126+0000) 2019-10-13T07:55:54.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:54.770+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.776459+0000) 2019-10-13T07:55:54.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:54.774+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.776943+0000) 2019-10-13T07:55:54.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:54.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:54.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:54.774+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.777782+0000) 2019-10-13T07:55:54.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:54.774+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.778748+0000) 2019-10-13T07:55:54.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:54.774+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.779284+0000) 2019-10-13T07:55:54.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:54.774+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.779934+0000) 2019-10-13T07:55:54.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:54.778+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:54.783572+0000) 2019-10-13T07:55:57.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:57.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:57.123399+0000) 2019-10-13T07:55:57.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:57.534+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:57.537731+0000) 2019-10-13T07:55:59.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:55:59.778+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.782205+0000) 2019-10-13T07:55:59.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:55:59.778+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.782911+0000) 2019-10-13T07:55:59.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:55:59.778+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.783291+0000) 2019-10-13T07:55:59.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:55:59.778+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:55:59.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:55:59.778+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.783756+0000) 2019-10-13T07:55:59.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:55:59.778+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.784368+0000) 2019-10-13T07:55:59.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:55:59.778+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.784959+0000) 2019-10-13T07:55:59.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:55:59.778+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.784997+0000) 2019-10-13T07:55:59.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:55:59.786+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:55:59.790831+0000) 2019-10-13T07:56:00.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:00.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:00.664792+0000) 2019-10-13T07:56:00.752 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:56:00.753 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:00.757 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:56:00.757 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:00.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:00.758+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:00.762703+0000) 2019-10-13T07:56:01.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:01.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:01.559320+0000) 2019-10-13T07:56:01.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:01.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:01.559958+0000) 2019-10-13T07:56:03.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:03.670+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:03.672553+0000) 2019-10-13T07:56:04.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:04.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.283237+0000) 2019-10-13T07:56:04.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:04.782+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.786547+0000) 2019-10-13T07:56:04.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:04.782+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.787592+0000) 2019-10-13T07:56:04.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:04.782+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.788321+0000) 2019-10-13T07:56:04.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:04.782+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:04.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:04.782+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.788606+0000) 2019-10-13T07:56:04.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:04.782+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.788600+0000) 2019-10-13T07:56:04.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:04.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.789093+0000) 2019-10-13T07:56:04.790 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:04.782+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.788998+0000) 2019-10-13T07:56:04.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:04.794+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:04.798966+0000) 2019-10-13T07:56:07.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:07.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:07.123636+0000) 2019-10-13T07:56:07.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:07.534+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:07.537997+0000) 2019-10-13T07:56:09.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:09.786+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.792008+0000) 2019-10-13T07:56:09.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:09.786+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.792583+0000) 2019-10-13T07:56:09.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:09.786+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.793551+0000) 2019-10-13T07:56:09.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:09.786+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.793484+0000) 2019-10-13T07:56:09.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:09.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61818 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:09.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:09.790+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.793839+0000) 2019-10-13T07:56:09.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:09.790+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.794441+0000) 2019-10-13T07:56:09.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:09.790+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.794616+0000) 2019-10-13T07:56:09.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:09.794+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:09.797869+0000) 2019-10-13T07:56:10.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:10.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:10.664965+0000) 2019-10-13T07:56:10.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:10.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:10.762905+0000) 2019-10-13T07:56:11.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:11.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:11.559521+0000) 2019-10-13T07:56:11.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:11.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:11.560161+0000) 2019-10-13T07:56:13.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:13.674+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:13.679095+0000) 2019-10-13T07:56:14.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:14.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.283444+0000) 2019-10-13T07:56:14.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:14.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.796899+0000) 2019-10-13T07:56:14.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:14.790+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.797622+0000) 2019-10-13T07:56:14.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:14.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:14.800 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:14.794+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.799712+0000) 2019-10-13T07:56:14.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:14.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.800058+0000) 2019-10-13T07:56:14.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:14.794+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.799880+0000) 2019-10-13T07:56:14.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:14.794+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.800656+0000) 2019-10-13T07:56:14.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:14.798+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.801849+0000) 2019-10-13T07:56:14.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:14.806+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:14.808296+0000) 2019-10-13T07:56:17.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:17.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:17.123840+0000) 2019-10-13T07:56:17.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:17.538+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:17.538227+0000) 2019-10-13T07:56:19.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:19.798+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.801775+0000) 2019-10-13T07:56:19.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:19.797+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.803096+0000) 2019-10-13T07:56:19.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:19.797+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.803703+0000) 2019-10-13T07:56:19.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:19.797+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.804186+0000) 2019-10-13T07:56:19.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:19.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:19.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:19.797+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.804897+0000) 2019-10-13T07:56:19.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:19.797+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.805180+0000) 2019-10-13T07:56:19.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:19.801+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.806949+0000) 2019-10-13T07:56:19.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:19.810+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:19.812392+0000) 2019-10-13T07:56:20.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:20.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:20.665137+0000) 2019-10-13T07:56:20.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:20.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:20.763077+0000) 2019-10-13T07:56:21.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:21.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:21.559751+0000) 2019-10-13T07:56:21.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:21.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:21.560375+0000) 2019-10-13T07:56:23.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:23.682+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:23.685511+0000) 2019-10-13T07:56:24.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:24.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.283600+0000) 2019-10-13T07:56:24.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:24.801+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.807453+0000) 2019-10-13T07:56:24.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:24.801+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.808621+0000) 2019-10-13T07:56:24.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:24.801+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.809153+0000) 2019-10-13T07:56:24.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:24.805+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:24.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:24.806+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.811072+0000) 2019-10-13T07:56:24.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:24.805+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.812342+0000) 2019-10-13T07:56:24.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:24.809+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.814024+0000) 2019-10-13T07:56:24.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:24.809+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.814364+0000) 2019-10-13T07:56:24.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:24.814+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:24.816380+0000) 2019-10-13T07:56:27.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:27.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:27.124042+0000) 2019-10-13T07:56:27.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:27.533+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:27.538448+0000) 2019-10-13T07:56:29.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:29.809+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.814071+0000) 2019-10-13T07:56:29.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:29.809+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.814407+0000) 2019-10-13T07:56:29.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:29.809+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.814849+0000) 2019-10-13T07:56:29.816 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:29.809+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.815469+0000) 2019-10-13T07:56:29.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:29.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:29.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:29.810+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.815637+0000) 2019-10-13T07:56:29.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:29.809+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.816624+0000) 2019-10-13T07:56:29.820 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:29.813+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.818157+0000) 2019-10-13T07:56:29.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:29.818+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:29.820616+0000) 2019-10-13T07:56:30.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:30.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:30.665354+0000) 2019-10-13T07:56:30.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:30.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:30.763249+0000) 2019-10-13T07:56:30.824 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:56:30.825 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:30.829 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:56:30.829 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:56:31.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:31.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:31.559975+0000) 2019-10-13T07:56:31.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:31.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:31.560578+0000) 2019-10-13T07:56:33.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:33.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:33.691943+0000) 2019-10-13T07:56:34.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:34.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.283828+0000) 2019-10-13T07:56:34.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:34.817+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.819725+0000) 2019-10-13T07:56:34.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:34.814+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.819394+0000) 2019-10-13T07:56:34.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:34.817+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.820623+0000) 2019-10-13T07:56:34.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:34.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.821141+0000) 2019-10-13T07:56:34.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:34.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:34.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:34.821+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.821721+0000) 2019-10-13T07:56:34.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:34.821+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.822590+0000) 2019-10-13T07:56:34.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:34.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.823479+0000) 2019-10-13T07:56:34.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:34.822+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:34.825983+0000) 2019-10-13T07:56:37.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:37.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:37.124237+0000) 2019-10-13T07:56:37.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:37.537+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:37.538647+0000) 2019-10-13T07:56:39.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:39.821+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.824705+0000) 2019-10-13T07:56:39.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:39.821+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.824453+0000) 2019-10-13T07:56:39.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:39.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:39.828 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:39.825+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.828052+0000) 2019-10-13T07:56:39.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:39.822+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.827779+0000) 2019-10-13T07:56:39.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:39.825+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.829146+0000) 2019-10-13T07:56:39.831 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:39.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.830320+0000) 2019-10-13T07:56:39.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:39.829+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.830041+0000) 2019-10-13T07:56:39.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:39.830+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:39.832771+0000) 2019-10-13T07:56:40.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:40.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:40.665538+0000) 2019-10-13T07:56:40.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:40.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:40.763430+0000) 2019-10-13T07:56:41.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:41.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:41.560169+0000) 2019-10-13T07:56:41.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:41.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:41.560842+0000) 2019-10-13T07:56:43.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:43.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:43.697827+0000) 2019-10-13T07:56:44.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:44.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.284027+0000) 2019-10-13T07:56:44.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:44.829+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.830179+0000) 2019-10-13T07:56:44.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:44.829+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.830463+0000) 2019-10-13T07:56:44.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:44.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.831180+0000) 2019-10-13T07:56:44.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:44.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.832043+0000) 2019-10-13T07:56:44.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:44.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:44.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:44.830+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.832508+0000) 2019-10-13T07:56:44.834 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:44.833+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.834023+0000) 2019-10-13T07:56:44.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:44.833+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.835500+0000) 2019-10-13T07:56:44.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:44.838+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:44.841041+0000) 2019-10-13T07:56:47.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:47.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:47.124438+0000) 2019-10-13T07:56:47.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:47.537+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:47.538849+0000) 2019-10-13T07:56:49.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:49.833+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.835914+0000) 2019-10-13T07:56:49.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:49.834+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.836236+0000) 2019-10-13T07:56:49.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:49.837+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.837805+0000) 2019-10-13T07:56:49.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:49.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:49.839 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:49.837+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.838366+0000) 2019-10-13T07:56:49.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:49.837+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.838996+0000) 2019-10-13T07:56:49.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:49.837+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.839927+0000) 2019-10-13T07:56:49.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:49.837+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.840804+0000) 2019-10-13T07:56:49.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:49.838+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:49.843838+0000) 2019-10-13T07:56:50.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:50.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:50.665716+0000) 2019-10-13T07:56:50.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:50.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:50.763604+0000) 2019-10-13T07:56:51.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:51.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:51.560387+0000) 2019-10-13T07:56:51.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:51.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:51.561041+0000) 2019-10-13T07:56:53.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:53.702+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:53.704377+0000) 2019-10-13T07:56:54.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:54.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.284225+0000) 2019-10-13T07:56:54.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:54.841+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.841828+0000) 2019-10-13T07:56:54.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:54.841+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.843085+0000) 2019-10-13T07:56:54.844 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:54.841+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.843383+0000) 2019-10-13T07:56:54.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:54.838+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.843383+0000) 2019-10-13T07:56:54.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:54.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62099 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:54.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:54.841+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.844035+0000) 2019-10-13T07:56:54.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:54.841+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.844348+0000) 2019-10-13T07:56:54.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:54.841+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.845717+0000) 2019-10-13T07:56:54.849 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:54.846+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:54.848173+0000) 2019-10-13T07:56:57.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:57.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:57.124638+0000) 2019-10-13T07:56:57.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:57.537+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:57.539074+0000) 2019-10-13T07:56:59.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:56:59.845+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.846544+0000) 2019-10-13T07:56:59.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:56:59.845+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.847311+0000) 2019-10-13T07:56:59.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:56:59.841+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.847587+0000) 2019-10-13T07:56:59.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:56:59.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:56:59.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:56:59.849+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.850396+0000) 2019-10-13T07:56:59.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:56:59.849+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.850724+0000) 2019-10-13T07:56:59.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:56:59.849+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.851378+0000) 2019-10-13T07:56:59.852 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:56:59.849+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.851520+0000) 2019-10-13T07:56:59.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:56:59.853+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:56:59.858201+0000) 2019-10-13T07:57:00.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:00.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:00.665956+0000) 2019-10-13T07:57:00.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:00.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:00.763781+0000) 2019-10-13T07:57:00.966 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:57:00.966 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:00.970 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:57:00.970 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:01.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:01.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:01.560612+0000) 2019-10-13T07:57:01.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:01.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:01.561268+0000) 2019-10-13T07:57:03.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:03.709+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:03.710764+0000) 2019-10-13T07:57:04.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:04.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.284433+0000) 2019-10-13T07:57:04.854 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:04.849+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.853294+0000) 2019-10-13T07:57:04.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:04.853+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.853984+0000) 2019-10-13T07:57:04.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:04.849+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.853834+0000) 2019-10-13T07:57:04.855 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:04.853+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.854804+0000) 2019-10-13T07:57:04.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:04.853+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.855575+0000) 2019-10-13T07:57:04.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:04.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:04.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:04.853+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.856635+0000) 2019-10-13T07:57:04.858 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:04.853+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.857653+0000) 2019-10-13T07:57:04.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:04.853+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:04.859476+0000) 2019-10-13T07:57:07.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:07.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:07.124874+0000) 2019-10-13T07:57:07.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:07.537+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:07.539279+0000) 2019-10-13T07:57:09.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:09.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:09.861 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:09.857+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.860551+0000) 2019-10-13T07:57:09.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:09.857+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.860962+0000) 2019-10-13T07:57:09.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:09.857+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.861533+0000) 2019-10-13T07:57:09.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:09.857+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.861839+0000) 2019-10-13T07:57:09.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:09.861+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.862872+0000) 2019-10-13T07:57:09.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:09.861+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.863704+0000) 2019-10-13T07:57:09.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:09.861+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.864200+0000) 2019-10-13T07:57:09.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:09.865+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:09.869379+0000) 2019-10-13T07:57:10.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:10.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:10.666138+0000) 2019-10-13T07:57:10.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:10.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:10.763953+0000) 2019-10-13T07:57:11.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:11.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:11.560812+0000) 2019-10-13T07:57:11.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:11.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:11.561467+0000) 2019-10-13T07:57:13.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:13.713+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:13.717383+0000) 2019-10-13T07:57:14.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:14.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.284642+0000) 2019-10-13T07:57:14.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:14.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:14.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:14.861+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.865409+0000) 2019-10-13T07:57:14.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:14.865+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.866241+0000) 2019-10-13T07:57:14.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:14.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.866832+0000) 2019-10-13T07:57:14.869 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:14.865+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.867506+0000) 2019-10-13T07:57:14.869 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:14.865+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.866925+0000) 2019-10-13T07:57:14.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:14.865+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.869492+0000) 2019-10-13T07:57:14.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:14.865+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.869502+0000) 2019-10-13T07:57:14.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:14.869+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:14.875555+0000) 2019-10-13T07:57:17.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:17.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:17.125064+0000) 2019-10-13T07:57:17.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:17.536+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:17.539482+0000) 2019-10-13T07:57:19.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:19.865+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.868543+0000) 2019-10-13T07:57:19.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:19.864+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.869426+0000) 2019-10-13T07:57:19.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:19.868+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.870270+0000) 2019-10-13T07:57:19.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:19.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:19.871 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:19.868+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.870597+0000) 2019-10-13T07:57:19.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:19.868+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.870947+0000) 2019-10-13T07:57:19.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:19.868+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.872144+0000) 2019-10-13T07:57:19.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:19.868+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.872679+0000) 2019-10-13T07:57:19.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:19.873+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:19.878244+0000) 2019-10-13T07:57:20.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:20.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:20.666312+0000) 2019-10-13T07:57:20.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:20.760+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:20.764125+0000) 2019-10-13T07:57:21.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:21.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:21.561021+0000) 2019-10-13T07:57:21.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:21.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:21.561680+0000) 2019-10-13T07:57:23.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:23.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:23.723889+0000) 2019-10-13T07:57:24.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:24.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.284856+0000) 2019-10-13T07:57:24.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:24.869+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.873225+0000) 2019-10-13T07:57:24.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:24.872+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.874261+0000) 2019-10-13T07:57:24.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:24.872+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.874523+0000) 2019-10-13T07:57:24.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:24.872+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.875394+0000) 2019-10-13T07:57:24.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:24.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:24.877 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:24.872+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.875913+0000) 2019-10-13T07:57:24.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:24.872+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.876437+0000) 2019-10-13T07:57:24.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:24.872+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.876481+0000) 2019-10-13T07:57:24.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:24.877+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:24.881109+0000) 2019-10-13T07:57:27.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:27.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:27.125286+0000) 2019-10-13T07:57:27.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:27.536+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:27.539683+0000) 2019-10-13T07:57:29.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:29.873+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.878043+0000) 2019-10-13T07:57:29.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:29.876+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.879239+0000) 2019-10-13T07:57:29.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:29.876+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.880046+0000) 2019-10-13T07:57:29.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:29.876+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.880427+0000) 2019-10-13T07:57:29.881 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:29.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:29.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:29.876+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.880969+0000) 2019-10-13T07:57:29.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:29.876+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.881360+0000) 2019-10-13T07:57:29.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:29.880+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.883317+0000) 2019-10-13T07:57:29.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:29.881+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:29.885286+0000) 2019-10-13T07:57:30.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:30.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:30.666507+0000) 2019-10-13T07:57:30.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:30.760+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:30.764301+0000) 2019-10-13T07:57:31.062 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:57:31.063 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:31.067 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:57:31.067 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:57:31.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:31.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:31.561221+0000) 2019-10-13T07:57:31.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:31.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:31.561882+0000) 2019-10-13T07:57:33.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:33.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:33.729743+0000) 2019-10-13T07:57:34.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:34.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.285110+0000) 2019-10-13T07:57:34.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:34.877+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.883226+0000) 2019-10-13T07:57:34.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:34.880+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.885259+0000) 2019-10-13T07:57:34.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:34.884+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.886007+0000) 2019-10-13T07:57:34.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:34.880+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.885691+0000) 2019-10-13T07:57:34.888 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:34.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.886236+0000) 2019-10-13T07:57:34.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:34.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:34.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:34.884+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.886730+0000) 2019-10-13T07:57:34.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:34.884+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.887502+0000) 2019-10-13T07:57:34.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:34.885+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:34.891230+0000) 2019-10-13T07:57:37.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:37.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:37.125527+0000) 2019-10-13T07:57:37.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:37.536+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:37.539881+0000) 2019-10-13T07:57:39.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:39.885+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.890669+0000) 2019-10-13T07:57:39.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:39.888+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.891617+0000) 2019-10-13T07:57:39.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:39.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:39.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:39.888+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.892278+0000) 2019-10-13T07:57:39.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:39.888+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.892569+0000) 2019-10-13T07:57:39.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:39.888+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.892907+0000) 2019-10-13T07:57:39.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:39.892+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.893956+0000) 2019-10-13T07:57:39.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:39.892+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.894377+0000) 2019-10-13T07:57:39.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:39.897+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:39.900567+0000) 2019-10-13T07:57:40.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:40.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:40.666706+0000) 2019-10-13T07:57:40.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:40.760+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:40.764498+0000) 2019-10-13T07:57:41.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:41.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:41.561420+0000) 2019-10-13T07:57:41.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:41.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:41.562114+0000) 2019-10-13T07:57:43.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:43.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:43.732326+0000) 2019-10-13T07:57:44.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:44.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.285304+0000) 2019-10-13T07:57:44.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:44.892+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.894350+0000) 2019-10-13T07:57:44.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:44.892+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.895947+0000) 2019-10-13T07:57:44.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:44.892+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.897042+0000) 2019-10-13T07:57:44.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:44.892+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.897284+0000) 2019-10-13T07:57:44.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:44.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62410 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:44.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:44.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.898262+0000) 2019-10-13T07:57:44.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:44.893+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.898208+0000) 2019-10-13T07:57:44.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:44.896+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.899286+0000) 2019-10-13T07:57:44.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:44.897+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:44.902155+0000) 2019-10-13T07:57:47.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:47.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:47.125796+0000) 2019-10-13T07:57:47.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:47.536+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:47.540083+0000) 2019-10-13T07:57:49.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:49.900+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.902184+0000) 2019-10-13T07:57:49.903 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:49.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.902797+0000) 2019-10-13T07:57:49.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:49.900+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.903243+0000) 2019-10-13T07:57:49.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:49.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:49.904 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:49.900+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.904247+0000) 2019-10-13T07:57:49.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:49.900+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.904769+0000) 2019-10-13T07:57:49.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:49.900+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.904545+0000) 2019-10-13T07:57:49.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:49.901+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.905424+0000) 2019-10-13T07:57:49.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:49.909+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:49.912640+0000) 2019-10-13T07:57:50.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:50.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:50.666919+0000) 2019-10-13T07:57:50.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:50.760+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:50.764667+0000) 2019-10-13T07:57:51.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:51.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:51.561616+0000) 2019-10-13T07:57:51.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:51.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:51.562307+0000) 2019-10-13T07:57:53.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:53.737+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:53.738680+0000) 2019-10-13T07:57:54.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:54.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.285502+0000) 2019-10-13T07:57:54.907 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:54.904+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.906791+0000) 2019-10-13T07:57:54.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:54.904+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.907170+0000) 2019-10-13T07:57:54.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:54.904+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.907779+0000) 2019-10-13T07:57:54.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:54.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:54.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:54.904+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.908468+0000) 2019-10-13T07:57:54.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:54.904+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.908902+0000) 2019-10-13T07:57:54.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:54.905+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.909219+0000) 2019-10-13T07:57:54.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:54.908+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.910149+0000) 2019-10-13T07:57:54.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:54.913+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:54.916941+0000) 2019-10-13T07:57:57.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:57.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:57.125990+0000) 2019-10-13T07:57:57.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:57.536+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:57.540282+0000) 2019-10-13T07:57:59.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:57:59.905+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.911315+0000) 2019-10-13T07:57:59.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:57:59.908+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.912450+0000) 2019-10-13T07:57:59.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:57:59.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:57:59.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:57:59.912+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.913821+0000) 2019-10-13T07:57:59.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:57:59.908+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.913471+0000) 2019-10-13T07:57:59.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:57:59.912+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.914213+0000) 2019-10-13T07:57:59.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:57:59.912+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.914853+0000) 2019-10-13T07:57:59.916 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:57:59.912+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.915352+0000) 2019-10-13T07:57:59.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:57:59.913+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:57:59.918596+0000) 2019-10-13T07:58:00.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:00.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:00.667102+0000) 2019-10-13T07:58:00.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:00.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:00.764875+0000) 2019-10-13T07:58:01.193 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:58:01.193 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:01.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:58:01.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:01.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:01.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:01.561835+0000) 2019-10-13T07:58:01.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:01.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:01.562510+0000) 2019-10-13T07:58:03.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:03.741+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:03.745181+0000) 2019-10-13T07:58:04.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:04.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.285711+0000) 2019-10-13T07:58:04.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:04.912+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.917600+0000) 2019-10-13T07:58:04.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:04.916+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.918200+0000) 2019-10-13T07:58:04.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:04.916+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.918741+0000) 2019-10-13T07:58:04.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:04.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:04.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:04.917+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.920446+0000) 2019-10-13T07:58:04.923 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:04.920+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.922298+0000) 2019-10-13T07:58:04.924 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:04.916+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.921186+0000) 2019-10-13T07:58:04.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:04.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.925312+0000) 2019-10-13T07:58:04.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:04.921+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:04.927472+0000) 2019-10-13T07:58:07.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:07.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:07.126196+0000) 2019-10-13T07:58:07.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:07.540+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:07.540515+0000) 2019-10-13T07:58:09.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:09.920+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.923439+0000) 2019-10-13T07:58:09.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:09.920+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.924516+0000) 2019-10-13T07:58:09.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:09.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.925003+0000) 2019-10-13T07:58:09.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:09.920+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.925227+0000) 2019-10-13T07:58:09.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:09.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:09.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:09.920+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.926289+0000) 2019-10-13T07:58:09.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:09.924+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.927516+0000) 2019-10-13T07:58:09.928 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:09.924+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.927181+0000) 2019-10-13T07:58:09.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:09.928+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:09.933456+0000) 2019-10-13T07:58:10.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:10.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:10.667294+0000) 2019-10-13T07:58:10.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:10.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:10.765080+0000) 2019-10-13T07:58:11.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:11.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:11.562038+0000) 2019-10-13T07:58:11.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:11.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:11.562711+0000) 2019-10-13T07:58:13.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:13.748+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:13.751818+0000) 2019-10-13T07:58:14.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:14.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.285922+0000) 2019-10-13T07:58:14.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:14.923+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.929164+0000) 2019-10-13T07:58:14.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:14.923+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.929638+0000) 2019-10-13T07:58:14.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:14.927+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:14.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:14.927+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.932445+0000) 2019-10-13T07:58:14.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:14.927+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.932951+0000) 2019-10-13T07:58:14.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:14.928+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.933322+0000) 2019-10-13T07:58:14.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:14.931+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.935313+0000) 2019-10-13T07:58:14.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:14.931+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.935726+0000) 2019-10-13T07:58:14.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:14.936+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:14.942389+0000) 2019-10-13T07:58:17.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:17.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:17.126417+0000) 2019-10-13T07:58:17.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:17.535+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:17.540741+0000) 2019-10-13T07:58:19.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:19.928+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.934678+0000) 2019-10-13T07:58:19.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:19.931+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.935852+0000) 2019-10-13T07:58:19.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:19.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:19.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:19.931+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.937568+0000) 2019-10-13T07:58:19.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:19.931+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.937678+0000) 2019-10-13T07:58:19.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:19.935+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.938524+0000) 2019-10-13T07:58:19.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:19.935+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.939106+0000) 2019-10-13T07:58:19.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:19.935+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.939730+0000) 2019-10-13T07:58:19.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:19.940+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:19.946383+0000) 2019-10-13T07:58:20.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:20.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:20.667507+0000) 2019-10-13T07:58:20.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:20.759+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:20.765254+0000) 2019-10-13T07:58:21.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:21.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:21.562239+0000) 2019-10-13T07:58:21.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:21.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:21.562908+0000) 2019-10-13T07:58:23.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:23.756+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:23.758280+0000) 2019-10-13T07:58:24.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:24.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.286125+0000) 2019-10-13T07:58:24.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:24.936+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.940213+0000) 2019-10-13T07:58:24.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:24.939+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.941721+0000) 2019-10-13T07:58:24.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:24.939+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.942733+0000) 2019-10-13T07:58:24.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:24.939+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.942494+0000) 2019-10-13T07:58:24.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:24.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:24.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:24.939+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.943242+0000) 2019-10-13T07:58:24.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:24.939+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.943090+0000) 2019-10-13T07:58:24.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:24.939+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.943764+0000) 2019-10-13T07:58:24.951 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:24.944+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:24.950236+0000) 2019-10-13T07:58:27.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:27.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:27.126615+0000) 2019-10-13T07:58:27.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:27.539+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:27.540943+0000) 2019-10-13T07:58:29.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:29.943+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.946900+0000) 2019-10-13T07:58:29.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:29.940+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.947256+0000) 2019-10-13T07:58:29.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:29.943+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.947333+0000) 2019-10-13T07:58:29.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:29.943+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.948086+0000) 2019-10-13T07:58:29.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:29.943+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.948434+0000) 2019-10-13T07:58:29.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:29.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:29.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:29.943+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.949549+0000) 2019-10-13T07:58:29.951 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:29.947+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.950557+0000) 2019-10-13T07:58:29.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:29.952+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:29.956574+0000) 2019-10-13T07:58:30.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:30.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:30.667726+0000) 2019-10-13T07:58:30.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:30.759+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:30.765435+0000) 2019-10-13T07:58:31.291 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:58:31.291 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:31.295 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:58:31.295 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:58:31.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:31.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:31.562428+0000) 2019-10-13T07:58:31.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:31.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:31.563129+0000) 2019-10-13T07:58:33.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:33.764+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:33.769788+0000) 2019-10-13T07:58:34.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:34.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.286322+0000) 2019-10-13T07:58:34.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:34.944+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.951490+0000) 2019-10-13T07:58:34.953 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:34.947+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.951826+0000) 2019-10-13T07:58:34.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:34.947+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.952740+0000) 2019-10-13T07:58:34.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:34.947+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.953111+0000) 2019-10-13T07:58:34.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:34.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:34.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:34.951+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.954068+0000) 2019-10-13T07:58:34.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:34.951+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.954831+0000) 2019-10-13T07:58:34.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:34.951+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.954740+0000) 2019-10-13T07:58:34.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:34.956+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:34.961108+0000) 2019-10-13T07:58:37.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:37.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:37.126833+0000) 2019-10-13T07:58:37.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:37.539+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:37.541153+0000) 2019-10-13T07:58:39.957 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:39.951+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.956668+0000) 2019-10-13T07:58:39.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:39.952+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.956888+0000) 2019-10-13T07:58:39.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:39.955+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.958456+0000) 2019-10-13T07:58:39.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:39.955+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.959129+0000) 2019-10-13T07:58:39.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:39.955+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.959697+0000) 2019-10-13T07:58:39.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:39.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:39.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:39.955+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.960298+0000) 2019-10-13T07:58:39.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:39.955+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.960956+0000) 2019-10-13T07:58:39.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:39.960+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:39.964239+0000) 2019-10-13T07:58:40.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:40.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:40.667922+0000) 2019-10-13T07:58:40.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:40.759+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:40.765642+0000) 2019-10-13T07:58:41.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:41.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:41.562640+0000) 2019-10-13T07:58:41.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:41.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:41.563379+0000) 2019-10-13T07:58:43.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:43.772+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:43.776171+0000) 2019-10-13T07:58:44.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:44.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.286532+0000) 2019-10-13T07:58:44.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:44.959+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.963175+0000) 2019-10-13T07:58:44.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:44.959+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.963589+0000) 2019-10-13T07:58:44.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:44.959+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.965027+0000) 2019-10-13T07:58:44.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:44.960+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.964811+0000) 2019-10-13T07:58:44.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:44.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:44.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:44.963+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.966507+0000) 2019-10-13T07:58:44.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:44.963+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.967540+0000) 2019-10-13T07:58:44.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:44.963+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.968630+0000) 2019-10-13T07:58:44.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:44.968+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:44.973911+0000) 2019-10-13T07:58:47.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:47.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:47.127067+0000) 2019-10-13T07:58:47.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:47.535+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:47.541345+0000) 2019-10-13T07:58:49.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:49.967+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.970397+0000) 2019-10-13T07:58:49.971 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:49.967+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.970770+0000) 2019-10-13T07:58:49.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:49.964+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.970787+0000) 2019-10-13T07:58:49.972 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:49.967+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.970957+0000) 2019-10-13T07:58:49.973 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:49.967+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.971651+0000) 2019-10-13T07:58:49.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:49.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:49.973 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:49.967+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.972354+0000) 2019-10-13T07:58:49.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:49.967+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.973306+0000) 2019-10-13T07:58:49.977 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:49.972+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:49.976761+0000) 2019-10-13T07:58:50.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:50.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:50.668116+0000) 2019-10-13T07:58:50.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:50.763+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:50.765855+0000) 2019-10-13T07:58:51.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:51.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:51.562871+0000) 2019-10-13T07:58:51.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:51.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:51.563582+0000) 2019-10-13T07:58:53.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:53.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:53.782604+0000) 2019-10-13T07:58:54.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:54.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.286743+0000) 2019-10-13T07:58:54.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:54.971+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.975368+0000) 2019-10-13T07:58:54.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:54.971+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.976057+0000) 2019-10-13T07:58:54.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:54.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:54.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:54.975+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.978089+0000) 2019-10-13T07:58:54.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:54.975+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.978378+0000) 2019-10-13T07:58:54.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:54.975+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.978727+0000) 2019-10-13T07:58:54.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:54.975+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.979179+0000) 2019-10-13T07:58:54.980 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:54.972+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.979103+0000) 2019-10-13T07:58:54.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:54.980+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:54.985459+0000) 2019-10-13T07:58:57.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:57.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:57.127271+0000) 2019-10-13T07:58:57.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:57.535+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:57.541549+0000) 2019-10-13T07:58:59.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:58:59.979+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.982375+0000) 2019-10-13T07:58:59.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:58:59.979+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.982862+0000) 2019-10-13T07:58:59.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:58:59.976+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.983444+0000) 2019-10-13T07:58:59.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:58:59.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:58:59.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:58:59.979+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.983340+0000) 2019-10-13T07:58:59.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:58:59.979+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.984742+0000) 2019-10-13T07:58:59.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:58:59.983+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.986247+0000) 2019-10-13T07:58:59.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:58:59.983+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.987740+0000) 2019-10-13T07:58:59.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:58:59.984+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:58:59.991485+0000) 2019-10-13T07:59:00.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:00.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:00.668317+0000) 2019-10-13T07:59:00.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:00.763+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:00.766043+0000) 2019-10-13T07:59:01.404 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:59:01.404 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:01.409 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:59:01.409 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:01.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:01.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:01.563098+0000) 2019-10-13T07:59:01.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:01.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:01.563789+0000) 2019-10-13T07:59:03.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:03.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:03.788942+0000) 2019-10-13T07:59:04.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:04.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.286949+0000) 2019-10-13T07:59:04.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:04.983+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.987470+0000) 2019-10-13T07:59:04.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:04.983+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.988023+0000) 2019-10-13T07:59:04.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:04.980+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.988053+0000) 2019-10-13T07:59:04.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:04.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62897 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:04.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:04.983+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.988718+0000) 2019-10-13T07:59:04.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:04.983+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.989054+0000) 2019-10-13T07:59:04.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:04.987+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.989914+0000) 2019-10-13T07:59:04.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:04.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.989950+0000) 2019-10-13T07:59:04.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:04.992+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:04.992967+0000) 2019-10-13T07:59:07.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:07.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:07.127466+0000) 2019-10-13T07:59:07.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:07.539+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:07.541776+0000) 2019-10-13T07:59:09.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:09.986+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.992213+0000) 2019-10-13T07:59:09.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:09.992+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.992406+0000) 2019-10-13T07:59:09.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:09.986+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.993361+0000) 2019-10-13T07:59:09.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:09.990+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.993993+0000) 2019-10-13T07:59:09.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:09.990+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.994005+0000) 2019-10-13T07:59:09.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:09.990+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.994371+0000) 2019-10-13T07:59:09.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:09.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:09.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:09.990+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:09.995878+0000) 2019-10-13T07:59:10.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:10.000+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:10.000755+0000) 2019-10-13T07:59:10.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:10.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:10.668445+0000) 2019-10-13T07:59:10.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:10.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:10.766240+0000) 2019-10-13T07:59:11.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:11.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:11.563296+0000) 2019-10-13T07:59:11.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:11.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:11.564022+0000) 2019-10-13T07:59:13.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:13.791+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:13.795333+0000) 2019-10-13T07:59:14.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:14.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.287181+0000) 2019-10-13T07:59:14.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:14.990+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.997593+0000) 2019-10-13T07:59:14.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:14.994+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.998540+0000) 2019-10-13T07:59:14.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:14.994+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.999122+0000) 2019-10-13T07:59:15.000 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:14.994+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.999295+0000) 2019-10-13T07:59:15.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:14.995+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:14.999256+0000) 2019-10-13T07:59:15.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:14.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:15.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:14.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:15.000360+0000) 2019-10-13T07:59:15.001 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:14.994+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:15.000611+0000) 2019-10-13T07:59:15.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:15.003+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:15.007085+0000) 2019-10-13T07:59:17.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:17.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:17.127659+0000) 2019-10-13T07:59:17.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:17.538+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:17.541996+0000) 2019-10-13T07:59:20.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:19.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.003715+0000) 2019-10-13T07:59:20.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:19.998+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.005127+0000) 2019-10-13T07:59:20.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:19.998+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.005467+0000) 2019-10-13T07:59:20.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:20.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.006234+0000) 2019-10-13T07:59:20.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:20.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:20.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:20.002+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.006570+0000) 2019-10-13T07:59:20.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:20.002+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.006819+0000) 2019-10-13T07:59:20.008 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:20.002+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.007592+0000) 2019-10-13T07:59:20.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:20.007+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.009410+0000) 2019-10-13T07:59:20.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:20.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.668611+0000) 2019-10-13T07:59:20.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:20.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:20.766446+0000) 2019-10-13T07:59:21.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:21.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:21.563505+0000) 2019-10-13T07:59:21.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:21.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:21.564249+0000) 2019-10-13T07:59:23.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:23.799+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:23.801894+0000) 2019-10-13T07:59:24.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:24.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:24.287409+0000) 2019-10-13T07:59:25.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:25.006+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.010282+0000) 2019-10-13T07:59:25.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:25.006+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.010950+0000) 2019-10-13T07:59:25.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:25.006+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.011342+0000) 2019-10-13T07:59:25.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:25.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.011252+0000) 2019-10-13T07:59:25.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:25.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:25.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:25.006+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.012218+0000) 2019-10-13T07:59:25.015 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:25.010+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.014988+0000) 2019-10-13T07:59:25.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:25.006+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.013462+0000) 2019-10-13T07:59:25.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:25.015+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:25.019269+0000) 2019-10-13T07:59:27.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:27.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:27.127875+0000) 2019-10-13T07:59:27.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:27.538+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:27.542215+0000) 2019-10-13T07:59:30.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:30.010+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.014798+0000) 2019-10-13T07:59:30.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:30.010+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.015412+0000) 2019-10-13T07:59:30.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:30.010+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.017300+0000) 2019-10-13T07:59:30.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:30.010+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.017561+0000) 2019-10-13T07:59:30.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:30.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:30.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:30.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.018112+0000) 2019-10-13T07:59:30.020 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:30.014+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.019087+0000) 2019-10-13T07:59:30.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:30.014+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.020008+0000) 2019-10-13T07:59:30.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:30.023+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.025113+0000) 2019-10-13T07:59:30.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:30.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.668788+0000) 2019-10-13T07:59:30.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:30.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:30.766623+0000) 2019-10-13T07:59:31.543 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T07:59:31.544 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:31.547 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T07:59:31.547 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T07:59:31.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:31.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:31.563703+0000) 2019-10-13T07:59:31.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:31.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:31.564411+0000) 2019-10-13T07:59:33.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:33.807+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:33.808487+0000) 2019-10-13T07:59:34.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:34.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:34.287614+0000) 2019-10-13T07:59:35.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:35.018+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.022187+0000) 2019-10-13T07:59:35.023 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:35.018+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.022416+0000) 2019-10-13T07:59:35.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:35.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.023027+0000) 2019-10-13T07:59:35.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:35.018+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.023327+0000) 2019-10-13T07:59:35.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:35.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:35.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:35.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.024103+0000) 2019-10-13T07:59:35.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:35.018+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.025107+0000) 2019-10-13T07:59:35.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:35.023+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.025255+0000) 2019-10-13T07:59:35.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:35.027+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:35.030531+0000) 2019-10-13T07:59:37.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:37.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:37.128085+0000) 2019-10-13T07:59:37.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:37.538+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:37.542419+0000) 2019-10-13T07:59:40.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:40.022+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.027075+0000) 2019-10-13T07:59:40.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:40.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.026905+0000) 2019-10-13T07:59:40.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:40.022+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.027667+0000) 2019-10-13T07:59:40.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:40.022+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.028258+0000) 2019-10-13T07:59:40.029 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:40.022+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.028626+0000) 2019-10-13T07:59:40.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:40.022+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.028982+0000) 2019-10-13T07:59:40.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:40.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:40.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:40.022+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.029150+0000) 2019-10-13T07:59:40.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:40.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.036374+0000) 2019-10-13T07:59:40.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:40.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.668969+0000) 2019-10-13T07:59:40.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:40.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:40.766856+0000) 2019-10-13T07:59:41.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:41.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:41.563940+0000) 2019-10-13T07:59:41.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:41.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:41.564642+0000) 2019-10-13T07:59:43.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:43.811+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:43.815045+0000) 2019-10-13T07:59:44.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:44.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:44.287841+0000) 2019-10-13T07:59:45.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:45.026+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.032494+0000) 2019-10-13T07:59:45.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:45.026+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.033082+0000) 2019-10-13T07:59:45.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:45.030+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.033784+0000) 2019-10-13T07:59:45.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:45.030+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.033945+0000) 2019-10-13T07:59:45.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:45.030+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.034368+0000) 2019-10-13T07:59:45.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:45.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:45.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:45.031+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.034777+0000) 2019-10-13T07:59:45.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:45.030+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.035669+0000) 2019-10-13T07:59:45.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:45.039+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:45.042860+0000) 2019-10-13T07:59:47.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:47.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:47.128287+0000) 2019-10-13T07:59:47.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:47.542+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:47.542626+0000) 2019-10-13T07:59:50.037 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:50.030+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.036806+0000) 2019-10-13T07:59:50.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:50.034+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.038170+0000) 2019-10-13T07:59:50.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:50.034+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.038616+0000) 2019-10-13T07:59:50.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:50.034+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.039352+0000) 2019-10-13T07:59:50.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:50.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.039055+0000) 2019-10-13T07:59:50.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:50.034+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.039937+0000) 2019-10-13T07:59:50.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:50.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:50.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:50.039+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.040314+0000) 2019-10-13T07:59:50.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:50.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.046719+0000) 2019-10-13T07:59:50.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:50.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.669121+0000) 2019-10-13T07:59:50.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:50.762+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:50.767037+0000) 2019-10-13T07:59:51.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:51.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:51.564136+0000) 2019-10-13T07:59:51.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:51.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:51.564839+0000) 2019-10-13T07:59:53.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:53.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:53.825338+0000) 2019-10-13T07:59:54.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:54.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:54.288071+0000) 2019-10-13T07:59:55.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:55.038+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.044220+0000) 2019-10-13T07:59:55.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T07:59:55.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.044472+0000) 2019-10-13T07:59:55.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T07:59:55.038+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.044790+0000) 2019-10-13T07:59:55.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:55.038+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.045320+0000) 2019-10-13T07:59:55.047 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T07:59:55.043+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.045119+0000) 2019-10-13T07:59:55.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T07:59:55.042+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.046019+0000) 2019-10-13T07:59:55.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T07:59:55.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T07:59:55.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T07:59:55.042+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.046229+0000) 2019-10-13T07:59:55.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T07:59:55.051+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:55.055498+0000) 2019-10-13T07:59:57.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T07:59:57.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:57.128501+0000) 2019-10-13T07:59:57.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T07:59:57.538+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T06:59:57.542824+0000) 2019-10-13T08:00:00.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:00.047+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.049376+0000) 2019-10-13T08:00:00.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:00.046+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.050627+0000) 2019-10-13T08:00:00.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:00.046+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.051179+0000) 2019-10-13T08:00:00.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:00.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:00.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:00.046+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.051749+0000) 2019-10-13T08:00:00.052 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:00.046+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.052089+0000) 2019-10-13T08:00:00.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:00.046+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.052492+0000) 2019-10-13T08:00:00.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:00.050+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.053956+0000) 2019-10-13T08:00:00.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:00.059+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.060533+0000) 2019-10-13T08:00:00.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:00.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.669258+0000) 2019-10-13T08:00:00.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:00.766+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:00.767169+0000) 2019-10-13T08:00:01.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:01.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:01.564345+0000) 2019-10-13T08:00:01.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:01.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:01.565055+0000) 2019-10-13T08:00:01.674 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:00:01.674 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:01.678 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:00:01.678 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:03.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:03.827+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:03.831753+0000) 2019-10-13T08:00:04.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:04.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:04.288299+0000) 2019-10-13T08:00:05.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:05.050+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.055038+0000) 2019-10-13T08:00:05.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:05.051+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.054925+0000) 2019-10-13T08:00:05.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:05.050+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.056251+0000) 2019-10-13T08:00:05.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:05.050+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.056442+0000) 2019-10-13T08:00:05.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:05.050+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.056890+0000) 2019-10-13T08:00:05.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:05.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:05.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:05.054+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.058036+0000) 2019-10-13T08:00:05.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:05.054+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.058476+0000) 2019-10-13T08:00:05.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:05.059+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:05.063801+0000) 2019-10-13T08:00:07.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:07.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:07.128725+0000) 2019-10-13T08:00:07.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:07.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:07.543026+0000) 2019-10-13T08:00:10.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:10.053+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.061150+0000) 2019-10-13T08:00:10.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:10.057+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.061730+0000) 2019-10-13T08:00:10.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:10.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:10.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:10.057+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.063537+0000) 2019-10-13T08:00:10.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:10.057+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.062976+0000) 2019-10-13T08:00:10.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:10.057+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.064301+0000) 2019-10-13T08:00:10.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:10.057+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.065202+0000) 2019-10-13T08:00:10.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:10.063+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.065775+0000) 2019-10-13T08:00:10.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:10.063+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.066339+0000) 2019-10-13T08:00:10.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:10.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.669428+0000) 2019-10-13T08:00:10.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:10.761+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:10.767360+0000) 2019-10-13T08:00:11.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:11.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:11.564554+0000) 2019-10-13T08:00:11.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:11.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:11.565281+0000) 2019-10-13T08:00:13.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:13.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:13.838029+0000) 2019-10-13T08:00:14.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:14.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:14.288512+0000) 2019-10-13T08:00:15.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:15.061+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.066455+0000) 2019-10-13T08:00:15.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:15.061+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.067389+0000) 2019-10-13T08:00:15.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:15.061+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.067315+0000) 2019-10-13T08:00:15.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:15.061+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.067576+0000) 2019-10-13T08:00:15.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:15.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:15.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:15.063+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.067971+0000) 2019-10-13T08:00:15.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:15.061+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.068101+0000) 2019-10-13T08:00:15.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:15.061+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.068883+0000) 2019-10-13T08:00:15.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:15.067+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:15.071538+0000) 2019-10-13T08:00:17.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:17.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:17.128921+0000) 2019-10-13T08:00:17.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:17.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:17.543219+0000) 2019-10-13T08:00:20.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:20.065+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.071765+0000) 2019-10-13T08:00:20.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:20.065+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.072639+0000) 2019-10-13T08:00:20.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:20.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:20.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:20.065+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.072997+0000) 2019-10-13T08:00:20.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:20.065+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.073283+0000) 2019-10-13T08:00:20.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:20.065+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.073617+0000) 2019-10-13T08:00:20.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:20.069+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.074103+0000) 2019-10-13T08:00:20.076 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:20.070+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.075353+0000) 2019-10-13T08:00:20.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:20.074+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.077060+0000) 2019-10-13T08:00:20.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:20.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.669606+0000) 2019-10-13T08:00:20.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:20.765+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:20.767557+0000) 2019-10-13T08:00:21.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:21.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:21.564749+0000) 2019-10-13T08:00:21.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:21.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:21.565484+0000) 2019-10-13T08:00:23.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:23.842+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:23.844642+0000) 2019-10-13T08:00:24.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:24.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:24.288709+0000) 2019-10-13T08:00:25.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:25.073+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.076447+0000) 2019-10-13T08:00:25.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:25.073+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.077042+0000) 2019-10-13T08:00:25.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:25.073+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.077611+0000) 2019-10-13T08:00:25.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:25.077+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.078024+0000) 2019-10-13T08:00:25.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:25.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:25.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:25.074+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.078242+0000) 2019-10-13T08:00:25.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:25.077+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.078666+0000) 2019-10-13T08:00:25.080 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:25.077+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.079626+0000) 2019-10-13T08:00:25.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:25.082+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:25.084996+0000) 2019-10-13T08:00:27.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:27.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:27.129155+0000) 2019-10-13T08:00:27.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:27.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:27.543417+0000) 2019-10-13T08:00:30.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:30.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.080944+0000) 2019-10-13T08:00:30.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:30.077+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.081422+0000) 2019-10-13T08:00:30.083 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:30.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:30.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:30.081+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.084325+0000) 2019-10-13T08:00:30.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:30.081+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.084642+0000) 2019-10-13T08:00:30.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:30.082+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.084453+0000) 2019-10-13T08:00:30.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:30.081+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.085314+0000) 2019-10-13T08:00:30.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:30.085+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.085854+0000) 2019-10-13T08:00:30.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:30.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.089933+0000) 2019-10-13T08:00:30.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:30.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.669783+0000) 2019-10-13T08:00:30.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:30.765+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:30.767760+0000) 2019-10-13T08:00:31.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:31.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:31.564985+0000) 2019-10-13T08:00:31.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:31.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:31.565678+0000) 2019-10-13T08:00:31.792 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:00:31.792 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:31.796 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:00:31.796 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:00:33.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:33.846+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:33.851055+0000) 2019-10-13T08:00:34.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:34.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:34.288912+0000) 2019-10-13T08:00:35.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:35.082+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.085907+0000) 2019-10-13T08:00:35.087 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:35.085+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.087158+0000) 2019-10-13T08:00:35.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:35.085+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.087453+0000) 2019-10-13T08:00:35.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:35.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.088087+0000) 2019-10-13T08:00:35.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:35.085+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.088725+0000) 2019-10-13T08:00:35.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:35.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:35.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:35.085+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.089491+0000) 2019-10-13T08:00:35.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:35.089+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.089886+0000) 2019-10-13T08:00:35.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:35.090+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:35.094774+0000) 2019-10-13T08:00:37.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:37.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:37.129354+0000) 2019-10-13T08:00:37.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:37.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:37.543637+0000) 2019-10-13T08:00:40.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:40.089+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.092449+0000) 2019-10-13T08:00:40.093 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:40.089+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.092900+0000) 2019-10-13T08:00:40.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:40.093+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.094360+0000) 2019-10-13T08:00:40.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:40.093+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.094656+0000) 2019-10-13T08:00:40.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:40.093+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.094410+0000) 2019-10-13T08:00:40.096 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:40.090+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.094672+0000) 2019-10-13T08:00:40.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:40.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:40.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:40.093+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.095901+0000) 2019-10-13T08:00:40.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:40.098+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.102313+0000) 2019-10-13T08:00:40.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:40.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.669997+0000) 2019-10-13T08:00:40.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:40.765+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:40.767932+0000) 2019-10-13T08:00:41.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:41.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:41.565173+0000) 2019-10-13T08:00:41.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:41.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:41.565904+0000) 2019-10-13T08:00:43.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:43.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:43.857469+0000) 2019-10-13T08:00:44.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:44.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:44.289141+0000) 2019-10-13T08:00:45.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:45.097+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.099199+0000) 2019-10-13T08:00:45.100 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:45.097+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.099742+0000) 2019-10-13T08:00:45.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:45.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.100083+0000) 2019-10-13T08:00:45.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:45.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.100406+0000) 2019-10-13T08:00:45.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:45.097+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.100582+0000) 2019-10-13T08:00:45.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:45.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:45.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:45.101+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.102257+0000) 2019-10-13T08:00:45.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:45.098+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.102117+0000) 2019-10-13T08:00:45.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:45.102+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:45.106428+0000) 2019-10-13T08:00:47.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:47.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:47.129557+0000) 2019-10-13T08:00:47.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:47.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:47.543836+0000) 2019-10-13T08:00:50.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:50.101+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.103155+0000) 2019-10-13T08:00:50.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:50.098+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.103556+0000) 2019-10-13T08:00:50.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:50.101+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.104314+0000) 2019-10-13T08:00:50.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:50.101+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.105059+0000) 2019-10-13T08:00:50.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:50.101+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.105519+0000) 2019-10-13T08:00:50.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:50.102+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.105355+0000) 2019-10-13T08:00:50.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:50.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63534 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:50.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:50.105+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.105961+0000) 2019-10-13T08:00:50.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:50.105+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.106014+0000) 2019-10-13T08:00:50.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:50.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.670181+0000) 2019-10-13T08:00:50.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:50.765+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:50.768104+0000) 2019-10-13T08:00:51.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:51.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:51.565386+0000) 2019-10-13T08:00:51.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:51.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:51.566123+0000) 2019-10-13T08:00:53.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:53.858+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:53.863880+0000) 2019-10-13T08:00:54.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:54.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:54.289375+0000) 2019-10-13T08:00:55.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:00:55.105+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.109550+0000) 2019-10-13T08:00:55.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:00:55.109+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.110232+0000) 2019-10-13T08:00:55.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:00:55.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:00:55.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:00:55.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.112879+0000) 2019-10-13T08:00:55.420 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:00:55.114+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.118309+0000) 2019-10-13T08:00:55.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:55.109+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.112404+0000) 2019-10-13T08:00:55.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:00:55.109+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.112651+0000) 2019-10-13T08:00:55.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:00:55.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.113005+0000) 2019-10-13T08:00:55.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:55.113+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:55.114223+0000) 2019-10-13T08:00:57.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:00:57.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:57.129810+0000) 2019-10-13T08:00:57.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:00:57.541+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:00:57.544058+0000) 2019-10-13T08:01:00.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:00.113+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.115110+0000) 2019-10-13T08:01:00.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:00.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.115175+0000) 2019-10-13T08:01:00.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:00.113+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.116400+0000) 2019-10-13T08:01:00.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:00.113+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.116831+0000) 2019-10-13T08:01:00.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:00.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:00.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:00.117+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.117927+0000) 2019-10-13T08:01:00.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:00.117+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.118320+0000) 2019-10-13T08:01:00.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:00.117+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.119517+0000) 2019-10-13T08:01:00.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:00.118+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.123168+0000) 2019-10-13T08:01:00.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:00.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.670375+0000) 2019-10-13T08:01:00.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:00.765+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:00.768285+0000) 2019-10-13T08:01:01.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:01.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:01.565580+0000) 2019-10-13T08:01:01.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:01.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:01.566319+0000) 2019-10-13T08:01:01.924 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:01:01.924 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:01.928 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:01:01.928 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:03.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:03.866+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:03.870187+0000) 2019-10-13T08:01:04.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:04.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:04.289569+0000) 2019-10-13T08:01:05.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:05.116+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.121064+0000) 2019-10-13T08:01:05.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:05.116+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.121657+0000) 2019-10-13T08:01:05.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:05.120+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.121994+0000) 2019-10-13T08:01:05.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:05.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.122524+0000) 2019-10-13T08:01:05.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:05.120+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.122675+0000) 2019-10-13T08:01:05.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:05.120+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:05.124 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:05.118+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.122793+0000) 2019-10-13T08:01:05.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:05.120+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.124586+0000) 2019-10-13T08:01:05.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:05.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:05.130501+0000) 2019-10-13T08:01:07.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:07.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:07.129988+0000) 2019-10-13T08:01:07.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:07.540+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:07.544293+0000) 2019-10-13T08:01:10.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:10.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.125033+0000) 2019-10-13T08:01:10.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:10.124+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.126558+0000) 2019-10-13T08:01:10.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:10.124+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.126857+0000) 2019-10-13T08:01:10.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:10.124+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.127244+0000) 2019-10-13T08:01:10.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:10.124+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.127675+0000) 2019-10-13T08:01:10.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:10.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:10.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:10.124+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.129519+0000) 2019-10-13T08:01:10.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:10.128+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.130113+0000) 2019-10-13T08:01:10.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:10.134+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.136289+0000) 2019-10-13T08:01:10.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:10.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.670550+0000) 2019-10-13T08:01:10.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:10.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:10.768480+0000) 2019-10-13T08:01:11.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:11.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:11.565773+0000) 2019-10-13T08:01:11.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:11.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:11.566546+0000) 2019-10-13T08:01:13.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:13.874+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:13.876820+0000) 2019-10-13T08:01:14.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:14.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:14.289811+0000) 2019-10-13T08:01:15.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:15.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.131931+0000) 2019-10-13T08:01:15.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:15.128+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.132374+0000) 2019-10-13T08:01:15.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:15.130+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.132298+0000) 2019-10-13T08:01:15.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:15.128+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.133003+0000) 2019-10-13T08:01:15.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:15.128+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.133528+0000) 2019-10-13T08:01:15.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:15.132+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.134726+0000) 2019-10-13T08:01:15.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:15.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:15.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:15.132+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.135233+0000) 2019-10-13T08:01:15.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:15.134+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:15.140011+0000) 2019-10-13T08:01:17.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:17.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:17.130206+0000) 2019-10-13T08:01:17.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:17.540+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:17.544493+0000) 2019-10-13T08:01:20.138 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:20.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.136796+0000) 2019-10-13T08:01:20.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:20.136+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.138161+0000) 2019-10-13T08:01:20.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:20.136+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.140055+0000) 2019-10-13T08:01:20.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:20.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:20.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:20.136+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.140384+0000) 2019-10-13T08:01:20.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:20.136+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.140732+0000) 2019-10-13T08:01:20.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:20.140+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.142019+0000) 2019-10-13T08:01:20.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:20.140+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.142518+0000) 2019-10-13T08:01:20.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:20.146+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.149519+0000) 2019-10-13T08:01:20.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:20.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.670730+0000) 2019-10-13T08:01:20.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:20.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:20.768657+0000) 2019-10-13T08:01:21.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:21.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:21.565964+0000) 2019-10-13T08:01:21.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:21.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:21.566779+0000) 2019-10-13T08:01:23.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:23.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:23.883201+0000) 2019-10-13T08:01:24.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:24.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:24.290014+0000) 2019-10-13T08:01:25.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:25.140+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.143899+0000) 2019-10-13T08:01:25.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:25.137+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.143740+0000) 2019-10-13T08:01:25.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:25.140+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.144649+0000) 2019-10-13T08:01:25.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:25.140+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.145600+0000) 2019-10-13T08:01:25.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:25.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:25.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:25.144+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.146026+0000) 2019-10-13T08:01:25.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:25.144+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.146485+0000) 2019-10-13T08:01:25.148 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:25.144+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.146764+0000) 2019-10-13T08:01:25.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:25.149+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:25.152266+0000) 2019-10-13T08:01:27.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:27.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:27.130418+0000) 2019-10-13T08:01:27.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:27.540+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:27.544687+0000) 2019-10-13T08:01:30.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:30.144+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.148724+0000) 2019-10-13T08:01:30.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:30.144+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.148944+0000) 2019-10-13T08:01:30.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:30.144+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.149517+0000) 2019-10-13T08:01:30.151 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:30.145+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.150658+0000) 2019-10-13T08:01:30.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:30.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:30.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:30.148+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.152226+0000) 2019-10-13T08:01:30.153 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:30.148+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.152661+0000) 2019-10-13T08:01:30.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:30.152+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.154922+0000) 2019-10-13T08:01:30.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:30.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.158862+0000) 2019-10-13T08:01:30.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:30.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.670912+0000) 2019-10-13T08:01:30.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:30.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:30.768831+0000) 2019-10-13T08:01:31.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:31.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:31.566161+0000) 2019-10-13T08:01:31.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:31.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:31.567011+0000) 2019-10-13T08:01:32.006 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:01:32.006 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:32.010 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:01:32.010 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:01:33.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:33.885+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:33.889748+0000) 2019-10-13T08:01:34.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:34.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:34.290213+0000) 2019-10-13T08:01:35.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:35.152+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.154464+0000) 2019-10-13T08:01:35.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:35.152+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.155691+0000) 2019-10-13T08:01:35.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:35.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.156315+0000) 2019-10-13T08:01:35.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:35.152+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.157130+0000) 2019-10-13T08:01:35.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:35.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:35.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:35.153+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.157412+0000) 2019-10-13T08:01:35.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:35.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.157531+0000) 2019-10-13T08:01:35.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:35.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.158869+0000) 2019-10-13T08:01:35.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:35.157+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:35.162536+0000) 2019-10-13T08:01:37.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:37.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:37.130634+0000) 2019-10-13T08:01:37.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:37.540+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:37.544888+0000) 2019-10-13T08:01:40.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:40.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.160265+0000) 2019-10-13T08:01:40.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:40.156+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.160448+0000) 2019-10-13T08:01:40.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:40.156+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.161541+0000) 2019-10-13T08:01:40.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:40.160+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.161918+0000) 2019-10-13T08:01:40.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:40.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:40.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:40.160+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.162900+0000) 2019-10-13T08:01:40.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:40.160+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.163674+0000) 2019-10-13T08:01:40.165 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:40.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.163442+0000) 2019-10-13T08:01:40.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:40.165+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.169113+0000) 2019-10-13T08:01:40.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:40.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.671093+0000) 2019-10-13T08:01:40.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:40.768+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:40.769036+0000) 2019-10-13T08:01:41.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:41.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:41.566355+0000) 2019-10-13T08:01:41.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:41.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:41.567241+0000) 2019-10-13T08:01:43.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:43.893+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:43.896187+0000) 2019-10-13T08:01:44.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:44.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:44.290417+0000) 2019-10-13T08:01:45.166 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:45.160+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.165288+0000) 2019-10-13T08:01:45.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:45.164+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.165896+0000) 2019-10-13T08:01:45.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:45.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:45.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:45.164+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.168706+0000) 2019-10-13T08:01:45.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:45.164+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.168855+0000) 2019-10-13T08:01:45.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:45.165+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.168819+0000) 2019-10-13T08:01:45.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:45.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.171110+0000) 2019-10-13T08:01:45.172 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:45.164+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.169707+0000) 2019-10-13T08:01:45.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:45.173+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:45.177854+0000) 2019-10-13T08:01:47.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:47.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:47.130831+0000) 2019-10-13T08:01:47.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:47.544+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:47.545090+0000) 2019-10-13T08:01:50.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:50.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:50.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:50.169+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.173149+0000) 2019-10-13T08:01:50.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:50.168+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.173717+0000) 2019-10-13T08:01:50.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:50.172+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.173878+0000) 2019-10-13T08:01:50.175 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:50.172+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.174484+0000) 2019-10-13T08:01:50.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:50.172+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.174792+0000) 2019-10-13T08:01:50.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:50.172+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.174219+0000) 2019-10-13T08:01:50.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:50.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.176113+0000) 2019-10-13T08:01:50.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:50.177+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.181574+0000) 2019-10-13T08:01:50.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:50.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.671268+0000) 2019-10-13T08:01:50.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:50.764+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:50.769247+0000) 2019-10-13T08:01:51.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:51.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:51.566548+0000) 2019-10-13T08:01:51.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:51.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:51.567437+0000) 2019-10-13T08:01:53.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:53.901+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:53.902670+0000) 2019-10-13T08:01:54.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:54.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:54.290661+0000) 2019-10-13T08:01:55.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:01:55.172+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.176411+0000) 2019-10-13T08:01:55.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:01:55.176+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.177850+0000) 2019-10-13T08:01:55.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:01:55.172+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.177518+0000) 2019-10-13T08:01:55.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:55.176+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.178178+0000) 2019-10-13T08:01:55.179 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:01:55.173+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.177997+0000) 2019-10-13T08:01:55.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:01:55.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:01:55.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:01:55.176+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.178930+0000) 2019-10-13T08:01:55.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:55.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.179946+0000) 2019-10-13T08:01:55.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:01:55.185+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:55.188227+0000) 2019-10-13T08:01:57.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:01:57.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:57.131035+0000) 2019-10-13T08:01:57.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:01:57.544+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:01:57.545300+0000) 2019-10-13T08:02:00.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:00.180+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.181834+0000) 2019-10-13T08:02:00.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:00.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:00.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:00.180+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.185317+0000) 2019-10-13T08:02:00.186 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:00.184+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.185825+0000) 2019-10-13T08:02:00.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:00.184+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.186313+0000) 2019-10-13T08:02:00.188 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:00.184+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.187325+0000) 2019-10-13T08:02:00.192 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:00.181+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.187633+0000) 2019-10-13T08:02:00.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:00.185+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.189720+0000) 2019-10-13T08:02:00.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:00.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.188804+0000) 2019-10-13T08:02:00.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:00.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.671448+0000) 2019-10-13T08:02:00.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:00.763+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:00.769421+0000) 2019-10-13T08:02:01.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:01.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:01.566738+0000) 2019-10-13T08:02:01.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:01.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:01.567633+0000) 2019-10-13T08:02:02.154 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:02:02.155 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:02.161 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:02:02.161 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:03.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:03.905+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:03.909245+0000) 2019-10-13T08:02:04.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:04.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:04.290866+0000) 2019-10-13T08:02:05.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:05.183+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.188632+0000) 2019-10-13T08:02:05.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:05.183+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.189351+0000) 2019-10-13T08:02:05.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:05.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:05.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:05.187+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.191119+0000) 2019-10-13T08:02:05.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:05.187+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.191306+0000) 2019-10-13T08:02:05.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:05.185+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.191394+0000) 2019-10-13T08:02:05.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:05.187+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.191767+0000) 2019-10-13T08:02:05.196 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:05.187+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.192311+0000) 2019-10-13T08:02:05.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:05.193+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:05.198728+0000) 2019-10-13T08:02:07.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:07.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:07.131236+0000) 2019-10-13T08:02:07.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:07.543+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:07.545515+0000) 2019-10-13T08:02:10.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:10.191+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.193929+0000) 2019-10-13T08:02:10.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:10.191+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.194947+0000) 2019-10-13T08:02:10.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:10.191+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.195261+0000) 2019-10-13T08:02:10.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:10.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:10.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:10.193+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.196544+0000) 2019-10-13T08:02:10.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:10.191+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.197487+0000) 2019-10-13T08:02:10.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:10.195+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.199346+0000) 2019-10-13T08:02:10.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:10.195+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.198457+0000) 2019-10-13T08:02:10.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:10.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.203428+0000) 2019-10-13T08:02:10.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:10.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.671643+0000) 2019-10-13T08:02:10.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:10.763+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:10.769590+0000) 2019-10-13T08:02:11.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:11.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:11.566957+0000) 2019-10-13T08:02:11.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:11.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:11.567824+0000) 2019-10-13T08:02:13.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:13.913+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:13.915629+0000) 2019-10-13T08:02:14.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:14.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:14.291101+0000) 2019-10-13T08:02:15.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:15.195+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.200131+0000) 2019-10-13T08:02:15.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:15.195+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.200519+0000) 2019-10-13T08:02:15.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:15.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:15.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:15.199+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.203162+0000) 2019-10-13T08:02:15.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:15.197+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.203097+0000) 2019-10-13T08:02:15.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:15.199+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.204029+0000) 2019-10-13T08:02:15.205 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:15.199+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.204549+0000) 2019-10-13T08:02:15.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:15.199+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.205122+0000) 2019-10-13T08:02:15.213 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:15.209+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:15.212206+0000) 2019-10-13T08:02:17.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:17.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:17.131439+0000) 2019-10-13T08:02:17.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:17.543+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:17.545731+0000) 2019-10-13T08:02:20.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:20.203+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.205992+0000) 2019-10-13T08:02:20.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:20.203+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.206382+0000) 2019-10-13T08:02:20.207 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:20.201+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.206372+0000) 2019-10-13T08:02:20.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:20.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:20.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:20.203+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.208250+0000) 2019-10-13T08:02:20.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:20.203+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.208512+0000) 2019-10-13T08:02:20.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:20.203+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.209301+0000) 2019-10-13T08:02:20.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:20.203+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.208852+0000) 2019-10-13T08:02:20.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:20.213+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.216540+0000) 2019-10-13T08:02:20.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:20.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.671823+0000) 2019-10-13T08:02:20.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:20.767+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:20.769786+0000) 2019-10-13T08:02:21.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:21.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:21.567157+0000) 2019-10-13T08:02:21.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:21.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:21.568032+0000) 2019-10-13T08:02:23.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:23.921+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:23.922079+0000) 2019-10-13T08:02:24.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:24.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:24.291337+0000) 2019-10-13T08:02:25.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:25.207+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:25.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:25.211+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.214083+0000) 2019-10-13T08:02:25.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:25.211+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.214524+0000) 2019-10-13T08:02:25.215 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:25.211+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.214854+0000) 2019-10-13T08:02:25.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:25.211+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.215173+0000) 2019-10-13T08:02:25.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:25.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.215969+0000) 2019-10-13T08:02:25.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:25.211+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.217124+0000) 2019-10-13T08:02:25.218 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:25.215+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.217375+0000) 2019-10-13T08:02:25.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:25.217+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:25.223177+0000) 2019-10-13T08:02:27.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:27.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:27.131643+0000) 2019-10-13T08:02:27.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:27.543+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:27.545975+0000) 2019-10-13T08:02:30.219 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:30.215+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.218513+0000) 2019-10-13T08:02:30.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:30.215+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.219058+0000) 2019-10-13T08:02:30.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:30.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:30.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:30.215+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.219596+0000) 2019-10-13T08:02:30.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:30.215+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.220756+0000) 2019-10-13T08:02:30.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:30.215+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.221097+0000) 2019-10-13T08:02:30.222 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:30.217+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.221055+0000) 2019-10-13T08:02:30.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:30.219+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.222024+0000) 2019-10-13T08:02:30.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:30.225+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.228184+0000) 2019-10-13T08:02:30.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:30.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.672000+0000) 2019-10-13T08:02:30.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:30.767+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:30.769981+0000) 2019-10-13T08:02:31.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:31.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:31.567311+0000) 2019-10-13T08:02:31.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:31.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:31.568275+0000) 2019-10-13T08:02:32.208 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:02:32.208 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:32.212 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:02:32.212 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:02:33.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:33.924+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:33.928476+0000) 2019-10-13T08:02:34.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:34.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:34.291536+0000) 2019-10-13T08:02:35.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:35.219+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.222254+0000) 2019-10-13T08:02:35.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:35.219+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.223860+0000) 2019-10-13T08:02:35.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:35.216+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.223795+0000) 2019-10-13T08:02:35.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:35.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:35.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:35.219+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.225136+0000) 2019-10-13T08:02:35.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:35.219+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.225534+0000) 2019-10-13T08:02:35.226 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:35.223+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.226015+0000) 2019-10-13T08:02:35.227 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:35.223+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.226495+0000) 2019-10-13T08:02:35.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:35.228+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:35.233461+0000) 2019-10-13T08:02:37.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:37.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:37.131838+0000) 2019-10-13T08:02:37.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:37.543+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:37.546193+0000) 2019-10-13T08:02:40.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:40.223+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.228009+0000) 2019-10-13T08:02:40.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:40.227+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.230303+0000) 2019-10-13T08:02:40.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:40.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:40.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:40.227+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.231171+0000) 2019-10-13T08:02:40.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:40.224+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.231290+0000) 2019-10-13T08:02:40.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:40.227+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.232322+0000) 2019-10-13T08:02:40.234 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:40.227+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.233323+0000) 2019-10-13T08:02:40.236 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:40.227+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.232889+0000) 2019-10-13T08:02:40.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:40.232+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.237171+0000) 2019-10-13T08:02:40.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:40.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.672202+0000) 2019-10-13T08:02:40.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:40.767+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:40.770158+0000) 2019-10-13T08:02:41.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:41.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:41.567539+0000) 2019-10-13T08:02:41.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:41.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:41.568476+0000) 2019-10-13T08:02:43.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:43.932+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:43.934904+0000) 2019-10-13T08:02:44.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:44.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:44.291750+0000) 2019-10-13T08:02:45.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:45.231+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.234643+0000) 2019-10-13T08:02:45.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:45.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:45.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:45.231+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.236328+0000) 2019-10-13T08:02:45.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:45.231+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.236820+0000) 2019-10-13T08:02:45.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:45.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.237528+0000) 2019-10-13T08:02:45.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:45.235+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.237877+0000) 2019-10-13T08:02:45.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:45.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.238100+0000) 2019-10-13T08:02:45.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:45.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.238410+0000) 2019-10-13T08:02:45.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:45.240+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:45.245107+0000) 2019-10-13T08:02:47.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:47.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:47.132040+0000) 2019-10-13T08:02:47.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:47.543+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:47.546405+0000) 2019-10-13T08:02:50.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:50.235+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.239095+0000) 2019-10-13T08:02:50.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:50.235+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.239653+0000) 2019-10-13T08:02:50.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:50.235+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.240034+0000) 2019-10-13T08:02:50.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:50.235+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.240587+0000) 2019-10-13T08:02:50.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:50.239+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.241790+0000) 2019-10-13T08:02:50.243 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:50.236+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.241819+0000) 2019-10-13T08:02:50.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:50.239+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.242056+0000) 2019-10-13T08:02:50.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:50.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:50.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:50.244+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.248627+0000) 2019-10-13T08:02:50.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:50.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.672381+0000) 2019-10-13T08:02:50.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:50.767+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:50.770331+0000) 2019-10-13T08:02:51.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:51.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:51.567733+0000) 2019-10-13T08:02:51.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:51.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:51.568668+0000) 2019-10-13T08:02:53.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:53.940+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:53.941461+0000) 2019-10-13T08:02:54.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:54.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:54.291911+0000) 2019-10-13T08:02:55.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:02:55.240+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.245254+0000) 2019-10-13T08:02:55.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:02:55.243+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.246625+0000) 2019-10-13T08:02:55.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:55.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.247055+0000) 2019-10-13T08:02:55.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:02:55.243+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.247526+0000) 2019-10-13T08:02:55.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:02:55.243+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.247830+0000) 2019-10-13T08:02:55.248 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:02:55.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:02:55.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:55.243+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.248678+0000) 2019-10-13T08:02:55.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:02:55.243+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.248297+0000) 2019-10-13T08:02:55.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:02:55.248+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:55.255830+0000) 2019-10-13T08:02:57.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:02:57.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:57.132254+0000) 2019-10-13T08:02:57.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:02:57.542+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:02:57.546604+0000) 2019-10-13T08:03:00.251 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:00.244+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.250665+0000) 2019-10-13T08:03:00.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:00.246+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.252936+0000) 2019-10-13T08:03:00.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:00.246+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.253413+0000) 2019-10-13T08:03:00.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:00.246+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.253588+0000) 2019-10-13T08:03:00.255 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:00.250+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.254262+0000) 2019-10-13T08:03:00.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:00.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:00.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:00.250+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.255702+0000) 2019-10-13T08:03:00.256 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:00.250+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.256108+0000) 2019-10-13T08:03:00.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:00.256+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.261800+0000) 2019-10-13T08:03:00.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:00.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.672577+0000) 2019-10-13T08:03:00.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:00.766+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:00.770524+0000) 2019-10-13T08:03:01.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:01.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:01.567920+0000) 2019-10-13T08:03:01.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:01.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:01.568866+0000) 2019-10-13T08:03:02.330 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:03:02.331 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:02.334 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:03:02.335 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:03.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:03.944+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:03.947898+0000) 2019-10-13T08:03:04.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:04.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:04.292098+0000) 2019-10-13T08:03:05.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:05.250+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.257669+0000) 2019-10-13T08:03:05.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:05.252+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.257554+0000) 2019-10-13T08:03:05.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:05.254+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.258486+0000) 2019-10-13T08:03:05.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:05.254+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.258868+0000) 2019-10-13T08:03:05.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:05.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:05.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:05.254+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.259185+0000) 2019-10-13T08:03:05.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:05.254+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.259628+0000) 2019-10-13T08:03:05.261 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:05.254+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.260486+0000) 2019-10-13T08:03:05.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:05.260+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:05.265206+0000) 2019-10-13T08:03:07.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:07.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:07.132456+0000) 2019-10-13T08:03:07.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:07.542+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:07.546808+0000) 2019-10-13T08:03:10.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:10.258+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.262863+0000) 2019-10-13T08:03:10.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:10.258+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.263072+0000) 2019-10-13T08:03:10.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:10.256+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.263413+0000) 2019-10-13T08:03:10.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:10.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:10.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:10.262+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.265857+0000) 2019-10-13T08:03:10.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:10.262+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.266103+0000) 2019-10-13T08:03:10.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:10.262+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.266411+0000) 2019-10-13T08:03:10.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:10.262+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.268085+0000) 2019-10-13T08:03:10.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:10.268+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.273718+0000) 2019-10-13T08:03:10.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:10.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.672754+0000) 2019-10-13T08:03:10.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:10.766+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:10.770700+0000) 2019-10-13T08:03:11.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:11.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:11.568072+0000) 2019-10-13T08:03:11.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:11.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:11.569080+0000) 2019-10-13T08:03:13.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:13.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:13.953940+0000) 2019-10-13T08:03:14.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:14.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:14.292298+0000) 2019-10-13T08:03:15.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:15.262+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.268227+0000) 2019-10-13T08:03:15.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:15.266+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.270027+0000) 2019-10-13T08:03:15.270 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:15.266+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.270339+0000) 2019-10-13T08:03:15.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:15.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:15.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:15.266+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.271101+0000) 2019-10-13T08:03:15.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:15.266+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.271682+0000) 2019-10-13T08:03:15.272 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:15.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.271580+0000) 2019-10-13T08:03:15.273 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:15.266+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.272366+0000) 2019-10-13T08:03:15.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:15.272+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:15.276828+0000) 2019-10-13T08:03:17.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:17.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:17.132654+0000) 2019-10-13T08:03:17.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:17.546+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:17.547016+0000) 2019-10-13T08:03:20.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:20.270+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.274716+0000) 2019-10-13T08:03:20.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:20.270+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.275609+0000) 2019-10-13T08:03:20.277 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:20.270+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.276376+0000) 2019-10-13T08:03:20.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:20.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:20.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:20.270+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.276939+0000) 2019-10-13T08:03:20.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:20.272+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.276900+0000) 2019-10-13T08:03:20.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:20.274+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.277944+0000) 2019-10-13T08:03:20.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:20.274+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.278606+0000) 2019-10-13T08:03:20.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:20.276+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.282136+0000) 2019-10-13T08:03:20.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:20.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.672993+0000) 2019-10-13T08:03:20.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:20.766+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:20.770891+0000) 2019-10-13T08:03:21.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:21.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:21.568266+0000) 2019-10-13T08:03:21.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:21.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:21.569280+0000) 2019-10-13T08:03:23.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:23.960+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:23.960494+0000) 2019-10-13T08:03:24.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:24.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:24.292499+0000) 2019-10-13T08:03:25.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:25.274+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.279984+0000) 2019-10-13T08:03:25.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:25.274+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.279684+0000) 2019-10-13T08:03:25.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:25.274+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.281268+0000) 2019-10-13T08:03:25.282 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:25.278+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.281729+0000) 2019-10-13T08:03:25.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:25.278+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.282367+0000) 2019-10-13T08:03:25.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:25.278+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.282695+0000) 2019-10-13T08:03:25.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:25.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64480 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:25.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:25.276+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.282248+0000) 2019-10-13T08:03:25.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:25.284+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:25.288800+0000) 2019-10-13T08:03:27.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:27.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:27.132856+0000) 2019-10-13T08:03:27.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:27.542+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:27.547216+0000) 2019-10-13T08:03:30.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:30.282+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.286024+0000) 2019-10-13T08:03:30.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:30.282+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.286449+0000) 2019-10-13T08:03:30.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:30.282+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.286976+0000) 2019-10-13T08:03:30.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:30.282+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.287439+0000) 2019-10-13T08:03:30.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:30.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:30.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:30.282+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.288295+0000) 2019-10-13T08:03:30.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:30.282+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.288717+0000) 2019-10-13T08:03:30.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:30.288+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.288952+0000) 2019-10-13T08:03:30.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:30.288+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.294785+0000) 2019-10-13T08:03:30.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:30.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.673171+0000) 2019-10-13T08:03:30.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:30.770+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:30.771068+0000) 2019-10-13T08:03:31.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:31.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:31.568435+0000) 2019-10-13T08:03:31.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:31.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:31.569482+0000) 2019-10-13T08:03:32.428 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:03:32.428 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:32.432 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:03:32.432 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:03:33.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:33.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:33.966940+0000) 2019-10-13T08:03:34.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:34.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:34.292702+0000) 2019-10-13T08:03:35.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:35.286+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.292072+0000) 2019-10-13T08:03:35.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:35.288+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.291873+0000) 2019-10-13T08:03:35.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:35.286+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.292796+0000) 2019-10-13T08:03:35.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:35.286+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.293021+0000) 2019-10-13T08:03:35.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:35.286+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.293608+0000) 2019-10-13T08:03:35.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:35.290+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:35.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:35.290+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.293955+0000) 2019-10-13T08:03:35.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:35.290+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.294783+0000) 2019-10-13T08:03:35.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:35.300+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:35.300404+0000) 2019-10-13T08:03:37.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:37.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:37.133063+0000) 2019-10-13T08:03:37.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:37.546+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:37.547415+0000) 2019-10-13T08:03:40.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:40.290+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.297290+0000) 2019-10-13T08:03:40.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:40.295+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.298881+0000) 2019-10-13T08:03:40.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:40.294+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:40.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:40.294+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.300133+0000) 2019-10-13T08:03:40.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:40.294+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.300906+0000) 2019-10-13T08:03:40.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:40.294+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.301246+0000) 2019-10-13T08:03:40.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:40.298+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.301983+0000) 2019-10-13T08:03:40.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:40.294+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.300999+0000) 2019-10-13T08:03:40.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:40.303+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.307705+0000) 2019-10-13T08:03:40.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:40.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.673354+0000) 2019-10-13T08:03:40.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:40.770+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:40.771245+0000) 2019-10-13T08:03:41.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:41.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:41.568582+0000) 2019-10-13T08:03:41.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:41.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:41.569712+0000) 2019-10-13T08:03:43.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:43.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:43.973411+0000) 2019-10-13T08:03:44.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:44.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:44.292903+0000) 2019-10-13T08:03:45.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:45.298+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.302906+0000) 2019-10-13T08:03:45.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:45.299+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.302909+0000) 2019-10-13T08:03:45.305 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:45.298+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:45.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:45.302+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.305652+0000) 2019-10-13T08:03:45.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:45.302+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.306288+0000) 2019-10-13T08:03:45.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:45.302+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.306855+0000) 2019-10-13T08:03:45.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:45.302+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.306865+0000) 2019-10-13T08:03:45.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:45.302+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.308436+0000) 2019-10-13T08:03:45.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:45.311+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:45.313901+0000) 2019-10-13T08:03:47.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:47.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:47.133242+0000) 2019-10-13T08:03:47.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:47.546+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:47.547645+0000) 2019-10-13T08:03:50.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:50.303+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.308001+0000) 2019-10-13T08:03:50.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:50.302+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.308987+0000) 2019-10-13T08:03:50.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:50.302+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.309698+0000) 2019-10-13T08:03:50.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:50.306+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.310275+0000) 2019-10-13T08:03:50.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:50.306+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:50.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:50.306+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.310804+0000) 2019-10-13T08:03:50.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:50.306+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.311001+0000) 2019-10-13T08:03:50.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:50.306+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.311625+0000) 2019-10-13T08:03:50.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:50.319+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.320188+0000) 2019-10-13T08:03:50.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:50.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.673573+0000) 2019-10-13T08:03:50.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:50.766+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:50.771449+0000) 2019-10-13T08:03:51.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:51.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:51.568751+0000) 2019-10-13T08:03:51.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:51.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:51.569904+0000) 2019-10-13T08:03:53.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:53.983+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:53.984632+0000) 2019-10-13T08:03:54.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:54.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:54.293062+0000) 2019-10-13T08:03:55.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:55.309+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.314309+0000) 2019-10-13T08:03:55.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:03:55.309+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.314596+0000) 2019-10-13T08:03:55.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:03:55.311+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.314577+0000) 2019-10-13T08:03:55.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:55.309+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.315559+0000) 2019-10-13T08:03:55.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:03:55.309+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.315917+0000) 2019-10-13T08:03:55.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:03:55.309+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.316518+0000) 2019-10-13T08:03:55.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:03:55.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:03:55.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:03:55.309+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.317689+0000) 2019-10-13T08:03:55.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:03:55.319+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:55.323050+0000) 2019-10-13T08:03:57.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:03:57.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:57.133463+0000) 2019-10-13T08:03:57.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:03:57.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:03:57.547845+0000) 2019-10-13T08:04:00.321 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:00.315+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.319836+0000) 2019-10-13T08:04:00.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:00.313+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.320754+0000) 2019-10-13T08:04:00.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:00.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:00.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:00.317+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.323054+0000) 2019-10-13T08:04:00.324 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:00.317+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.323196+0000) 2019-10-13T08:04:00.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:00.317+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.324551+0000) 2019-10-13T08:04:00.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:00.317+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.324940+0000) 2019-10-13T08:04:00.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:00.317+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.325459+0000) 2019-10-13T08:04:00.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:00.327+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.329427+0000) 2019-10-13T08:04:00.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:00.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.673777+0000) 2019-10-13T08:04:00.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:00.769+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:00.771643+0000) 2019-10-13T08:04:01.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:01.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:01.568951+0000) 2019-10-13T08:04:01.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:01.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:01.570103+0000) 2019-10-13T08:04:02.730 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:04:02.730 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:02.734 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:04:02.734 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:03.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:03.987+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:03.991062+0000) 2019-10-13T08:04:04.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:04.289+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:04.293263+0000) 2019-10-13T08:04:05.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:05.321+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.326369+0000) 2019-10-13T08:04:05.327 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:05.323+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.326127+0000) 2019-10-13T08:04:05.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:05.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.328470+0000) 2019-10-13T08:04:05.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:05.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:05.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:05.321+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.328963+0000) 2019-10-13T08:04:05.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:05.321+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.329714+0000) 2019-10-13T08:04:05.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:05.325+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.329861+0000) 2019-10-13T08:04:05.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:05.325+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.331718+0000) 2019-10-13T08:04:05.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:05.331+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:05.334442+0000) 2019-10-13T08:04:07.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:07.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:07.133705+0000) 2019-10-13T08:04:07.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:07.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:07.548038+0000) 2019-10-13T08:04:10.332 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:10.329+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.331540+0000) 2019-10-13T08:04:10.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:10.329+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.332298+0000) 2019-10-13T08:04:10.333 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:10.327+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.332050+0000) 2019-10-13T08:04:10.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:10.329+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.333455+0000) 2019-10-13T08:04:10.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:10.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:10.334 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:10.333+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.333814+0000) 2019-10-13T08:04:10.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:10.333+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.334831+0000) 2019-10-13T08:04:10.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:10.333+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.335628+0000) 2019-10-13T08:04:10.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:10.339+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.341067+0000) 2019-10-13T08:04:10.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:10.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.673952+0000) 2019-10-13T08:04:10.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:10.769+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:10.771834+0000) 2019-10-13T08:04:11.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:11.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:11.569172+0000) 2019-10-13T08:04:11.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:11.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:11.570329+0000) 2019-10-13T08:04:13.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:13.995+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:13.997543+0000) 2019-10-13T08:04:14.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:14.289+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:14.293482+0000) 2019-10-13T08:04:15.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:15.333+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.337412+0000) 2019-10-13T08:04:15.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:15.337+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.338087+0000) 2019-10-13T08:04:15.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:15.337+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.338494+0000) 2019-10-13T08:04:15.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:15.337+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.338640+0000) 2019-10-13T08:04:15.339 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:15.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.338550+0000) 2019-10-13T08:04:15.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:15.337+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.339631+0000) 2019-10-13T08:04:15.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:15.337+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.340242+0000) 2019-10-13T08:04:15.341 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:15.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:15.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:15.343+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:15.345646+0000) 2019-10-13T08:04:17.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:17.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:17.133936+0000) 2019-10-13T08:04:17.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:17.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:17.548264+0000) 2019-10-13T08:04:20.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:20.341+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.344492+0000) 2019-10-13T08:04:20.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:20.341+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.344956+0000) 2019-10-13T08:04:20.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:20.343+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.345074+0000) 2019-10-13T08:04:20.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:20.341+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.345207+0000) 2019-10-13T08:04:20.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:20.341+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.345709+0000) 2019-10-13T08:04:20.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:20.345+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.346088+0000) 2019-10-13T08:04:20.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:20.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:20.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:20.345+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.347103+0000) 2019-10-13T08:04:20.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:20.351+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.352371+0000) 2019-10-13T08:04:20.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:20.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.674167+0000) 2019-10-13T08:04:20.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:20.769+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:20.772043+0000) 2019-10-13T08:04:21.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:21.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:21.569361+0000) 2019-10-13T08:04:21.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:21.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:21.570549+0000) 2019-10-13T08:04:24.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:23.999+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.003983+0000) 2019-10-13T08:04:24.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:24.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:24.293682+0000) 2019-10-13T08:04:25.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:25.349+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.351057+0000) 2019-10-13T08:04:25.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:25.349+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.351728+0000) 2019-10-13T08:04:25.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:25.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:25.353 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:25.349+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.352791+0000) 2019-10-13T08:04:25.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:25.349+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.353089+0000) 2019-10-13T08:04:25.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:25.351+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.353429+0000) 2019-10-13T08:04:25.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:25.349+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.353622+0000) 2019-10-13T08:04:25.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:25.353+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.354429+0000) 2019-10-13T08:04:25.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:25.359+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:25.360496+0000) 2019-10-13T08:04:27.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:27.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:27.134131+0000) 2019-10-13T08:04:27.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:27.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:27.548474+0000) 2019-10-13T08:04:30.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:30.353+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.355202+0000) 2019-10-13T08:04:30.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:30.353+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.356017+0000) 2019-10-13T08:04:30.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:30.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.356711+0000) 2019-10-13T08:04:30.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:30.355+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.356901+0000) 2019-10-13T08:04:30.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:30.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:30.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:30.353+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.357494+0000) 2019-10-13T08:04:30.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:30.357+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.358114+0000) 2019-10-13T08:04:30.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:30.357+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.358563+0000) 2019-10-13T08:04:30.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:30.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.361276+0000) 2019-10-13T08:04:30.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:30.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.674361+0000) 2019-10-13T08:04:30.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:30.769+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:30.772249+0000) 2019-10-13T08:04:31.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:31.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:31.569555+0000) 2019-10-13T08:04:31.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:31.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:31.570744+0000) 2019-10-13T08:04:32.787 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:04:32.787 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:32.791 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:04:32.791 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:04:34.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:34.007+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.010381+0000) 2019-10-13T08:04:34.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:34.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:34.293916+0000) 2019-10-13T08:04:35.360 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:35.357+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.360320+0000) 2019-10-13T08:04:35.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:35.357+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.361022+0000) 2019-10-13T08:04:35.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:35.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:35.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:35.361+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.363265+0000) 2019-10-13T08:04:35.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:35.361+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.363533+0000) 2019-10-13T08:04:35.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:35.359+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.363374+0000) 2019-10-13T08:04:35.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:35.361+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.364153+0000) 2019-10-13T08:04:35.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:35.361+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.365573+0000) 2019-10-13T08:04:35.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:35.363+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:35.367869+0000) 2019-10-13T08:04:37.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:37.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:37.134364+0000) 2019-10-13T08:04:37.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:37.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:37.548717+0000) 2019-10-13T08:04:40.367 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:40.365+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.366808+0000) 2019-10-13T08:04:40.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:40.365+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.367136+0000) 2019-10-13T08:04:40.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:40.365+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.367821+0000) 2019-10-13T08:04:40.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:40.365+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.368007+0000) 2019-10-13T08:04:40.369 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:40.367+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.368117+0000) 2019-10-13T08:04:40.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:40.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:40.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:40.365+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.369600+0000) 2019-10-13T08:04:40.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:40.369+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.369929+0000) 2019-10-13T08:04:40.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:40.371+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.374699+0000) 2019-10-13T08:04:40.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:40.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.674538+0000) 2019-10-13T08:04:40.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:40.769+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:40.772425+0000) 2019-10-13T08:04:41.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:41.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:41.569772+0000) 2019-10-13T08:04:41.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:41.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:41.570942+0000) 2019-10-13T08:04:44.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:44.014+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.016930+0000) 2019-10-13T08:04:44.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:44.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:44.294111+0000) 2019-10-13T08:04:45.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:45.369+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.371800+0000) 2019-10-13T08:04:45.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:45.369+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.372502+0000) 2019-10-13T08:04:45.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:45.369+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.372717+0000) 2019-10-13T08:04:45.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:45.370+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.372900+0000) 2019-10-13T08:04:45.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:45.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:45.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:45.373+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.374551+0000) 2019-10-13T08:04:45.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:45.373+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.374617+0000) 2019-10-13T08:04:45.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:45.373+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.374846+0000) 2019-10-13T08:04:45.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:45.374+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:45.380034+0000) 2019-10-13T08:04:47.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:47.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:47.134579+0000) 2019-10-13T08:04:47.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:47.545+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:47.548909+0000) 2019-10-13T08:04:50.378 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:50.377+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.377926+0000) 2019-10-13T08:04:50.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:50.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:50.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:50.377+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.380442+0000) 2019-10-13T08:04:50.382 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:50.377+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.380927+0000) 2019-10-13T08:04:50.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:50.377+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.381393+0000) 2019-10-13T08:04:50.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:50.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.382708+0000) 2019-10-13T08:04:50.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:50.381+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.381896+0000) 2019-10-13T08:04:50.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:50.381+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.384107+0000) 2019-10-13T08:04:50.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:50.382+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.386782+0000) 2019-10-13T08:04:50.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:50.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.674706+0000) 2019-10-13T08:04:50.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:50.768+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:50.772605+0000) 2019-10-13T08:04:51.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:51.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:51.571164+0000) 2019-10-13T08:04:51.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:51.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:51.570005+0000) 2019-10-13T08:04:54.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:54.018+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.023458+0000) 2019-10-13T08:04:54.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:54.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:54.294336+0000) 2019-10-13T08:04:55.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:04:55.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.382803+0000) 2019-10-13T08:04:55.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:04:55.380+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.383884+0000) 2019-10-13T08:04:55.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:04:55.380+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.384758+0000) 2019-10-13T08:04:55.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:55.380+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.385366+0000) 2019-10-13T08:04:55.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:04:55.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.385433+0000) 2019-10-13T08:04:55.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:04:55.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:04:55.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:04:55.384+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.386711+0000) 2019-10-13T08:04:55.387 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:55.384+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.387357+0000) 2019-10-13T08:04:55.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:04:55.386+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:55.391053+0000) 2019-10-13T08:04:57.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:04:57.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:57.134792+0000) 2019-10-13T08:04:57.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:04:57.544+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:04:57.549107+0000) 2019-10-13T08:05:00.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:00.386+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.388367+0000) 2019-10-13T08:05:00.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:00.384+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.389480+0000) 2019-10-13T08:05:00.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:00.388+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.390360+0000) 2019-10-13T08:05:00.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:00.388+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.390684+0000) 2019-10-13T08:05:00.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:00.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.390910+0000) 2019-10-13T08:05:00.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:00.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:00.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:00.388+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.392214+0000) 2019-10-13T08:05:00.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:00.388+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.392979+0000) 2019-10-13T08:05:00.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:00.398+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.401847+0000) 2019-10-13T08:05:00.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:00.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.674905+0000) 2019-10-13T08:05:00.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:00.768+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:00.772817+0000) 2019-10-13T08:05:01.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:01.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:01.570236+0000) 2019-10-13T08:05:01.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:01.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:01.571356+0000) 2019-10-13T08:05:02.925 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:05:02.925 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:02.929 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:05:02.929 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:04.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:04.026+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.029783+0000) 2019-10-13T08:05:04.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:04.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:04.294576+0000) 2019-10-13T08:05:05.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:05.392+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.395527+0000) 2019-10-13T08:05:05.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:05.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.395377+0000) 2019-10-13T08:05:05.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:05.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.396563+0000) 2019-10-13T08:05:05.397 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:05.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:05.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:05.396+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.397989+0000) 2019-10-13T08:05:05.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:05.396+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.398417+0000) 2019-10-13T08:05:05.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:05.396+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.400405+0000) 2019-10-13T08:05:05.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:05.396+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.400958+0000) 2019-10-13T08:05:05.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:05.402+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:05.406554+0000) 2019-10-13T08:05:07.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:07.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:07.134986+0000) 2019-10-13T08:05:07.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:07.548+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:07.549310+0000) 2019-10-13T08:05:10.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:10.396+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.400992+0000) 2019-10-13T08:05:10.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:10.400+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.402260+0000) 2019-10-13T08:05:10.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:10.400+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.401917+0000) 2019-10-13T08:05:10.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:10.400+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.402731+0000) 2019-10-13T08:05:10.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:10.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65118 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:10.405 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:10.398+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.402864+0000) 2019-10-13T08:05:10.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:10.400+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.403707+0000) 2019-10-13T08:05:10.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:10.400+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.404296+0000) 2019-10-13T08:05:10.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:10.402+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.407245+0000) 2019-10-13T08:05:10.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:10.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.675135+0000) 2019-10-13T08:05:10.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:10.768+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:10.772993+0000) 2019-10-13T08:05:11.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:11.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:11.570435+0000) 2019-10-13T08:05:11.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:11.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:11.571556+0000) 2019-10-13T08:05:14.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:14.034+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.036283+0000) 2019-10-13T08:05:14.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:14.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:14.294779+0000) 2019-10-13T08:05:15.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:15.404+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.405801+0000) 2019-10-13T08:05:15.407 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:15.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.406797+0000) 2019-10-13T08:05:15.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:15.404+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.408468+0000) 2019-10-13T08:05:15.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:15.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:15.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:15.406+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.408253+0000) 2019-10-13T08:05:15.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:15.408+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.410050+0000) 2019-10-13T08:05:15.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:15.408+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.410404+0000) 2019-10-13T08:05:15.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:15.408+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.410757+0000) 2019-10-13T08:05:15.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:15.414+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:15.416308+0000) 2019-10-13T08:05:17.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:17.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:17.135218+0000) 2019-10-13T08:05:17.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:17.548+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:17.549512+0000) 2019-10-13T08:05:20.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:20.408+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.412383+0000) 2019-10-13T08:05:20.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:20.408+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.412616+0000) 2019-10-13T08:05:20.413 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:20.408+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.412983+0000) 2019-10-13T08:05:20.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:20.408+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.413311+0000) 2019-10-13T08:05:20.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:20.410+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.413801+0000) 2019-10-13T08:05:20.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:20.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65180 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:20.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:20.408+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.413660+0000) 2019-10-13T08:05:20.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:20.412+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.414912+0000) 2019-10-13T08:05:20.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:20.414+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.418900+0000) 2019-10-13T08:05:20.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:20.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.675317+0000) 2019-10-13T08:05:20.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:20.772+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:20.773171+0000) 2019-10-13T08:05:21.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:21.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:21.570628+0000) 2019-10-13T08:05:21.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:21.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:21.571767+0000) 2019-10-13T08:05:24.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:24.042+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.042866+0000) 2019-10-13T08:05:24.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:24.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:24.294984+0000) 2019-10-13T08:05:25.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:25.416+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.418183+0000) 2019-10-13T08:05:25.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:25.416+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.418790+0000) 2019-10-13T08:05:25.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:25.416+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.419264+0000) 2019-10-13T08:05:25.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:25.416+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.418930+0000) 2019-10-13T08:05:25.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:25.416+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:25.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:25.416+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.420952+0000) 2019-10-13T08:05:25.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:25.418+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.421757+0000) 2019-10-13T08:05:25.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:25.420+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.422075+0000) 2019-10-13T08:05:25.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:25.426+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:25.428783+0000) 2019-10-13T08:05:27.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:27.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:27.135425+0000) 2019-10-13T08:05:27.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:27.548+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:27.549712+0000) 2019-10-13T08:05:30.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:30.420+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.423738+0000) 2019-10-13T08:05:30.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:30.420+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.424324+0000) 2019-10-13T08:05:30.425 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:30.420+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.425062+0000) 2019-10-13T08:05:30.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:30.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:30.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:30.424+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.427409+0000) 2019-10-13T08:05:30.428 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:30.422+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.427109+0000) 2019-10-13T08:05:30.428 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:30.424+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.427991+0000) 2019-10-13T08:05:30.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:30.424+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.428644+0000) 2019-10-13T08:05:30.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:30.430+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.433382+0000) 2019-10-13T08:05:30.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:30.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.675499+0000) 2019-10-13T08:05:30.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:30.772+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:30.773386+0000) 2019-10-13T08:05:31.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:31.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:31.570823+0000) 2019-10-13T08:05:31.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:31.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:31.571997+0000) 2019-10-13T08:05:33.049 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:05:33.049 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:33.053 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:05:33.053 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:05:34.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:34.046+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.049237+0000) 2019-10-13T08:05:34.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:34.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:34.295224+0000) 2019-10-13T08:05:35.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:35.428+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.429952+0000) 2019-10-13T08:05:35.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:35.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.429992+0000) 2019-10-13T08:05:35.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:35.438+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.441050+0000) 2019-10-13T08:05:35.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:35.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:35.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:35.428+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.432613+0000) 2019-10-13T08:05:35.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:35.428+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.433056+0000) 2019-10-13T08:05:35.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:35.428+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.433660+0000) 2019-10-13T08:05:35.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:35.432+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.434129+0000) 2019-10-13T08:05:35.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:35.432+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:35.435146+0000) 2019-10-13T08:05:37.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:37.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:37.135593+0000) 2019-10-13T08:05:37.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:37.548+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:37.550006+0000) 2019-10-13T08:05:40.435 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:40.430+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.435011+0000) 2019-10-13T08:05:40.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:40.432+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.436050+0000) 2019-10-13T08:05:40.436 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:40.432+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.436743+0000) 2019-10-13T08:05:40.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:40.432+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.437138+0000) 2019-10-13T08:05:40.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:40.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:40.439 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:40.436+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.439238+0000) 2019-10-13T08:05:40.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:40.436+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.439983+0000) 2019-10-13T08:05:40.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:40.436+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.440971+0000) 2019-10-13T08:05:40.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:40.442+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.445541+0000) 2019-10-13T08:05:40.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:40.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.675682+0000) 2019-10-13T08:05:40.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:40.768+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:40.773563+0000) 2019-10-13T08:05:41.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:41.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:41.571049+0000) 2019-10-13T08:05:41.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:41.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:41.572246+0000) 2019-10-13T08:05:44.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:44.050+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:44.055700+0000) 2019-10-13T08:05:44.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:44.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:44.295424+0000) 2019-10-13T08:05:45.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:45.436+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.441482+0000) 2019-10-13T08:05:45.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:45.440+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.442267+0000) 2019-10-13T08:05:45.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:45.438+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.442452+0000) 2019-10-13T08:05:45.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:45.440+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.443209+0000) 2019-10-13T08:05:45.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:45.440+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.443615+0000) 2019-10-13T08:05:45.444 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:45.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:45.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:45.440+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.443962+0000) 2019-10-13T08:05:45.446 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:45.440+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.444729+0000) 2019-10-13T08:05:45.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:45.442+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:45.447805+0000) 2019-10-13T08:05:47.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:47.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:47.135811+0000) 2019-10-13T08:05:47.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:47.547+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:47.550210+0000) 2019-10-13T08:05:50.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:50.443+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.447987+0000) 2019-10-13T08:05:50.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:50.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.448349+0000) 2019-10-13T08:05:50.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:50.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:50.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:50.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.449454+0000) 2019-10-13T08:05:50.450 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:50.447+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.450482+0000) 2019-10-13T08:05:50.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:50.447+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.450962+0000) 2019-10-13T08:05:50.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:50.447+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.451140+0000) 2019-10-13T08:05:50.451 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:50.447+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.451699+0000) 2019-10-13T08:05:50.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:50.453+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.457425+0000) 2019-10-13T08:05:50.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:50.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.675821+0000) 2019-10-13T08:05:50.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:50.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:50.773765+0000) 2019-10-13T08:05:51.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:51.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:51.571261+0000) 2019-10-13T08:05:51.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:51.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:51.572448+0000) 2019-10-13T08:05:54.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:54.057+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:54.062352+0000) 2019-10-13T08:05:54.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:54.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:54.295644+0000) 2019-10-13T08:05:55.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:05:55.447+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.452517+0000) 2019-10-13T08:05:55.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:05:55.447+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.453293+0000) 2019-10-13T08:05:55.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:55.451+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.454155+0000) 2019-10-13T08:05:55.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:05:55.451+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.454800+0000) 2019-10-13T08:05:55.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:05:55.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:05:55.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:55.451+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.455257+0000) 2019-10-13T08:05:55.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:05:55.451+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.455541+0000) 2019-10-13T08:05:55.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:05:55.449+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.453153+0000) 2019-10-13T08:05:55.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:05:55.461+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:55.464297+0000) 2019-10-13T08:05:57.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:05:57.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:57.136012+0000) 2019-10-13T08:05:57.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:05:57.547+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:05:57.550416+0000) 2019-10-13T08:06:00.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:00.453+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.458169+0000) 2019-10-13T08:06:00.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:00.455+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.459016+0000) 2019-10-13T08:06:00.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:00.455+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.459808+0000) 2019-10-13T08:06:00.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:00.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:00.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:00.455+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.461480+0000) 2019-10-13T08:06:00.462 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:00.455+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.461714+0000) 2019-10-13T08:06:00.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:00.459+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.462525+0000) 2019-10-13T08:06:00.463 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:00.459+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.463293+0000) 2019-10-13T08:06:00.470 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:00.465+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.470142+0000) 2019-10-13T08:06:00.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:00.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.675996+0000) 2019-10-13T08:06:00.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:00.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:00.773964+0000) 2019-10-13T08:06:01.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:01.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:01.571460+0000) 2019-10-13T08:06:01.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:01.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:01.572640+0000) 2019-10-13T08:06:03.323 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:06:03.323 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:03.327 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:06:03.327 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:04.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:04.065+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:04.068802+0000) 2019-10-13T08:06:04.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:04.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:04.295847+0000) 2019-10-13T08:06:05.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:05.459+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.464185+0000) 2019-10-13T08:06:05.465 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:05.459+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.464615+0000) 2019-10-13T08:06:05.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:05.459+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.465218+0000) 2019-10-13T08:06:05.466 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:05.463+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.465868+0000) 2019-10-13T08:06:05.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:05.461+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.465643+0000) 2019-10-13T08:06:05.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:05.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:05.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:05.463+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.467364+0000) 2019-10-13T08:06:05.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:05.463+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.468001+0000) 2019-10-13T08:06:05.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:05.469+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:05.472314+0000) 2019-10-13T08:06:07.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:07.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:07.136219+0000) 2019-10-13T08:06:07.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:07.547+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:07.550628+0000) 2019-10-13T08:06:10.470 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:10.465+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.469524+0000) 2019-10-13T08:06:10.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:10.467+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.470386+0000) 2019-10-13T08:06:10.471 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:10.467+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.471336+0000) 2019-10-13T08:06:10.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:10.467+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.471485+0000) 2019-10-13T08:06:10.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:10.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:10.473 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:10.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.471836+0000) 2019-10-13T08:06:10.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:10.467+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.472637+0000) 2019-10-13T08:06:10.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:10.467+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.473314+0000) 2019-10-13T08:06:10.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:10.469+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.476003+0000) 2019-10-13T08:06:10.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:10.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.676199+0000) 2019-10-13T08:06:10.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:10.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:10.774162+0000) 2019-10-13T08:06:11.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:11.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:11.571673+0000) 2019-10-13T08:06:11.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:11.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:11.572836+0000) 2019-10-13T08:06:14.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:14.073+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:14.075136+0000) 2019-10-13T08:06:14.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:14.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:14.296048+0000) 2019-10-13T08:06:15.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:15.471+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.474992+0000) 2019-10-13T08:06:15.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:15.469+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.474798+0000) 2019-10-13T08:06:15.476 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:15.471+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.476094+0000) 2019-10-13T08:06:15.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:15.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:15.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:15.471+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.477167+0000) 2019-10-13T08:06:15.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:15.471+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.477709+0000) 2019-10-13T08:06:15.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:15.475+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.478031+0000) 2019-10-13T08:06:15.480 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:15.475+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.478715+0000) 2019-10-13T08:06:15.484 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:15.477+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:15.483370+0000) 2019-10-13T08:06:17.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:17.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:17.136415+0000) 2019-10-13T08:06:17.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:17.547+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:17.550827+0000) 2019-10-13T08:06:20.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:20.473+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.479987+0000) 2019-10-13T08:06:20.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:20.479+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.481888+0000) 2019-10-13T08:06:20.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:20.479+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.482113+0000) 2019-10-13T08:06:20.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:20.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:20.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:20.479+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.483848+0000) 2019-10-13T08:06:20.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:20.479+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.484750+0000) 2019-10-13T08:06:20.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:20.479+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.485184+0000) 2019-10-13T08:06:20.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:20.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.485922+0000) 2019-10-13T08:06:20.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:20.485+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.489375+0000) 2019-10-13T08:06:20.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:20.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.676375+0000) 2019-10-13T08:06:20.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:20.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:20.774369+0000) 2019-10-13T08:06:21.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:21.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:21.571868+0000) 2019-10-13T08:06:21.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:21.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:21.573052+0000) 2019-10-13T08:06:24.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:24.077+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:24.081648+0000) 2019-10-13T08:06:24.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:24.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:24.296276+0000) 2019-10-13T08:06:25.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:25.483+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.486935+0000) 2019-10-13T08:06:25.488 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:25.481+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.488060+0000) 2019-10-13T08:06:25.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:25.483+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.489169+0000) 2019-10-13T08:06:25.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:25.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:25.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:25.487+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.489908+0000) 2019-10-13T08:06:25.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:25.487+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.490275+0000) 2019-10-13T08:06:25.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:25.487+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.490636+0000) 2019-10-13T08:06:25.492 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:25.487+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.491981+0000) 2019-10-13T08:06:25.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:25.493+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:25.496712+0000) 2019-10-13T08:06:27.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:27.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:27.136619+0000) 2019-10-13T08:06:27.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:27.547+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:27.551046+0000) 2019-10-13T08:06:30.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:30.485+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.492029+0000) 2019-10-13T08:06:30.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:30.491+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.494152+0000) 2019-10-13T08:06:30.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:30.491+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.494435+0000) 2019-10-13T08:06:30.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:30.491+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.494659+0000) 2019-10-13T08:06:30.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:30.491+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.494846+0000) 2019-10-13T08:06:30.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:30.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:30.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:30.491+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.495409+0000) 2019-10-13T08:06:30.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:30.491+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.496373+0000) 2019-10-13T08:06:30.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:30.497+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.503554+0000) 2019-10-13T08:06:30.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:30.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.676588+0000) 2019-10-13T08:06:30.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:30.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:30.774546+0000) 2019-10-13T08:06:31.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:31.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:31.572065+0000) 2019-10-13T08:06:31.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:31.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:31.573308+0000) 2019-10-13T08:06:33.512 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:06:33.512 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:33.516 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:06:33.516 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:06:34.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:34.085+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:34.088197+0000) 2019-10-13T08:06:34.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:34.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:34.296480+0000) 2019-10-13T08:06:35.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:35.495+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.498908+0000) 2019-10-13T08:06:35.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:35.493+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.499185+0000) 2019-10-13T08:06:35.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:35.495+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.500268+0000) 2019-10-13T08:06:35.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:35.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:35.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:35.499+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.501931+0000) 2019-10-13T08:06:35.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:35.499+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.502442+0000) 2019-10-13T08:06:35.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:35.499+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.502776+0000) 2019-10-13T08:06:35.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:35.499+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.503109+0000) 2019-10-13T08:06:35.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:35.501+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:35.506253+0000) 2019-10-13T08:06:37.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:37.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:37.136852+0000) 2019-10-13T08:06:37.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:37.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:37.551246+0000) 2019-10-13T08:06:40.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:40.499+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.504596+0000) 2019-10-13T08:06:40.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:40.499+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.505339+0000) 2019-10-13T08:06:40.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:40.503+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.505917+0000) 2019-10-13T08:06:40.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:40.503+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.506413+0000) 2019-10-13T08:06:40.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:40.503+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.506635+0000) 2019-10-13T08:06:40.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:40.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:40.508 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:40.503+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.507902+0000) 2019-10-13T08:06:40.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:40.501+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.507776+0000) 2019-10-13T08:06:40.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:40.509+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.514092+0000) 2019-10-13T08:06:40.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:40.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.676767+0000) 2019-10-13T08:06:40.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:40.771+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:40.774743+0000) 2019-10-13T08:06:41.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:41.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:41.572298+0000) 2019-10-13T08:06:41.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:41.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:41.573526+0000) 2019-10-13T08:06:44.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:44.093+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:44.094693+0000) 2019-10-13T08:06:44.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:44.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:44.296648+0000) 2019-10-13T08:06:45.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:45.506+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.510551+0000) 2019-10-13T08:06:45.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:45.506+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.510890+0000) 2019-10-13T08:06:45.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:45.506+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.512779+0000) 2019-10-13T08:06:45.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:45.509+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.512540+0000) 2019-10-13T08:06:45.514 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:45.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:45.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:45.506+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.513478+0000) 2019-10-13T08:06:45.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:45.510+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.514110+0000) 2019-10-13T08:06:45.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:45.510+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.514208+0000) 2019-10-13T08:06:45.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:45.513+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:45.517207+0000) 2019-10-13T08:06:47.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:47.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:47.137011+0000) 2019-10-13T08:06:47.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:47.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:47.551454+0000) 2019-10-13T08:06:50.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:50.510+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.516393+0000) 2019-10-13T08:06:50.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:50.513+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.516408+0000) 2019-10-13T08:06:50.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:50.510+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.517123+0000) 2019-10-13T08:06:50.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:50.514+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.517837+0000) 2019-10-13T08:06:50.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:50.514+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.518461+0000) 2019-10-13T08:06:50.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:50.514+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.519052+0000) 2019-10-13T08:06:50.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:50.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:50.519 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:50.514+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.519363+0000) 2019-10-13T08:06:50.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:50.521+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.526024+0000) 2019-10-13T08:06:50.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:50.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.676949+0000) 2019-10-13T08:06:50.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:50.774+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:50.774960+0000) 2019-10-13T08:06:51.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:51.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:51.572506+0000) 2019-10-13T08:06:51.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:51.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:51.573734+0000) 2019-10-13T08:06:54.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:54.097+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:54.101160+0000) 2019-10-13T08:06:54.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:54.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:54.296852+0000) 2019-10-13T08:06:55.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:06:55.518+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.522457+0000) 2019-10-13T08:06:55.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:06:55.518+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.523275+0000) 2019-10-13T08:06:55.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:06:55.517+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.523223+0000) 2019-10-13T08:06:55.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:06:55.518+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.524187+0000) 2019-10-13T08:06:55.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:06:55.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:06:55.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:06:55.518+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.525107+0000) 2019-10-13T08:06:55.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:55.518+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.524705+0000) 2019-10-13T08:06:55.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:55.522+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.526166+0000) 2019-10-13T08:06:55.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:06:55.525+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:55.528678+0000) 2019-10-13T08:06:57.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:06:57.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:57.137208+0000) 2019-10-13T08:06:57.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:06:57.546+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:06:57.551647+0000) 2019-10-13T08:07:00.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:00.522+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.528706+0000) 2019-10-13T08:07:00.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:00.526+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.529773+0000) 2019-10-13T08:07:00.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:00.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:00.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:00.524+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.530506+0000) 2019-10-13T08:07:00.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:00.526+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.531451+0000) 2019-10-13T08:07:00.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:00.526+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.532221+0000) 2019-10-13T08:07:00.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:00.526+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.532729+0000) 2019-10-13T08:07:00.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:00.526+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.532753+0000) 2019-10-13T08:07:00.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:00.528+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.535271+0000) 2019-10-13T08:07:00.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:00.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.677125+0000) 2019-10-13T08:07:00.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:00.770+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:00.775206+0000) 2019-10-13T08:07:01.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:01.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:01.572734+0000) 2019-10-13T08:07:01.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:01.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:01.573934+0000) 2019-10-13T08:07:03.761 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:07:03.761 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:03.765 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:07:03.765 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:04.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:04.100+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:04.107586+0000) 2019-10-13T08:07:04.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:04.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:04.297048+0000) 2019-10-13T08:07:05.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:05.528+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.533161+0000) 2019-10-13T08:07:05.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:05.530+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.534036+0000) 2019-10-13T08:07:05.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:05.530+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.535898+0000) 2019-10-13T08:07:05.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:05.530+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.536309+0000) 2019-10-13T08:07:05.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:05.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65820 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:05.538 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:05.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.537926+0000) 2019-10-13T08:07:05.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:05.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.537384+0000) 2019-10-13T08:07:05.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:05.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.540472+0000) 2019-10-13T08:07:05.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:05.540+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:05.546213+0000) 2019-10-13T08:07:07.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:07.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:07.137417+0000) 2019-10-13T08:07:07.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:07.546+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:07.551876+0000) 2019-10-13T08:07:10.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:10.532+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.539110+0000) 2019-10-13T08:07:10.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:10.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.540267+0000) 2019-10-13T08:07:10.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:10.534+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.541303+0000) 2019-10-13T08:07:10.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:10.534+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.541606+0000) 2019-10-13T08:07:10.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:10.538+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.541993+0000) 2019-10-13T08:07:10.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:10.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:10.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:10.538+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.542747+0000) 2019-10-13T08:07:10.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:10.538+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.543226+0000) 2019-10-13T08:07:10.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:10.540+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.545512+0000) 2019-10-13T08:07:10.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:10.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.677313+0000) 2019-10-13T08:07:10.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:10.774+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:10.775383+0000) 2019-10-13T08:07:11.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:11.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:11.572928+0000) 2019-10-13T08:07:11.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:11.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:11.574161+0000) 2019-10-13T08:07:14.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:14.112+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:14.113992+0000) 2019-10-13T08:07:14.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:14.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:14.297249+0000) 2019-10-13T08:07:15.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:15.542+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.545703+0000) 2019-10-13T08:07:15.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:15.542+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.546706+0000) 2019-10-13T08:07:15.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:15.542+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.547742+0000) 2019-10-13T08:07:15.548 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:15.542+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.548160+0000) 2019-10-13T08:07:15.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:15.542+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.548263+0000) 2019-10-13T08:07:15.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:15.544+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.548146+0000) 2019-10-13T08:07:15.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:15.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:15.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:15.546+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.549812+0000) 2019-10-13T08:07:15.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:15.548+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:15.555272+0000) 2019-10-13T08:07:17.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:17.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:17.137647+0000) 2019-10-13T08:07:17.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:17.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:17.552110+0000) 2019-10-13T08:07:20.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:20.546+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.552882+0000) 2019-10-13T08:07:20.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:20.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.553244+0000) 2019-10-13T08:07:20.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:20.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.553441+0000) 2019-10-13T08:07:20.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:20.550+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.554081+0000) 2019-10-13T08:07:20.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:20.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65911 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:20.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:20.550+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.554834+0000) 2019-10-13T08:07:20.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:20.550+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.555246+0000) 2019-10-13T08:07:20.557 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:20.550+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.556220+0000) 2019-10-13T08:07:20.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:20.556+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.561809+0000) 2019-10-13T08:07:20.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:20.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.677492+0000) 2019-10-13T08:07:20.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:20.774+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:20.775577+0000) 2019-10-13T08:07:21.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:21.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:21.573126+0000) 2019-10-13T08:07:21.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:21.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:21.574392+0000) 2019-10-13T08:07:24.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:24.116+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:24.120555+0000) 2019-10-13T08:07:24.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:24.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:24.297472+0000) 2019-10-13T08:07:25.559 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:25.554+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.558638+0000) 2019-10-13T08:07:25.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:25.554+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.559134+0000) 2019-10-13T08:07:25.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:25.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:25.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:25.554+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.561067+0000) 2019-10-13T08:07:25.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:25.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.561684+0000) 2019-10-13T08:07:25.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:25.556+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.562140+0000) 2019-10-13T08:07:25.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:25.558+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.563647+0000) 2019-10-13T08:07:25.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:25.558+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.561995+0000) 2019-10-13T08:07:25.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:25.564+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:25.569006+0000) 2019-10-13T08:07:27.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:27.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:27.137924+0000) 2019-10-13T08:07:29.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:27.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:27.552312+0000) 2019-10-13T08:07:30.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:30.558+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.563585+0000) 2019-10-13T08:07:30.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:30.558+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.564361+0000) 2019-10-13T08:07:30.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:30.558+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.565282+0000) 2019-10-13T08:07:30.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:30.562+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.566114+0000) 2019-10-13T08:07:30.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:30.562+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.566468+0000) 2019-10-13T08:07:30.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:30.560+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.566336+0000) 2019-10-13T08:07:30.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:30.562+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.567643+0000) 2019-10-13T08:07:30.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:30.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:30.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:30.568+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.572817+0000) 2019-10-13T08:07:30.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:30.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.677673+0000) 2019-10-13T08:07:30.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:30.770+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:30.775757+0000) 2019-10-13T08:07:31.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:31.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:31.573334+0000) 2019-10-13T08:07:31.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:31.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:31.574621+0000) 2019-10-13T08:07:33.846 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:07:33.846 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:33.850 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:07:33.850 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:07:34.128 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:34.124+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:34.127061+0000) 2019-10-13T08:07:34.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:34.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:34.297723+0000) 2019-10-13T08:07:35.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:35.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.572038+0000) 2019-10-13T08:07:35.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:35.566+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.572449+0000) 2019-10-13T08:07:35.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:35.566+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.572778+0000) 2019-10-13T08:07:35.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:35.566+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.573184+0000) 2019-10-13T08:07:35.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:35.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:35.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:35.570+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.573808+0000) 2019-10-13T08:07:35.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:35.568+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.573032+0000) 2019-10-13T08:07:35.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:35.570+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.574983+0000) 2019-10-13T08:07:35.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:35.576+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:35.580597+0000) 2019-10-13T08:07:37.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:37.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:37.138123+0000) 2019-10-13T08:07:37.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:37.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:37.552515+0000) 2019-10-13T08:07:40.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:40.570+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.577294+0000) 2019-10-13T08:07:40.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:40.570+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.577681+0000) 2019-10-13T08:07:40.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:40.574+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.578358+0000) 2019-10-13T08:07:40.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:40.574+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.578533+0000) 2019-10-13T08:07:40.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:40.574+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.579020+0000) 2019-10-13T08:07:40.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:40.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:40.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:40.572+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.579550+0000) 2019-10-13T08:07:40.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:40.574+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.580405+0000) 2019-10-13T08:07:40.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:40.580+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.587752+0000) 2019-10-13T08:07:40.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:40.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.677867+0000) 2019-10-13T08:07:40.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:40.774+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:40.775930+0000) 2019-10-13T08:07:41.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:41.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:41.573530+0000) 2019-10-13T08:07:41.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:41.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:41.574811+0000) 2019-10-13T08:07:44.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:44.132+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:44.133606+0000) 2019-10-13T08:07:44.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:44.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:44.297976+0000) 2019-10-13T08:07:45.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:45.577+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.582423+0000) 2019-10-13T08:07:45.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:45.576+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.582475+0000) 2019-10-13T08:07:45.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:45.577+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.584835+0000) 2019-10-13T08:07:45.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:45.577+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.585022+0000) 2019-10-13T08:07:45.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:45.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:45.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:45.577+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.585508+0000) 2019-10-13T08:07:45.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:45.581+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.586544+0000) 2019-10-13T08:07:45.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:45.581+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.586972+0000) 2019-10-13T08:07:45.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:45.584+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:45.592052+0000) 2019-10-13T08:07:47.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:47.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:47.138320+0000) 2019-10-13T08:07:47.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:47.549+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:47.552715+0000) 2019-10-13T08:07:50.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:50.584+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.587929+0000) 2019-10-13T08:07:50.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:50.581+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.589625+0000) 2019-10-13T08:07:50.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:50.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:50.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:50.585+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.591368+0000) 2019-10-13T08:07:50.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:50.585+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.592166+0000) 2019-10-13T08:07:50.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:50.585+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.591738+0000) 2019-10-13T08:07:50.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:50.585+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.593198+0000) 2019-10-13T08:07:50.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:50.589+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.594078+0000) 2019-10-13T08:07:50.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:50.592+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.599795+0000) 2019-10-13T08:07:50.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:50.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.678040+0000) 2019-10-13T08:07:50.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:50.773+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:50.776105+0000) 2019-10-13T08:07:51.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:51.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:51.573721+0000) 2019-10-13T08:07:51.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:51.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:51.574978+0000) 2019-10-13T08:07:54.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:54.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:54.139983+0000) 2019-10-13T08:07:54.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:54.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:54.298178+0000) 2019-10-13T08:07:55.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:07:55.589+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.594722+0000) 2019-10-13T08:07:55.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:07:55.589+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.595001+0000) 2019-10-13T08:07:55.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:07:55.589+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.596606+0000) 2019-10-13T08:07:55.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:07:55.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:07:55.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:07:55.589+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.596850+0000) 2019-10-13T08:07:55.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:07:55.592+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.596817+0000) 2019-10-13T08:07:55.598 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:55.593+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.598124+0000) 2019-10-13T08:07:55.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:55.593+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.600440+0000) 2019-10-13T08:07:55.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:07:55.600+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:55.606585+0000) 2019-10-13T08:07:57.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:07:57.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:57.138560+0000) 2019-10-13T08:07:57.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:07:57.549+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:07:57.552914+0000) 2019-10-13T08:08:00.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:00.597+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.600157+0000) 2019-10-13T08:08:00.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:00.597+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.600783+0000) 2019-10-13T08:08:00.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:00.597+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.601531+0000) 2019-10-13T08:08:00.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:00.601+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.602356+0000) 2019-10-13T08:08:00.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:00.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:00.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:00.600+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.602255+0000) 2019-10-13T08:08:00.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:00.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.603098+0000) 2019-10-13T08:08:00.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:00.601+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.604375+0000) 2019-10-13T08:08:00.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:00.608+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.609390+0000) 2019-10-13T08:08:00.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:00.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.678230+0000) 2019-10-13T08:08:00.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:00.773+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:00.776279+0000) 2019-10-13T08:08:01.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:01.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:01.573948+0000) 2019-10-13T08:08:01.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:01.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:01.575153+0000) 2019-10-13T08:08:03.957 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:08:03.957 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:03.960 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:08:03.961 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:04.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:04.143+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:04.146559+0000) 2019-10-13T08:08:04.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:04.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:04.298390+0000) 2019-10-13T08:08:05.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:05.605+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.607904+0000) 2019-10-13T08:08:05.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:05.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:05.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:05.605+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.609203+0000) 2019-10-13T08:08:05.609 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:05.605+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.609535+0000) 2019-10-13T08:08:05.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:05.609+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.610210+0000) 2019-10-13T08:08:05.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:05.609+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.610779+0000) 2019-10-13T08:08:05.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:05.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.611036+0000) 2019-10-13T08:08:05.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:05.609+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.612002+0000) 2019-10-13T08:08:05.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:05.615+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:05.619125+0000) 2019-10-13T08:08:07.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:07.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:07.138794+0000) 2019-10-13T08:08:07.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:07.549+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:07.553116+0000) 2019-10-13T08:08:10.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:10.609+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.612051+0000) 2019-10-13T08:08:10.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:10.609+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.612359+0000) 2019-10-13T08:08:10.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:10.609+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.613439+0000) 2019-10-13T08:08:10.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:10.613+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.613927+0000) 2019-10-13T08:08:10.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:10.613+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.614221+0000) 2019-10-13T08:08:10.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:10.611+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.614021+0000) 2019-10-13T08:08:10.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:10.613+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.614842+0000) 2019-10-13T08:08:10.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:10.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:10.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:10.619+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.621400+0000) 2019-10-13T08:08:10.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:10.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.678429+0000) 2019-10-13T08:08:10.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:10.773+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:10.776460+0000) 2019-10-13T08:08:11.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:11.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:11.574123+0000) 2019-10-13T08:08:11.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:11.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:11.575328+0000) 2019-10-13T08:08:14.154 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:14.151+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:14.153231+0000) 2019-10-13T08:08:14.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:14.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:14.298596+0000) 2019-10-13T08:08:15.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:15.615+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.618412+0000) 2019-10-13T08:08:15.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:15.617+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.619576+0000) 2019-10-13T08:08:15.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:15.617+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.619974+0000) 2019-10-13T08:08:15.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:15.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:15.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:15.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.621470+0000) 2019-10-13T08:08:15.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:15.621+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.622130+0000) 2019-10-13T08:08:15.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:15.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.624657+0000) 2019-10-13T08:08:15.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:15.621+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.623080+0000) 2019-10-13T08:08:15.626 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:15.621+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:15.623187+0000) 2019-10-13T08:08:17.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:17.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:17.138993+0000) 2019-10-13T08:08:17.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:17.549+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:17.553312+0000) 2019-10-13T08:08:20.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:20.621+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.624688+0000) 2019-10-13T08:08:20.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:20.621+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.625167+0000) 2019-10-13T08:08:20.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:20.621+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.625650+0000) 2019-10-13T08:08:20.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:20.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.625680+0000) 2019-10-13T08:08:20.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:20.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:20.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:20.625+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.627227+0000) 2019-10-13T08:08:20.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:20.625+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.628385+0000) 2019-10-13T08:08:20.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:20.625+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.629154+0000) 2019-10-13T08:08:20.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:20.631+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.633970+0000) 2019-10-13T08:08:20.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:20.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.678624+0000) 2019-10-13T08:08:20.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:20.773+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:20.776626+0000) 2019-10-13T08:08:21.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:21.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:21.574302+0000) 2019-10-13T08:08:21.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:21.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:21.575502+0000) 2019-10-13T08:08:24.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:24.155+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:24.159778+0000) 2019-10-13T08:08:24.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:24.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:24.298804+0000) 2019-10-13T08:08:25.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:25.629+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.630053+0000) 2019-10-13T08:08:25.632 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:25.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:25.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:25.629+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.632446+0000) 2019-10-13T08:08:25.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:25.629+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.633505+0000) 2019-10-13T08:08:25.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:25.633+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.633783+0000) 2019-10-13T08:08:25.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:25.631+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.634569+0000) 2019-10-13T08:08:25.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:25.633+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.635736+0000) 2019-10-13T08:08:25.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:25.633+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.636062+0000) 2019-10-13T08:08:25.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:25.635+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:25.639426+0000) 2019-10-13T08:08:27.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:27.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:27.139188+0000) 2019-10-13T08:08:27.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:27.549+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:27.553544+0000) 2019-10-13T08:08:30.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:30.633+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.637175+0000) 2019-10-13T08:08:30.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:30.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:30.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:30.637+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.637850+0000) 2019-10-13T08:08:30.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:30.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.637770+0000) 2019-10-13T08:08:30.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:30.637+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.638223+0000) 2019-10-13T08:08:30.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:30.637+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.638650+0000) 2019-10-13T08:08:30.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:30.637+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.638989+0000) 2019-10-13T08:08:30.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:30.635+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.636346+0000) 2019-10-13T08:08:30.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:30.639+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.642506+0000) 2019-10-13T08:08:30.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:30.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.678816+0000) 2019-10-13T08:08:30.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:30.773+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:30.776812+0000) 2019-10-13T08:08:31.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:31.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:31.574425+0000) 2019-10-13T08:08:31.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:31.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:31.575674+0000) 2019-10-13T08:08:34.072 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:08:34.072 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:34.076 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:08:34.076 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:08:34.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:34.163+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:34.166237+0000) 2019-10-13T08:08:34.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:34.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:34.299016+0000) 2019-10-13T08:08:35.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:35.639+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.641385+0000) 2019-10-13T08:08:35.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:35.641+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.642249+0000) 2019-10-13T08:08:35.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:35.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:35.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:35.641+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.644608+0000) 2019-10-13T08:08:35.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:35.641+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.644641+0000) 2019-10-13T08:08:35.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:35.641+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.645078+0000) 2019-10-13T08:08:35.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:35.641+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.645552+0000) 2019-10-13T08:08:35.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:35.645+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.646625+0000) 2019-10-13T08:08:35.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:35.651+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:35.652149+0000) 2019-10-13T08:08:37.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:37.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:37.139380+0000) 2019-10-13T08:08:37.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:37.553+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:37.553771+0000) 2019-10-13T08:08:40.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:40.644+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.648042+0000) 2019-10-13T08:08:40.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:40.644+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.648731+0000) 2019-10-13T08:08:40.649 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:40.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:40.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:40.648+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.649850+0000) 2019-10-13T08:08:40.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:40.648+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.650198+0000) 2019-10-13T08:08:40.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:40.647+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.650354+0000) 2019-10-13T08:08:40.653 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:40.648+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.651298+0000) 2019-10-13T08:08:40.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:40.648+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.653003+0000) 2019-10-13T08:08:40.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:40.651+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.655549+0000) 2019-10-13T08:08:40.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:40.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.678989+0000) 2019-10-13T08:08:40.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:40.772+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:40.777030+0000) 2019-10-13T08:08:41.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:41.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:41.574599+0000) 2019-10-13T08:08:41.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:41.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:41.575839+0000) 2019-10-13T08:08:44.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:44.171+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:44.172613+0000) 2019-10-13T08:08:44.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:44.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:44.299209+0000) 2019-10-13T08:08:45.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:45.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.651964+0000) 2019-10-13T08:08:45.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:45.648+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.652568+0000) 2019-10-13T08:08:45.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:45.648+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.653386+0000) 2019-10-13T08:08:45.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:45.652+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.653820+0000) 2019-10-13T08:08:45.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:45.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:45.655 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:45.652+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.654851+0000) 2019-10-13T08:08:45.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:45.652+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.655083+0000) 2019-10-13T08:08:45.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:45.651+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.655567+0000) 2019-10-13T08:08:45.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:45.659+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:45.663721+0000) 2019-10-13T08:08:47.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:47.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:47.139588+0000) 2019-10-13T08:08:47.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:47.552+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:47.553992+0000) 2019-10-13T08:08:50.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:50.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.658882+0000) 2019-10-13T08:08:50.660 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:50.656+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.659231+0000) 2019-10-13T08:08:50.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:50.656+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.660066+0000) 2019-10-13T08:08:50.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:50.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:50.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:50.656+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.660443+0000) 2019-10-13T08:08:50.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:50.656+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.660816+0000) 2019-10-13T08:08:50.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:50.656+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.661321+0000) 2019-10-13T08:08:50.662 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:50.660+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.662091+0000) 2019-10-13T08:08:50.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:50.663+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.665674+0000) 2019-10-13T08:08:50.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:50.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.679166+0000) 2019-10-13T08:08:50.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:50.772+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:50.777174+0000) 2019-10-13T08:08:51.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:51.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:51.574774+0000) 2019-10-13T08:08:51.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:51.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:51.576015+0000) 2019-10-13T08:08:54.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:54.175+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:54.179097+0000) 2019-10-13T08:08:54.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:54.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:54.299422+0000) 2019-10-13T08:08:55.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:55.660+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.663549+0000) 2019-10-13T08:08:55.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:08:55.660+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.663874+0000) 2019-10-13T08:08:55.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:08:55.660+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.664685+0000) 2019-10-13T08:08:55.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:55.660+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.664812+0000) 2019-10-13T08:08:55.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:08:55.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.664768+0000) 2019-10-13T08:08:55.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:08:55.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66489 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:08:55.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:08:55.664+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.666954+0000) 2019-10-13T08:08:55.668 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:08:55.664+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.666241+0000) 2019-10-13T08:08:55.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:08:55.667+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:55.669607+0000) 2019-10-13T08:08:57.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:08:57.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:57.139837+0000) 2019-10-13T08:08:57.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:08:57.552+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:08:57.554146+0000) 2019-10-13T08:09:00.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:00.664+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.669277+0000) 2019-10-13T08:09:00.670 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:00.664+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.669691+0000) 2019-10-13T08:09:00.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:00.668+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.670653+0000) 2019-10-13T08:09:00.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:00.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:00.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:00.668+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.671162+0000) 2019-10-13T08:09:00.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:00.667+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.671288+0000) 2019-10-13T08:09:00.674 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:00.672+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.673823+0000) 2019-10-13T08:09:00.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:00.668+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.672622+0000) 2019-10-13T08:09:00.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:00.675+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.677911+0000) 2019-10-13T08:09:00.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:00.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.679285+0000) 2019-10-13T08:09:00.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:00.776+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:00.777384+0000) 2019-10-13T08:09:01.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:01.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:01.574984+0000) 2019-10-13T08:09:01.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:01.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:01.576187+0000) 2019-10-13T08:09:04.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:04.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:04.185525+0000) 2019-10-13T08:09:04.207 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:09:04.207 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:04.214 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:09:04.214 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:04.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:04.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:04.299586+0000) 2019-10-13T08:09:05.675 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:05.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.674922+0000) 2019-10-13T08:09:05.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:05.672+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.676268+0000) 2019-10-13T08:09:05.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:05.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:05.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:05.672+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.675981+0000) 2019-10-13T08:09:05.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:05.672+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.676497+0000) 2019-10-13T08:09:05.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:05.672+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.676673+0000) 2019-10-13T08:09:05.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:05.672+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.677091+0000) 2019-10-13T08:09:05.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:05.675+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.677643+0000) 2019-10-13T08:09:05.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:05.679+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:05.680335+0000) 2019-10-13T08:09:07.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:07.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:07.140073+0000) 2019-10-13T08:09:07.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:07.552+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:07.554354+0000) 2019-10-13T08:09:10.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:10.676+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.679416+0000) 2019-10-13T08:09:10.681 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:10.676+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.680788+0000) 2019-10-13T08:09:10.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:10.676+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.680871+0000) 2019-10-13T08:09:10.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:10.676+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.681332+0000) 2019-10-13T08:09:10.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:10.680+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.682638+0000) 2019-10-13T08:09:10.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:10.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:10.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:10.680+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.683155+0000) 2019-10-13T08:09:10.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:10.678+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.683111+0000) 2019-10-13T08:09:10.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:10.686+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.688281+0000) 2019-10-13T08:09:10.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:10.772+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:10.777568+0000) 2019-10-13T08:09:11.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:11.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:11.575197+0000) 2019-10-13T08:09:11.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:11.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:11.576367+0000) 2019-10-13T08:09:14.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:14.186+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:14.191909+0000) 2019-10-13T08:09:14.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:14.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:14.299765+0000) 2019-10-13T08:09:15.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:15.684+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.685917+0000) 2019-10-13T08:09:15.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:15.684+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.686952+0000) 2019-10-13T08:09:15.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:15.682+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.686756+0000) 2019-10-13T08:09:15.687 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:15.684+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.687766+0000) 2019-10-13T08:09:15.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:15.684+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.689029+0000) 2019-10-13T08:09:15.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:15.684+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.689036+0000) 2019-10-13T08:09:15.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:15.684+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.689196+0000) 2019-10-13T08:09:15.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:15.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:15.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:15.690+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:15.694425+0000) 2019-10-13T08:09:17.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:17.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:17.140270+0000) 2019-10-13T08:09:17.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:17.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:17.429964+0000) 2019-10-13T08:09:17.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:17.552+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:17.554546+0000) 2019-10-13T08:09:20.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:20.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.693380+0000) 2019-10-13T08:09:20.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:20.688+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.693564+0000) 2019-10-13T08:09:20.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:20.692+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.694144+0000) 2019-10-13T08:09:20.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:20.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:20.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:20.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.695698+0000) 2019-10-13T08:09:20.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:20.692+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.697457+0000) 2019-10-13T08:09:20.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:20.696+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.698411+0000) 2019-10-13T08:09:20.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:20.696+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.698566+0000) 2019-10-13T08:09:20.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:20.698+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.703198+0000) 2019-10-13T08:09:20.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:20.776+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:20.777772+0000) 2019-10-13T08:09:21.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:21.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:21.575407+0000) 2019-10-13T08:09:21.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:21.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:21.576568+0000) 2019-10-13T08:09:24.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:24.194+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:24.198255+0000) 2019-10-13T08:09:24.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:24.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:24.299940+0000) 2019-10-13T08:09:25.699 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:25.696+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.699015+0000) 2019-10-13T08:09:25.700 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:25.696+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.700299+0000) 2019-10-13T08:09:25.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:25.696+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.700392+0000) 2019-10-13T08:09:25.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:25.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:25.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:25.700+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.702529+0000) 2019-10-13T08:09:25.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:25.700+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.703898+0000) 2019-10-13T08:09:25.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:25.698+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.703751+0000) 2019-10-13T08:09:25.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:25.700+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.704613+0000) 2019-10-13T08:09:25.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:25.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:25.716252+0000) 2019-10-13T08:09:27.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:27.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:27.140502+0000) 2019-10-13T08:09:27.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:27.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:27.430167+0000) 2019-10-13T08:09:27.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:27.552+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:27.554741+0000) 2019-10-13T08:09:30.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:30.700+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.705717+0000) 2019-10-13T08:09:30.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:30.702+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.706071+0000) 2019-10-13T08:09:30.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:30.704+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.706230+0000) 2019-10-13T08:09:30.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:30.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:30.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:30.704+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.708435+0000) 2019-10-13T08:09:30.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:30.704+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.709490+0000) 2019-10-13T08:09:30.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:30.704+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.709364+0000) 2019-10-13T08:09:30.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:30.708+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.709841+0000) 2019-10-13T08:09:30.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:30.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.716568+0000) 2019-10-13T08:09:30.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:30.776+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:30.777947+0000) 2019-10-13T08:09:31.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:31.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:31.575611+0000) 2019-10-13T08:09:31.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:31.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:31.576810+0000) 2019-10-13T08:09:34.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:34.206+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:34.209714+0000) 2019-10-13T08:09:34.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:34.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:34.300158+0000) 2019-10-13T08:09:34.369 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:09:34.369 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:34.373 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:09:34.373 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:09:35.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:35.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.711992+0000) 2019-10-13T08:09:35.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:35.706+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.712055+0000) 2019-10-13T08:09:35.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:35.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.713160+0000) 2019-10-13T08:09:35.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:35.707+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.713449+0000) 2019-10-13T08:09:35.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:35.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:35.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:35.711+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.714101+0000) 2019-10-13T08:09:35.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:35.711+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.714861+0000) 2019-10-13T08:09:35.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:35.718+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.720804+0000) 2019-10-13T08:09:35.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:35.711+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:35.714486+0000) 2019-10-13T08:09:37.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:37.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:37.140710+0000) 2019-10-13T08:09:37.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:37.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:37.430401+0000) 2019-10-13T08:09:37.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:37.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:37.554992+0000) 2019-10-13T08:09:40.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:40.711+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.717368+0000) 2019-10-13T08:09:40.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:40.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.717257+0000) 2019-10-13T08:09:40.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:40.715+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.718611+0000) 2019-10-13T08:09:40.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:40.715+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.718673+0000) 2019-10-13T08:09:40.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:40.715+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.719685+0000) 2019-10-13T08:09:40.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:40.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66764 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:40.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:40.715+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.719863+0000) 2019-10-13T08:09:40.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:40.715+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.720158+0000) 2019-10-13T08:09:40.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:40.722+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.724225+0000) 2019-10-13T08:09:40.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:40.775+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:40.778163+0000) 2019-10-13T08:09:41.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:41.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:41.575784+0000) 2019-10-13T08:09:41.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:41.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:41.577004+0000) 2019-10-13T08:09:44.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:44.214+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:44.216254+0000) 2019-10-13T08:09:44.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:44.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:44.300423+0000) 2019-10-13T08:09:45.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:45.719+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.722589+0000) 2019-10-13T08:09:45.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:45.719+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.724681+0000) 2019-10-13T08:09:45.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:45.722+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.725088+0000) 2019-10-13T08:09:45.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:45.723+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.725863+0000) 2019-10-13T08:09:45.726 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:45.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:45.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:45.723+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.726177+0000) 2019-10-13T08:09:45.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:45.723+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.726754+0000) 2019-10-13T08:09:45.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:45.723+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.726817+0000) 2019-10-13T08:09:45.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:45.726+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:45.730088+0000) 2019-10-13T08:09:47.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:47.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:47.140914+0000) 2019-10-13T08:09:47.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:47.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:47.430603+0000) 2019-10-13T08:09:47.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:47.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:47.555190+0000) 2019-10-13T08:09:50.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:50.727+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.730831+0000) 2019-10-13T08:09:50.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:50.727+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.731461+0000) 2019-10-13T08:09:50.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:50.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:50.732 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:50.727+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.732762+0000) 2019-10-13T08:09:50.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:50.730+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.732810+0000) 2019-10-13T08:09:50.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:50.731+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.734230+0000) 2019-10-13T08:09:50.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:50.731+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.734539+0000) 2019-10-13T08:09:50.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:50.731+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.735745+0000) 2019-10-13T08:09:50.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:50.734+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.738002+0000) 2019-10-13T08:09:50.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:50.775+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:50.778354+0000) 2019-10-13T08:09:51.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:51.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:51.575962+0000) 2019-10-13T08:09:51.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:51.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:51.577180+0000) 2019-10-13T08:09:54.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:54.222+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:54.222616+0000) 2019-10-13T08:09:54.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:54.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:54.300673+0000) 2019-10-13T08:09:55.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:55.731+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.735696+0000) 2019-10-13T08:09:55.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:09:55.731+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.736584+0000) 2019-10-13T08:09:55.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:55.731+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.737163+0000) 2019-10-13T08:09:55.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:09:55.734+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.736934+0000) 2019-10-13T08:09:55.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:09:55.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:09:55.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:09:55.735+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.737945+0000) 2019-10-13T08:09:55.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:55.735+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.738327+0000) 2019-10-13T08:09:55.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:09:55.735+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.738847+0000) 2019-10-13T08:09:55.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:09:55.742+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:55.744425+0000) 2019-10-13T08:09:57.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:09:57.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:57.141108+0000) 2019-10-13T08:09:57.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:09:57.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:57.430810+0000) 2019-10-13T08:09:57.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:09:57.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:09:57.555370+0000) 2019-10-13T08:10:00.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:00.739+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.741837+0000) 2019-10-13T08:10:00.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:00.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66881 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:00.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:00.739+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.743641+0000) 2019-10-13T08:10:00.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:00.739+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.744345+0000) 2019-10-13T08:10:00.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:00.739+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.744617+0000) 2019-10-13T08:10:00.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:00.742+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.744845+0000) 2019-10-13T08:10:00.745 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:00.739+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.745619+0000) 2019-10-13T08:10:00.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:00.743+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.745974+0000) 2019-10-13T08:10:00.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:00.746+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.749600+0000) 2019-10-13T08:10:00.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:00.775+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:00.778522+0000) 2019-10-13T08:10:01.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:01.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:01.576165+0000) 2019-10-13T08:10:01.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:01.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:01.577374+0000) 2019-10-13T08:10:04.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:04.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:04.229163+0000) 2019-10-13T08:10:04.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:04.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:04.300883+0000) 2019-10-13T08:10:04.490 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:10:04.490 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:04.494 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:10:04.494 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:05.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:05.743+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.746703+0000) 2019-10-13T08:10:05.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:05.743+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.746876+0000) 2019-10-13T08:10:05.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:05.743+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.748419+0000) 2019-10-13T08:10:05.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:05.743+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.748906+0000) 2019-10-13T08:10:05.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:05.743+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.749327+0000) 2019-10-13T08:10:05.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:05.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:05.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:05.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.749644+0000) 2019-10-13T08:10:05.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:05.747+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.749802+0000) 2019-10-13T08:10:05.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:05.758+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:05.760707+0000) 2019-10-13T08:10:07.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:07.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:07.141315+0000) 2019-10-13T08:10:07.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:07.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:07.430960+0000) 2019-10-13T08:10:07.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:07.555+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:07.555582+0000) 2019-10-13T08:10:10.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:10.747+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.753205+0000) 2019-10-13T08:10:10.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:10.747+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.753641+0000) 2019-10-13T08:10:10.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:10.751+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.753957+0000) 2019-10-13T08:10:10.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:10.751+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.754349+0000) 2019-10-13T08:10:10.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:10.750+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.754184+0000) 2019-10-13T08:10:10.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:10.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:10.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:10.751+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.756749+0000) 2019-10-13T08:10:10.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:10.751+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.757422+0000) 2019-10-13T08:10:10.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:10.758+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.762394+0000) 2019-10-13T08:10:10.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:10.775+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:10.778719+0000) 2019-10-13T08:10:11.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:11.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:11.576342+0000) 2019-10-13T08:10:11.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:11.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:11.577566+0000) 2019-10-13T08:10:14.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:14.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:14.235634+0000) 2019-10-13T08:10:14.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:14.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:14.301083+0000) 2019-10-13T08:10:15.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:15.755+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.760250+0000) 2019-10-13T08:10:15.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:15.755+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.761287+0000) 2019-10-13T08:10:15.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:15.759+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.761837+0000) 2019-10-13T08:10:15.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:15.759+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66968 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:15.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:15.759+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.762175+0000) 2019-10-13T08:10:15.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:15.757+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.762095+0000) 2019-10-13T08:10:15.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:15.759+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.762113+0000) 2019-10-13T08:10:15.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:15.759+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.763454+0000) 2019-10-13T08:10:15.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:15.765+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:15.769492+0000) 2019-10-13T08:10:17.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:17.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:17.141548+0000) 2019-10-13T08:10:17.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:17.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:17.431168+0000) 2019-10-13T08:10:17.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:17.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:17.555780+0000) 2019-10-13T08:10:20.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:20.759+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.764724+0000) 2019-10-13T08:10:20.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:20.759+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.765287+0000) 2019-10-13T08:10:20.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:20.763+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.765779+0000) 2019-10-13T08:10:20.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:20.763+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.766794+0000) 2019-10-13T08:10:20.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:20.763+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.767234+0000) 2019-10-13T08:10:20.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:20.761+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.767167+0000) 2019-10-13T08:10:20.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:20.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:20.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:20.763+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.768396+0000) 2019-10-13T08:10:20.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:20.769+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.773097+0000) 2019-10-13T08:10:20.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:20.775+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:20.778866+0000) 2019-10-13T08:10:21.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:21.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:21.576526+0000) 2019-10-13T08:10:21.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:21.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:21.577762+0000) 2019-10-13T08:10:24.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:24.241+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:24.242123+0000) 2019-10-13T08:10:24.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:24.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:24.301312+0000) 2019-10-13T08:10:25.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:25.767+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.770905+0000) 2019-10-13T08:10:25.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:25.767+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.771445+0000) 2019-10-13T08:10:25.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:25.767+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.772400+0000) 2019-10-13T08:10:25.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:25.767+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.772868+0000) 2019-10-13T08:10:25.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:25.771+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.773884+0000) 2019-10-13T08:10:25.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:25.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:25.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:25.769+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.774099+0000) 2019-10-13T08:10:25.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:25.771+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.776393+0000) 2019-10-13T08:10:25.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:25.777+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:25.781214+0000) 2019-10-13T08:10:27.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:27.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:27.141797+0000) 2019-10-13T08:10:27.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:27.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:27.431369+0000) 2019-10-13T08:10:27.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:27.551+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:27.555979+0000) 2019-10-13T08:10:30.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:30.771+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.777023+0000) 2019-10-13T08:10:30.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:30.775+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.778729+0000) 2019-10-13T08:10:30.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:30.775+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.779436+0000) 2019-10-13T08:10:30.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:30.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:30.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:30.775+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.780619+0000) 2019-10-13T08:10:30.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:30.777+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.780400+0000) 2019-10-13T08:10:30.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:30.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.781995+0000) 2019-10-13T08:10:30.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:30.779+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.783744+0000) 2019-10-13T08:10:30.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:30.781+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:30.785072+0000) 2019-10-13T08:10:31.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:31.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:31.576709+0000) 2019-10-13T08:10:31.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:31.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:31.577948+0000) 2019-10-13T08:10:34.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:34.245+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:34.248655+0000) 2019-10-13T08:10:34.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:34.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:34.301473+0000) 2019-10-13T08:10:34.623 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:10:34.623 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:34.627 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:10:34.627 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:10:35.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:35.778+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.783785+0000) 2019-10-13T08:10:35.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:35.778+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.783874+0000) 2019-10-13T08:10:35.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:35.778+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.784327+0000) 2019-10-13T08:10:35.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:35.778+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.784653+0000) 2019-10-13T08:10:35.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:35.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.784969+0000) 2019-10-13T08:10:35.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:35.782+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:35.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:35.782+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.786459+0000) 2019-10-13T08:10:35.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:35.782+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.786831+0000) 2019-10-13T08:10:35.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:35.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:35.792285+0000) 2019-10-13T08:10:37.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:37.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:37.142003+0000) 2019-10-13T08:10:37.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:37.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:37.431561+0000) 2019-10-13T08:10:37.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:37.526+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:37.529561+0000) 2019-10-13T08:10:37.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:37.554+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:37.556179+0000) 2019-10-13T08:10:40.793 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:40.786+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.790189+0000) 2019-10-13T08:10:40.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:40.786+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.790560+0000) 2019-10-13T08:10:40.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:40.786+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.792537+0000) 2019-10-13T08:10:40.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:40.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:40.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:40.786+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.793508+0000) 2019-10-13T08:10:40.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:40.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.793896+0000) 2019-10-13T08:10:40.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:40.789+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.794558+0000) 2019-10-13T08:10:40.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:40.790+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.794758+0000) 2019-10-13T08:10:40.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:40.793+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:40.797967+0000) 2019-10-13T08:10:41.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:41.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:41.576917+0000) 2019-10-13T08:10:41.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:41.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:41.578080+0000) 2019-10-13T08:10:44.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:44.253+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:44.255172+0000) 2019-10-13T08:10:44.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:44.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:44.301684+0000) 2019-10-13T08:10:45.796 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:45.790+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.796301+0000) 2019-10-13T08:10:45.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:45.790+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.797477+0000) 2019-10-13T08:10:45.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:45.794+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.798061+0000) 2019-10-13T08:10:45.798 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:45.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:45.798 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:45.794+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.798247+0000) 2019-10-13T08:10:45.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:45.793+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.798221+0000) 2019-10-13T08:10:45.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:45.794+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.799005+0000) 2019-10-13T08:10:45.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:45.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.801647+0000) 2019-10-13T08:10:45.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:45.801+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:45.804550+0000) 2019-10-13T08:10:47.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:47.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:47.142204+0000) 2019-10-13T08:10:47.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:47.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:47.431762+0000) 2019-10-13T08:10:47.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:47.526+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:47.529801+0000) 2019-10-13T08:10:47.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:47.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:47.556373+0000) 2019-10-13T08:10:50.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:50.794+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.801120+0000) 2019-10-13T08:10:50.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:50.798+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.802517+0000) 2019-10-13T08:10:50.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:50.797+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.802428+0000) 2019-10-13T08:10:50.803 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:50.798+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.802926+0000) 2019-10-13T08:10:50.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:50.798+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.803701+0000) 2019-10-13T08:10:50.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:50.798+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:50.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:50.798+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.804169+0000) 2019-10-13T08:10:50.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:50.798+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.805411+0000) 2019-10-13T08:10:50.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:50.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:50.812682+0000) 2019-10-13T08:10:51.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:51.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:51.577108+0000) 2019-10-13T08:10:51.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:51.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:51.578253+0000) 2019-10-13T08:10:54.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:54.257+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:54.261786+0000) 2019-10-13T08:10:54.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:54.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:54.301918+0000) 2019-10-13T08:10:55.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:55.802+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.807694+0000) 2019-10-13T08:10:55.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:10:55.802+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.808130+0000) 2019-10-13T08:10:55.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:10:55.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:10:55.810 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:55.806+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.810674+0000) 2019-10-13T08:10:55.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:55.806+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.811241+0000) 2019-10-13T08:10:55.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:10:55.806+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.811545+0000) 2019-10-13T08:10:55.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:10:55.805+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.811427+0000) 2019-10-13T08:10:55.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:55.810+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.814153+0000) 2019-10-13T08:10:55.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:10:55.813+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:55.816934+0000) 2019-10-13T08:10:57.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:10:57.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:57.142418+0000) 2019-10-13T08:10:57.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:10:57.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:57.432025+0000) 2019-10-13T08:10:57.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:10:57.526+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:57.529995+0000) 2019-10-13T08:10:57.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:10:57.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:10:57.556591+0000) 2019-10-13T08:11:00.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:00.810+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.813885+0000) 2019-10-13T08:11:00.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:00.809+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.813895+0000) 2019-10-13T08:11:00.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:00.810+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.814778+0000) 2019-10-13T08:11:00.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:00.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:00.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:00.810+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.817126+0000) 2019-10-13T08:11:00.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:00.810+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.817303+0000) 2019-10-13T08:11:00.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:00.810+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.816649+0000) 2019-10-13T08:11:00.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:00.814+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.818334+0000) 2019-10-13T08:11:00.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:00.817+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:00.822282+0000) 2019-10-13T08:11:01.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:01.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:01.577288+0000) 2019-10-13T08:11:01.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:01.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:01.578447+0000) 2019-10-13T08:11:04.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:04.265+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:04.268215+0000) 2019-10-13T08:11:04.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:04.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:04.302117+0000) 2019-10-13T08:11:04.758 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:11:04.759 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:04.763 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:11:04.763 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:05.820 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:05.814+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.820296+0000) 2019-10-13T08:11:05.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:05.814+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.820709+0000) 2019-10-13T08:11:05.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:05.814+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.821105+0000) 2019-10-13T08:11:05.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:05.814+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.821328+0000) 2019-10-13T08:11:05.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:05.818+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.822360+0000) 2019-10-13T08:11:05.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:05.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:05.823 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:05.818+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.822834+0000) 2019-10-13T08:11:05.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:05.817+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.822407+0000) 2019-10-13T08:11:05.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:05.821+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:05.824783+0000) 2019-10-13T08:11:07.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:07.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:07.142619+0000) 2019-10-13T08:11:07.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:07.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:07.432223+0000) 2019-10-13T08:11:07.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:07.526+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:07.530192+0000) 2019-10-13T08:11:07.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:07.554+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:07.556790+0000) 2019-10-13T08:11:10.826 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:10.822+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.825492+0000) 2019-10-13T08:11:10.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:10.822+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.827012+0000) 2019-10-13T08:11:10.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:10.822+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.827502+0000) 2019-10-13T08:11:10.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:10.822+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.827850+0000) 2019-10-13T08:11:10.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:10.821+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.827611+0000) 2019-10-13T08:11:10.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:10.822+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.828410+0000) 2019-10-13T08:11:10.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:10.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:10.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:10.822+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.828924+0000) 2019-10-13T08:11:10.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:10.829+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:10.832721+0000) 2019-10-13T08:11:11.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:11.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:11.577463+0000) 2019-10-13T08:11:11.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:11.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:11.578619+0000) 2019-10-13T08:11:14.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:14.273+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:14.274700+0000) 2019-10-13T08:11:14.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:14.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:14.302348+0000) 2019-10-13T08:11:15.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:15.826+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.831690+0000) 2019-10-13T08:11:15.832 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:15.826+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.832439+0000) 2019-10-13T08:11:15.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:15.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:15.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:15.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.834185+0000) 2019-10-13T08:11:15.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:15.830+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.834980+0000) 2019-10-13T08:11:15.835 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:15.830+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.835633+0000) 2019-10-13T08:11:15.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:15.830+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.835667+0000) 2019-10-13T08:11:15.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:15.830+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.836240+0000) 2019-10-13T08:11:15.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:15.837+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:15.841718+0000) 2019-10-13T08:11:17.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:17.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:17.142815+0000) 2019-10-13T08:11:17.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:17.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:17.432425+0000) 2019-10-13T08:11:17.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:17.526+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:17.530370+0000) 2019-10-13T08:11:17.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:17.554+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:17.556988+0000) 2019-10-13T08:11:20.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:20.834+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.837090+0000) 2019-10-13T08:11:20.839 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:20.834+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.839168+0000) 2019-10-13T08:11:20.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:20.834+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.839514+0000) 2019-10-13T08:11:20.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:20.833+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.839393+0000) 2019-10-13T08:11:20.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:20.834+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:20.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:20.834+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.840491+0000) 2019-10-13T08:11:20.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:20.834+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.840769+0000) 2019-10-13T08:11:20.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:20.834+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.841071+0000) 2019-10-13T08:11:20.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:20.841+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:20.845709+0000) 2019-10-13T08:11:21.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:21.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:21.577648+0000) 2019-10-13T08:11:21.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:21.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:21.578822+0000) 2019-10-13T08:11:24.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:24.276+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:24.281226+0000) 2019-10-13T08:11:24.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:24.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:24.302548+0000) 2019-10-13T08:11:25.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:25.838+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.843903+0000) 2019-10-13T08:11:25.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:25.842+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:25.846 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:25.842+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.846268+0000) 2019-10-13T08:11:25.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:25.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.846185+0000) 2019-10-13T08:11:25.847 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:25.842+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.846874+0000) 2019-10-13T08:11:25.847 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:25.842+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.847279+0000) 2019-10-13T08:11:25.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:25.842+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.847508+0000) 2019-10-13T08:11:25.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:25.842+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.848101+0000) 2019-10-13T08:11:25.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:25.844+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:25.851420+0000) 2019-10-13T08:11:27.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:27.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:27.143053+0000) 2019-10-13T08:11:27.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:27.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:27.432629+0000) 2019-10-13T08:11:27.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:27.530+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:27.530572+0000) 2019-10-13T08:11:27.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:27.550+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:27.557180+0000) 2019-10-13T08:11:30.849 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:30.841+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.849441+0000) 2019-10-13T08:11:30.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:30.845+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.850111+0000) 2019-10-13T08:11:30.850 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:30.845+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.850376+0000) 2019-10-13T08:11:30.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:30.844+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.850367+0000) 2019-10-13T08:11:30.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:30.845+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.851415+0000) 2019-10-13T08:11:30.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:30.845+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:30.852 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:30.845+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.852299+0000) 2019-10-13T08:11:30.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:30.845+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.853349+0000) 2019-10-13T08:11:30.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:30.852+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:30.858693+0000) 2019-10-13T08:11:31.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:31.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:31.577829+0000) 2019-10-13T08:11:31.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:31.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:31.579015+0000) 2019-10-13T08:11:34.288 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:34.280+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:34.287565+0000) 2019-10-13T08:11:34.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:34.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:34.302750+0000) 2019-10-13T08:11:35.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:11:35.062 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:35.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:11:35.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:11:35.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:35.849+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.856581+0000) 2019-10-13T08:11:35.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:35.849+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.856368+0000) 2019-10-13T08:11:35.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:35.849+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.855790+0000) 2019-10-13T08:11:35.858 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:35.849+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.857081+0000) 2019-10-13T08:11:35.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:35.852+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.856832+0000) 2019-10-13T08:11:35.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:35.853+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.857831+0000) 2019-10-13T08:11:35.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:35.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:35.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:35.853+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.858415+0000) 2019-10-13T08:11:35.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:35.856+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:35.860868+0000) 2019-10-13T08:11:37.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:37.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:37.143300+0000) 2019-10-13T08:11:37.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:37.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:37.432857+0000) 2019-10-13T08:11:37.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:37.529+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:37.530781+0000) 2019-10-13T08:11:37.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:37.553+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:37.557379+0000) 2019-10-13T08:11:40.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:40.857+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.863527+0000) 2019-10-13T08:11:40.865 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:40.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:40.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:40.857+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.864388+0000) 2019-10-13T08:11:40.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:40.860+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.865289+0000) 2019-10-13T08:11:40.867 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:40.857+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.864970+0000) 2019-10-13T08:11:40.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:40.857+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.865212+0000) 2019-10-13T08:11:40.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:40.861+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.866090+0000) 2019-10-13T08:11:40.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:40.861+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.868761+0000) 2019-10-13T08:11:40.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:40.864+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:40.869374+0000) 2019-10-13T08:11:41.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:41.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:41.578025+0000) 2019-10-13T08:11:41.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:41.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:41.579227+0000) 2019-10-13T08:11:44.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:44.288+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:44.293989+0000) 2019-10-13T08:11:44.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:44.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:44.302955+0000) 2019-10-13T08:11:45.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:45.865+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.867717+0000) 2019-10-13T08:11:45.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:45.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.868702+0000) 2019-10-13T08:11:45.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:45.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.868486+0000) 2019-10-13T08:11:45.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:45.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.869702+0000) 2019-10-13T08:11:45.870 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:45.869+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.870093+0000) 2019-10-13T08:11:45.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:45.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:45.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:45.869+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.870431+0000) 2019-10-13T08:11:45.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:45.869+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.870732+0000) 2019-10-13T08:11:45.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:45.868+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:45.874969+0000) 2019-10-13T08:11:47.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:47.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:47.143542+0000) 2019-10-13T08:11:47.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:47.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:47.433064+0000) 2019-10-13T08:11:47.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:47.529+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:47.530994+0000) 2019-10-13T08:11:47.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:47.553+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:47.557578+0000) 2019-10-13T08:11:50.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:50.869+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.873604+0000) 2019-10-13T08:11:50.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:50.873+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.874414+0000) 2019-10-13T08:11:50.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:50.873+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.875373+0000) 2019-10-13T08:11:50.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:50.868+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.875159+0000) 2019-10-13T08:11:50.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:50.873+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.876178+0000) 2019-10-13T08:11:50.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:50.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:50.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:50.873+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.876526+0000) 2019-10-13T08:11:50.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:50.873+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.877290+0000) 2019-10-13T08:11:50.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:50.876+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:50.881718+0000) 2019-10-13T08:11:51.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:51.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:51.578211+0000) 2019-10-13T08:11:51.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:51.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:51.579398+0000) 2019-10-13T08:11:54.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:54.296+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:54.300403+0000) 2019-10-13T08:11:54.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:54.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:54.303182+0000) 2019-10-13T08:11:55.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:11:55.877+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.880033+0000) 2019-10-13T08:11:55.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:55.877+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.880471+0000) 2019-10-13T08:11:55.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:11:55.877+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.880823+0000) 2019-10-13T08:11:55.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:11:55.876+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.881895+0000) 2019-10-13T08:11:55.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:11:55.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:11:55.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:55.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.883154+0000) 2019-10-13T08:11:55.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:55.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.883987+0000) 2019-10-13T08:11:55.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:55.881+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.885164+0000) 2019-10-13T08:11:55.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:11:55.884+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:55.891988+0000) 2019-10-13T08:11:57.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:11:57.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:57.143749+0000) 2019-10-13T08:11:57.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:11:57.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:57.433262+0000) 2019-10-13T08:11:57.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:11:57.529+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:57.531217+0000) 2019-10-13T08:11:57.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:11:57.557+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:11:57.557815+0000) 2019-10-13T08:12:00.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:00.885+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.885833+0000) 2019-10-13T08:12:00.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:00.885+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.886138+0000) 2019-10-13T08:12:00.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:00.885+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.886534+0000) 2019-10-13T08:12:00.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:00.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:00.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:00.885+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.887267+0000) 2019-10-13T08:12:00.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:00.885+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.888154+0000) 2019-10-13T08:12:00.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:00.885+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.889039+0000) 2019-10-13T08:12:00.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:00.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.891510+0000) 2019-10-13T08:12:00.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:00.888+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:00.892521+0000) 2019-10-13T08:12:01.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:01.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:01.578400+0000) 2019-10-13T08:12:01.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:01.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:01.579567+0000) 2019-10-13T08:12:04.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:04.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:04.303379+0000) 2019-10-13T08:12:04.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:04.300+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:04.307022+0000) 2019-10-13T08:12:05.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:12:05.118 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:05.122 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:12:05.123 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:05.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:05.889+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.890819+0000) 2019-10-13T08:12:05.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:05.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.890832+0000) 2019-10-13T08:12:05.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:05.889+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.891763+0000) 2019-10-13T08:12:05.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:05.889+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.892228+0000) 2019-10-13T08:12:05.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:05.889+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.892948+0000) 2019-10-13T08:12:05.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:05.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:05.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:05.889+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.893665+0000) 2019-10-13T08:12:05.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:05.893+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.894284+0000) 2019-10-13T08:12:05.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:05.896+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:05.900547+0000) 2019-10-13T08:12:07.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:07.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:07.143944+0000) 2019-10-13T08:12:07.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:07.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:07.433479+0000) 2019-10-13T08:12:07.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:07.529+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:07.531429+0000) 2019-10-13T08:12:07.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:07.557+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:07.558012+0000) 2019-10-13T08:12:10.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:10.893+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.895652+0000) 2019-10-13T08:12:10.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:10.893+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.897399+0000) 2019-10-13T08:12:10.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:10.892+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.897714+0000) 2019-10-13T08:12:10.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:10.897+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:10.899 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:10.897+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.898734+0000) 2019-10-13T08:12:10.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:10.897+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.898876+0000) 2019-10-13T08:12:10.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:10.897+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.899302+0000) 2019-10-13T08:12:10.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:10.897+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.900079+0000) 2019-10-13T08:12:10.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:10.900+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:10.905080+0000) 2019-10-13T08:12:11.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:11.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:11.578615+0000) 2019-10-13T08:12:11.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:11.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:11.579739+0000) 2019-10-13T08:12:14.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:14.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:14.303608+0000) 2019-10-13T08:12:14.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:14.312+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:14.313511+0000) 2019-10-13T08:12:15.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:15.901+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.902732+0000) 2019-10-13T08:12:15.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:15.896+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.902808+0000) 2019-10-13T08:12:15.905 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:15.901+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.903913+0000) 2019-10-13T08:12:15.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:15.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:15.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:15.901+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.904509+0000) 2019-10-13T08:12:15.906 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:15.901+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.904251+0000) 2019-10-13T08:12:15.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:15.905+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.906522+0000) 2019-10-13T08:12:15.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:15.905+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.906342+0000) 2019-10-13T08:12:15.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:15.904+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:15.908632+0000) 2019-10-13T08:12:17.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:17.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:17.144164+0000) 2019-10-13T08:12:17.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:17.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:17.433673+0000) 2019-10-13T08:12:17.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:17.529+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:17.531646+0000) 2019-10-13T08:12:17.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:17.557+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:17.558210+0000) 2019-10-13T08:12:20.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:20.905+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.908286+0000) 2019-10-13T08:12:20.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:20.905+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.908947+0000) 2019-10-13T08:12:20.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:20.905+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.909614+0000) 2019-10-13T08:12:20.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:20.909+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:20.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:20.904+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.910820+0000) 2019-10-13T08:12:20.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:20.909+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.911561+0000) 2019-10-13T08:12:20.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:20.909+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.912059+0000) 2019-10-13T08:12:20.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:20.909+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.913087+0000) 2019-10-13T08:12:20.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:20.908+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:20.915937+0000) 2019-10-13T08:12:21.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:21.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:21.578827+0000) 2019-10-13T08:12:21.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:21.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:21.579946+0000) 2019-10-13T08:12:24.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:24.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:24.303810+0000) 2019-10-13T08:12:24.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:24.320+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:24.320133+0000) 2019-10-13T08:12:25.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:25.912+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.914129+0000) 2019-10-13T08:12:25.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:25.912+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.914562+0000) 2019-10-13T08:12:25.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:25.912+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.915900+0000) 2019-10-13T08:12:25.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:25.912+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.915004+0000) 2019-10-13T08:12:25.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:25.912+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.915612+0000) 2019-10-13T08:12:25.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:25.912+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.916286+0000) 2019-10-13T08:12:25.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:25.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:25.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:25.916+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.917904+0000) 2019-10-13T08:12:25.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:25.920+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:25.922749+0000) 2019-10-13T08:12:27.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:27.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:27.144364+0000) 2019-10-13T08:12:27.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:27.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:27.433872+0000) 2019-10-13T08:12:27.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:27.528+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:27.531847+0000) 2019-10-13T08:12:27.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:27.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:27.558411+0000) 2019-10-13T08:12:30.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:30.916+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.920492+0000) 2019-10-13T08:12:30.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:30.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.921031+0000) 2019-10-13T08:12:30.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:30.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.922061+0000) 2019-10-13T08:12:30.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:30.920+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.922877+0000) 2019-10-13T08:12:30.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:30.919+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.922723+0000) 2019-10-13T08:12:30.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:30.920+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:30.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:30.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.923624+0000) 2019-10-13T08:12:30.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:30.920+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.924088+0000) 2019-10-13T08:12:30.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:30.927+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:30.928080+0000) 2019-10-13T08:12:31.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:31.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:31.579027+0000) 2019-10-13T08:12:31.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:31.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:31.580151+0000) 2019-10-13T08:12:34.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:34.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:34.304054+0000) 2019-10-13T08:12:34.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:34.323+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:34.326520+0000) 2019-10-13T08:12:35.365 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:12:35.366 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:35.369 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:12:35.369 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:12:35.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:35.924+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.926507+0000) 2019-10-13T08:12:35.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:35.923+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.926945+0000) 2019-10-13T08:12:35.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:35.924+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.928099+0000) 2019-10-13T08:12:35.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:35.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:35.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:35.924+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.929262+0000) 2019-10-13T08:12:35.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:35.928+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.931250+0000) 2019-10-13T08:12:35.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:35.928+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.930432+0000) 2019-10-13T08:12:35.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:35.928+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.930168+0000) 2019-10-13T08:12:35.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:35.935+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:35.936207+0000) 2019-10-13T08:12:37.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:37.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:37.144573+0000) 2019-10-13T08:12:37.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:37.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:37.434061+0000) 2019-10-13T08:12:37.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:37.528+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:37.532046+0000) 2019-10-13T08:12:37.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:37.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:37.558610+0000) 2019-10-13T08:12:40.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:40.928+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.932915+0000) 2019-10-13T08:12:40.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:40.931+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.932995+0000) 2019-10-13T08:12:40.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:40.928+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.933240+0000) 2019-10-13T08:12:40.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:40.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:40.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:40.932+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.935224+0000) 2019-10-13T08:12:40.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:40.932+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.935915+0000) 2019-10-13T08:12:40.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:40.932+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.936441+0000) 2019-10-13T08:12:40.937 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:40.932+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.935487+0000) 2019-10-13T08:12:40.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:40.935+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:40.939301+0000) 2019-10-13T08:12:41.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:41.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:41.579206+0000) 2019-10-13T08:12:41.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:41.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:41.580356+0000) 2019-10-13T08:12:44.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:44.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:44.304251+0000) 2019-10-13T08:12:44.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:44.331+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:44.333583+0000) 2019-10-13T08:12:45.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:45.936+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.939302+0000) 2019-10-13T08:12:45.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:45.936+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.939685+0000) 2019-10-13T08:12:45.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:45.936+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:45.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:45.936+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.940898+0000) 2019-10-13T08:12:45.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:45.936+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.940800+0000) 2019-10-13T08:12:45.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:45.936+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.941305+0000) 2019-10-13T08:12:45.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:45.939+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.941919+0000) 2019-10-13T08:12:45.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:45.940+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.943001+0000) 2019-10-13T08:12:45.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:45.947+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:45.948315+0000) 2019-10-13T08:12:47.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:47.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:47.144770+0000) 2019-10-13T08:12:47.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:47.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:47.434263+0000) 2019-10-13T08:12:47.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:47.528+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:47.532242+0000) 2019-10-13T08:12:47.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:47.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:47.558810+0000) 2019-10-13T08:12:50.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:50.940+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.943590+0000) 2019-10-13T08:12:50.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:50.940+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.944247+0000) 2019-10-13T08:12:50.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:50.940+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.944732+0000) 2019-10-13T08:12:50.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:50.940+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.945237+0000) 2019-10-13T08:12:50.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:50.944+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.945802+0000) 2019-10-13T08:12:50.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:50.943+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.945475+0000) 2019-10-13T08:12:50.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:50.944+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:50.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:50.944+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.947439+0000) 2019-10-13T08:12:50.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:50.951+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:50.953142+0000) 2019-10-13T08:12:51.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:51.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:51.579394+0000) 2019-10-13T08:12:51.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:51.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:51.580578+0000) 2019-10-13T08:12:54.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:54.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:54.304456+0000) 2019-10-13T08:12:54.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:54.339+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:54.340073+0000) 2019-10-13T08:12:55.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:55.948+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.950450+0000) 2019-10-13T08:12:55.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:55.948+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.952078+0000) 2019-10-13T08:12:55.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:12:55.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:12:55.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:55.952+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.954202+0000) 2019-10-13T08:12:55.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:55.952+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.954428+0000) 2019-10-13T08:12:55.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:12:55.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.954137+0000) 2019-10-13T08:12:55.965 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:12:55.952+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.954920+0000) 2019-10-13T08:12:55.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:12:55.952+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.955856+0000) 2019-10-13T08:12:55.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:12:55.959+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:55.961391+0000) 2019-10-13T08:12:57.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:12:57.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:57.144997+0000) 2019-10-13T08:12:57.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:12:57.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:57.434479+0000) 2019-10-13T08:12:57.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:12:57.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:57.532476+0000) 2019-10-13T08:12:57.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:12:57.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:12:57.559041+0000) 2019-10-13T08:13:00.957 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:00.952+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.956864+0000) 2019-10-13T08:13:00.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:00.956+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.959173+0000) 2019-10-13T08:13:00.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:00.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:00.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:00.956+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.959739+0000) 2019-10-13T08:13:00.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:00.956+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.960356+0000) 2019-10-13T08:13:00.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:00.959+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.960376+0000) 2019-10-13T08:13:00.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:00.956+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.961175+0000) 2019-10-13T08:13:00.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:00.960+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.962346+0000) 2019-10-13T08:13:00.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:00.963+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:00.966353+0000) 2019-10-13T08:13:01.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:01.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:01.579596+0000) 2019-10-13T08:13:01.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:01.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:01.580782+0000) 2019-10-13T08:13:04.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:04.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:04.304617+0000) 2019-10-13T08:13:04.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:04.343+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:04.346596+0000) 2019-10-13T08:13:05.439 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:13:05.439 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:05.450 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:13:05.450 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:05.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:05.960+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.962763+0000) 2019-10-13T08:13:05.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:05.959+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.963143+0000) 2019-10-13T08:13:05.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:05.960+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.963999+0000) 2019-10-13T08:13:05.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:05.960+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.964364+0000) 2019-10-13T08:13:05.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:05.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:05.966 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:05.964+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.966107+0000) 2019-10-13T08:13:05.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:05.964+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.966783+0000) 2019-10-13T08:13:05.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:05.964+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.967798+0000) 2019-10-13T08:13:05.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:05.967+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:05.969035+0000) 2019-10-13T08:13:07.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:07.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:07.145260+0000) 2019-10-13T08:13:07.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:07.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:07.434675+0000) 2019-10-13T08:13:07.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:07.528+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:07.532704+0000) 2019-10-13T08:13:07.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:07.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:07.559238+0000) 2019-10-13T08:13:10.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:10.964+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.968829+0000) 2019-10-13T08:13:10.971 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:10.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:10.971 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:10.968+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.971392+0000) 2019-10-13T08:13:10.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:10.968+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.971686+0000) 2019-10-13T08:13:10.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:10.968+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.972639+0000) 2019-10-13T08:13:10.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:10.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.972672+0000) 2019-10-13T08:13:10.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:10.972+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.974001+0000) 2019-10-13T08:13:10.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:10.972+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.974812+0000) 2019-10-13T08:13:10.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:10.975+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:10.979260+0000) 2019-10-13T08:13:11.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:11.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:11.579781+0000) 2019-10-13T08:13:11.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:11.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:11.580958+0000) 2019-10-13T08:13:14.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:14.304+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:14.304833+0000) 2019-10-13T08:13:14.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:14.351+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:14.353065+0000) 2019-10-13T08:13:15.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:15.972+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.974549+0000) 2019-10-13T08:13:15.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:15.972+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.974915+0000) 2019-10-13T08:13:15.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:15.972+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.975626+0000) 2019-10-13T08:13:15.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:15.972+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.976123+0000) 2019-10-13T08:13:15.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:15.972+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.976634+0000) 2019-10-13T08:13:15.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:15.975+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.976338+0000) 2019-10-13T08:13:15.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:15.972+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.977423+0000) 2019-10-13T08:13:15.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:15.976+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:15.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:15.983+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:15.985222+0000) 2019-10-13T08:13:17.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:17.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:17.145471+0000) 2019-10-13T08:13:17.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:17.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:17.434875+0000) 2019-10-13T08:13:17.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:17.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:17.532900+0000) 2019-10-13T08:13:17.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:17.556+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:17.559433+0000) 2019-10-13T08:13:20.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:20.976+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.981436+0000) 2019-10-13T08:13:20.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:20.979+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.981959+0000) 2019-10-13T08:13:20.982 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:20.980+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.982561+0000) 2019-10-13T08:13:20.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:20.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.983205+0000) 2019-10-13T08:13:20.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:20.980+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.983423+0000) 2019-10-13T08:13:20.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:20.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:20.984 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:20.980+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.984177+0000) 2019-10-13T08:13:20.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:20.980+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.985076+0000) 2019-10-13T08:13:20.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:20.987+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:20.990214+0000) 2019-10-13T08:13:21.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:21.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:21.579957+0000) 2019-10-13T08:13:21.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:21.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:21.581153+0000) 2019-10-13T08:13:24.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:24.303+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:24.305036+0000) 2019-10-13T08:13:24.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:24.355+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:24.359658+0000) 2019-10-13T08:13:25.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:25.983+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.987252+0000) 2019-10-13T08:13:25.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:25.983+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.988374+0000) 2019-10-13T08:13:25.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:25.983+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.988704+0000) 2019-10-13T08:13:25.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:25.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:25.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:25.987+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.990675+0000) 2019-10-13T08:13:25.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:25.987+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.990559+0000) 2019-10-13T08:13:25.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:25.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.991429+0000) 2019-10-13T08:13:25.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:25.987+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.992622+0000) 2019-10-13T08:13:25.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:25.995+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:25.998825+0000) 2019-10-13T08:13:27.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:27.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:27.145635+0000) 2019-10-13T08:13:27.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:27.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:27.435071+0000) 2019-10-13T08:13:27.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:27.531+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:27.533098+0000) 2019-10-13T08:13:27.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:27.555+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:27.559675+0000) 2019-10-13T08:13:30.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:30.991+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.994064+0000) 2019-10-13T08:13:30.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:30.991+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.994868+0000) 2019-10-13T08:13:30.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:30.991+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.994910+0000) 2019-10-13T08:13:30.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:30.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:30.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:30.991+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.995564+0000) 2019-10-13T08:13:30.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:30.991+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.996685+0000) 2019-10-13T08:13:30.997 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:30.991+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.996421+0000) 2019-10-13T08:13:30.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:30.995+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:30.998885+0000) 2019-10-13T08:13:31.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:31.003+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:31.005275+0000) 2019-10-13T08:13:31.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:31.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:31.580136+0000) 2019-10-13T08:13:31.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:31.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:31.581326+0000) 2019-10-13T08:13:34.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:34.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:34.305249+0000) 2019-10-13T08:13:34.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:34.366+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:34.370540+0000) 2019-10-13T08:13:35.514 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:13:35.514 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:35.517 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:13:35.518 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:13:35.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:35.995+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.998909+0000) 2019-10-13T08:13:35.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:35.995+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:35.999747+0000) 2019-10-13T08:13:36.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:35.995+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.000694+0000) 2019-10-13T08:13:36.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:35.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:36.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:35.995+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.000997+0000) 2019-10-13T08:13:36.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:35.995+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.001420+0000) 2019-10-13T08:13:36.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:35.998+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.001445+0000) 2019-10-13T08:13:36.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:35.999+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.002786+0000) 2019-10-13T08:13:36.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:36.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:36.008626+0000) 2019-10-13T08:13:37.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:37.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:37.145842+0000) 2019-10-13T08:13:37.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:37.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:37.435277+0000) 2019-10-13T08:13:37.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:37.527+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:37.533326+0000) 2019-10-13T08:13:37.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:37.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:37.559889+0000) 2019-10-13T08:13:41.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:40.999+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.004595+0000) 2019-10-13T08:13:41.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:40.999+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.005319+0000) 2019-10-13T08:13:41.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:41.003+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.005854+0000) 2019-10-13T08:13:41.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:41.002+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.005730+0000) 2019-10-13T08:13:41.006 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:41.003+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.006589+0000) 2019-10-13T08:13:41.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:41.003+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.006936+0000) 2019-10-13T08:13:41.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:41.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68200 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:41.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:41.003+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.008834+0000) 2019-10-13T08:13:41.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:41.010+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.012650+0000) 2019-10-13T08:13:41.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:41.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.580317+0000) 2019-10-13T08:13:41.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:41.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:41.581491+0000) 2019-10-13T08:13:44.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:44.303+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:44.305450+0000) 2019-10-13T08:13:44.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:44.374+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:44.377000+0000) 2019-10-13T08:13:46.010 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:46.007+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.010724+0000) 2019-10-13T08:13:46.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:46.007+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.011220+0000) 2019-10-13T08:13:46.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:46.007+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.012007+0000) 2019-10-13T08:13:46.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:46.007+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.011763+0000) 2019-10-13T08:13:46.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:46.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:46.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:46.007+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.013013+0000) 2019-10-13T08:13:46.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:46.010+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.013657+0000) 2019-10-13T08:13:46.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:46.011+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.014666+0000) 2019-10-13T08:13:46.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:46.014+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:46.019489+0000) 2019-10-13T08:13:47.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:47.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:47.145995+0000) 2019-10-13T08:13:47.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:47.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:47.435469+0000) 2019-10-13T08:13:47.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:47.527+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:47.533518+0000) 2019-10-13T08:13:47.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:47.555+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:47.560103+0000) 2019-10-13T08:13:51.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:51.011+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.016513+0000) 2019-10-13T08:13:51.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:51.011+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.016878+0000) 2019-10-13T08:13:51.017 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:51.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.017235+0000) 2019-10-13T08:13:51.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:51.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68259 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:51.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:51.014+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.018723+0000) 2019-10-13T08:13:51.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:51.015+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.019988+0000) 2019-10-13T08:13:51.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:51.015+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.020954+0000) 2019-10-13T08:13:51.021 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:51.015+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.021493+0000) 2019-10-13T08:13:51.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:51.022+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.024603+0000) 2019-10-13T08:13:51.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:51.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.580497+0000) 2019-10-13T08:13:51.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:51.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:51.581663+0000) 2019-10-13T08:13:54.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:54.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:54.305648+0000) 2019-10-13T08:13:54.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:54.382+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:54.383482+0000) 2019-10-13T08:13:56.022 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:56.019+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.021948+0000) 2019-10-13T08:13:56.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:56.019+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.022588+0000) 2019-10-13T08:13:56.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:56.019+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.023396+0000) 2019-10-13T08:13:56.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:13:56.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:13:56.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:13:56.022+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.024671+0000) 2019-10-13T08:13:56.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:13:56.019+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.025512+0000) 2019-10-13T08:13:56.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:56.023+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.026006+0000) 2019-10-13T08:13:56.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:13:56.023+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.026855+0000) 2019-10-13T08:13:56.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:13:56.030+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:56.032862+0000) 2019-10-13T08:13:57.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:13:57.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:57.146149+0000) 2019-10-13T08:13:57.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:13:57.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:57.435694+0000) 2019-10-13T08:13:57.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:13:57.527+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:57.533720+0000) 2019-10-13T08:13:57.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:13:57.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:13:57.560291+0000) 2019-10-13T08:14:01.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:01.022+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.027939+0000) 2019-10-13T08:14:01.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:01.023+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.029546+0000) 2019-10-13T08:14:01.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:01.027+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.030064+0000) 2019-10-13T08:14:01.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:01.027+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.030590+0000) 2019-10-13T08:14:01.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:01.027+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.030780+0000) 2019-10-13T08:14:01.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:01.027+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.031117+0000) 2019-10-13T08:14:01.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:01.027+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.031313+0000) 2019-10-13T08:14:01.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:01.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:01.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:01.038+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.040150+0000) 2019-10-13T08:14:01.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:01.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.580693+0000) 2019-10-13T08:14:01.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:01.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:01.581841+0000) 2019-10-13T08:14:04.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:04.303+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:04.305846+0000) 2019-10-13T08:14:04.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:04.386+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:04.390021+0000) 2019-10-13T08:14:05.644 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:14:05.644 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:05.648 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:14:05.648 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:06.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:06.031+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.034461+0000) 2019-10-13T08:14:06.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:06.031+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.034723+0000) 2019-10-13T08:14:06.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:06.031+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.035936+0000) 2019-10-13T08:14:06.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:06.031+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.036422+0000) 2019-10-13T08:14:06.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:06.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:06.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:06.034+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.037988+0000) 2019-10-13T08:14:06.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:06.031+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.036942+0000) 2019-10-13T08:14:06.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:06.035+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.040659+0000) 2019-10-13T08:14:06.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:06.042+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:06.045330+0000) 2019-10-13T08:14:07.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:07.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:07.146345+0000) 2019-10-13T08:14:07.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:07.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:07.435891+0000) 2019-10-13T08:14:07.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:07.531+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:07.533953+0000) 2019-10-13T08:14:07.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:07.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:07.560505+0000) 2019-10-13T08:14:11.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:11.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.040277+0000) 2019-10-13T08:14:11.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:11.038+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.040138+0000) 2019-10-13T08:14:11.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:11.035+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.040990+0000) 2019-10-13T08:14:11.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:11.039+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.041880+0000) 2019-10-13T08:14:11.042 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:11.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:11.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:11.039+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.042367+0000) 2019-10-13T08:14:11.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:11.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.043151+0000) 2019-10-13T08:14:11.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:11.039+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.043473+0000) 2019-10-13T08:14:11.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:11.046+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.050285+0000) 2019-10-13T08:14:11.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:11.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.580878+0000) 2019-10-13T08:14:11.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:11.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:11.582030+0000) 2019-10-13T08:14:14.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:14.303+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:14.306010+0000) 2019-10-13T08:14:14.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:14.394+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:14.396396+0000) 2019-10-13T08:14:16.045 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:16.039+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.045336+0000) 2019-10-13T08:14:16.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:16.043+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.046148+0000) 2019-10-13T08:14:16.047 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:16.042+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.046423+0000) 2019-10-13T08:14:16.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:16.043+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.047311+0000) 2019-10-13T08:14:16.048 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:16.043+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.048251+0000) 2019-10-13T08:14:16.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:16.043+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.048431+0000) 2019-10-13T08:14:16.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:16.043+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.048909+0000) 2019-10-13T08:14:16.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:16.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68410 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:16.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:16.050+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:16.055701+0000) 2019-10-13T08:14:17.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:17.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:17.146547+0000) 2019-10-13T08:14:17.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:17.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:17.436090+0000) 2019-10-13T08:14:17.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:17.531+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:17.534159+0000) 2019-10-13T08:14:17.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:17.555+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:17.560735+0000) 2019-10-13T08:14:21.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:21.046+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.051381+0000) 2019-10-13T08:14:21.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:21.050+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.053848+0000) 2019-10-13T08:14:21.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:21.050+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.053583+0000) 2019-10-13T08:14:21.054 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:21.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:21.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:21.050+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.055421+0000) 2019-10-13T08:14:21.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:21.050+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.055718+0000) 2019-10-13T08:14:21.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:21.050+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.056178+0000) 2019-10-13T08:14:21.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:21.050+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.057277+0000) 2019-10-13T08:14:21.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:21.058+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.062185+0000) 2019-10-13T08:14:21.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:21.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.581079+0000) 2019-10-13T08:14:21.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:21.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:21.582230+0000) 2019-10-13T08:14:24.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:24.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:24.306241+0000) 2019-10-13T08:14:24.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:24.398+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:24.403094+0000) 2019-10-13T08:14:26.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:26.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.059289+0000) 2019-10-13T08:14:26.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:26.054+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.059792+0000) 2019-10-13T08:14:26.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:26.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68472 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:26.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:26.054+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.060773+0000) 2019-10-13T08:14:26.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:26.058+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.060527+0000) 2019-10-13T08:14:26.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:26.054+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.061514+0000) 2019-10-13T08:14:26.062 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:26.058+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.061976+0000) 2019-10-13T08:14:26.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:26.058+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.062615+0000) 2019-10-13T08:14:26.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:26.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:26.069460+0000) 2019-10-13T08:14:27.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:27.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:27.146762+0000) 2019-10-13T08:14:27.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:27.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:27.436291+0000) 2019-10-13T08:14:27.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:27.530+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:27.534358+0000) 2019-10-13T08:14:27.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:27.558+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:27.560946+0000) 2019-10-13T08:14:31.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:31.058+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.064027+0000) 2019-10-13T08:14:31.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:31.058+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.064281+0000) 2019-10-13T08:14:31.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:31.062+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.064273+0000) 2019-10-13T08:14:31.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:31.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:31.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:31.062+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.066883+0000) 2019-10-13T08:14:31.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:31.062+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.067355+0000) 2019-10-13T08:14:31.067 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:31.062+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.067894+0000) 2019-10-13T08:14:31.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:31.062+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.069317+0000) 2019-10-13T08:14:31.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:31.070+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.075375+0000) 2019-10-13T08:14:31.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:31.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.581256+0000) 2019-10-13T08:14:31.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:31.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:31.582409+0000) 2019-10-13T08:14:34.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:34.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:34.306459+0000) 2019-10-13T08:14:34.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:34.406+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:34.409796+0000) 2019-10-13T08:14:35.748 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:14:35.749 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:35.752 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:14:35.753 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:14:36.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:36.066+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.070703+0000) 2019-10-13T08:14:36.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:36.066+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.072023+0000) 2019-10-13T08:14:36.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:36.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:36.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:36.070+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.073887+0000) 2019-10-13T08:14:36.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:36.070+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.074237+0000) 2019-10-13T08:14:36.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:36.070+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.074338+0000) 2019-10-13T08:14:36.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:36.070+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.075272+0000) 2019-10-13T08:14:36.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:36.070+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.075855+0000) 2019-10-13T08:14:36.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:36.078+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:36.082871+0000) 2019-10-13T08:14:37.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:37.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:37.146961+0000) 2019-10-13T08:14:37.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:37.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:37.436490+0000) 2019-10-13T08:14:37.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:37.530+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:37.534600+0000) 2019-10-13T08:14:37.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:37.558+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:37.561152+0000) 2019-10-13T08:14:41.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:41.070+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.076172+0000) 2019-10-13T08:14:41.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:41.070+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.076836+0000) 2019-10-13T08:14:41.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:41.070+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.077165+0000) 2019-10-13T08:14:41.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:41.074+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.078287+0000) 2019-10-13T08:14:41.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:41.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.078324+0000) 2019-10-13T08:14:41.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:41.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:41.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:41.074+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.078739+0000) 2019-10-13T08:14:41.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:41.074+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.080333+0000) 2019-10-13T08:14:41.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:41.081+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.085372+0000) 2019-10-13T08:14:41.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:41.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.581396+0000) 2019-10-13T08:14:41.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:41.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:41.582585+0000) 2019-10-13T08:14:44.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:44.306+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:44.306673+0000) 2019-10-13T08:14:44.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:44.417+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:44.420701+0000) 2019-10-13T08:14:46.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:46.078+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.082503+0000) 2019-10-13T08:14:46.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:46.077+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.083702+0000) 2019-10-13T08:14:46.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:46.078+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.084521+0000) 2019-10-13T08:14:46.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:46.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:46.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:46.082+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.087080+0000) 2019-10-13T08:14:46.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:46.082+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.086203+0000) 2019-10-13T08:14:46.087 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:46.082+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.087959+0000) 2019-10-13T08:14:46.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:46.082+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.088913+0000) 2019-10-13T08:14:46.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:46.089+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:46.093757+0000) 2019-10-13T08:14:47.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:47.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:47.147222+0000) 2019-10-13T08:14:47.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:47.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:47.436718+0000) 2019-10-13T08:14:47.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:47.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:47.534820+0000) 2019-10-13T08:14:47.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:47.554+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:47.561305+0000) 2019-10-13T08:14:51.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:51.082+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.088967+0000) 2019-10-13T08:14:51.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:51.082+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.089269+0000) 2019-10-13T08:14:51.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:51.086+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.090730+0000) 2019-10-13T08:14:51.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:51.086+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.090534+0000) 2019-10-13T08:14:51.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:51.085+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.091037+0000) 2019-10-13T08:14:51.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:51.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:51.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:51.086+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.092395+0000) 2019-10-13T08:14:51.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:51.086+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.093273+0000) 2019-10-13T08:14:51.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:51.093+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.098227+0000) 2019-10-13T08:14:51.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:51.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.581570+0000) 2019-10-13T08:14:51.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:51.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:51.582785+0000) 2019-10-13T08:14:54.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:54.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:54.306874+0000) 2019-10-13T08:14:54.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:54.425+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:54.427102+0000) 2019-10-13T08:14:56.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:14:56.090+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.095039+0000) 2019-10-13T08:14:56.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:56.090+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.096365+0000) 2019-10-13T08:14:56.096 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:14:56.093+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.096413+0000) 2019-10-13T08:14:56.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:14:56.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:14:56.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:56.094+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.098070+0000) 2019-10-13T08:14:56.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:14:56.094+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.098048+0000) 2019-10-13T08:14:56.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:56.094+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.098590+0000) 2019-10-13T08:14:56.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:56.094+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.098941+0000) 2019-10-13T08:14:56.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:14:56.097+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:56.102013+0000) 2019-10-13T08:14:57.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:14:57.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:57.147452+0000) 2019-10-13T08:14:57.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:14:57.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:57.436913+0000) 2019-10-13T08:14:57.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:14:57.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:57.535044+0000) 2019-10-13T08:14:57.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:14:57.558+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:14:57.561503+0000) 2019-10-13T08:15:01.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:01.094+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.101055+0000) 2019-10-13T08:15:01.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:01.094+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.101438+0000) 2019-10-13T08:15:01.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:01.098+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.102717+0000) 2019-10-13T08:15:01.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:01.097+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.103238+0000) 2019-10-13T08:15:01.104 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:01.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:01.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:01.098+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.104379+0000) 2019-10-13T08:15:01.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:01.098+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.104593+0000) 2019-10-13T08:15:01.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:01.098+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.105250+0000) 2019-10-13T08:15:01.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:01.105+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.111097+0000) 2019-10-13T08:15:01.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:01.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.581778+0000) 2019-10-13T08:15:01.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:01.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:01.582979+0000) 2019-10-13T08:15:04.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:04.306+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:04.307087+0000) 2019-10-13T08:15:04.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:04.429+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:04.433637+0000) 2019-10-13T08:15:05.866 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:15:05.866 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:05.870 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:15:05.870 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:06.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:06.102+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.108466+0000) 2019-10-13T08:15:06.110 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:06.106+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.110248+0000) 2019-10-13T08:15:06.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:06.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:06.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:06.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.110744+0000) 2019-10-13T08:15:06.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:06.106+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.111134+0000) 2019-10-13T08:15:06.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:06.106+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.111183+0000) 2019-10-13T08:15:06.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:06.105+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.111872+0000) 2019-10-13T08:15:06.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:06.106+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.111937+0000) 2019-10-13T08:15:06.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:06.117+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:06.120512+0000) 2019-10-13T08:15:07.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:07.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:07.147607+0000) 2019-10-13T08:15:07.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:07.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:07.437107+0000) 2019-10-13T08:15:07.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:07.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:07.535272+0000) 2019-10-13T08:15:07.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:07.554+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:07.561701+0000) 2019-10-13T08:15:11.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:11.109+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.114080+0000) 2019-10-13T08:15:11.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:11.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.115531+0000) 2019-10-13T08:15:11.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:11.110+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.115042+0000) 2019-10-13T08:15:11.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:11.110+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.115912+0000) 2019-10-13T08:15:11.116 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:11.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:11.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:11.114+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.118100+0000) 2019-10-13T08:15:11.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:11.110+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.117711+0000) 2019-10-13T08:15:11.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:11.114+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.118793+0000) 2019-10-13T08:15:11.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:11.121+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.126348+0000) 2019-10-13T08:15:11.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:11.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.581959+0000) 2019-10-13T08:15:11.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:11.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:11.583175+0000) 2019-10-13T08:15:14.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:14.306+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:14.307296+0000) 2019-10-13T08:15:14.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:14.437+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:14.440143+0000) 2019-10-13T08:15:16.120 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:16.113+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.120431+0000) 2019-10-13T08:15:16.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:16.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.120894+0000) 2019-10-13T08:15:16.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:16.117+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.121940+0000) 2019-10-13T08:15:16.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:16.117+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.122022+0000) 2019-10-13T08:15:16.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:16.117+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.122722+0000) 2019-10-13T08:15:16.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:16.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:16.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:16.117+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.123798+0000) 2019-10-13T08:15:16.124 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:16.117+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.124207+0000) 2019-10-13T08:15:16.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:16.129+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:16.132795+0000) 2019-10-13T08:15:17.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:17.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:17.147826+0000) 2019-10-13T08:15:17.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:17.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:17.437306+0000) 2019-10-13T08:15:17.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:17.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:17.535475+0000) 2019-10-13T08:15:17.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:17.557+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:17.561946+0000) 2019-10-13T08:15:21.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:21.121+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.126620+0000) 2019-10-13T08:15:21.128 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:21.121+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.128762+0000) 2019-10-13T08:15:21.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:21.121+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.129301+0000) 2019-10-13T08:15:21.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:21.121+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.129393+0000) 2019-10-13T08:15:21.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:21.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:21.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:21.125+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.130394+0000) 2019-10-13T08:15:21.131 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:21.125+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.130564+0000) 2019-10-13T08:15:21.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:21.125+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.131588+0000) 2019-10-13T08:15:21.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:21.129+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.132975+0000) 2019-10-13T08:15:21.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:21.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.582136+0000) 2019-10-13T08:15:21.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:21.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:21.583349+0000) 2019-10-13T08:15:24.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:24.305+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:24.307527+0000) 2019-10-13T08:15:24.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:24.441+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:24.446544+0000) 2019-10-13T08:15:26.133 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:26.129+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.133082+0000) 2019-10-13T08:15:26.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:26.129+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.133485+0000) 2019-10-13T08:15:26.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:26.129+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.133821+0000) 2019-10-13T08:15:26.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:26.129+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.134594+0000) 2019-10-13T08:15:26.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:26.129+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.135001+0000) 2019-10-13T08:15:26.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:26.129+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.135292+0000) 2019-10-13T08:15:26.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:26.129+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.135892+0000) 2019-10-13T08:15:26.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:26.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:26.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:26.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:26.140487+0000) 2019-10-13T08:15:27.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:27.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:27.148030+0000) 2019-10-13T08:15:27.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:27.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:27.437546+0000) 2019-10-13T08:15:27.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:27.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:27.535678+0000) 2019-10-13T08:15:27.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:27.557+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:27.562151+0000) 2019-10-13T08:15:31.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:31.137+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.141315+0000) 2019-10-13T08:15:31.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:31.137+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.141469+0000) 2019-10-13T08:15:31.142 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:31.137+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.141298+0000) 2019-10-13T08:15:31.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:31.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:31.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:31.141+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.142393+0000) 2019-10-13T08:15:31.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:31.141+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.143047+0000) 2019-10-13T08:15:31.143 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:31.141+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.143252+0000) 2019-10-13T08:15:31.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:31.141+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.144661+0000) 2019-10-13T08:15:31.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:31.145+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.149392+0000) 2019-10-13T08:15:31.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:31.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.582312+0000) 2019-10-13T08:15:31.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:31.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:31.583521+0000) 2019-10-13T08:15:34.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:34.305+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:34.307734+0000) 2019-10-13T08:15:34.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:34.449+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:34.453088+0000) 2019-10-13T08:15:36.124 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:15:36.124 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:36.127 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:15:36.128 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:15:36.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:36.141+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.144725+0000) 2019-10-13T08:15:36.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:36.141+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.144401+0000) 2019-10-13T08:15:36.145 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:36.141+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.145096+0000) 2019-10-13T08:15:36.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:36.141+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.145708+0000) 2019-10-13T08:15:36.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:36.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:36.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:36.145+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.147527+0000) 2019-10-13T08:15:36.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:36.145+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.147988+0000) 2019-10-13T08:15:36.148 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:36.145+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.148490+0000) 2019-10-13T08:15:36.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:36.149+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:36.152220+0000) 2019-10-13T08:15:37.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:37.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:37.148233+0000) 2019-10-13T08:15:37.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:37.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:37.437784+0000) 2019-10-13T08:15:37.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:37.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:37.535878+0000) 2019-10-13T08:15:37.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:37.561+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:37.562352+0000) 2019-10-13T08:15:41.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:41.149+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.150689+0000) 2019-10-13T08:15:41.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:41.149+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.151474+0000) 2019-10-13T08:15:41.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:41.149+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.152575+0000) 2019-10-13T08:15:41.153 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:41.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.152374+0000) 2019-10-13T08:15:41.153 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:41.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:41.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:41.149+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.153517+0000) 2019-10-13T08:15:41.154 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:41.153+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.154538+0000) 2019-10-13T08:15:41.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:41.153+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.154468+0000) 2019-10-13T08:15:41.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:41.153+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.158750+0000) 2019-10-13T08:15:41.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:41.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.582520+0000) 2019-10-13T08:15:41.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:41.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:41.583724+0000) 2019-10-13T08:15:44.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:44.305+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:44.307936+0000) 2019-10-13T08:15:44.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:44.453+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:44.459537+0000) 2019-10-13T08:15:46.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:46.157+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.158019+0000) 2019-10-13T08:15:46.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:46.157+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.158297+0000) 2019-10-13T08:15:46.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:46.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68959 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:46.160 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:46.153+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.158936+0000) 2019-10-13T08:15:46.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:46.157+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.159572+0000) 2019-10-13T08:15:46.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:46.157+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.159549+0000) 2019-10-13T08:15:46.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:46.157+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.159804+0000) 2019-10-13T08:15:46.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:46.157+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.160775+0000) 2019-10-13T08:15:46.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:46.161+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:46.167116+0000) 2019-10-13T08:15:47.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:47.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:47.148435+0000) 2019-10-13T08:15:47.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:47.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:47.438037+0000) 2019-10-13T08:15:47.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:47.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:47.536109+0000) 2019-10-13T08:15:47.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:47.561+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:47.562571+0000) 2019-10-13T08:15:51.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:51.161+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.162365+0000) 2019-10-13T08:15:51.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:51.161+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.162612+0000) 2019-10-13T08:15:51.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:51.156+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.162803+0000) 2019-10-13T08:15:51.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:51.161+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.163603+0000) 2019-10-13T08:15:51.164 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:51.161+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.163554+0000) 2019-10-13T08:15:51.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:51.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:51.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:51.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.165589+0000) 2019-10-13T08:15:51.166 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:51.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.166277+0000) 2019-10-13T08:15:51.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:51.164+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.169266+0000) 2019-10-13T08:15:51.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:51.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.582696+0000) 2019-10-13T08:15:51.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:51.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:51.583913+0000) 2019-10-13T08:15:54.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:54.305+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:54.308142+0000) 2019-10-13T08:15:54.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:54.464+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:54.465565+0000) 2019-10-13T08:15:56.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:56.165+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.168318+0000) 2019-10-13T08:15:56.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:56.165+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.169007+0000) 2019-10-13T08:15:56.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:15:56.164+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.169100+0000) 2019-10-13T08:15:56.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:56.169+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.170048+0000) 2019-10-13T08:15:56.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:15:56.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:15:56.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:15:56.169+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.171562+0000) 2019-10-13T08:15:56.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:56.169+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.172545+0000) 2019-10-13T08:15:56.173 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:15:56.169+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.172634+0000) 2019-10-13T08:15:56.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:15:56.172+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:56.177912+0000) 2019-10-13T08:15:57.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:15:57.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:57.148614+0000) 2019-10-13T08:15:57.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:15:57.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:57.438246+0000) 2019-10-13T08:15:57.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:15:57.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:57.536307+0000) 2019-10-13T08:15:57.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:15:57.561+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:15:57.562719+0000) 2019-10-13T08:16:01.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:01.173+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.174542+0000) 2019-10-13T08:16:01.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:01.168+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.174726+0000) 2019-10-13T08:16:01.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:01.173+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.175988+0000) 2019-10-13T08:16:01.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:01.173+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.176593+0000) 2019-10-13T08:16:01.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:01.173+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.176269+0000) 2019-10-13T08:16:01.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:01.173+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.177233+0000) 2019-10-13T08:16:01.178 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:01.173+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.177557+0000) 2019-10-13T08:16:01.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:01.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:01.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:01.176+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.183982+0000) 2019-10-13T08:16:01.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:01.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.582872+0000) 2019-10-13T08:16:01.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:01.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:01.584084+0000) 2019-10-13T08:16:04.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:04.305+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:04.308340+0000) 2019-10-13T08:16:04.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:04.468+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:04.471929+0000) 2019-10-13T08:16:06.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:06.177+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.179973+0000) 2019-10-13T08:16:06.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:06.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:06.184 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:06.181+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.183856+0000) 2019-10-13T08:16:06.184 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:06.180+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.184322+0000) 2019-10-13T08:16:06.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:06.181+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.185170+0000) 2019-10-13T08:16:06.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:06.181+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.185488+0000) 2019-10-13T08:16:06.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:06.185+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.186228+0000) 2019-10-13T08:16:06.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:06.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.186743+0000) 2019-10-13T08:16:06.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:06.184+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:06.189178+0000) 2019-10-13T08:16:06.196 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:16:06.196 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:06.201 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:16:06.201 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:07.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:07.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:07.148751+0000) 2019-10-13T08:16:07.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:07.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:07.438447+0000) 2019-10-13T08:16:07.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:07.533+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:07.536502+0000) 2019-10-13T08:16:07.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:07.561+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:07.562946+0000) 2019-10-13T08:16:11.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:11.184+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.188144+0000) 2019-10-13T08:16:11.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:11.185+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.189379+0000) 2019-10-13T08:16:11.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:11.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:11.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:11.189+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.191769+0000) 2019-10-13T08:16:11.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:11.189+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.191408+0000) 2019-10-13T08:16:11.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:11.189+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.192985+0000) 2019-10-13T08:16:11.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:11.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.192336+0000) 2019-10-13T08:16:11.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:11.189+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.193521+0000) 2019-10-13T08:16:11.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:11.192+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.196626+0000) 2019-10-13T08:16:11.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:11.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.583066+0000) 2019-10-13T08:16:11.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:11.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:11.584306+0000) 2019-10-13T08:16:14.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:14.304+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:14.308539+0000) 2019-10-13T08:16:14.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:14.476+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:14.478168+0000) 2019-10-13T08:16:16.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:16.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.193139+0000) 2019-10-13T08:16:16.196 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:16.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:16.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:16.192+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.196562+0000) 2019-10-13T08:16:16.196 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:16.192+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.196474+0000) 2019-10-13T08:16:16.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:16.192+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.197232+0000) 2019-10-13T08:16:16.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:16.196+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.198231+0000) 2019-10-13T08:16:16.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:16.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.199027+0000) 2019-10-13T08:16:16.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:16.196+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.199751+0000) 2019-10-13T08:16:16.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:16.196+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:16.202885+0000) 2019-10-13T08:16:17.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:17.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:17.148928+0000) 2019-10-13T08:16:17.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:17.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:17.438664+0000) 2019-10-13T08:16:17.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:17.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:17.536721+0000) 2019-10-13T08:16:17.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:17.560+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:17.563103+0000) 2019-10-13T08:16:21.199 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:21.196+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.199600+0000) 2019-10-13T08:16:21.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:21.196+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.199959+0000) 2019-10-13T08:16:21.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:21.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:21.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:21.200+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.202625+0000) 2019-10-13T08:16:21.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:21.200+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.202821+0000) 2019-10-13T08:16:21.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:21.196+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.202890+0000) 2019-10-13T08:16:21.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:21.200+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.203833+0000) 2019-10-13T08:16:21.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:21.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.204502+0000) 2019-10-13T08:16:21.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:21.204+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.209714+0000) 2019-10-13T08:16:21.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:21.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.583251+0000) 2019-10-13T08:16:21.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:21.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:21.584515+0000) 2019-10-13T08:16:24.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:24.304+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:24.308738+0000) 2019-10-13T08:16:24.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:24.488+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:24.489640+0000) 2019-10-13T08:16:26.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:26.204+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.206293+0000) 2019-10-13T08:16:26.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:26.204+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.206658+0000) 2019-10-13T08:16:26.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:26.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.207571+0000) 2019-10-13T08:16:26.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:26.204+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.207691+0000) 2019-10-13T08:16:26.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:26.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69203 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:26.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:26.200+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.208067+0000) 2019-10-13T08:16:26.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:26.208+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.209883+0000) 2019-10-13T08:16:26.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:26.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.213126+0000) 2019-10-13T08:16:26.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:26.208+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:26.213785+0000) 2019-10-13T08:16:27.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:27.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:27.149135+0000) 2019-10-13T08:16:27.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:27.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:27.438876+0000) 2019-10-13T08:16:27.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:27.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:27.536908+0000) 2019-10-13T08:16:27.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:27.560+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:27.563306+0000) 2019-10-13T08:16:31.211 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:31.208+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.211063+0000) 2019-10-13T08:16:31.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:31.208+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.211817+0000) 2019-10-13T08:16:31.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:31.204+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.211822+0000) 2019-10-13T08:16:31.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:31.208+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.213302+0000) 2019-10-13T08:16:31.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:31.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.213545+0000) 2019-10-13T08:16:31.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:31.212+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.214031+0000) 2019-10-13T08:16:31.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:31.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:31.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:31.212+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.214356+0000) 2019-10-13T08:16:31.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:31.216+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.221518+0000) 2019-10-13T08:16:31.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:31.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.583450+0000) 2019-10-13T08:16:31.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:31.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:31.584683+0000) 2019-10-13T08:16:34.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:34.308+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:34.308961+0000) 2019-10-13T08:16:34.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:34.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:34.496053+0000) 2019-10-13T08:16:36.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:36.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.217687+0000) 2019-10-13T08:16:36.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:36.212+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.218100+0000) 2019-10-13T08:16:36.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:36.216+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.218254+0000) 2019-10-13T08:16:36.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:36.216+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.218998+0000) 2019-10-13T08:16:36.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:36.216+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.221029+0000) 2019-10-13T08:16:36.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:36.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:36.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:36.216+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.221342+0000) 2019-10-13T08:16:36.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:36.220+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.222107+0000) 2019-10-13T08:16:36.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:36.224+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:36.231205+0000) 2019-10-13T08:16:36.261 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:16:36.261 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:36.265 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:16:36.265 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:16:37.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:37.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:37.149311+0000) 2019-10-13T08:16:37.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:37.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:37.439101+0000) 2019-10-13T08:16:37.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:37.536+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:37.537070+0000) 2019-10-13T08:16:37.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:37.560+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:37.563541+0000) 2019-10-13T08:16:41.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:41.220+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.224212+0000) 2019-10-13T08:16:41.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:41.224+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.226429+0000) 2019-10-13T08:16:41.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:41.224+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.226918+0000) 2019-10-13T08:16:41.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:41.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:41.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:41.224+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.227631+0000) 2019-10-13T08:16:41.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:41.224+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.227939+0000) 2019-10-13T08:16:41.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:41.224+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.228880+0000) 2019-10-13T08:16:41.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:41.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.583651+0000) 2019-10-13T08:16:41.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:41.224+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.228235+0000) 2019-10-13T08:16:41.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:41.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.232357+0000) 2019-10-13T08:16:41.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:41.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:41.584871+0000) 2019-10-13T08:16:44.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:44.308+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:44.309170+0000) 2019-10-13T08:16:44.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:44.496+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:44.502513+0000) 2019-10-13T08:16:46.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:46.224+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.230544+0000) 2019-10-13T08:16:46.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:46.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.231584+0000) 2019-10-13T08:16:46.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:46.228+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.231852+0000) 2019-10-13T08:16:46.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:46.228+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.232652+0000) 2019-10-13T08:16:46.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:46.228+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.232709+0000) 2019-10-13T08:16:46.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:46.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.232559+0000) 2019-10-13T08:16:46.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:46.228+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.233143+0000) 2019-10-13T08:16:46.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:46.228+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:46.233536+0000) 2019-10-13T08:16:46.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:46.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:47.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:47.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:47.149484+0000) 2019-10-13T08:16:47.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:47.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:47.439296+0000) 2019-10-13T08:16:47.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:47.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:47.537292+0000) 2019-10-13T08:16:47.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:47.560+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:47.563772+0000) 2019-10-13T08:16:51.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:51.232+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.237591+0000) 2019-10-13T08:16:51.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:51.236+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.237885+0000) 2019-10-13T08:16:51.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:51.236+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.237709+0000) 2019-10-13T08:16:51.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:51.236+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.238755+0000) 2019-10-13T08:16:51.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:51.236+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.238927+0000) 2019-10-13T08:16:51.239 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:51.236+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.239345+0000) 2019-10-13T08:16:51.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:51.236+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.239895+0000) 2019-10-13T08:16:51.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:51.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:51.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:51.240+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.242810+0000) 2019-10-13T08:16:51.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:51.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.583862+0000) 2019-10-13T08:16:51.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:51.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:51.585079+0000) 2019-10-13T08:16:54.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:54.304+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:54.309368+0000) 2019-10-13T08:16:54.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:54.507+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:54.508939+0000) 2019-10-13T08:16:56.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:56.240+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.243037+0000) 2019-10-13T08:16:56.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:16:56.240+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.243575+0000) 2019-10-13T08:16:56.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:56.240+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.243901+0000) 2019-10-13T08:16:56.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:56.244+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.245901+0000) 2019-10-13T08:16:56.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:16:56.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:16:56.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:56.244+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.246679+0000) 2019-10-13T08:16:56.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:16:56.243+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.246589+0000) 2019-10-13T08:16:56.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:16:56.248+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.249814+0000) 2019-10-13T08:16:56.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:16:56.247+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:56.251298+0000) 2019-10-13T08:16:57.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:16:57.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:57.149667+0000) 2019-10-13T08:16:57.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:16:57.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:57.439497+0000) 2019-10-13T08:16:57.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:16:57.532+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:57.537486+0000) 2019-10-13T08:16:57.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:16:57.560+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:16:57.563974+0000) 2019-10-13T08:17:01.250 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:01.248+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.250154+0000) 2019-10-13T08:17:01.251 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:01.248+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.251943+0000) 2019-10-13T08:17:01.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:01.248+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.252599+0000) 2019-10-13T08:17:01.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:01.248+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.252716+0000) 2019-10-13T08:17:01.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:01.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:01.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:01.248+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.253207+0000) 2019-10-13T08:17:01.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:01.251+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.253582+0000) 2019-10-13T08:17:01.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:01.248+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.253650+0000) 2019-10-13T08:17:01.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:01.251+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.255456+0000) 2019-10-13T08:17:01.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:01.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.584058+0000) 2019-10-13T08:17:01.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:01.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:01.585255+0000) 2019-10-13T08:17:04.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:04.304+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:04.309598+0000) 2019-10-13T08:17:04.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:04.511+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:04.515518+0000) 2019-10-13T08:17:06.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:06.252+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.256233+0000) 2019-10-13T08:17:06.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:06.252+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.256735+0000) 2019-10-13T08:17:06.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:06.255+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.256833+0000) 2019-10-13T08:17:06.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:06.252+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.257548+0000) 2019-10-13T08:17:06.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:06.256+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.258448+0000) 2019-10-13T08:17:06.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:06.256+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.259048+0000) 2019-10-13T08:17:06.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:06.256+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.259285+0000) 2019-10-13T08:17:06.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:06.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:06.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:06.263+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:06.265907+0000) 2019-10-13T08:17:06.368 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:17:06.368 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:06.371 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:17:06.371 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:07.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:07.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:07.149850+0000) 2019-10-13T08:17:07.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:07.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:07.439712+0000) 2019-10-13T08:17:07.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:07.536+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:07.537682+0000) 2019-10-13T08:17:07.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:07.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:07.564183+0000) 2019-10-13T08:17:11.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:11.259+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.263215+0000) 2019-10-13T08:17:11.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:11.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:11.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:11.263+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.266325+0000) 2019-10-13T08:17:11.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:11.263+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.266713+0000) 2019-10-13T08:17:11.267 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:11.263+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.267293+0000) 2019-10-13T08:17:11.268 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:11.263+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.268025+0000) 2019-10-13T08:17:11.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:11.263+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.267039+0000) 2019-10-13T08:17:11.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:11.263+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.269072+0000) 2019-10-13T08:17:11.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:11.267+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.269033+0000) 2019-10-13T08:17:11.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:11.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.584239+0000) 2019-10-13T08:17:11.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:11.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:11.585493+0000) 2019-10-13T08:17:14.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:14.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:14.309843+0000) 2019-10-13T08:17:14.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:14.519+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:14.522006+0000) 2019-10-13T08:17:16.269 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:16.267+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.268875+0000) 2019-10-13T08:17:16.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:16.267+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.271214+0000) 2019-10-13T08:17:16.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:16.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:16.272 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:16.267+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.271984+0000) 2019-10-13T08:17:16.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:16.267+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.272739+0000) 2019-10-13T08:17:16.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:16.271+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.273944+0000) 2019-10-13T08:17:16.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:16.271+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.273984+0000) 2019-10-13T08:17:16.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:16.271+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.274228+0000) 2019-10-13T08:17:16.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:16.275+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:16.277422+0000) 2019-10-13T08:17:17.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:17.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:17.150015+0000) 2019-10-13T08:17:17.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:17.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:17.439918+0000) 2019-10-13T08:17:17.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:17.535+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:17.537909+0000) 2019-10-13T08:17:17.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:17.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:17.564417+0000) 2019-10-13T08:17:21.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:21.271+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.275094+0000) 2019-10-13T08:17:21.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:21.271+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.274792+0000) 2019-10-13T08:17:21.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:21.271+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.276767+0000) 2019-10-13T08:17:21.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:21.271+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.277049+0000) 2019-10-13T08:17:21.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:21.271+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.277476+0000) 2019-10-13T08:17:21.277 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:21.275+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.277793+0000) 2019-10-13T08:17:21.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:21.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:21.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:21.275+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.278150+0000) 2019-10-13T08:17:21.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:21.279+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.281367+0000) 2019-10-13T08:17:21.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:21.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.584415+0000) 2019-10-13T08:17:21.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:21.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:21.585686+0000) 2019-10-13T08:17:24.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:24.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:24.310074+0000) 2019-10-13T08:17:24.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:24.527+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:24.528602+0000) 2019-10-13T08:17:26.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:26.279+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.281805+0000) 2019-10-13T08:17:26.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:26.279+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.283571+0000) 2019-10-13T08:17:26.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:26.279+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.283379+0000) 2019-10-13T08:17:26.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:26.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:26.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:26.279+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.284813+0000) 2019-10-13T08:17:26.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:26.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.285349+0000) 2019-10-13T08:17:26.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:26.279+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.285310+0000) 2019-10-13T08:17:26.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:26.283+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.285826+0000) 2019-10-13T08:17:26.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:26.287+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:26.291307+0000) 2019-10-13T08:17:27.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:27.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:27.150159+0000) 2019-10-13T08:17:27.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:27.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:27.440148+0000) 2019-10-13T08:17:29.666 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:27.535+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:27.538112+0000) 2019-10-13T08:17:29.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:27.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:27.564618+0000) 2019-10-13T08:17:31.287 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:31.283+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.287181+0000) 2019-10-13T08:17:31.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:31.283+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.287483+0000) 2019-10-13T08:17:31.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:31.283+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.287827+0000) 2019-10-13T08:17:31.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:31.283+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.289292+0000) 2019-10-13T08:17:31.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:31.287+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.289156+0000) 2019-10-13T08:17:31.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:31.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:31.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:31.287+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.290401+0000) 2019-10-13T08:17:31.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:31.287+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.290944+0000) 2019-10-13T08:17:31.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:31.291+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.294469+0000) 2019-10-13T08:17:31.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:31.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.584625+0000) 2019-10-13T08:17:31.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:31.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:31.585857+0000) 2019-10-13T08:17:34.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:34.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:34.310289+0000) 2019-10-13T08:17:34.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:34.531+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:34.535113+0000) 2019-10-13T08:17:36.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:36.287+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.292675+0000) 2019-10-13T08:17:36.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:36.291+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.293991+0000) 2019-10-13T08:17:36.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:36.291+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.294928+0000) 2019-10-13T08:17:36.295 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:36.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.295218+0000) 2019-10-13T08:17:36.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:36.291+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.295719+0000) 2019-10-13T08:17:36.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:36.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:36.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:36.291+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.295991+0000) 2019-10-13T08:17:36.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:36.291+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.296744+0000) 2019-10-13T08:17:36.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:36.299+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:36.300302+0000) 2019-10-13T08:17:36.468 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:17:36.469 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:36.472 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:17:36.472 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:17:37.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:37.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:37.150360+0000) 2019-10-13T08:17:37.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:37.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:37.440370+0000) 2019-10-13T08:17:37.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:37.535+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:37.538357+0000) 2019-10-13T08:17:37.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:37.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:37.564783+0000) 2019-10-13T08:17:41.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:41.295+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.299347+0000) 2019-10-13T08:17:41.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:41.295+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.300037+0000) 2019-10-13T08:17:41.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:41.299+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.300849+0000) 2019-10-13T08:17:41.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:41.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:41.302 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:41.299+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.301904+0000) 2019-10-13T08:17:41.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:41.299+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.302423+0000) 2019-10-13T08:17:41.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:41.299+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.302670+0000) 2019-10-13T08:17:41.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:41.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.302872+0000) 2019-10-13T08:17:41.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:41.303+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.306606+0000) 2019-10-13T08:17:41.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:41.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.584836+0000) 2019-10-13T08:17:41.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:41.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:41.586021+0000) 2019-10-13T08:17:44.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:44.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:44.310505+0000) 2019-10-13T08:17:44.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:44.539+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:44.542594+0000) 2019-10-13T08:17:46.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:46.299+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.305061+0000) 2019-10-13T08:17:46.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:46.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.305377+0000) 2019-10-13T08:17:46.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:46.303+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.306287+0000) 2019-10-13T08:17:46.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:46.303+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.306922+0000) 2019-10-13T08:17:46.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:46.303+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.307224+0000) 2019-10-13T08:17:46.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:46.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:46.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:46.303+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.308045+0000) 2019-10-13T08:17:46.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:46.303+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.308525+0000) 2019-10-13T08:17:46.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:46.311+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:46.312922+0000) 2019-10-13T08:17:47.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:47.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:47.150556+0000) 2019-10-13T08:17:47.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:47.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:47.440534+0000) 2019-10-13T08:17:47.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:47.535+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:47.538559+0000) 2019-10-13T08:17:47.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:47.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:47.564974+0000) 2019-10-13T08:17:51.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:51.307+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.310776+0000) 2019-10-13T08:17:51.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:51.307+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.311778+0000) 2019-10-13T08:17:51.313 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:51.307+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.312976+0000) 2019-10-13T08:17:51.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:51.311+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.313852+0000) 2019-10-13T08:17:51.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:51.311+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.313818+0000) 2019-10-13T08:17:51.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:51.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:51.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:51.311+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.315054+0000) 2019-10-13T08:17:51.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:51.311+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.314738+0000) 2019-10-13T08:17:51.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:51.319+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.320393+0000) 2019-10-13T08:17:51.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:51.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.585050+0000) 2019-10-13T08:17:51.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:51.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:51.586227+0000) 2019-10-13T08:17:54.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:54.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:54.310662+0000) 2019-10-13T08:17:54.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:54.547+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:54.549082+0000) 2019-10-13T08:17:56.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:56.315+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.317969+0000) 2019-10-13T08:17:56.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:56.315+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.318494+0000) 2019-10-13T08:17:56.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:17:56.315+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.319297+0000) 2019-10-13T08:17:56.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:17:56.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.319195+0000) 2019-10-13T08:17:56.320 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:17:56.315+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.320004+0000) 2019-10-13T08:17:56.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:17:56.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:17:56.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:56.315+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.320549+0000) 2019-10-13T08:17:56.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:56.315+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.320892+0000) 2019-10-13T08:17:56.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:17:56.319+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:56.322959+0000) 2019-10-13T08:17:57.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:17:57.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:57.150734+0000) 2019-10-13T08:17:57.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:17:57.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:57.440729+0000) 2019-10-13T08:17:57.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:17:57.535+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:57.538763+0000) 2019-10-13T08:17:57.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:17:57.559+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:17:57.565176+0000) 2019-10-13T08:18:01.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:01.319+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.324028+0000) 2019-10-13T08:18:01.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:01.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.324698+0000) 2019-10-13T08:18:01.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:01.319+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.324695+0000) 2019-10-13T08:18:01.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:01.322+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.325201+0000) 2019-10-13T08:18:01.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:01.319+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.325266+0000) 2019-10-13T08:18:01.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:01.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69781 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:01.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:01.323+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.326851+0000) 2019-10-13T08:18:01.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:01.323+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.327324+0000) 2019-10-13T08:18:01.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:01.330+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.333003+0000) 2019-10-13T08:18:01.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:01.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.585225+0000) 2019-10-13T08:18:01.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:01.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:01.586398+0000) 2019-10-13T08:18:04.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:04.307+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:04.310917+0000) 2019-10-13T08:18:04.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:04.554+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:04.555613+0000) 2019-10-13T08:18:06.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:06.322+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.329668+0000) 2019-10-13T08:18:06.330 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:06.326+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.330212+0000) 2019-10-13T08:18:06.331 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:06.326+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.330368+0000) 2019-10-13T08:18:06.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:06.326+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.330470+0000) 2019-10-13T08:18:06.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:06.326+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.331848+0000) 2019-10-13T08:18:06.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:06.326+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.332066+0000) 2019-10-13T08:18:06.332 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:06.326+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.332314+0000) 2019-10-13T08:18:06.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:06.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:06.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:06.334+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:06.338547+0000) 2019-10-13T08:18:06.584 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:18:06.585 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:06.588 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:18:06.588 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:07.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:07.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:07.150916+0000) 2019-10-13T08:18:07.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:07.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:07.440965+0000) 2019-10-13T08:18:07.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:07.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:07.538965+0000) 2019-10-13T08:18:07.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:07.558+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:07.565368+0000) 2019-10-13T08:18:11.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:11.330+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.335914+0000) 2019-10-13T08:18:11.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:11.330+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.336857+0000) 2019-10-13T08:18:11.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:11.330+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.337312+0000) 2019-10-13T08:18:11.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:11.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.337599+0000) 2019-10-13T08:18:11.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:11.334+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.337517+0000) 2019-10-13T08:18:11.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:11.334+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.338530+0000) 2019-10-13T08:18:11.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:11.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:11.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:11.334+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.339319+0000) 2019-10-13T08:18:11.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:11.342+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.345092+0000) 2019-10-13T08:18:11.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:11.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.585401+0000) 2019-10-13T08:18:11.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:11.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:11.586526+0000) 2019-10-13T08:18:14.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:14.306+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:14.311147+0000) 2019-10-13T08:18:14.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:14.558+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:14.562060+0000) 2019-10-13T08:18:16.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:16.338+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.342103+0000) 2019-10-13T08:18:16.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:16.338+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.341895+0000) 2019-10-13T08:18:16.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:16.338+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.343272+0000) 2019-10-13T08:18:16.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:16.338+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.343551+0000) 2019-10-13T08:18:16.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:16.338+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.344186+0000) 2019-10-13T08:18:16.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:16.338+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.344545+0000) 2019-10-13T08:18:16.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:16.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:16.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:16.338+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.345698+0000) 2019-10-13T08:18:16.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:16.346+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:16.350765+0000) 2019-10-13T08:18:17.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:17.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:17.151089+0000) 2019-10-13T08:18:17.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:17.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:17.441195+0000) 2019-10-13T08:18:17.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:17.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:17.539168+0000) 2019-10-13T08:18:17.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:17.562+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:17.565558+0000) 2019-10-13T08:18:21.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:21.342+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.349243+0000) 2019-10-13T08:18:21.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:21.346+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.350963+0000) 2019-10-13T08:18:21.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:21.346+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.351486+0000) 2019-10-13T08:18:21.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:21.346+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.351780+0000) 2019-10-13T08:18:21.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:21.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:21.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:21.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.352739+0000) 2019-10-13T08:18:21.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:21.350+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.352896+0000) 2019-10-13T08:18:21.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:21.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.353183+0000) 2019-10-13T08:18:21.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:21.358+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.361676+0000) 2019-10-13T08:18:21.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:21.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.585606+0000) 2019-10-13T08:18:21.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:21.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:21.586654+0000) 2019-10-13T08:18:24.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:24.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:24.311349+0000) 2019-10-13T08:18:24.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:24.566+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:24.568527+0000) 2019-10-13T08:18:26.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:26.350+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.355139+0000) 2019-10-13T08:18:26.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:26.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.355717+0000) 2019-10-13T08:18:26.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:26.350+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.355393+0000) 2019-10-13T08:18:26.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:26.350+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.356699+0000) 2019-10-13T08:18:26.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:26.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.357303+0000) 2019-10-13T08:18:26.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:26.354+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.359014+0000) 2019-10-13T08:18:26.359 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:26.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:26.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:26.354+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.359802+0000) 2019-10-13T08:18:26.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:26.362+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:26.367274+0000) 2019-10-13T08:18:27.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:27.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:27.151225+0000) 2019-10-13T08:18:27.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:27.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:27.441391+0000) 2019-10-13T08:18:27.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:27.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:27.539384+0000) 2019-10-13T08:18:27.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:27.562+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:27.565781+0000) 2019-10-13T08:18:31.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:31.358+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.361317+0000) 2019-10-13T08:18:31.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:31.358+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.362682+0000) 2019-10-13T08:18:31.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:31.358+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.363161+0000) 2019-10-13T08:18:31.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:31.358+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.363407+0000) 2019-10-13T08:18:31.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:31.358+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.364478+0000) 2019-10-13T08:18:31.364 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:31.358+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.364931+0000) 2019-10-13T08:18:31.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:31.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.365364+0000) 2019-10-13T08:18:31.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:31.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:31.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:31.366+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.370979+0000) 2019-10-13T08:18:31.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:31.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.585780+0000) 2019-10-13T08:18:31.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:31.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:31.586854+0000) 2019-10-13T08:18:34.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:34.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:34.311611+0000) 2019-10-13T08:18:37.115 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:18:37.115 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:37.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:34.570+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:34.574976+0000) 2019-10-13T08:18:37.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:36.374+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.377868+0000) 2019-10-13T08:18:37.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:36.366+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.368497+0000) 2019-10-13T08:18:37.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:36.366+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.369798+0000) 2019-10-13T08:18:37.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:36.362+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.369641+0000) 2019-10-13T08:18:37.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:36.366+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.370221+0000) 2019-10-13T08:18:37.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:36.366+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.371346+0000) 2019-10-13T08:18:37.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:36.366+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.371778+0000) 2019-10-13T08:18:37.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:36.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:37.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:36.366+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:36.373669+0000) 2019-10-13T08:18:37.122 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:18:37.122 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:18:37.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:37.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:37.151408+0000) 2019-10-13T08:18:37.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:37.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:37.441589+0000) 2019-10-13T08:18:37.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:37.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:37.539622+0000) 2019-10-13T08:18:37.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:37.562+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:37.565971+0000) 2019-10-13T08:18:41.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:41.370+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.376013+0000) 2019-10-13T08:18:41.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:41.370+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.376795+0000) 2019-10-13T08:18:41.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:41.370+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.377491+0000) 2019-10-13T08:18:41.379 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:41.374+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.379272+0000) 2019-10-13T08:18:41.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:41.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:41.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:41.374+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.379934+0000) 2019-10-13T08:18:41.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:41.374+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.381152+0000) 2019-10-13T08:18:41.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:41.378+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.381780+0000) 2019-10-13T08:18:41.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:41.382+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.387630+0000) 2019-10-13T08:18:41.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:41.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.585914+0000) 2019-10-13T08:18:41.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:41.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:41.587027+0000) 2019-10-13T08:18:44.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:44.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:44.311842+0000) 2019-10-13T08:18:44.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:44.578+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:44.581513+0000) 2019-10-13T08:18:46.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:46.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.382957+0000) 2019-10-13T08:18:46.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:46.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.383955+0000) 2019-10-13T08:18:46.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:46.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.384102+0000) 2019-10-13T08:18:46.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:46.378+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.384734+0000) 2019-10-13T08:18:46.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:46.378+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.385423+0000) 2019-10-13T08:18:46.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:46.378+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.385624+0000) 2019-10-13T08:18:46.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:46.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:46.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:46.382+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.385944+0000) 2019-10-13T08:18:46.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:46.390+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:46.392241+0000) 2019-10-13T08:18:47.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:47.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:47.151596+0000) 2019-10-13T08:18:47.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:47.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:47.441779+0000) 2019-10-13T08:18:47.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:47.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:47.539831+0000) 2019-10-13T08:18:47.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:47.562+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:47.566127+0000) 2019-10-13T08:18:51.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:51.382+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.389323+0000) 2019-10-13T08:18:51.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:51.386+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.390577+0000) 2019-10-13T08:18:51.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:51.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.390728+0000) 2019-10-13T08:18:51.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:51.386+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.391300+0000) 2019-10-13T08:18:51.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:51.386+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.391777+0000) 2019-10-13T08:18:51.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:51.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:51.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:51.386+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.391892+0000) 2019-10-13T08:18:51.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:51.386+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.392720+0000) 2019-10-13T08:18:51.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:51.390+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.395954+0000) 2019-10-13T08:18:51.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:51.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.586077+0000) 2019-10-13T08:18:51.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:51.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:51.587198+0000) 2019-10-13T08:18:54.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:54.306+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:54.312046+0000) 2019-10-13T08:18:54.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:54.582+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:54.588065+0000) 2019-10-13T08:18:56.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:18:56.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.395409+0000) 2019-10-13T08:18:56.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:18:56.390+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.396384+0000) 2019-10-13T08:18:56.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:56.390+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.396988+0000) 2019-10-13T08:18:56.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:18:56.394+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.396862+0000) 2019-10-13T08:18:56.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:56.390+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.397583+0000) 2019-10-13T08:18:56.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:18:56.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:18:56.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:56.394+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.398791+0000) 2019-10-13T08:18:56.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:56.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.399249+0000) 2019-10-13T08:18:56.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:18:56.398+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:56.403808+0000) 2019-10-13T08:18:57.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:18:57.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:57.151778+0000) 2019-10-13T08:18:57.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:18:57.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:57.441949+0000) 2019-10-13T08:18:57.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:18:57.534+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:57.540045+0000) 2019-10-13T08:18:57.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:18:57.562+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:18:57.566293+0000) 2019-10-13T08:19:01.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:01.398+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.401394+0000) 2019-10-13T08:19:01.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:01.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.402031+0000) 2019-10-13T08:19:01.403 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:01.398+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.403369+0000) 2019-10-13T08:19:01.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:01.398+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.403884+0000) 2019-10-13T08:19:01.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:01.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:01.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:01.402+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.406183+0000) 2019-10-13T08:19:01.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:01.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.407164+0000) 2019-10-13T08:19:01.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:01.402+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.407592+0000) 2019-10-13T08:19:01.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:01.406+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.409865+0000) 2019-10-13T08:19:01.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:01.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.586257+0000) 2019-10-13T08:19:01.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:01.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:01.587371+0000) 2019-10-13T08:19:04.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:04.309+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:04.312201+0000) 2019-10-13T08:19:04.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:04.590+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:04.594437+0000) 2019-10-13T08:19:06.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:06.401+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.407643+0000) 2019-10-13T08:19:06.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:06.401+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.408100+0000) 2019-10-13T08:19:06.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:06.401+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.408994+0000) 2019-10-13T08:19:06.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:06.405+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.410618+0000) 2019-10-13T08:19:06.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:06.405+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.410945+0000) 2019-10-13T08:19:06.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:06.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:06.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:06.405+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.412325+0000) 2019-10-13T08:19:06.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:06.409+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.413359+0000) 2019-10-13T08:19:06.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:06.413+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:06.417553+0000) 2019-10-13T08:19:07.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:07.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:07.152006+0000) 2019-10-13T08:19:07.161 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:19:07.161 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:07.165 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:19:07.165 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:07.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:07.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:07.442170+0000) 2019-10-13T08:19:07.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:07.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:07.540242+0000) 2019-10-13T08:19:07.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:07.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:07.566516+0000) 2019-10-13T08:19:11.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:11.409+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.415094+0000) 2019-10-13T08:19:11.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:11.409+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.415441+0000) 2019-10-13T08:19:11.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:11.409+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.415927+0000) 2019-10-13T08:19:11.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:11.409+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.416771+0000) 2019-10-13T08:19:11.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:11.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:11.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:11.409+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.417231+0000) 2019-10-13T08:19:11.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:11.413+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.417812+0000) 2019-10-13T08:19:11.418 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:11.413+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.418331+0000) 2019-10-13T08:19:11.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:11.417+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.423113+0000) 2019-10-13T08:19:11.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:11.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.586457+0000) 2019-10-13T08:19:11.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:11.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:11.587547+0000) 2019-10-13T08:19:14.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:14.309+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:14.312433+0000) 2019-10-13T08:19:14.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:14.597+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:14.600855+0000) 2019-10-13T08:19:16.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:16.417+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.420266+0000) 2019-10-13T08:19:16.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:16.413+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.420979+0000) 2019-10-13T08:19:16.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:16.413+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.421402+0000) 2019-10-13T08:19:16.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:16.417+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.422525+0000) 2019-10-13T08:19:16.423 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:16.417+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.421968+0000) 2019-10-13T08:19:16.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:16.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:16.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:16.417+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.423012+0000) 2019-10-13T08:19:16.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:16.417+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.424659+0000) 2019-10-13T08:19:16.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:16.425+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:16.431843+0000) 2019-10-13T08:19:17.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:17.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:17.152194+0000) 2019-10-13T08:19:17.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:17.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:17.442386+0000) 2019-10-13T08:19:17.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:17.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:17.540433+0000) 2019-10-13T08:19:17.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:17.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:17.566716+0000) 2019-10-13T08:19:21.425 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:21.421+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.425637+0000) 2019-10-13T08:19:21.428 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:21.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:21.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:21.425+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.429113+0000) 2019-10-13T08:19:21.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:21.425+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.429233+0000) 2019-10-13T08:19:21.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:21.425+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.429594+0000) 2019-10-13T08:19:21.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:21.429+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.430004+0000) 2019-10-13T08:19:21.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:21.425+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.429862+0000) 2019-10-13T08:19:21.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:21.429+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.431070+0000) 2019-10-13T08:19:21.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:21.429+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.434574+0000) 2019-10-13T08:19:21.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:21.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.586646+0000) 2019-10-13T08:19:21.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:21.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:21.587713+0000) 2019-10-13T08:19:24.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:24.309+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:24.312638+0000) 2019-10-13T08:19:24.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:24.605+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:24.607386+0000) 2019-10-13T08:19:26.432 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:26.429+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.432061+0000) 2019-10-13T08:19:26.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:26.429+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.432164+0000) 2019-10-13T08:19:26.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:26.429+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.432794+0000) 2019-10-13T08:19:26.433 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:26.429+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.433031+0000) 2019-10-13T08:19:26.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:26.433+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.433778+0000) 2019-10-13T08:19:26.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:26.433+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.434080+0000) 2019-10-13T08:19:26.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:26.433+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:26.434 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:26.429+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.434420+0000) 2019-10-13T08:19:26.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:26.437+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:26.440760+0000) 2019-10-13T08:19:27.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:27.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:27.152379+0000) 2019-10-13T08:19:27.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:27.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:27.442607+0000) 2019-10-13T08:19:27.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:27.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:27.540599+0000) 2019-10-13T08:19:27.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:27.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:27.566920+0000) 2019-10-13T08:19:31.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:31.433+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.437681+0000) 2019-10-13T08:19:31.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:31.437+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.437966+0000) 2019-10-13T08:19:31.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:31.437+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.438482+0000) 2019-10-13T08:19:31.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:31.433+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.438321+0000) 2019-10-13T08:19:31.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:31.437+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.439491+0000) 2019-10-13T08:19:31.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:31.437+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.439859+0000) 2019-10-13T08:19:31.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:31.437+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.440618+0000) 2019-10-13T08:19:31.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:31.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:31.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:31.441+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.447632+0000) 2019-10-13T08:19:31.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:31.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.586820+0000) 2019-10-13T08:19:31.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:31.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:31.587886+0000) 2019-10-13T08:19:34.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:34.309+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:34.312876+0000) 2019-10-13T08:19:34.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:34.613+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:34.613909+0000) 2019-10-13T08:19:36.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:36.445+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.446312+0000) 2019-10-13T08:19:36.447 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:36.445+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.447234+0000) 2019-10-13T08:19:36.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:36.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:36.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:36.445+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.447580+0000) 2019-10-13T08:19:36.448 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:36.445+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.447882+0000) 2019-10-13T08:19:36.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:36.445+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.448661+0000) 2019-10-13T08:19:36.449 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:36.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.448436+0000) 2019-10-13T08:19:36.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:36.445+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.449512+0000) 2019-10-13T08:19:36.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:36.449+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:36.455016+0000) 2019-10-13T08:19:37.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:37.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:37.152565+0000) 2019-10-13T08:19:37.304 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:19:37.304 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:37.307 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:19:37.307 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:19:37.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:37.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:37.442818+0000) 2019-10-13T08:19:37.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:37.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:37.540828+0000) 2019-10-13T08:19:37.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:37.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:37.567153+0000) 2019-10-13T08:19:41.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:41.449+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.449858+0000) 2019-10-13T08:19:44.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:41.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.451411+0000) 2019-10-13T08:19:44.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:41.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.588063+0000) 2019-10-13T08:19:44.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:41.453+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.458571+0000) 2019-10-13T08:19:44.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:41.449+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.450782+0000) 2019-10-13T08:19:44.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:41.449+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.452197+0000) 2019-10-13T08:19:44.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:41.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:44.539 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:41.449+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.453100+0000) 2019-10-13T08:19:44.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:44.309+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:44.313032+0000) 2019-10-13T08:19:44.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:41.449+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.453713+0000) 2019-10-13T08:19:44.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:41.453+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.454161+0000) 2019-10-13T08:19:44.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:41.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:41.586954+0000) 2019-10-13T08:19:44.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:44.621+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:44.624677+0000) 2019-10-13T08:19:46.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:46.453+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.456145+0000) 2019-10-13T08:19:46.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:46.453+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.456342+0000) 2019-10-13T08:19:46.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:46.453+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.456706+0000) 2019-10-13T08:19:46.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:46.457+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.457860+0000) 2019-10-13T08:19:46.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:46.453+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.457774+0000) 2019-10-13T08:19:46.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:46.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70409 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:46.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:46.457+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.458493+0000) 2019-10-13T08:19:46.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:46.457+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.458923+0000) 2019-10-13T08:19:46.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:46.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:46.462723+0000) 2019-10-13T08:19:47.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:47.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:47.152752+0000) 2019-10-13T08:19:47.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:47.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:47.443028+0000) 2019-10-13T08:19:47.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:47.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:47.541028+0000) 2019-10-13T08:19:47.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:47.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:47.567353+0000) 2019-10-13T08:19:51.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:51.457+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.463369+0000) 2019-10-13T08:19:51.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:51.461+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.462176+0000) 2019-10-13T08:19:51.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:51.461+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.462797+0000) 2019-10-13T08:19:51.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:51.461+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.463191+0000) 2019-10-13T08:19:51.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:51.461+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.463451+0000) 2019-10-13T08:19:51.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:51.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:51.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:51.461+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.465257+0000) 2019-10-13T08:19:51.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:51.465+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.466188+0000) 2019-10-13T08:19:51.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:51.469+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.474261+0000) 2019-10-13T08:19:51.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:51.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.587131+0000) 2019-10-13T08:19:51.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:51.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:51.588264+0000) 2019-10-13T08:19:54.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:54.313+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:54.313235+0000) 2019-10-13T08:19:54.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:54.625+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:54.631146+0000) 2019-10-13T08:19:56.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:56.465+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.467718+0000) 2019-10-13T08:19:56.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:19:56.465+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.468182+0000) 2019-10-13T08:19:56.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:19:56.465+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.468914+0000) 2019-10-13T08:19:56.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:19:56.465+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.469192+0000) 2019-10-13T08:19:56.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:56.469+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.470463+0000) 2019-10-13T08:19:56.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:56.469+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.470601+0000) 2019-10-13T08:19:56.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:19:56.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:19:56.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:56.469+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.471596+0000) 2019-10-13T08:19:56.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:19:56.473+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:56.476443+0000) 2019-10-13T08:19:57.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:19:57.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:57.152940+0000) 2019-10-13T08:19:57.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:19:57.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:57.443237+0000) 2019-10-13T08:19:57.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:19:57.537+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:57.541230+0000) 2019-10-13T08:19:57.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:19:57.565+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:19:57.567555+0000) 2019-10-13T08:20:01.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:01.472+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.474321+0000) 2019-10-13T08:20:01.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:01.472+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.474653+0000) 2019-10-13T08:20:01.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:01.472+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.475372+0000) 2019-10-13T08:20:01.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:01.469+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.475532+0000) 2019-10-13T08:20:01.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:01.472+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.476479+0000) 2019-10-13T08:20:01.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:01.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:01.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:01.472+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.477037+0000) 2019-10-13T08:20:01.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:01.472+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.477567+0000) 2019-10-13T08:20:01.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:01.481+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.485037+0000) 2019-10-13T08:20:01.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:01.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.587309+0000) 2019-10-13T08:20:01.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:01.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:01.588391+0000) 2019-10-13T08:20:04.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:04.308+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:04.313438+0000) 2019-10-13T08:20:04.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:04.637+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:04.637745+0000) 2019-10-13T08:20:06.479 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:06.473+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.479228+0000) 2019-10-13T08:20:06.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:06.476+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.481332+0000) 2019-10-13T08:20:06.482 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:06.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:06.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:06.480+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.483081+0000) 2019-10-13T08:20:06.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:06.480+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.483547+0000) 2019-10-13T08:20:06.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:06.480+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.484091+0000) 2019-10-13T08:20:06.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:06.480+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.484529+0000) 2019-10-13T08:20:06.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:06.480+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.484843+0000) 2019-10-13T08:20:06.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:06.481+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:06.487397+0000) 2019-10-13T08:20:07.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:07.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:07.153126+0000) 2019-10-13T08:20:07.411 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:20:07.411 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:07.442 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:20:07.442 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:07.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:07.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:07.443406+0000) 2019-10-13T08:20:07.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:07.536+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:07.541399+0000) 2019-10-13T08:20:07.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:07.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:07.567803+0000) 2019-10-13T08:20:11.486 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:11.481+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.485582+0000) 2019-10-13T08:20:11.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:11.484+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.486614+0000) 2019-10-13T08:20:11.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:11.484+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.487303+0000) 2019-10-13T08:20:11.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:11.484+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.487632+0000) 2019-10-13T08:20:11.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:11.484+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.487875+0000) 2019-10-13T08:20:11.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:11.484+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:11.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:11.484+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.488836+0000) 2019-10-13T08:20:11.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:11.484+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.488699+0000) 2019-10-13T08:20:11.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:11.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.493474+0000) 2019-10-13T08:20:11.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:11.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.587455+0000) 2019-10-13T08:20:11.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:11.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:11.588573+0000) 2019-10-13T08:20:14.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:14.308+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:14.313659+0000) 2019-10-13T08:20:14.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:14.640+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:14.644131+0000) 2019-10-13T08:20:16.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:16.488+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.491081+0000) 2019-10-13T08:20:16.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:16.488+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.491603+0000) 2019-10-13T08:20:16.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:16.488+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.492092+0000) 2019-10-13T08:20:16.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:16.488+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.492637+0000) 2019-10-13T08:20:16.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:16.488+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.493151+0000) 2019-10-13T08:20:16.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:16.488+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.493057+0000) 2019-10-13T08:20:16.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:16.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:16.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:16.492+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.495123+0000) 2019-10-13T08:20:16.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:16.492+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:16.499579+0000) 2019-10-13T08:20:17.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:17.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:17.153333+0000) 2019-10-13T08:20:17.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:17.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:17.443597+0000) 2019-10-13T08:20:17.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:17.536+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:17.541625+0000) 2019-10-13T08:20:17.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:17.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:17.567998+0000) 2019-10-13T08:20:21.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:21.492+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.496639+0000) 2019-10-13T08:20:21.499 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:21.496+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.499260+0000) 2019-10-13T08:20:21.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:21.496+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.499315+0000) 2019-10-13T08:20:21.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:21.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:21.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:21.496+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.500008+0000) 2019-10-13T08:20:21.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:21.496+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.500269+0000) 2019-10-13T08:20:21.501 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:21.496+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.500425+0000) 2019-10-13T08:20:21.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:21.500+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.502678+0000) 2019-10-13T08:20:21.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:21.504+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.508644+0000) 2019-10-13T08:20:21.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:21.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.587659+0000) 2019-10-13T08:20:21.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:21.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:21.588763+0000) 2019-10-13T08:20:24.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:24.312+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:24.313853+0000) 2019-10-13T08:20:24.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:24.648+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:24.650697+0000) 2019-10-13T08:20:26.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:26.500+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.503083+0000) 2019-10-13T08:20:26.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:26.500+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.504356+0000) 2019-10-13T08:20:26.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:26.500+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.505283+0000) 2019-10-13T08:20:26.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:26.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:26.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:26.504+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.506635+0000) 2019-10-13T08:20:26.507 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:26.504+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.507346+0000) 2019-10-13T08:20:26.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:26.504+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.507709+0000) 2019-10-13T08:20:26.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:26.504+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.508152+0000) 2019-10-13T08:20:26.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:26.504+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:26.511271+0000) 2019-10-13T08:20:27.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:27.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:27.153561+0000) 2019-10-13T08:20:27.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:27.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:27.443776+0000) 2019-10-13T08:20:27.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:27.540+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:27.541850+0000) 2019-10-13T08:20:27.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:27.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:27.568198+0000) 2019-10-13T08:20:31.510 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:31.508+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.510091+0000) 2019-10-13T08:20:31.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:31.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.510531+0000) 2019-10-13T08:20:31.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:31.508+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.510558+0000) 2019-10-13T08:20:31.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:31.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70679 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:31.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:31.508+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.511606+0000) 2019-10-13T08:20:31.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:31.508+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.512444+0000) 2019-10-13T08:20:31.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:31.508+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.513303+0000) 2019-10-13T08:20:31.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:31.508+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.513481+0000) 2019-10-13T08:20:31.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:31.512+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.518790+0000) 2019-10-13T08:20:31.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:31.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.587867+0000) 2019-10-13T08:20:31.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:31.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:31.588934+0000) 2019-10-13T08:20:34.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:34.312+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:34.314032+0000) 2019-10-13T08:20:34.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:34.656+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:34.657199+0000) 2019-10-13T08:20:36.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:36.512+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.514218+0000) 2019-10-13T08:20:36.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:36.508+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.514117+0000) 2019-10-13T08:20:36.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:36.512+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.514983+0000) 2019-10-13T08:20:36.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:36.512+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.516207+0000) 2019-10-13T08:20:36.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:36.512+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.515906+0000) 2019-10-13T08:20:36.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:36.512+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.516479+0000) 2019-10-13T08:20:36.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:36.512+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.517497+0000) 2019-10-13T08:20:36.518 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:36.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:36.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:36.520+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:36.524621+0000) 2019-10-13T08:20:37.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:37.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:37.153775+0000) 2019-10-13T08:20:37.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:37.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:37.443904+0000) 2019-10-13T08:20:37.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:37.540+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:37.542050+0000) 2019-10-13T08:20:37.554 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:20:37.554 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:37.558 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:20:37.558 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:20:37.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:37.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:37.568399+0000) 2019-10-13T08:20:41.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:41.520+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.521798+0000) 2019-10-13T08:20:41.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:41.520+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.522078+0000) 2019-10-13T08:20:41.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:41.520+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.523258+0000) 2019-10-13T08:20:41.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:41.520+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.523887+0000) 2019-10-13T08:20:41.524 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:41.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:41.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:41.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.524911+0000) 2019-10-13T08:20:41.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:41.524+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.527175+0000) 2019-10-13T08:20:41.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:41.524+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.526626+0000) 2019-10-13T08:20:41.533 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:41.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.532732+0000) 2019-10-13T08:20:41.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:41.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.588049+0000) 2019-10-13T08:20:41.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:41.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:41.589125+0000) 2019-10-13T08:20:44.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:44.312+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:44.314230+0000) 2019-10-13T08:20:44.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:44.656+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:44.663669+0000) 2019-10-13T08:20:46.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:46.524+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.528003+0000) 2019-10-13T08:20:46.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:46.524+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.528671+0000) 2019-10-13T08:20:46.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:46.524+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.528923+0000) 2019-10-13T08:20:46.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:46.524+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.529289+0000) 2019-10-13T08:20:46.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:46.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:46.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:46.528+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.530648+0000) 2019-10-13T08:20:46.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:46.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.530965+0000) 2019-10-13T08:20:46.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:46.528+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.532119+0000) 2019-10-13T08:20:46.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:46.532+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:46.539495+0000) 2019-10-13T08:20:47.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:47.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:47.153985+0000) 2019-10-13T08:20:47.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:47.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:47.444038+0000) 2019-10-13T08:20:47.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:47.540+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:47.542296+0000) 2019-10-13T08:20:47.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:47.564+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:47.568621+0000) 2019-10-13T08:20:51.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:51.528+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.533541+0000) 2019-10-13T08:20:51.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:51.532+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.533998+0000) 2019-10-13T08:20:51.534 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:51.532+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.534958+0000) 2019-10-13T08:20:51.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:51.532+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.535289+0000) 2019-10-13T08:20:51.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:51.528+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.535133+0000) 2019-10-13T08:20:51.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:51.532+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:51.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:51.532+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.536638+0000) 2019-10-13T08:20:51.539 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:51.532+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.537469+0000) 2019-10-13T08:20:51.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:51.536+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.542638+0000) 2019-10-13T08:20:51.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:51.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.588221+0000) 2019-10-13T08:20:51.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:51.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:51.589334+0000) 2019-10-13T08:20:54.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:54.312+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:54.314473+0000) 2019-10-13T08:20:54.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:54.668+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:54.670016+0000) 2019-10-13T08:20:56.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:56.535+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.539367+0000) 2019-10-13T08:20:56.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:20:56.532+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.539888+0000) 2019-10-13T08:20:56.540 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:56.535+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.540789+0000) 2019-10-13T08:20:56.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:20:56.535+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.541448+0000) 2019-10-13T08:20:56.541 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:20:56.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:20:56.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:56.539+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.541819+0000) 2019-10-13T08:20:56.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:20:56.539+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.542170+0000) 2019-10-13T08:20:56.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:56.539+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.542533+0000) 2019-10-13T08:20:56.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:20:56.540+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:56.545656+0000) 2019-10-13T08:20:57.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:20:57.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:57.154163+0000) 2019-10-13T08:20:57.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:20:57.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:57.444197+0000) 2019-10-13T08:20:57.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:20:57.539+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:57.542521+0000) 2019-10-13T08:20:57.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:20:57.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:20:57.568832+0000) 2019-10-13T08:21:01.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:01.539+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.545373+0000) 2019-10-13T08:21:01.545 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:01.543+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.545944+0000) 2019-10-13T08:21:01.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:01.543+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.548172+0000) 2019-10-13T08:21:01.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:01.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:01.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:01.544+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.548419+0000) 2019-10-13T08:21:01.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:01.543+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.549407+0000) 2019-10-13T08:21:01.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:01.547+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.550459+0000) 2019-10-13T08:21:01.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:01.547+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.550064+0000) 2019-10-13T08:21:01.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:01.548+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.553541+0000) 2019-10-13T08:21:01.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:01.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.588388+0000) 2019-10-13T08:21:01.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:01.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:01.589474+0000) 2019-10-13T08:21:04.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:04.311+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:04.314709+0000) 2019-10-13T08:21:04.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:04.672+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:04.676540+0000) 2019-10-13T08:21:06.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:06.544+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.550597+0000) 2019-10-13T08:21:06.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:06.547+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.551687+0000) 2019-10-13T08:21:06.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:06.547+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.551797+0000) 2019-10-13T08:21:06.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:06.547+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.552743+0000) 2019-10-13T08:21:06.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:06.547+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.553275+0000) 2019-10-13T08:21:06.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:06.547+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.553533+0000) 2019-10-13T08:21:06.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:06.551+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.554880+0000) 2019-10-13T08:21:06.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:06.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:06.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:06.552+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:06.559780+0000) 2019-10-13T08:21:07.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:07.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:07.154391+0000) 2019-10-13T08:21:07.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:07.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:07.444364+0000) 2019-10-13T08:21:07.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:07.539+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:07.542697+0000) 2019-10-13T08:21:07.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:07.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:07.569050+0000) 2019-10-13T08:21:07.650 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:21:07.651 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:07.654 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:21:07.655 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:11.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:11.555+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.559040+0000) 2019-10-13T08:21:11.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:11.555+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.559487+0000) 2019-10-13T08:21:11.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:11.555+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.559660+0000) 2019-10-13T08:21:11.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:11.555+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.560381+0000) 2019-10-13T08:21:11.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:11.556+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.560276+0000) 2019-10-13T08:21:11.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:11.555+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.561188+0000) 2019-10-13T08:21:11.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:11.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:11.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:11.559+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.561901+0000) 2019-10-13T08:21:11.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:11.568+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.568294+0000) 2019-10-13T08:21:11.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:11.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.588562+0000) 2019-10-13T08:21:11.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:11.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:11.589629+0000) 2019-10-13T08:21:14.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:14.311+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:14.314910+0000) 2019-10-13T08:21:14.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:14.680+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:14.683076+0000) 2019-10-13T08:21:16.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:16.563+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.566184+0000) 2019-10-13T08:21:16.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:16.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.566148+0000) 2019-10-13T08:21:16.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:16.563+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.567776+0000) 2019-10-13T08:21:16.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:16.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:16.568 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:16.563+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.568504+0000) 2019-10-13T08:21:16.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:16.563+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.568607+0000) 2019-10-13T08:21:16.569 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:16.563+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.569071+0000) 2019-10-13T08:21:16.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:16.563+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.569522+0000) 2019-10-13T08:21:16.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:16.572+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:16.573085+0000) 2019-10-13T08:21:17.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:17.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:17.154570+0000) 2019-10-13T08:21:17.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:17.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:17.444540+0000) 2019-10-13T08:21:17.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:17.539+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:17.542903+0000) 2019-10-13T08:21:17.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:17.567+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:17.569260+0000) 2019-10-13T08:21:21.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:21.567+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.571174+0000) 2019-10-13T08:21:21.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:21.567+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.571371+0000) 2019-10-13T08:21:21.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:21.567+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.572080+0000) 2019-10-13T08:21:21.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:21.567+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.571772+0000) 2019-10-13T08:21:21.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:21.567+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.572879+0000) 2019-10-13T08:21:21.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:21.567+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.573524+0000) 2019-10-13T08:21:21.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:21.571+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:21.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:21.571+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.574726+0000) 2019-10-13T08:21:21.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:21.575+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.579896+0000) 2019-10-13T08:21:21.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:21.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.588711+0000) 2019-10-13T08:21:21.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:21.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:21.589797+0000) 2019-10-13T08:21:24.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:24.311+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:24.315141+0000) 2019-10-13T08:21:24.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:24.687+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:24.689532+0000) 2019-10-13T08:21:26.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:26.571+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.577257+0000) 2019-10-13T08:21:26.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:26.575+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.577151+0000) 2019-10-13T08:21:26.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:26.575+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.578582+0000) 2019-10-13T08:21:26.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:26.575+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.579366+0000) 2019-10-13T08:21:26.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:26.575+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.579845+0000) 2019-10-13T08:21:26.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:26.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:26.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:26.575+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.580318+0000) 2019-10-13T08:21:26.581 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:26.575+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.580991+0000) 2019-10-13T08:21:26.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:26.583+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:26.584932+0000) 2019-10-13T08:21:27.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:27.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.154742+0000) 2019-10-13T08:21:27.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:27.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.444668+0000) 2019-10-13T08:21:27.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:27.539+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.543108+0000) 2019-10-13T08:21:27.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:27.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:27.569482+0000) 2019-10-13T08:21:31.583 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:31.579+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.583308+0000) 2019-10-13T08:21:31.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:31.579+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.583872+0000) 2019-10-13T08:21:31.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:31.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.585019+0000) 2019-10-13T08:21:31.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:31.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:31.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:31.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.585982+0000) 2019-10-13T08:21:31.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:31.583+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.586379+0000) 2019-10-13T08:21:31.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:31.583+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.586829+0000) 2019-10-13T08:21:31.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:31.583+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.587797+0000) 2019-10-13T08:21:31.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:31.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.588835+0000) 2019-10-13T08:21:31.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:31.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.589927+0000) 2019-10-13T08:21:31.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:31.587+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:31.589825+0000) 2019-10-13T08:21:34.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:34.311+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:34.315368+0000) 2019-10-13T08:21:34.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:34.691+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:34.695843+0000) 2019-10-13T08:21:36.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:36.587+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.589727+0000) 2019-10-13T08:21:36.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:36.587+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.590538+0000) 2019-10-13T08:21:36.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:36.587+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.590484+0000) 2019-10-13T08:21:36.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:36.587+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.591112+0000) 2019-10-13T08:21:36.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:36.587+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:36.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:36.587+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.592865+0000) 2019-10-13T08:21:36.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:36.591+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.592194+0000) 2019-10-13T08:21:36.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:36.587+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.593642+0000) 2019-10-13T08:21:36.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:36.591+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:36.595714+0000) 2019-10-13T08:21:37.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:37.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.154950+0000) 2019-10-13T08:21:37.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:37.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.444846+0000) 2019-10-13T08:21:37.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:37.539+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.543282+0000) 2019-10-13T08:21:37.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:37.563+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:37.569658+0000) 2019-10-13T08:21:37.847 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:21:37.847 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:37.851 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:21:37.851 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:21:41.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:41.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.589504+0000) 2019-10-13T08:21:41.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:41.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.590811+0000) 2019-10-13T08:21:41.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:41.591+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.596038+0000) 2019-10-13T08:21:41.596 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:41.591+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.596753+0000) 2019-10-13T08:21:41.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:41.591+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.596925+0000) 2019-10-13T08:21:41.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:41.595+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.597827+0000) 2019-10-13T08:21:41.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:41.595+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.598288+0000) 2019-10-13T08:21:41.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:41.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:41.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:41.595+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.598995+0000) 2019-10-13T08:21:41.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:41.595+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.599356+0000) 2019-10-13T08:21:41.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:41.599+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:41.603642+0000) 2019-10-13T08:21:44.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:44.311+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:44.315568+0000) 2019-10-13T08:21:44.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:44.699+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:44.702339+0000) 2019-10-13T08:21:46.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:46.599+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.603776+0000) 2019-10-13T08:21:46.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:46.599+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.604493+0000) 2019-10-13T08:21:46.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:46.599+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.605077+0000) 2019-10-13T08:21:46.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:46.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:46.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:46.603+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.605810+0000) 2019-10-13T08:21:46.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:46.603+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.606595+0000) 2019-10-13T08:21:46.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:46.603+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.607151+0000) 2019-10-13T08:21:46.607 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:46.603+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.607517+0000) 2019-10-13T08:21:46.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:46.607+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:46.611757+0000) 2019-10-13T08:21:47.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:47.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.155161+0000) 2019-10-13T08:21:47.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:47.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.445018+0000) 2019-10-13T08:21:47.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:47.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.543452+0000) 2019-10-13T08:21:47.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:47.567+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:47.569832+0000) 2019-10-13T08:21:51.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:51.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.589716+0000) 2019-10-13T08:21:51.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:51.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.591021+0000) 2019-10-13T08:21:51.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:51.603+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.608503+0000) 2019-10-13T08:21:51.609 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:51.603+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.609217+0000) 2019-10-13T08:21:51.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:51.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.609955+0000) 2019-10-13T08:21:51.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:51.607+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.610157+0000) 2019-10-13T08:21:51.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:51.607+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.610625+0000) 2019-10-13T08:21:51.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:51.607+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.611478+0000) 2019-10-13T08:21:51.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:51.607+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.611310+0000) 2019-10-13T08:21:51.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:51.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:51.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:51.615+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:51.616745+0000) 2019-10-13T08:21:54.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:54.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:54.315763+0000) 2019-10-13T08:21:54.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:54.707+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:54.708912+0000) 2019-10-13T08:21:56.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:21:56.610+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.615507+0000) 2019-10-13T08:21:56.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:56.610+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.616190+0000) 2019-10-13T08:21:56.617 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:21:56.614+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.617884+0000) 2019-10-13T08:21:56.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:21:56.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:21:56.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:56.614+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.618549+0000) 2019-10-13T08:21:56.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:56.614+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.618769+0000) 2019-10-13T08:21:56.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:56.614+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.619375+0000) 2019-10-13T08:21:56.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:21:56.615+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.619272+0000) 2019-10-13T08:21:56.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:21:56.623+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:56.625601+0000) 2019-10-13T08:21:57.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:21:57.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.155296+0000) 2019-10-13T08:21:57.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:21:57.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.445149+0000) 2019-10-13T08:21:57.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:21:57.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.543668+0000) 2019-10-13T08:21:57.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:21:57.566+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:21:57.570007+0000) 2019-10-13T08:22:01.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:01.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.589981+0000) 2019-10-13T08:22:01.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:01.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.591183+0000) 2019-10-13T08:22:01.622 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:01.618+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.621892+0000) 2019-10-13T08:22:01.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:01.618+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.622168+0000) 2019-10-13T08:22:01.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:01.618+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.622970+0000) 2019-10-13T08:22:01.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:01.618+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.623552+0000) 2019-10-13T08:22:01.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:01.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:01.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:01.618+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.624170+0000) 2019-10-13T08:22:01.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:01.623+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.624530+0000) 2019-10-13T08:22:01.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:01.618+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.625428+0000) 2019-10-13T08:22:01.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:01.627+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:01.629495+0000) 2019-10-13T08:22:04.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:04.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:04.315965+0000) 2019-10-13T08:22:04.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:04.711+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:04.715268+0000) 2019-10-13T08:22:06.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:06.622+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.627364+0000) 2019-10-13T08:22:06.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:06.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.628188+0000) 2019-10-13T08:22:06.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:06.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.628014+0000) 2019-10-13T08:22:06.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:06.622+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.629098+0000) 2019-10-13T08:22:06.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:06.622+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.629492+0000) 2019-10-13T08:22:06.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:06.626+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.629782+0000) 2019-10-13T08:22:06.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:06.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:06.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:06.626+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.630964+0000) 2019-10-13T08:22:06.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:06.635+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:06.636384+0000) 2019-10-13T08:22:07.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:07.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.155471+0000) 2019-10-13T08:22:07.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:07.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.445289+0000) 2019-10-13T08:22:07.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:07.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.543817+0000) 2019-10-13T08:22:07.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:07.566+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:07.570185+0000) 2019-10-13T08:22:07.971 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:22:07.971 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:07.980 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:22:07.980 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:11.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:11.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.590182+0000) 2019-10-13T08:22:11.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:11.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.591399+0000) 2019-10-13T08:22:11.634 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:11.630+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.634374+0000) 2019-10-13T08:22:11.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:11.630+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.634567+0000) 2019-10-13T08:22:11.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:11.631+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.634909+0000) 2019-10-13T08:22:11.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:11.630+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.636074+0000) 2019-10-13T08:22:11.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:11.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:11.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:11.630+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.636879+0000) 2019-10-13T08:22:11.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:11.630+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.637109+0000) 2019-10-13T08:22:11.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:11.634+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.638375+0000) 2019-10-13T08:22:11.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:11.639+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:11.641982+0000) 2019-10-13T08:22:14.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:14.314+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:14.316232+0000) 2019-10-13T08:22:14.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:14.719+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:14.721681+0000) 2019-10-13T08:22:16.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:16.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.639559+0000) 2019-10-13T08:22:16.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:16.634+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.641037+0000) 2019-10-13T08:22:16.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:16.634+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.641426+0000) 2019-10-13T08:22:16.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:16.638+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.642935+0000) 2019-10-13T08:22:16.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:16.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:16.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:16.638+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.643579+0000) 2019-10-13T08:22:16.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:16.638+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.644115+0000) 2019-10-13T08:22:16.644 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:16.638+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.644409+0000) 2019-10-13T08:22:16.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:16.643+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:16.646329+0000) 2019-10-13T08:22:17.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:17.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.155657+0000) 2019-10-13T08:22:17.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:17.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.445461+0000) 2019-10-13T08:22:17.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:17.542+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.543968+0000) 2019-10-13T08:22:17.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:17.566+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:17.570364+0000) 2019-10-13T08:22:21.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:21.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.590374+0000) 2019-10-13T08:22:21.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:21.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.591614+0000) 2019-10-13T08:22:21.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:21.642+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.646588+0000) 2019-10-13T08:22:21.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:21.642+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.648453+0000) 2019-10-13T08:22:21.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:21.642+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.648258+0000) 2019-10-13T08:22:21.649 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:21.642+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:21.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:21.642+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.649282+0000) 2019-10-13T08:22:21.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:21.647+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.648529+0000) 2019-10-13T08:22:21.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:21.642+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.649664+0000) 2019-10-13T08:22:21.651 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:21.646+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.650841+0000) 2019-10-13T08:22:21.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:21.655+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:21.656279+0000) 2019-10-13T08:22:24.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:24.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:24.316467+0000) 2019-10-13T08:22:24.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:24.730+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:24.732719+0000) 2019-10-13T08:22:26.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:26.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.651875+0000) 2019-10-13T08:22:26.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:26.646+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.653208+0000) 2019-10-13T08:22:26.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:26.650+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.654922+0000) 2019-10-13T08:22:26.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:26.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:26.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:26.650+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.656206+0000) 2019-10-13T08:22:26.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:26.650+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.656713+0000) 2019-10-13T08:22:26.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:26.650+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.657015+0000) 2019-10-13T08:22:26.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:26.650+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.657535+0000) 2019-10-13T08:22:26.668 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:26.658+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:26.661755+0000) 2019-10-13T08:22:27.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:27.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.155851+0000) 2019-10-13T08:22:27.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:27.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.445614+0000) 2019-10-13T08:22:27.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:27.542+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.544146+0000) 2019-10-13T08:22:27.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:27.566+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:27.570501+0000) 2019-10-13T08:22:31.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:31.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.590567+0000) 2019-10-13T08:22:31.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:31.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.591813+0000) 2019-10-13T08:22:31.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:31.654+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.658999+0000) 2019-10-13T08:22:31.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:31.654+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.659696+0000) 2019-10-13T08:22:31.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:31.654+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.661012+0000) 2019-10-13T08:22:31.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:31.654+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.660618+0000) 2019-10-13T08:22:31.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:31.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:31.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:31.654+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.661441+0000) 2019-10-13T08:22:31.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:31.658+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.661135+0000) 2019-10-13T08:22:31.662 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:31.658+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.662898+0000) 2019-10-13T08:22:31.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:31.666+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:31.670445+0000) 2019-10-13T08:22:34.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:34.314+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:34.316678+0000) 2019-10-13T08:22:34.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:34.734+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:34.739174+0000) 2019-10-13T08:22:36.666 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:36.662+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.666087+0000) 2019-10-13T08:22:36.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:36.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.666848+0000) 2019-10-13T08:22:36.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:36.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:36.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:36.662+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.666876+0000) 2019-10-13T08:22:36.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:36.662+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.668022+0000) 2019-10-13T08:22:36.668 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:36.662+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.668151+0000) 2019-10-13T08:22:36.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:36.662+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.668910+0000) 2019-10-13T08:22:36.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:36.662+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.669159+0000) 2019-10-13T08:22:36.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:36.670+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:36.675922+0000) 2019-10-13T08:22:37.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:37.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.156071+0000) 2019-10-13T08:22:37.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:37.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.445770+0000) 2019-10-13T08:22:37.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:37.538+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.544323+0000) 2019-10-13T08:22:37.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:37.570+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:37.570681+0000) 2019-10-13T08:22:38.145 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:22:38.145 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:38.149 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:22:38.150 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:22:41.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:41.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.590762+0000) 2019-10-13T08:22:41.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:41.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.592010+0000) 2019-10-13T08:22:41.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:41.666+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.670334+0000) 2019-10-13T08:22:41.672 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:41.666+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.672418+0000) 2019-10-13T08:22:41.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:41.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:41.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:41.670+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.673393+0000) 2019-10-13T08:22:41.675 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:41.670+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.675167+0000) 2019-10-13T08:22:41.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:41.670+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.675680+0000) 2019-10-13T08:22:41.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:41.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.676725+0000) 2019-10-13T08:22:41.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:41.670+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.677194+0000) 2019-10-13T08:22:41.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:41.678+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:41.681829+0000) 2019-10-13T08:22:44.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:44.310+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:44.316876+0000) 2019-10-13T08:22:44.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:44.742+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:44.745592+0000) 2019-10-13T08:22:46.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:46.674+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.678679+0000) 2019-10-13T08:22:46.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:46.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:46.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:46.674+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.679130+0000) 2019-10-13T08:22:46.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:46.674+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.679326+0000) 2019-10-13T08:22:46.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:46.674+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.680545+0000) 2019-10-13T08:22:46.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:46.674+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.681410+0000) 2019-10-13T08:22:46.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:46.674+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.680961+0000) 2019-10-13T08:22:46.682 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:46.678+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.682209+0000) 2019-10-13T08:22:46.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:46.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:46.682962+0000) 2019-10-13T08:22:47.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:47.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.156260+0000) 2019-10-13T08:22:47.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:47.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.445908+0000) 2019-10-13T08:22:47.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:47.542+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.544497+0000) 2019-10-13T08:22:47.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:47.566+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:47.570880+0000) 2019-10-13T08:22:51.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:51.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.590995+0000) 2019-10-13T08:22:51.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:51.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.592263+0000) 2019-10-13T08:22:51.682 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:51.677+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.682313+0000) 2019-10-13T08:22:51.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:51.677+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.683326+0000) 2019-10-13T08:22:51.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:51.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.683807+0000) 2019-10-13T08:22:51.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:51.677+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.684343+0000) 2019-10-13T08:22:51.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:51.677+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:51.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:51.677+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.685218+0000) 2019-10-13T08:22:51.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:51.682+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.685225+0000) 2019-10-13T08:22:51.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:51.681+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.686766+0000) 2019-10-13T08:22:51.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:51.690+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:51.692913+0000) 2019-10-13T08:22:54.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:54.313+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:54.317107+0000) 2019-10-13T08:22:54.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:54.750+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:54.751959+0000) 2019-10-13T08:22:56.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:56.681+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.688520+0000) 2019-10-13T08:22:56.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:22:56.685+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.690518+0000) 2019-10-13T08:22:56.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:22:56.685+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.690999+0000) 2019-10-13T08:22:56.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:22:56.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71542 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:22:56.691 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:56.685+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.691895+0000) 2019-10-13T08:22:56.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:56.685+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.692450+0000) 2019-10-13T08:22:56.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:56.685+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.692839+0000) 2019-10-13T08:22:56.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:22:56.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.693209+0000) 2019-10-13T08:22:56.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:22:56.694+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:56.697396+0000) 2019-10-13T08:22:57.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:22:57.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.156447+0000) 2019-10-13T08:22:57.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:22:57.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.446044+0000) 2019-10-13T08:22:57.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:22:57.541+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.544677+0000) 2019-10-13T08:22:57.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:22:57.569+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:22:57.571062+0000) 2019-10-13T08:23:01.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:01.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.591196+0000) 2019-10-13T08:23:01.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:01.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.592463+0000) 2019-10-13T08:23:01.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:01.689+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.694565+0000) 2019-10-13T08:23:01.695 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:01.689+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.694983+0000) 2019-10-13T08:23:01.696 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:01.689+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.696505+0000) 2019-10-13T08:23:01.696 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:01.689+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.696715+0000) 2019-10-13T08:23:01.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:01.689+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:01.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:01.694+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.697138+0000) 2019-10-13T08:23:01.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:01.693+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.697899+0000) 2019-10-13T08:23:01.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:01.693+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.698281+0000) 2019-10-13T08:23:01.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:01.702+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:01.705497+0000) 2019-10-13T08:23:04.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:04.313+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:04.317341+0000) 2019-10-13T08:23:04.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:04.762+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:04.763038+0000) 2019-10-13T08:23:06.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:06.697+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.701152+0000) 2019-10-13T08:23:06.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:06.697+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.701606+0000) 2019-10-13T08:23:06.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:06.701+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.702355+0000) 2019-10-13T08:23:06.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:06.701+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.703135+0000) 2019-10-13T08:23:06.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:06.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:06.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:06.701+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.703905+0000) 2019-10-13T08:23:06.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:06.698+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.703790+0000) 2019-10-13T08:23:06.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:06.701+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.705247+0000) 2019-10-13T08:23:06.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:06.706+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:06.708606+0000) 2019-10-13T08:23:07.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:07.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:07.156593+0000) 2019-10-13T08:23:07.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:07.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:07.446239+0000) 2019-10-13T08:23:07.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:07.541+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:07.544826+0000) 2019-10-13T08:23:07.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:07.569+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:07.571238+0000) 2019-10-13T08:23:08.269 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:23:08.269 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:08.273 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:23:08.273 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:11.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:11.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.591425+0000) 2019-10-13T08:23:11.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:11.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.592659+0000) 2019-10-13T08:23:11.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:11.709+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.709812+0000) 2019-10-13T08:23:11.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:11.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:11.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:11.706+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.709436+0000) 2019-10-13T08:23:11.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:11.709+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.710531+0000) 2019-10-13T08:23:11.711 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:11.709+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.711357+0000) 2019-10-13T08:23:11.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:11.709+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.710770+0000) 2019-10-13T08:23:11.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:11.709+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.711506+0000) 2019-10-13T08:23:11.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:11.709+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.712646+0000) 2019-10-13T08:23:11.713 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:11.710+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:11.713499+0000) 2019-10-13T08:23:14.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:14.313+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:14.317543+0000) 2019-10-13T08:23:14.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:14.766+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:14.769532+0000) 2019-10-13T08:23:16.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:16.713+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.713844+0000) 2019-10-13T08:23:16.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:16.713+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.716313+0000) 2019-10-13T08:23:16.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:16.713+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.716316+0000) 2019-10-13T08:23:16.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:16.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:16.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:16.713+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.717069+0000) 2019-10-13T08:23:16.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:16.713+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.717690+0000) 2019-10-13T08:23:16.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:16.714+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.717748+0000) 2019-10-13T08:23:16.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:16.713+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.717389+0000) 2019-10-13T08:23:16.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:16.722+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:16.725158+0000) 2019-10-13T08:23:17.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:17.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:17.156781+0000) 2019-10-13T08:23:17.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:17.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:17.446429+0000) 2019-10-13T08:23:17.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:17.541+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:17.544992+0000) 2019-10-13T08:23:17.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:17.569+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:17.571416+0000) 2019-10-13T08:23:21.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:21.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.591668+0000) 2019-10-13T08:23:21.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:21.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.592917+0000) 2019-10-13T08:23:21.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:21.717+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.719873+0000) 2019-10-13T08:23:21.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:21.717+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.720340+0000) 2019-10-13T08:23:21.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:21.717+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.721419+0000) 2019-10-13T08:23:21.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:21.717+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.721476+0000) 2019-10-13T08:23:21.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:21.718+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.721517+0000) 2019-10-13T08:23:21.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:21.721+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.722589+0000) 2019-10-13T08:23:21.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:21.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:21.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:21.721+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.723830+0000) 2019-10-13T08:23:21.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:21.726+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:21.728874+0000) 2019-10-13T08:23:24.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:24.317+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:24.317775+0000) 2019-10-13T08:23:24.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:24.774+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:24.775869+0000) 2019-10-13T08:23:26.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:26.725+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.726532+0000) 2019-10-13T08:23:26.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:26.725+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.727547+0000) 2019-10-13T08:23:26.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:26.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:26.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:26.725+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.729369+0000) 2019-10-13T08:23:26.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:26.725+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.729616+0000) 2019-10-13T08:23:26.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:26.729+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.730127+0000) 2019-10-13T08:23:26.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:26.729+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.730323+0000) 2019-10-13T08:23:26.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:26.726+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.730645+0000) 2019-10-13T08:23:26.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:26.730+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:26.733842+0000) 2019-10-13T08:23:27.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:27.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:27.157002+0000) 2019-10-13T08:23:27.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:27.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:27.446630+0000) 2019-10-13T08:23:27.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:27.541+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:27.545192+0000) 2019-10-13T08:23:27.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:27.569+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:27.571610+0000) 2019-10-13T08:23:31.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:31.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.591872+0000) 2019-10-13T08:23:31.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:31.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.593132+0000) 2019-10-13T08:23:31.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:31.729+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.732770+0000) 2019-10-13T08:23:31.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:31.733+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.733735+0000) 2019-10-13T08:23:31.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:31.733+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.734331+0000) 2019-10-13T08:23:31.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:31.733+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.734478+0000) 2019-10-13T08:23:31.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:31.733+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.735231+0000) 2019-10-13T08:23:31.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:31.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:31.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:31.733+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.735028+0000) 2019-10-13T08:23:31.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:31.733+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.736767+0000) 2019-10-13T08:23:31.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:31.737+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:31.741034+0000) 2019-10-13T08:23:34.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:34.317+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:34.318000+0000) 2019-10-13T08:23:34.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:34.785+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:34.787603+0000) 2019-10-13T08:23:36.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:36.737+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.738388+0000) 2019-10-13T08:23:36.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:36.737+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.738638+0000) 2019-10-13T08:23:36.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:36.733+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.739144+0000) 2019-10-13T08:23:36.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:36.737+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.739974+0000) 2019-10-13T08:23:36.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:36.737+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.741256+0000) 2019-10-13T08:23:36.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:36.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:36.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:36.741+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.742471+0000) 2019-10-13T08:23:36.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:36.741+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.742969+0000) 2019-10-13T08:23:36.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:36.741+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:36.747908+0000) 2019-10-13T08:23:37.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:37.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:37.157219+0000) 2019-10-13T08:23:37.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:37.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:37.446804+0000) 2019-10-13T08:23:37.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:37.541+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:37.545365+0000) 2019-10-13T08:23:37.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:37.569+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:37.571783+0000) 2019-10-13T08:23:38.392 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:23:38.392 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:38.396 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:23:38.396 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:23:41.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:41.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.592097+0000) 2019-10-13T08:23:41.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:41.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.593326+0000) 2019-10-13T08:23:41.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:41.745+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.746596+0000) 2019-10-13T08:23:41.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:41.745+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.746643+0000) 2019-10-13T08:23:41.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:41.745+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.747351+0000) 2019-10-13T08:23:41.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:41.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:41.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:41.745+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.748130+0000) 2019-10-13T08:23:41.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:41.745+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.748399+0000) 2019-10-13T08:23:41.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:41.745+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.748772+0000) 2019-10-13T08:23:41.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:41.745+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.748679+0000) 2019-10-13T08:23:41.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:41.749+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:41.756057+0000) 2019-10-13T08:23:44.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:44.317+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:44.318224+0000) 2019-10-13T08:23:44.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:44.789+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:44.794090+0000) 2019-10-13T08:23:46.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:46.748+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.751292+0000) 2019-10-13T08:23:46.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:46.748+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.752233+0000) 2019-10-13T08:23:46.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:46.748+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.752561+0000) 2019-10-13T08:23:46.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:46.749+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.752326+0000) 2019-10-13T08:23:46.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:46.748+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.753457+0000) 2019-10-13T08:23:46.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:46.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:46.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:46.752+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.754380+0000) 2019-10-13T08:23:46.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:46.752+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.754792+0000) 2019-10-13T08:23:46.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:46.753+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:46.759364+0000) 2019-10-13T08:23:47.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:47.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:47.157404+0000) 2019-10-13T08:23:47.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:47.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:47.447015+0000) 2019-10-13T08:23:47.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:47.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:47.545521+0000) 2019-10-13T08:23:47.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:47.568+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:47.571914+0000) 2019-10-13T08:23:51.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:51.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.592295+0000) 2019-10-13T08:23:51.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:51.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.593555+0000) 2019-10-13T08:23:51.757 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:51.752+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.757445+0000) 2019-10-13T08:23:51.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:51.756+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.759269+0000) 2019-10-13T08:23:51.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:51.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:51.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:51.756+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.760312+0000) 2019-10-13T08:23:51.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:51.756+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.760740+0000) 2019-10-13T08:23:51.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:51.757+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.762032+0000) 2019-10-13T08:23:51.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:51.760+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.762611+0000) 2019-10-13T08:23:51.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:51.760+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.762285+0000) 2019-10-13T08:23:51.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:51.761+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:51.766014+0000) 2019-10-13T08:23:54.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:54.316+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:54.318438+0000) 2019-10-13T08:23:54.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:54.797+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:54.800661+0000) 2019-10-13T08:23:56.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:56.760+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.762719+0000) 2019-10-13T08:23:56.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:23:56.760+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.763483+0000) 2019-10-13T08:23:56.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:56.760+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.764615+0000) 2019-10-13T08:23:56.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:56.760+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.765449+0000) 2019-10-13T08:23:56.766 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:23:56.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71890 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:23:56.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:56.764+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.766254+0000) 2019-10-13T08:23:56.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:23:56.764+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.766567+0000) 2019-10-13T08:23:56.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:23:56.761+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.766944+0000) 2019-10-13T08:23:56.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:23:56.765+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:56.771841+0000) 2019-10-13T08:23:57.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:23:57.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:57.157584+0000) 2019-10-13T08:23:57.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:23:57.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:57.447227+0000) 2019-10-13T08:23:57.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:23:57.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:57.545666+0000) 2019-10-13T08:23:57.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:23:57.568+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:23:57.572091+0000) 2019-10-13T08:24:01.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:01.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.592518+0000) 2019-10-13T08:24:01.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:01.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.593775+0000) 2019-10-13T08:24:01.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:01.768+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.769845+0000) 2019-10-13T08:24:01.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:01.768+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.770290+0000) 2019-10-13T08:24:01.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:01.765+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.770087+0000) 2019-10-13T08:24:01.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:01.768+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.771549+0000) 2019-10-13T08:24:01.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:01.768+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.772262+0000) 2019-10-13T08:24:01.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:01.768+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.772579+0000) 2019-10-13T08:24:01.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:01.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:01.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:01.768+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.772916+0000) 2019-10-13T08:24:01.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:01.773+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:01.777034+0000) 2019-10-13T08:24:04.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:04.316+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:04.318636+0000) 2019-10-13T08:24:04.807 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:04.805+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:04.807271+0000) 2019-10-13T08:24:06.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:06.772+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.775683+0000) 2019-10-13T08:24:06.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:06.772+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.776531+0000) 2019-10-13T08:24:06.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:06.776+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.778822+0000) 2019-10-13T08:24:06.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:06.776+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.779070+0000) 2019-10-13T08:24:06.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:06.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71948 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:06.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:06.776+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.779870+0000) 2019-10-13T08:24:06.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:06.773+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.779830+0000) 2019-10-13T08:24:06.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:06.776+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.780899+0000) 2019-10-13T08:24:06.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:06.781+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:06.786318+0000) 2019-10-13T08:24:07.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:07.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:07.157804+0000) 2019-10-13T08:24:07.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:07.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:07.447406+0000) 2019-10-13T08:24:07.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:07.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:07.545840+0000) 2019-10-13T08:24:07.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:07.568+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:07.572261+0000) 2019-10-13T08:24:08.542 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:24:08.542 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:08.546 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:24:08.546 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:11.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:11.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.592724+0000) 2019-10-13T08:24:11.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:11.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.593938+0000) 2019-10-13T08:24:11.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:11.780+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.784328+0000) 2019-10-13T08:24:11.785 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:11.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:11.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:11.784+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.786180+0000) 2019-10-13T08:24:11.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:11.784+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.786436+0000) 2019-10-13T08:24:11.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:11.784+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.787218+0000) 2019-10-13T08:24:11.787 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:11.784+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.787360+0000) 2019-10-13T08:24:11.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:11.784+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.787766+0000) 2019-10-13T08:24:11.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:11.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.787575+0000) 2019-10-13T08:24:11.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:11.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:11.793895+0000) 2019-10-13T08:24:14.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:14.316+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:14.318851+0000) 2019-10-13T08:24:14.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:14.809+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:14.813809+0000) 2019-10-13T08:24:16.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:16.788+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.790269+0000) 2019-10-13T08:24:16.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:16.788+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.792646+0000) 2019-10-13T08:24:16.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:16.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:16.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:16.788+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.793270+0000) 2019-10-13T08:24:16.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:16.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.793315+0000) 2019-10-13T08:24:16.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:16.792+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.794015+0000) 2019-10-13T08:24:16.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:16.792+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.796052+0000) 2019-10-13T08:24:16.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:16.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.796185+0000) 2019-10-13T08:24:16.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:16.793+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:16.800038+0000) 2019-10-13T08:24:17.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:17.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:17.158021+0000) 2019-10-13T08:24:17.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:17.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:17.447610+0000) 2019-10-13T08:24:17.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:17.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:17.546021+0000) 2019-10-13T08:24:17.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:17.568+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:17.572438+0000) 2019-10-13T08:24:21.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:21.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.592916+0000) 2019-10-13T08:24:21.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:21.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.594154+0000) 2019-10-13T08:24:21.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:21.792+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.796112+0000) 2019-10-13T08:24:21.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:21.792+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.796915+0000) 2019-10-13T08:24:21.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:21.796+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.798744+0000) 2019-10-13T08:24:21.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:21.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:21.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:21.796+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.799798+0000) 2019-10-13T08:24:21.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:21.793+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.799953+0000) 2019-10-13T08:24:21.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:21.796+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.800818+0000) 2019-10-13T08:24:21.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:21.800+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.802704+0000) 2019-10-13T08:24:21.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:21.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:21.802204+0000) 2019-10-13T08:24:24.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:24.316+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:24.319057+0000) 2019-10-13T08:24:24.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:24.817+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:24.820547+0000) 2019-10-13T08:24:26.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:26.800+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.802687+0000) 2019-10-13T08:24:26.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:26.800+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.803174+0000) 2019-10-13T08:24:26.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:26.800+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.804624+0000) 2019-10-13T08:24:26.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:26.800+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.805055+0000) 2019-10-13T08:24:26.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:26.800+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.805495+0000) 2019-10-13T08:24:26.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:26.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:26.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:26.804+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.806597+0000) 2019-10-13T08:24:26.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:26.801+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.806596+0000) 2019-10-13T08:24:26.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:26.805+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:26.811582+0000) 2019-10-13T08:24:27.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:27.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:27.158203+0000) 2019-10-13T08:24:27.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:27.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:27.447787+0000) 2019-10-13T08:24:27.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:27.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:27.546174+0000) 2019-10-13T08:24:27.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:27.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:27.572572+0000) 2019-10-13T08:24:31.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:31.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.593114+0000) 2019-10-13T08:24:31.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:31.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.594384+0000) 2019-10-13T08:24:31.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:31.804+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.809098+0000) 2019-10-13T08:24:31.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:31.804+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.809371+0000) 2019-10-13T08:24:31.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:31.808+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.810014+0000) 2019-10-13T08:24:31.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:31.808+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.811430+0000) 2019-10-13T08:24:31.811 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:31.808+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.811940+0000) 2019-10-13T08:24:31.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:31.808+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.812736+0000) 2019-10-13T08:24:31.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:31.808+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:31.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:31.809+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.812656+0000) 2019-10-13T08:24:31.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:31.813+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:31.817631+0000) 2019-10-13T08:24:34.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:34.316+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:34.319265+0000) 2019-10-13T08:24:34.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:34.825+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:34.827073+0000) 2019-10-13T08:24:36.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:36.812+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.815884+0000) 2019-10-13T08:24:36.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:36.812+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.817017+0000) 2019-10-13T08:24:36.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:36.816+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.818073+0000) 2019-10-13T08:24:36.818 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:36.816+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.818552+0000) 2019-10-13T08:24:36.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:36.816+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.818991+0000) 2019-10-13T08:24:36.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:36.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:36.819 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:36.816+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.820009+0000) 2019-10-13T08:24:36.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:36.817+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.820224+0000) 2019-10-13T08:24:36.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:36.821+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:36.825831+0000) 2019-10-13T08:24:37.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:37.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:37.158382+0000) 2019-10-13T08:24:37.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:37.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:37.447964+0000) 2019-10-13T08:24:37.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:37.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:37.546321+0000) 2019-10-13T08:24:37.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:37.568+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:37.572703+0000) 2019-10-13T08:24:38.650 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:24:38.650 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:38.654 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:24:38.654 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:24:41.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:41.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.593365+0000) 2019-10-13T08:24:41.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:41.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.594583+0000) 2019-10-13T08:24:41.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:41.820+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.824579+0000) 2019-10-13T08:24:41.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:41.820+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.825697+0000) 2019-10-13T08:24:41.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:41.824+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:41.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:41.824+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.826524+0000) 2019-10-13T08:24:41.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:41.824+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.827165+0000) 2019-10-13T08:24:41.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:41.820+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.826718+0000) 2019-10-13T08:24:41.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:41.824+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.827715+0000) 2019-10-13T08:24:41.828 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:41.824+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.828963+0000) 2019-10-13T08:24:41.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:41.824+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:41.829426+0000) 2019-10-13T08:24:44.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:44.315+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:44.319494+0000) 2019-10-13T08:24:44.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:44.832+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:44.833550+0000) 2019-10-13T08:24:46.829 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:46.823+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.828920+0000) 2019-10-13T08:24:46.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:46.823+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.829255+0000) 2019-10-13T08:24:46.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:46.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:46.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:46.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.832130+0000) 2019-10-13T08:24:46.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:46.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.832329+0000) 2019-10-13T08:24:46.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:46.827+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.832943+0000) 2019-10-13T08:24:46.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:46.827+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.833448+0000) 2019-10-13T08:24:46.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:46.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.832805+0000) 2019-10-13T08:24:46.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:46.832+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:46.837171+0000) 2019-10-13T08:24:47.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:47.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:47.158600+0000) 2019-10-13T08:24:47.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:47.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:47.448176+0000) 2019-10-13T08:24:47.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:47.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:47.546454+0000) 2019-10-13T08:24:47.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:47.567+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:47.572839+0000) 2019-10-13T08:24:51.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:51.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.593565+0000) 2019-10-13T08:24:51.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:51.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.594816+0000) 2019-10-13T08:24:51.835 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:51.831+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.835133+0000) 2019-10-13T08:24:51.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:51.831+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.836016+0000) 2019-10-13T08:24:51.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:51.831+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.837027+0000) 2019-10-13T08:24:51.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:51.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:51.838 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:51.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.838960+0000) 2019-10-13T08:24:51.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:51.835+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.839274+0000) 2019-10-13T08:24:51.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:51.832+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.839970+0000) 2019-10-13T08:24:51.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:51.836+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.841371+0000) 2019-10-13T08:24:51.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:51.835+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:51.841336+0000) 2019-10-13T08:24:54.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:54.315+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:54.319696+0000) 2019-10-13T08:24:54.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:54.836+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:54.840047+0000) 2019-10-13T08:24:56.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:56.839+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.842147+0000) 2019-10-13T08:24:56.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:24:56.839+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.842456+0000) 2019-10-13T08:24:56.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:56.839+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.843158+0000) 2019-10-13T08:24:56.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:24:56.836+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.842962+0000) 2019-10-13T08:24:56.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:56.839+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.843937+0000) 2019-10-13T08:24:56.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:24:56.839+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.844175+0000) 2019-10-13T08:24:56.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:56.839+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.844479+0000) 2019-10-13T08:24:56.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:24:56.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:24:56.849 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:24:56.844+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:56.849376+0000) 2019-10-13T08:24:57.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:24:57.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:57.158774+0000) 2019-10-13T08:24:57.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:24:57.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:57.448354+0000) 2019-10-13T08:24:57.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:24:57.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:57.546599+0000) 2019-10-13T08:24:57.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:24:57.571+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:24:57.573012+0000) 2019-10-13T08:25:01.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:01.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.593786+0000) 2019-10-13T08:25:01.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:01.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.595042+0000) 2019-10-13T08:25:01.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:01.843+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.847895+0000) 2019-10-13T08:25:01.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:01.843+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.849227+0000) 2019-10-13T08:25:01.851 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:01.847+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.851266+0000) 2019-10-13T08:25:01.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:01.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:01.853 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:01.847+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.852265+0000) 2019-10-13T08:25:01.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:01.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.852520+0000) 2019-10-13T08:25:01.854 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:01.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.852449+0000) 2019-10-13T08:25:01.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:01.851+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.855125+0000) 2019-10-13T08:25:01.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:01.852+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:01.857019+0000) 2019-10-13T08:25:04.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:04.319+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:04.319904+0000) 2019-10-13T08:25:04.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:04.844+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:04.847269+0000) 2019-10-13T08:25:06.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:06.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.854354+0000) 2019-10-13T08:25:06.855 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:06.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.854618+0000) 2019-10-13T08:25:06.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:06.851+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.855396+0000) 2019-10-13T08:25:06.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:06.851+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.855759+0000) 2019-10-13T08:25:06.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:06.851+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.857626+0000) 2019-10-13T08:25:06.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:06.852+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.857339+0000) 2019-10-13T08:25:06.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:06.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:06.858 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:06.855+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.858816+0000) 2019-10-13T08:25:06.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:06.860+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:06.864661+0000) 2019-10-13T08:25:07.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:07.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:07.158962+0000) 2019-10-13T08:25:07.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:07.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:07.448546+0000) 2019-10-13T08:25:07.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:07.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:07.546734+0000) 2019-10-13T08:25:07.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:07.571+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:07.573160+0000) 2019-10-13T08:25:08.727 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:25:08.727 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:08.730 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:25:08.731 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:11.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:11.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.594019+0000) 2019-10-13T08:25:11.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:11.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.595228+0000) 2019-10-13T08:25:11.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:11.855+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.861147+0000) 2019-10-13T08:25:11.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:11.855+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.861549+0000) 2019-10-13T08:25:11.862 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:11.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.862255+0000) 2019-10-13T08:25:11.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:11.859+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.862694+0000) 2019-10-13T08:25:11.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:11.859+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.863454+0000) 2019-10-13T08:25:11.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:11.859+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.863674+0000) 2019-10-13T08:25:11.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:11.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:11.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:11.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.864011+0000) 2019-10-13T08:25:11.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:11.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:11.869441+0000) 2019-10-13T08:25:14.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:14.319+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:14.320106+0000) 2019-10-13T08:25:14.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:14.852+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:14.853775+0000) 2019-10-13T08:25:16.867 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:16.863+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.867634+0000) 2019-10-13T08:25:16.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:16.863+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.867978+0000) 2019-10-13T08:25:16.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:16.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.867846+0000) 2019-10-13T08:25:16.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:16.863+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.868750+0000) 2019-10-13T08:25:16.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:16.863+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.869404+0000) 2019-10-13T08:25:16.869 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:16.867+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.869859+0000) 2019-10-13T08:25:16.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:16.867+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:16.871 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:16.867+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.870644+0000) 2019-10-13T08:25:16.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:16.872+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:16.876684+0000) 2019-10-13T08:25:17.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:17.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:17.159173+0000) 2019-10-13T08:25:17.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:17.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:17.448722+0000) 2019-10-13T08:25:17.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:17.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:17.546915+0000) 2019-10-13T08:25:17.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:17.567+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:17.573290+0000) 2019-10-13T08:25:21.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:21.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.594236+0000) 2019-10-13T08:25:21.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:21.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.595455+0000) 2019-10-13T08:25:21.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:21.868+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.874035+0000) 2019-10-13T08:25:21.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:21.871+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.874944+0000) 2019-10-13T08:25:21.875 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:21.871+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.875464+0000) 2019-10-13T08:25:21.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:21.871+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.875809+0000) 2019-10-13T08:25:21.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:21.871+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.876232+0000) 2019-10-13T08:25:21.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:21.871+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.876736+0000) 2019-10-13T08:25:21.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:21.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:21.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:21.871+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.877181+0000) 2019-10-13T08:25:21.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:21.880+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:21.884391+0000) 2019-10-13T08:25:24.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:24.319+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:24.320338+0000) 2019-10-13T08:25:24.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:24.860+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:24.860440+0000) 2019-10-13T08:25:26.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:26.875+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.880846+0000) 2019-10-13T08:25:26.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:26.876+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.880578+0000) 2019-10-13T08:25:26.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:26.875+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.881376+0000) 2019-10-13T08:25:26.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:26.879+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.882268+0000) 2019-10-13T08:25:26.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:26.879+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.883171+0000) 2019-10-13T08:25:26.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:26.879+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.883604+0000) 2019-10-13T08:25:26.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:26.879+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.883821+0000) 2019-10-13T08:25:26.886 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:26.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:26.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:26.884+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:26.888803+0000) 2019-10-13T08:25:27.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:27.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:27.159351+0000) 2019-10-13T08:25:27.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:27.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:27.448898+0000) 2019-10-13T08:25:27.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:27.543+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:27.547089+0000) 2019-10-13T08:25:27.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:27.571+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:27.573449+0000) 2019-10-13T08:25:31.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:31.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.594483+0000) 2019-10-13T08:25:31.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:31.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.595697+0000) 2019-10-13T08:25:31.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:31.880+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.886421+0000) 2019-10-13T08:25:31.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:31.883+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.887799+0000) 2019-10-13T08:25:31.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:31.883+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.888683+0000) 2019-10-13T08:25:31.889 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:31.883+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.889045+0000) 2019-10-13T08:25:31.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:31.887+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.889770+0000) 2019-10-13T08:25:31.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:31.887+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.890982+0000) 2019-10-13T08:25:31.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:31.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:31.893 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:31.887+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.892669+0000) 2019-10-13T08:25:31.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:31.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:31.894896+0000) 2019-10-13T08:25:34.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:34.315+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:34.320545+0000) 2019-10-13T08:25:34.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:34.864+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:34.867020+0000) 2019-10-13T08:25:36.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:36.887+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.893466+0000) 2019-10-13T08:25:36.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:36.888+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.895728+0000) 2019-10-13T08:25:36.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:36.891+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.896925+0000) 2019-10-13T08:25:36.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:36.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72479 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:36.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:36.895+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.898107+0000) 2019-10-13T08:25:36.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:36.895+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.898542+0000) 2019-10-13T08:25:36.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:36.895+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.898361+0000) 2019-10-13T08:25:36.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:36.895+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.899188+0000) 2019-10-13T08:25:36.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:36.900+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:36.905637+0000) 2019-10-13T08:25:37.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:37.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:37.159537+0000) 2019-10-13T08:25:37.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:37.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:37.449107+0000) 2019-10-13T08:25:37.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:37.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:37.547275+0000) 2019-10-13T08:25:37.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:37.567+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:37.573602+0000) 2019-10-13T08:25:38.789 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:25:38.790 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:38.801 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:25:38.801 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:25:41.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:41.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.594698+0000) 2019-10-13T08:25:41.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:41.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.595900+0000) 2019-10-13T08:25:41.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:41.894+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.901587+0000) 2019-10-13T08:25:41.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:41.898+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.902016+0000) 2019-10-13T08:25:41.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:41.900+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.902116+0000) 2019-10-13T08:25:41.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:41.898+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.902984+0000) 2019-10-13T08:25:41.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:41.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.903409+0000) 2019-10-13T08:25:41.905 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:41.898+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.904385+0000) 2019-10-13T08:25:41.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:41.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:41.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:41.898+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.905092+0000) 2019-10-13T08:25:41.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:41.908+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:41.910298+0000) 2019-10-13T08:25:44.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:44.318+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:44.320748+0000) 2019-10-13T08:25:44.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:44.871+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:44.873601+0000) 2019-10-13T08:25:46.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:46.903+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.907517+0000) 2019-10-13T08:25:46.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:46.902+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.908427+0000) 2019-10-13T08:25:46.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:46.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.909001+0000) 2019-10-13T08:25:46.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:46.902+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.909487+0000) 2019-10-13T08:25:46.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:46.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:46.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:46.906+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.911227+0000) 2019-10-13T08:25:46.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:46.906+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.911948+0000) 2019-10-13T08:25:46.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:46.906+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.912769+0000) 2019-10-13T08:25:46.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:46.915+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:46.917713+0000) 2019-10-13T08:25:47.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:47.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:47.159727+0000) 2019-10-13T08:25:47.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:47.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:47.449285+0000) 2019-10-13T08:25:47.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:47.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:47.547454+0000) 2019-10-13T08:25:47.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:47.570+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:47.573789+0000) 2019-10-13T08:25:51.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:51.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.594885+0000) 2019-10-13T08:25:51.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:51.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.596129+0000) 2019-10-13T08:25:51.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:51.910+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.914036+0000) 2019-10-13T08:25:51.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:51.910+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.914670+0000) 2019-10-13T08:25:51.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:51.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.914737+0000) 2019-10-13T08:25:51.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:51.910+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.915175+0000) 2019-10-13T08:25:51.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:51.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.915354+0000) 2019-10-13T08:25:51.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:51.910+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.916718+0000) 2019-10-13T08:25:51.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:51.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:51.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:51.914+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.918010+0000) 2019-10-13T08:25:51.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:51.923+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:51.927061+0000) 2019-10-13T08:25:54.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:54.314+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:54.320971+0000) 2019-10-13T08:25:54.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:54.879+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:54.880241+0000) 2019-10-13T08:25:56.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:56.914+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.919837+0000) 2019-10-13T08:25:56.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:25:56.915+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.919729+0000) 2019-10-13T08:25:56.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:56.914+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.920598+0000) 2019-10-13T08:25:56.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:56.914+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.921007+0000) 2019-10-13T08:25:56.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:25:56.918+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.922252+0000) 2019-10-13T08:25:56.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:25:56.918+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.922404+0000) 2019-10-13T08:25:56.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:25:56.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:25:56.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:56.918+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.923412+0000) 2019-10-13T08:25:56.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:25:56.923+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:56.927699+0000) 2019-10-13T08:25:57.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:25:57.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:57.159939+0000) 2019-10-13T08:25:57.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:25:57.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:57.449462+0000) 2019-10-13T08:25:57.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:25:57.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:57.547629+0000) 2019-10-13T08:25:57.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:25:57.570+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:25:57.574001+0000) 2019-10-13T08:26:01.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:01.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.595089+0000) 2019-10-13T08:26:01.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:01.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.596371+0000) 2019-10-13T08:26:01.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:01.922+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.925728+0000) 2019-10-13T08:26:01.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:01.922+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.926189+0000) 2019-10-13T08:26:01.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:01.922+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.926727+0000) 2019-10-13T08:26:01.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:01.922+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.927450+0000) 2019-10-13T08:26:01.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:01.922+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.927481+0000) 2019-10-13T08:26:01.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:01.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.928045+0000) 2019-10-13T08:26:01.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:01.922+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.928858+0000) 2019-10-13T08:26:01.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:01.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:01.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:01.935+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:01.936660+0000) 2019-10-13T08:26:04.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:04.314+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:04.321177+0000) 2019-10-13T08:26:04.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:04.883+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:04.886732+0000) 2019-10-13T08:26:06.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:06.926+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.933660+0000) 2019-10-13T08:26:06.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:06.930+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.936148+0000) 2019-10-13T08:26:06.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:06.930+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.936344+0000) 2019-10-13T08:26:06.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:06.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:06.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:06.930+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.937184+0000) 2019-10-13T08:26:06.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:06.934+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.938050+0000) 2019-10-13T08:26:06.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:06.930+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.937482+0000) 2019-10-13T08:26:06.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:06.935+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.939694+0000) 2019-10-13T08:26:06.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:06.943+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:06.945140+0000) 2019-10-13T08:26:07.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:07.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:07.160155+0000) 2019-10-13T08:26:07.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:07.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:07.449673+0000) 2019-10-13T08:26:07.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:07.542+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:07.547830+0000) 2019-10-13T08:26:07.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:07.570+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:07.574189+0000) 2019-10-13T08:26:08.905 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:26:08.905 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:08.909 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:26:08.909 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:11.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:11.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.595281+0000) 2019-10-13T08:26:11.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:11.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.596570+0000) 2019-10-13T08:26:11.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:11.934+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.939228+0000) 2019-10-13T08:26:11.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:11.934+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.940492+0000) 2019-10-13T08:26:11.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:11.939+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.940418+0000) 2019-10-13T08:26:11.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:11.934+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.941401+0000) 2019-10-13T08:26:11.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:11.938+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.941878+0000) 2019-10-13T08:26:11.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:11.938+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.942604+0000) 2019-10-13T08:26:11.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:11.938+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.942686+0000) 2019-10-13T08:26:11.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:11.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:11.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:11.947+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:11.948254+0000) 2019-10-13T08:26:14.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:14.314+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:14.321379+0000) 2019-10-13T08:26:14.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:14.891+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:14.893173+0000) 2019-10-13T08:26:16.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:16.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.946354+0000) 2019-10-13T08:26:16.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:16.942+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.946662+0000) 2019-10-13T08:26:16.947 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:16.942+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.947099+0000) 2019-10-13T08:26:16.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:16.943+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.947107+0000) 2019-10-13T08:26:16.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:16.942+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.948244+0000) 2019-10-13T08:26:16.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:16.942+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.948560+0000) 2019-10-13T08:26:16.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:16.942+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.949096+0000) 2019-10-13T08:26:16.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:16.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:16.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:16.951+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:16.955418+0000) 2019-10-13T08:26:17.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:17.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:17.160340+0000) 2019-10-13T08:26:17.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:17.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:17.449921+0000) 2019-10-13T08:26:17.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:17.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:17.548035+0000) 2019-10-13T08:26:17.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:17.570+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:17.574370+0000) 2019-10-13T08:26:21.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:21.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.595434+0000) 2019-10-13T08:26:21.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:21.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.596816+0000) 2019-10-13T08:26:21.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:21.951+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.952871+0000) 2019-10-13T08:26:21.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:21.950+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.953858+0000) 2019-10-13T08:26:21.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:21.950+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.954757+0000) 2019-10-13T08:26:21.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:21.950+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.954758+0000) 2019-10-13T08:26:21.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:21.950+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.955282+0000) 2019-10-13T08:26:21.955 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:21.950+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.955920+0000) 2019-10-13T08:26:21.956 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:21.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:21.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:21.950+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.956854+0000) 2019-10-13T08:26:21.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:21.955+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:21.960000+0000) 2019-10-13T08:26:24.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:24.318+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:24.321617+0000) 2019-10-13T08:26:24.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:24.895+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:24.899847+0000) 2019-10-13T08:26:26.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:26.954+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.959634+0000) 2019-10-13T08:26:26.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:26.954+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.960291+0000) 2019-10-13T08:26:26.961 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:26.954+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.960890+0000) 2019-10-13T08:26:26.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:26.954+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.961217+0000) 2019-10-13T08:26:26.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:26.959+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.961210+0000) 2019-10-13T08:26:26.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:26.958+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.961894+0000) 2019-10-13T08:26:26.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:26.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:26.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:26.958+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.963201+0000) 2019-10-13T08:26:26.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:26.963+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:26.966168+0000) 2019-10-13T08:26:27.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:27.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:27.160570+0000) 2019-10-13T08:26:27.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:27.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:27.450099+0000) 2019-10-13T08:26:27.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:27.542+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:27.548209+0000) 2019-10-13T08:26:27.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:27.574+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:27.574545+0000) 2019-10-13T08:26:31.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:31.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.595650+0000) 2019-10-13T08:26:31.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:31.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.597026+0000) 2019-10-13T08:26:31.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:31.962+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.966054+0000) 2019-10-13T08:26:31.966 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:31.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.966892+0000) 2019-10-13T08:26:31.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:31.962+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.967256+0000) 2019-10-13T08:26:31.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:31.962+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.968109+0000) 2019-10-13T08:26:31.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:31.967+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.968169+0000) 2019-10-13T08:26:31.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:31.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:31.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:31.962+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.968864+0000) 2019-10-13T08:26:31.970 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:31.966+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.970041+0000) 2019-10-13T08:26:31.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:31.971+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:31.972463+0000) 2019-10-13T08:26:34.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:34.318+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:34.321854+0000) 2019-10-13T08:26:34.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:34.907+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:34.909311+0000) 2019-10-13T08:26:36.973 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:36.966+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.973406+0000) 2019-10-13T08:26:36.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:36.970+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.973891+0000) 2019-10-13T08:26:36.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:36.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.974281+0000) 2019-10-13T08:26:36.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:36.970+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.975358+0000) 2019-10-13T08:26:36.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:36.970+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:36.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:36.970+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.976086+0000) 2019-10-13T08:26:36.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:36.970+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.976508+0000) 2019-10-13T08:26:36.977 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:36.970+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.976881+0000) 2019-10-13T08:26:36.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:36.979+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:36.981542+0000) 2019-10-13T08:26:37.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:37.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:37.160759+0000) 2019-10-13T08:26:37.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:37.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:37.450332+0000) 2019-10-13T08:26:37.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:37.545+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:37.548386+0000) 2019-10-13T08:26:37.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:37.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:37.574722+0000) 2019-10-13T08:26:39.206 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:26:39.206 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:39.210 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:26:39.210 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:26:41.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:41.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.595857+0000) 2019-10-13T08:26:41.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:41.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.597252+0000) 2019-10-13T08:26:41.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:41.975+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.978273+0000) 2019-10-13T08:26:41.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:41.973+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.979382+0000) 2019-10-13T08:26:41.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:41.973+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.979709+0000) 2019-10-13T08:26:41.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:41.973+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.980792+0000) 2019-10-13T08:26:41.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:41.973+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.981417+0000) 2019-10-13T08:26:41.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:41.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:41.982 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:41.977+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.982005+0000) 2019-10-13T08:26:41.982 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:41.977+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.982474+0000) 2019-10-13T08:26:41.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:41.983+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:41.985822+0000) 2019-10-13T08:26:44.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:44.321+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:44.322057+0000) 2019-10-13T08:26:44.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:44.911+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:44.915796+0000) 2019-10-13T08:26:46.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:46.977+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.985059+0000) 2019-10-13T08:26:46.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:46.977+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.985720+0000) 2019-10-13T08:26:46.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:46.983+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.985637+0000) 2019-10-13T08:26:46.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:46.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.986445+0000) 2019-10-13T08:26:46.987 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:46.981+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.987388+0000) 2019-10-13T08:26:46.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:46.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:46.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:46.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.988007+0000) 2019-10-13T08:26:46.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:46.981+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.988787+0000) 2019-10-13T08:26:46.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:46.991+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:46.992914+0000) 2019-10-13T08:26:47.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:47.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:47.161004+0000) 2019-10-13T08:26:47.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:47.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:47.450510+0000) 2019-10-13T08:26:47.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:47.545+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:47.548585+0000) 2019-10-13T08:26:47.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:47.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:47.574896+0000) 2019-10-13T08:26:51.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:51.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.596046+0000) 2019-10-13T08:26:51.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:51.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.597479+0000) 2019-10-13T08:26:51.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:51.985+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.991772+0000) 2019-10-13T08:26:51.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:51.985+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.992192+0000) 2019-10-13T08:26:51.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:51.985+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.992969+0000) 2019-10-13T08:26:51.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:51.985+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.993292+0000) 2019-10-13T08:26:51.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:51.985+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.993687+0000) 2019-10-13T08:26:51.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:51.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:51.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:51.989+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.995783+0000) 2019-10-13T08:26:51.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:51.990+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:51.993532+0000) 2019-10-13T08:26:52.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:51.998+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:52.000757+0000) 2019-10-13T08:26:54.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:54.321+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:54.322268+0000) 2019-10-13T08:26:54.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:54.918+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:54.922188+0000) 2019-10-13T08:26:56.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:26:56.993+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.997620+0000) 2019-10-13T08:26:56.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:56.997+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.997918+0000) 2019-10-13T08:26:56.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:56.997+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:56.998408+0000) 2019-10-13T08:26:57.000 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:26:56.997+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:26:57.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:56.997+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.000716+0000) 2019-10-13T08:26:57.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:26:56.998+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.000940+0000) 2019-10-13T08:26:57.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:26:56.997+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.001243+0000) 2019-10-13T08:26:57.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:57.001+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.002409+0000) 2019-10-13T08:26:57.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:26:57.006+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.008328+0000) 2019-10-13T08:26:57.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:26:57.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.161170+0000) 2019-10-13T08:26:57.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:26:57.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.450716+0000) 2019-10-13T08:26:57.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:26:57.545+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.548765+0000) 2019-10-13T08:26:57.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:26:57.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:26:57.575074+0000) 2019-10-13T08:27:01.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:01.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.596241+0000) 2019-10-13T08:27:01.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:01.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:01.597707+0000) 2019-10-13T08:27:02.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:02.001+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.003557+0000) 2019-10-13T08:27:02.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:02.001+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.004049+0000) 2019-10-13T08:27:02.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:02.001+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.004471+0000) 2019-10-13T08:27:02.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:02.001+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.004617+0000) 2019-10-13T08:27:02.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:02.002+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.004650+0000) 2019-10-13T08:27:02.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:02.001+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.005583+0000) 2019-10-13T08:27:02.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:02.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:02.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:02.005+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.006967+0000) 2019-10-13T08:27:02.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:02.010+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:02.012359+0000) 2019-10-13T08:27:04.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:04.321+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:04.322477+0000) 2019-10-13T08:27:04.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:04.926+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:04.928707+0000) 2019-10-13T08:27:07.010 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:07.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.010208+0000) 2019-10-13T08:27:07.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:07.009+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.010402+0000) 2019-10-13T08:27:07.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:07.009+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.011196+0000) 2019-10-13T08:27:07.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:07.009+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.011690+0000) 2019-10-13T08:27:07.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:07.006+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.011610+0000) 2019-10-13T08:27:07.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:07.009+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:07.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:07.009+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.012770+0000) 2019-10-13T08:27:07.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:07.009+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.013446+0000) 2019-10-13T08:27:07.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:07.014+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.019253+0000) 2019-10-13T08:27:07.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:07.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.161350+0000) 2019-10-13T08:27:07.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:07.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.450893+0000) 2019-10-13T08:27:07.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:07.545+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.548941+0000) 2019-10-13T08:27:07.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:07.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:07.575277+0000) 2019-10-13T08:27:09.322 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:27:09.322 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:09.326 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:27:09.326 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:11.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:11.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.596437+0000) 2019-10-13T08:27:11.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:11.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:11.597925+0000) 2019-10-13T08:27:12.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:12.013+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.015986+0000) 2019-10-13T08:27:12.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:12.013+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.016649+0000) 2019-10-13T08:27:12.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:12.017+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.018226+0000) 2019-10-13T08:27:12.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:12.014+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.018275+0000) 2019-10-13T08:27:12.019 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:12.017+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.019150+0000) 2019-10-13T08:27:12.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:12.017+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.018532+0000) 2019-10-13T08:27:12.020 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:12.017+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.019436+0000) 2019-10-13T08:27:12.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:12.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:12.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:12.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:12.023185+0000) 2019-10-13T08:27:14.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:14.321+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:14.322699+0000) 2019-10-13T08:27:14.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:14.934+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:14.935055+0000) 2019-10-13T08:27:17.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:17.021+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.022977+0000) 2019-10-13T08:27:17.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:17.018+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.022908+0000) 2019-10-13T08:27:17.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:17.021+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.024239+0000) 2019-10-13T08:27:17.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:17.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.024866+0000) 2019-10-13T08:27:17.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:17.021+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.025574+0000) 2019-10-13T08:27:17.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:17.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73068 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:17.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:17.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.025934+0000) 2019-10-13T08:27:17.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:17.025+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.026688+0000) 2019-10-13T08:27:17.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:17.026+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.030147+0000) 2019-10-13T08:27:17.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:17.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.161534+0000) 2019-10-13T08:27:17.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:17.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.451072+0000) 2019-10-13T08:27:17.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:17.545+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.549099+0000) 2019-10-13T08:27:17.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:17.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:17.575451+0000) 2019-10-13T08:27:21.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:21.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.596661+0000) 2019-10-13T08:27:21.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:21.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:21.598163+0000) 2019-10-13T08:27:22.029 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:22.025+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.029392+0000) 2019-10-13T08:27:22.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:22.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.030398+0000) 2019-10-13T08:27:22.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:22.026+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.030278+0000) 2019-10-13T08:27:22.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:22.029+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.031119+0000) 2019-10-13T08:27:22.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:22.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:22.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:22.029+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.031729+0000) 2019-10-13T08:27:22.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:22.029+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.031945+0000) 2019-10-13T08:27:22.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:22.029+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.032534+0000) 2019-10-13T08:27:22.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:22.030+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:22.035129+0000) 2019-10-13T08:27:24.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:24.321+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:24.322891+0000) 2019-10-13T08:27:24.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:24.938+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:24.941502+0000) 2019-10-13T08:27:27.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:27.033+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.035891+0000) 2019-10-13T08:27:27.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:27.033+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.036292+0000) 2019-10-13T08:27:27.037 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:27.033+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.036774+0000) 2019-10-13T08:27:27.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:27.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:27.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:27.034+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.039167+0000) 2019-10-13T08:27:27.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:27.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.040122+0000) 2019-10-13T08:27:27.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:27.037+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.039958+0000) 2019-10-13T08:27:27.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:27.037+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.039799+0000) 2019-10-13T08:27:27.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:27.038+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.043889+0000) 2019-10-13T08:27:27.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:27.161+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.161776+0000) 2019-10-13T08:27:27.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:27.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.451269+0000) 2019-10-13T08:27:27.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:27.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.549282+0000) 2019-10-13T08:27:27.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:27.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:27.575623+0000) 2019-10-13T08:27:31.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:31.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.596868+0000) 2019-10-13T08:27:31.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:31.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:31.598409+0000) 2019-10-13T08:27:32.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:32.037+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.041116+0000) 2019-10-13T08:27:32.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:32.041+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.042140+0000) 2019-10-13T08:27:32.042 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:32.041+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.041731+0000) 2019-10-13T08:27:32.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:32.041+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.042867+0000) 2019-10-13T08:27:32.043 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:32.038+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.042948+0000) 2019-10-13T08:27:32.044 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:32.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:32.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:32.041+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.044170+0000) 2019-10-13T08:27:32.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:32.041+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.045046+0000) 2019-10-13T08:27:32.052 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:32.046+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:32.051621+0000) 2019-10-13T08:27:34.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:34.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:34.323098+0000) 2019-10-13T08:27:34.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:34.946+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:34.948069+0000) 2019-10-13T08:27:37.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:37.044+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.047591+0000) 2019-10-13T08:27:37.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:37.044+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.047936+0000) 2019-10-13T08:27:37.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:37.044+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.048443+0000) 2019-10-13T08:27:37.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:37.044+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.049014+0000) 2019-10-13T08:27:37.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:37.044+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.049598+0000) 2019-10-13T08:27:37.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:37.048+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.050140+0000) 2019-10-13T08:27:37.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:37.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:37.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:37.046+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.050088+0000) 2019-10-13T08:27:37.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:37.050+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.052842+0000) 2019-10-13T08:27:37.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:37.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.161957+0000) 2019-10-13T08:27:37.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:37.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.451474+0000) 2019-10-13T08:27:37.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:37.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.549458+0000) 2019-10-13T08:27:37.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:37.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:37.575804+0000) 2019-10-13T08:27:39.550 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:27:39.550 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:39.554 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:27:39.554 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:27:41.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:41.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.597061+0000) 2019-10-13T08:27:41.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:41.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:41.598662+0000) 2019-10-13T08:27:42.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:42.048+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.052944+0000) 2019-10-13T08:27:42.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:42.048+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.053624+0000) 2019-10-13T08:27:42.054 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:42.052+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.054078+0000) 2019-10-13T08:27:42.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:42.052+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.055560+0000) 2019-10-13T08:27:42.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:42.052+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.056054+0000) 2019-10-13T08:27:42.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:42.050+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.055750+0000) 2019-10-13T08:27:42.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:42.052+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.056071+0000) 2019-10-13T08:27:42.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:42.052+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:42.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:42.058+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:42.062821+0000) 2019-10-13T08:27:44.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:44.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:44.323299+0000) 2019-10-13T08:27:44.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:44.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:44.959726+0000) 2019-10-13T08:27:47.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:47.056+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.060933+0000) 2019-10-13T08:27:47.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:47.056+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.061394+0000) 2019-10-13T08:27:47.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:47.060+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.061792+0000) 2019-10-13T08:27:47.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:47.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73244 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:47.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:47.060+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.064073+0000) 2019-10-13T08:27:47.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:47.060+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.065214+0000) 2019-10-13T08:27:47.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:47.062+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.065101+0000) 2019-10-13T08:27:47.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:47.060+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.065330+0000) 2019-10-13T08:27:47.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:47.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.070213+0000) 2019-10-13T08:27:47.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:47.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.162140+0000) 2019-10-13T08:27:47.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:47.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.451708+0000) 2019-10-13T08:27:47.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:47.544+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.549672+0000) 2019-10-13T08:27:47.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:47.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:47.575977+0000) 2019-10-13T08:27:51.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:51.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.597262+0000) 2019-10-13T08:27:51.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:51.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:51.598882+0000) 2019-10-13T08:27:52.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:52.064+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.067373+0000) 2019-10-13T08:27:52.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:52.064+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.068356+0000) 2019-10-13T08:27:52.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:52.064+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.068458+0000) 2019-10-13T08:27:52.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:52.064+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.069637+0000) 2019-10-13T08:27:52.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:52.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:52.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:52.064+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.069222+0000) 2019-10-13T08:27:52.070 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:52.066+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.069361+0000) 2019-10-13T08:27:52.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:52.068+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.070536+0000) 2019-10-13T08:27:52.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:52.070+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:52.075853+0000) 2019-10-13T08:27:54.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:54.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:54.323500+0000) 2019-10-13T08:27:54.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:54.966+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:54.966202+0000) 2019-10-13T08:27:57.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:57.068+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.073694+0000) 2019-10-13T08:27:57.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:27:57.069+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.073552+0000) 2019-10-13T08:27:57.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:57.072+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.074345+0000) 2019-10-13T08:27:57.077 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:27:57.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:27:57.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:57.072+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.077541+0000) 2019-10-13T08:27:57.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:27:57.076+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.078157+0000) 2019-10-13T08:27:57.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:27:57.076+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.079010+0000) 2019-10-13T08:27:57.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:57.076+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.079434+0000) 2019-10-13T08:27:57.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:27:57.077+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.081696+0000) 2019-10-13T08:27:57.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:27:57.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.162356+0000) 2019-10-13T08:27:57.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:27:57.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.451916+0000) 2019-10-13T08:27:57.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:27:57.548+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.549842+0000) 2019-10-13T08:27:57.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:27:57.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:27:57.576145+0000) 2019-10-13T08:28:01.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:01.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:01.597459+0000) 2019-10-13T08:28:01.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:01.597+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:01.599082+0000) 2019-10-13T08:28:02.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:02.080+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.082428+0000) 2019-10-13T08:28:02.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:02.080+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.082593+0000) 2019-10-13T08:28:02.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:02.080+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.083108+0000) 2019-10-13T08:28:02.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:02.080+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.084271+0000) 2019-10-13T08:28:02.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:02.080+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:02.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:02.081+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.084150+0000) 2019-10-13T08:28:02.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:02.080+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.085048+0000) 2019-10-13T08:28:02.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:02.084+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.086372+0000) 2019-10-13T08:28:02.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:02.085+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:02.089396+0000) 2019-10-13T08:28:04.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:04.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:04.323703+0000) 2019-10-13T08:28:04.973 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:04.969+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:04.972802+0000) 2019-10-13T08:28:07.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:07.084+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.087700+0000) 2019-10-13T08:28:07.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:07.081+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.087779+0000) 2019-10-13T08:28:07.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:07.088+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.090338+0000) 2019-10-13T08:28:07.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:07.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:07.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:07.088+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.090805+0000) 2019-10-13T08:28:07.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:07.088+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.091193+0000) 2019-10-13T08:28:07.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:07.088+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.092312+0000) 2019-10-13T08:28:07.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:07.088+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.092631+0000) 2019-10-13T08:28:07.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:07.093+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.096917+0000) 2019-10-13T08:28:07.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:07.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.162553+0000) 2019-10-13T08:28:07.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:07.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.452123+0000) 2019-10-13T08:28:07.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:07.548+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.550038+0000) 2019-10-13T08:28:07.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:07.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:07.576355+0000) 2019-10-13T08:28:09.658 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:28:09.659 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:09.662 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:28:09.662 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:11.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:11.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:11.597684+0000) 2019-10-13T08:28:11.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:11.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:11.599310+0000) 2019-10-13T08:28:12.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:12.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.095964+0000) 2019-10-13T08:28:12.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:12.092+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.096469+0000) 2019-10-13T08:28:12.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:12.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:12.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:12.092+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.096993+0000) 2019-10-13T08:28:12.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:12.092+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.097163+0000) 2019-10-13T08:28:12.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:12.093+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.097153+0000) 2019-10-13T08:28:12.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:12.096+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.098185+0000) 2019-10-13T08:28:12.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:12.096+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.099210+0000) 2019-10-13T08:28:12.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:12.101+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:12.106215+0000) 2019-10-13T08:28:14.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:14.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:14.323923+0000) 2019-10-13T08:28:14.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:14.977+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:14.979341+0000) 2019-10-13T08:28:17.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:17.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.100189+0000) 2019-10-13T08:28:17.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:17.096+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.101198+0000) 2019-10-13T08:28:17.101 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:17.097+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.100993+0000) 2019-10-13T08:28:17.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:17.100+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.102502+0000) 2019-10-13T08:28:17.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:17.100+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.102876+0000) 2019-10-13T08:28:17.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:17.100+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:17.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:17.100+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.103292+0000) 2019-10-13T08:28:17.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:17.100+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.103699+0000) 2019-10-13T08:28:17.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:17.105+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.110810+0000) 2019-10-13T08:28:17.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:17.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.162764+0000) 2019-10-13T08:28:17.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:17.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.452331+0000) 2019-10-13T08:28:17.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:17.548+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.550272+0000) 2019-10-13T08:28:17.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:17.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:17.576566+0000) 2019-10-13T08:28:21.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:21.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:21.597915+0000) 2019-10-13T08:28:21.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:21.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:21.599517+0000) 2019-10-13T08:28:22.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:22.104+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.106793+0000) 2019-10-13T08:28:22.109 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:22.104+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.107561+0000) 2019-10-13T08:28:22.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:22.104+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.108200+0000) 2019-10-13T08:28:22.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:22.104+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.108944+0000) 2019-10-13T08:28:22.109 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:22.104+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.108764+0000) 2019-10-13T08:28:22.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:22.105+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.109052+0000) 2019-10-13T08:28:22.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:22.108+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:22.110 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:22.108+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.109913+0000) 2019-10-13T08:28:22.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:22.109+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:22.114059+0000) 2019-10-13T08:28:24.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:24.320+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:24.324075+0000) 2019-10-13T08:28:24.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:24.981+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:24.987813+0000) 2019-10-13T08:28:27.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:27.112+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.114852+0000) 2019-10-13T08:28:27.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:27.112+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.114864+0000) 2019-10-13T08:28:27.116 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:27.112+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73480 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:27.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:27.112+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.117140+0000) 2019-10-13T08:28:27.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:27.112+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.117401+0000) 2019-10-13T08:28:27.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:27.116+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.118180+0000) 2019-10-13T08:28:27.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:27.113+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.118214+0000) 2019-10-13T08:28:27.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:27.112+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.117585+0000) 2019-10-13T08:28:27.125 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:27.121+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.125031+0000) 2019-10-13T08:28:27.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:27.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.162950+0000) 2019-10-13T08:28:27.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:27.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.452509+0000) 2019-10-13T08:28:27.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:27.548+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.550427+0000) 2019-10-13T08:28:27.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:27.572+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:27.576747+0000) 2019-10-13T08:28:31.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:31.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:31.598162+0000) 2019-10-13T08:28:31.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:31.597+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:31.599714+0000) 2019-10-13T08:28:32.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:32.115+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.119892+0000) 2019-10-13T08:28:32.120 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:32.115+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.120154+0000) 2019-10-13T08:28:32.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:32.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.120445+0000) 2019-10-13T08:28:32.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:32.115+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.121340+0000) 2019-10-13T08:28:32.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:32.119+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.122378+0000) 2019-10-13T08:28:32.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:32.119+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:32.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:32.119+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.123165+0000) 2019-10-13T08:28:32.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:32.119+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.123354+0000) 2019-10-13T08:28:32.128 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:32.125+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:32.128428+0000) 2019-10-13T08:28:34.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:34.323+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:34.324236+0000) 2019-10-13T08:28:34.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:34.989+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:34.994520+0000) 2019-10-13T08:28:37.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:37.123+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.127067+0000) 2019-10-13T08:28:37.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:37.123+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.129101+0000) 2019-10-13T08:28:37.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:37.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:37.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:37.123+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.129713+0000) 2019-10-13T08:28:37.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:37.125+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.129635+0000) 2019-10-13T08:28:37.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:37.127+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.130416+0000) 2019-10-13T08:28:37.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:37.127+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.131394+0000) 2019-10-13T08:28:37.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:37.127+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.130544+0000) 2019-10-13T08:28:37.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:37.129+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.135504+0000) 2019-10-13T08:28:37.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:37.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.163143+0000) 2019-10-13T08:28:37.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:37.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.452683+0000) 2019-10-13T08:28:37.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:37.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.550617+0000) 2019-10-13T08:28:37.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:37.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:37.576925+0000) 2019-10-13T08:28:39.782 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:28:39.782 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:39.790 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:28:39.790 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:28:41.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:41.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:41.598362+0000) 2019-10-13T08:28:41.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:41.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:41.599930+0000) 2019-10-13T08:28:42.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:42.131+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.134367+0000) 2019-10-13T08:28:42.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:42.131+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.134089+0000) 2019-10-13T08:28:42.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:42.131+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.135691+0000) 2019-10-13T08:28:42.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:42.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.135715+0000) 2019-10-13T08:28:42.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:42.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:42.137 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:42.131+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.136594+0000) 2019-10-13T08:28:42.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:42.131+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.137047+0000) 2019-10-13T08:28:42.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:42.133+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.138144+0000) 2019-10-13T08:28:42.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:42.135+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:42.139936+0000) 2019-10-13T08:28:44.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:44.323+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:44.324395+0000) 2019-10-13T08:28:45.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:44.997+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:45.001036+0000) 2019-10-13T08:28:47.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:47.135+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.139151+0000) 2019-10-13T08:28:47.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:47.135+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.141517+0000) 2019-10-13T08:28:47.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:47.135+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.141706+0000) 2019-10-13T08:28:47.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:47.139+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:47.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:47.139+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.142192+0000) 2019-10-13T08:28:47.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:47.139+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.142402+0000) 2019-10-13T08:28:47.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:47.137+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.142277+0000) 2019-10-13T08:28:47.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:47.139+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.143191+0000) 2019-10-13T08:28:47.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:47.141+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.146639+0000) 2019-10-13T08:28:47.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:47.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.163340+0000) 2019-10-13T08:28:47.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:47.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.452890+0000) 2019-10-13T08:28:47.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:47.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.550790+0000) 2019-10-13T08:28:47.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:47.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:47.577120+0000) 2019-10-13T08:28:51.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:51.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:51.598568+0000) 2019-10-13T08:28:51.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:51.597+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:51.600140+0000) 2019-10-13T08:28:52.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:52.139+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.145465+0000) 2019-10-13T08:28:52.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:52.143+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.146107+0000) 2019-10-13T08:28:52.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:52.143+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.146942+0000) 2019-10-13T08:28:52.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:52.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73628 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:52.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:52.143+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.149285+0000) 2019-10-13T08:28:52.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:52.145+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.149389+0000) 2019-10-13T08:28:52.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:52.147+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.149708+0000) 2019-10-13T08:28:52.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:52.147+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.150322+0000) 2019-10-13T08:28:52.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:52.149+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:52.154715+0000) 2019-10-13T08:28:54.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:54.319+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:54.324571+0000) 2019-10-13T08:28:55.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:55.005+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:55.007424+0000) 2019-10-13T08:28:57.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:57.147+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.152753+0000) 2019-10-13T08:28:57.153 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:28:57.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.152389+0000) 2019-10-13T08:28:57.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:57.147+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.153118+0000) 2019-10-13T08:28:57.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:28:57.147+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.153650+0000) 2019-10-13T08:28:57.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:28:57.151+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.154114+0000) 2019-10-13T08:28:57.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:57.151+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.155091+0000) 2019-10-13T08:28:57.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:28:57.151+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:28:57.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:57.151+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.155498+0000) 2019-10-13T08:28:57.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:28:57.157+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.160660+0000) 2019-10-13T08:28:57.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:28:57.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.163459+0000) 2019-10-13T08:28:57.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:28:57.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.453066+0000) 2019-10-13T08:28:57.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:28:57.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.550968+0000) 2019-10-13T08:28:57.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:28:57.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:28:57.577288+0000) 2019-10-13T08:29:01.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:01.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:01.598768+0000) 2019-10-13T08:29:01.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:01.597+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:01.600346+0000) 2019-10-13T08:29:02.160 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:02.152+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.159089+0000) 2019-10-13T08:29:02.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:02.155+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.159519+0000) 2019-10-13T08:29:02.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:02.155+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.160120+0000) 2019-10-13T08:29:02.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:02.159+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:02.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:02.159+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.161907+0000) 2019-10-13T08:29:02.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:02.159+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.162578+0000) 2019-10-13T08:29:02.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:02.159+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.163243+0000) 2019-10-13T08:29:02.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:02.159+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.165463+0000) 2019-10-13T08:29:02.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:02.164+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:02.169903+0000) 2019-10-13T08:29:04.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:04.319+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:04.324761+0000) 2019-10-13T08:29:05.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:05.012+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:05.013975+0000) 2019-10-13T08:29:07.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:07.159+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.164965+0000) 2019-10-13T08:29:07.167 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:07.163+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.167032+0000) 2019-10-13T08:29:07.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:07.160+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.167150+0000) 2019-10-13T08:29:07.168 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:07.163+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.167927+0000) 2019-10-13T08:29:07.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:07.163+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.168404+0000) 2019-10-13T08:29:07.169 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:07.163+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:07.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:07.163+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.168317+0000) 2019-10-13T08:29:07.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:07.163+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.168904+0000) 2019-10-13T08:29:07.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:07.168+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.173507+0000) 2019-10-13T08:29:07.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:07.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.453198+0000) 2019-10-13T08:29:07.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:07.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.551137+0000) 2019-10-13T08:29:07.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:07.571+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:07.577462+0000) 2019-10-13T08:29:09.897 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:29:09.897 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:09.901 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:29:09.901 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:11.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:11.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:11.598992+0000) 2019-10-13T08:29:11.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:11.596+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:11.600570+0000) 2019-10-13T08:29:12.172 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:12.167+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.172388+0000) 2019-10-13T08:29:12.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:12.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:12.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:12.168+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.174083+0000) 2019-10-13T08:29:12.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:12.171+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.174732+0000) 2019-10-13T08:29:12.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:12.171+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.174980+0000) 2019-10-13T08:29:12.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:12.171+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.175455+0000) 2019-10-13T08:29:12.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:12.171+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.175897+0000) 2019-10-13T08:29:12.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:12.171+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.176413+0000) 2019-10-13T08:29:12.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:12.176+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:12.182234+0000) 2019-10-13T08:29:13.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:13.911+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:13.914232+0000) 2019-10-13T08:29:14.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:14.323+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:14.324928+0000) 2019-10-13T08:29:15.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:15.016+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:15.020470+0000) 2019-10-13T08:29:17.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:17.175+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.178467+0000) 2019-10-13T08:29:17.179 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:17.175+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.178953+0000) 2019-10-13T08:29:17.179 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:17.172+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.178734+0000) 2019-10-13T08:29:17.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:17.175+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.179591+0000) 2019-10-13T08:29:17.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:17.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:17.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:17.175+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.181097+0000) 2019-10-13T08:29:17.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:17.175+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.181161+0000) 2019-10-13T08:29:17.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:17.179+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.181944+0000) 2019-10-13T08:29:17.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:17.176+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.183384+0000) 2019-10-13T08:29:17.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:17.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.453382+0000) 2019-10-13T08:29:17.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:17.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.551316+0000) 2019-10-13T08:29:17.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:17.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:17.577633+0000) 2019-10-13T08:29:21.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:21.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:21.599189+0000) 2019-10-13T08:29:21.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:21.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:21.600797+0000) 2019-10-13T08:29:22.183 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:22.176+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.182755+0000) 2019-10-13T08:29:22.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:22.179+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.183737+0000) 2019-10-13T08:29:22.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:22.179+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.185317+0000) 2019-10-13T08:29:22.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:22.183+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.186011+0000) 2019-10-13T08:29:22.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:22.183+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.186381+0000) 2019-10-13T08:29:22.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:22.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:22.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:22.183+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.187839+0000) 2019-10-13T08:29:22.188 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:22.183+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.188441+0000) 2019-10-13T08:29:22.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:22.188+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:22.193866+0000) 2019-10-13T08:29:23.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:23.911+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:23.914461+0000) 2019-10-13T08:29:24.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:24.323+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:24.325086+0000) 2019-10-13T08:29:25.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:25.024+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:25.027010+0000) 2019-10-13T08:29:27.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:27.184+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.190072+0000) 2019-10-13T08:29:27.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:27.187+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.191168+0000) 2019-10-13T08:29:27.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:27.187+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.191471+0000) 2019-10-13T08:29:27.193 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:27.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:27.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:27.187+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.193475+0000) 2019-10-13T08:29:27.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:27.191+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.193920+0000) 2019-10-13T08:29:27.195 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:27.191+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.194228+0000) 2019-10-13T08:29:27.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:27.191+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.194595+0000) 2019-10-13T08:29:27.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:27.196+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.201760+0000) 2019-10-13T08:29:27.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:27.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.453564+0000) 2019-10-13T08:29:27.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:27.547+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.551494+0000) 2019-10-13T08:29:27.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:27.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:27.577806+0000) 2019-10-13T08:29:31.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:31.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:31.599337+0000) 2019-10-13T08:29:31.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:31.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:31.600992+0000) 2019-10-13T08:29:32.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:32.190+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.196420+0000) 2019-10-13T08:29:32.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:32.190+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.196943+0000) 2019-10-13T08:29:32.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:32.190+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.197652+0000) 2019-10-13T08:29:32.198 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:32.194+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.197728+0000) 2019-10-13T08:29:32.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:32.194+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.198147+0000) 2019-10-13T08:29:32.199 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:32.192+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.199116+0000) 2019-10-13T08:29:32.200 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:32.194+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.199708+0000) 2019-10-13T08:29:32.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:32.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:32.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:32.200+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:32.205035+0000) 2019-10-13T08:29:33.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:33.910+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:33.914658+0000) 2019-10-13T08:29:34.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:34.318+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:34.325295+0000) 2019-10-13T08:29:35.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:35.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:35.037122+0000) 2019-10-13T08:29:37.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:37.198+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.203772+0000) 2019-10-13T08:29:37.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:37.198+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.204411+0000) 2019-10-13T08:29:37.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:37.200+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.204367+0000) 2019-10-13T08:29:37.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:37.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:37.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:37.202+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.207013+0000) 2019-10-13T08:29:37.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:37.202+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.207728+0000) 2019-10-13T08:29:37.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:37.202+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.207826+0000) 2019-10-13T08:29:37.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:37.202+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.208870+0000) 2019-10-13T08:29:37.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:37.212+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.216306+0000) 2019-10-13T08:29:37.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:37.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.453780+0000) 2019-10-13T08:29:37.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:37.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.551669+0000) 2019-10-13T08:29:37.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:37.574+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:37.577979+0000) 2019-10-13T08:29:40.022 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:29:40.022 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:40.026 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:29:40.026 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:29:41.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:41.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:41.599535+0000) 2019-10-13T08:29:41.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:41.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:41.601214+0000) 2019-10-13T08:29:42.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:42.206+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.211514+0000) 2019-10-13T08:29:42.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:42.204+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.211879+0000) 2019-10-13T08:29:42.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:42.206+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.212164+0000) 2019-10-13T08:29:42.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:42.206+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.212901+0000) 2019-10-13T08:29:42.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:42.206+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.213323+0000) 2019-10-13T08:29:42.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:42.210+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:42.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:42.210+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.213925+0000) 2019-10-13T08:29:42.214 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:42.210+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.214179+0000) 2019-10-13T08:29:42.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:42.216+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:42.221104+0000) 2019-10-13T08:29:43.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:43.914+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:43.914854+0000) 2019-10-13T08:29:44.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:44.318+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:44.325451+0000) 2019-10-13T08:29:45.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:45.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:45.043758+0000) 2019-10-13T08:29:47.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:47.212+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.217441+0000) 2019-10-13T08:29:47.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:47.210+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.217710+0000) 2019-10-13T08:29:47.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:47.214+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.218222+0000) 2019-10-13T08:29:47.219 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:47.214+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.218581+0000) 2019-10-13T08:29:47.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:47.214+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.219381+0000) 2019-10-13T08:29:47.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:47.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:47.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:47.214+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.220358+0000) 2019-10-13T08:29:47.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:47.214+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.220631+0000) 2019-10-13T08:29:47.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:47.220+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.225443+0000) 2019-10-13T08:29:47.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:47.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.453983+0000) 2019-10-13T08:29:47.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:47.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.551868+0000) 2019-10-13T08:29:47.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:47.574+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:47.578162+0000) 2019-10-13T08:29:51.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:51.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:51.599730+0000) 2019-10-13T08:29:51.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:51.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:51.601432+0000) 2019-10-13T08:29:52.224 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:52.218+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.223681+0000) 2019-10-13T08:29:52.224 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:52.218+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.224010+0000) 2019-10-13T08:29:52.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:52.216+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.223893+0000) 2019-10-13T08:29:52.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:52.218+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.224590+0000) 2019-10-13T08:29:52.226 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:52.222+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.225827+0000) 2019-10-13T08:29:52.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:52.222+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.226281+0000) 2019-10-13T08:29:52.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:52.222+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.226637+0000) 2019-10-13T08:29:52.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:52.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:52.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:52.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:52.232729+0000) 2019-10-13T08:29:53.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:53.910+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:53.915042+0000) 2019-10-13T08:29:54.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:54.322+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:54.325677+0000) 2019-10-13T08:29:55.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:55.052+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:55.053740+0000) 2019-10-13T08:29:57.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:57.226+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.231032+0000) 2019-10-13T08:29:57.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:57.226+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.231467+0000) 2019-10-13T08:29:57.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:29:57.226+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.232689+0000) 2019-10-13T08:29:57.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:29:57.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:29:57.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:29:57.230+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.234088+0000) 2019-10-13T08:29:57.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:29:57.230+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.234594+0000) 2019-10-13T08:29:57.235 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:29:57.228+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.235284+0000) 2019-10-13T08:29:57.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:57.230+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.236127+0000) 2019-10-13T08:29:57.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:29:57.236+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.243678+0000) 2019-10-13T08:29:57.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:29:57.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.454184+0000) 2019-10-13T08:29:57.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:29:57.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.552057+0000) 2019-10-13T08:29:57.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:29:57.574+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:29:57.578338+0000) 2019-10-13T08:30:01.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:01.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:01.599940+0000) 2019-10-13T08:30:01.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:01.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:01.601627+0000) 2019-10-13T08:30:02.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:02.234+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.237796+0000) 2019-10-13T08:30:02.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:02.234+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.238379+0000) 2019-10-13T08:30:02.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:02.234+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.238637+0000) 2019-10-13T08:30:02.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:02.234+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.239977+0000) 2019-10-13T08:30:02.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:02.234+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:02.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:02.236+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.240122+0000) 2019-10-13T08:30:02.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:02.234+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.240207+0000) 2019-10-13T08:30:02.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:02.234+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.240979+0000) 2019-10-13T08:30:02.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:02.236+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:02.243365+0000) 2019-10-13T08:30:03.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:03.910+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:03.915238+0000) 2019-10-13T08:30:04.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:04.322+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:04.325911+0000) 2019-10-13T08:30:05.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:05.060+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:05.060197+0000) 2019-10-13T08:30:07.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:07.238+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.243610+0000) 2019-10-13T08:30:07.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:07.240+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.243476+0000) 2019-10-13T08:30:07.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:07.238+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.244337+0000) 2019-10-13T08:30:07.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:07.238+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.244818+0000) 2019-10-13T08:30:07.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:07.238+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.245091+0000) 2019-10-13T08:30:07.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:07.238+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.245682+0000) 2019-10-13T08:30:07.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:07.242+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.246039+0000) 2019-10-13T08:30:07.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:07.242+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:07.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:07.252+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.252372+0000) 2019-10-13T08:30:07.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:07.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.454382+0000) 2019-10-13T08:30:07.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:07.546+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.552226+0000) 2019-10-13T08:30:07.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:07.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:07.578511+0000) 2019-10-13T08:30:10.135 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:30:10.135 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:10.139 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:30:10.139 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:11.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:11.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:11.600135+0000) 2019-10-13T08:30:11.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:11.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:11.601826+0000) 2019-10-13T08:30:12.251 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:12.246+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.250726+0000) 2019-10-13T08:30:12.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:12.246+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.251253+0000) 2019-10-13T08:30:12.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:12.246+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.251722+0000) 2019-10-13T08:30:12.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:12.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74099 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:12.253 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:12.251+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.252867+0000) 2019-10-13T08:30:12.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:12.246+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.253696+0000) 2019-10-13T08:30:12.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:12.250+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.254010+0000) 2019-10-13T08:30:12.255 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:12.250+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.255590+0000) 2019-10-13T08:30:12.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:12.259+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:12.260580+0000) 2019-10-13T08:30:13.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:13.914+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:13.915444+0000) 2019-10-13T08:30:14.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:14.322+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:14.326105+0000) 2019-10-13T08:30:15.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:15.063+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:15.066692+0000) 2019-10-13T08:30:17.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:17.250+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.257054+0000) 2019-10-13T08:30:17.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:17.250+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.257646+0000) 2019-10-13T08:30:17.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:17.254+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.258159+0000) 2019-10-13T08:30:17.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:17.255+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.258629+0000) 2019-10-13T08:30:17.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:17.254+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.259389+0000) 2019-10-13T08:30:17.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:17.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74129 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:17.260 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:17.254+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.259646+0000) 2019-10-13T08:30:17.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:17.254+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.260317+0000) 2019-10-13T08:30:17.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:17.259+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.263196+0000) 2019-10-13T08:30:17.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:17.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.454582+0000) 2019-10-13T08:30:17.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:17.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.552407+0000) 2019-10-13T08:30:17.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:17.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:17.578724+0000) 2019-10-13T08:30:21.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:21.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:21.600345+0000) 2019-10-13T08:30:21.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:21.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:21.602028+0000) 2019-10-13T08:30:22.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:22.258+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.263351+0000) 2019-10-13T08:30:22.264 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:22.258+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.263642+0000) 2019-10-13T08:30:22.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:22.259+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.263481+0000) 2019-10-13T08:30:22.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:22.258+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.265246+0000) 2019-10-13T08:30:22.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:22.258+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.264941+0000) 2019-10-13T08:30:22.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:22.258+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.265574+0000) 2019-10-13T08:30:22.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:22.262+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.265977+0000) 2019-10-13T08:30:22.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:22.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:22.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:22.271+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:22.272607+0000) 2019-10-13T08:30:23.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:23.914+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:23.915697+0000) 2019-10-13T08:30:24.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:24.326+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:24.326305+0000) 2019-10-13T08:30:25.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:25.071+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:25.073228+0000) 2019-10-13T08:30:27.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:27.265+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.270015+0000) 2019-10-13T08:30:27.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:27.267+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.269920+0000) 2019-10-13T08:30:27.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:27.265+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.270997+0000) 2019-10-13T08:30:27.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:27.265+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.271510+0000) 2019-10-13T08:30:27.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:27.265+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.271912+0000) 2019-10-13T08:30:27.272 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:27.265+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.272413+0000) 2019-10-13T08:30:27.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:27.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:27.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:27.269+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.273872+0000) 2019-10-13T08:30:27.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:27.275+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.276734+0000) 2019-10-13T08:30:27.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:27.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.454778+0000) 2019-10-13T08:30:27.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:27.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.552605+0000) 2019-10-13T08:30:27.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:27.573+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:27.578930+0000) 2019-10-13T08:30:31.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:31.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:31.600599+0000) 2019-10-13T08:30:31.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:31.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:31.602230+0000) 2019-10-13T08:30:32.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:32.269+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.276529+0000) 2019-10-13T08:30:32.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:32.269+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.276963+0000) 2019-10-13T08:30:32.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:32.275+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.277116+0000) 2019-10-13T08:30:32.278 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:32.273+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.278378+0000) 2019-10-13T08:30:32.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:32.273+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.278960+0000) 2019-10-13T08:30:32.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:32.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:32.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:32.273+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.281024+0000) 2019-10-13T08:30:32.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:32.273+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.281556+0000) 2019-10-13T08:30:32.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:32.279+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:32.283114+0000) 2019-10-13T08:30:33.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:33.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:33.915943+0000) 2019-10-13T08:30:34.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:34.325+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:34.326538+0000) 2019-10-13T08:30:35.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:35.079+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:35.079737+0000) 2019-10-13T08:30:37.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:37.277+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.283836+0000) 2019-10-13T08:30:37.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:37.277+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.284447+0000) 2019-10-13T08:30:37.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:37.277+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.284616+0000) 2019-10-13T08:30:37.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:37.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74247 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:37.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:37.283+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.286705+0000) 2019-10-13T08:30:37.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:37.281+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.287493+0000) 2019-10-13T08:30:37.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:37.281+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.287832+0000) 2019-10-13T08:30:37.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:37.281+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.288630+0000) 2019-10-13T08:30:37.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:37.291+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.294187+0000) 2019-10-13T08:30:37.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:37.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.454982+0000) 2019-10-13T08:30:37.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:37.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.552810+0000) 2019-10-13T08:30:37.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:37.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:37.579101+0000) 2019-10-13T08:30:40.250 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:30:40.250 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:40.254 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:30:40.254 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:30:41.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:41.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:41.600794+0000) 2019-10-13T08:30:41.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:41.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:41.602458+0000) 2019-10-13T08:30:42.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:42.289+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.290267+0000) 2019-10-13T08:30:42.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:42.287+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.290776+0000) 2019-10-13T08:30:42.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:42.289+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.291589+0000) 2019-10-13T08:30:42.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:42.289+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.292030+0000) 2019-10-13T08:30:42.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:42.289+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:42.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:42.289+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.292907+0000) 2019-10-13T08:30:42.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:42.289+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.293013+0000) 2019-10-13T08:30:42.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:42.289+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.293140+0000) 2019-10-13T08:30:42.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:42.295+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:42.298958+0000) 2019-10-13T08:30:43.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:43.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:43.916147+0000) 2019-10-13T08:30:44.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:44.325+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:44.326735+0000) 2019-10-13T08:30:45.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:45.087+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:45.091047+0000) 2019-10-13T08:30:47.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:47.291+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.295289+0000) 2019-10-13T08:30:47.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:47.293+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.296150+0000) 2019-10-13T08:30:47.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:47.293+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.296390+0000) 2019-10-13T08:30:47.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:47.293+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.297494+0000) 2019-10-13T08:30:47.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:47.297+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.298214+0000) 2019-10-13T08:30:47.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:47.297+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.298826+0000) 2019-10-13T08:30:47.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:47.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:47.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:47.297+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.299720+0000) 2019-10-13T08:30:47.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:47.303+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.304214+0000) 2019-10-13T08:30:47.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:47.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.455175+0000) 2019-10-13T08:30:47.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:47.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.552940+0000) 2019-10-13T08:30:47.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:47.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:47.579275+0000) 2019-10-13T08:30:51.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:51.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:51.600986+0000) 2019-10-13T08:30:51.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:51.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:51.602659+0000) 2019-10-13T08:30:52.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:52.301+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.302503+0000) 2019-10-13T08:30:52.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:52.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.302925+0000) 2019-10-13T08:30:52.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:52.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.304884+0000) 2019-10-13T08:30:52.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:52.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.304704+0000) 2019-10-13T08:30:52.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:52.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:52.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:52.301+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.305639+0000) 2019-10-13T08:30:52.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:52.305+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.306297+0000) 2019-10-13T08:30:52.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:52.305+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.305799+0000) 2019-10-13T08:30:52.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:52.311+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:52.312579+0000) 2019-10-13T08:30:53.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:53.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:53.916373+0000) 2019-10-13T08:30:54.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:54.325+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:54.326963+0000) 2019-10-13T08:30:55.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:55.095+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:55.097561+0000) 2019-10-13T08:30:57.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:30:57.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.307890+0000) 2019-10-13T08:30:57.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:30:57.305+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.308951+0000) 2019-10-13T08:30:57.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:57.309+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.309820+0000) 2019-10-13T08:30:57.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:57.309+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.310096+0000) 2019-10-13T08:30:57.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:30:57.309+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.310628+0000) 2019-10-13T08:30:57.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:30:57.309+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.310965+0000) 2019-10-13T08:30:57.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:57.309+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.311282+0000) 2019-10-13T08:30:57.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:30:57.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:30:57.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:30:57.315+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.318029+0000) 2019-10-13T08:30:57.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:30:57.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.455354+0000) 2019-10-13T08:30:57.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:30:57.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.553138+0000) 2019-10-13T08:30:57.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:30:57.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:30:57.579454+0000) 2019-10-13T08:31:01.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:01.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:01.601183+0000) 2019-10-13T08:31:01.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:01.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:01.602857+0000) 2019-10-13T08:31:02.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:02.313+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.316476+0000) 2019-10-13T08:31:02.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:02.313+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.316560+0000) 2019-10-13T08:31:02.317 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:02.315+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.316388+0000) 2019-10-13T08:31:02.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:02.313+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.317642+0000) 2019-10-13T08:31:02.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:02.317+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.318024+0000) 2019-10-13T08:31:02.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:02.317+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.318495+0000) 2019-10-13T08:31:02.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:02.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:02.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:02.317+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.319349+0000) 2019-10-13T08:31:02.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:02.319+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:02.323193+0000) 2019-10-13T08:31:03.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:03.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:03.916574+0000) 2019-10-13T08:31:04.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:04.325+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:04.327217+0000) 2019-10-13T08:31:05.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:05.099+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:05.104058+0000) 2019-10-13T08:31:07.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:07.321+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.323624+0000) 2019-10-13T08:31:07.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:07.319+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.323423+0000) 2019-10-13T08:31:07.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:07.321+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.324252+0000) 2019-10-13T08:31:07.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:07.321+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.324819+0000) 2019-10-13T08:31:07.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:07.321+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.325296+0000) 2019-10-13T08:31:07.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:07.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:07.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:07.325+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.325940+0000) 2019-10-13T08:31:07.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:07.325+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.326243+0000) 2019-10-13T08:31:07.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:07.331+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.332615+0000) 2019-10-13T08:31:07.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:07.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.455529+0000) 2019-10-13T08:31:07.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:07.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.553312+0000) 2019-10-13T08:31:07.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:07.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:07.579633+0000) 2019-10-13T08:31:10.362 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:31:10.363 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:10.367 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:31:10.367 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:11.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:11.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:11.601375+0000) 2019-10-13T08:31:11.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:11.599+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:11.603056+0000) 2019-10-13T08:31:12.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:12.323+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.327502+0000) 2019-10-13T08:31:12.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:12.325+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.328912+0000) 2019-10-13T08:31:12.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:12.325+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.328978+0000) 2019-10-13T08:31:12.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:12.325+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.329515+0000) 2019-10-13T08:31:12.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:12.329+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.329842+0000) 2019-10-13T08:31:12.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:12.329+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.330441+0000) 2019-10-13T08:31:12.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:12.329+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.332071+0000) 2019-10-13T08:31:12.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:12.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:12.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:12.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:12.338353+0000) 2019-10-13T08:31:13.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:13.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:13.916778+0000) 2019-10-13T08:31:14.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:14.325+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:14.327445+0000) 2019-10-13T08:31:15.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:15.106+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:15.110565+0000) 2019-10-13T08:31:17.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:17.333+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.336256+0000) 2019-10-13T08:31:17.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:17.333+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.336980+0000) 2019-10-13T08:31:17.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:17.333+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.337262+0000) 2019-10-13T08:31:17.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:17.334+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.337157+0000) 2019-10-13T08:31:17.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:17.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:17.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:17.337+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.339669+0000) 2019-10-13T08:31:17.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:17.341+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.342545+0000) 2019-10-13T08:31:17.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:17.341+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.343391+0000) 2019-10-13T08:31:17.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:17.342+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.344738+0000) 2019-10-13T08:31:17.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:17.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.455739+0000) 2019-10-13T08:31:17.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:17.549+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.553492+0000) 2019-10-13T08:31:17.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:17.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:17.579804+0000) 2019-10-13T08:31:21.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:21.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:21.601571+0000) 2019-10-13T08:31:21.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:21.602+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:21.603255+0000) 2019-10-13T08:31:22.342 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:22.340+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.342213+0000) 2019-10-13T08:31:22.343 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:22.338+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.342607+0000) 2019-10-13T08:31:22.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:22.340+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.343379+0000) 2019-10-13T08:31:22.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:22.340+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.343633+0000) 2019-10-13T08:31:22.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:22.340+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.344764+0000) 2019-10-13T08:31:22.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:22.340+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:22.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:22.340+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.345553+0000) 2019-10-13T08:31:22.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:22.340+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.345458+0000) 2019-10-13T08:31:22.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:22.346+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:22.348837+0000) 2019-10-13T08:31:23.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:23.912+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:23.917000+0000) 2019-10-13T08:31:24.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:24.324+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:24.327644+0000) 2019-10-13T08:31:25.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:25.114+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:25.117037+0000) 2019-10-13T08:31:27.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:27.344+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.349174+0000) 2019-10-13T08:31:27.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:27.344+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.349652+0000) 2019-10-13T08:31:27.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:27.348+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.350280+0000) 2019-10-13T08:31:27.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:27.348+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.350722+0000) 2019-10-13T08:31:27.351 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:27.346+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.350561+0000) 2019-10-13T08:31:27.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:27.348+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.351369+0000) 2019-10-13T08:31:27.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:27.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74542 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:27.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:27.348+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.352668+0000) 2019-10-13T08:31:27.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:27.354+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.357206+0000) 2019-10-13T08:31:27.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:27.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.455949+0000) 2019-10-13T08:31:27.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:27.548+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.553698+0000) 2019-10-13T08:31:27.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:27.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:27.579982+0000) 2019-10-13T08:31:31.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:31.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:31.601771+0000) 2019-10-13T08:31:31.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:31.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:31.603461+0000) 2019-10-13T08:31:32.356 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:32.352+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.356056+0000) 2019-10-13T08:31:32.357 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:32.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:32.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:32.356+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.358418+0000) 2019-10-13T08:31:32.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:32.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.358763+0000) 2019-10-13T08:31:32.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:32.354+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.358559+0000) 2019-10-13T08:31:32.359 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:32.356+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.359091+0000) 2019-10-13T08:31:32.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:32.356+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.361125+0000) 2019-10-13T08:31:32.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:32.356+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.361263+0000) 2019-10-13T08:31:32.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:32.358+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:32.363700+0000) 2019-10-13T08:31:33.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:33.916+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:33.917202+0000) 2019-10-13T08:31:34.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:34.324+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:34.327871+0000) 2019-10-13T08:31:35.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:35.118+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:35.123554+0000) 2019-10-13T08:31:37.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:37.356+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.361381+0000) 2019-10-13T08:31:37.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:37.360+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.362878+0000) 2019-10-13T08:31:37.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:37.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:37.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:37.358+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.363826+0000) 2019-10-13T08:31:37.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:37.360+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.364690+0000) 2019-10-13T08:31:37.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:37.360+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.365305+0000) 2019-10-13T08:31:37.366 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:37.364+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.365797+0000) 2019-10-13T08:31:37.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:37.364+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.366436+0000) 2019-10-13T08:31:37.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:37.362+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.367884+0000) 2019-10-13T08:31:37.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:37.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.456160+0000) 2019-10-13T08:31:37.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:37.552+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.553970+0000) 2019-10-13T08:31:37.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:37.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:37.580154+0000) 2019-10-13T08:31:40.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:31:40.432 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:40.436 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:31:40.436 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:31:41.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:41.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:41.601987+0000) 2019-10-13T08:31:41.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:41.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:41.603654+0000) 2019-10-13T08:31:42.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:42.364+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.366561+0000) 2019-10-13T08:31:42.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:42.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.367257+0000) 2019-10-13T08:31:42.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:42.364+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.367916+0000) 2019-10-13T08:31:42.369 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:42.366+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.368822+0000) 2019-10-13T08:31:42.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:42.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:42.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:42.368+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.371104+0000) 2019-10-13T08:31:42.371 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:42.368+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.371300+0000) 2019-10-13T08:31:42.371 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:42.368+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.371644+0000) 2019-10-13T08:31:42.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:42.374+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:42.376279+0000) 2019-10-13T08:31:43.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:43.912+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:43.917399+0000) 2019-10-13T08:31:44.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:44.324+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:44.328071+0000) 2019-10-13T08:31:45.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:45.126+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:45.130071+0000) 2019-10-13T08:31:47.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:47.368+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.373364+0000) 2019-10-13T08:31:47.374 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:47.372+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.374451+0000) 2019-10-13T08:31:47.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:47.372+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.375455+0000) 2019-10-13T08:31:47.376 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:47.372+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.375844+0000) 2019-10-13T08:31:47.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:47.372+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.376524+0000) 2019-10-13T08:31:47.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:47.372+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.376778+0000) 2019-10-13T08:31:47.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:47.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:47.377 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:47.374+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.376368+0000) 2019-10-13T08:31:47.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:47.374+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.379946+0000) 2019-10-13T08:31:47.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:47.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.456355+0000) 2019-10-13T08:31:47.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:47.552+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.554163+0000) 2019-10-13T08:31:47.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:47.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:47.580351+0000) 2019-10-13T08:31:51.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:51.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:51.602216+0000) 2019-10-13T08:31:51.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:51.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:51.603856+0000) 2019-10-13T08:31:52.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:52.376+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.380346+0000) 2019-10-13T08:31:52.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:52.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:52.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:52.380+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.383040+0000) 2019-10-13T08:31:52.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:52.380+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.383200+0000) 2019-10-13T08:31:52.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:52.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.383498+0000) 2019-10-13T08:31:52.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:52.380+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.384094+0000) 2019-10-13T08:31:52.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:52.380+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.384833+0000) 2019-10-13T08:31:52.386 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:52.382+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.385733+0000) 2019-10-13T08:31:52.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:52.386+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:52.389669+0000) 2019-10-13T08:31:53.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:53.912+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:53.917595+0000) 2019-10-13T08:31:54.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:54.324+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:54.328316+0000) 2019-10-13T08:31:55.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:55.138+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:55.139874+0000) 2019-10-13T08:31:57.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:31:57.384+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.385889+0000) 2019-10-13T08:31:57.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:31:57.384+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.388949+0000) 2019-10-13T08:31:57.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:57.384+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.389137+0000) 2019-10-13T08:31:57.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:31:57.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74723 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:31:57.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:31:57.388+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.390635+0000) 2019-10-13T08:31:57.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:57.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.391032+0000) 2019-10-13T08:31:57.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:57.388+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.389771+0000) 2019-10-13T08:31:57.391 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:31:57.386+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.391177+0000) 2019-10-13T08:31:57.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:31:57.390+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.392504+0000) 2019-10-13T08:31:57.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:31:57.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.456553+0000) 2019-10-13T08:31:57.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:31:57.552+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.554383+0000) 2019-10-13T08:31:57.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:31:57.580+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:31:57.580524+0000) 2019-10-13T08:32:01.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:01.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:01.602411+0000) 2019-10-13T08:32:01.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:01.598+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:01.604058+0000) 2019-10-13T08:32:02.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:02.388+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.392652+0000) 2019-10-13T08:32:02.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:02.388+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.393706+0000) 2019-10-13T08:32:02.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:02.392+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.394097+0000) 2019-10-13T08:32:02.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:02.392+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.394624+0000) 2019-10-13T08:32:02.395 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:02.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.394790+0000) 2019-10-13T08:32:02.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:02.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:02.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:02.392+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.395826+0000) 2019-10-13T08:32:02.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:02.392+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.396211+0000) 2019-10-13T08:32:02.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:02.398+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:02.401622+0000) 2019-10-13T08:32:03.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:03.916+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:03.917816+0000) 2019-10-13T08:32:04.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:04.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:04.328546+0000) 2019-10-13T08:32:05.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:05.142+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:05.145970+0000) 2019-10-13T08:32:07.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:07.396+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.398611+0000) 2019-10-13T08:32:07.400 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:07.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.400078+0000) 2019-10-13T08:32:07.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:07.396+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.401555+0000) 2019-10-13T08:32:07.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:07.400+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.401982+0000) 2019-10-13T08:32:07.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:07.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:07.403 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:07.400+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.402279+0000) 2019-10-13T08:32:07.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:07.400+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.402659+0000) 2019-10-13T08:32:07.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:07.400+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.403315+0000) 2019-10-13T08:32:07.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:07.406+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.409772+0000) 2019-10-13T08:32:07.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:07.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.456729+0000) 2019-10-13T08:32:07.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:07.552+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.554561+0000) 2019-10-13T08:32:07.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:07.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:07.580722+0000) 2019-10-13T08:32:10.565 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:32:10.565 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:10.569 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:32:10.569 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:11.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:11.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:11.602608+0000) 2019-10-13T08:32:11.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:11.602+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:11.604288+0000) 2019-10-13T08:32:12.405 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:12.400+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.405001+0000) 2019-10-13T08:32:12.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:12.404+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.406084+0000) 2019-10-13T08:32:12.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:12.404+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.406549+0000) 2019-10-13T08:32:12.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:12.404+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.407553+0000) 2019-10-13T08:32:12.408 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:12.402+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.407414+0000) 2019-10-13T08:32:12.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:12.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:12.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:12.404+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.408643+0000) 2019-10-13T08:32:12.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:12.404+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.409230+0000) 2019-10-13T08:32:12.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:12.410+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:12.413886+0000) 2019-10-13T08:32:13.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:13.916+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:13.918037+0000) 2019-10-13T08:32:14.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:14.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:14.328755+0000) 2019-10-13T08:32:15.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:15.150+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:15.152367+0000) 2019-10-13T08:32:17.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:17.408+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.411708+0000) 2019-10-13T08:32:17.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:17.408+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.412370+0000) 2019-10-13T08:32:17.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:17.408+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.412823+0000) 2019-10-13T08:32:17.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:17.412+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.413770+0000) 2019-10-13T08:32:17.414 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:17.412+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.414089+0000) 2019-10-13T08:32:17.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:17.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:17.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:17.412+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.414931+0000) 2019-10-13T08:32:17.415 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:17.410+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.414841+0000) 2019-10-13T08:32:17.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:17.418+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.420763+0000) 2019-10-13T08:32:17.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:17.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.456908+0000) 2019-10-13T08:32:17.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:17.552+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.554697+0000) 2019-10-13T08:32:17.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:17.576+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:17.580930+0000) 2019-10-13T08:32:21.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:21.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:21.602796+0000) 2019-10-13T08:32:21.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:21.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:21.604529+0000) 2019-10-13T08:32:22.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:22.411+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.417492+0000) 2019-10-13T08:32:22.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:22.415+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.419371+0000) 2019-10-13T08:32:22.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:22.415+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.419937+0000) 2019-10-13T08:32:22.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:22.415+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.421065+0000) 2019-10-13T08:32:22.421 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:22.417+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.420894+0000) 2019-10-13T08:32:22.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:22.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:22.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:22.419+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.422383+0000) 2019-10-13T08:32:22.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:22.419+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.423034+0000) 2019-10-13T08:32:22.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:22.421+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:22.427863+0000) 2019-10-13T08:32:23.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:23.915+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:23.918234+0000) 2019-10-13T08:32:24.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:24.323+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:24.328953+0000) 2019-10-13T08:32:25.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:25.157+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:25.158974+0000) 2019-10-13T08:32:27.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:27.419+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.423992+0000) 2019-10-13T08:32:27.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:27.419+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.424630+0000) 2019-10-13T08:32:27.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:27.419+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.425258+0000) 2019-10-13T08:32:27.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:27.419+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.425673+0000) 2019-10-13T08:32:27.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:27.421+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.425949+0000) 2019-10-13T08:32:27.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:27.423+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.427042+0000) 2019-10-13T08:32:27.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:27.423+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.427497+0000) 2019-10-13T08:32:27.428 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:27.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:27.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:27.429+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.433663+0000) 2019-10-13T08:32:27.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:27.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.457107+0000) 2019-10-13T08:32:27.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:27.551+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.554871+0000) 2019-10-13T08:32:27.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:27.579+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:27.581081+0000) 2019-10-13T08:32:31.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:31.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:31.603026+0000) 2019-10-13T08:32:31.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:31.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:31.604729+0000) 2019-10-13T08:32:32.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:32.427+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.431295+0000) 2019-10-13T08:32:32.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:32.427+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.431779+0000) 2019-10-13T08:32:32.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:32.427+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.432437+0000) 2019-10-13T08:32:32.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:32.427+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.432530+0000) 2019-10-13T08:32:32.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:32.429+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.433268+0000) 2019-10-13T08:32:32.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:32.427+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.433653+0000) 2019-10-13T08:32:32.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:32.431+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.434045+0000) 2019-10-13T08:32:32.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:32.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:32.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:32.433+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:32.439130+0000) 2019-10-13T08:32:33.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:33.915+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:33.918450+0000) 2019-10-13T08:32:34.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:34.327+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:34.329186+0000) 2019-10-13T08:32:35.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:35.161+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:35.165535+0000) 2019-10-13T08:32:37.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:37.435+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.437726+0000) 2019-10-13T08:32:37.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:37.435+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.438239+0000) 2019-10-13T08:32:37.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:37.435+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.439487+0000) 2019-10-13T08:32:37.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:37.435+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.439265+0000) 2019-10-13T08:32:37.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:37.435+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.440008+0000) 2019-10-13T08:32:37.440 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:37.435+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.440899+0000) 2019-10-13T08:32:37.441 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:37.437+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.440633+0000) 2019-10-13T08:32:37.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:37.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:37.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:37.441+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.444328+0000) 2019-10-13T08:32:37.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:37.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.457287+0000) 2019-10-13T08:32:37.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:37.551+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.555108+0000) 2019-10-13T08:32:37.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:37.579+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:37.581274+0000) 2019-10-13T08:32:40.696 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:32:40.697 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:40.700 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:32:40.700 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:32:41.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:41.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:41.603255+0000) 2019-10-13T08:32:41.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:41.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:41.604923+0000) 2019-10-13T08:32:42.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:42.439+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.443440+0000) 2019-10-13T08:32:42.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:42.439+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.444091+0000) 2019-10-13T08:32:42.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:42.443+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.446014+0000) 2019-10-13T08:32:42.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:42.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.446566+0000) 2019-10-13T08:32:42.447 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:42.441+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.446452+0000) 2019-10-13T08:32:42.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:42.443+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.447335+0000) 2019-10-13T08:32:42.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:42.443+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.447674+0000) 2019-10-13T08:32:42.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:42.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:42.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:42.445+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:42.450592+0000) 2019-10-13T08:32:43.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:43.915+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:43.918654+0000) 2019-10-13T08:32:44.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:44.327+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:44.329406+0000) 2019-10-13T08:32:45.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:45.169+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:45.172050+0000) 2019-10-13T08:32:47.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:47.447+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.451627+0000) 2019-10-13T08:32:47.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:47.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.451587+0000) 2019-10-13T08:32:47.453 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:47.447+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.452478+0000) 2019-10-13T08:32:47.455 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:47.451+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.454702+0000) 2019-10-13T08:32:47.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:47.451+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.454841+0000) 2019-10-13T08:32:47.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:47.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:47.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:47.451+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.456538+0000) 2019-10-13T08:32:47.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:47.451+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.456916+0000) 2019-10-13T08:32:47.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:47.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.457414+0000) 2019-10-13T08:32:47.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:47.453+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.458563+0000) 2019-10-13T08:32:47.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:47.551+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.555286+0000) 2019-10-13T08:32:47.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:47.575+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:47.581490+0000) 2019-10-13T08:32:51.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:51.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:51.603465+0000) 2019-10-13T08:32:51.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:51.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:51.605122+0000) 2019-10-13T08:32:52.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:52.455+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.457968+0000) 2019-10-13T08:32:52.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:52.455+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.458978+0000) 2019-10-13T08:32:52.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:52.453+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.459227+0000) 2019-10-13T08:32:52.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:52.455+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.460196+0000) 2019-10-13T08:32:52.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:52.455+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.460755+0000) 2019-10-13T08:32:52.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:52.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:52.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:52.455+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.461440+0000) 2019-10-13T08:32:52.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:52.459+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.462122+0000) 2019-10-13T08:32:52.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:52.461+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:52.464536+0000) 2019-10-13T08:32:53.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:53.915+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:53.918852+0000) 2019-10-13T08:32:54.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:54.327+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:54.329601+0000) 2019-10-13T08:32:55.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:55.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:55.178538+0000) 2019-10-13T08:32:57.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:57.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.457621+0000) 2019-10-13T08:32:57.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:32:57.461+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.464564+0000) 2019-10-13T08:32:57.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:32:57.459+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.465690+0000) 2019-10-13T08:32:57.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:32:57.463+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.466193+0000) 2019-10-13T08:32:57.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:32:57.463+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.466795+0000) 2019-10-13T08:32:57.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:57.463+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.467136+0000) 2019-10-13T08:32:57.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:32:57.463+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.467824+0000) 2019-10-13T08:32:57.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:57.463+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.468127+0000) 2019-10-13T08:32:57.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:32:57.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:32:57.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:32:57.465+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.471717+0000) 2019-10-13T08:32:57.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:32:57.555+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.555497+0000) 2019-10-13T08:32:57.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:32:57.579+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:32:57.581668+0000) 2019-10-13T08:33:01.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:01.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:01.603691+0000) 2019-10-13T08:33:01.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:01.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:01.605317+0000) 2019-10-13T08:33:02.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:02.467+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.471221+0000) 2019-10-13T08:33:02.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:02.469+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.472369+0000) 2019-10-13T08:33:02.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:02.467+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.473247+0000) 2019-10-13T08:33:02.474 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:02.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.473514+0000) 2019-10-13T08:33:02.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:02.471+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.473948+0000) 2019-10-13T08:33:02.475 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:02.471+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.474597+0000) 2019-10-13T08:33:02.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:02.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.474893+0000) 2019-10-13T08:33:02.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:02.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:02.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:02.477+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:02.481032+0000) 2019-10-13T08:33:03.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:03.915+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:03.919055+0000) 2019-10-13T08:33:04.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:04.327+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:04.329814+0000) 2019-10-13T08:33:05.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:05.181+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:05.185069+0000) 2019-10-13T08:33:07.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:07.455+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.457845+0000) 2019-10-13T08:33:07.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:07.475+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.478081+0000) 2019-10-13T08:33:07.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:07.475+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.479783+0000) 2019-10-13T08:33:07.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:07.475+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.480213+0000) 2019-10-13T08:33:07.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:07.477+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.480156+0000) 2019-10-13T08:33:07.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:07.475+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.480971+0000) 2019-10-13T08:33:07.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:07.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:07.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:07.475+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.481632+0000) 2019-10-13T08:33:07.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:07.479+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.482315+0000) 2019-10-13T08:33:07.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:07.481+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.485477+0000) 2019-10-13T08:33:07.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:07.555+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.555672+0000) 2019-10-13T08:33:07.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:07.579+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:07.581902+0000) 2019-10-13T08:33:10.760 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:33:10.761 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:10.764 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:33:10.764 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:11.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:11.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:11.603917+0000) 2019-10-13T08:33:11.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:11.601+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:11.605525+0000) 2019-10-13T08:33:12.485 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:12.479+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.485417+0000) 2019-10-13T08:33:12.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:12.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.487070+0000) 2019-10-13T08:33:12.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:12.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:12.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:12.483+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.488527+0000) 2019-10-13T08:33:12.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:12.483+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.488871+0000) 2019-10-13T08:33:12.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:12.487+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.491447+0000) 2019-10-13T08:33:12.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:12.485+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.491993+0000) 2019-10-13T08:33:12.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:12.487+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.492682+0000) 2019-10-13T08:33:12.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:12.493+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:12.497253+0000) 2019-10-13T08:33:13.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:13.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:13.919237+0000) 2019-10-13T08:33:14.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:14.327+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:14.330032+0000) 2019-10-13T08:33:15.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:15.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:15.191578+0000) 2019-10-13T08:33:17.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:17.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.458043+0000) 2019-10-13T08:33:17.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:17.486+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.492749+0000) 2019-10-13T08:33:17.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:17.489+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.492933+0000) 2019-10-13T08:33:17.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:17.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:17.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:17.490+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.494077+0000) 2019-10-13T08:33:17.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:17.490+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.494499+0000) 2019-10-13T08:33:17.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:17.490+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.495112+0000) 2019-10-13T08:33:17.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:17.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.495556+0000) 2019-10-13T08:33:17.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:17.490+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.495933+0000) 2019-10-13T08:33:17.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:17.501+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.506012+0000) 2019-10-13T08:33:17.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:17.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.555850+0000) 2019-10-13T08:33:17.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:17.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:17.582082+0000) 2019-10-13T08:33:21.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:21.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:21.604116+0000) 2019-10-13T08:33:21.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:21.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:21.605731+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:22.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.497444+0000) 2019-10-13T08:33:22.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:22.494+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.498044+0000) 2019-10-13T08:33:22.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:22.494+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.498825+0000) 2019-10-13T08:33:22.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:22.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.499541+0000) 2019-10-13T08:33:22.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:22.494+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.499321+0000) 2019-10-13T08:33:22.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:22.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:22.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:22.497+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.500328+0000) 2019-10-13T08:33:22.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:22.498+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.502277+0000) 2019-10-13T08:33:22.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:22.505+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:22.508236+0000) 2019-10-13T08:33:23.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:23.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:23.919444+0000) 2019-10-13T08:33:24.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:24.326+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:24.330240+0000) 2019-10-13T08:33:25.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:25.201+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:25.203061+0000) 2019-10-13T08:33:27.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:27.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.458271+0000) 2019-10-13T08:33:27.503 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:27.498+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.503350+0000) 2019-10-13T08:33:27.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:27.496+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.503816+0000) 2019-10-13T08:33:27.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:27.498+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.504578+0000) 2019-10-13T08:33:27.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:27.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:27.508 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:27.502+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.507546+0000) 2019-10-13T08:33:27.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:27.502+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.507973+0000) 2019-10-13T08:33:27.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:27.502+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.508456+0000) 2019-10-13T08:33:27.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:27.502+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.508757+0000) 2019-10-13T08:33:27.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:27.508+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.512488+0000) 2019-10-13T08:33:27.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:27.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.556026+0000) 2019-10-13T08:33:27.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:27.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:27.582258+0000) 2019-10-13T08:33:31.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:31.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:31.604327+0000) 2019-10-13T08:33:31.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:31.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:31.605949+0000) 2019-10-13T08:33:32.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:32.506+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.509731+0000) 2019-10-13T08:33:32.510 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:32.506+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.510066+0000) 2019-10-13T08:33:32.511 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:32.506+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.510570+0000) 2019-10-13T08:33:32.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:32.506+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.511005+0000) 2019-10-13T08:33:32.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:32.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.511276+0000) 2019-10-13T08:33:32.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:32.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:32.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:32.506+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.513313+0000) 2019-10-13T08:33:32.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:32.510+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.514240+0000) 2019-10-13T08:33:32.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:32.512+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:32.518563+0000) 2019-10-13T08:33:33.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:33.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:33.919645+0000) 2019-10-13T08:33:34.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:34.326+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:34.330443+0000) 2019-10-13T08:33:35.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:35.208+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:35.209594+0000) 2019-10-13T08:33:37.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:37.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.458496+0000) 2019-10-13T08:33:37.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:37.510+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.516216+0000) 2019-10-13T08:33:37.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:37.510+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.516547+0000) 2019-10-13T08:33:37.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:37.510+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.517054+0000) 2019-10-13T08:33:37.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:37.510+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.517288+0000) 2019-10-13T08:33:37.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:37.514+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.518177+0000) 2019-10-13T08:33:37.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:37.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:37.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:37.512+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.518314+0000) 2019-10-13T08:33:37.520 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:37.514+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.519111+0000) 2019-10-13T08:33:37.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:37.520+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.524945+0000) 2019-10-13T08:33:37.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:37.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.556210+0000) 2019-10-13T08:33:37.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:37.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:37.582446+0000) 2019-10-13T08:33:40.882 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:33:40.882 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:40.886 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:33:40.886 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:33:41.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:41.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:41.604551+0000) 2019-10-13T08:33:41.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:41.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:41.606170+0000) 2019-10-13T08:33:42.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:42.514+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.521390+0000) 2019-10-13T08:33:42.524 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:42.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:42.525 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:42.518+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.524981+0000) 2019-10-13T08:33:42.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:42.518+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.525525+0000) 2019-10-13T08:33:42.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:42.522+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.526192+0000) 2019-10-13T08:33:42.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:42.522+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.526556+0000) 2019-10-13T08:33:42.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:42.522+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.525727+0000) 2019-10-13T08:33:42.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:42.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.527553+0000) 2019-10-13T08:33:42.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:42.524+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:42.529145+0000) 2019-10-13T08:33:43.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:43.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:43.919855+0000) 2019-10-13T08:33:44.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:44.330+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:44.330646+0000) 2019-10-13T08:33:45.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:45.216+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:45.220588+0000) 2019-10-13T08:33:47.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:47.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.458731+0000) 2019-10-13T08:33:47.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:47.522+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.527688+0000) 2019-10-13T08:33:47.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:47.522+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.528230+0000) 2019-10-13T08:33:47.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:47.522+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.528659+0000) 2019-10-13T08:33:47.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:47.522+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.529548+0000) 2019-10-13T08:33:47.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:47.526+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.530041+0000) 2019-10-13T08:33:47.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:47.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:47.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:47.526+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.531421+0000) 2019-10-13T08:33:47.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:47.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.531426+0000) 2019-10-13T08:33:47.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:47.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.534680+0000) 2019-10-13T08:33:47.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:47.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.556405+0000) 2019-10-13T08:33:47.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:47.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:47.582636+0000) 2019-10-13T08:33:51.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:51.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:51.604782+0000) 2019-10-13T08:33:51.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:51.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:51.606370+0000) 2019-10-13T08:33:52.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:52.530+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.533978+0000) 2019-10-13T08:33:52.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:52.530+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.535424+0000) 2019-10-13T08:33:52.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:52.528+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.535692+0000) 2019-10-13T08:33:52.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:52.530+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.536366+0000) 2019-10-13T08:33:52.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:52.530+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.537019+0000) 2019-10-13T08:33:52.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:52.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:52.538 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:52.530+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.537544+0000) 2019-10-13T08:33:52.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:52.534+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.538027+0000) 2019-10-13T08:33:52.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:52.536+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:52.541759+0000) 2019-10-13T08:33:53.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:53.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:53.920069+0000) 2019-10-13T08:33:54.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:54.330+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:54.330844+0000) 2019-10-13T08:33:55.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:55.220+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:55.227069+0000) 2019-10-13T08:33:57.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:57.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.458930+0000) 2019-10-13T08:33:57.540 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:33:57.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.540156+0000) 2019-10-13T08:33:57.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:57.534+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.541007+0000) 2019-10-13T08:33:57.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:33:57.534+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.541655+0000) 2019-10-13T08:33:57.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:33:57.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.541887+0000) 2019-10-13T08:33:57.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:33:57.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:33:57.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:33:57.538+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.544400+0000) 2019-10-13T08:33:57.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:57.538+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.545276+0000) 2019-10-13T08:33:57.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:33:57.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.546899+0000) 2019-10-13T08:33:57.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:33:57.540+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.547812+0000) 2019-10-13T08:33:57.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:33:57.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.556570+0000) 2019-10-13T08:33:57.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:33:57.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:33:57.582818+0000) 2019-10-13T08:34:01.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:01.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:01.604976+0000) 2019-10-13T08:34:01.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:01.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:01.606573+0000) 2019-10-13T08:34:02.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:02.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.547445+0000) 2019-10-13T08:34:02.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:02.540+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.547916+0000) 2019-10-13T08:34:02.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:02.542+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.547894+0000) 2019-10-13T08:34:02.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:02.542+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.548768+0000) 2019-10-13T08:34:02.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:02.546+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:02.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:02.546+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.550944+0000) 2019-10-13T08:34:02.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:02.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.551321+0000) 2019-10-13T08:34:02.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:02.546+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.552037+0000) 2019-10-13T08:34:02.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:02.552+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:02.557426+0000) 2019-10-13T08:34:03.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:03.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:03.920232+0000) 2019-10-13T08:34:04.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:04.330+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:04.331062+0000) 2019-10-13T08:34:05.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:05.236+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:05.238139+0000) 2019-10-13T08:34:07.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:07.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.459139+0000) 2019-10-13T08:34:07.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:07.550+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.553274+0000) 2019-10-13T08:34:07.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.553846+0000) 2019-10-13T08:34:07.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.554502+0000) 2019-10-13T08:34:07.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.555148+0000) 2019-10-13T08:34:07.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.555447+0000) 2019-10-13T08:34:07.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.556389+0000) 2019-10-13T08:34:07.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:07.552+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.556176+0000) 2019-10-13T08:34:07.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:07.550+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.556702+0000) 2019-10-13T08:34:07.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:07.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:07.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:07.556+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.561687+0000) 2019-10-13T08:34:07.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:07.578+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:07.582967+0000) 2019-10-13T08:34:11.062 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:34:11.063 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:11.066 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:34:11.066 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:11.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:11.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:11.605177+0000) 2019-10-13T08:34:11.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:11.600+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:11.606723+0000) 2019-10-13T08:34:12.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:12.553+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.560841+0000) 2019-10-13T08:34:12.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:12.553+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.561322+0000) 2019-10-13T08:34:12.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:12.557+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.561843+0000) 2019-10-13T08:34:12.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:12.557+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.562673+0000) 2019-10-13T08:34:12.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:12.557+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.562950+0000) 2019-10-13T08:34:12.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:12.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:12.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:12.556+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.563970+0000) 2019-10-13T08:34:12.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:12.557+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.564819+0000) 2019-10-13T08:34:12.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:12.564+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:12.570156+0000) 2019-10-13T08:34:13.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:13.917+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:13.920435+0000) 2019-10-13T08:34:14.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:14.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:14.331258+0000) 2019-10-13T08:34:15.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:15.244+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:15.244741+0000) 2019-10-13T08:34:17.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:17.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.459368+0000) 2019-10-13T08:34:17.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:17.553+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.556873+0000) 2019-10-13T08:34:17.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:17.561+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.566484+0000) 2019-10-13T08:34:17.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:17.561+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.568557+0000) 2019-10-13T08:34:17.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:17.561+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.569086+0000) 2019-10-13T08:34:17.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:17.561+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.569417+0000) 2019-10-13T08:34:17.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:17.561+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.569509+0000) 2019-10-13T08:34:17.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:17.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:17.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:17.565+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.570336+0000) 2019-10-13T08:34:17.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:17.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.570367+0000) 2019-10-13T08:34:17.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:17.568+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.574529+0000) 2019-10-13T08:34:17.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:17.577+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:17.583106+0000) 2019-10-13T08:34:21.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:21.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:21.605386+0000) 2019-10-13T08:34:21.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:21.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:21.606926+0000) 2019-10-13T08:34:22.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:22.565+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.573251+0000) 2019-10-13T08:34:22.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:22.569+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.573953+0000) 2019-10-13T08:34:22.574 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:22.569+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.574630+0000) 2019-10-13T08:34:22.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:22.569+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.574688+0000) 2019-10-13T08:34:22.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:22.569+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.575010+0000) 2019-10-13T08:34:22.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:22.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:22.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:22.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.577946+0000) 2019-10-13T08:34:22.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:22.572+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.577848+0000) 2019-10-13T08:34:22.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:22.576+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:22.582191+0000) 2019-10-13T08:34:23.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:23.913+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:23.920637+0000) 2019-10-13T08:34:24.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:24.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:24.331490+0000) 2019-10-13T08:34:25.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:25.252+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:25.256055+0000) 2019-10-13T08:34:27.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:27.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.459597+0000) 2019-10-13T08:34:27.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:27.553+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.557068+0000) 2019-10-13T08:34:27.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:27.577+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.580383+0000) 2019-10-13T08:34:27.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:27.577+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.581275+0000) 2019-10-13T08:34:27.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:27.581+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.583234+0000) 2019-10-13T08:34:27.584 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:27.581+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.583907+0000) 2019-10-13T08:34:27.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:27.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:27.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:27.581+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.583969+0000) 2019-10-13T08:34:27.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:27.580+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.584012+0000) 2019-10-13T08:34:27.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:27.581+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.585157+0000) 2019-10-13T08:34:27.587 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:27.585+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.587223+0000) 2019-10-13T08:34:27.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:27.588+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:27.592904+0000) 2019-10-13T08:34:31.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:31.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:31.605637+0000) 2019-10-13T08:34:31.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:31.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:31.607123+0000) 2019-10-13T08:34:32.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:32.585+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.587230+0000) 2019-10-13T08:34:32.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:32.585+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.587731+0000) 2019-10-13T08:34:32.589 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:32.585+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.589013+0000) 2019-10-13T08:34:32.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:32.585+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.589193+0000) 2019-10-13T08:34:32.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:32.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.589685+0000) 2019-10-13T08:34:32.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:32.588+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.589541+0000) 2019-10-13T08:34:32.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:32.589+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.590121+0000) 2019-10-13T08:34:32.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:32.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:32.596 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:32.592+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:32.595694+0000) 2019-10-13T08:34:33.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:33.917+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:33.920850+0000) 2019-10-13T08:34:34.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:34.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:34.331707+0000) 2019-10-13T08:34:35.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:35.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:35.262579+0000) 2019-10-13T08:34:37.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:37.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.459821+0000) 2019-10-13T08:34:37.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:37.553+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.557267+0000) 2019-10-13T08:34:37.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:37.581+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.583442+0000) 2019-10-13T08:34:37.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:37.591+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.592899+0000) 2019-10-13T08:34:37.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:37.593+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.594051+0000) 2019-10-13T08:34:37.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:37.593+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.595802+0000) 2019-10-13T08:34:37.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:37.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.596843+0000) 2019-10-13T08:34:37.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:37.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:37.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:37.593+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.597655+0000) 2019-10-13T08:34:37.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:37.597+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.598172+0000) 2019-10-13T08:34:37.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:37.597+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.598664+0000) 2019-10-13T08:34:37.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:37.603+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:37.604790+0000) 2019-10-13T08:34:41.193 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:34:41.193 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:41.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:34:41.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:34:41.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:41.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:41.605872+0000) 2019-10-13T08:34:41.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:41.603+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:41.607322+0000) 2019-10-13T08:34:42.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:42.597+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.601119+0000) 2019-10-13T08:34:42.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:42.597+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.601277+0000) 2019-10-13T08:34:42.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:42.601+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.601782+0000) 2019-10-13T08:34:42.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:42.601+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.602382+0000) 2019-10-13T08:34:42.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:42.601+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.602767+0000) 2019-10-13T08:34:42.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:42.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75709 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:42.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:42.601+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.603848+0000) 2019-10-13T08:34:42.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:42.599+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.603872+0000) 2019-10-13T08:34:42.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:42.607+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:42.610159+0000) 2019-10-13T08:34:43.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:43.917+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:43.921047+0000) 2019-10-13T08:34:44.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:44.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:44.331912+0000) 2019-10-13T08:34:45.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:45.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:45.271131+0000) 2019-10-13T08:34:47.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:47.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.460050+0000) 2019-10-13T08:34:47.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:47.553+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.557489+0000) 2019-10-13T08:34:47.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:47.581+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.583638+0000) 2019-10-13T08:34:47.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:47.605+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.606025+0000) 2019-10-13T08:34:47.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:47.605+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.607020+0000) 2019-10-13T08:34:47.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:47.605+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.607804+0000) 2019-10-13T08:34:47.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:47.605+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.609467+0000) 2019-10-13T08:34:47.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:47.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.609905+0000) 2019-10-13T08:34:47.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:47.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75741 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:47.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:47.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.610038+0000) 2019-10-13T08:34:47.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:47.609+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.611555+0000) 2019-10-13T08:34:47.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:47.615+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:47.617694+0000) 2019-10-13T08:34:51.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:51.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:51.606066+0000) 2019-10-13T08:34:51.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:51.603+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:51.607522+0000) 2019-10-13T08:34:52.614 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:52.613+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.614107+0000) 2019-10-13T08:34:52.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:52.613+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.614477+0000) 2019-10-13T08:34:52.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:52.613+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.616076+0000) 2019-10-13T08:34:52.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:52.611+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.615966+0000) 2019-10-13T08:34:52.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:52.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:52.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:52.617+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.618189+0000) 2019-10-13T08:34:52.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:52.617+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.618367+0000) 2019-10-13T08:34:52.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:52.617+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.618774+0000) 2019-10-13T08:34:52.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:52.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:52.625376+0000) 2019-10-13T08:34:53.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:53.917+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:53.921242+0000) 2019-10-13T08:34:54.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:54.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:54.332108+0000) 2019-10-13T08:34:55.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:55.275+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:55.277616+0000) 2019-10-13T08:34:57.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:57.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.460251+0000) 2019-10-13T08:34:57.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:57.553+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.557699+0000) 2019-10-13T08:34:57.584 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:57.581+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.583840+0000) 2019-10-13T08:34:57.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:34:57.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.620191+0000) 2019-10-13T08:34:57.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:34:57.617+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.620809+0000) 2019-10-13T08:34:57.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:34:57.619+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.623115+0000) 2019-10-13T08:34:57.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:34:57.621+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.622951+0000) 2019-10-13T08:34:57.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:34:57.621+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.622645+0000) 2019-10-13T08:34:57.627 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:34:57.621+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.623173+0000) 2019-10-13T08:34:57.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:34:57.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:34:57.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:34:57.621+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.625032+0000) 2019-10-13T08:34:57.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:34:57.627+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:34:57.631167+0000) 2019-10-13T08:35:01.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:01.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:01.606282+0000) 2019-10-13T08:35:01.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:01.603+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:01.607720+0000) 2019-10-13T08:35:02.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:02.625+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.626377+0000) 2019-10-13T08:35:02.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:02.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.627132+0000) 2019-10-13T08:35:02.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:02.625+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.627269+0000) 2019-10-13T08:35:02.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:02.625+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.628033+0000) 2019-10-13T08:35:02.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:02.625+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.628559+0000) 2019-10-13T08:35:02.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:02.625+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.628863+0000) 2019-10-13T08:35:02.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:02.625+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.629421+0000) 2019-10-13T08:35:02.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:02.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:02.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:02.635+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:02.637681+0000) 2019-10-13T08:35:03.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:03.917+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:03.921448+0000) 2019-10-13T08:35:04.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:04.329+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:04.332329+0000) 2019-10-13T08:35:05.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:05.283+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:05.284155+0000) 2019-10-13T08:35:07.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:07.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.460451+0000) 2019-10-13T08:35:07.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:07.557+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.557934+0000) 2019-10-13T08:35:07.584 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:07.581+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.584035+0000) 2019-10-13T08:35:07.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:07.629+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.633328+0000) 2019-10-13T08:35:07.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:07.633+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.634030+0000) 2019-10-13T08:35:07.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:07.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:07.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:07.633+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.637427+0000) 2019-10-13T08:35:07.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:07.633+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.637530+0000) 2019-10-13T08:35:07.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:07.635+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.638189+0000) 2019-10-13T08:35:07.639 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:07.637+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.638147+0000) 2019-10-13T08:35:07.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:07.637+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.638998+0000) 2019-10-13T08:35:07.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:07.639+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:07.643287+0000) 2019-10-13T08:35:11.333 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:35:11.333 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:11.337 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:35:11.337 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:11.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:11.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:11.606482+0000) 2019-10-13T08:35:11.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:11.607+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:11.607876+0000) 2019-10-13T08:35:12.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:12.636+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.639455+0000) 2019-10-13T08:35:13.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:12.639+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.643178+0000) 2019-10-13T08:35:13.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:12.647+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.651175+0000) 2019-10-13T08:35:13.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:12.636+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.640139+0000) 2019-10-13T08:35:13.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:12.636+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.640357+0000) 2019-10-13T08:35:13.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:12.640+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.642024+0000) 2019-10-13T08:35:13.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:12.640+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.642853+0000) 2019-10-13T08:35:13.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:12.640+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:12.643081+0000) 2019-10-13T08:35:13.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:12.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:13.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:13.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:13.921654+0000) 2019-10-13T08:35:14.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:14.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:14.332530+0000) 2019-10-13T08:35:15.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:15.287+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:15.290770+0000) 2019-10-13T08:35:17.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:17.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.460679+0000) 2019-10-13T08:35:17.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:17.556+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.558158+0000) 2019-10-13T08:35:17.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:17.580+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.584272+0000) 2019-10-13T08:35:17.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:17.644+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.647643+0000) 2019-10-13T08:35:17.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:17.644+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.648507+0000) 2019-10-13T08:35:17.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:17.648+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.650091+0000) 2019-10-13T08:35:17.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:17.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75924 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:17.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:17.648+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.650591+0000) 2019-10-13T08:35:17.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:17.648+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.650824+0000) 2019-10-13T08:35:17.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:17.648+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.651896+0000) 2019-10-13T08:35:17.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:17.651+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.652169+0000) 2019-10-13T08:35:17.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:17.655+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:17.659375+0000) 2019-10-13T08:35:21.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:21.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:21.606675+0000) 2019-10-13T08:35:21.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:21.607+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:21.608122+0000) 2019-10-13T08:35:22.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:22.652+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.654293+0000) 2019-10-13T08:35:22.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:22.652+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.654608+0000) 2019-10-13T08:35:22.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:22.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:22.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:22.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.656554+0000) 2019-10-13T08:35:22.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:22.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.656759+0000) 2019-10-13T08:35:22.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:22.652+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.657108+0000) 2019-10-13T08:35:22.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:22.652+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.657594+0000) 2019-10-13T08:35:22.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:22.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.659162+0000) 2019-10-13T08:35:22.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:22.663+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:22.664951+0000) 2019-10-13T08:35:23.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:23.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:23.921848+0000) 2019-10-13T08:35:24.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:24.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:24.332729+0000) 2019-10-13T08:35:25.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:25.295+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:25.297324+0000) 2019-10-13T08:35:27.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:27.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.460837+0000) 2019-10-13T08:35:27.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:27.556+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.558414+0000) 2019-10-13T08:35:27.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:27.580+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.584506+0000) 2019-10-13T08:35:27.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:27.656+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.660587+0000) 2019-10-13T08:35:27.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:27.656+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.660788+0000) 2019-10-13T08:35:27.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:27.656+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.661456+0000) 2019-10-13T08:35:27.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:27.660+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.662007+0000) 2019-10-13T08:35:27.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:27.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.662123+0000) 2019-10-13T08:35:27.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:27.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:27.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:27.659+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.663191+0000) 2019-10-13T08:35:27.666 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:27.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.666058+0000) 2019-10-13T08:35:27.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:27.671+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:27.672401+0000) 2019-10-13T08:35:31.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:31.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:31.606900+0000) 2019-10-13T08:35:31.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:31.607+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:31.608321+0000) 2019-10-13T08:35:32.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:32.664+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.666438+0000) 2019-10-13T08:35:32.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:32.667+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.670814+0000) 2019-10-13T08:35:32.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:32.675+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.676416+0000) 2019-10-13T08:35:32.680 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:32.664+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.667003+0000) 2019-10-13T08:35:32.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:32.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:32.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:32.668+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.670322+0000) 2019-10-13T08:35:32.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:32.668+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.670855+0000) 2019-10-13T08:35:32.681 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:32.668+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.671599+0000) 2019-10-13T08:35:32.681 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:32.668+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:32.672913+0000) 2019-10-13T08:35:33.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:33.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:33.922006+0000) 2019-10-13T08:35:34.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:34.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:34.332925+0000) 2019-10-13T08:35:35.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:35.299+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:35.303954+0000) 2019-10-13T08:35:37.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:37.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.461038+0000) 2019-10-13T08:35:37.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:37.556+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.558645+0000) 2019-10-13T08:35:37.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:37.580+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.584706+0000) 2019-10-13T08:35:37.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:37.668+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.673012+0000) 2019-10-13T08:35:37.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:37.672+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.674382+0000) 2019-10-13T08:35:37.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:37.672+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.674844+0000) 2019-10-13T08:35:37.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:37.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.675133+0000) 2019-10-13T08:35:37.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:37.672+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.675574+0000) 2019-10-13T08:35:37.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:37.672+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.676455+0000) 2019-10-13T08:35:37.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:37.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:37.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:37.675+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.676497+0000) 2019-10-13T08:35:37.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:37.679+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:37.682019+0000) 2019-10-13T08:35:41.479 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:35:41.479 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:41.483 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:35:41.483 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:35:41.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:41.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:41.607099+0000) 2019-10-13T08:35:41.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:41.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:41.608540+0000) 2019-10-13T08:35:42.680 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:42.676+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.680235+0000) 2019-10-13T08:35:42.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:42.678+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.681746+0000) 2019-10-13T08:35:42.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:42.680+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.682653+0000) 2019-10-13T08:35:42.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:42.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:42.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:42.680+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.683045+0000) 2019-10-13T08:35:42.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:42.680+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.683330+0000) 2019-10-13T08:35:42.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:42.680+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.683536+0000) 2019-10-13T08:35:42.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:42.680+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.684244+0000) 2019-10-13T08:35:42.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:42.686+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:42.689356+0000) 2019-10-13T08:35:43.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:43.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:43.922222+0000) 2019-10-13T08:35:44.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:44.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:44.333153+0000) 2019-10-13T08:35:45.316 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:45.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:45.315013+0000) 2019-10-13T08:35:47.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:47.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.461236+0000) 2019-10-13T08:35:47.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:47.556+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.558843+0000) 2019-10-13T08:35:47.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:47.584+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.584905+0000) 2019-10-13T08:35:47.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:47.684+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.686228+0000) 2019-10-13T08:35:47.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:47.684+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.687540+0000) 2019-10-13T08:35:47.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:47.684+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.687704+0000) 2019-10-13T08:35:47.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:47.684+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.688072+0000) 2019-10-13T08:35:47.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:47.686+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.688439+0000) 2019-10-13T08:35:47.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:47.684+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.688685+0000) 2019-10-13T08:35:47.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:47.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:47.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:47.684+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.689685+0000) 2019-10-13T08:35:47.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:47.690+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:47.695778+0000) 2019-10-13T08:35:51.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:51.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:51.607303+0000) 2019-10-13T08:35:51.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:51.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:51.608748+0000) 2019-10-13T08:35:52.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:52.688+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.692404+0000) 2019-10-13T08:35:52.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:52.690+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.693789+0000) 2019-10-13T08:35:52.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:52.692+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.694481+0000) 2019-10-13T08:35:52.696 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:52.692+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.695890+0000) 2019-10-13T08:35:52.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:52.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:52.696 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:52.692+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.696647+0000) 2019-10-13T08:35:52.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:52.692+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.696866+0000) 2019-10-13T08:35:52.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:52.692+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.697571+0000) 2019-10-13T08:35:52.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:52.702+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:52.707755+0000) 2019-10-13T08:35:53.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:53.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:53.922442+0000) 2019-10-13T08:35:54.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:54.332+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:54.333382+0000) 2019-10-13T08:35:55.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:55.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:55.325910+0000) 2019-10-13T08:35:57.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:57.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.461441+0000) 2019-10-13T08:35:57.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:57.556+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.559093+0000) 2019-10-13T08:35:57.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:57.580+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.585098+0000) 2019-10-13T08:35:57.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:35:57.696+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.700443+0000) 2019-10-13T08:35:57.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:35:57.696+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.701292+0000) 2019-10-13T08:35:57.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:35:57.698+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.701211+0000) 2019-10-13T08:35:57.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:35:57.700+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.702059+0000) 2019-10-13T08:35:57.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:35:57.700+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.702765+0000) 2019-10-13T08:35:57.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:35:57.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:35:57.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:35:57.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.703104+0000) 2019-10-13T08:35:57.704 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:35:57.700+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.703917+0000) 2019-10-13T08:35:57.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:35:57.706+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:35:57.709745+0000) 2019-10-13T08:36:01.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:01.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:01.607533+0000) 2019-10-13T08:36:01.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:01.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:01.608948+0000) 2019-10-13T08:36:02.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:02.704+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.706168+0000) 2019-10-13T08:36:02.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:02.704+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.708655+0000) 2019-10-13T08:36:02.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:02.704+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.709291+0000) 2019-10-13T08:36:02.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:02.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:02.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:02.704+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.709629+0000) 2019-10-13T08:36:02.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:02.708+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.709985+0000) 2019-10-13T08:36:02.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:02.706+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.709810+0000) 2019-10-13T08:36:02.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:02.708+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.710946+0000) 2019-10-13T08:36:02.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:02.714+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:02.716414+0000) 2019-10-13T08:36:03.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:03.920+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:03.922675+0000) 2019-10-13T08:36:04.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:04.328+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:04.333581+0000) 2019-10-13T08:36:05.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:05.330+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:05.332451+0000) 2019-10-13T08:36:07.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:07.455+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.461637+0000) 2019-10-13T08:36:07.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:07.555+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.559292+0000) 2019-10-13T08:36:07.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:07.583+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.585293+0000) 2019-10-13T08:36:07.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:07.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.712504+0000) 2019-10-13T08:36:07.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:07.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.712734+0000) 2019-10-13T08:36:07.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:07.711+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.713896+0000) 2019-10-13T08:36:07.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:07.711+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.714138+0000) 2019-10-13T08:36:07.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:07.711+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.714762+0000) 2019-10-13T08:36:07.715 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:07.711+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.715138+0000) 2019-10-13T08:36:07.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:07.710+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.715268+0000) 2019-10-13T08:36:07.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:07.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:07.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:07.718+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:07.722213+0000) 2019-10-13T08:36:11.586 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:36:11.586 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:11.590 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:36:11.590 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:11.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:11.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:11.607762+0000) 2019-10-13T08:36:11.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:11.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:11.609176+0000) 2019-10-13T08:36:12.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:12.715+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.719274+0000) 2019-10-13T08:36:12.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:12.718+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.721053+0000) 2019-10-13T08:36:12.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:12.719+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.722463+0000) 2019-10-13T08:36:12.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:12.719+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:12.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:12.719+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.723534+0000) 2019-10-13T08:36:12.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:12.719+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.724726+0000) 2019-10-13T08:36:12.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:12.719+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.725609+0000) 2019-10-13T08:36:12.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:12.723+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.726382+0000) 2019-10-13T08:36:12.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:12.726+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:12.728254+0000) 2019-10-13T08:36:13.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:13.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:13.922874+0000) 2019-10-13T08:36:14.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:14.331+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:14.333791+0000) 2019-10-13T08:36:15.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:15.334+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:15.338951+0000) 2019-10-13T08:36:17.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:17.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.461871+0000) 2019-10-13T08:36:17.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:17.555+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.559499+0000) 2019-10-13T08:36:17.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:17.579+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.585494+0000) 2019-10-13T08:36:17.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:17.723+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.727278+0000) 2019-10-13T08:36:17.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:17.722+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.727450+0000) 2019-10-13T08:36:17.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:17.723+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.728216+0000) 2019-10-13T08:36:17.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:17.723+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.728495+0000) 2019-10-13T08:36:17.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:17.723+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.728878+0000) 2019-10-13T08:36:17.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:17.723+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.729296+0000) 2019-10-13T08:36:17.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:17.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:17.731 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:17.727+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.731134+0000) 2019-10-13T08:36:17.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:17.730+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:17.733322+0000) 2019-10-13T08:36:21.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:21.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:21.607958+0000) 2019-10-13T08:36:21.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:21.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:21.609374+0000) 2019-10-13T08:36:22.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:22.727+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.733468+0000) 2019-10-13T08:36:22.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:22.731+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.735307+0000) 2019-10-13T08:36:22.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:22.731+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.736011+0000) 2019-10-13T08:36:22.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:22.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:22.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:22.731+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.736529+0000) 2019-10-13T08:36:22.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:22.731+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.737025+0000) 2019-10-13T08:36:22.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:22.734+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.737052+0000) 2019-10-13T08:36:22.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:22.735+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.737985+0000) 2019-10-13T08:36:22.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:22.734+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:22.738999+0000) 2019-10-13T08:36:23.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:23.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:23.923067+0000) 2019-10-13T08:36:24.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:24.331+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:24.333950+0000) 2019-10-13T08:36:25.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:25.342+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:25.345534+0000) 2019-10-13T08:36:27.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:27.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.462059+0000) 2019-10-13T08:36:27.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:27.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.559702+0000) 2019-10-13T08:36:27.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:27.583+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.585668+0000) 2019-10-13T08:36:27.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:27.735+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.739834+0000) 2019-10-13T08:36:27.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:27.735+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.740217+0000) 2019-10-13T08:36:27.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:27.738+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.740271+0000) 2019-10-13T08:36:27.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:27.735+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.741152+0000) 2019-10-13T08:36:27.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:27.735+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.741368+0000) 2019-10-13T08:36:27.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:27.739+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.742403+0000) 2019-10-13T08:36:27.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:27.739+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.742791+0000) 2019-10-13T08:36:27.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:27.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:27.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:27.742+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:27.747510+0000) 2019-10-13T08:36:31.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:31.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:31.609569+0000) 2019-10-13T08:36:31.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:31.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:31.608178+0000) 2019-10-13T08:36:32.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:32.742+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.745799+0000) 2019-10-13T08:36:32.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:32.743+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.748491+0000) 2019-10-13T08:36:32.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:32.743+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.748771+0000) 2019-10-13T08:36:32.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:32.743+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.749564+0000) 2019-10-13T08:36:32.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:32.747+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.749968+0000) 2019-10-13T08:36:32.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:32.747+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.750790+0000) 2019-10-13T08:36:32.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:32.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:32.751 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:32.747+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.751438+0000) 2019-10-13T08:36:32.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:32.754+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:32.758541+0000) 2019-10-13T08:36:33.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:33.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:33.923269+0000) 2019-10-13T08:36:34.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:34.331+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:34.334159+0000) 2019-10-13T08:36:35.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:35.350+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:35.352061+0000) 2019-10-13T08:36:37.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:37.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.462258+0000) 2019-10-13T08:36:37.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:37.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.559925+0000) 2019-10-13T08:36:37.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:37.583+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.585916+0000) 2019-10-13T08:36:37.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:37.751+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.754347+0000) 2019-10-13T08:36:37.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:37.751+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.755809+0000) 2019-10-13T08:36:37.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:37.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:37.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:37.751+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.757529+0000) 2019-10-13T08:36:37.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:37.755+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.757868+0000) 2019-10-13T08:36:37.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:37.754+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.757714+0000) 2019-10-13T08:36:37.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:37.755+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.758272+0000) 2019-10-13T08:36:37.759 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:37.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.758897+0000) 2019-10-13T08:36:37.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:37.762+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:37.764339+0000) 2019-10-13T08:36:41.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:41.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:41.608390+0000) 2019-10-13T08:36:41.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:41.606+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:41.609767+0000) 2019-10-13T08:36:41.661 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:36:41.661 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:41.665 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:36:41.665 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:36:42.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:42.755+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.760500+0000) 2019-10-13T08:36:42.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:42.755+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.761173+0000) 2019-10-13T08:36:42.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:42.755+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.761605+0000) 2019-10-13T08:36:42.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:42.758+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.761668+0000) 2019-10-13T08:36:42.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:42.759+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.763252+0000) 2019-10-13T08:36:42.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:42.759+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.763673+0000) 2019-10-13T08:36:42.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:42.759+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.764101+0000) 2019-10-13T08:36:42.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:42.759+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:42.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:42.766+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:42.770173+0000) 2019-10-13T08:36:43.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:43.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:43.923425+0000) 2019-10-13T08:36:44.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:44.331+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:44.334395+0000) 2019-10-13T08:36:45.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:45.362+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:45.363514+0000) 2019-10-13T08:36:47.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:47.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.462466+0000) 2019-10-13T08:36:47.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:47.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.560122+0000) 2019-10-13T08:36:47.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:47.583+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.586076+0000) 2019-10-13T08:36:47.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:47.761+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.766663+0000) 2019-10-13T08:36:47.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:47.763+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.768270+0000) 2019-10-13T08:36:47.769 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:47.763+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.768649+0000) 2019-10-13T08:36:47.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:47.763+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.768815+0000) 2019-10-13T08:36:47.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:47.763+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.769464+0000) 2019-10-13T08:36:47.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:47.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:47.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:47.767+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.770768+0000) 2019-10-13T08:36:47.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:47.767+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.771199+0000) 2019-10-13T08:36:47.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:47.773+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:47.776697+0000) 2019-10-13T08:36:51.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:51.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:51.608591+0000) 2019-10-13T08:36:51.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:51.609+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:51.609961+0000) 2019-10-13T08:36:52.774 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:52.771+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.774207+0000) 2019-10-13T08:36:52.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:52.771+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.774693+0000) 2019-10-13T08:36:52.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:52.769+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.774633+0000) 2019-10-13T08:36:52.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:52.771+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.775442+0000) 2019-10-13T08:36:52.776 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:52.771+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.775776+0000) 2019-10-13T08:36:52.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:52.771+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.776210+0000) 2019-10-13T08:36:52.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:52.771+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.777035+0000) 2019-10-13T08:36:52.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:52.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:52.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:52.777+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:52.784007+0000) 2019-10-13T08:36:53.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:53.919+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:53.923647+0000) 2019-10-13T08:36:54.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:54.331+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:54.334592+0000) 2019-10-13T08:36:55.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:55.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:55.369968+0000) 2019-10-13T08:36:57.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:57.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.462665+0000) 2019-10-13T08:36:57.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:57.555+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.560324+0000) 2019-10-13T08:36:57.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:57.583+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.586300+0000) 2019-10-13T08:36:57.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:36:57.775+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.780942+0000) 2019-10-13T08:36:57.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:36:57.775+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.781268+0000) 2019-10-13T08:36:57.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:36:57.779+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.782074+0000) 2019-10-13T08:36:57.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:36:57.779+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.782292+0000) 2019-10-13T08:36:57.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:36:57.779+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.782888+0000) 2019-10-13T08:36:57.783 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:36:57.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:36:57.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:36:57.777+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.783319+0000) 2019-10-13T08:36:57.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:36:57.779+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.784149+0000) 2019-10-13T08:36:57.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:36:57.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:36:57.792496+0000) 2019-10-13T08:37:01.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:01.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:01.608825+0000) 2019-10-13T08:37:01.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:01.609+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:01.610155+0000) 2019-10-13T08:37:02.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:02.783+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.786712+0000) 2019-10-13T08:37:02.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:02.781+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.787168+0000) 2019-10-13T08:37:02.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:02.783+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.787286+0000) 2019-10-13T08:37:02.789 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:02.783+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.788260+0000) 2019-10-13T08:37:02.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:02.783+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.788560+0000) 2019-10-13T08:37:02.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:02.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:02.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:02.787+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.790531+0000) 2019-10-13T08:37:02.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:02.787+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.790956+0000) 2019-10-13T08:37:02.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:02.793+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:02.797410+0000) 2019-10-13T08:37:03.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:03.922+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:03.923858+0000) 2019-10-13T08:37:04.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:04.330+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:04.334806+0000) 2019-10-13T08:37:05.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:05.373+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:05.376494+0000) 2019-10-13T08:37:07.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:07.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.462864+0000) 2019-10-13T08:37:07.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:07.554+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.560523+0000) 2019-10-13T08:37:07.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:07.582+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.586521+0000) 2019-10-13T08:37:07.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:07.786+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.793095+0000) 2019-10-13T08:37:07.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:07.786+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.793649+0000) 2019-10-13T08:37:07.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:07.789+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.793548+0000) 2019-10-13T08:37:07.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:07.790+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.796376+0000) 2019-10-13T08:37:07.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:07.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:07.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:07.790+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.797284+0000) 2019-10-13T08:37:07.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:07.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.797594+0000) 2019-10-13T08:37:07.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:07.794+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.798058+0000) 2019-10-13T08:37:07.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:07.797+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:07.803951+0000) 2019-10-13T08:37:11.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:11.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:11.609012+0000) 2019-10-13T08:37:11.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:11.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:11.610347+0000) 2019-10-13T08:37:11.826 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:37:11.826 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:11.830 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:37:11.830 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:12.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:12.793+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.799200+0000) 2019-10-13T08:37:12.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:12.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.800419+0000) 2019-10-13T08:37:12.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:12.794+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.800834+0000) 2019-10-13T08:37:12.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:12.794+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.801479+0000) 2019-10-13T08:37:12.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:12.798+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.801786+0000) 2019-10-13T08:37:12.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:12.798+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.802445+0000) 2019-10-13T08:37:12.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:12.798+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.803180+0000) 2019-10-13T08:37:12.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:12.798+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:12.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:12.805+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:12.808435+0000) 2019-10-13T08:37:13.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:13.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:13.924094+0000) 2019-10-13T08:37:14.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:14.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:14.335041+0000) 2019-10-13T08:37:15.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:15.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:15.383002+0000) 2019-10-13T08:37:17.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:17.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.463106+0000) 2019-10-13T08:37:17.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:17.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.560739+0000) 2019-10-13T08:37:17.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:17.582+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.586725+0000) 2019-10-13T08:37:17.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:17.802+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.806547+0000) 2019-10-13T08:37:17.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:17.802+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.807014+0000) 2019-10-13T08:37:17.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:17.801+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.807102+0000) 2019-10-13T08:37:17.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:17.802+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.808000+0000) 2019-10-13T08:37:17.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:17.802+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.808352+0000) 2019-10-13T08:37:17.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:17.802+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.809190+0000) 2019-10-13T08:37:17.811 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:17.802+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.809695+0000) 2019-10-13T08:37:17.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:17.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:17.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:17.813+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:17.816679+0000) 2019-10-13T08:37:21.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:21.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:21.609163+0000) 2019-10-13T08:37:21.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:21.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:21.610548+0000) 2019-10-13T08:37:22.813 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:22.806+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.813126+0000) 2019-10-13T08:37:22.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:22.810+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.813995+0000) 2019-10-13T08:37:22.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:22.810+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.814536+0000) 2019-10-13T08:37:22.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:22.809+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.814585+0000) 2019-10-13T08:37:22.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:22.810+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.815306+0000) 2019-10-13T08:37:22.816 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:22.810+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.815862+0000) 2019-10-13T08:37:22.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:22.810+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.816517+0000) 2019-10-13T08:37:22.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:22.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:22.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:22.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:22.822964+0000) 2019-10-13T08:37:23.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:23.922+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:23.924314+0000) 2019-10-13T08:37:24.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:24.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:24.335254+0000) 2019-10-13T08:37:25.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:25.385+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:25.389592+0000) 2019-10-13T08:37:27.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:27.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.463303+0000) 2019-10-13T08:37:27.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:27.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.560932+0000) 2019-10-13T08:37:27.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:27.586+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.586929+0000) 2019-10-13T08:37:27.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:27.814+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.821108+0000) 2019-10-13T08:37:27.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:27.814+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.821473+0000) 2019-10-13T08:37:27.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:27.818+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.821988+0000) 2019-10-13T08:37:27.823 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:27.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:27.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:27.818+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.823841+0000) 2019-10-13T08:37:27.824 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:27.818+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.824402+0000) 2019-10-13T08:37:27.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:27.818+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.825029+0000) 2019-10-13T08:37:27.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:27.821+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.824880+0000) 2019-10-13T08:37:27.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:27.821+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:27.825756+0000) 2019-10-13T08:37:31.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:31.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:31.609373+0000) 2019-10-13T08:37:31.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:31.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:31.610750+0000) 2019-10-13T08:37:32.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:32.822+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.827010+0000) 2019-10-13T08:37:32.827 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:32.822+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.827440+0000) 2019-10-13T08:37:32.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:32.822+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.829574+0000) 2019-10-13T08:37:32.830 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:32.822+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.829233+0000) 2019-10-13T08:37:32.830 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:32.826+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.830146+0000) 2019-10-13T08:37:32.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:32.826+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76759 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:32.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:32.826+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.831221+0000) 2019-10-13T08:37:32.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:32.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.832573+0000) 2019-10-13T08:37:32.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:32.833+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:32.838314+0000) 2019-10-13T08:37:33.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:33.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:33.924537+0000) 2019-10-13T08:37:34.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:34.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:34.335460+0000) 2019-10-13T08:37:35.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:35.393+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:35.398696+0000) 2019-10-13T08:37:37.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:37.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.463499+0000) 2019-10-13T08:37:37.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:37.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.561137+0000) 2019-10-13T08:37:37.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:37.586+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.587131+0000) 2019-10-13T08:37:37.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:37.826+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.833647+0000) 2019-10-13T08:37:37.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:37.830+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.834265+0000) 2019-10-13T08:37:37.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:37.833+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.836931+0000) 2019-10-13T08:37:37.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:37.837+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.841570+0000) 2019-10-13T08:37:37.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:37.830+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.834796+0000) 2019-10-13T08:37:37.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:37.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:37.864 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:37.834+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.837895+0000) 2019-10-13T08:37:37.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:37.834+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.838533+0000) 2019-10-13T08:37:37.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:37.834+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:37.838827+0000) 2019-10-13T08:37:41.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:41.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:41.609566+0000) 2019-10-13T08:37:41.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:41.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:41.610965+0000) 2019-10-13T08:37:42.135 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:37:42.135 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:42.165 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:37:42.165 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:37:42.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:42.834+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.839868+0000) 2019-10-13T08:37:42.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:42.834+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.840749+0000) 2019-10-13T08:37:42.841 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:42.834+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.841311+0000) 2019-10-13T08:37:42.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:42.838+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.842239+0000) 2019-10-13T08:37:42.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:42.837+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.842093+0000) 2019-10-13T08:37:42.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:42.838+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.843092+0000) 2019-10-13T08:37:42.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:42.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:42.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:42.838+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.844113+0000) 2019-10-13T08:37:42.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:42.845+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:42.849421+0000) 2019-10-13T08:37:43.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:43.922+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:43.924775+0000) 2019-10-13T08:37:44.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:44.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:44.335661+0000) 2019-10-13T08:37:45.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:45.401+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:45.405209+0000) 2019-10-13T08:37:47.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:47.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.463703+0000) 2019-10-13T08:37:47.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:47.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.561367+0000) 2019-10-13T08:37:47.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:47.586+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.587324+0000) 2019-10-13T08:37:47.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:47.842+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.847142+0000) 2019-10-13T08:37:47.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:47.841+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.847138+0000) 2019-10-13T08:37:47.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:47.842+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.848069+0000) 2019-10-13T08:37:47.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:47.842+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.849348+0000) 2019-10-13T08:37:47.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:47.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76855 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:47.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:47.846+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.850828+0000) 2019-10-13T08:37:47.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:47.846+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.851448+0000) 2019-10-13T08:37:47.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:47.846+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.852218+0000) 2019-10-13T08:37:47.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:47.853+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:47.856691+0000) 2019-10-13T08:37:51.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:51.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:51.609780+0000) 2019-10-13T08:37:51.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:51.605+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:51.611155+0000) 2019-10-13T08:37:52.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:52.850+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.853175+0000) 2019-10-13T08:37:52.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:52.848+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.854016+0000) 2019-10-13T08:37:52.855 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:52.850+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.854705+0000) 2019-10-13T08:37:52.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:52.850+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.856851+0000) 2019-10-13T08:37:52.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:52.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.856777+0000) 2019-10-13T08:37:52.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:52.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76885 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:52.858 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:52.854+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.857872+0000) 2019-10-13T08:37:52.858 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:52.854+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.858451+0000) 2019-10-13T08:37:52.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:52.860+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:52.865213+0000) 2019-10-13T08:37:53.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:53.918+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:53.924992+0000) 2019-10-13T08:37:54.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:54.330+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:54.335863+0000) 2019-10-13T08:37:55.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:55.408+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:55.416021+0000) 2019-10-13T08:37:57.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:57.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.463958+0000) 2019-10-13T08:37:57.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:57.554+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.561527+0000) 2019-10-13T08:37:57.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:57.586+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.587536+0000) 2019-10-13T08:37:57.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:37:57.854+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.860075+0000) 2019-10-13T08:37:57.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:37:57.854+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.860674+0000) 2019-10-13T08:37:57.861 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:37:57.854+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.861167+0000) 2019-10-13T08:37:57.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:37:57.854+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.861531+0000) 2019-10-13T08:37:57.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:37:57.856+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.862041+0000) 2019-10-13T08:37:57.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:37:57.858+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.863041+0000) 2019-10-13T08:37:57.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:37:57.858+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:37:57.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:37:57.858+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.863354+0000) 2019-10-13T08:37:57.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:37:57.864+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:37:57.869062+0000) 2019-10-13T08:38:01.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:01.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:01.610008+0000) 2019-10-13T08:38:01.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:01.604+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:01.611308+0000) 2019-10-13T08:38:02.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:02.861+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.866798+0000) 2019-10-13T08:38:02.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:02.861+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.867833+0000) 2019-10-13T08:38:02.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:02.860+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.867649+0000) 2019-10-13T08:38:02.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:02.865+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.870029+0000) 2019-10-13T08:38:02.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:02.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:02.871 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:02.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.870645+0000) 2019-10-13T08:38:02.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:02.865+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.871091+0000) 2019-10-13T08:38:02.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:02.865+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.871322+0000) 2019-10-13T08:38:02.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:02.872+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:02.876250+0000) 2019-10-13T08:38:03.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:03.921+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:03.925199+0000) 2019-10-13T08:38:04.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:04.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:04.336091+0000) 2019-10-13T08:38:05.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:05.420+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:05.421896+0000) 2019-10-13T08:38:07.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:07.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.464177+0000) 2019-10-13T08:38:07.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:07.557+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.561723+0000) 2019-10-13T08:38:07.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:07.585+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.587735+0000) 2019-10-13T08:38:07.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:07.869+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.873075+0000) 2019-10-13T08:38:07.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:07.869+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.874192+0000) 2019-10-13T08:38:07.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:07.868+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.875756+0000) 2019-10-13T08:38:07.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:07.869+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.876518+0000) 2019-10-13T08:38:07.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:07.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:07.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:07.873+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.878082+0000) 2019-10-13T08:38:07.878 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:07.873+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.878379+0000) 2019-10-13T08:38:07.879 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:07.873+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.879289+0000) 2019-10-13T08:38:07.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:07.880+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:07.885621+0000) 2019-10-13T08:38:11.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:11.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:11.610213+0000) 2019-10-13T08:38:11.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:11.608+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:11.611505+0000) 2019-10-13T08:38:12.221 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:38:12.221 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:12.225 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:38:12.225 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:12.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:12.873+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.881609+0000) 2019-10-13T08:38:12.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:12.877+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.882261+0000) 2019-10-13T08:38:12.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:12.876+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.882227+0000) 2019-10-13T08:38:12.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:12.877+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.883088+0000) 2019-10-13T08:38:12.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:12.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:12.885 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:12.877+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.883712+0000) 2019-10-13T08:38:12.885 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:12.877+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.884148+0000) 2019-10-13T08:38:12.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:12.877+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.884687+0000) 2019-10-13T08:38:12.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:12.884+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:12.889280+0000) 2019-10-13T08:38:13.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:13.921+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:13.925409+0000) 2019-10-13T08:38:14.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:14.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:14.336324+0000) 2019-10-13T08:38:15.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:15.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:15.428383+0000) 2019-10-13T08:38:17.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:17.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.464378+0000) 2019-10-13T08:38:17.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:17.561+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.561981+0000) 2019-10-13T08:38:17.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:17.585+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.587939+0000) 2019-10-13T08:38:17.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:17.885+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.887032+0000) 2019-10-13T08:38:17.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:17.885+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.887633+0000) 2019-10-13T08:38:17.888 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:17.885+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.888253+0000) 2019-10-13T08:38:17.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:17.885+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.888929+0000) 2019-10-13T08:38:17.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:17.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:17.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:17.889+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.890243+0000) 2019-10-13T08:38:17.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:17.884+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.890015+0000) 2019-10-13T08:38:17.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:17.889+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.892889+0000) 2019-10-13T08:38:17.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:17.892+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:17.897194+0000) 2019-10-13T08:38:21.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:21.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:21.610406+0000) 2019-10-13T08:38:21.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:21.608+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:21.611704+0000) 2019-10-13T08:38:22.893 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:22.889+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.892900+0000) 2019-10-13T08:38:22.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:22.889+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.893605+0000) 2019-10-13T08:38:22.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:22.893+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.894477+0000) 2019-10-13T08:38:22.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:22.893+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.895008+0000) 2019-10-13T08:38:22.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:22.888+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.895175+0000) 2019-10-13T08:38:22.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:22.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:22.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:22.893+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.896397+0000) 2019-10-13T08:38:22.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:22.893+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.896996+0000) 2019-10-13T08:38:22.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:22.896+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:22.900551+0000) 2019-10-13T08:38:23.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:23.925+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:23.925626+0000) 2019-10-13T08:38:24.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:24.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:24.336521+0000) 2019-10-13T08:38:25.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:25.432+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:25.435030+0000) 2019-10-13T08:38:27.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:27.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.464573+0000) 2019-10-13T08:38:27.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:27.561+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.562180+0000) 2019-10-13T08:38:27.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:27.585+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.588139+0000) 2019-10-13T08:38:27.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:27.897+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.899813+0000) 2019-10-13T08:38:27.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:27.897+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.900370+0000) 2019-10-13T08:38:27.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:27.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.900724+0000) 2019-10-13T08:38:27.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:27.897+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.901710+0000) 2019-10-13T08:38:27.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:27.901+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.901937+0000) 2019-10-13T08:38:27.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:27.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:27.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:27.901+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.902203+0000) 2019-10-13T08:38:27.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:27.901+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.903177+0000) 2019-10-13T08:38:27.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:27.900+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:27.907795+0000) 2019-10-13T08:38:31.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:31.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:31.610616+0000) 2019-10-13T08:38:31.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:31.608+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:31.611932+0000) 2019-10-13T08:38:32.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:32.900+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.904600+0000) 2019-10-13T08:38:32.906 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:32.905+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.906464+0000) 2019-10-13T08:38:32.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:32.905+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.907142+0000) 2019-10-13T08:38:32.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:32.905+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.907912+0000) 2019-10-13T08:38:32.908 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:32.905+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.908405+0000) 2019-10-13T08:38:32.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:32.905+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:32.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:32.905+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.908958+0000) 2019-10-13T08:38:32.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:32.905+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.909471+0000) 2019-10-13T08:38:32.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:32.908+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:32.913103+0000) 2019-10-13T08:38:33.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:33.925+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:33.925831+0000) 2019-10-13T08:38:34.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:34.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:34.336757+0000) 2019-10-13T08:38:35.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:35.440+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:35.443349+0000) 2019-10-13T08:38:37.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:37.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.464792+0000) 2019-10-13T08:38:37.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:37.561+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.562380+0000) 2019-10-13T08:38:37.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:37.585+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.588349+0000) 2019-10-13T08:38:37.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:37.909+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.912484+0000) 2019-10-13T08:38:37.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:37.909+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.913026+0000) 2019-10-13T08:38:37.914 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:37.909+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.913592+0000) 2019-10-13T08:38:37.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:37.913+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.914037+0000) 2019-10-13T08:38:37.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:37.913+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.914879+0000) 2019-10-13T08:38:37.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:37.908+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.914823+0000) 2019-10-13T08:38:37.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:37.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:37.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:37.913+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.916564+0000) 2019-10-13T08:38:37.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:37.916+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:37.921196+0000) 2019-10-13T08:38:41.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:41.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:41.610842+0000) 2019-10-13T08:38:41.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:41.608+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:41.612125+0000) 2019-10-13T08:38:42.353 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:38:42.354 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:42.357 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:38:42.357 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:38:42.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:42.917+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.919744+0000) 2019-10-13T08:38:42.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:42.917+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.920509+0000) 2019-10-13T08:38:42.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:42.917+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.920929+0000) 2019-10-13T08:38:42.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:42.917+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.921372+0000) 2019-10-13T08:38:42.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:42.916+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.922179+0000) 2019-10-13T08:38:42.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:42.921+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.923037+0000) 2019-10-13T08:38:42.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:42.921+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.923473+0000) 2019-10-13T08:38:42.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:42.921+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:42.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:42.924+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:42.930471+0000) 2019-10-13T08:38:43.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:43.925+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:43.926032+0000) 2019-10-13T08:38:44.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:44.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:44.336986+0000) 2019-10-13T08:38:45.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:45.448+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:45.450053+0000) 2019-10-13T08:38:47.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:47.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.464996+0000) 2019-10-13T08:38:47.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:47.561+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.562584+0000) 2019-10-13T08:38:47.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:47.585+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.588562+0000) 2019-10-13T08:38:47.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:47.925+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.926678+0000) 2019-10-13T08:38:47.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:47.920+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.926887+0000) 2019-10-13T08:38:47.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:47.925+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.927855+0000) 2019-10-13T08:38:47.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:47.925+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.929382+0000) 2019-10-13T08:38:47.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:47.925+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.929670+0000) 2019-10-13T08:38:47.930 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:47.929+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.930003+0000) 2019-10-13T08:38:47.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:47.929+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.930411+0000) 2019-10-13T08:38:47.931 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:47.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:47.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:47.928+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:47.934046+0000) 2019-10-13T08:38:51.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:51.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:51.611063+0000) 2019-10-13T08:38:51.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:51.612+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:51.612322+0000) 2019-10-13T08:38:52.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:52.933+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.935040+0000) 2019-10-13T08:38:52.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:52.932+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.935165+0000) 2019-10-13T08:38:52.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:52.933+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.935635+0000) 2019-10-13T08:38:52.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:52.933+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.936080+0000) 2019-10-13T08:38:52.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:52.933+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.936476+0000) 2019-10-13T08:38:52.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:52.933+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.936860+0000) 2019-10-13T08:38:52.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:52.933+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.937200+0000) 2019-10-13T08:38:52.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:52.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:52.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:52.940+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:52.943611+0000) 2019-10-13T08:38:53.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:53.925+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:53.926245+0000) 2019-10-13T08:38:54.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:54.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:54.337220+0000) 2019-10-13T08:38:55.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:55.456+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:55.456657+0000) 2019-10-13T08:38:57.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:57.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.465191+0000) 2019-10-13T08:38:57.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:57.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.562780+0000) 2019-10-13T08:38:57.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:57.584+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.588728+0000) 2019-10-13T08:38:57.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:38:57.936+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.941394+0000) 2019-10-13T08:38:57.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:38:57.940+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.942151+0000) 2019-10-13T08:38:57.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:38:57.940+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.942284+0000) 2019-10-13T08:38:57.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:38:57.940+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.944527+0000) 2019-10-13T08:38:57.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:38:57.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:38:57.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:38:57.940+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.944690+0000) 2019-10-13T08:38:57.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:38:57.944+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.945346+0000) 2019-10-13T08:38:57.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:38:57.944+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.947437+0000) 2019-10-13T08:38:57.951 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:38:57.948+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:38:57.950940+0000) 2019-10-13T08:39:01.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:01.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:01.611285+0000) 2019-10-13T08:39:01.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:01.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:01.612523+0000) 2019-10-13T08:39:02.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:02.944+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.948489+0000) 2019-10-13T08:39:02.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:02.944+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.948895+0000) 2019-10-13T08:39:02.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:02.948+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.949847+0000) 2019-10-13T08:39:02.950 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:02.948+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.950091+0000) 2019-10-13T08:39:02.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:02.948+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.950283+0000) 2019-10-13T08:39:02.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:02.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77324 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:02.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:02.948+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.950970+0000) 2019-10-13T08:39:02.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:02.947+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.951321+0000) 2019-10-13T08:39:02.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:02.955+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:02.956748+0000) 2019-10-13T08:39:03.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:03.924+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:03.926422+0000) 2019-10-13T08:39:04.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:04.332+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:04.337421+0000) 2019-10-13T08:39:05.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:05.459+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:05.463224+0000) 2019-10-13T08:39:07.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:07.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.465395+0000) 2019-10-13T08:39:07.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:07.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.562974+0000) 2019-10-13T08:39:07.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:07.588+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.588923+0000) 2019-10-13T08:39:07.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:07.952+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.954650+0000) 2019-10-13T08:39:07.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:07.952+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.955307+0000) 2019-10-13T08:39:07.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:07.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.955340+0000) 2019-10-13T08:39:07.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:07.952+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.956465+0000) 2019-10-13T08:39:07.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:07.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:07.959 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:07.956+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.958760+0000) 2019-10-13T08:39:07.959 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:07.956+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.958905+0000) 2019-10-13T08:39:07.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:07.956+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.959264+0000) 2019-10-13T08:39:07.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:07.963+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:07.964805+0000) 2019-10-13T08:39:11.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:11.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:11.611470+0000) 2019-10-13T08:39:12.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:11.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:11.612717+0000) 2019-10-13T08:39:12.459 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:39:12.459 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:12.463 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:39:12.463 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:12.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:12.960+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.962522+0000) 2019-10-13T08:39:12.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:12.960+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.962721+0000) 2019-10-13T08:39:12.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:12.960+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.963077+0000) 2019-10-13T08:39:12.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:12.959+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.963036+0000) 2019-10-13T08:39:12.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:12.960+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.964116+0000) 2019-10-13T08:39:12.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:12.960+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.964781+0000) 2019-10-13T08:39:12.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:12.960+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.965182+0000) 2019-10-13T08:39:12.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:12.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:12.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:12.967+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:12.970414+0000) 2019-10-13T08:39:13.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:13.924+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:13.926601+0000) 2019-10-13T08:39:14.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:14.332+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:14.337648+0000) 2019-10-13T08:39:15.470 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:15.467+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:15.469747+0000) 2019-10-13T08:39:17.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:17.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.465610+0000) 2019-10-13T08:39:17.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:17.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.563191+0000) 2019-10-13T08:39:17.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:17.584+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.589119+0000) 2019-10-13T08:39:17.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:17.964+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.968234+0000) 2019-10-13T08:39:17.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:17.964+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.969269+0000) 2019-10-13T08:39:17.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:17.968+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.970696+0000) 2019-10-13T08:39:17.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:17.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:17.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:17.968+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.972776+0000) 2019-10-13T08:39:17.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:17.968+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.973120+0000) 2019-10-13T08:39:17.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:17.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.973141+0000) 2019-10-13T08:39:17.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:17.972+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.973860+0000) 2019-10-13T08:39:17.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:17.975+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:17.979340+0000) 2019-10-13T08:39:21.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:21.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:21.611664+0000) 2019-10-13T08:39:21.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:21.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:21.612906+0000) 2019-10-13T08:39:22.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:22.972+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.975936+0000) 2019-10-13T08:39:22.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:22.971+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.975875+0000) 2019-10-13T08:39:22.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:22.976+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:22.979 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:22.976+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.978645+0000) 2019-10-13T08:39:22.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:22.976+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.978873+0000) 2019-10-13T08:39:22.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:22.976+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.979115+0000) 2019-10-13T08:39:22.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:22.976+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.979518+0000) 2019-10-13T08:39:22.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:22.976+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.980241+0000) 2019-10-13T08:39:22.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:22.983+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:22.984405+0000) 2019-10-13T08:39:23.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:23.924+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:23.926814+0000) 2019-10-13T08:39:24.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:24.336+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:24.337849+0000) 2019-10-13T08:39:25.477 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:25.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:25.476385+0000) 2019-10-13T08:39:27.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:27.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.465837+0000) 2019-10-13T08:39:27.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:27.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.563378+0000) 2019-10-13T08:39:27.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:27.588+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.589317+0000) 2019-10-13T08:39:27.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:27.976+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.980918+0000) 2019-10-13T08:39:27.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:27.980+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.984146+0000) 2019-10-13T08:39:27.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:27.980+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.984757+0000) 2019-10-13T08:39:27.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:27.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77484 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:27.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:27.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.984951+0000) 2019-10-13T08:39:27.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:27.983+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.985250+0000) 2019-10-13T08:39:27.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:27.984+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.985697+0000) 2019-10-13T08:39:27.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:27.984+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.987532+0000) 2019-10-13T08:39:27.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:27.991+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:27.992861+0000) 2019-10-13T08:39:31.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:31.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:31.611880+0000) 2019-10-13T08:39:31.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:31.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:31.613100+0000) 2019-10-13T08:39:32.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:32.983+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.987988+0000) 2019-10-13T08:39:32.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:32.988+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.990335+0000) 2019-10-13T08:39:32.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:32.988+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.990634+0000) 2019-10-13T08:39:32.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:32.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:32.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:32.988+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.992032+0000) 2019-10-13T08:39:32.992 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:32.988+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.992572+0000) 2019-10-13T08:39:32.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:32.988+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.993179+0000) 2019-10-13T08:39:32.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:32.992+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.993777+0000) 2019-10-13T08:39:32.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:32.995+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:32.998771+0000) 2019-10-13T08:39:33.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:33.924+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:33.926999+0000) 2019-10-13T08:39:34.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:34.336+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:34.338043+0000) 2019-10-13T08:39:35.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:35.479+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:35.482924+0000) 2019-10-13T08:39:37.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:37.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.466041+0000) 2019-10-13T08:39:37.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:37.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.563613+0000) 2019-10-13T08:39:37.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:37.584+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.589522+0000) 2019-10-13T08:39:37.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:37.992+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.995308+0000) 2019-10-13T08:39:37.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:37.992+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.996062+0000) 2019-10-13T08:39:37.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:37.992+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.996763+0000) 2019-10-13T08:39:37.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:37.995+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.997285+0000) 2019-10-13T08:39:37.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:37.996+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.998140+0000) 2019-10-13T08:39:37.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:37.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:37.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:37.996+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.998123+0000) 2019-10-13T08:39:37.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:37.996+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:37.998835+0000) 2019-10-13T08:39:38.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:37.999+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:38.000519+0000) 2019-10-13T08:39:41.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:41.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:41.612098+0000) 2019-10-13T08:39:41.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:41.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:41.613296+0000) 2019-10-13T08:39:42.565 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:39:42.565 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:42.569 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:39:42.569 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:39:43.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:42.999+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.001790+0000) 2019-10-13T08:39:43.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:43.000+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.002911+0000) 2019-10-13T08:39:43.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:43.000+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.003312+0000) 2019-10-13T08:39:43.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:43.000+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.004072+0000) 2019-10-13T08:39:43.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:43.000+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.004607+0000) 2019-10-13T08:39:43.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:43.000+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.004349+0000) 2019-10-13T08:39:43.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:43.000+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.005024+0000) 2019-10-13T08:39:43.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:43.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:43.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:43.007+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.008808+0000) 2019-10-13T08:39:43.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:43.924+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:43.927184+0000) 2019-10-13T08:39:44.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:44.336+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:44.338211+0000) 2019-10-13T08:39:45.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:45.487+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:45.489476+0000) 2019-10-13T08:39:47.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:47.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:47.466262+0000) 2019-10-13T08:39:47.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:47.560+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:47.563817+0000) 2019-10-13T08:39:47.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:47.588+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:47.589722+0000) 2019-10-13T08:39:48.009 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:48.004+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.008037+0000) 2019-10-13T08:39:48.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:48.003+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.007753+0000) 2019-10-13T08:39:48.010 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:48.004+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.009441+0000) 2019-10-13T08:39:48.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:48.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.010471+0000) 2019-10-13T08:39:48.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:48.008+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.009794+0000) 2019-10-13T08:39:48.012 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:48.008+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.010607+0000) 2019-10-13T08:39:48.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:48.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:48.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:48.008+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.012205+0000) 2019-10-13T08:39:48.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:48.015+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:48.017080+0000) 2019-10-13T08:39:51.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:51.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:51.612324+0000) 2019-10-13T08:39:51.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:51.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:51.613494+0000) 2019-10-13T08:39:53.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:53.012+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.017649+0000) 2019-10-13T08:39:53.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:53.016+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.017958+0000) 2019-10-13T08:39:53.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:53.015+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.017441+0000) 2019-10-13T08:39:53.019 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:53.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:53.020 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:53.016+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.018777+0000) 2019-10-13T08:39:53.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:53.016+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.019199+0000) 2019-10-13T08:39:53.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:53.019+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.021085+0000) 2019-10-13T08:39:53.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:53.020+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.022274+0000) 2019-10-13T08:39:53.023 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:53.020+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.022814+0000) 2019-10-13T08:39:53.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:53.923+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:53.927359+0000) 2019-10-13T08:39:54.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:54.335+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:54.338445+0000) 2019-10-13T08:39:55.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:55.495+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:55.495997+0000) 2019-10-13T08:39:57.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:57.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:57.466462+0000) 2019-10-13T08:39:57.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:57.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:57.564016+0000) 2019-10-13T08:39:57.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:57.587+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:57.589990+0000) 2019-10-13T08:39:58.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:39:58.015+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.021559+0000) 2019-10-13T08:39:58.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:39:58.019+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.023871+0000) 2019-10-13T08:39:58.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:39:58.019+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.023907+0000) 2019-10-13T08:39:58.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:39:58.019+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.024255+0000) 2019-10-13T08:39:58.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:39:58.019+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.024619+0000) 2019-10-13T08:39:58.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:39:58.019+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.025082+0000) 2019-10-13T08:39:58.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:39:58.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:39:58.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:39:58.023+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.025355+0000) 2019-10-13T08:39:58.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:39:58.023+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:39:58.025379+0000) 2019-10-13T08:40:01.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:01.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:01.612522+0000) 2019-10-13T08:40:01.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:01.611+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:01.613726+0000) 2019-10-13T08:40:03.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:03.023+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.028373+0000) 2019-10-13T08:40:03.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:03.027+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.030595+0000) 2019-10-13T08:40:03.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:03.027+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.030219+0000) 2019-10-13T08:40:03.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:03.027+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.031583+0000) 2019-10-13T08:40:03.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:03.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:03.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:03.027+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.032009+0000) 2019-10-13T08:40:03.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:03.027+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.032429+0000) 2019-10-13T08:40:03.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:03.027+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.033481+0000) 2019-10-13T08:40:03.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:03.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.038598+0000) 2019-10-13T08:40:03.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:03.923+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:03.927554+0000) 2019-10-13T08:40:04.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:04.335+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:04.338650+0000) 2019-10-13T08:40:05.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:05.498+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:05.502564+0000) 2019-10-13T08:40:07.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:07.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:07.466673+0000) 2019-10-13T08:40:07.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:07.563+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:07.564216+0000) 2019-10-13T08:40:07.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:07.587+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:07.590257+0000) 2019-10-13T08:40:08.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:08.031+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.037199+0000) 2019-10-13T08:40:08.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:08.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:08.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:08.035+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.038040+0000) 2019-10-13T08:40:08.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:08.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.038380+0000) 2019-10-13T08:40:08.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:08.034+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.038533+0000) 2019-10-13T08:40:08.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:08.035+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.038718+0000) 2019-10-13T08:40:08.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:08.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.039563+0000) 2019-10-13T08:40:08.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:08.035+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.040566+0000) 2019-10-13T08:40:08.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:08.038+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:08.042579+0000) 2019-10-13T08:40:11.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:11.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:11.612718+0000) 2019-10-13T08:40:11.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:11.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:11.613922+0000) 2019-10-13T08:40:12.792 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:40:12.792 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:12.796 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:40:12.796 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:13.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:13.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.040929+0000) 2019-10-13T08:40:13.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:13.035+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.041303+0000) 2019-10-13T08:40:13.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:13.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.041666+0000) 2019-10-13T08:40:13.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:13.038+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.041759+0000) 2019-10-13T08:40:13.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:13.039+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.042431+0000) 2019-10-13T08:40:13.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:13.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.043575+0000) 2019-10-13T08:40:13.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:13.039+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.043659+0000) 2019-10-13T08:40:13.044 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:13.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:13.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:13.046+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.050675+0000) 2019-10-13T08:40:13.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:13.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:13.927742+0000) 2019-10-13T08:40:14.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:14.335+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:14.338849+0000) 2019-10-13T08:40:15.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:15.506+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:15.509248+0000) 2019-10-13T08:40:17.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:17.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:17.466872+0000) 2019-10-13T08:40:17.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:17.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:17.564450+0000) 2019-10-13T08:40:17.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:17.587+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:17.590475+0000) 2019-10-13T08:40:18.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:18.043+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.048042+0000) 2019-10-13T08:40:18.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:18.043+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.049660+0000) 2019-10-13T08:40:18.051 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:18.046+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.050565+0000) 2019-10-13T08:40:18.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:18.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:18.053 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:18.047+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.053029+0000) 2019-10-13T08:40:18.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:18.047+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.053592+0000) 2019-10-13T08:40:18.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:18.051+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.053923+0000) 2019-10-13T08:40:18.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:18.051+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.054441+0000) 2019-10-13T08:40:18.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:18.054+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:18.057700+0000) 2019-10-13T08:40:21.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:21.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:21.612934+0000) 2019-10-13T08:40:21.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:21.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:21.614135+0000) 2019-10-13T08:40:23.055 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:23.050+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.054671+0000) 2019-10-13T08:40:23.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:23.051+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.055774+0000) 2019-10-13T08:40:23.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:23.051+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.056575+0000) 2019-10-13T08:40:23.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:23.055+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.058275+0000) 2019-10-13T08:40:23.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:23.055+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.058553+0000) 2019-10-13T08:40:23.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:23.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:23.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:23.055+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.060535+0000) 2019-10-13T08:40:23.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:23.055+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.061032+0000) 2019-10-13T08:40:23.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:23.062+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.064616+0000) 2019-10-13T08:40:23.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:23.923+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:23.927955+0000) 2019-10-13T08:40:24.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:24.335+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:24.339055+0000) 2019-10-13T08:40:25.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:25.510+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:25.515642+0000) 2019-10-13T08:40:27.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:27.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:27.467071+0000) 2019-10-13T08:40:27.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:27.563+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:27.564650+0000) 2019-10-13T08:40:27.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:27.587+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:27.590720+0000) 2019-10-13T08:40:28.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:28.059+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.062788+0000) 2019-10-13T08:40:28.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:28.059+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.063169+0000) 2019-10-13T08:40:28.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:28.059+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.063654+0000) 2019-10-13T08:40:28.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:28.059+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.064084+0000) 2019-10-13T08:40:28.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:28.058+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.063830+0000) 2019-10-13T08:40:28.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:28.063+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.065847+0000) 2019-10-13T08:40:28.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:28.063+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.066129+0000) 2019-10-13T08:40:28.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:28.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:28.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:28.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:28.071379+0000) 2019-10-13T08:40:31.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:31.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:31.613127+0000) 2019-10-13T08:40:31.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:31.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:31.614323+0000) 2019-10-13T08:40:33.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:33.063+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.069289+0000) 2019-10-13T08:40:33.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:33.067+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.070904+0000) 2019-10-13T08:40:33.071 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:33.067+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.071722+0000) 2019-10-13T08:40:33.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:33.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:33.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:33.067+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.073489+0000) 2019-10-13T08:40:33.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:33.070+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.073448+0000) 2019-10-13T08:40:33.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:33.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.074041+0000) 2019-10-13T08:40:33.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:33.071+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.074147+0000) 2019-10-13T08:40:33.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:33.070+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.076008+0000) 2019-10-13T08:40:33.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:33.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:33.928143+0000) 2019-10-13T08:40:34.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:34.335+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:34.339281+0000) 2019-10-13T08:40:35.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:35.518+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:35.522200+0000) 2019-10-13T08:40:37.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:37.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:37.467298+0000) 2019-10-13T08:40:37.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:37.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:37.564844+0000) 2019-10-13T08:40:37.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:37.587+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:37.590945+0000) 2019-10-13T08:40:38.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:38.071+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.076230+0000) 2019-10-13T08:40:38.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:38.071+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.077428+0000) 2019-10-13T08:40:38.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:38.075+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.078345+0000) 2019-10-13T08:40:38.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:38.074+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.078255+0000) 2019-10-13T08:40:38.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:38.075+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.079003+0000) 2019-10-13T08:40:38.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:38.075+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.079712+0000) 2019-10-13T08:40:38.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:38.075+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:38.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:38.075+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.080496+0000) 2019-10-13T08:40:38.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:38.078+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:38.083589+0000) 2019-10-13T08:40:41.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:41.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:41.613360+0000) 2019-10-13T08:40:41.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:41.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:41.614517+0000) 2019-10-13T08:40:42.900 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:40:42.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:42.904 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:40:42.904 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:40:43.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:43.079+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.082670+0000) 2019-10-13T08:40:43.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:43.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.083502+0000) 2019-10-13T08:40:43.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:43.079+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.084234+0000) 2019-10-13T08:40:43.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:43.079+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.084695+0000) 2019-10-13T08:40:43.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:43.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:43.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:43.079+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.085082+0000) 2019-10-13T08:40:43.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:43.082+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.084977+0000) 2019-10-13T08:40:43.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:43.083+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.086400+0000) 2019-10-13T08:40:43.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:43.086+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.091229+0000) 2019-10-13T08:40:43.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:43.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:43.928363+0000) 2019-10-13T08:40:44.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:44.339+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:44.339482+0000) 2019-10-13T08:40:45.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:45.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:45.528854+0000) 2019-10-13T08:40:47.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:47.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:47.467495+0000) 2019-10-13T08:40:47.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:47.559+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:47.565039+0000) 2019-10-13T08:40:47.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:47.591+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:47.591146+0000) 2019-10-13T08:40:48.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:48.083+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.087935+0000) 2019-10-13T08:40:48.089 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:48.083+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.087984+0000) 2019-10-13T08:40:48.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:48.083+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.088490+0000) 2019-10-13T08:40:48.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:48.087+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.091262+0000) 2019-10-13T08:40:48.091 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:48.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:48.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:48.086+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.091843+0000) 2019-10-13T08:40:48.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:48.091+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.094509+0000) 2019-10-13T08:40:48.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:48.091+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.095574+0000) 2019-10-13T08:40:48.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:48.090+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:48.095406+0000) 2019-10-13T08:40:51.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:51.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:51.613578+0000) 2019-10-13T08:40:51.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:51.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:51.614737+0000) 2019-10-13T08:40:53.095 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:53.090+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.094358+0000) 2019-10-13T08:40:53.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:53.090+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.095605+0000) 2019-10-13T08:40:53.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:53.090+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.095092+0000) 2019-10-13T08:40:53.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:53.090+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.094524+0000) 2019-10-13T08:40:53.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:53.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.094247+0000) 2019-10-13T08:40:53.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:53.090+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.095932+0000) 2019-10-13T08:40:53.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:53.090+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.095345+0000) 2019-10-13T08:40:53.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:53.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:53.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:53.098+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.102168+0000) 2019-10-13T08:40:53.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:53.922+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:53.928545+0000) 2019-10-13T08:40:54.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:54.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:54.339684+0000) 2019-10-13T08:40:55.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:55.534+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:55.535330+0000) 2019-10-13T08:40:57.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:57.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:57.467703+0000) 2019-10-13T08:40:57.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:57.562+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:57.565254+0000) 2019-10-13T08:40:57.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:57.590+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:57.591404+0000) 2019-10-13T08:40:58.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:40:58.094+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.101424+0000) 2019-10-13T08:40:58.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:40:58.098+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.101954+0000) 2019-10-13T08:40:58.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:40:58.098+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.102092+0000) 2019-10-13T08:40:58.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:40:58.098+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.103219+0000) 2019-10-13T08:40:58.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:40:58.098+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.103227+0000) 2019-10-13T08:40:58.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:40:58.098+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.103642+0000) 2019-10-13T08:40:58.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:40:58.098+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.104416+0000) 2019-10-13T08:40:58.104 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:40:58.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:40:58.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:40:58.106+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:40:58.111223+0000) 2019-10-13T08:41:01.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:01.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:01.613772+0000) 2019-10-13T08:41:01.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:01.610+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:01.614933+0000) 2019-10-13T08:41:03.108 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:03.102+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.107803+0000) 2019-10-13T08:41:03.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:03.102+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.108226+0000) 2019-10-13T08:41:03.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:03.102+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.109431+0000) 2019-10-13T08:41:03.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:03.106+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.109450+0000) 2019-10-13T08:41:03.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:03.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.110771+0000) 2019-10-13T08:41:03.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:03.106+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.110849+0000) 2019-10-13T08:41:03.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:03.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:03.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:03.106+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.111643+0000) 2019-10-13T08:41:03.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:03.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.115737+0000) 2019-10-13T08:41:03.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:03.926+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:03.928760+0000) 2019-10-13T08:41:04.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:04.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:04.339919+0000) 2019-10-13T08:41:05.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:05.538+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:05.541856+0000) 2019-10-13T08:41:07.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:07.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:07.467950+0000) 2019-10-13T08:41:07.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:07.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:07.565447+0000) 2019-10-13T08:41:07.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:07.590+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:07.591659+0000) 2019-10-13T08:41:08.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:08.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.114252+0000) 2019-10-13T08:41:08.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:08.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.114498+0000) 2019-10-13T08:41:08.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:08.110+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.114820+0000) 2019-10-13T08:41:08.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:08.110+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.115259+0000) 2019-10-13T08:41:08.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:08.110+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.116148+0000) 2019-10-13T08:41:08.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:08.110+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.116265+0000) 2019-10-13T08:41:08.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:08.110+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.116805+0000) 2019-10-13T08:41:08.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:08.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:08.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:08.118+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:08.121330+0000) 2019-10-13T08:41:11.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:11.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:11.613970+0000) 2019-10-13T08:41:11.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:11.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:11.615126+0000) 2019-10-13T08:41:13.098 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:41:13.098 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:13.102 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:41:13.102 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:13.120 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:13.114+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.118876+0000) 2019-10-13T08:41:13.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:13.114+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.119413+0000) 2019-10-13T08:41:13.120 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:13.114+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.120428+0000) 2019-10-13T08:41:13.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:13.114+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.120940+0000) 2019-10-13T08:41:13.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:13.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.121427+0000) 2019-10-13T08:41:13.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:13.117+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.121328+0000) 2019-10-13T08:41:13.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:13.118+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.122891+0000) 2019-10-13T08:41:13.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:13.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:13.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:13.121+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.126031+0000) 2019-10-13T08:41:13.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:13.926+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:13.928976+0000) 2019-10-13T08:41:14.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:14.334+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:14.340120+0000) 2019-10-13T08:41:15.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:15.545+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:15.548288+0000) 2019-10-13T08:41:17.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:17.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:17.468149+0000) 2019-10-13T08:41:17.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:17.558+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:17.565640+0000) 2019-10-13T08:41:17.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:17.586+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:17.591886+0000) 2019-10-13T08:41:18.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:18.122+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.126491+0000) 2019-10-13T08:41:18.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:18.122+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.126924+0000) 2019-10-13T08:41:18.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:18.121+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.126884+0000) 2019-10-13T08:41:18.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:18.122+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.128003+0000) 2019-10-13T08:41:18.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:18.122+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.128583+0000) 2019-10-13T08:41:18.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:18.122+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.129121+0000) 2019-10-13T08:41:18.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:18.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:18.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:18.126+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.131077+0000) 2019-10-13T08:41:18.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:18.129+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:18.135148+0000) 2019-10-13T08:41:21.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:21.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:21.614165+0000) 2019-10-13T08:41:21.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:21.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:21.615325+0000) 2019-10-13T08:41:23.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:23.130+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.133865+0000) 2019-10-13T08:41:23.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:23.130+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.134320+0000) 2019-10-13T08:41:23.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:23.130+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.134680+0000) 2019-10-13T08:41:23.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:23.130+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.134954+0000) 2019-10-13T08:41:23.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:23.130+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.135258+0000) 2019-10-13T08:41:23.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:23.130+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.136107+0000) 2019-10-13T08:41:23.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:23.133+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.136082+0000) 2019-10-13T08:41:23.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:23.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:23.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:23.141+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.144255+0000) 2019-10-13T08:41:23.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:23.926+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:23.929165+0000) 2019-10-13T08:41:24.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:24.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:24.340340+0000) 2019-10-13T08:41:25.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:25.549+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:25.554834+0000) 2019-10-13T08:41:27.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:27.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:27.468350+0000) 2019-10-13T08:41:27.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:27.562+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:27.565840+0000) 2019-10-13T08:41:27.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:27.590+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:27.592088+0000) 2019-10-13T08:41:28.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:28.134+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.140332+0000) 2019-10-13T08:41:28.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:28.134+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.140839+0000) 2019-10-13T08:41:28.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:28.134+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.141441+0000) 2019-10-13T08:41:28.142 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:28.138+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.142092+0000) 2019-10-13T08:41:28.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:28.137+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.142362+0000) 2019-10-13T08:41:28.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:28.138+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.143317+0000) 2019-10-13T08:41:28.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:28.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:28.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:28.138+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.144163+0000) 2019-10-13T08:41:28.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:28.141+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:28.147732+0000) 2019-10-13T08:41:31.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:31.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:31.614392+0000) 2019-10-13T08:41:31.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:31.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:31.615571+0000) 2019-10-13T08:41:33.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:33.142+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.146761+0000) 2019-10-13T08:41:33.147 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:33.141+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.146682+0000) 2019-10-13T08:41:33.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:33.142+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.147860+0000) 2019-10-13T08:41:33.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:33.146+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.149811+0000) 2019-10-13T08:41:33.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:33.146+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.150240+0000) 2019-10-13T08:41:33.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:33.146+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:33.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:33.146+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.150587+0000) 2019-10-13T08:41:33.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:33.146+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.151249+0000) 2019-10-13T08:41:33.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:33.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.156817+0000) 2019-10-13T08:41:33.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:33.926+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:33.929382+0000) 2019-10-13T08:41:34.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:34.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:34.340536+0000) 2019-10-13T08:41:35.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:35.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:35.561448+0000) 2019-10-13T08:41:37.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:37.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:37.468570+0000) 2019-10-13T08:41:37.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:37.562+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:37.566041+0000) 2019-10-13T08:41:37.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:37.590+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:37.592288+0000) 2019-10-13T08:41:38.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:38.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.153930+0000) 2019-10-13T08:41:38.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:38.150+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.154616+0000) 2019-10-13T08:41:38.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:38.150+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.155005+0000) 2019-10-13T08:41:38.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:38.150+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.156501+0000) 2019-10-13T08:41:38.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:38.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78298 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:38.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:38.150+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.157035+0000) 2019-10-13T08:41:38.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:38.150+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.157719+0000) 2019-10-13T08:41:38.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:38.154+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.159211+0000) 2019-10-13T08:41:38.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:38.161+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:38.164713+0000) 2019-10-13T08:41:41.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:41.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:41.614585+0000) 2019-10-13T08:41:41.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:41.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:41.615840+0000) 2019-10-13T08:41:43.160 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:43.153+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.159417+0000) 2019-10-13T08:41:43.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:43.154+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.160524+0000) 2019-10-13T08:41:43.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:43.154+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.161211+0000) 2019-10-13T08:41:43.163 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:43.158+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.163306+0000) 2019-10-13T08:41:43.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:43.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:43.164 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:43.158+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.164279+0000) 2019-10-13T08:41:43.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:43.158+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.163692+0000) 2019-10-13T08:41:43.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:43.158+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.164626+0000) 2019-10-13T08:41:43.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:43.161+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.166353+0000) 2019-10-13T08:41:43.243 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:41:43.243 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:43.247 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:41:43.247 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:41:43.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:43.926+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:43.929576+0000) 2019-10-13T08:41:44.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:44.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:44.340697+0000) 2019-10-13T08:41:45.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:45.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:45.567989+0000) 2019-10-13T08:41:47.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:47.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:47.468805+0000) 2019-10-13T08:41:47.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:47.561+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:47.566199+0000) 2019-10-13T08:41:47.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:47.589+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:47.592481+0000) 2019-10-13T08:41:48.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:48.161+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.166465+0000) 2019-10-13T08:41:48.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:48.161+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.166966+0000) 2019-10-13T08:41:48.168 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:48.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.167909+0000) 2019-10-13T08:41:48.168 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:48.165+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.168139+0000) 2019-10-13T08:41:48.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:48.161+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.169195+0000) 2019-10-13T08:41:48.169 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:48.161+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.169591+0000) 2019-10-13T08:41:48.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:48.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:48.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:48.165+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.170105+0000) 2019-10-13T08:41:48.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:48.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:48.174380+0000) 2019-10-13T08:41:51.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:51.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:51.614815+0000) 2019-10-13T08:41:51.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:51.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:51.616033+0000) 2019-10-13T08:41:53.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:53.165+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.172819+0000) 2019-10-13T08:41:53.173 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:53.169+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.172844+0000) 2019-10-13T08:41:53.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:53.169+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.174224+0000) 2019-10-13T08:41:53.174 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:53.169+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.174482+0000) 2019-10-13T08:41:53.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:53.169+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.175031+0000) 2019-10-13T08:41:53.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:53.169+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.175563+0000) 2019-10-13T08:41:53.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:53.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:53.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:53.169+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.176609+0000) 2019-10-13T08:41:53.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:53.177+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.181855+0000) 2019-10-13T08:41:53.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:53.925+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:53.929798+0000) 2019-10-13T08:41:54.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:54.333+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:54.340897+0000) 2019-10-13T08:41:55.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:55.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:55.574641+0000) 2019-10-13T08:41:57.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:57.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:57.469024+0000) 2019-10-13T08:41:57.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:57.565+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:57.566397+0000) 2019-10-13T08:41:57.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:57.589+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:57.592715+0000) 2019-10-13T08:41:58.180 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:41:58.173+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.179910+0000) 2019-10-13T08:41:58.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:41:58.173+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.180348+0000) 2019-10-13T08:41:58.181 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:41:58.173+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.181300+0000) 2019-10-13T08:41:58.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:41:58.177+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.182933+0000) 2019-10-13T08:41:58.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:41:58.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:41:58.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:41:58.177+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.183202+0000) 2019-10-13T08:41:58.184 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:41:58.177+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.183350+0000) 2019-10-13T08:41:58.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:41:58.177+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.184473+0000) 2019-10-13T08:41:58.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:41:58.185+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:41:58.191044+0000) 2019-10-13T08:42:01.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:01.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:01.615011+0000) 2019-10-13T08:42:01.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:01.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:01.616278+0000) 2019-10-13T08:42:03.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:03.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.186786+0000) 2019-10-13T08:42:03.187 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:03.181+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.186870+0000) 2019-10-13T08:42:03.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:03.185+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.188915+0000) 2019-10-13T08:42:03.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:03.185+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.189095+0000) 2019-10-13T08:42:03.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:03.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:03.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:03.189+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.190285+0000) 2019-10-13T08:42:03.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:03.189+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.190714+0000) 2019-10-13T08:42:03.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:03.189+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.190771+0000) 2019-10-13T08:42:03.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:03.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.193881+0000) 2019-10-13T08:42:03.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:03.929+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:03.930039+0000) 2019-10-13T08:42:04.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:04.337+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:04.341099+0000) 2019-10-13T08:42:05.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:05.577+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:05.581174+0000) 2019-10-13T08:42:07.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:07.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:07.469189+0000) 2019-10-13T08:42:07.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:07.565+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:07.566587+0000) 2019-10-13T08:42:07.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:07.589+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:07.592914+0000) 2019-10-13T08:42:08.193 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:08.189+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.192629+0000) 2019-10-13T08:42:08.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:08.189+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.193289+0000) 2019-10-13T08:42:08.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:08.193+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.193913+0000) 2019-10-13T08:42:08.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:08.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.194715+0000) 2019-10-13T08:42:08.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:08.193+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.195720+0000) 2019-10-13T08:42:08.196 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:08.189+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.195295+0000) 2019-10-13T08:42:08.196 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:08.193+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.195916+0000) 2019-10-13T08:42:08.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:08.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:08.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:08.197+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:08.200475+0000) 2019-10-13T08:42:11.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:11.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:11.615206+0000) 2019-10-13T08:42:11.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:11.613+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:11.616471+0000) 2019-10-13T08:42:13.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:13.197+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.200073+0000) 2019-10-13T08:42:13.202 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:13.201+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.201834+0000) 2019-10-13T08:42:13.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:13.201+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.202053+0000) 2019-10-13T08:42:13.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:13.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.201927+0000) 2019-10-13T08:42:13.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:13.201+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.202699+0000) 2019-10-13T08:42:13.203 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:13.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:13.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:13.201+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.203417+0000) 2019-10-13T08:42:13.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:13.201+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.204126+0000) 2019-10-13T08:42:13.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:13.205+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.210133+0000) 2019-10-13T08:42:13.516 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:42:13.516 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:13.521 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:42:13.521 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:13.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:13.929+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:13.930189+0000) 2019-10-13T08:42:14.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:14.337+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:14.341303+0000) 2019-10-13T08:42:15.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:15.585+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:15.587664+0000) 2019-10-13T08:42:17.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:17.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:17.469414+0000) 2019-10-13T08:42:17.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:17.565+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:17.566787+0000) 2019-10-13T08:42:17.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:17.589+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:17.593153+0000) 2019-10-13T08:42:18.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:18.205+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.206476+0000) 2019-10-13T08:42:18.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:18.205+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.206747+0000) 2019-10-13T08:42:18.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:18.205+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.207150+0000) 2019-10-13T08:42:18.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:18.204+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.208779+0000) 2019-10-13T08:42:18.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:18.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78546 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:18.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:18.205+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.209714+0000) 2019-10-13T08:42:18.210 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:18.209+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.210284+0000) 2019-10-13T08:42:18.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:18.209+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.210912+0000) 2019-10-13T08:42:18.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:18.212+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:18.216855+0000) 2019-10-13T08:42:21.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:21.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:21.615365+0000) 2019-10-13T08:42:21.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:21.612+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:21.616679+0000) 2019-10-13T08:42:23.213 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:23.209+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.212518+0000) 2019-10-13T08:42:23.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:23.209+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.213697+0000) 2019-10-13T08:42:23.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:23.208+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.214481+0000) 2019-10-13T08:42:23.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:23.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.215168+0000) 2019-10-13T08:42:23.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:23.213+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.215717+0000) 2019-10-13T08:42:23.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:23.213+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.214493+0000) 2019-10-13T08:42:23.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:23.213+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.216139+0000) 2019-10-13T08:42:23.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:23.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:23.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:23.216+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.220201+0000) 2019-10-13T08:42:23.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:23.929+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:23.930331+0000) 2019-10-13T08:42:24.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:24.341+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:24.341504+0000) 2019-10-13T08:42:25.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:25.596+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:25.599248+0000) 2019-10-13T08:42:27.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:27.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:27.469611+0000) 2019-10-13T08:42:27.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:27.565+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:27.566991+0000) 2019-10-13T08:42:27.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:27.593+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:27.593348+0000) 2019-10-13T08:42:28.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:28.212+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.218801+0000) 2019-10-13T08:42:28.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:28.217+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.221378+0000) 2019-10-13T08:42:28.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:28.221+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.222013+0000) 2019-10-13T08:42:28.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:28.221+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.222125+0000) 2019-10-13T08:42:28.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:28.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78609 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:28.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:28.221+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.223505+0000) 2019-10-13T08:42:28.224 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:28.221+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.223869+0000) 2019-10-13T08:42:28.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:28.221+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.225205+0000) 2019-10-13T08:42:28.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:28.224+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:28.228607+0000) 2019-10-13T08:42:31.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:31.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:31.615578+0000) 2019-10-13T08:42:31.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:31.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:31.616877+0000) 2019-10-13T08:42:33.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:33.225+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.226117+0000) 2019-10-13T08:42:33.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:33.225+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.226427+0000) 2019-10-13T08:42:33.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:33.225+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.228694+0000) 2019-10-13T08:42:33.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:33.225+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.228985+0000) 2019-10-13T08:42:33.230 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:33.225+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.229467+0000) 2019-10-13T08:42:33.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:33.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:33.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:33.229+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.230561+0000) 2019-10-13T08:42:33.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:33.224+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.230249+0000) 2019-10-13T08:42:33.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:33.228+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.235835+0000) 2019-10-13T08:42:33.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:33.929+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:33.930555+0000) 2019-10-13T08:42:34.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:34.341+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:34.341768+0000) 2019-10-13T08:42:35.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:35.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:35.605742+0000) 2019-10-13T08:42:37.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:37.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:37.469851+0000) 2019-10-13T08:42:37.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:37.565+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:37.567227+0000) 2019-10-13T08:42:37.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:37.593+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:37.593583+0000) 2019-10-13T08:42:38.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:38.229+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.233323+0000) 2019-10-13T08:42:38.234 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:38.229+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.233632+0000) 2019-10-13T08:42:38.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:38.233+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.234038+0000) 2019-10-13T08:42:38.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:38.233+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.235113+0000) 2019-10-13T08:42:38.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:38.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:38.237 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:38.233+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.236743+0000) 2019-10-13T08:42:38.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:38.232+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.236600+0000) 2019-10-13T08:42:38.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:38.233+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.237483+0000) 2019-10-13T08:42:38.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:38.240+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:38.245420+0000) 2019-10-13T08:42:41.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:41.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:41.615762+0000) 2019-10-13T08:42:41.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:41.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:41.617065+0000) 2019-10-13T08:42:43.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:43.237+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.239270+0000) 2019-10-13T08:42:43.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:43.237+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.239880+0000) 2019-10-13T08:42:43.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:43.237+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.240687+0000) 2019-10-13T08:42:43.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:43.237+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.241062+0000) 2019-10-13T08:42:43.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:43.236+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.242052+0000) 2019-10-13T08:42:43.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:43.241+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.244528+0000) 2019-10-13T08:42:43.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:43.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:43.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:43.241+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.245199+0000) 2019-10-13T08:42:43.247 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:43.240+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.247105+0000) 2019-10-13T08:42:43.646 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:42:43.646 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:43.650 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:42:43.650 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:42:43.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:43.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:43.930739+0000) 2019-10-13T08:42:44.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:44.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:44.341975+0000) 2019-10-13T08:42:45.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:45.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:45.612143+0000) 2019-10-13T08:42:47.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:47.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:47.470023+0000) 2019-10-13T08:42:47.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:47.564+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:47.567421+0000) 2019-10-13T08:42:47.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:47.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:47.593837+0000) 2019-10-13T08:42:48.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:48.244+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.246879+0000) 2019-10-13T08:42:48.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:48.244+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.248971+0000) 2019-10-13T08:42:48.250 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:48.248+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.250044+0000) 2019-10-13T08:42:48.251 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:48.248+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.250924+0000) 2019-10-13T08:42:48.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:48.248+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.250329+0000) 2019-10-13T08:42:48.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:48.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:48.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:48.248+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.251802+0000) 2019-10-13T08:42:48.253 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:48.248+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.252890+0000) 2019-10-13T08:42:48.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:48.252+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:48.257918+0000) 2019-10-13T08:42:51.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:51.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:51.615974+0000) 2019-10-13T08:42:51.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:51.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:51.617260+0000) 2019-10-13T08:42:53.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:53.252+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.254594+0000) 2019-10-13T08:42:53.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:53.252+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.254980+0000) 2019-10-13T08:42:53.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:53.252+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:53.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:53.252+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.257959+0000) 2019-10-13T08:42:53.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:53.256+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.258766+0000) 2019-10-13T08:42:53.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:53.256+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.259016+0000) 2019-10-13T08:42:53.260 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:53.256+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.259380+0000) 2019-10-13T08:42:53.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:53.256+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.259643+0000) 2019-10-13T08:42:53.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:53.256+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.262253+0000) 2019-10-13T08:42:53.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:53.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:53.930924+0000) 2019-10-13T08:42:54.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:54.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:54.342195+0000) 2019-10-13T08:42:55.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:55.612+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:55.618553+0000) 2019-10-13T08:42:57.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:57.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:57.470205+0000) 2019-10-13T08:42:57.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:57.564+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:57.567618+0000) 2019-10-13T08:42:57.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:57.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:57.594002+0000) 2019-10-13T08:42:58.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:42:58.256+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.260758+0000) 2019-10-13T08:42:58.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:42:58.260+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.262101+0000) 2019-10-13T08:42:58.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:42:58.260+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.262676+0000) 2019-10-13T08:42:58.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:42:58.260+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.263559+0000) 2019-10-13T08:42:58.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:42:58.256+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.263367+0000) 2019-10-13T08:42:58.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:42:58.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:42:58.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:42:58.260+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.263996+0000) 2019-10-13T08:42:58.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:42:58.260+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.264426+0000) 2019-10-13T08:42:58.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:42:58.264+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:42:58.270061+0000) 2019-10-13T08:43:01.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:01.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:01.616171+0000) 2019-10-13T08:43:01.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:01.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:01.617456+0000) 2019-10-13T08:43:03.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:03.264+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.267010+0000) 2019-10-13T08:43:03.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:03.268+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.269810+0000) 2019-10-13T08:43:03.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:03.264+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.269517+0000) 2019-10-13T08:43:03.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:03.268+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.270219+0000) 2019-10-13T08:43:03.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:03.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:03.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:03.268+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.270699+0000) 2019-10-13T08:43:03.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:03.264+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.270777+0000) 2019-10-13T08:43:03.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:03.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.271963+0000) 2019-10-13T08:43:03.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:03.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.273329+0000) 2019-10-13T08:43:03.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:03.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:03.931117+0000) 2019-10-13T08:43:04.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:04.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:04.342395+0000) 2019-10-13T08:43:05.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:05.620+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:05.625066+0000) 2019-10-13T08:43:07.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:07.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:07.470439+0000) 2019-10-13T08:43:07.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:07.564+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:07.567836+0000) 2019-10-13T08:43:07.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:07.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:07.594241+0000) 2019-10-13T08:43:08.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:08.272+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.274556+0000) 2019-10-13T08:43:08.275 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:08.272+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.274916+0000) 2019-10-13T08:43:08.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:08.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.274977+0000) 2019-10-13T08:43:08.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:08.272+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.276065+0000) 2019-10-13T08:43:08.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:08.272+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.276401+0000) 2019-10-13T08:43:08.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:08.272+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.275704+0000) 2019-10-13T08:43:08.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:08.272+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.276764+0000) 2019-10-13T08:43:08.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:08.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:08.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:08.276+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:08.281174+0000) 2019-10-13T08:43:11.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:11.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:11.616388+0000) 2019-10-13T08:43:11.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:11.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:11.617651+0000) 2019-10-13T08:43:13.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:13.276+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.280222+0000) 2019-10-13T08:43:13.281 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:13.276+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.281065+0000) 2019-10-13T08:43:13.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:13.276+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.281811+0000) 2019-10-13T08:43:13.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:13.280+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.283063+0000) 2019-10-13T08:43:13.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:13.280+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.282557+0000) 2019-10-13T08:43:13.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:13.280+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.283460+0000) 2019-10-13T08:43:13.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:13.280+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.283924+0000) 2019-10-13T08:43:13.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:13.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:13.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:13.284+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.290964+0000) 2019-10-13T08:43:13.697 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:43:13.697 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:13.701 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:43:13.701 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:13.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:13.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:13.931299+0000) 2019-10-13T08:43:14.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:14.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:14.342638+0000) 2019-10-13T08:43:15.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:15.628+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:15.631413+0000) 2019-10-13T08:43:17.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:17.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:17.470640+0000) 2019-10-13T08:43:17.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:17.564+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:17.568042+0000) 2019-10-13T08:43:17.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:17.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:17.594462+0000) 2019-10-13T08:43:18.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:18.284+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.288102+0000) 2019-10-13T08:43:18.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:18.284+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.288450+0000) 2019-10-13T08:43:18.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:18.284+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.288972+0000) 2019-10-13T08:43:18.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:18.288+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.290723+0000) 2019-10-13T08:43:18.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:18.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:18.292 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:18.288+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.292069+0000) 2019-10-13T08:43:18.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:18.288+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.292446+0000) 2019-10-13T08:43:18.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:18.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.294383+0000) 2019-10-13T08:43:18.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:18.300+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:18.301691+0000) 2019-10-13T08:43:21.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:21.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:21.616595+0000) 2019-10-13T08:43:21.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:21.616+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:21.617842+0000) 2019-10-13T08:43:23.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:23.292+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.295377+0000) 2019-10-13T08:43:23.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:23.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.296118+0000) 2019-10-13T08:43:23.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:23.292+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.296243+0000) 2019-10-13T08:43:23.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:23.292+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.297171+0000) 2019-10-13T08:43:23.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:23.296+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.297831+0000) 2019-10-13T08:43:23.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:23.295+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.297764+0000) 2019-10-13T08:43:23.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:23.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:23.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:23.296+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.298427+0000) 2019-10-13T08:43:23.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:23.299+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.303370+0000) 2019-10-13T08:43:23.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:23.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:23.931479+0000) 2019-10-13T08:43:24.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:24.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:24.342843+0000) 2019-10-13T08:43:25.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:25.635+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:25.638038+0000) 2019-10-13T08:43:27.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:27.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:27.470846+0000) 2019-10-13T08:43:27.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:27.564+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:27.568235+0000) 2019-10-13T08:43:27.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:27.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:27.594680+0000) 2019-10-13T08:43:28.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:28.300+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.301732+0000) 2019-10-13T08:43:28.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:28.300+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.302437+0000) 2019-10-13T08:43:28.303 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:28.300+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.303622+0000) 2019-10-13T08:43:28.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:28.300+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.304100+0000) 2019-10-13T08:43:28.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:28.300+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.304606+0000) 2019-10-13T08:43:28.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:28.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.304795+0000) 2019-10-13T08:43:28.306 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:28.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.304698+0000) 2019-10-13T08:43:28.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:28.300+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:28.309 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:28.307+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:28.309010+0000) 2019-10-13T08:43:31.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:31.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:31.616786+0000) 2019-10-13T08:43:31.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:31.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:31.618035+0000) 2019-10-13T08:43:33.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:33.304+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.308847+0000) 2019-10-13T08:43:33.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:33.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.310100+0000) 2019-10-13T08:43:33.310 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:33.308+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.310324+0000) 2019-10-13T08:43:33.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:33.308+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.310613+0000) 2019-10-13T08:43:33.312 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:33.308+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.312053+0000) 2019-10-13T08:43:33.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:33.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:33.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:33.311+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.312379+0000) 2019-10-13T08:43:33.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:33.308+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.313537+0000) 2019-10-13T08:43:33.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:33.319+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.320724+0000) 2019-10-13T08:43:33.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:33.928+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:33.931655+0000) 2019-10-13T08:43:34.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:34.340+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:34.343050+0000) 2019-10-13T08:43:35.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:35.643+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:35.644636+0000) 2019-10-13T08:43:37.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:37.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:37.471083+0000) 2019-10-13T08:43:37.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:37.568+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:37.568464+0000) 2019-10-13T08:43:37.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:37.592+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:37.594885+0000) 2019-10-13T08:43:38.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:38.312+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.314901+0000) 2019-10-13T08:43:38.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:38.312+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.316209+0000) 2019-10-13T08:43:38.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:38.312+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.316561+0000) 2019-10-13T08:43:38.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:38.312+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.317385+0000) 2019-10-13T08:43:38.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:38.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.317782+0000) 2019-10-13T08:43:38.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:38.316+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.318657+0000) 2019-10-13T08:43:38.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:38.316+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.319048+0000) 2019-10-13T08:43:38.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:38.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:38.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:38.323+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:38.325954+0000) 2019-10-13T08:43:41.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:41.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:41.616993+0000) 2019-10-13T08:43:41.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:41.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:41.618226+0000) 2019-10-13T08:43:43.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:43.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.322639+0000) 2019-10-13T08:43:43.324 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:43.319+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.324056+0000) 2019-10-13T08:43:43.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:43.319+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.324332+0000) 2019-10-13T08:43:43.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:43.319+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.325555+0000) 2019-10-13T08:43:43.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:43.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:43.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:43.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.326216+0000) 2019-10-13T08:43:43.327 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:43.323+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.326366+0000) 2019-10-13T08:43:43.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:43.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.327736+0000) 2019-10-13T08:43:43.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:43.331+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.332780+0000) 2019-10-13T08:43:43.812 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:43:43.812 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:43.816 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:43:43.816 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:43:43.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:43.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:43.931843+0000) 2019-10-13T08:43:44.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:44.339+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:44.343261+0000) 2019-10-13T08:43:45.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:45.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:45.651169+0000) 2019-10-13T08:43:47.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:47.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:47.471228+0000) 2019-10-13T08:43:47.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:47.567+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:47.568682+0000) 2019-10-13T08:43:47.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:47.591+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:47.595086+0000) 2019-10-13T08:43:48.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:48.327+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.328342+0000) 2019-10-13T08:43:48.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:48.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.329448+0000) 2019-10-13T08:43:48.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:48.327+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.331485+0000) 2019-10-13T08:43:48.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:48.327+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.331979+0000) 2019-10-13T08:43:48.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:48.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:48.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:48.327+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.332921+0000) 2019-10-13T08:43:48.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:48.327+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.333552+0000) 2019-10-13T08:43:48.334 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:48.331+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.334224+0000) 2019-10-13T08:43:48.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:48.335+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:48.338999+0000) 2019-10-13T08:43:51.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:51.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:51.617182+0000) 2019-10-13T08:43:51.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:51.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:51.618422+0000) 2019-10-13T08:43:53.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:53.331+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.336043+0000) 2019-10-13T08:43:53.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:53.331+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.336576+0000) 2019-10-13T08:43:53.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:53.331+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.337085+0000) 2019-10-13T08:43:53.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:53.331+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.337469+0000) 2019-10-13T08:43:53.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:53.335+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.338024+0000) 2019-10-13T08:43:53.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:53.335+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.338003+0000) 2019-10-13T08:43:53.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:53.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:53.340 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:53.335+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.340361+0000) 2019-10-13T08:43:53.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:53.343+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.346123+0000) 2019-10-13T08:43:53.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:53.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:53.932032+0000) 2019-10-13T08:43:54.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:54.339+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:54.343463+0000) 2019-10-13T08:43:55.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:55.655+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:55.657702+0000) 2019-10-13T08:43:57.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:57.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:57.471439+0000) 2019-10-13T08:43:57.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:57.567+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:57.568884+0000) 2019-10-13T08:43:57.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:57.591+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:57.595287+0000) 2019-10-13T08:43:58.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:43:58.339+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.342860+0000) 2019-10-13T08:43:58.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:43:58.339+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.344294+0000) 2019-10-13T08:43:58.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:43:58.339+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.344750+0000) 2019-10-13T08:43:58.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:43:58.339+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.344862+0000) 2019-10-13T08:43:58.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:43:58.339+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.345207+0000) 2019-10-13T08:43:58.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:43:58.343+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.345635+0000) 2019-10-13T08:43:58.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:43:58.343+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.346601+0000) 2019-10-13T08:43:58.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:43:58.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:43:58.352 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:43:58.351+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:43:58.352299+0000) 2019-10-13T08:44:01.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:01.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:01.617380+0000) 2019-10-13T08:44:01.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:01.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:01.618634+0000) 2019-10-13T08:44:03.349 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:03.347+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.349188+0000) 2019-10-13T08:44:03.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:03.347+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.350647+0000) 2019-10-13T08:44:03.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:03.347+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.351273+0000) 2019-10-13T08:44:03.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:03.347+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.352075+0000) 2019-10-13T08:44:03.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:03.347+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.351710+0000) 2019-10-13T08:44:03.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:03.347+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.352706+0000) 2019-10-13T08:44:03.353 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:03.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:03.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:03.347+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.353364+0000) 2019-10-13T08:44:03.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:03.355+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.359089+0000) 2019-10-13T08:44:03.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:03.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:03.932237+0000) 2019-10-13T08:44:04.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:04.343+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:04.343692+0000) 2019-10-13T08:44:05.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:05.663+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:05.664316+0000) 2019-10-13T08:44:07.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:07.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:07.471659+0000) 2019-10-13T08:44:07.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:07.563+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:07.569114+0000) 2019-10-13T08:44:07.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:07.591+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:07.595486+0000) 2019-10-13T08:44:08.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:08.355+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.359288+0000) 2019-10-13T08:44:08.361 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:08.351+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.357038+0000) 2019-10-13T08:44:08.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:08.355+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.358152+0000) 2019-10-13T08:44:08.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:08.355+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.358457+0000) 2019-10-13T08:44:08.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:08.355+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:08.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:08.355+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.360263+0000) 2019-10-13T08:44:08.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:08.355+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.360694+0000) 2019-10-13T08:44:08.363 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:08.359+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.361292+0000) 2019-10-13T08:44:08.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:08.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:08.363922+0000) 2019-10-13T08:44:11.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:11.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:11.617572+0000) 2019-10-13T08:44:11.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:11.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:11.618858+0000) 2019-10-13T08:44:13.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:13.359+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.363001+0000) 2019-10-13T08:44:13.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:13.359+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.363816+0000) 2019-10-13T08:44:13.365 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:13.359+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.364585+0000) 2019-10-13T08:44:13.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:13.359+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.364965+0000) 2019-10-13T08:44:13.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:13.359+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.365226+0000) 2019-10-13T08:44:13.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:13.363+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.366067+0000) 2019-10-13T08:44:13.366 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:13.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.366521+0000) 2019-10-13T08:44:13.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:13.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:13.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:13.367+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.371979+0000) 2019-10-13T08:44:13.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:13.927+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:13.932446+0000) 2019-10-13T08:44:13.952 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:44:13.952 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:13.956 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:44:13.956 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:14.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:14.339+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:14.343894+0000) 2019-10-13T08:44:15.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:15.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:15.670832+0000) 2019-10-13T08:44:17.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:17.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:17.471858+0000) 2019-10-13T08:44:17.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:17.563+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:17.569293+0000) 2019-10-13T08:44:17.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:17.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:17.595687+0000) 2019-10-13T08:44:18.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:18.367+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.370400+0000) 2019-10-13T08:44:18.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:18.367+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.370768+0000) 2019-10-13T08:44:18.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:18.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.370596+0000) 2019-10-13T08:44:18.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:18.367+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.371524+0000) 2019-10-13T08:44:18.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:18.367+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.372105+0000) 2019-10-13T08:44:18.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:18.367+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.371800+0000) 2019-10-13T08:44:18.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:18.367+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.372995+0000) 2019-10-13T08:44:18.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:18.367+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:18.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:18.375+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:18.378580+0000) 2019-10-13T08:44:21.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:21.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:21.617780+0000) 2019-10-13T08:44:21.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:21.615+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:21.619082+0000) 2019-10-13T08:44:23.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:23.375+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.378793+0000) 2019-10-13T08:44:23.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:23.375+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.379560+0000) 2019-10-13T08:44:23.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:23.375+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:23.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:23.375+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.379941+0000) 2019-10-13T08:44:23.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:23.375+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.380259+0000) 2019-10-13T08:44:23.381 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:23.379+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.380202+0000) 2019-10-13T08:44:23.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:23.375+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.381631+0000) 2019-10-13T08:44:23.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:23.379+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.382149+0000) 2019-10-13T08:44:23.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:23.383+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.387229+0000) 2019-10-13T08:44:23.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:23.931+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:23.932627+0000) 2019-10-13T08:44:24.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:24.343+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:24.344075+0000) 2019-10-13T08:44:25.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:25.675+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:25.677758+0000) 2019-10-13T08:44:27.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:27.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:27.472098+0000) 2019-10-13T08:44:27.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:27.563+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:27.569497+0000) 2019-10-13T08:44:27.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:27.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:27.595901+0000) 2019-10-13T08:44:28.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:28.379+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.383394+0000) 2019-10-13T08:44:28.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:28.379+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.383809+0000) 2019-10-13T08:44:28.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:28.383+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.385984+0000) 2019-10-13T08:44:28.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:28.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:28.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:28.383+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.386456+0000) 2019-10-13T08:44:28.387 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:28.382+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.386436+0000) 2019-10-13T08:44:28.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:28.383+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.387512+0000) 2019-10-13T08:44:28.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:28.383+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.387742+0000) 2019-10-13T08:44:28.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:28.386+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:28.390699+0000) 2019-10-13T08:44:31.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:31.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:31.617989+0000) 2019-10-13T08:44:31.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:31.614+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:31.619308+0000) 2019-10-13T08:44:33.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:33.383+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.388776+0000) 2019-10-13T08:44:33.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:33.383+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.389089+0000) 2019-10-13T08:44:33.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:33.386+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.390717+0000) 2019-10-13T08:44:33.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:33.387+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.389816+0000) 2019-10-13T08:44:33.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:33.387+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.390475+0000) 2019-10-13T08:44:33.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:33.387+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.391795+0000) 2019-10-13T08:44:33.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:33.387+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.393075+0000) 2019-10-13T08:44:33.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:33.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:33.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:33.394+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.399918+0000) 2019-10-13T08:44:33.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:33.931+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:33.932806+0000) 2019-10-13T08:44:34.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:34.343+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:34.344278+0000) 2019-10-13T08:44:35.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:35.682+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:35.684216+0000) 2019-10-13T08:44:37.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:37.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:37.472301+0000) 2019-10-13T08:44:37.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:37.566+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:37.569712+0000) 2019-10-13T08:44:37.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:37.590+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:37.596095+0000) 2019-10-13T08:44:38.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:38.390+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.396697+0000) 2019-10-13T08:44:38.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:38.394+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.396781+0000) 2019-10-13T08:44:38.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:38.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.397661+0000) 2019-10-13T08:44:38.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:38.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.398155+0000) 2019-10-13T08:44:38.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:38.394+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.398409+0000) 2019-10-13T08:44:38.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:38.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.398901+0000) 2019-10-13T08:44:38.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:38.394+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.399629+0000) 2019-10-13T08:44:38.401 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:38.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:38.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:38.402+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:38.405536+0000) 2019-10-13T08:44:41.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:41.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:41.618185+0000) 2019-10-13T08:44:41.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:41.614+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:41.619499+0000) 2019-10-13T08:44:43.405 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:43.398+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.404598+0000) 2019-10-13T08:44:43.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:43.398+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.405007+0000) 2019-10-13T08:44:43.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:43.398+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.405412+0000) 2019-10-13T08:44:43.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:43.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.405931+0000) 2019-10-13T08:44:43.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:43.402+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.406174+0000) 2019-10-13T08:44:43.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:43.402+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.406049+0000) 2019-10-13T08:44:43.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:43.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79440 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:43.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:43.402+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.408167+0000) 2019-10-13T08:44:43.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:43.410+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.413130+0000) 2019-10-13T08:44:43.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:43.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:43.932991+0000) 2019-10-13T08:44:44.076 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:44:44.076 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:44.080 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:44:44.080 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:44:44.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:44.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:44.344417+0000) 2019-10-13T08:44:45.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:45.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:45.690588+0000) 2019-10-13T08:44:47.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:47.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:47.472497+0000) 2019-10-13T08:44:47.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:47.566+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:47.569937+0000) 2019-10-13T08:44:47.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:47.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:47.596295+0000) 2019-10-13T08:44:48.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:48.406+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.410845+0000) 2019-10-13T08:44:48.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:48.406+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.412149+0000) 2019-10-13T08:44:48.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:48.406+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.412755+0000) 2019-10-13T08:44:48.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:48.410+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.412526+0000) 2019-10-13T08:44:48.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:48.406+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.413287+0000) 2019-10-13T08:44:48.414 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:48.406+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.413607+0000) 2019-10-13T08:44:48.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:48.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:48.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:48.410+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.414655+0000) 2019-10-13T08:44:48.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:48.418+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:48.420363+0000) 2019-10-13T08:44:51.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:51.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:51.618333+0000) 2019-10-13T08:44:51.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:51.618+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:51.619697+0000) 2019-10-13T08:44:53.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:53.410+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.416692+0000) 2019-10-13T08:44:53.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:53.414+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.419140+0000) 2019-10-13T08:44:53.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:53.414+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.419353+0000) 2019-10-13T08:44:53.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:53.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:53.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:53.414+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.420466+0000) 2019-10-13T08:44:53.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:53.414+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.420699+0000) 2019-10-13T08:44:53.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:53.414+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.421124+0000) 2019-10-13T08:44:53.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:53.414+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.421637+0000) 2019-10-13T08:44:53.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:53.426+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.428540+0000) 2019-10-13T08:44:53.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:53.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:53.933174+0000) 2019-10-13T08:44:54.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:54.342+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:54.344601+0000) 2019-10-13T08:44:55.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:55.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:55.697101+0000) 2019-10-13T08:44:57.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:57.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:57.472724+0000) 2019-10-13T08:44:57.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:57.566+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:57.570094+0000) 2019-10-13T08:44:57.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:57.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:57.596496+0000) 2019-10-13T08:44:58.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:44:58.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.423915+0000) 2019-10-13T08:44:58.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:44:58.418+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.424013+0000) 2019-10-13T08:44:58.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:44:58.422+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.425386+0000) 2019-10-13T08:44:58.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:44:58.422+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.426211+0000) 2019-10-13T08:44:58.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:44:58.422+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.426467+0000) 2019-10-13T08:44:58.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:44:58.422+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:44:58.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:44:58.422+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.426975+0000) 2019-10-13T08:44:58.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:44:58.422+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.427672+0000) 2019-10-13T08:44:58.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:44:58.426+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:44:58.430882+0000) 2019-10-13T08:45:01.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:01.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:01.618566+0000) 2019-10-13T08:45:01.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:01.614+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:01.619892+0000) 2019-10-13T08:45:03.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:03.426+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.429810+0000) 2019-10-13T08:45:03.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:03.426+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.430507+0000) 2019-10-13T08:45:03.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:03.426+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.430918+0000) 2019-10-13T08:45:03.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:03.426+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.431485+0000) 2019-10-13T08:45:03.432 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:03.426+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.432431+0000) 2019-10-13T08:45:03.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:03.430+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.432208+0000) 2019-10-13T08:45:03.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:03.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:03.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:03.430+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.434023+0000) 2019-10-13T08:45:03.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:03.438+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.440752+0000) 2019-10-13T08:45:03.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:03.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:03.933351+0000) 2019-10-13T08:45:04.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:04.342+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:04.344776+0000) 2019-10-13T08:45:05.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:05.702+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:05.703574+0000) 2019-10-13T08:45:07.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:07.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:07.472949+0000) 2019-10-13T08:45:07.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:07.566+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:07.570317+0000) 2019-10-13T08:45:07.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:07.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:07.596731+0000) 2019-10-13T08:45:08.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:08.430+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.436566+0000) 2019-10-13T08:45:08.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:08.430+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.437347+0000) 2019-10-13T08:45:08.438 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:08.434+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.437903+0000) 2019-10-13T08:45:08.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:08.434+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.438768+0000) 2019-10-13T08:45:08.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:08.434+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.439189+0000) 2019-10-13T08:45:08.439 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:08.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:08.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:08.434+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.439528+0000) 2019-10-13T08:45:08.440 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:08.434+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.440110+0000) 2019-10-13T08:45:08.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:08.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:08.443016+0000) 2019-10-13T08:45:11.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:11.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:11.618761+0000) 2019-10-13T08:45:11.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:11.618+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:11.620140+0000) 2019-10-13T08:45:13.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:13.438+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.442443+0000) 2019-10-13T08:45:13.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:13.438+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.442871+0000) 2019-10-13T08:45:13.445 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:13.438+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.445070+0000) 2019-10-13T08:45:13.447 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:13.442+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.445733+0000) 2019-10-13T08:45:13.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:13.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:13.447 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:13.442+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.446550+0000) 2019-10-13T08:45:13.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:13.442+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.447417+0000) 2019-10-13T08:45:13.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:13.442+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.447986+0000) 2019-10-13T08:45:13.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:13.450+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.452830+0000) 2019-10-13T08:45:13.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:13.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:13.933570+0000) 2019-10-13T08:45:14.169 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:45:14.169 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:14.173 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:45:14.173 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:14.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:14.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:14.344978+0000) 2019-10-13T08:45:15.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:15.706+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:15.710118+0000) 2019-10-13T08:45:17.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:17.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:17.473143+0000) 2019-10-13T08:45:17.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:17.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:17.570518+0000) 2019-10-13T08:45:17.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:17.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:17.596935+0000) 2019-10-13T08:45:18.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:18.442+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.449164+0000) 2019-10-13T08:45:18.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:18.446+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.452116+0000) 2019-10-13T08:45:18.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:18.446+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.452714+0000) 2019-10-13T08:45:18.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:18.450+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.452563+0000) 2019-10-13T08:45:18.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:18.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:18.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:18.446+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.453683+0000) 2019-10-13T08:45:18.455 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:18.450+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.453992+0000) 2019-10-13T08:45:18.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:18.450+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.454690+0000) 2019-10-13T08:45:18.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:18.454+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:18.457514+0000) 2019-10-13T08:45:21.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:21.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:21.618957+0000) 2019-10-13T08:45:21.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:21.618+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:21.620349+0000) 2019-10-13T08:45:23.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:23.450+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.457040+0000) 2019-10-13T08:45:23.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:23.450+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.457583+0000) 2019-10-13T08:45:23.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:23.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.458488+0000) 2019-10-13T08:45:23.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:23.454+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.458209+0000) 2019-10-13T08:45:23.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:23.454+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.459333+0000) 2019-10-13T08:45:23.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:23.454+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.459610+0000) 2019-10-13T08:45:23.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:23.454+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.460002+0000) 2019-10-13T08:45:23.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:23.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:23.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:23.458+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.463964+0000) 2019-10-13T08:45:23.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:23.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:23.933785+0000) 2019-10-13T08:45:24.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:24.338+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:24.345158+0000) 2019-10-13T08:45:25.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:25.714+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:25.716667+0000) 2019-10-13T08:45:27.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:27.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:27.473328+0000) 2019-10-13T08:45:27.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:27.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:27.570751+0000) 2019-10-13T08:45:27.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:27.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:27.597150+0000) 2019-10-13T08:45:28.465 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:28.458+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.464644+0000) 2019-10-13T08:45:28.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:28.458+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.465415+0000) 2019-10-13T08:45:28.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:28.462+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.465389+0000) 2019-10-13T08:45:28.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:28.462+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.466154+0000) 2019-10-13T08:45:28.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:28.462+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.466781+0000) 2019-10-13T08:45:28.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:28.462+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.467476+0000) 2019-10-13T08:45:28.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:28.462+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.467716+0000) 2019-10-13T08:45:28.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:28.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79723 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:28.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:28.470+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:28.473187+0000) 2019-10-13T08:45:31.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:31.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:31.619183+0000) 2019-10-13T08:45:31.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:31.618+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:31.620542+0000) 2019-10-13T08:45:33.471 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:33.466+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.471031+0000) 2019-10-13T08:45:33.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:33.466+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.471151+0000) 2019-10-13T08:45:33.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:33.466+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.471733+0000) 2019-10-13T08:45:33.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:33.466+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.472345+0000) 2019-10-13T08:45:33.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:33.466+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.472675+0000) 2019-10-13T08:45:33.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:33.470+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.473051+0000) 2019-10-13T08:45:33.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:33.470+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.473801+0000) 2019-10-13T08:45:33.474 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:33.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:33.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:33.478+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.480636+0000) 2019-10-13T08:45:33.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:33.929+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:33.933921+0000) 2019-10-13T08:45:34.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:34.341+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:34.345294+0000) 2019-10-13T08:45:35.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:35.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:35.723114+0000) 2019-10-13T08:45:37.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:37.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:37.473528+0000) 2019-10-13T08:45:37.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:37.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:37.570943+0000) 2019-10-13T08:45:37.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:37.589+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:37.597377+0000) 2019-10-13T08:45:38.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:38.473+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.478320+0000) 2019-10-13T08:45:38.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:38.473+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.479002+0000) 2019-10-13T08:45:38.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:38.473+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.479394+0000) 2019-10-13T08:45:38.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:38.473+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.480861+0000) 2019-10-13T08:45:38.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:38.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:38.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:38.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.481964+0000) 2019-10-13T08:45:38.482 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:38.477+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.482149+0000) 2019-10-13T08:45:38.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:38.477+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.485364+0000) 2019-10-13T08:45:38.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:38.481+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:38.487907+0000) 2019-10-13T08:45:41.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:41.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:41.619404+0000) 2019-10-13T08:45:41.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:41.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:41.620770+0000) 2019-10-13T08:45:43.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:43.477+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.484139+0000) 2019-10-13T08:45:43.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:43.477+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.484756+0000) 2019-10-13T08:45:43.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:43.481+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.486136+0000) 2019-10-13T08:45:43.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:43.481+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.486366+0000) 2019-10-13T08:45:43.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:43.481+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.486293+0000) 2019-10-13T08:45:43.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:43.481+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.487766+0000) 2019-10-13T08:45:43.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:43.481+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.488077+0000) 2019-10-13T08:45:43.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:43.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:43.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:43.489+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.492616+0000) 2019-10-13T08:45:43.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:43.933+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:43.934109+0000) 2019-10-13T08:45:44.311 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:45:44.311 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:44.315 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:45:44.315 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:45:44.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:44.337+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:44.345475+0000) 2019-10-13T08:45:45.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:45.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:45.729517+0000) 2019-10-13T08:45:47.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:47.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:47.473768+0000) 2019-10-13T08:45:47.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:47.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:47.571144+0000) 2019-10-13T08:45:47.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:47.593+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:47.597540+0000) 2019-10-13T08:45:48.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:48.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.491156+0000) 2019-10-13T08:45:48.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:48.489+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.492288+0000) 2019-10-13T08:45:48.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:48.489+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.492575+0000) 2019-10-13T08:45:48.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:48.489+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.493687+0000) 2019-10-13T08:45:48.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:48.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:48.496 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:48.489+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.494970+0000) 2019-10-13T08:45:48.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:48.493+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.495734+0000) 2019-10-13T08:45:48.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:48.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.495911+0000) 2019-10-13T08:45:48.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:48.493+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:48.497772+0000) 2019-10-13T08:45:51.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:51.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:51.619620+0000) 2019-10-13T08:45:51.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:51.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:51.620968+0000) 2019-10-13T08:45:53.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:53.497+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.499086+0000) 2019-10-13T08:45:53.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:53.497+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.499379+0000) 2019-10-13T08:45:53.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:53.497+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.500199+0000) 2019-10-13T08:45:53.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:53.497+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.500504+0000) 2019-10-13T08:45:53.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:53.497+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.501309+0000) 2019-10-13T08:45:53.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:53.497+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.501564+0000) 2019-10-13T08:45:53.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:53.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79875 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:53.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:53.501+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.502025+0000) 2019-10-13T08:45:53.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:53.501+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.507497+0000) 2019-10-13T08:45:53.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:53.933+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:53.934290+0000) 2019-10-13T08:45:54.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:54.341+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:54.345659+0000) 2019-10-13T08:45:55.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:55.733+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:55.736064+0000) 2019-10-13T08:45:57.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:57.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:57.473985+0000) 2019-10-13T08:45:57.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:57.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:57.571395+0000) 2019-10-13T08:45:57.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:57.597+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:57.597798+0000) 2019-10-13T08:45:58.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:45:58.501+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.504790+0000) 2019-10-13T08:45:58.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:45:58.505+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.506451+0000) 2019-10-13T08:45:58.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:45:58.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.507545+0000) 2019-10-13T08:45:58.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:45:58.505+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.508589+0000) 2019-10-13T08:45:58.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:45:58.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:45:58.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:45:58.505+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.508625+0000) 2019-10-13T08:45:58.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:45:58.505+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.508472+0000) 2019-10-13T08:45:58.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:45:58.505+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.509427+0000) 2019-10-13T08:45:58.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:45:58.509+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:45:58.513881+0000) 2019-10-13T08:46:01.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:01.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:01.619813+0000) 2019-10-13T08:46:01.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:01.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:01.621157+0000) 2019-10-13T08:46:03.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:03.513+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.513788+0000) 2019-10-13T08:46:03.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:03.513+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.514303+0000) 2019-10-13T08:46:03.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:03.513+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.514586+0000) 2019-10-13T08:46:03.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:03.513+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.515183+0000) 2019-10-13T08:46:03.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:03.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:03.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:03.513+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.516616+0000) 2019-10-13T08:46:03.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:03.513+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.516967+0000) 2019-10-13T08:46:03.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:03.513+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.517812+0000) 2019-10-13T08:46:03.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:03.521+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.524417+0000) 2019-10-13T08:46:03.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:03.933+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:03.934477+0000) 2019-10-13T08:46:04.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:04.345+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:04.345838+0000) 2019-10-13T08:46:05.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:05.741+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:05.742533+0000) 2019-10-13T08:46:07.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:07.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:07.474214+0000) 2019-10-13T08:46:07.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:07.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:07.571623+0000) 2019-10-13T08:46:07.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:07.597+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:07.598013+0000) 2019-10-13T08:46:08.520 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:08.517+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.519975+0000) 2019-10-13T08:46:08.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:08.517+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.520661+0000) 2019-10-13T08:46:08.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:08.521+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.522058+0000) 2019-10-13T08:46:08.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:08.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:08.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:08.517+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.522319+0000) 2019-10-13T08:46:08.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:08.521+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.523550+0000) 2019-10-13T08:46:08.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:08.521+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.523942+0000) 2019-10-13T08:46:08.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:08.521+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.524571+0000) 2019-10-13T08:46:08.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:08.525+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:08.529110+0000) 2019-10-13T08:46:11.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:11.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:11.620040+0000) 2019-10-13T08:46:11.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:11.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:11.621386+0000) 2019-10-13T08:46:13.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:13.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.526546+0000) 2019-10-13T08:46:13.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:13.521+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.526504+0000) 2019-10-13T08:46:13.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:13.525+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.528622+0000) 2019-10-13T08:46:13.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:13.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:13.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:13.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.529810+0000) 2019-10-13T08:46:13.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:13.529+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.530163+0000) 2019-10-13T08:46:13.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:13.529+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.530735+0000) 2019-10-13T08:46:13.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:13.529+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.531279+0000) 2019-10-13T08:46:13.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:13.533+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.536962+0000) 2019-10-13T08:46:13.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:13.933+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:13.934680+0000) 2019-10-13T08:46:14.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:14.345+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:14.345964+0000) 2019-10-13T08:46:14.441 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:46:14.442 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:14.446 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:46:14.446 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:15.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:15.745+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:15.749309+0000) 2019-10-13T08:46:17.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:17.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:17.474441+0000) 2019-10-13T08:46:17.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:17.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:17.571817+0000) 2019-10-13T08:46:17.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:17.597+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:17.598207+0000) 2019-10-13T08:46:18.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:18.529+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.532724+0000) 2019-10-13T08:46:18.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:18.529+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.533011+0000) 2019-10-13T08:46:18.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:18.533+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.534128+0000) 2019-10-13T08:46:18.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:18.529+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.534175+0000) 2019-10-13T08:46:18.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:18.533+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.534939+0000) 2019-10-13T08:46:18.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:18.533+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.535495+0000) 2019-10-13T08:46:18.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:18.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:18.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:18.533+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.536389+0000) 2019-10-13T08:46:18.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:18.537+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:18.540848+0000) 2019-10-13T08:46:21.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:21.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:21.620245+0000) 2019-10-13T08:46:21.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:21.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:21.621588+0000) 2019-10-13T08:46:23.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:23.537+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.538783+0000) 2019-10-13T08:46:23.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:23.537+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.540523+0000) 2019-10-13T08:46:23.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:23.537+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.541068+0000) 2019-10-13T08:46:23.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:23.537+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.540930+0000) 2019-10-13T08:46:23.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:23.541+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.541868+0000) 2019-10-13T08:46:23.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:23.541+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.542238+0000) 2019-10-13T08:46:23.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:23.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:23.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:23.541+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.542880+0000) 2019-10-13T08:46:23.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:23.541+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.546068+0000) 2019-10-13T08:46:23.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:23.933+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:23.934888+0000) 2019-10-13T08:46:24.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:24.345+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:24.346142+0000) 2019-10-13T08:46:25.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:25.753+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:25.756005+0000) 2019-10-13T08:46:27.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:27.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:27.474662+0000) 2019-10-13T08:46:27.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:27.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:27.572022+0000) 2019-10-13T08:46:27.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:27.597+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:27.598347+0000) 2019-10-13T08:46:28.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:28.541+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.545372+0000) 2019-10-13T08:46:28.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:28.545+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.546934+0000) 2019-10-13T08:46:28.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:28.545+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.547624+0000) 2019-10-13T08:46:28.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:28.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.548069+0000) 2019-10-13T08:46:28.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:28.545+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.548903+0000) 2019-10-13T08:46:28.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:28.545+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.548632+0000) 2019-10-13T08:46:28.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:28.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:28.550 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:28.549+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.549855+0000) 2019-10-13T08:46:28.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:28.549+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:28.552627+0000) 2019-10-13T08:46:31.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:31.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:31.620473+0000) 2019-10-13T08:46:31.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:31.617+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:31.621778+0000) 2019-10-13T08:46:33.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:33.548+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.552714+0000) 2019-10-13T08:46:33.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:33.549+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.554148+0000) 2019-10-13T08:46:33.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:33.552+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.555246+0000) 2019-10-13T08:46:33.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:33.552+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.555599+0000) 2019-10-13T08:46:33.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:33.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:33.557 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:33.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.556834+0000) 2019-10-13T08:46:33.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:33.552+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.557077+0000) 2019-10-13T08:46:33.559 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:33.556+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.558685+0000) 2019-10-13T08:46:33.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:33.557+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.561614+0000) 2019-10-13T08:46:33.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:33.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:33.935069+0000) 2019-10-13T08:46:34.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:34.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:34.346271+0000) 2019-10-13T08:46:35.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:35.757+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:35.762492+0000) 2019-10-13T08:46:37.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:37.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:37.474870+0000) 2019-10-13T08:46:37.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:37.568+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:37.572227+0000) 2019-10-13T08:46:37.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:37.596+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:37.598555+0000) 2019-10-13T08:46:38.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:38.556+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.560123+0000) 2019-10-13T08:46:38.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:38.560+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.561990+0000) 2019-10-13T08:46:38.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:38.560+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.562269+0000) 2019-10-13T08:46:38.562 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:38.560+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.562546+0000) 2019-10-13T08:46:38.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:38.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80151 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:38.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:38.557+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.562941+0000) 2019-10-13T08:46:38.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:38.560+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.563843+0000) 2019-10-13T08:46:38.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:38.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.567202+0000) 2019-10-13T08:46:38.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:38.565+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:38.569050+0000) 2019-10-13T08:46:41.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:41.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:41.620694+0000) 2019-10-13T08:46:41.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:41.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:41.621972+0000) 2019-10-13T08:46:43.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:43.564+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.566745+0000) 2019-10-13T08:46:43.567 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:43.564+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.567152+0000) 2019-10-13T08:46:43.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:43.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.568984+0000) 2019-10-13T08:46:43.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:43.564+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.569432+0000) 2019-10-13T08:46:43.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:43.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.569630+0000) 2019-10-13T08:46:43.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:43.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:43.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:43.568+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.570983+0000) 2019-10-13T08:46:43.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:43.568+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.571236+0000) 2019-10-13T08:46:43.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:43.572+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.576783+0000) 2019-10-13T08:46:43.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:43.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:43.935272+0000) 2019-10-13T08:46:44.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:44.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:44.346445+0000) 2019-10-13T08:46:44.773 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:46:44.773 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:44.777 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:46:44.777 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:46:45.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:45.768+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:45.769088+0000) 2019-10-13T08:46:47.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:47.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:47.475099+0000) 2019-10-13T08:46:47.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:47.568+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:47.572424+0000) 2019-10-13T08:46:47.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:47.596+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:47.598756+0000) 2019-10-13T08:46:48.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:48.568+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.573383+0000) 2019-10-13T08:46:48.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:48.572+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.574227+0000) 2019-10-13T08:46:48.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:48.572+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.575288+0000) 2019-10-13T08:46:48.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:48.572+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.575683+0000) 2019-10-13T08:46:48.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:48.572+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.575996+0000) 2019-10-13T08:46:48.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:48.572+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.576582+0000) 2019-10-13T08:46:48.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:48.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80213 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:48.577 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:48.572+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.577173+0000) 2019-10-13T08:46:48.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:48.576+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:48.583700+0000) 2019-10-13T08:46:51.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:51.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:51.620909+0000) 2019-10-13T08:46:51.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:51.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:51.622170+0000) 2019-10-13T08:46:53.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:53.576+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.580958+0000) 2019-10-13T08:46:53.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:53.580+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.581968+0000) 2019-10-13T08:46:53.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:53.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:53.584 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:53.580+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.584349+0000) 2019-10-13T08:46:53.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:53.580+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.584640+0000) 2019-10-13T08:46:53.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:53.584+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.585793+0000) 2019-10-13T08:46:53.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:53.584+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.586167+0000) 2019-10-13T08:46:53.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:53.584+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.586956+0000) 2019-10-13T08:46:53.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:53.588+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.593346+0000) 2019-10-13T08:46:53.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:53.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:53.935465+0000) 2019-10-13T08:46:54.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:54.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:54.346635+0000) 2019-10-13T08:46:55.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:55.772+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:55.775682+0000) 2019-10-13T08:46:57.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:57.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:57.475329+0000) 2019-10-13T08:46:57.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:57.572+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:57.572620+0000) 2019-10-13T08:46:57.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:57.596+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:57.598920+0000) 2019-10-13T08:46:58.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:46:58.584+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.587764+0000) 2019-10-13T08:46:58.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:46:58.584+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.588198+0000) 2019-10-13T08:46:58.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:46:58.584+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.588526+0000) 2019-10-13T08:46:58.589 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:46:58.584+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.588948+0000) 2019-10-13T08:46:58.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:46:58.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80273 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:46:58.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:46:58.588+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.590795+0000) 2019-10-13T08:46:58.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:46:58.588+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.592321+0000) 2019-10-13T08:46:58.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:46:58.588+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.590898+0000) 2019-10-13T08:46:58.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:46:58.592+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:46:58.598218+0000) 2019-10-13T08:47:01.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:01.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:01.621107+0000) 2019-10-13T08:47:01.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:01.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:01.622373+0000) 2019-10-13T08:47:03.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:03.592+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.593980+0000) 2019-10-13T08:47:03.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:03.588+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.594711+0000) 2019-10-13T08:47:03.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:03.592+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.596720+0000) 2019-10-13T08:47:03.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:03.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:03.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:03.592+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.597633+0000) 2019-10-13T08:47:03.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:03.596+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.598052+0000) 2019-10-13T08:47:03.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:03.596+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.598353+0000) 2019-10-13T08:47:03.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:03.596+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.598728+0000) 2019-10-13T08:47:03.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:03.600+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.605224+0000) 2019-10-13T08:47:03.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:03.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:03.935684+0000) 2019-10-13T08:47:04.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:04.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:04.346818+0000) 2019-10-13T08:47:05.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:05.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:05.782323+0000) 2019-10-13T08:47:07.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:07.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:07.475478+0000) 2019-10-13T08:47:07.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:07.568+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:07.572841+0000) 2019-10-13T08:47:07.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:07.596+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:07.599135+0000) 2019-10-13T08:47:08.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:08.596+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.600864+0000) 2019-10-13T08:47:08.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:08.596+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.601277+0000) 2019-10-13T08:47:08.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:08.596+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.601645+0000) 2019-10-13T08:47:08.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:08.600+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.602054+0000) 2019-10-13T08:47:08.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:08.596+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.601903+0000) 2019-10-13T08:47:08.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:08.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:08.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:08.600+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.603798+0000) 2019-10-13T08:47:08.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:08.600+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.603830+0000) 2019-10-13T08:47:08.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:08.604+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:08.609643+0000) 2019-10-13T08:47:11.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:11.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:11.621363+0000) 2019-10-13T08:47:11.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:11.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:11.622574+0000) 2019-10-13T08:47:13.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:13.604+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.607370+0000) 2019-10-13T08:47:13.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:13.604+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.609484+0000) 2019-10-13T08:47:13.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:13.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:13.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:13.608+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.610992+0000) 2019-10-13T08:47:13.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:13.608+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.611434+0000) 2019-10-13T08:47:13.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:13.608+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.611468+0000) 2019-10-13T08:47:13.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:13.604+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.611767+0000) 2019-10-13T08:47:13.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:13.608+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.612648+0000) 2019-10-13T08:47:13.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:13.608+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.615794+0000) 2019-10-13T08:47:13.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:13.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:13.935872+0000) 2019-10-13T08:47:14.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:14.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:14.347011+0000) 2019-10-13T08:47:14.887 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:47:14.887 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:14.891 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:47:14.891 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:15.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:15.784+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:15.788873+0000) 2019-10-13T08:47:17.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:17.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:17.475701+0000) 2019-10-13T08:47:17.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:17.568+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:17.573029+0000) 2019-10-13T08:47:17.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:17.596+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:17.599353+0000) 2019-10-13T08:47:18.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:18.612+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.615179+0000) 2019-10-13T08:47:18.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:18.612+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.616252+0000) 2019-10-13T08:47:18.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:18.612+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.616716+0000) 2019-10-13T08:47:18.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:18.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80395 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:18.618 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:18.616+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.618348+0000) 2019-10-13T08:47:18.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:18.612+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.618226+0000) 2019-10-13T08:47:18.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:18.616+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.618658+0000) 2019-10-13T08:47:18.621 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:18.616+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.619742+0000) 2019-10-13T08:47:18.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:18.620+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:18.625494+0000) 2019-10-13T08:47:21.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:21.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:21.621559+0000) 2019-10-13T08:47:21.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:21.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:21.622773+0000) 2019-10-13T08:47:23.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:23.616+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.620740+0000) 2019-10-13T08:47:23.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:23.620+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.622229+0000) 2019-10-13T08:47:23.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:23.620+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.622545+0000) 2019-10-13T08:47:23.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:23.620+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.623659+0000) 2019-10-13T08:47:23.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:23.620+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.623915+0000) 2019-10-13T08:47:23.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:23.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:23.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:23.620+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.624369+0000) 2019-10-13T08:47:23.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:23.620+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.624722+0000) 2019-10-13T08:47:23.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:23.624+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.631636+0000) 2019-10-13T08:47:23.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:23.932+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:23.936054+0000) 2019-10-13T08:47:24.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:24.344+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:24.347196+0000) 2019-10-13T08:47:25.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:25.792+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:25.795464+0000) 2019-10-13T08:47:27.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:27.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:27.475856+0000) 2019-10-13T08:47:27.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:27.567+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:27.573222+0000) 2019-10-13T08:47:27.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:27.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:27.599505+0000) 2019-10-13T08:47:28.627 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:28.623+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.627054+0000) 2019-10-13T08:47:28.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:28.623+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.628372+0000) 2019-10-13T08:47:28.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:28.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.628575+0000) 2019-10-13T08:47:28.630 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:28.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:28.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:28.627+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.631401+0000) 2019-10-13T08:47:28.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:28.627+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.631079+0000) 2019-10-13T08:47:28.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:28.627+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.632461+0000) 2019-10-13T08:47:28.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:28.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.633166+0000) 2019-10-13T08:47:28.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:28.628+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:28.635106+0000) 2019-10-13T08:47:31.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:31.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:31.621781+0000) 2019-10-13T08:47:31.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:31.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:31.622970+0000) 2019-10-13T08:47:33.635 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:33.631+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.634803+0000) 2019-10-13T08:47:33.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:33.631+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.636470+0000) 2019-10-13T08:47:33.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:33.632+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.636766+0000) 2019-10-13T08:47:33.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:33.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:33.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:33.631+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.637673+0000) 2019-10-13T08:47:33.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:33.635+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.638407+0000) 2019-10-13T08:47:33.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:33.635+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.638555+0000) 2019-10-13T08:47:33.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:33.635+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.639296+0000) 2019-10-13T08:47:33.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:33.640+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.645849+0000) 2019-10-13T08:47:33.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:33.935+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:33.936240+0000) 2019-10-13T08:47:34.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:34.343+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:34.347398+0000) 2019-10-13T08:47:35.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:35.800+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:35.801942+0000) 2019-10-13T08:47:37.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:37.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:37.476067+0000) 2019-10-13T08:47:37.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:37.567+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:37.573420+0000) 2019-10-13T08:47:37.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:37.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:37.599662+0000) 2019-10-13T08:47:38.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:38.636+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.640014+0000) 2019-10-13T08:47:38.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:38.635+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.641404+0000) 2019-10-13T08:47:38.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:38.639+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.642345+0000) 2019-10-13T08:47:38.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:38.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:38.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:38.639+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.644615+0000) 2019-10-13T08:47:38.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:38.639+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.644843+0000) 2019-10-13T08:47:38.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:38.639+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.645235+0000) 2019-10-13T08:47:38.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:38.639+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.645665+0000) 2019-10-13T08:47:38.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:38.652+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:38.652459+0000) 2019-10-13T08:47:41.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:41.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.621932+0000) 2019-10-13T08:47:41.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:41.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:41.623164+0000) 2019-10-13T08:47:43.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:43.643+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.647598+0000) 2019-10-13T08:47:43.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:43.643+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.647794+0000) 2019-10-13T08:47:43.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:43.643+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.648374+0000) 2019-10-13T08:47:43.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:43.643+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.648959+0000) 2019-10-13T08:47:43.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:43.648+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.648890+0000) 2019-10-13T08:47:43.651 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:43.647+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.650757+0000) 2019-10-13T08:47:43.651 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:43.647+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.650571+0000) 2019-10-13T08:47:43.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:43.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80545 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:43.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:43.652+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.655525+0000) 2019-10-13T08:47:43.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:43.931+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:43.936423+0000) 2019-10-13T08:47:44.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:44.343+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:44.347577+0000) 2019-10-13T08:47:45.034 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:47:45.035 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:45.038 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:47:45.039 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:47:45.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:45.807+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:45.808600+0000) 2019-10-13T08:47:47.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:47.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:47.476266+0000) 2019-10-13T08:47:47.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:47.567+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:47.573631+0000) 2019-10-13T08:47:47.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:47.599+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:47.599850+0000) 2019-10-13T08:47:48.655 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:48.651+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.654617+0000) 2019-10-13T08:47:48.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:48.651+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.655126+0000) 2019-10-13T08:47:48.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:48.651+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.654865+0000) 2019-10-13T08:47:48.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:48.651+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.656412+0000) 2019-10-13T08:47:48.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:48.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:48.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:48.655+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.658748+0000) 2019-10-13T08:47:48.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:48.655+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.659321+0000) 2019-10-13T08:47:48.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:48.655+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.660086+0000) 2019-10-13T08:47:48.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:48.663+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:48.664436+0000) 2019-10-13T08:47:51.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:51.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.622080+0000) 2019-10-13T08:47:51.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:51.619+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:51.623355+0000) 2019-10-13T08:47:53.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:53.655+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.661219+0000) 2019-10-13T08:47:53.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:53.655+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.661685+0000) 2019-10-13T08:47:53.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:53.659+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.663307+0000) 2019-10-13T08:47:53.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:53.659+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.663413+0000) 2019-10-13T08:47:53.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:53.659+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.663361+0000) 2019-10-13T08:47:53.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:53.659+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.664317+0000) 2019-10-13T08:47:53.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:53.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:53.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:53.659+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.665263+0000) 2019-10-13T08:47:53.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:53.667+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.670631+0000) 2019-10-13T08:47:53.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:53.931+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:53.936610+0000) 2019-10-13T08:47:54.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:54.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:54.347756+0000) 2019-10-13T08:47:55.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:55.811+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:55.815238+0000) 2019-10-13T08:47:57.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:57.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:57.476466+0000) 2019-10-13T08:47:57.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:57.571+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:57.573825+0000) 2019-10-13T08:47:57.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:57.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:57.600011+0000) 2019-10-13T08:47:58.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:47:58.663+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.668597+0000) 2019-10-13T08:47:58.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:47:58.667+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.670520+0000) 2019-10-13T08:47:58.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:47:58.667+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.670655+0000) 2019-10-13T08:47:58.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:47:58.667+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.671049+0000) 2019-10-13T08:47:58.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:47:58.667+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.671023+0000) 2019-10-13T08:47:58.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:47:58.667+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:47:58.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:47:58.667+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.671675+0000) 2019-10-13T08:47:58.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:47:58.667+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.671157+0000) 2019-10-13T08:47:58.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:47:58.675+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:47:58.677619+0000) 2019-10-13T08:48:01.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:01.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.622293+0000) 2019-10-13T08:48:01.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:01.619+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:01.623552+0000) 2019-10-13T08:48:03.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:03.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.676139+0000) 2019-10-13T08:48:03.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:03.671+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.676274+0000) 2019-10-13T08:48:03.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:03.671+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.676886+0000) 2019-10-13T08:48:03.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:03.675+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.677499+0000) 2019-10-13T08:48:03.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:03.675+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.678338+0000) 2019-10-13T08:48:03.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:03.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:03.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:03.675+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.679347+0000) 2019-10-13T08:48:03.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:03.675+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.679702+0000) 2019-10-13T08:48:03.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:03.679+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.683032+0000) 2019-10-13T08:48:03.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:03.935+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:03.936799+0000) 2019-10-13T08:48:04.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:04.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:04.347939+0000) 2019-10-13T08:48:05.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:05.819+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:05.821410+0000) 2019-10-13T08:48:07.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:07.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:07.476654+0000) 2019-10-13T08:48:07.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:07.571+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:07.574016+0000) 2019-10-13T08:48:07.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:07.599+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:07.600240+0000) 2019-10-13T08:48:08.685 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:08.679+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.684502+0000) 2019-10-13T08:48:08.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:08.679+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.684848+0000) 2019-10-13T08:48:08.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:08.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:08.686 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:08.679+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.685283+0000) 2019-10-13T08:48:08.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:08.683+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.686148+0000) 2019-10-13T08:48:08.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:08.683+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.686823+0000) 2019-10-13T08:48:08.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:08.683+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.687099+0000) 2019-10-13T08:48:08.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:08.683+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.688323+0000) 2019-10-13T08:48:08.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:08.683+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:08.687909+0000) 2019-10-13T08:48:11.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:11.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.622526+0000) 2019-10-13T08:48:11.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:11.619+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:11.623743+0000) 2019-10-13T08:48:13.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:13.687+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.691238+0000) 2019-10-13T08:48:13.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:13.687+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.691648+0000) 2019-10-13T08:48:13.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:13.687+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.692124+0000) 2019-10-13T08:48:13.693 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:13.687+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.692464+0000) 2019-10-13T08:48:13.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:13.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:13.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:13.687+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.692993+0000) 2019-10-13T08:48:13.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:13.691+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.694599+0000) 2019-10-13T08:48:13.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:13.691+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.694418+0000) 2019-10-13T08:48:13.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:13.695+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.699012+0000) 2019-10-13T08:48:13.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:13.931+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:13.936939+0000) 2019-10-13T08:48:14.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:14.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:14.348119+0000) 2019-10-13T08:48:15.161 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:48:15.162 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:15.165 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:48:15.165 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:15.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:15.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:15.827550+0000) 2019-10-13T08:48:17.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:17.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:17.476920+0000) 2019-10-13T08:48:17.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:17.571+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:17.574237+0000) 2019-10-13T08:48:17.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:17.595+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:17.600434+0000) 2019-10-13T08:48:18.697 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:18.691+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.696498+0000) 2019-10-13T08:48:18.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:18.691+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.697042+0000) 2019-10-13T08:48:18.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:18.695+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.697776+0000) 2019-10-13T08:48:18.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:18.695+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.698356+0000) 2019-10-13T08:48:18.699 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:18.695+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.698704+0000) 2019-10-13T08:48:18.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:18.695+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.699067+0000) 2019-10-13T08:48:18.700 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:18.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:18.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:18.695+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.699385+0000) 2019-10-13T08:48:18.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:18.699+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:18.703961+0000) 2019-10-13T08:48:21.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:21.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.622722+0000) 2019-10-13T08:48:21.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:21.619+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:21.623931+0000) 2019-10-13T08:48:23.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:23.699+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.702804+0000) 2019-10-13T08:48:23.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:23.699+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.703186+0000) 2019-10-13T08:48:23.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:23.699+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.705362+0000) 2019-10-13T08:48:23.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:23.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:23.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:23.703+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.706707+0000) 2019-10-13T08:48:23.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:23.703+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.707286+0000) 2019-10-13T08:48:23.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:23.703+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.708008+0000) 2019-10-13T08:48:23.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:23.703+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.706556+0000) 2019-10-13T08:48:23.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:23.707+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.711004+0000) 2019-10-13T08:48:23.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:23.934+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:23.937119+0000) 2019-10-13T08:48:24.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:24.346+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:24.348296+0000) 2019-10-13T08:48:25.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:25.831+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:25.833911+0000) 2019-10-13T08:48:27.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:27.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:27.477117+0000) 2019-10-13T08:48:27.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:27.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:27.574424+0000) 2019-10-13T08:48:27.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:27.594+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:27.600631+0000) 2019-10-13T08:48:28.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:28.702+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.709718+0000) 2019-10-13T08:48:28.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:28.706+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.710570+0000) 2019-10-13T08:48:28.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:28.706+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.710772+0000) 2019-10-13T08:48:28.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:28.707+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.710505+0000) 2019-10-13T08:48:28.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:28.706+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.711422+0000) 2019-10-13T08:48:28.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:28.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:28.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:28.706+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.713121+0000) 2019-10-13T08:48:28.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:28.710+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.714083+0000) 2019-10-13T08:48:28.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:28.715+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:28.716083+0000) 2019-10-13T08:48:31.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:31.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.622914+0000) 2019-10-13T08:48:31.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:31.623+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:31.624124+0000) 2019-10-13T08:48:33.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:33.710+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.716472+0000) 2019-10-13T08:48:33.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:33.710+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.717311+0000) 2019-10-13T08:48:33.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:33.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:33.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:33.714+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.719234+0000) 2019-10-13T08:48:33.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:33.715+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.719906+0000) 2019-10-13T08:48:33.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:33.714+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.721359+0000) 2019-10-13T08:48:33.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:33.718+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.722407+0000) 2019-10-13T08:48:33.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:33.718+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.721881+0000) 2019-10-13T08:48:33.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:33.723+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.726710+0000) 2019-10-13T08:48:33.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:33.930+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:33.937281+0000) 2019-10-13T08:48:34.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:34.346+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:34.348482+0000) 2019-10-13T08:48:35.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:35.839+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:35.840012+0000) 2019-10-13T08:48:37.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:37.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:37.477321+0000) 2019-10-13T08:48:37.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:37.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:37.574625+0000) 2019-10-13T08:48:37.601 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:37.598+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:37.600855+0000) 2019-10-13T08:48:38.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:38.718+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.722487+0000) 2019-10-13T08:48:38.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:38.718+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.724252+0000) 2019-10-13T08:48:38.726 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:38.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:38.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:38.718+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.725400+0000) 2019-10-13T08:48:38.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:38.719+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.724050+0000) 2019-10-13T08:48:38.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:38.722+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.726595+0000) 2019-10-13T08:48:38.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:38.722+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.726753+0000) 2019-10-13T08:48:38.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:38.722+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.727325+0000) 2019-10-13T08:48:38.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:38.727+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:38.731026+0000) 2019-10-13T08:48:41.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:41.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.623135+0000) 2019-10-13T08:48:41.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:41.623+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:41.624319+0000) 2019-10-13T08:48:43.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:43.722+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.727836+0000) 2019-10-13T08:48:43.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:43.722+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.728866+0000) 2019-10-13T08:48:43.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:43.726+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.730775+0000) 2019-10-13T08:48:43.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:43.726+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.730972+0000) 2019-10-13T08:48:43.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:43.726+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.731462+0000) 2019-10-13T08:48:43.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:43.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:43.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:43.727+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.731903+0000) 2019-10-13T08:48:43.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:43.726+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.732707+0000) 2019-10-13T08:48:43.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:43.739+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.740342+0000) 2019-10-13T08:48:43.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:43.934+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:43.937452+0000) 2019-10-13T08:48:44.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:44.342+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:44.348693+0000) 2019-10-13T08:48:45.210 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:48:45.210 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:45.214 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:48:45.214 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:48:45.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:45.843+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:45.846564+0000) 2019-10-13T08:48:47.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:47.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:47.477564+0000) 2019-10-13T08:48:47.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:47.574+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:47.574855+0000) 2019-10-13T08:48:47.601 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:47.598+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:47.601069+0000) 2019-10-13T08:48:48.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:48.730+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.734827+0000) 2019-10-13T08:48:48.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:48.730+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.735557+0000) 2019-10-13T08:48:48.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:48.730+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.735838+0000) 2019-10-13T08:48:48.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:48.730+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.737719+0000) 2019-10-13T08:48:48.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:48.734+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.738108+0000) 2019-10-13T08:48:48.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:48.734+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.739389+0000) 2019-10-13T08:48:48.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:48.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:48.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:48.735+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.739313+0000) 2019-10-13T08:48:48.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:48.739+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:48.741863+0000) 2019-10-13T08:48:51.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:51.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.623339+0000) 2019-10-13T08:48:51.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:51.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:51.624529+0000) 2019-10-13T08:48:53.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:53.738+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.742819+0000) 2019-10-13T08:48:53.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:53.738+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.743096+0000) 2019-10-13T08:48:53.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:53.738+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.743524+0000) 2019-10-13T08:48:53.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:53.742+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.746177+0000) 2019-10-13T08:48:53.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:53.742+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.746410+0000) 2019-10-13T08:48:53.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:53.742+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:53.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:53.742+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.747484+0000) 2019-10-13T08:48:53.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:53.742+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.747411+0000) 2019-10-13T08:48:53.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:53.750+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.752385+0000) 2019-10-13T08:48:53.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:53.934+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:53.937634+0000) 2019-10-13T08:48:54.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:54.346+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:54.348875+0000) 2019-10-13T08:48:55.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:55.850+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:55.853070+0000) 2019-10-13T08:48:57.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:57.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:57.477799+0000) 2019-10-13T08:48:57.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:57.574+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:57.575083+0000) 2019-10-13T08:48:57.601 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:57.598+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:57.601307+0000) 2019-10-13T08:48:58.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:48:58.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.750053+0000) 2019-10-13T08:48:58.751 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:48:58.746+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.750942+0000) 2019-10-13T08:48:58.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:48:58.746+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.751489+0000) 2019-10-13T08:48:58.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:48:58.746+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.751657+0000) 2019-10-13T08:48:58.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:48:58.746+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.752347+0000) 2019-10-13T08:48:58.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:48:58.750+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.752127+0000) 2019-10-13T08:48:58.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:48:58.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:48:58.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:48:58.750+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.753934+0000) 2019-10-13T08:48:58.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:48:58.758+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:48:58.760185+0000) 2019-10-13T08:49:01.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:01.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.623545+0000) 2019-10-13T08:49:01.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:01.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:01.624730+0000) 2019-10-13T08:49:03.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:03.754+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.756221+0000) 2019-10-13T08:49:03.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:03.750+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.757532+0000) 2019-10-13T08:49:03.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:03.754+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.759088+0000) 2019-10-13T08:49:03.759 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:03.754+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.759674+0000) 2019-10-13T08:49:03.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:03.754+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.759923+0000) 2019-10-13T08:49:03.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:03.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:03.761 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:03.754+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.760801+0000) 2019-10-13T08:49:03.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:03.758+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.762106+0000) 2019-10-13T08:49:03.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:03.762+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.766315+0000) 2019-10-13T08:49:03.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:03.934+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:03.937819+0000) 2019-10-13T08:49:04.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:04.346+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:04.349072+0000) 2019-10-13T08:49:05.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:05.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:05.859682+0000) 2019-10-13T08:49:07.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:07.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:07.478000+0000) 2019-10-13T08:49:07.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:07.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:07.575288+0000) 2019-10-13T08:49:07.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:07.598+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:07.601521+0000) 2019-10-13T08:49:08.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:08.758+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.763997+0000) 2019-10-13T08:49:08.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:08.762+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.764296+0000) 2019-10-13T08:49:08.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:08.758+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.765356+0000) 2019-10-13T08:49:08.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:08.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:08.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:08.762+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.767611+0000) 2019-10-13T08:49:08.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:08.762+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.767957+0000) 2019-10-13T08:49:08.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:08.762+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.768401+0000) 2019-10-13T08:49:08.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:08.762+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.768774+0000) 2019-10-13T08:49:08.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:08.766+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:08.771314+0000) 2019-10-13T08:49:11.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:11.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.623736+0000) 2019-10-13T08:49:11.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:11.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:11.624940+0000) 2019-10-13T08:49:13.771 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:13.766+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.770418+0000) 2019-10-13T08:49:13.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:13.766+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.769982+0000) 2019-10-13T08:49:13.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:13.766+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.771075+0000) 2019-10-13T08:49:13.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:13.766+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.771425+0000) 2019-10-13T08:49:13.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:13.766+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.771669+0000) 2019-10-13T08:49:13.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:13.766+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.772338+0000) 2019-10-13T08:49:13.773 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:13.766+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.772555+0000) 2019-10-13T08:49:13.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:13.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:13.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:13.778+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.780263+0000) 2019-10-13T08:49:13.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:13.934+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:13.937978+0000) 2019-10-13T08:49:14.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:14.342+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:14.349277+0000) 2019-10-13T08:49:15.350 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:49:15.350 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:15.359 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:49:15.359 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:15.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:15.862+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:15.866059+0000) 2019-10-13T08:49:17.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:17.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:17.478193+0000) 2019-10-13T08:49:17.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:17.570+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:17.575486+0000) 2019-10-13T08:49:17.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:17.598+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:17.601718+0000) 2019-10-13T08:49:18.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:18.770+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.777282+0000) 2019-10-13T08:49:18.778 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:18.774+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.777775+0000) 2019-10-13T08:49:18.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:18.774+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.777549+0000) 2019-10-13T08:49:18.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:18.774+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.778710+0000) 2019-10-13T08:49:18.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:18.774+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.779222+0000) 2019-10-13T08:49:18.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:18.774+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.779449+0000) 2019-10-13T08:49:18.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:18.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:18.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:18.774+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.780726+0000) 2019-10-13T08:49:18.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:18.782+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:18.785133+0000) 2019-10-13T08:49:21.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:21.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.623958+0000) 2019-10-13T08:49:21.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:21.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:21.625148+0000) 2019-10-13T08:49:23.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:23.777+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.783881+0000) 2019-10-13T08:49:23.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:23.777+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.784844+0000) 2019-10-13T08:49:23.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:23.777+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.785261+0000) 2019-10-13T08:49:23.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:23.777+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.785592+0000) 2019-10-13T08:49:23.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:23.781+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.786159+0000) 2019-10-13T08:49:23.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:23.782+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.786237+0000) 2019-10-13T08:49:23.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:23.781+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.787033+0000) 2019-10-13T08:49:23.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:23.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:23.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:23.786+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.791347+0000) 2019-10-13T08:49:23.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:23.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:23.938189+0000) 2019-10-13T08:49:24.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:24.345+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:24.349453+0000) 2019-10-13T08:49:25.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:25.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:25.872483+0000) 2019-10-13T08:49:27.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:27.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:27.478402+0000) 2019-10-13T08:49:27.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:27.569+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:27.575682+0000) 2019-10-13T08:49:27.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:27.601+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:27.601952+0000) 2019-10-13T08:49:28.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:28.785+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.790619+0000) 2019-10-13T08:49:28.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:28.785+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.792766+0000) 2019-10-13T08:49:28.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:28.785+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.793049+0000) 2019-10-13T08:49:28.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:28.790+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.793519+0000) 2019-10-13T08:49:28.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:28.789+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.793928+0000) 2019-10-13T08:49:28.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:28.789+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:28.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:28.789+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.794760+0000) 2019-10-13T08:49:28.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:28.789+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.795856+0000) 2019-10-13T08:49:28.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:28.798+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:28.800276+0000) 2019-10-13T08:49:31.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:31.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.624176+0000) 2019-10-13T08:49:31.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:31.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:31.625377+0000) 2019-10-13T08:49:33.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:33.794+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.796877+0000) 2019-10-13T08:49:33.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:33.793+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.798277+0000) 2019-10-13T08:49:33.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:33.793+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.798618+0000) 2019-10-13T08:49:33.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:33.793+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.799499+0000) 2019-10-13T08:49:33.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:33.793+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.800418+0000) 2019-10-13T08:49:33.801 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:33.793+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.801018+0000) 2019-10-13T08:49:33.801 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:33.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:33.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:33.793+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.800732+0000) 2019-10-13T08:49:33.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:33.802+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.805995+0000) 2019-10-13T08:49:33.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:33.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:33.938408+0000) 2019-10-13T08:49:34.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:34.345+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:34.349550+0000) 2019-10-13T08:49:35.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:35.878+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:35.878989+0000) 2019-10-13T08:49:37.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:37.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:37.478614+0000) 2019-10-13T08:49:37.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:37.573+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:37.575889+0000) 2019-10-13T08:49:37.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:37.601+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:37.602178+0000) 2019-10-13T08:49:38.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:38.798+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.804019+0000) 2019-10-13T08:49:38.805 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:38.801+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.805041+0000) 2019-10-13T08:49:38.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:38.805+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.806021+0000) 2019-10-13T08:49:38.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:38.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.806046+0000) 2019-10-13T08:49:38.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:38.805+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.806592+0000) 2019-10-13T08:49:38.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:38.805+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.806985+0000) 2019-10-13T08:49:38.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:38.805+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:38.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:38.805+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.807754+0000) 2019-10-13T08:49:38.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:38.810+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:38.812903+0000) 2019-10-13T08:49:41.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:41.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.624367+0000) 2019-10-13T08:49:41.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:41.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:41.625573+0000) 2019-10-13T08:49:43.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:43.809+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.811609+0000) 2019-10-13T08:49:43.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:43.813+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.813779+0000) 2019-10-13T08:49:43.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:43.813+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.814135+0000) 2019-10-13T08:49:43.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:43.813+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.814674+0000) 2019-10-13T08:49:43.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:43.810+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.814655+0000) 2019-10-13T08:49:43.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:43.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:43.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:43.813+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.815599+0000) 2019-10-13T08:49:43.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:43.813+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.815897+0000) 2019-10-13T08:49:43.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:43.814+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.819529+0000) 2019-10-13T08:49:43.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:43.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:43.938601+0000) 2019-10-13T08:49:44.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:44.349+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:44.349782+0000) 2019-10-13T08:49:45.443 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:49:45.443 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:45.450 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:49:45.450 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:49:45.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:45.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:45.885726+0000) 2019-10-13T08:49:47.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:47.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:47.478846+0000) 2019-10-13T08:49:47.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:47.573+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:47.576115+0000) 2019-10-13T08:49:47.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:47.601+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:47.602376+0000) 2019-10-13T08:49:48.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:48.817+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.817982+0000) 2019-10-13T08:49:48.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:48.817+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.817968+0000) 2019-10-13T08:49:48.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:48.814+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.819861+0000) 2019-10-13T08:49:48.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:48.817+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.820667+0000) 2019-10-13T08:49:48.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:48.817+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.821058+0000) 2019-10-13T08:49:48.822 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:48.821+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.822018+0000) 2019-10-13T08:49:48.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:48.821+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.822203+0000) 2019-10-13T08:49:48.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:48.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:48.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:48.826+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:48.828432+0000) 2019-10-13T08:49:51.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:51.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.624562+0000) 2019-10-13T08:49:51.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:51.622+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:51.625769+0000) 2019-10-13T08:49:53.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:53.825+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.826125+0000) 2019-10-13T08:49:53.828 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:53.825+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.828156+0000) 2019-10-13T08:49:53.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:53.825+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.828860+0000) 2019-10-13T08:49:53.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:53.825+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.829624+0000) 2019-10-13T08:49:53.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:53.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:53.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:53.826+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.829989+0000) 2019-10-13T08:49:53.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:53.829+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.830650+0000) 2019-10-13T08:49:53.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:53.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.830891+0000) 2019-10-13T08:49:53.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:53.830+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.834455+0000) 2019-10-13T08:49:53.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:53.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:53.938824+0000) 2019-10-13T08:49:54.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:54.349+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:54.349962+0000) 2019-10-13T08:49:55.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:55.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:55.894887+0000) 2019-10-13T08:49:57.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:57.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:57.479052+0000) 2019-10-13T08:49:57.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:57.573+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:57.576340+0000) 2019-10-13T08:49:57.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:57.601+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:57.602607+0000) 2019-10-13T08:49:58.834 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:49:58.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.833515+0000) 2019-10-13T08:49:58.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:49:58.829+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.833502+0000) 2019-10-13T08:49:58.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:49:58.829+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.834282+0000) 2019-10-13T08:49:58.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:49:58.833+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.835091+0000) 2019-10-13T08:49:58.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:49:58.833+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.835349+0000) 2019-10-13T08:49:58.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:49:58.833+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.835768+0000) 2019-10-13T08:49:58.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:49:58.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:49:58.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:49:58.833+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.836234+0000) 2019-10-13T08:49:58.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:49:58.833+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:49:58.839541+0000) 2019-10-13T08:50:01.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:01.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.624759+0000) 2019-10-13T08:50:01.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:01.621+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:01.625998+0000) 2019-10-13T08:50:03.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:03.837+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.839510+0000) 2019-10-13T08:50:03.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:03.837+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.839930+0000) 2019-10-13T08:50:03.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:03.837+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.841210+0000) 2019-10-13T08:50:03.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:03.841+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.842378+0000) 2019-10-13T08:50:03.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:03.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:03.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:03.841+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.844082+0000) 2019-10-13T08:50:03.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:03.841+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.844652+0000) 2019-10-13T08:50:03.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:03.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.844920+0000) 2019-10-13T08:50:03.849 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:03.845+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.848857+0000) 2019-10-13T08:50:03.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:03.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:03.939074+0000) 2019-10-13T08:50:04.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:04.349+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:04.350138+0000) 2019-10-13T08:50:05.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:05.897+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:05.901384+0000) 2019-10-13T08:50:07.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:07.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:07.479285+0000) 2019-10-13T08:50:07.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:07.573+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:07.576537+0000) 2019-10-13T08:50:07.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:07.601+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:07.602801+0000) 2019-10-13T08:50:08.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:08.845+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.846159+0000) 2019-10-13T08:50:08.847 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:08.845+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.846402+0000) 2019-10-13T08:50:08.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:08.841+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.846899+0000) 2019-10-13T08:50:08.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:08.845+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.847699+0000) 2019-10-13T08:50:08.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:08.845+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.848103+0000) 2019-10-13T08:50:08.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:08.845+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.848498+0000) 2019-10-13T08:50:08.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:08.845+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.849712+0000) 2019-10-13T08:50:08.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:08.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:08.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:08.853+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:08.856525+0000) 2019-10-13T08:50:11.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:11.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.624960+0000) 2019-10-13T08:50:11.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:11.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:11.626228+0000) 2019-10-13T08:50:13.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:13.853+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.854133+0000) 2019-10-13T08:50:13.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:13.853+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.854435+0000) 2019-10-13T08:50:13.855 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:13.853+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.854806+0000) 2019-10-13T08:50:13.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:13.853+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.855454+0000) 2019-10-13T08:50:13.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:13.849+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.855447+0000) 2019-10-13T08:50:13.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:13.853+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.856345+0000) 2019-10-13T08:50:13.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:13.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:13.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:13.853+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.856807+0000) 2019-10-13T08:50:13.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:13.853+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.858543+0000) 2019-10-13T08:50:13.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:13.937+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:13.939293+0000) 2019-10-13T08:50:14.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:14.349+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:14.350309+0000) 2019-10-13T08:50:15.585 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:50:15.585 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:15.592 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:50:15.592 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:15.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:15.905+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:15.907837+0000) 2019-10-13T08:50:17.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:17.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:17.479503+0000) 2019-10-13T08:50:17.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:17.572+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:17.576767+0000) 2019-10-13T08:50:17.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:17.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:17.603002+0000) 2019-10-13T08:50:18.860 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:18.856+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.860199+0000) 2019-10-13T08:50:18.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:18.860+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.861965+0000) 2019-10-13T08:50:18.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:18.860+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.862190+0000) 2019-10-13T08:50:18.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:18.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:18.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:18.857+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.862583+0000) 2019-10-13T08:50:18.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:18.860+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.863525+0000) 2019-10-13T08:50:18.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:18.860+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.864930+0000) 2019-10-13T08:50:18.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:18.860+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.865449+0000) 2019-10-13T08:50:18.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:18.865+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:18.868434+0000) 2019-10-13T08:50:21.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:21.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.625182+0000) 2019-10-13T08:50:21.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:21.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:21.626447+0000) 2019-10-13T08:50:23.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:23.864+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.865887+0000) 2019-10-13T08:50:23.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:23.864+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.866511+0000) 2019-10-13T08:50:23.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:23.861+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.866464+0000) 2019-10-13T08:50:23.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:23.864+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.867658+0000) 2019-10-13T08:50:23.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:23.864+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.867889+0000) 2019-10-13T08:50:23.868 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:23.864+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.868334+0000) 2019-10-13T08:50:23.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:23.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:23.871 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:23.868+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.870687+0000) 2019-10-13T08:50:23.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:23.869+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.875704+0000) 2019-10-13T08:50:23.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:23.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:23.939457+0000) 2019-10-13T08:50:24.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:24.348+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:24.350496+0000) 2019-10-13T08:50:25.917 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:25.909+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:25.914492+0000) 2019-10-13T08:50:27.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:27.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:27.479654+0000) 2019-10-13T08:50:27.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:27.576+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:27.576966+0000) 2019-10-13T08:50:27.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:27.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:27.603198+0000) 2019-10-13T08:50:28.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:28.868+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.872571+0000) 2019-10-13T08:50:28.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:28.872+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.874661+0000) 2019-10-13T08:50:28.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:28.872+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.875360+0000) 2019-10-13T08:50:28.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:28.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81554 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:28.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:28.873+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.876673+0000) 2019-10-13T08:50:28.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:28.876+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.878338+0000) 2019-10-13T08:50:28.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:28.876+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.878633+0000) 2019-10-13T08:50:28.879 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:28.876+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.879652+0000) 2019-10-13T08:50:28.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:28.877+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:28.881685+0000) 2019-10-13T08:50:31.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:31.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.625378+0000) 2019-10-13T08:50:31.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:31.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:31.626650+0000) 2019-10-13T08:50:33.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:33.876+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.879780+0000) 2019-10-13T08:50:33.880 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:33.876+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.880412+0000) 2019-10-13T08:50:33.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:33.876+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.880905+0000) 2019-10-13T08:50:33.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:33.876+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.881212+0000) 2019-10-13T08:50:33.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:33.877+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.881185+0000) 2019-10-13T08:50:33.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:33.880+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.882250+0000) 2019-10-13T08:50:33.882 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:33.880+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.882656+0000) 2019-10-13T08:50:33.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:33.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81582 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:33.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:33.885+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.889239+0000) 2019-10-13T08:50:33.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:33.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:33.939655+0000) 2019-10-13T08:50:34.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:34.348+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:34.350705+0000) 2019-10-13T08:50:35.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:35.917+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:35.921027+0000) 2019-10-13T08:50:37.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:37.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:37.479854+0000) 2019-10-13T08:50:37.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:37.572+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:37.577160+0000) 2019-10-13T08:50:37.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:37.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:37.603402+0000) 2019-10-13T08:50:38.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:38.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.886912+0000) 2019-10-13T08:50:38.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:38.884+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.887319+0000) 2019-10-13T08:50:38.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:38.884+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.887973+0000) 2019-10-13T08:50:38.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:38.884+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.889011+0000) 2019-10-13T08:50:38.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:38.884+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.888483+0000) 2019-10-13T08:50:38.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:38.884+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.889596+0000) 2019-10-13T08:50:38.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:38.885+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.889841+0000) 2019-10-13T08:50:38.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:38.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:38.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:38.889+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:38.896013+0000) 2019-10-13T08:50:41.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:41.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.625528+0000) 2019-10-13T08:50:41.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:41.621+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:41.626843+0000) 2019-10-13T08:50:43.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:43.892+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.894227+0000) 2019-10-13T08:50:43.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:43.892+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.894620+0000) 2019-10-13T08:50:43.895 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:43.892+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.894823+0000) 2019-10-13T08:50:43.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:43.892+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.896897+0000) 2019-10-13T08:50:43.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:43.893+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.897389+0000) 2019-10-13T08:50:43.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:43.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:43.899 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:43.892+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.897186+0000) 2019-10-13T08:50:43.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:43.896+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.898624+0000) 2019-10-13T08:50:43.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:43.897+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.903468+0000) 2019-10-13T08:50:43.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:43.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:43.939859+0000) 2019-10-13T08:50:44.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:44.348+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:44.350887+0000) 2019-10-13T08:50:45.715 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:50:45.716 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:45.720 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:50:45.720 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:50:45.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:45.925+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:45.927577+0000) 2019-10-13T08:50:47.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:47.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:47.480012+0000) 2019-10-13T08:50:47.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:47.572+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:47.577352+0000) 2019-10-13T08:50:47.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:47.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:47.603600+0000) 2019-10-13T08:50:48.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:48.896+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.900759+0000) 2019-10-13T08:50:48.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:48.900+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.902458+0000) 2019-10-13T08:50:48.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:48.900+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.902959+0000) 2019-10-13T08:50:48.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:48.900+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.904604+0000) 2019-10-13T08:50:48.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:48.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:48.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:48.900+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.904886+0000) 2019-10-13T08:50:48.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:48.901+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.904654+0000) 2019-10-13T08:50:48.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:48.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.905706+0000) 2019-10-13T08:50:48.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:48.901+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:48.907349+0000) 2019-10-13T08:50:51.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:51.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.625729+0000) 2019-10-13T08:50:51.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:51.621+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:51.627037+0000) 2019-10-13T08:50:53.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:53.901+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.907669+0000) 2019-10-13T08:50:53.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:53.904+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.908957+0000) 2019-10-13T08:50:53.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:53.904+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.909304+0000) 2019-10-13T08:50:53.910 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:53.908+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.909953+0000) 2019-10-13T08:50:53.910 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:53.908+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.909736+0000) 2019-10-13T08:50:53.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:53.908+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.910338+0000) 2019-10-13T08:50:53.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:53.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:53.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:53.908+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.910820+0000) 2019-10-13T08:50:53.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:53.909+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.914560+0000) 2019-10-13T08:50:53.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:53.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:53.940043+0000) 2019-10-13T08:50:54.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:54.348+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:54.351059+0000) 2019-10-13T08:50:55.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:55.929+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:55.933436+0000) 2019-10-13T08:50:57.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:57.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:57.480206+0000) 2019-10-13T08:50:57.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:57.576+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:57.577552+0000) 2019-10-13T08:50:57.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:57.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:57.603827+0000) 2019-10-13T08:50:58.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:50:58.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.914807+0000) 2019-10-13T08:50:58.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:50:58.912+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.915020+0000) 2019-10-13T08:50:58.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:50:58.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.916440+0000) 2019-10-13T08:50:58.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:50:58.912+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.917314+0000) 2019-10-13T08:50:58.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:50:58.916+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.918055+0000) 2019-10-13T08:50:58.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:50:58.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:50:58.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:50:58.916+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.919007+0000) 2019-10-13T08:50:58.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:50:58.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.919301+0000) 2019-10-13T08:50:58.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:50:58.921+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:50:58.925214+0000) 2019-10-13T08:51:01.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:01.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.625974+0000) 2019-10-13T08:51:01.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:01.621+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:01.627277+0000) 2019-10-13T08:51:03.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:03.916+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.921715+0000) 2019-10-13T08:51:03.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:03.916+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.921952+0000) 2019-10-13T08:51:03.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:03.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.924461+0000) 2019-10-13T08:51:03.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:03.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.925357+0000) 2019-10-13T08:51:03.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:03.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.925721+0000) 2019-10-13T08:51:03.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:03.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81764 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:03.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:03.924+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.926521+0000) 2019-10-13T08:51:03.927 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:03.924+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.926982+0000) 2019-10-13T08:51:03.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:03.928+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.932979+0000) 2019-10-13T08:51:03.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:03.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:03.940156+0000) 2019-10-13T08:51:04.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:04.348+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:04.351245+0000) 2019-10-13T08:51:05.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:05.936+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:05.940064+0000) 2019-10-13T08:51:07.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:07.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:07.480420+0000) 2019-10-13T08:51:07.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:07.576+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:07.577783+0000) 2019-10-13T08:51:07.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:07.600+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:07.604016+0000) 2019-10-13T08:51:08.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:08.924+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.928872+0000) 2019-10-13T08:51:08.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:08.924+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.929747+0000) 2019-10-13T08:51:08.931 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:08.924+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.929564+0000) 2019-10-13T08:51:08.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:08.928+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.930664+0000) 2019-10-13T08:51:08.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:08.928+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.930735+0000) 2019-10-13T08:51:08.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:08.928+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.931523+0000) 2019-10-13T08:51:08.932 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:08.928+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.932442+0000) 2019-10-13T08:51:08.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:08.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:08.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:08.936+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:08.941432+0000) 2019-10-13T08:51:11.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:11.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.626195+0000) 2019-10-13T08:51:11.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:11.620+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:11.627457+0000) 2019-10-13T08:51:13.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:13.932+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.936055+0000) 2019-10-13T08:51:13.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:13.932+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.937040+0000) 2019-10-13T08:51:13.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:13.932+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.937438+0000) 2019-10-13T08:51:13.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:13.932+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.937669+0000) 2019-10-13T08:51:13.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:13.936+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.938799+0000) 2019-10-13T08:51:13.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:13.932+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.938743+0000) 2019-10-13T08:51:13.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:13.936+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.939595+0000) 2019-10-13T08:51:13.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:13.936+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:13.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:13.936+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.940288+0000) 2019-10-13T08:51:13.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:13.940+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:13.945731+0000) 2019-10-13T08:51:14.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:14.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:14.351425+0000) 2019-10-13T08:51:15.899 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:51:15.899 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:15.903 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:51:15.903 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:15.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:15.940+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:15.946478+0000) 2019-10-13T08:51:17.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:17.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:17.480630+0000) 2019-10-13T08:51:17.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:17.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:17.577982+0000) 2019-10-13T08:51:17.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:17.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:17.604227+0000) 2019-10-13T08:51:18.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:18.936+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.942831+0000) 2019-10-13T08:51:18.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:18.939+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.943666+0000) 2019-10-13T08:51:18.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:18.939+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.943998+0000) 2019-10-13T08:51:18.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:18.939+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.944370+0000) 2019-10-13T08:51:18.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:18.939+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.945263+0000) 2019-10-13T08:51:18.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:18.939+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.945507+0000) 2019-10-13T08:51:18.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:18.943+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.946941+0000) 2019-10-13T08:51:18.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:18.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:18.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:18.944+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:18.951960+0000) 2019-10-13T08:51:21.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:21.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.626379+0000) 2019-10-13T08:51:21.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:21.624+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:21.627675+0000) 2019-10-13T08:51:23.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:23.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.940485+0000) 2019-10-13T08:51:23.950 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:23.947+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.950482+0000) 2019-10-13T08:51:23.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:23.947+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.950908+0000) 2019-10-13T08:51:23.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:23.947+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.951304+0000) 2019-10-13T08:51:23.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:23.944+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.951601+0000) 2019-10-13T08:51:23.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:23.947+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.952309+0000) 2019-10-13T08:51:23.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:23.947+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.953101+0000) 2019-10-13T08:51:23.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:23.951+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.953884+0000) 2019-10-13T08:51:23.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:23.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:23.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:23.952+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:23.957450+0000) 2019-10-13T08:51:24.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:24.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:24.351639+0000) 2019-10-13T08:51:25.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:25.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:25.952880+0000) 2019-10-13T08:51:27.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:27.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:27.480793+0000) 2019-10-13T08:51:27.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:27.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:27.578186+0000) 2019-10-13T08:51:27.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:27.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:27.604447+0000) 2019-10-13T08:51:28.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:28.955+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.957894+0000) 2019-10-13T08:51:28.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:28.955+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.958596+0000) 2019-10-13T08:51:28.959 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:28.955+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.958647+0000) 2019-10-13T08:51:28.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:28.955+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.959217+0000) 2019-10-13T08:51:28.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:28.955+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.960684+0000) 2019-10-13T08:51:28.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:28.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:28.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:28.956+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.961284+0000) 2019-10-13T08:51:28.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:28.959+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.962146+0000) 2019-10-13T08:51:28.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:28.960+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:28.966777+0000) 2019-10-13T08:51:31.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:31.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.626593+0000) 2019-10-13T08:51:31.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:31.624+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:31.627864+0000) 2019-10-13T08:51:33.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:33.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.940685+0000) 2019-10-13T08:51:33.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:33.959+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.964339+0000) 2019-10-13T08:51:33.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:33.959+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.964472+0000) 2019-10-13T08:51:33.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:33.959+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.965059+0000) 2019-10-13T08:51:33.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:33.960+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.965917+0000) 2019-10-13T08:51:33.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:33.963+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.966684+0000) 2019-10-13T08:51:33.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:33.963+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.966974+0000) 2019-10-13T08:51:33.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:33.963+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.967438+0000) 2019-10-13T08:51:33.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:33.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81948 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:33.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:33.964+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:33.971506+0000) 2019-10-13T08:51:34.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:34.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:34.351774+0000) 2019-10-13T08:51:35.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:35.956+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:35.959532+0000) 2019-10-13T08:51:37.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:37.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:37.481009+0000) 2019-10-13T08:51:37.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:37.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:37.578393+0000) 2019-10-13T08:51:37.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:37.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:37.604659+0000) 2019-10-13T08:51:38.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:38.968+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.971747+0000) 2019-10-13T08:51:38.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:38.967+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.972630+0000) 2019-10-13T08:51:38.973 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:38.967+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.973198+0000) 2019-10-13T08:51:38.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:38.967+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.973632+0000) 2019-10-13T08:51:38.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:38.971+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.974319+0000) 2019-10-13T08:51:38.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:38.971+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.974677+0000) 2019-10-13T08:51:38.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:38.971+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.975178+0000) 2019-10-13T08:51:38.976 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:38.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:38.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:38.976+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:38.981274+0000) 2019-10-13T08:51:41.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:41.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.626804+0000) 2019-10-13T08:51:41.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:41.624+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:41.628074+0000) 2019-10-13T08:51:43.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:43.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.940857+0000) 2019-10-13T08:51:43.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:43.975+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.980024+0000) 2019-10-13T08:51:43.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:43.975+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.979930+0000) 2019-10-13T08:51:43.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:43.972+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.979995+0000) 2019-10-13T08:51:43.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:43.975+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.980883+0000) 2019-10-13T08:51:43.981 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:43.975+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.981400+0000) 2019-10-13T08:51:43.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:43.975+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.981684+0000) 2019-10-13T08:51:43.982 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:43.979+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.982080+0000) 2019-10-13T08:51:43.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:43.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:43.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:43.984+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:43.989312+0000) 2019-10-13T08:51:44.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:44.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:44.351910+0000) 2019-10-13T08:51:45.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:45.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:45.965978+0000) 2019-10-13T08:51:46.193 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:51:46.194 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:46.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:51:46.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:51:47.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:47.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:47.481210+0000) 2019-10-13T08:51:47.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:47.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:47.578612+0000) 2019-10-13T08:51:47.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:47.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:47.604857+0000) 2019-10-13T08:51:48.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:48.980+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.986161+0000) 2019-10-13T08:51:48.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:48.983+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.987164+0000) 2019-10-13T08:51:48.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:48.983+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.988063+0000) 2019-10-13T08:51:48.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:48.983+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.988382+0000) 2019-10-13T08:51:48.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:48.983+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.989342+0000) 2019-10-13T08:51:48.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:48.987+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.989839+0000) 2019-10-13T08:51:48.990 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:48.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:48.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:48.987+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.990812+0000) 2019-10-13T08:51:48.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:48.988+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:48.995329+0000) 2019-10-13T08:51:51.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:51.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.627025+0000) 2019-10-13T08:51:51.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:51.624+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:51.628344+0000) 2019-10-13T08:51:53.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:53.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.941088+0000) 2019-10-13T08:51:53.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:53.987+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.992869+0000) 2019-10-13T08:51:53.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:53.987+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.993379+0000) 2019-10-13T08:51:53.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:53.991+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.994785+0000) 2019-10-13T08:51:53.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:53.988+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.995206+0000) 2019-10-13T08:51:53.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:53.991+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.996430+0000) 2019-10-13T08:51:53.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:53.991+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.996614+0000) 2019-10-13T08:51:53.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:53.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:53.997 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:53.991+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:53.997384+0000) 2019-10-13T08:51:54.003 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:53.996+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.003183+0000) 2019-10-13T08:51:54.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:54.351+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:54.352139+0000) 2019-10-13T08:51:55.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:55.972+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:55.975013+0000) 2019-10-13T08:51:57.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:57.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:57.481451+0000) 2019-10-13T08:51:57.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:57.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:57.578813+0000) 2019-10-13T08:51:57.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:57.599+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:57.605048+0000) 2019-10-13T08:51:58.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:51:58.995+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:58.999336+0000) 2019-10-13T08:51:59.001 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:51:58.995+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.001481+0000) 2019-10-13T08:51:59.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:51:58.996+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.001295+0000) 2019-10-13T08:51:59.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:51:58.999+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.002109+0000) 2019-10-13T08:51:59.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:51:58.999+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.002444+0000) 2019-10-13T08:51:59.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:51:58.999+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.003408+0000) 2019-10-13T08:51:59.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:51:58.999+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:51:59.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:51:58.999+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.004981+0000) 2019-10-13T08:51:59.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:51:59.000+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:51:59.007102+0000) 2019-10-13T08:52:01.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:01.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.627235+0000) 2019-10-13T08:52:01.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:01.628+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:01.628546+0000) 2019-10-13T08:52:03.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:03.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:03.941300+0000) 2019-10-13T08:52:04.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:04.004+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.007414+0000) 2019-10-13T08:52:04.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:04.003+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.008715+0000) 2019-10-13T08:52:04.010 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:04.007+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.010657+0000) 2019-10-13T08:52:04.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:04.007+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.011071+0000) 2019-10-13T08:52:04.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:04.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:04.012 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:04.007+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.011883+0000) 2019-10-13T08:52:04.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:04.007+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.012183+0000) 2019-10-13T08:52:04.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:04.007+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.013005+0000) 2019-10-13T08:52:04.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:04.016+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.019475+0000) 2019-10-13T08:52:04.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:04.347+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:04.352320+0000) 2019-10-13T08:52:05.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:05.980+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:05.981551+0000) 2019-10-13T08:52:07.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:07.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:07.481649+0000) 2019-10-13T08:52:07.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:07.575+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:07.579006+0000) 2019-10-13T08:52:07.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:07.599+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:07.605242+0000) 2019-10-13T08:52:09.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:09.011+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.014985+0000) 2019-10-13T08:52:09.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:09.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.015810+0000) 2019-10-13T08:52:09.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:09.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:09.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:09.015+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.018916+0000) 2019-10-13T08:52:09.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:09.016+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.018850+0000) 2019-10-13T08:52:09.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:09.015+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.020084+0000) 2019-10-13T08:52:09.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:09.015+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.020446+0000) 2019-10-13T08:52:09.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:09.020+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.021161+0000) 2019-10-13T08:52:09.022 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:09.015+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:09.021644+0000) 2019-10-13T08:52:11.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:11.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.627443+0000) 2019-10-13T08:52:11.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:11.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:11.628699+0000) 2019-10-13T08:52:13.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:13.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:13.941533+0000) 2019-10-13T08:52:14.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:14.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.022025+0000) 2019-10-13T08:52:14.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:14.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.022831+0000) 2019-10-13T08:52:14.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:14.018+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.023480+0000) 2019-10-13T08:52:14.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:14.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82192 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:14.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:14.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.024964+0000) 2019-10-13T08:52:14.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:14.022+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.025940+0000) 2019-10-13T08:52:14.027 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:14.022+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.027312+0000) 2019-10-13T08:52:14.028 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:14.022+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.026742+0000) 2019-10-13T08:52:14.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:14.027+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.028369+0000) 2019-10-13T08:52:14.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:14.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:14.352496+0000) 2019-10-13T08:52:15.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:15.983+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:15.988029+0000) 2019-10-13T08:52:16.290 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:52:16.291 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:16.294 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:52:16.295 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:17.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:17.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:17.481870+0000) 2019-10-13T08:52:17.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:17.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:17.579202+0000) 2019-10-13T08:52:17.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:17.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:17.605472+0000) 2019-10-13T08:52:19.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:19.022+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.029029+0000) 2019-10-13T08:52:19.029 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:19.022+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.029495+0000) 2019-10-13T08:52:19.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:19.026+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.030776+0000) 2019-10-13T08:52:19.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:19.027+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.030468+0000) 2019-10-13T08:52:19.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:19.026+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.031324+0000) 2019-10-13T08:52:19.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:19.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:19.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:19.026+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.031673+0000) 2019-10-13T08:52:19.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:19.026+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.032266+0000) 2019-10-13T08:52:19.036 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:19.031+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:19.035808+0000) 2019-10-13T08:52:21.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:21.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.627640+0000) 2019-10-13T08:52:21.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:21.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:21.628926+0000) 2019-10-13T08:52:23.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:23.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:23.941723+0000) 2019-10-13T08:52:24.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:24.030+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.035046+0000) 2019-10-13T08:52:24.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:24.030+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.035419+0000) 2019-10-13T08:52:24.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:24.030+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.035963+0000) 2019-10-13T08:52:24.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:24.030+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.036488+0000) 2019-10-13T08:52:24.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:24.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:24.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:24.035+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.038468+0000) 2019-10-13T08:52:24.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:24.034+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.039447+0000) 2019-10-13T08:52:24.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:24.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.040007+0000) 2019-10-13T08:52:24.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:24.039+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.042944+0000) 2019-10-13T08:52:24.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:24.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:24.352686+0000) 2019-10-13T08:52:25.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:25.991+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:25.994623+0000) 2019-10-13T08:52:27.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:27.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:27.482077+0000) 2019-10-13T08:52:27.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:27.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:27.579469+0000) 2019-10-13T08:52:27.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:27.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:27.605672+0000) 2019-10-13T08:52:29.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:29.039+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.041411+0000) 2019-10-13T08:52:29.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:29.038+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.042459+0000) 2019-10-13T08:52:29.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:29.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.043063+0000) 2019-10-13T08:52:29.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:29.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.043551+0000) 2019-10-13T08:52:29.044 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:29.038+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.044244+0000) 2019-10-13T08:52:29.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:29.038+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.044885+0000) 2019-10-13T08:52:29.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:29.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:29.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:29.038+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.045716+0000) 2019-10-13T08:52:29.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:29.047+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:29.049713+0000) 2019-10-13T08:52:31.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:31.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.627897+0000) 2019-10-13T08:52:31.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:31.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:31.629142+0000) 2019-10-13T08:52:33.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:33.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:33.941952+0000) 2019-10-13T08:52:34.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:34.042+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.048604+0000) 2019-10-13T08:52:34.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:34.042+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.049127+0000) 2019-10-13T08:52:34.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:34.047+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.049743+0000) 2019-10-13T08:52:34.050 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:34.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.050618+0000) 2019-10-13T08:52:34.051 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:34.046+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.050968+0000) 2019-10-13T08:52:34.052 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:34.046+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.051631+0000) 2019-10-13T08:52:34.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:34.046+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.052010+0000) 2019-10-13T08:52:34.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:34.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:34.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:34.051+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.055545+0000) 2019-10-13T08:52:34.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:34.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:34.352897+0000) 2019-10-13T08:52:36.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:35.999+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:36.001033+0000) 2019-10-13T08:52:37.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:37.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:37.482273+0000) 2019-10-13T08:52:37.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:37.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:37.579665+0000) 2019-10-13T08:52:37.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:37.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:37.605918+0000) 2019-10-13T08:52:39.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:39.050+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.055550+0000) 2019-10-13T08:52:39.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:39.051+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.055379+0000) 2019-10-13T08:52:39.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:39.050+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.056409+0000) 2019-10-13T08:52:39.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:39.050+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.057116+0000) 2019-10-13T08:52:39.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:39.054+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.058071+0000) 2019-10-13T08:52:39.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:39.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:39.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:39.054+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.059435+0000) 2019-10-13T08:52:39.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:39.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.060005+0000) 2019-10-13T08:52:39.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:39.063+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:39.065710+0000) 2019-10-13T08:52:41.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:41.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.628152+0000) 2019-10-13T08:52:41.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:41.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:41.629335+0000) 2019-10-13T08:52:43.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:43.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:43.942159+0000) 2019-10-13T08:52:44.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:44.058+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.062439+0000) 2019-10-13T08:52:44.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:44.058+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.064632+0000) 2019-10-13T08:52:44.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:44.058+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.065531+0000) 2019-10-13T08:52:44.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:44.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:44.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:44.063+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.065514+0000) 2019-10-13T08:52:44.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:44.062+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.066658+0000) 2019-10-13T08:52:44.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:44.062+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.067761+0000) 2019-10-13T08:52:44.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:44.062+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.067329+0000) 2019-10-13T08:52:44.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:44.067+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.071180+0000) 2019-10-13T08:52:44.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:44.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:44.353079+0000) 2019-10-13T08:52:46.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:46.003+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:46.007432+0000) 2019-10-13T08:52:46.429 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:52:46.429 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:46.433 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:52:46.433 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:52:47.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:47.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:47.482472+0000) 2019-10-13T08:52:47.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:47.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:47.579885+0000) 2019-10-13T08:52:47.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:47.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:47.606115+0000) 2019-10-13T08:52:49.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:49.062+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.068802+0000) 2019-10-13T08:52:49.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:49.066+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.070251+0000) 2019-10-13T08:52:49.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:49.066+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.070623+0000) 2019-10-13T08:52:49.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:49.066+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.071325+0000) 2019-10-13T08:52:49.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:49.066+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.071170+0000) 2019-10-13T08:52:49.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:49.066+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.071725+0000) 2019-10-13T08:52:49.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:49.067+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.072032+0000) 2019-10-13T08:52:49.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:49.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:49.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:49.075+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:49.078103+0000) 2019-10-13T08:52:51.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:51.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.628379+0000) 2019-10-13T08:52:51.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:51.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:51.629532+0000) 2019-10-13T08:52:53.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:53.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:53.942347+0000) 2019-10-13T08:52:54.076 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:54.070+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.076298+0000) 2019-10-13T08:52:54.077 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:54.075+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.076194+0000) 2019-10-13T08:52:54.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:54.070+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.077038+0000) 2019-10-13T08:52:54.078 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:54.070+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.077708+0000) 2019-10-13T08:52:54.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:54.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:54.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:54.074+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.081109+0000) 2019-10-13T08:52:54.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:54.074+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.081513+0000) 2019-10-13T08:52:54.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:54.074+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.081575+0000) 2019-10-13T08:52:54.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:54.083+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.087858+0000) 2019-10-13T08:52:54.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:54.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:54.353275+0000) 2019-10-13T08:52:56.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:56.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:56.018867+0000) 2019-10-13T08:52:57.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:57.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:57.482689+0000) 2019-10-13T08:52:57.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:57.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:57.580077+0000) 2019-10-13T08:52:57.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:57.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:57.606267+0000) 2019-10-13T08:52:59.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:52:59.078+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.084161+0000) 2019-10-13T08:52:59.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:52:59.078+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.084653+0000) 2019-10-13T08:52:59.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:52:59.078+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.085010+0000) 2019-10-13T08:52:59.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:52:59.083+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.084800+0000) 2019-10-13T08:52:59.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:52:59.082+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.086065+0000) 2019-10-13T08:52:59.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:52:59.082+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.086172+0000) 2019-10-13T08:52:59.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:52:59.082+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.087682+0000) 2019-10-13T08:52:59.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:52:59.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:52:59.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:52:59.091+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:52:59.093764+0000) 2019-10-13T08:53:01.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:01.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.628613+0000) 2019-10-13T08:53:01.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:01.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:01.629777+0000) 2019-10-13T08:53:03.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:03.938+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:03.942564+0000) 2019-10-13T08:53:04.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:04.086+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.091023+0000) 2019-10-13T08:53:04.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:04.086+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.091599+0000) 2019-10-13T08:53:04.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:04.086+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.092003+0000) 2019-10-13T08:53:04.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:04.086+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.092498+0000) 2019-10-13T08:53:04.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:04.091+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.092689+0000) 2019-10-13T08:53:04.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:04.090+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.094339+0000) 2019-10-13T08:53:04.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:04.090+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.094750+0000) 2019-10-13T08:53:04.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:04.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:04.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:04.095+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.099167+0000) 2019-10-13T08:53:04.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:04.346+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:04.353474+0000) 2019-10-13T08:53:06.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:06.023+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:06.025389+0000) 2019-10-13T08:53:07.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:07.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:07.482926+0000) 2019-10-13T08:53:07.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:07.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:07.580283+0000) 2019-10-13T08:53:07.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:07.602+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:07.606481+0000) 2019-10-13T08:53:09.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:09.093+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.098684+0000) 2019-10-13T08:53:09.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:09.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.099961+0000) 2019-10-13T08:53:09.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:09.093+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.101031+0000) 2019-10-13T08:53:09.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:09.097+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.101776+0000) 2019-10-13T08:53:09.102 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:09.097+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.101716+0000) 2019-10-13T08:53:09.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:09.099+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.101860+0000) 2019-10-13T08:53:09.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:09.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.102793+0000) 2019-10-13T08:53:09.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:09.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82528 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:09.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:09.103+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:09.105106+0000) 2019-10-13T08:53:11.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:11.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.628810+0000) 2019-10-13T08:53:11.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:11.627+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:11.629992+0000) 2019-10-13T08:53:13.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:13.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:13.942762+0000) 2019-10-13T08:53:14.108 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:14.101+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.107613+0000) 2019-10-13T08:53:14.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:14.101+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.108095+0000) 2019-10-13T08:53:14.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:14.101+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.108991+0000) 2019-10-13T08:53:14.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:14.101+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.109483+0000) 2019-10-13T08:53:14.110 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:14.105+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.109931+0000) 2019-10-13T08:53:14.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:14.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:14.111 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:14.107+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.110144+0000) 2019-10-13T08:53:14.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:14.105+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.112913+0000) 2019-10-13T08:53:14.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:14.111+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.113669+0000) 2019-10-13T08:53:14.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:14.349+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:14.353686+0000) 2019-10-13T08:53:16.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:16.030+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:16.031908+0000) 2019-10-13T08:53:16.534 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:53:16.534 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:16.538 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:53:16.538 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:17.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:17.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:17.483127+0000) 2019-10-13T08:53:17.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:17.573+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:17.580481+0000) 2019-10-13T08:53:17.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:17.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:17.606710+0000) 2019-10-13T08:53:19.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:19.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.114298+0000) 2019-10-13T08:53:19.115 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:19.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.114175+0000) 2019-10-13T08:53:19.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:19.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.115144+0000) 2019-10-13T08:53:19.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:19.109+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.115996+0000) 2019-10-13T08:53:19.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:19.109+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.116221+0000) 2019-10-13T08:53:19.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:19.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.116807+0000) 2019-10-13T08:53:19.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:19.109+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.117619+0000) 2019-10-13T08:53:19.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:19.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:19.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:19.118+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:19.121241+0000) 2019-10-13T08:53:21.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:21.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.629006+0000) 2019-10-13T08:53:21.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:21.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:21.630186+0000) 2019-10-13T08:53:23.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:23.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:23.942911+0000) 2019-10-13T08:53:24.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:24.117+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.120576+0000) 2019-10-13T08:53:24.121 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:24.117+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.121193+0000) 2019-10-13T08:53:24.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:24.117+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.121476+0000) 2019-10-13T08:53:24.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:24.118+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.121147+0000) 2019-10-13T08:53:24.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:24.121+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.122323+0000) 2019-10-13T08:53:24.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:24.121+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.122743+0000) 2019-10-13T08:53:24.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:24.121+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.123275+0000) 2019-10-13T08:53:24.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:24.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:24.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:24.122+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.126501+0000) 2019-10-13T08:53:24.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:24.353+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:24.353867+0000) 2019-10-13T08:53:26.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:26.034+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:26.038575+0000) 2019-10-13T08:53:27.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:27.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:27.483334+0000) 2019-10-13T08:53:27.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:27.577+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:27.580635+0000) 2019-10-13T08:53:27.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:27.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:27.606918+0000) 2019-10-13T08:53:29.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:29.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.127738+0000) 2019-10-13T08:53:29.129 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:29.125+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.128766+0000) 2019-10-13T08:53:29.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:29.125+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.129230+0000) 2019-10-13T08:53:29.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:29.129+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.131073+0000) 2019-10-13T08:53:29.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:29.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:29.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:29.129+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.132522+0000) 2019-10-13T08:53:29.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:29.129+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.133126+0000) 2019-10-13T08:53:29.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:29.129+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.133608+0000) 2019-10-13T08:53:29.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:29.134+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:29.137560+0000) 2019-10-13T08:53:31.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:31.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.629195+0000) 2019-10-13T08:53:31.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:31.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:31.630396+0000) 2019-10-13T08:53:33.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:33.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:33.943083+0000) 2019-10-13T08:53:34.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:34.133+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.136531+0000) 2019-10-13T08:53:34.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:34.133+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.137589+0000) 2019-10-13T08:53:34.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:34.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:34.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:34.134+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.139244+0000) 2019-10-13T08:53:34.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:34.137+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.140798+0000) 2019-10-13T08:53:34.141 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:34.137+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.140853+0000) 2019-10-13T08:53:34.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:34.137+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.141649+0000) 2019-10-13T08:53:34.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:34.141+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.142307+0000) 2019-10-13T08:53:34.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:34.142+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.145158+0000) 2019-10-13T08:53:34.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:34.353+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:34.354090+0000) 2019-10-13T08:53:36.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:36.042+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:36.045256+0000) 2019-10-13T08:53:37.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:37.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:37.483494+0000) 2019-10-13T08:53:37.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:37.577+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:37.580863+0000) 2019-10-13T08:53:37.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:37.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:37.607114+0000) 2019-10-13T08:53:39.142 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:39.141+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.142276+0000) 2019-10-13T08:53:39.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:39.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.142143+0000) 2019-10-13T08:53:39.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:39.141+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.143287+0000) 2019-10-13T08:53:39.143 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:39.141+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.143732+0000) 2019-10-13T08:53:39.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:39.141+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.144047+0000) 2019-10-13T08:53:39.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:39.141+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.144952+0000) 2019-10-13T08:53:39.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:39.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.145244+0000) 2019-10-13T08:53:39.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:39.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:39.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:39.146+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:39.150865+0000) 2019-10-13T08:53:41.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:41.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.629424+0000) 2019-10-13T08:53:41.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:41.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:41.630587+0000) 2019-10-13T08:53:43.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:43.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:43.943231+0000) 2019-10-13T08:53:44.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:44.149+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.150173+0000) 2019-10-13T08:53:44.151 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:44.149+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.150739+0000) 2019-10-13T08:53:44.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:44.149+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.151183+0000) 2019-10-13T08:53:44.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:44.149+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.153461+0000) 2019-10-13T08:53:44.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:44.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:44.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:44.150+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.154428+0000) 2019-10-13T08:53:44.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:44.153+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.155384+0000) 2019-10-13T08:53:44.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:44.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.155850+0000) 2019-10-13T08:53:44.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:44.158+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.161229+0000) 2019-10-13T08:53:44.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:44.353+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:44.354266+0000) 2019-10-13T08:53:46.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:46.046+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:46.051814+0000) 2019-10-13T08:53:46.657 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:53:46.657 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:46.660 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:53:46.661 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:53:47.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:47.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:47.483739+0000) 2019-10-13T08:53:47.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:47.577+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:47.581085+0000) 2019-10-13T08:53:47.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:47.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:47.607338+0000) 2019-10-13T08:53:49.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:49.153+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.157470+0000) 2019-10-13T08:53:49.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:49.157+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.158271+0000) 2019-10-13T08:53:49.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:49.157+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.157950+0000) 2019-10-13T08:53:49.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:49.154+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.158116+0000) 2019-10-13T08:53:49.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:49.157+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.158999+0000) 2019-10-13T08:53:49.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:49.157+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.159480+0000) 2019-10-13T08:53:49.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:49.157+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.160419+0000) 2019-10-13T08:53:49.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:49.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:49.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:49.162+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:49.166943+0000) 2019-10-13T08:53:51.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:51.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.629613+0000) 2019-10-13T08:53:51.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:51.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:51.630779+0000) 2019-10-13T08:53:53.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:53.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:53.943406+0000) 2019-10-13T08:53:54.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:54.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.164692+0000) 2019-10-13T08:53:54.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:54.161+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.165515+0000) 2019-10-13T08:53:54.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:54.165+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.166810+0000) 2019-10-13T08:53:54.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:54.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.166651+0000) 2019-10-13T08:53:54.168 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:54.165+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.167621+0000) 2019-10-13T08:53:54.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:54.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.167916+0000) 2019-10-13T08:53:54.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:54.165+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.168584+0000) 2019-10-13T08:53:54.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:54.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:54.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:54.170+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.172509+0000) 2019-10-13T08:53:54.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:54.353+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:54.354460+0000) 2019-10-13T08:53:56.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:56.058+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:56.061764+0000) 2019-10-13T08:53:57.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:57.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:57.483940+0000) 2019-10-13T08:53:57.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:57.577+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:57.581311+0000) 2019-10-13T08:53:57.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:57.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:57.607566+0000) 2019-10-13T08:53:59.172 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:53:59.169+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.171526+0000) 2019-10-13T08:53:59.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:53:59.169+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.172550+0000) 2019-10-13T08:53:59.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:53:59.169+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.173045+0000) 2019-10-13T08:53:59.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:53:59.169+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.173490+0000) 2019-10-13T08:53:59.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:53:59.173+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.174014+0000) 2019-10-13T08:53:59.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:53:59.173+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.174281+0000) 2019-10-13T08:53:59.175 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:53:59.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:53:59.176 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:53:59.170+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.175248+0000) 2019-10-13T08:53:59.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:53:59.178+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:53:59.180972+0000) 2019-10-13T08:54:01.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:01.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.629783+0000) 2019-10-13T08:54:01.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:01.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:01.630973+0000) 2019-10-13T08:54:03.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:03.941+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:03.943580+0000) 2019-10-13T08:54:04.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:04.177+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.179571+0000) 2019-10-13T08:54:04.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:04.174+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.179650+0000) 2019-10-13T08:54:04.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:04.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.180728+0000) 2019-10-13T08:54:04.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:04.177+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.181521+0000) 2019-10-13T08:54:04.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:04.181+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.182309+0000) 2019-10-13T08:54:04.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:04.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:04.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:04.181+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.182794+0000) 2019-10-13T08:54:04.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:04.181+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.183864+0000) 2019-10-13T08:54:04.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:04.182+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.187703+0000) 2019-10-13T08:54:04.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:04.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:04.354640+0000) 2019-10-13T08:54:06.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:06.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:06.068320+0000) 2019-10-13T08:54:07.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:07.480+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:07.484135+0000) 2019-10-13T08:54:07.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:07.576+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:07.581509+0000) 2019-10-13T08:54:07.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:07.604+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:07.607768+0000) 2019-10-13T08:54:09.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:09.184+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.187728+0000) 2019-10-13T08:54:09.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:09.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.187981+0000) 2019-10-13T08:54:09.189 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:09.184+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.188986+0000) 2019-10-13T08:54:09.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:09.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82899 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:09.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:09.188+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.190297+0000) 2019-10-13T08:54:09.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:09.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.190804+0000) 2019-10-13T08:54:09.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:09.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.191035+0000) 2019-10-13T08:54:09.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:09.194+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.197242+0000) 2019-10-13T08:54:09.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:09.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:09.193431+0000) 2019-10-13T08:54:11.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:11.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.629973+0000) 2019-10-13T08:54:11.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:11.626+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:11.631178+0000) 2019-10-13T08:54:13.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:13.940+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:13.943789+0000) 2019-10-13T08:54:14.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:14.188+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.193560+0000) 2019-10-13T08:54:14.195 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:14.192+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.194263+0000) 2019-10-13T08:54:14.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:14.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.194564+0000) 2019-10-13T08:54:14.195 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:14.192+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.195256+0000) 2019-10-13T08:54:14.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:14.192+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.196199+0000) 2019-10-13T08:54:14.196 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:14.192+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.196457+0000) 2019-10-13T08:54:14.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:14.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.196428+0000) 2019-10-13T08:54:14.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:14.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:14.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:14.194+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.198772+0000) 2019-10-13T08:54:14.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:14.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:14.354819+0000) 2019-10-13T08:54:16.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:16.074+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:16.074750+0000) 2019-10-13T08:54:16.788 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:54:16.788 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:16.792 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:54:16.792 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:17.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:17.480+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:17.484336+0000) 2019-10-13T08:54:17.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:17.580+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:17.581719+0000) 2019-10-13T08:54:17.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:17.604+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:17.607964+0000) 2019-10-13T08:54:19.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:19.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.200780+0000) 2019-10-13T08:54:19.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:19.196+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.200873+0000) 2019-10-13T08:54:19.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:19.196+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.201535+0000) 2019-10-13T08:54:19.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:19.200+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.201298+0000) 2019-10-13T08:54:19.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:19.200+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.202070+0000) 2019-10-13T08:54:19.203 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:19.200+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.202716+0000) 2019-10-13T08:54:19.203 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:19.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:19.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:19.198+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.202988+0000) 2019-10-13T08:54:19.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:19.206+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:19.208757+0000) 2019-10-13T08:54:21.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:21.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.630199+0000) 2019-10-13T08:54:21.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:21.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:21.631432+0000) 2019-10-13T08:54:23.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:23.940+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:23.943985+0000) 2019-10-13T08:54:24.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:24.204+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.207227+0000) 2019-10-13T08:54:24.208 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:24.204+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.207708+0000) 2019-10-13T08:54:24.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:24.204+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.208045+0000) 2019-10-13T08:54:24.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:24.201+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.207901+0000) 2019-10-13T08:54:24.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:24.204+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.208694+0000) 2019-10-13T08:54:24.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:24.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:24.211 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:24.208+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.210951+0000) 2019-10-13T08:54:24.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:24.208+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.211927+0000) 2019-10-13T08:54:24.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:24.213+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.218464+0000) 2019-10-13T08:54:24.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:24.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:24.354995+0000) 2019-10-13T08:54:26.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:26.077+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:26.081157+0000) 2019-10-13T08:54:27.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:27.480+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:27.484584+0000) 2019-10-13T08:54:27.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:27.580+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:27.581997+0000) 2019-10-13T08:54:27.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:27.604+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:27.608155+0000) 2019-10-13T08:54:29.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:29.208+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.213562+0000) 2019-10-13T08:54:29.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:29.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.215703+0000) 2019-10-13T08:54:29.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:29.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.217340+0000) 2019-10-13T08:54:29.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:29.212+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.217718+0000) 2019-10-13T08:54:29.218 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:29.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:29.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:29.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.218886+0000) 2019-10-13T08:54:29.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:29.213+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.219251+0000) 2019-10-13T08:54:29.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:29.216+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.220288+0000) 2019-10-13T08:54:29.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:29.221+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:29.226280+0000) 2019-10-13T08:54:31.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:31.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.630393+0000) 2019-10-13T08:54:31.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:31.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:31.631620+0000) 2019-10-13T08:54:33.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:33.940+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:33.944200+0000) 2019-10-13T08:54:34.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:34.216+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.221011+0000) 2019-10-13T08:54:34.223 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:34.220+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.223293+0000) 2019-10-13T08:54:34.226 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:34.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:34.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:34.221+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.225693+0000) 2019-10-13T08:54:34.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:34.224+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.226490+0000) 2019-10-13T08:54:34.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:34.224+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.226610+0000) 2019-10-13T08:54:34.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:34.224+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.226909+0000) 2019-10-13T08:54:34.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:34.224+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.227303+0000) 2019-10-13T08:54:34.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:34.225+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.230074+0000) 2019-10-13T08:54:34.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:34.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:34.355182+0000) 2019-10-13T08:54:36.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:36.085+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:36.087676+0000) 2019-10-13T08:54:37.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:37.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:37.484799+0000) 2019-10-13T08:54:37.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:37.580+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:37.582192+0000) 2019-10-13T08:54:37.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:37.604+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:37.608365+0000) 2019-10-13T08:54:39.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:39.228+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.230074+0000) 2019-10-13T08:54:39.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:39.228+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.232185+0000) 2019-10-13T08:54:39.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:39.228+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.232372+0000) 2019-10-13T08:54:39.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:39.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:39.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:39.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.233429+0000) 2019-10-13T08:54:39.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:39.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.233194+0000) 2019-10-13T08:54:39.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:39.232+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.234191+0000) 2019-10-13T08:54:39.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:39.232+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.234549+0000) 2019-10-13T08:54:39.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:39.233+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:39.238506+0000) 2019-10-13T08:54:41.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:41.625+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.631816+0000) 2019-10-13T08:54:41.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:41.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:41.630618+0000) 2019-10-13T08:54:43.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:43.944+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:43.944376+0000) 2019-10-13T08:54:44.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:44.232+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.236446+0000) 2019-10-13T08:54:44.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:44.236+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.238629+0000) 2019-10-13T08:54:44.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:44.236+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.239722+0000) 2019-10-13T08:54:44.240 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:44.236+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.240025+0000) 2019-10-13T08:54:44.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:44.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:44.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:44.236+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.240646+0000) 2019-10-13T08:54:44.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:44.236+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.240825+0000) 2019-10-13T08:54:44.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:44.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.240963+0000) 2019-10-13T08:54:44.247 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:44.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.246774+0000) 2019-10-13T08:54:44.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:44.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:44.355405+0000) 2019-10-13T08:54:46.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:46.093+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:46.094050+0000) 2019-10-13T08:54:46.869 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:54:46.869 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:46.873 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:54:46.873 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:54:47.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:47.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:47.485002+0000) 2019-10-13T08:54:47.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:47.580+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:47.582394+0000) 2019-10-13T08:54:47.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:47.608+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:47.608565+0000) 2019-10-13T08:54:49.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:49.237+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.243366+0000) 2019-10-13T08:54:49.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:49.240+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.244267+0000) 2019-10-13T08:54:49.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:49.244+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.245917+0000) 2019-10-13T08:54:49.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:49.244+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.246400+0000) 2019-10-13T08:54:49.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:49.244+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.246645+0000) 2019-10-13T08:54:49.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:49.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:49.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:49.244+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.247468+0000) 2019-10-13T08:54:49.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:49.244+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.247918+0000) 2019-10-13T08:54:49.253 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:49.249+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:49.253161+0000) 2019-10-13T08:54:51.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:51.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.630812+0000) 2019-10-13T08:54:51.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:51.629+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:51.632009+0000) 2019-10-13T08:54:53.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:53.940+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:53.944595+0000) 2019-10-13T08:54:54.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:54.248+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.250613+0000) 2019-10-13T08:54:54.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:54.248+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.251043+0000) 2019-10-13T08:54:54.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:54.245+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.251385+0000) 2019-10-13T08:54:54.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:54.248+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.251320+0000) 2019-10-13T08:54:54.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:54.248+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.252145+0000) 2019-10-13T08:54:54.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:54.248+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.252640+0000) 2019-10-13T08:54:54.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:54.248+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.252864+0000) 2019-10-13T08:54:54.254 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:54.252+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:54.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:54.253+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.258151+0000) 2019-10-13T08:54:54.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:54.352+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:54.355587+0000) 2019-10-13T08:54:56.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:56.097+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:56.100376+0000) 2019-10-13T08:54:57.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:57.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:57.485202+0000) 2019-10-13T08:54:57.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:57.580+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:57.582614+0000) 2019-10-13T08:54:57.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:57.604+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:57.608753+0000) 2019-10-13T08:54:59.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:54:59.256+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.258058+0000) 2019-10-13T08:54:59.259 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:54:59.256+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.258206+0000) 2019-10-13T08:54:59.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:54:59.256+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.259396+0000) 2019-10-13T08:54:59.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:54:59.256+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.260650+0000) 2019-10-13T08:54:59.261 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:54:59.257+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.260506+0000) 2019-10-13T08:54:59.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:54:59.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:54:59.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:54:59.260+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.262193+0000) 2019-10-13T08:54:59.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:54:59.260+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.262357+0000) 2019-10-13T08:54:59.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:54:59.265+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:54:59.268663+0000) 2019-10-13T08:55:01.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:01.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.631010+0000) 2019-10-13T08:55:01.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:01.629+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:01.632213+0000) 2019-10-13T08:55:03.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:03.943+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:03.944816+0000) 2019-10-13T08:55:04.265 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:04.259+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.265056+0000) 2019-10-13T08:55:04.267 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:04.261+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.264850+0000) 2019-10-13T08:55:04.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:04.259+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.265667+0000) 2019-10-13T08:55:04.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:04.263+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.266100+0000) 2019-10-13T08:55:04.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:04.263+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.266652+0000) 2019-10-13T08:55:04.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:04.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:04.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:04.263+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.268915+0000) 2019-10-13T08:55:04.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:04.267+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.270821+0000) 2019-10-13T08:55:04.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:04.269+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.273014+0000) 2019-10-13T08:55:04.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:04.351+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:04.355771+0000) 2019-10-13T08:55:06.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:06.101+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:06.106870+0000) 2019-10-13T08:55:07.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:07.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:07.485397+0000) 2019-10-13T08:55:07.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:07.579+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:07.582811+0000) 2019-10-13T08:55:07.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:07.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:07.608951+0000) 2019-10-13T08:55:09.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:09.267+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.271798+0000) 2019-10-13T08:55:09.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:09.267+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.272137+0000) 2019-10-13T08:55:09.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:09.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.273078+0000) 2019-10-13T08:55:09.274 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:09.267+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.273497+0000) 2019-10-13T08:55:09.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:09.271+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.273865+0000) 2019-10-13T08:55:09.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:09.271+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.274475+0000) 2019-10-13T08:55:09.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:09.269+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.274916+0000) 2019-10-13T08:55:09.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:09.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:09.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:09.277+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:09.280643+0000) 2019-10-13T08:55:11.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:11.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.631253+0000) 2019-10-13T08:55:11.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:11.629+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:11.632407+0000) 2019-10-13T08:55:13.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:13.939+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:13.945092+0000) 2019-10-13T08:55:14.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:14.275+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.279457+0000) 2019-10-13T08:55:14.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:14.275+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.280118+0000) 2019-10-13T08:55:14.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:14.279+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.281935+0000) 2019-10-13T08:55:14.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:14.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:14.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:14.279+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.283064+0000) 2019-10-13T08:55:14.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:14.279+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.283380+0000) 2019-10-13T08:55:14.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:14.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.284278+0000) 2019-10-13T08:55:14.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:14.281+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.286829+0000) 2019-10-13T08:55:14.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:14.285+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.288726+0000) 2019-10-13T08:55:14.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:14.355+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:14.355951+0000) 2019-10-13T08:55:16.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:16.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:16.113310+0000) 2019-10-13T08:55:16.992 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:55:16.992 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:16.999 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:55:17.000 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:17.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:17.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:17.485631+0000) 2019-10-13T08:55:17.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:17.579+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:17.583022+0000) 2019-10-13T08:55:17.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:17.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:17.609181+0000) 2019-10-13T08:55:19.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:19.281+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.286052+0000) 2019-10-13T08:55:19.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:19.283+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.287060+0000) 2019-10-13T08:55:19.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:19.283+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.287585+0000) 2019-10-13T08:55:19.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:19.283+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.287949+0000) 2019-10-13T08:55:19.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:19.283+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.288222+0000) 2019-10-13T08:55:19.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:19.283+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.288558+0000) 2019-10-13T08:55:19.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:19.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:19.291 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:19.283+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.289605+0000) 2019-10-13T08:55:19.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:19.285+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:19.291619+0000) 2019-10-13T08:55:21.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:21.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.631437+0000) 2019-10-13T08:55:21.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:21.629+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:21.632660+0000) 2019-10-13T08:55:23.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:23.943+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:23.945335+0000) 2019-10-13T08:55:24.293 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:24.287+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.292624+0000) 2019-10-13T08:55:24.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:24.287+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.292926+0000) 2019-10-13T08:55:24.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:24.287+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.293392+0000) 2019-10-13T08:55:24.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:24.289+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.293448+0000) 2019-10-13T08:55:24.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:24.291+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.294179+0000) 2019-10-13T08:55:24.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:24.291+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.295112+0000) 2019-10-13T08:55:24.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:24.291+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.295572+0000) 2019-10-13T08:55:24.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:24.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:24.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:24.297+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.300299+0000) 2019-10-13T08:55:24.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:24.355+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:24.356129+0000) 2019-10-13T08:55:26.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:26.117+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:26.119740+0000) 2019-10-13T08:55:27.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:27.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:27.485828+0000) 2019-10-13T08:55:27.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:27.579+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:27.583212+0000) 2019-10-13T08:55:27.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:27.607+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:27.609406+0000) 2019-10-13T08:55:29.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:29.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.299309+0000) 2019-10-13T08:55:29.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:29.295+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.300055+0000) 2019-10-13T08:55:29.300 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:29.295+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.300624+0000) 2019-10-13T08:55:29.301 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:29.295+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.301025+0000) 2019-10-13T08:55:29.302 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:29.296+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.301626+0000) 2019-10-13T08:55:29.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:29.299+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.302404+0000) 2019-10-13T08:55:29.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:29.299+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.302672+0000) 2019-10-13T08:55:29.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:29.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:29.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:29.300+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:29.305234+0000) 2019-10-13T08:55:31.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:31.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.631628+0000) 2019-10-13T08:55:31.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:31.628+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:31.632862+0000) 2019-10-13T08:55:33.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:33.943+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:33.945529+0000) 2019-10-13T08:55:34.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:34.303+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.306084+0000) 2019-10-13T08:55:34.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:34.303+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.307096+0000) 2019-10-13T08:55:34.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:34.300+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.306764+0000) 2019-10-13T08:55:34.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:34.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.309250+0000) 2019-10-13T08:55:34.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:34.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:34.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:34.307+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.310433+0000) 2019-10-13T08:55:34.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:34.307+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.311076+0000) 2019-10-13T08:55:34.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:34.307+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.312007+0000) 2019-10-13T08:55:34.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:34.312+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.317843+0000) 2019-10-13T08:55:34.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:34.351+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:34.356311+0000) 2019-10-13T08:55:36.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:36.124+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:36.126230+0000) 2019-10-13T08:55:37.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:37.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:37.486023+0000) 2019-10-13T08:55:37.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:37.579+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:37.583452+0000) 2019-10-13T08:55:37.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:37.603+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:37.609640+0000) 2019-10-13T08:55:39.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:39.307+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.313333+0000) 2019-10-13T08:55:39.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:39.311+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.314302+0000) 2019-10-13T08:55:39.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:39.311+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.314675+0000) 2019-10-13T08:55:39.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:39.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:39.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:39.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.317092+0000) 2019-10-13T08:55:39.317 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:39.312+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.317033+0000) 2019-10-13T08:55:39.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:39.315+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.318995+0000) 2019-10-13T08:55:39.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:39.315+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.320047+0000) 2019-10-13T08:55:39.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:39.320+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:39.326635+0000) 2019-10-13T08:55:41.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:41.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.631867+0000) 2019-10-13T08:55:41.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:41.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:41.633092+0000) 2019-10-13T08:55:43.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:43.943+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:43.945787+0000) 2019-10-13T08:55:44.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:44.315+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.320320+0000) 2019-10-13T08:55:44.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:44.315+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.320771+0000) 2019-10-13T08:55:44.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:44.315+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.321240+0000) 2019-10-13T08:55:44.322 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:44.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.321919+0000) 2019-10-13T08:55:44.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:44.316+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.322440+0000) 2019-10-13T08:55:44.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:44.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.323490+0000) 2019-10-13T08:55:44.324 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:44.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:44.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:44.319+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.324034+0000) 2019-10-13T08:55:44.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:44.320+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.326055+0000) 2019-10-13T08:55:44.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:44.351+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:44.356504+0000) 2019-10-13T08:55:46.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:46.128+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:46.132685+0000) 2019-10-13T08:55:47.108 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:55:47.108 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:47.112 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:55:47.112 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:55:47.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:47.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:47.486216+0000) 2019-10-13T08:55:47.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:47.583+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:47.583654+0000) 2019-10-13T08:55:47.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:47.607+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:47.609849+0000) 2019-10-13T08:55:49.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:49.320+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.326073+0000) 2019-10-13T08:55:49.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:49.323+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.326987+0000) 2019-10-13T08:55:49.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:49.323+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.327670+0000) 2019-10-13T08:55:49.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:49.323+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.328113+0000) 2019-10-13T08:55:49.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:49.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.328528+0000) 2019-10-13T08:55:49.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:49.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.328921+0000) 2019-10-13T08:55:49.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:49.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:49.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:49.327+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.331766+0000) 2019-10-13T08:55:49.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:49.332+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:49.336572+0000) 2019-10-13T08:55:51.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:51.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.632077+0000) 2019-10-13T08:55:51.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:51.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:51.633323+0000) 2019-10-13T08:55:53.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:53.943+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:53.946008+0000) 2019-10-13T08:55:54.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:54.331+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.333545+0000) 2019-10-13T08:55:54.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:54.331+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.334761+0000) 2019-10-13T08:55:54.335 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:54.331+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.334839+0000) 2019-10-13T08:55:54.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:54.328+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.334859+0000) 2019-10-13T08:55:54.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:54.331+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.335649+0000) 2019-10-13T08:55:54.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:54.331+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.335873+0000) 2019-10-13T08:55:54.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:54.331+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.336839+0000) 2019-10-13T08:55:54.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:54.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:54.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:54.336+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.342390+0000) 2019-10-13T08:55:54.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:54.351+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:54.356682+0000) 2019-10-13T08:55:56.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:56.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:56.139176+0000) 2019-10-13T08:55:57.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:57.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:57.486445+0000) 2019-10-13T08:55:57.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:57.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:57.583852+0000) 2019-10-13T08:55:57.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:57.606+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:57.610050+0000) 2019-10-13T08:55:59.341 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:55:59.334+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.341471+0000) 2019-10-13T08:55:59.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:55:59.336+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.341209+0000) 2019-10-13T08:55:59.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:55:59.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.342377+0000) 2019-10-13T08:55:59.344 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:55:59.338+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.343833+0000) 2019-10-13T08:55:59.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:55:59.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:55:59.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:55:59.342+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.346489+0000) 2019-10-13T08:55:59.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:55:59.342+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.346928+0000) 2019-10-13T08:55:59.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:55:59.342+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.346003+0000) 2019-10-13T08:55:59.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:55:59.348+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:55:59.352599+0000) 2019-10-13T08:56:01.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:01.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.632269+0000) 2019-10-13T08:56:01.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:01.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:01.633555+0000) 2019-10-13T08:56:03.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:03.942+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:03.946270+0000) 2019-10-13T08:56:04.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:04.342+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.349017+0000) 2019-10-13T08:56:04.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:04.342+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.349412+0000) 2019-10-13T08:56:04.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:04.344+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.351020+0000) 2019-10-13T08:56:04.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:04.346+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.351701+0000) 2019-10-13T08:56:04.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:04.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:04.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:04.346+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.353230+0000) 2019-10-13T08:56:04.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:04.350+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.353730+0000) 2019-10-13T08:56:04.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:04.350+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.355544+0000) 2019-10-13T08:56:04.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:04.350+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.356808+0000) 2019-10-13T08:56:04.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:04.352+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:04.358774+0000) 2019-10-13T08:56:06.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:06.144+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:06.145707+0000) 2019-10-13T08:56:07.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:07.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:07.486653+0000) 2019-10-13T08:56:07.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:07.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:07.584057+0000) 2019-10-13T08:56:07.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:07.606+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:07.610246+0000) 2019-10-13T08:56:09.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:09.350+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.356158+0000) 2019-10-13T08:56:09.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:09.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.358360+0000) 2019-10-13T08:56:09.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:09.354+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.358823+0000) 2019-10-13T08:56:09.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:09.354+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.359261+0000) 2019-10-13T08:56:09.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:09.352+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.359025+0000) 2019-10-13T08:56:09.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:09.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:09.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:09.354+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.360077+0000) 2019-10-13T08:56:09.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:09.354+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.359841+0000) 2019-10-13T08:56:09.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:09.360+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:09.367675+0000) 2019-10-13T08:56:11.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:11.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.632468+0000) 2019-10-13T08:56:11.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:11.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:11.633709+0000) 2019-10-13T08:56:13.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:13.942+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:13.946485+0000) 2019-10-13T08:56:14.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:14.354+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.357342+0000) 2019-10-13T08:56:14.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:14.358+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.363015+0000) 2019-10-13T08:56:14.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:14.358+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.365407+0000) 2019-10-13T08:56:14.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:14.362+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.365814+0000) 2019-10-13T08:56:14.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:14.362+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.366195+0000) 2019-10-13T08:56:14.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:14.362+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.366656+0000) 2019-10-13T08:56:14.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:14.360+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.367650+0000) 2019-10-13T08:56:14.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:14.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:14.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:14.362+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.368989+0000) 2019-10-13T08:56:14.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:14.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:14.375009+0000) 2019-10-13T08:56:16.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:16.148+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:16.152242+0000) 2019-10-13T08:56:17.225 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:56:17.225 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:17.229 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:56:17.229 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:17.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:17.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:17.486836+0000) 2019-10-13T08:56:17.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:17.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:17.584255+0000) 2019-10-13T08:56:17.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:17.606+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:17.610463+0000) 2019-10-13T08:56:19.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:19.366+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.371643+0000) 2019-10-13T08:56:19.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:19.366+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.373589+0000) 2019-10-13T08:56:19.374 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:19.370+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.373937+0000) 2019-10-13T08:56:19.374 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:19.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:19.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:19.370+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.375219+0000) 2019-10-13T08:56:19.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:19.370+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.375932+0000) 2019-10-13T08:56:19.376 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:19.370+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.376293+0000) 2019-10-13T08:56:19.377 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:19.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.376586+0000) 2019-10-13T08:56:19.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:19.372+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:19.377033+0000) 2019-10-13T08:56:21.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:21.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.632694+0000) 2019-10-13T08:56:21.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:21.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:21.633920+0000) 2019-10-13T08:56:23.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:23.942+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:23.946707+0000) 2019-10-13T08:56:24.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:24.354+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.357545+0000) 2019-10-13T08:56:24.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:24.374+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.377157+0000) 2019-10-13T08:56:24.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:24.374+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.378609+0000) 2019-10-13T08:56:24.380 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:24.374+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.379870+0000) 2019-10-13T08:56:24.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:24.374+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.380029+0000) 2019-10-13T08:56:24.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:24.374+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.380584+0000) 2019-10-13T08:56:24.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:24.374+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.381178+0000) 2019-10-13T08:56:24.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:24.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:24.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:24.376+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.381381+0000) 2019-10-13T08:56:24.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:24.384+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:24.388803+0000) 2019-10-13T08:56:26.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:26.152+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:26.160009+0000) 2019-10-13T08:56:27.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:27.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:27.487024+0000) 2019-10-13T08:56:27.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:27.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:27.584457+0000) 2019-10-13T08:56:27.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:27.606+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:27.610686+0000) 2019-10-13T08:56:29.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:29.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.384706+0000) 2019-10-13T08:56:29.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:29.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.385306+0000) 2019-10-13T08:56:29.386 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:29.384+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.385490+0000) 2019-10-13T08:56:29.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:29.382+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.386916+0000) 2019-10-13T08:56:29.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:29.382+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.387806+0000) 2019-10-13T08:56:29.388 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:29.382+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.388595+0000) 2019-10-13T08:56:29.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:29.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:29.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:29.382+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.389366+0000) 2019-10-13T08:56:29.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:29.388+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:29.391575+0000) 2019-10-13T08:56:31.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:31.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.632888+0000) 2019-10-13T08:56:31.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:31.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:31.634111+0000) 2019-10-13T08:56:33.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:33.946+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:33.946907+0000) 2019-10-13T08:56:34.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:34.354+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.357794+0000) 2019-10-13T08:56:34.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:34.386+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.391907+0000) 2019-10-13T08:56:34.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:34.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.393716+0000) 2019-10-13T08:56:34.394 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:34.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.394017+0000) 2019-10-13T08:56:34.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:34.390+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.395142+0000) 2019-10-13T08:56:34.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:34.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:34.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:34.391+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.395089+0000) 2019-10-13T08:56:34.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:34.390+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.396081+0000) 2019-10-13T08:56:34.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:34.390+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.397291+0000) 2019-10-13T08:56:34.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:34.399+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:34.403005+0000) 2019-10-13T08:56:36.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:36.163+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:36.166381+0000) 2019-10-13T08:56:37.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:37.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:37.487199+0000) 2019-10-13T08:56:37.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:37.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:37.584659+0000) 2019-10-13T08:56:37.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:37.606+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:37.610885+0000) 2019-10-13T08:56:39.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:39.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.399031+0000) 2019-10-13T08:56:39.400 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:39.394+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.399435+0000) 2019-10-13T08:56:39.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:39.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.400210+0000) 2019-10-13T08:56:39.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:39.394+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.400232+0000) 2019-10-13T08:56:39.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:39.394+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.401457+0000) 2019-10-13T08:56:39.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:39.398+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.401877+0000) 2019-10-13T08:56:39.402 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:39.399+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.401689+0000) 2019-10-13T08:56:39.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:39.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:39.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:39.407+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:39.408753+0000) 2019-10-13T08:56:41.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:41.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.633130+0000) 2019-10-13T08:56:41.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:41.631+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:41.634308+0000) 2019-10-13T08:56:43.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:43.946+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:43.947144+0000) 2019-10-13T08:56:44.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:44.354+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.358000+0000) 2019-10-13T08:56:44.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:44.398+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.405598+0000) 2019-10-13T08:56:44.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:44.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.405955+0000) 2019-10-13T08:56:44.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:44.403+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.406337+0000) 2019-10-13T08:56:44.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:44.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.407236+0000) 2019-10-13T08:56:44.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:44.402+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.407462+0000) 2019-10-13T08:56:44.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:44.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.408204+0000) 2019-10-13T08:56:44.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:44.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:44.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:44.402+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.408755+0000) 2019-10-13T08:56:44.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:44.411+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:44.415302+0000) 2019-10-13T08:56:46.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:46.171+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:46.172840+0000) 2019-10-13T08:56:47.467 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:56:47.468 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:47.471 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:56:47.471 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:56:47.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:47.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:47.487368+0000) 2019-10-13T08:56:47.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:47.578+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:47.584863+0000) 2019-10-13T08:56:47.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:47.610+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:47.611112+0000) 2019-10-13T08:56:49.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:49.406+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.412472+0000) 2019-10-13T08:56:49.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:49.410+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.414612+0000) 2019-10-13T08:56:49.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:49.410+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.414355+0000) 2019-10-13T08:56:49.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:49.410+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.415055+0000) 2019-10-13T08:56:49.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:49.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:49.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:49.410+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.416069+0000) 2019-10-13T08:56:49.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:49.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.416290+0000) 2019-10-13T08:56:49.417 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:49.410+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.416411+0000) 2019-10-13T08:56:49.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:49.419+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:49.422655+0000) 2019-10-13T08:56:51.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:51.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.633322+0000) 2019-10-13T08:56:51.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:51.631+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:51.634508+0000) 2019-10-13T08:56:53.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:53.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:53.947372+0000) 2019-10-13T08:56:54.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:54.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.358201+0000) 2019-10-13T08:56:54.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:54.413+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.418571+0000) 2019-10-13T08:56:54.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:54.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.418845+0000) 2019-10-13T08:56:54.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:54.413+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.419478+0000) 2019-10-13T08:56:54.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:54.413+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.420035+0000) 2019-10-13T08:56:54.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:54.413+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.420609+0000) 2019-10-13T08:56:54.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:54.413+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.420859+0000) 2019-10-13T08:56:54.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:54.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:54.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:54.419+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.421873+0000) 2019-10-13T08:56:54.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:54.427+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:54.428520+0000) 2019-10-13T08:56:56.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:56.175+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:56.179221+0000) 2019-10-13T08:56:57.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:57.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:57.487543+0000) 2019-10-13T08:56:57.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:57.577+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:57.585060+0000) 2019-10-13T08:56:57.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:57.605+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:57.611310+0000) 2019-10-13T08:56:59.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:56:59.421+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.425150+0000) 2019-10-13T08:56:59.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:56:59.421+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.425698+0000) 2019-10-13T08:56:59.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:56:59.421+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.427043+0000) 2019-10-13T08:56:59.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:56:59.423+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.426936+0000) 2019-10-13T08:56:59.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:56:59.421+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.427329+0000) 2019-10-13T08:56:59.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:56:59.421+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.428315+0000) 2019-10-13T08:56:59.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:56:59.421+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.429143+0000) 2019-10-13T08:56:59.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:56:59.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:56:59.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:56:59.431+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:56:59.432476+0000) 2019-10-13T08:57:01.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:01.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.633514+0000) 2019-10-13T08:57:01.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:01.631+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:01.634733+0000) 2019-10-13T08:57:03.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:03.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:03.947586+0000) 2019-10-13T08:57:04.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:04.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.358417+0000) 2019-10-13T08:57:04.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:04.427+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.431791+0000) 2019-10-13T08:57:04.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:04.425+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.432525+0000) 2019-10-13T08:57:04.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:04.425+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.432908+0000) 2019-10-13T08:57:04.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:04.425+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.433311+0000) 2019-10-13T08:57:04.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:04.429+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.434907+0000) 2019-10-13T08:57:04.435 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:04.429+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.435509+0000) 2019-10-13T08:57:04.436 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:04.429+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.435736+0000) 2019-10-13T08:57:04.436 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:04.429+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:04.440 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:04.435+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:04.439876+0000) 2019-10-13T08:57:06.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:06.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:06.185677+0000) 2019-10-13T08:57:07.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:07.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:07.487740+0000) 2019-10-13T08:57:07.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:07.581+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:07.585286+0000) 2019-10-13T08:57:07.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:07.609+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:07.611510+0000) 2019-10-13T08:57:09.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:09.437+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.439993+0000) 2019-10-13T08:57:09.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:09.437+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.440819+0000) 2019-10-13T08:57:09.441 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:09.439+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.440747+0000) 2019-10-13T08:57:09.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:09.441+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.442274+0000) 2019-10-13T08:57:09.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:09.441+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.442448+0000) 2019-10-13T08:57:09.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:09.441+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.442890+0000) 2019-10-13T08:57:09.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:09.441+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:09.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:09.441+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.443847+0000) 2019-10-13T08:57:09.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:09.443+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:09.447327+0000) 2019-10-13T08:57:11.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:11.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.633729+0000) 2019-10-13T08:57:11.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:11.631+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:11.634942+0000) 2019-10-13T08:57:13.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:13.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:13.947807+0000) 2019-10-13T08:57:14.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:14.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.358651+0000) 2019-10-13T08:57:14.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:14.443+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.445998+0000) 2019-10-13T08:57:14.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:14.445+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.447182+0000) 2019-10-13T08:57:14.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:14.445+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.447954+0000) 2019-10-13T08:57:14.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:14.445+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.448237+0000) 2019-10-13T08:57:14.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:14.445+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.447641+0000) 2019-10-13T08:57:14.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:14.445+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.448781+0000) 2019-10-13T08:57:14.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:14.445+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.449702+0000) 2019-10-13T08:57:14.450 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:14.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:14.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:14.451+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:14.455529+0000) 2019-10-13T08:57:16.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:16.195+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:16.196598+0000) 2019-10-13T08:57:17.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:17.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:17.487955+0000) 2019-10-13T08:57:17.520 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:57:17.520 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:17.524 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:57:17.524 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:17.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:17.581+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:17.585498+0000) 2019-10-13T08:57:17.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:17.609+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:17.611706+0000) 2019-10-13T08:57:19.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:19.453+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.454309+0000) 2019-10-13T08:57:19.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:19.453+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.454892+0000) 2019-10-13T08:57:19.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:19.455+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.457252+0000) 2019-10-13T08:57:19.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:19.457+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.457950+0000) 2019-10-13T08:57:19.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:19.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:19.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:19.457+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.459034+0000) 2019-10-13T08:57:19.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:19.457+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.459796+0000) 2019-10-13T08:57:19.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:19.457+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.460388+0000) 2019-10-13T08:57:19.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:19.459+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:19.461385+0000) 2019-10-13T08:57:21.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:21.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.633948+0000) 2019-10-13T08:57:21.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:21.631+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:21.635135+0000) 2019-10-13T08:57:23.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:23.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:23.948023+0000) 2019-10-13T08:57:24.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:24.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.358864+0000) 2019-10-13T08:57:24.464 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:24.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.461788+0000) 2019-10-13T08:57:24.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:24.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.462804+0000) 2019-10-13T08:57:24.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:24.461+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.463991+0000) 2019-10-13T08:57:24.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:24.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:24.466 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:24.461+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.465712+0000) 2019-10-13T08:57:24.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:24.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.466414+0000) 2019-10-13T08:57:24.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:24.463+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.465741+0000) 2019-10-13T08:57:24.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:24.465+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.466868+0000) 2019-10-13T08:57:24.470 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:24.467+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:24.468898+0000) 2019-10-13T08:57:26.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:26.199+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:26.203123+0000) 2019-10-13T08:57:27.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:27.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:27.488157+0000) 2019-10-13T08:57:27.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:27.585+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:27.585783+0000) 2019-10-13T08:57:27.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:27.609+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:27.611870+0000) 2019-10-13T08:57:29.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:29.465+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.468517+0000) 2019-10-13T08:57:29.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:29.465+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.469441+0000) 2019-10-13T08:57:29.470 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:29.467+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.469763+0000) 2019-10-13T08:57:29.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:29.469+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.470270+0000) 2019-10-13T08:57:29.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:29.469+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.470851+0000) 2019-10-13T08:57:29.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:29.469+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.471641+0000) 2019-10-13T08:57:29.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:29.469+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.471847+0000) 2019-10-13T08:57:29.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:29.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84137 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:29.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:29.475+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:29.478677+0000) 2019-10-13T08:57:31.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:31.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.634145+0000) 2019-10-13T08:57:31.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:31.635+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:31.635331+0000) 2019-10-13T08:57:33.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:33.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:33.948239+0000) 2019-10-13T08:57:34.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:34.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.359074+0000) 2019-10-13T08:57:34.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:34.473+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.477246+0000) 2019-10-13T08:57:34.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:34.477+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.479004+0000) 2019-10-13T08:57:34.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:34.475+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.479126+0000) 2019-10-13T08:57:34.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:34.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.479263+0000) 2019-10-13T08:57:34.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:34.477+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.480054+0000) 2019-10-13T08:57:34.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:34.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:34.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:34.477+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.480575+0000) 2019-10-13T08:57:34.481 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:34.477+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.481502+0000) 2019-10-13T08:57:34.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:34.483+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:34.486565+0000) 2019-10-13T08:57:36.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:36.211+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:36.214302+0000) 2019-10-13T08:57:37.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:37.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:37.488336+0000) 2019-10-13T08:57:37.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:37.585+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:37.586031+0000) 2019-10-13T08:57:37.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:37.609+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:37.612056+0000) 2019-10-13T08:57:39.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:39.481+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.483841+0000) 2019-10-13T08:57:39.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:39.485+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.486323+0000) 2019-10-13T08:57:39.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:39.485+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.486589+0000) 2019-10-13T08:57:39.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:39.482+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.486798+0000) 2019-10-13T08:57:39.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:39.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:39.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:39.485+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.487893+0000) 2019-10-13T08:57:39.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:39.485+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.488150+0000) 2019-10-13T08:57:39.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:39.485+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.488538+0000) 2019-10-13T08:57:39.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:39.490+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:39.494612+0000) 2019-10-13T08:57:41.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:41.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.634360+0000) 2019-10-13T08:57:41.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:41.630+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:41.635534+0000) 2019-10-13T08:57:43.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:43.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:43.948486+0000) 2019-10-13T08:57:44.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:44.357+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.359280+0000) 2019-10-13T08:57:44.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:44.489+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.490633+0000) 2019-10-13T08:57:44.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:44.493+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.494326+0000) 2019-10-13T08:57:44.495 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:44.490+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.493743+0000) 2019-10-13T08:57:44.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:44.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:44.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:44.493+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.494747+0000) 2019-10-13T08:57:44.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:44.493+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.495117+0000) 2019-10-13T08:57:44.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:44.493+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.495549+0000) 2019-10-13T08:57:44.497 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:44.493+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.495964+0000) 2019-10-13T08:57:44.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:44.494+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:44.499785+0000) 2019-10-13T08:57:46.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:46.218+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:46.220858+0000) 2019-10-13T08:57:47.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:47.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:47.488532+0000) 2019-10-13T08:57:47.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:47.585+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:47.586210+0000) 2019-10-13T08:57:47.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:47.609+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:47.612251+0000) 2019-10-13T08:57:47.649 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:57:47.649 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:47.653 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:57:47.653 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:57:49.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:49.497+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.498319+0000) 2019-10-13T08:57:49.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:49.497+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.498458+0000) 2019-10-13T08:57:49.499 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:49.497+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.498832+0000) 2019-10-13T08:57:49.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:49.497+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.499272+0000) 2019-10-13T08:57:49.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:49.497+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.499638+0000) 2019-10-13T08:57:49.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:49.498+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.500208+0000) 2019-10-13T08:57:49.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:49.497+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.501071+0000) 2019-10-13T08:57:49.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:49.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:49.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:49.502+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:49.507275+0000) 2019-10-13T08:57:51.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:51.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.634607+0000) 2019-10-13T08:57:51.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:51.630+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:51.635789+0000) 2019-10-13T08:57:53.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:53.944+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:53.948701+0000) 2019-10-13T08:57:54.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:54.356+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.359481+0000) 2019-10-13T08:57:54.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:54.504+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.505892+0000) 2019-10-13T08:57:54.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:54.504+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.506197+0000) 2019-10-13T08:57:54.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:54.504+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.506557+0000) 2019-10-13T08:57:54.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:54.504+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.507106+0000) 2019-10-13T08:57:54.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:54.502+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.507617+0000) 2019-10-13T08:57:54.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:54.504+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.507817+0000) 2019-10-13T08:57:54.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:54.504+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.509126+0000) 2019-10-13T08:57:54.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:54.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:54.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:54.510+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:54.514634+0000) 2019-10-13T08:57:56.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:56.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:56.232206+0000) 2019-10-13T08:57:57.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:57.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:57.488727+0000) 2019-10-13T08:57:57.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:57.584+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:57.586395+0000) 2019-10-13T08:57:57.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:57.608+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:57.612461+0000) 2019-10-13T08:57:59.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:57:59.508+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.512981+0000) 2019-10-13T08:57:59.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:57:59.508+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.513668+0000) 2019-10-13T08:57:59.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:57:59.512+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.514056+0000) 2019-10-13T08:57:59.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:57:59.512+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.514372+0000) 2019-10-13T08:57:59.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:57:59.510+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.514258+0000) 2019-10-13T08:57:59.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:57:59.512+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.516308+0000) 2019-10-13T08:57:59.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:57:59.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:57:59.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:57:59.512+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.516910+0000) 2019-10-13T08:57:59.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:57:59.514+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:57:59.519309+0000) 2019-10-13T08:58:01.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:01.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.634834+0000) 2019-10-13T08:58:01.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:01.630+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:01.636018+0000) 2019-10-13T08:58:03.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:03.948+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:03.948901+0000) 2019-10-13T08:58:04.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:04.356+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.359637+0000) 2019-10-13T08:58:04.520 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:04.516+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.519593+0000) 2019-10-13T08:58:04.520 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:04.514+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.519462+0000) 2019-10-13T08:58:04.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:04.516+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.520633+0000) 2019-10-13T08:58:04.521 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:04.516+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.521617+0000) 2019-10-13T08:58:04.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:04.520+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.522314+0000) 2019-10-13T08:58:04.523 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:04.520+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.522571+0000) 2019-10-13T08:58:04.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:04.520+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.523001+0000) 2019-10-13T08:58:04.524 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:04.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:04.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:04.526+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:04.528254+0000) 2019-10-13T08:58:06.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:06.238+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:06.238669+0000) 2019-10-13T08:58:07.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:07.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:07.488941+0000) 2019-10-13T08:58:07.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:07.584+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:07.586610+0000) 2019-10-13T08:58:07.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:07.608+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:07.612681+0000) 2019-10-13T08:58:09.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:09.524+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.527224+0000) 2019-10-13T08:58:09.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:09.524+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.529230+0000) 2019-10-13T08:58:09.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:09.526+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.528993+0000) 2019-10-13T08:58:09.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:09.528+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.530191+0000) 2019-10-13T08:58:09.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:09.528+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.530685+0000) 2019-10-13T08:58:09.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:09.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:09.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:09.528+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.531763+0000) 2019-10-13T08:58:09.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:09.528+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.532240+0000) 2019-10-13T08:58:09.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:09.534+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:09.536602+0000) 2019-10-13T08:58:11.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:11.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.635060+0000) 2019-10-13T08:58:11.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:11.634+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:11.636208+0000) 2019-10-13T08:58:13.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:13.944+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:13.949099+0000) 2019-10-13T08:58:14.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:14.356+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.359870+0000) 2019-10-13T08:58:14.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:14.532+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.534490+0000) 2019-10-13T08:58:14.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:14.532+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.534984+0000) 2019-10-13T08:58:14.536 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:14.532+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.535651+0000) 2019-10-13T08:58:14.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:14.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:14.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:14.536+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.538867+0000) 2019-10-13T08:58:14.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:14.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.538715+0000) 2019-10-13T08:58:14.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:14.536+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.539897+0000) 2019-10-13T08:58:14.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:14.536+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.540452+0000) 2019-10-13T08:58:14.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:14.542+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:14.547316+0000) 2019-10-13T08:58:16.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:16.242+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:16.245300+0000) 2019-10-13T08:58:17.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:17.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:17.489142+0000) 2019-10-13T08:58:17.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:17.584+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:17.586836+0000) 2019-10-13T08:58:17.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:17.608+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:17.612876+0000) 2019-10-13T08:58:17.708 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:58:17.709 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:17.712 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:58:17.712 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:19.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:19.540+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.541859+0000) 2019-10-13T08:58:19.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:19.538+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.542688+0000) 2019-10-13T08:58:19.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:19.540+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.543346+0000) 2019-10-13T08:58:19.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:19.540+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.543654+0000) 2019-10-13T08:58:19.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:19.540+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.544017+0000) 2019-10-13T08:58:19.545 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:19.540+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.544826+0000) 2019-10-13T08:58:19.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:19.540+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.544979+0000) 2019-10-13T08:58:19.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:19.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:19.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:19.546+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:19.549944+0000) 2019-10-13T08:58:21.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:21.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.635260+0000) 2019-10-13T08:58:21.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:21.634+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:21.636406+0000) 2019-10-13T08:58:23.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:23.948+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:23.949315+0000) 2019-10-13T08:58:24.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:24.356+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.360081+0000) 2019-10-13T08:58:24.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:24.544+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.549128+0000) 2019-10-13T08:58:24.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:24.544+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.549447+0000) 2019-10-13T08:58:24.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:24.546+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.549253+0000) 2019-10-13T08:58:24.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:24.548+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.550136+0000) 2019-10-13T08:58:24.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:24.548+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.550288+0000) 2019-10-13T08:58:24.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:24.548+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.551417+0000) 2019-10-13T08:58:24.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:24.548+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.552043+0000) 2019-10-13T08:58:24.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:24.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84479 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:24.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:24.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:24.559331+0000) 2019-10-13T08:58:26.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:26.246+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:26.251746+0000) 2019-10-13T08:58:27.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:27.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:27.489373+0000) 2019-10-13T08:58:27.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:27.584+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:27.587033+0000) 2019-10-13T08:58:27.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:27.608+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:27.613065+0000) 2019-10-13T08:58:29.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:29.552+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.556202+0000) 2019-10-13T08:58:29.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:29.552+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.556661+0000) 2019-10-13T08:58:29.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:29.554+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.556926+0000) 2019-10-13T08:58:29.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:29.552+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.557328+0000) 2019-10-13T08:58:29.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:29.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:29.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:29.556+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.559935+0000) 2019-10-13T08:58:29.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:29.556+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.561127+0000) 2019-10-13T08:58:29.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:29.556+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.560502+0000) 2019-10-13T08:58:29.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:29.558+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:29.563437+0000) 2019-10-13T08:58:31.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:31.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.635431+0000) 2019-10-13T08:58:31.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:31.634+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:31.636635+0000) 2019-10-13T08:58:33.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:33.944+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:33.949545+0000) 2019-10-13T08:58:34.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:34.360+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.360317+0000) 2019-10-13T08:58:34.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:34.560+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.564045+0000) 2019-10-13T08:58:34.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:34.560+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.564106+0000) 2019-10-13T08:58:34.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:34.560+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.564844+0000) 2019-10-13T08:58:34.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:34.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:34.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:34.562+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.566326+0000) 2019-10-13T08:58:34.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:34.564+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.567088+0000) 2019-10-13T08:58:34.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:34.564+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.567333+0000) 2019-10-13T08:58:34.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:34.564+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.567895+0000) 2019-10-13T08:58:34.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:34.566+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:34.569512+0000) 2019-10-13T08:58:36.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:36.258+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:36.258291+0000) 2019-10-13T08:58:37.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:37.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:37.489601+0000) 2019-10-13T08:58:37.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:37.584+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:37.587201+0000) 2019-10-13T08:58:37.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:37.612+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:37.613261+0000) 2019-10-13T08:58:39.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:39.564+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.569661+0000) 2019-10-13T08:58:39.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:39.568+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.570003+0000) 2019-10-13T08:58:39.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:39.568+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.571409+0000) 2019-10-13T08:58:39.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:39.566+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.571245+0000) 2019-10-13T08:58:39.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:39.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:39.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:39.568+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.573664+0000) 2019-10-13T08:58:39.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:39.572+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.573854+0000) 2019-10-13T08:58:39.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:39.572+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.574992+0000) 2019-10-13T08:58:39.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:39.574+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:39.578211+0000) 2019-10-13T08:58:41.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:41.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.635638+0000) 2019-10-13T08:58:41.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:41.634+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:41.636865+0000) 2019-10-13T08:58:43.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:43.948+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:43.949777+0000) 2019-10-13T08:58:44.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:44.356+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.360519+0000) 2019-10-13T08:58:44.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:44.572+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.576216+0000) 2019-10-13T08:58:44.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:44.572+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.576635+0000) 2019-10-13T08:58:44.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:44.572+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.577525+0000) 2019-10-13T08:58:44.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:44.576+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.580408+0000) 2019-10-13T08:58:44.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:44.578+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.580405+0000) 2019-10-13T08:58:44.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:44.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:44.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:44.576+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.581156+0000) 2019-10-13T08:58:44.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:44.576+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.581236+0000) 2019-10-13T08:58:44.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:44.582+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:44.586083+0000) 2019-10-13T08:58:46.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:46.261+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:46.264736+0000) 2019-10-13T08:58:47.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:47.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:47.489830+0000) 2019-10-13T08:58:47.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:47.583+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:47.587424+0000) 2019-10-13T08:58:47.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:47.607+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:47.613445+0000) 2019-10-13T08:58:47.826 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:58:47.826 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:47.830 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:58:47.830 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:58:49.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:49.579+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.583749+0000) 2019-10-13T08:58:49.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:49.583+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.587039+0000) 2019-10-13T08:58:49.587 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:49.583+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.586986+0000) 2019-10-13T08:58:49.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:49.583+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.587512+0000) 2019-10-13T08:58:49.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:49.585+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.589714+0000) 2019-10-13T08:58:49.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:49.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:49.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:49.583+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.589127+0000) 2019-10-13T08:58:49.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:49.583+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.589694+0000) 2019-10-13T08:58:49.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:49.589+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:49.593668+0000) 2019-10-13T08:58:51.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:51.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.635856+0000) 2019-10-13T08:58:51.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:51.633+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:51.637094+0000) 2019-10-13T08:58:53.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:53.947+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:53.949996+0000) 2019-10-13T08:58:54.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:54.355+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.360723+0000) 2019-10-13T08:58:54.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:54.587+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.591822+0000) 2019-10-13T08:58:54.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:54.587+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.592349+0000) 2019-10-13T08:58:54.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:54.587+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.593022+0000) 2019-10-13T08:58:54.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:54.587+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.593611+0000) 2019-10-13T08:58:54.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:54.591+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.594111+0000) 2019-10-13T08:58:54.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:54.591+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.595302+0000) 2019-10-13T08:58:54.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:54.589+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.595500+0000) 2019-10-13T08:58:54.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:54.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:54.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:54.593+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:54.599746+0000) 2019-10-13T08:58:56.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:56.269+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:56.271262+0000) 2019-10-13T08:58:57.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:57.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:57.490028+0000) 2019-10-13T08:58:57.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:57.583+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:57.587614+0000) 2019-10-13T08:58:57.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:57.607+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:57.613646+0000) 2019-10-13T08:58:59.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:58:59.595+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.600858+0000) 2019-10-13T08:58:59.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:58:59.595+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.601283+0000) 2019-10-13T08:58:59.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:58:59.595+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.601612+0000) 2019-10-13T08:58:59.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:58:59.599+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.601832+0000) 2019-10-13T08:58:59.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:58:59.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.601919+0000) 2019-10-13T08:58:59.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:58:59.599+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.602704+0000) 2019-10-13T08:58:59.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:58:59.599+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.603524+0000) 2019-10-13T08:58:59.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:58:59.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:58:59.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:58:59.601+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:58:59.605840+0000) 2019-10-13T08:59:01.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:01.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.636008+0000) 2019-10-13T08:59:01.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:01.633+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:01.637294+0000) 2019-10-13T08:59:03.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:03.947+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:03.950193+0000) 2019-10-13T08:59:04.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:04.359+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.360931+0000) 2019-10-13T08:59:04.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:04.603+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.606856+0000) 2019-10-13T08:59:04.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:04.603+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.608926+0000) 2019-10-13T08:59:04.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:04.607+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.610165+0000) 2019-10-13T08:59:04.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:04.607+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.610162+0000) 2019-10-13T08:59:04.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:04.605+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.609981+0000) 2019-10-13T08:59:04.611 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:04.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84731 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:04.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:04.607+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.610954+0000) 2019-10-13T08:59:04.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:04.607+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.613128+0000) 2019-10-13T08:59:04.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:04.609+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:04.614531+0000) 2019-10-13T08:59:06.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:06.273+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:06.277829+0000) 2019-10-13T08:59:07.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:07.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:07.490222+0000) 2019-10-13T08:59:07.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:07.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:07.587813+0000) 2019-10-13T08:59:07.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:07.611+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:07.613845+0000) 2019-10-13T08:59:09.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:09.611+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.614203+0000) 2019-10-13T08:59:09.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:09.609+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.613912+0000) 2019-10-13T08:59:09.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:09.611+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.614872+0000) 2019-10-13T08:59:09.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:09.611+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.615692+0000) 2019-10-13T08:59:09.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:09.611+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.616413+0000) 2019-10-13T08:59:09.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:09.611+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.616857+0000) 2019-10-13T08:59:09.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:09.611+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.617349+0000) 2019-10-13T08:59:09.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:09.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:09.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:09.617+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:09.620926+0000) 2019-10-13T08:59:11.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:11.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.636228+0000) 2019-10-13T08:59:11.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:11.633+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:11.637486+0000) 2019-10-13T08:59:13.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:13.947+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:13.950393+0000) 2019-10-13T08:59:14.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:14.359+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.361157+0000) 2019-10-13T08:59:14.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:14.615+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.621600+0000) 2019-10-13T08:59:14.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:14.619+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.623161+0000) 2019-10-13T08:59:14.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:14.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:14.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:14.619+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.625534+0000) 2019-10-13T08:59:14.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:14.623+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.625880+0000) 2019-10-13T08:59:14.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:14.621+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.626190+0000) 2019-10-13T08:59:14.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:14.623+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.627005+0000) 2019-10-13T08:59:14.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:14.623+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.627282+0000) 2019-10-13T08:59:14.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:14.625+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:14.628235+0000) 2019-10-13T08:59:16.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:16.281+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:16.284355+0000) 2019-10-13T08:59:17.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:17.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:17.490403+0000) 2019-10-13T08:59:17.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:17.583+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:17.588013+0000) 2019-10-13T08:59:17.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:17.611+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:17.614043+0000) 2019-10-13T08:59:17.927 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:59:17.928 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:17.931 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:59:17.931 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:19.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:19.625+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.628148+0000) 2019-10-13T08:59:19.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:19.623+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.629230+0000) 2019-10-13T08:59:19.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:19.627+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.630946+0000) 2019-10-13T08:59:19.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:19.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.631435+0000) 2019-10-13T08:59:19.632 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:19.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:19.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:19.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.632108+0000) 2019-10-13T08:59:19.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:19.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.633116+0000) 2019-10-13T08:59:19.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:19.627+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.633649+0000) 2019-10-13T08:59:19.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:19.633+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:19.639538+0000) 2019-10-13T08:59:21.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:21.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.636478+0000) 2019-10-13T08:59:21.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:21.637+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:21.637703+0000) 2019-10-13T08:59:23.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:23.947+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:23.950548+0000) 2019-10-13T08:59:24.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:24.355+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.361362+0000) 2019-10-13T08:59:24.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:24.629+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.634560+0000) 2019-10-13T08:59:24.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:24.631+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.635759+0000) 2019-10-13T08:59:24.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:24.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:24.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:24.635+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.639355+0000) 2019-10-13T08:59:24.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:24.635+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.639819+0000) 2019-10-13T08:59:24.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:24.635+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.640124+0000) 2019-10-13T08:59:24.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:24.635+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.640594+0000) 2019-10-13T08:59:24.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:24.635+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.641288+0000) 2019-10-13T08:59:24.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:24.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:24.642224+0000) 2019-10-13T08:59:26.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:26.289+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:26.290822+0000) 2019-10-13T08:59:27.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:27.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:27.490601+0000) 2019-10-13T08:59:27.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:27.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:27.588207+0000) 2019-10-13T08:59:27.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:27.611+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:27.614234+0000) 2019-10-13T08:59:29.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:29.639+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.642089+0000) 2019-10-13T08:59:29.643 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:29.639+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.642853+0000) 2019-10-13T08:59:29.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:29.639+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.643574+0000) 2019-10-13T08:59:29.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:29.639+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.643937+0000) 2019-10-13T08:59:29.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:29.641+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.644971+0000) 2019-10-13T08:59:29.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:29.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:29.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:29.643+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.646171+0000) 2019-10-13T08:59:29.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:29.643+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.646567+0000) 2019-10-13T08:59:29.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:29.645+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:29.648989+0000) 2019-10-13T08:59:31.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:31.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.636674+0000) 2019-10-13T08:59:31.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:31.637+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:31.637905+0000) 2019-10-13T08:59:33.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:33.947+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:33.950748+0000) 2019-10-13T08:59:34.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:34.355+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.361578+0000) 2019-10-13T08:59:34.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:34.643+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.649706+0000) 2019-10-13T08:59:34.652 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:34.647+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.650398+0000) 2019-10-13T08:59:34.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:34.647+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.650197+0000) 2019-10-13T08:59:34.652 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:34.647+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.650946+0000) 2019-10-13T08:59:34.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:34.647+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.651582+0000) 2019-10-13T08:59:34.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:34.649+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.652339+0000) 2019-10-13T08:59:34.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:34.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:34.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:34.647+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.652247+0000) 2019-10-13T08:59:34.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:34.649+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:34.655017+0000) 2019-10-13T08:59:36.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:36.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:36.297369+0000) 2019-10-13T08:59:37.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:37.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:37.490805+0000) 2019-10-13T08:59:37.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:37.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:37.588373+0000) 2019-10-13T08:59:37.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:37.611+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:37.614431+0000) 2019-10-13T08:59:39.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:39.651+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.656075+0000) 2019-10-13T08:59:39.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:39.655+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.658411+0000) 2019-10-13T08:59:39.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:39.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:39.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:39.655+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.660183+0000) 2019-10-13T08:59:39.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:39.655+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.660737+0000) 2019-10-13T08:59:39.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:39.655+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.661236+0000) 2019-10-13T08:59:39.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:39.659+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.662704+0000) 2019-10-13T08:59:39.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:39.657+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.662963+0000) 2019-10-13T08:59:39.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:39.661+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:39.666627+0000) 2019-10-13T08:59:41.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:41.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.636865+0000) 2019-10-13T08:59:41.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:41.633+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:41.638125+0000) 2019-10-13T08:59:43.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:43.946+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:43.950952+0000) 2019-10-13T08:59:44.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:44.358+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.361780+0000) 2019-10-13T08:59:44.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:44.658+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.663051+0000) 2019-10-13T08:59:44.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:44.658+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.663439+0000) 2019-10-13T08:59:44.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:44.661+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.664361+0000) 2019-10-13T08:59:44.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:44.658+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.665361+0000) 2019-10-13T08:59:44.666 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:44.662+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.665829+0000) 2019-10-13T08:59:44.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:44.662+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.666276+0000) 2019-10-13T08:59:44.667 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:44.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:44.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:44.662+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.666777+0000) 2019-10-13T08:59:44.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:44.669+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:44.672937+0000) 2019-10-13T08:59:46.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:46.301+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:46.303997+0000) 2019-10-13T08:59:47.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:47.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:47.491004+0000) 2019-10-13T08:59:47.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:47.582+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:47.588615+0000) 2019-10-13T08:59:47.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:47.610+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:47.614637+0000) 2019-10-13T08:59:48.029 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T08:59:48.029 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:48.033 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T08:59:48.033 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T08:59:49.670 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:49.666+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.669966+0000) 2019-10-13T08:59:49.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:49.666+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.670797+0000) 2019-10-13T08:59:49.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:49.666+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.671170+0000) 2019-10-13T08:59:49.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:49.666+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.671957+0000) 2019-10-13T08:59:49.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:49.665+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.671864+0000) 2019-10-13T08:59:49.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:49.666+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.673229+0000) 2019-10-13T08:59:49.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:49.670+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.674008+0000) 2019-10-13T08:59:49.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:49.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:49.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:49.673+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:49.677457+0000) 2019-10-13T08:59:51.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:51.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.637090+0000) 2019-10-13T08:59:51.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:51.632+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:51.638325+0000) 2019-10-13T08:59:53.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:53.946+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:53.951155+0000) 2019-10-13T08:59:54.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:54.358+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.361981+0000) 2019-10-13T08:59:54.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:54.670+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.677078+0000) 2019-10-13T08:59:54.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:54.674+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.678187+0000) 2019-10-13T08:59:54.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:54.672+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.679441+0000) 2019-10-13T08:59:54.681 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:54.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:54.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:54.678+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.681947+0000) 2019-10-13T08:59:54.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:54.678+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.682339+0000) 2019-10-13T08:59:54.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:54.678+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.682733+0000) 2019-10-13T08:59:54.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:54.678+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.683330+0000) 2019-10-13T08:59:54.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:54.680+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:54.687225+0000) 2019-10-13T08:59:56.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:56.308+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:56.310524+0000) 2019-10-13T08:59:57.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:57.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:57.491208+0000) 2019-10-13T08:59:57.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:57.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:57.588772+0000) 2019-10-13T08:59:57.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:57.614+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:57.614834+0000) 2019-10-13T08:59:59.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T08:59:59.678+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.685013+0000) 2019-10-13T08:59:59.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T08:59:59.678+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.685368+0000) 2019-10-13T08:59:59.686 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T08:59:59.678+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.685658+0000) 2019-10-13T08:59:59.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T08:59:59.682+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.686427+0000) 2019-10-13T08:59:59.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T08:59:59.680+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.686158+0000) 2019-10-13T08:59:59.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T08:59:59.682+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.687586+0000) 2019-10-13T08:59:59.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T08:59:59.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T08:59:59.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T08:59:59.682+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.688468+0000) 2019-10-13T08:59:59.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T08:59:59.684+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T07:59:59.690311+0000) 2019-10-13T09:00:01.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:01.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.637282+0000) 2019-10-13T09:00:01.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:01.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:01.638584+0000) 2019-10-13T09:00:03.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:03.950+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:03.951350+0000) 2019-10-13T09:00:04.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:04.358+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.362199+0000) 2019-10-13T09:00:04.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:04.686+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.691635+0000) 2019-10-13T09:00:04.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:04.686+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.692609+0000) 2019-10-13T09:00:04.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:04.686+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.693101+0000) 2019-10-13T09:00:04.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:04.686+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.693187+0000) 2019-10-13T09:00:04.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:04.686+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.693602+0000) 2019-10-13T09:00:04.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:04.690+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.694301+0000) 2019-10-13T09:00:04.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:04.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:04.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:04.684+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.690370+0000) 2019-10-13T09:00:04.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:04.692+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:04.697494+0000) 2019-10-13T09:00:06.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:06.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:06.316896+0000) 2019-10-13T09:00:07.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:07.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:07.491404+0000) 2019-10-13T09:00:07.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:07.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:07.589001+0000) 2019-10-13T09:00:07.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:07.614+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:07.615067+0000) 2019-10-13T09:00:09.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:09.694+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.698518+0000) 2019-10-13T09:00:09.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:09.694+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.700171+0000) 2019-10-13T09:00:09.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:09.694+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.700376+0000) 2019-10-13T09:00:09.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:09.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:09.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:09.696+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.703440+0000) 2019-10-13T09:00:09.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:09.704+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.709990+0000) 2019-10-13T09:00:09.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:09.698+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.702231+0000) 2019-10-13T09:00:09.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:09.698+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.702466+0000) 2019-10-13T09:00:09.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:09.698+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:09.702757+0000) 2019-10-13T09:00:11.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:11.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.637452+0000) 2019-10-13T09:00:11.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:11.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:11.638776+0000) 2019-10-13T09:00:13.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:13.950+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:13.951566+0000) 2019-10-13T09:00:14.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:14.358+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.362447+0000) 2019-10-13T09:00:14.705 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:14.698+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.704829+0000) 2019-10-13T09:00:14.705 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:14.698+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.705138+0000) 2019-10-13T09:00:14.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:14.698+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.705543+0000) 2019-10-13T09:00:14.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:14.702+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.707297+0000) 2019-10-13T09:00:14.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:14.702+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.707458+0000) 2019-10-13T09:00:14.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:14.700+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.707957+0000) 2019-10-13T09:00:14.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:14.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:14.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:14.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.709013+0000) 2019-10-13T09:00:14.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:14.708+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:14.713825+0000) 2019-10-13T09:00:16.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:16.320+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:16.323362+0000) 2019-10-13T09:00:17.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:17.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:17.491602+0000) 2019-10-13T09:00:17.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:17.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:17.589198+0000) 2019-10-13T09:00:17.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:17.614+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:17.615286+0000) 2019-10-13T09:00:18.086 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:00:18.086 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:18.090 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:00:18.090 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:19.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:19.706+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.711939+0000) 2019-10-13T09:00:19.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:19.706+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.712537+0000) 2019-10-13T09:00:19.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:19.706+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.713034+0000) 2019-10-13T09:00:19.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:19.706+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.713417+0000) 2019-10-13T09:00:19.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:19.710+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.714156+0000) 2019-10-13T09:00:19.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:19.710+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.714500+0000) 2019-10-13T09:00:19.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:19.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:19.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:19.708+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.715273+0000) 2019-10-13T09:00:19.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:19.712+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:19.717904+0000) 2019-10-13T09:00:21.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:21.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.637676+0000) 2019-10-13T09:00:21.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:21.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:21.638975+0000) 2019-10-13T09:00:23.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:23.946+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:23.951801+0000) 2019-10-13T09:00:24.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:24.358+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.362647+0000) 2019-10-13T09:00:24.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:24.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.719051+0000) 2019-10-13T09:00:24.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:24.714+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.719484+0000) 2019-10-13T09:00:24.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:24.714+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.720376+0000) 2019-10-13T09:00:24.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:24.714+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.720894+0000) 2019-10-13T09:00:24.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:24.714+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.721300+0000) 2019-10-13T09:00:24.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:24.714+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.721618+0000) 2019-10-13T09:00:24.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:24.716+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.721970+0000) 2019-10-13T09:00:24.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:24.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:24.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:24.724+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:24.730376+0000) 2019-10-13T09:00:26.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:26.328+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:26.329995+0000) 2019-10-13T09:00:27.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:27.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:27.491804+0000) 2019-10-13T09:00:27.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:27.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:27.589429+0000) 2019-10-13T09:00:27.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:27.614+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:27.615515+0000) 2019-10-13T09:00:29.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:29.722+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.727219+0000) 2019-10-13T09:00:29.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:29.722+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.727477+0000) 2019-10-13T09:00:29.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:29.722+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.728054+0000) 2019-10-13T09:00:29.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:29.722+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.728369+0000) 2019-10-13T09:00:29.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:29.724+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.728259+0000) 2019-10-13T09:00:29.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:29.722+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.729078+0000) 2019-10-13T09:00:29.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:29.722+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.729704+0000) 2019-10-13T09:00:29.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:29.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:29.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:29.732+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:29.737295+0000) 2019-10-13T09:00:31.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:31.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.637901+0000) 2019-10-13T09:00:31.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:31.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:31.639163+0000) 2019-10-13T09:00:33.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:33.950+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:33.952000+0000) 2019-10-13T09:00:34.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:34.362+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.362849+0000) 2019-10-13T09:00:34.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:34.730+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.734040+0000) 2019-10-13T09:00:34.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:34.730+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.736229+0000) 2019-10-13T09:00:34.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:34.730+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.736387+0000) 2019-10-13T09:00:34.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:34.730+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.736982+0000) 2019-10-13T09:00:34.738 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:34.734+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.738036+0000) 2019-10-13T09:00:34.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:34.734+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.738399+0000) 2019-10-13T09:00:34.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:34.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:34.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:34.732+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.738914+0000) 2019-10-13T09:00:34.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:34.740+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:34.745808+0000) 2019-10-13T09:00:36.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:36.336+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:36.336529+0000) 2019-10-13T09:00:37.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:37.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:37.491998+0000) 2019-10-13T09:00:37.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:37.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:37.589629+0000) 2019-10-13T09:00:37.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:37.614+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:37.615708+0000) 2019-10-13T09:00:39.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:39.737+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.741896+0000) 2019-10-13T09:00:39.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:39.737+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.742417+0000) 2019-10-13T09:00:39.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:39.737+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.742647+0000) 2019-10-13T09:00:39.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:39.737+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.743275+0000) 2019-10-13T09:00:39.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:39.736+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.743106+0000) 2019-10-13T09:00:39.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:39.737+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.744025+0000) 2019-10-13T09:00:39.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:39.737+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.744870+0000) 2019-10-13T09:00:39.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:39.740+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:39.744556+0000) 2019-10-13T09:00:39.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:39.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:41.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:41.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.638096+0000) 2019-10-13T09:00:41.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:41.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:41.639373+0000) 2019-10-13T09:00:43.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:43.945+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:43.952231+0000) 2019-10-13T09:00:44.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:44.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.363074+0000) 2019-10-13T09:00:44.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:44.741+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.748168+0000) 2019-10-13T09:00:44.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:44.741+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.749086+0000) 2019-10-13T09:00:44.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:44.741+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.749483+0000) 2019-10-13T09:00:44.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:44.745+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.750522+0000) 2019-10-13T09:00:44.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:44.744+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.750526+0000) 2019-10-13T09:00:44.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:44.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:44.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:44.745+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.753367+0000) 2019-10-13T09:00:44.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:44.745+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.753689+0000) 2019-10-13T09:00:44.758 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:44.752+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:44.757941+0000) 2019-10-13T09:00:46.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:46.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:46.343038+0000) 2019-10-13T09:00:47.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:47.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:47.492193+0000) 2019-10-13T09:00:47.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:47.585+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:47.589808+0000) 2019-10-13T09:00:47.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:47.613+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:47.615857+0000) 2019-10-13T09:00:48.198 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:00:48.198 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:48.202 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:00:48.202 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:00:49.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:49.749+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.756608+0000) 2019-10-13T09:00:49.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:49.749+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.757084+0000) 2019-10-13T09:00:49.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:49.752+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.757606+0000) 2019-10-13T09:00:49.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:49.753+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.758827+0000) 2019-10-13T09:00:49.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:49.753+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.759090+0000) 2019-10-13T09:00:49.760 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:49.753+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.759514+0000) 2019-10-13T09:00:49.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:49.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.759814+0000) 2019-10-13T09:00:49.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:49.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:49.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:49.760+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:49.765866+0000) 2019-10-13T09:00:51.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:51.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.638307+0000) 2019-10-13T09:00:51.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:51.636+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:51.639564+0000) 2019-10-13T09:00:53.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:53.949+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:53.952437+0000) 2019-10-13T09:00:54.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:54.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.363303+0000) 2019-10-13T09:00:54.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:54.761+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.763343+0000) 2019-10-13T09:00:54.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:54.761+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.764883+0000) 2019-10-13T09:00:54.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:54.761+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.765343+0000) 2019-10-13T09:00:54.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:54.764+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.766288+0000) 2019-10-13T09:00:54.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:54.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:54.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:54.765+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.768373+0000) 2019-10-13T09:00:54.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:54.765+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.768577+0000) 2019-10-13T09:00:54.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:54.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.769345+0000) 2019-10-13T09:00:54.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:54.772+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:54.775957+0000) 2019-10-13T09:00:56.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:56.348+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:56.349564+0000) 2019-10-13T09:00:57.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:57.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:57.492424+0000) 2019-10-13T09:00:57.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:57.589+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:57.590015+0000) 2019-10-13T09:00:57.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:57.613+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:57.616046+0000) 2019-10-13T09:00:59.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:00:59.769+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.770790+0000) 2019-10-13T09:00:59.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:00:59.767+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.770647+0000) 2019-10-13T09:00:59.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:00:59.769+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.771814+0000) 2019-10-13T09:00:59.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:00:59.769+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.772436+0000) 2019-10-13T09:00:59.773 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:00:59.769+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.772739+0000) 2019-10-13T09:00:59.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:00:59.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:00:59.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:00:59.773+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.775518+0000) 2019-10-13T09:00:59.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:00:59.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.775864+0000) 2019-10-13T09:00:59.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:00:59.775+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:00:59.779035+0000) 2019-10-13T09:01:01.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:01.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.638525+0000) 2019-10-13T09:01:01.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:01.635+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:01.639765+0000) 2019-10-13T09:01:03.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:03.949+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:03.952670+0000) 2019-10-13T09:01:04.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:04.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.363507+0000) 2019-10-13T09:01:04.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:04.775+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.777520+0000) 2019-10-13T09:01:04.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:04.777+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.779306+0000) 2019-10-13T09:01:04.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:04.777+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.780989+0000) 2019-10-13T09:01:04.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:04.777+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.780801+0000) 2019-10-13T09:01:04.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:04.781+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.781891+0000) 2019-10-13T09:01:04.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:04.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:04.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:04.781+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.782706+0000) 2019-10-13T09:01:04.783 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:04.781+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.783342+0000) 2019-10-13T09:01:04.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:04.787+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:04.790153+0000) 2019-10-13T09:01:06.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:06.355+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:06.356264+0000) 2019-10-13T09:01:07.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:07.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:07.492622+0000) 2019-10-13T09:01:07.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:07.589+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:07.590232+0000) 2019-10-13T09:01:07.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:07.613+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:07.616241+0000) 2019-10-13T09:01:09.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:09.785+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.786158+0000) 2019-10-13T09:01:09.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:09.785+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.786451+0000) 2019-10-13T09:01:09.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:09.785+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.786783+0000) 2019-10-13T09:01:09.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:09.785+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.787393+0000) 2019-10-13T09:01:09.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:09.787+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.788402+0000) 2019-10-13T09:01:09.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:09.785+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.789204+0000) 2019-10-13T09:01:09.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:09.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:09.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:09.785+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.789623+0000) 2019-10-13T09:01:09.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:09.795+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:09.796567+0000) 2019-10-13T09:01:11.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:11.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.638776+0000) 2019-10-13T09:01:11.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:11.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:11.639988+0000) 2019-10-13T09:01:13.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:13.949+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:13.952872+0000) 2019-10-13T09:01:14.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:14.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.363653+0000) 2019-10-13T09:01:14.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:14.789+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.792777+0000) 2019-10-13T09:01:14.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:14.791+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.792833+0000) 2019-10-13T09:01:14.794 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:14.789+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.793635+0000) 2019-10-13T09:01:14.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:14.793+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.794426+0000) 2019-10-13T09:01:14.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:14.793+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.794761+0000) 2019-10-13T09:01:14.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:14.793+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.795524+0000) 2019-10-13T09:01:14.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:14.793+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.796136+0000) 2019-10-13T09:01:14.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:14.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:14.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:14.799+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:14.801703+0000) 2019-10-13T09:01:16.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:16.359+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:16.362888+0000) 2019-10-13T09:01:17.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:17.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:17.492839+0000) 2019-10-13T09:01:17.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:17.589+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:17.590462+0000) 2019-10-13T09:01:17.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:17.613+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:17.616447+0000) 2019-10-13T09:01:18.313 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:01:18.313 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:18.316 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:01:18.317 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:19.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:19.797+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.799972+0000) 2019-10-13T09:01:19.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:19.797+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.800637+0000) 2019-10-13T09:01:19.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:19.801+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.803158+0000) 2019-10-13T09:01:19.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:19.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:19.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:19.801+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.803636+0000) 2019-10-13T09:01:19.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:19.799+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.803840+0000) 2019-10-13T09:01:19.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:19.801+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.805175+0000) 2019-10-13T09:01:19.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:19.801+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.805597+0000) 2019-10-13T09:01:19.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:19.807+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:19.809851+0000) 2019-10-13T09:01:21.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:21.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.638974+0000) 2019-10-13T09:01:21.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:21.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:21.640209+0000) 2019-10-13T09:01:23.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:23.949+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:23.953093+0000) 2019-10-13T09:01:24.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:24.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.363862+0000) 2019-10-13T09:01:24.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:24.805+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.808611+0000) 2019-10-13T09:01:24.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:24.805+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.809022+0000) 2019-10-13T09:01:24.810 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:24.805+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.809584+0000) 2019-10-13T09:01:24.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:24.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85612 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:24.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:24.807+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.810396+0000) 2019-10-13T09:01:24.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:24.809+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.811413+0000) 2019-10-13T09:01:24.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:24.809+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.811851+0000) 2019-10-13T09:01:24.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:24.813+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.813791+0000) 2019-10-13T09:01:24.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:24.815+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:24.816736+0000) 2019-10-13T09:01:26.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:26.367+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:26.369494+0000) 2019-10-13T09:01:27.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:27.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:27.493097+0000) 2019-10-13T09:01:27.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:27.589+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:27.590621+0000) 2019-10-13T09:01:27.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:27.613+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:27.616605+0000) 2019-10-13T09:01:29.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:29.811+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.814434+0000) 2019-10-13T09:01:29.816 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:29.813+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.816445+0000) 2019-10-13T09:01:29.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:29.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85643 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:29.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:29.813+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.817142+0000) 2019-10-13T09:01:29.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:29.813+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.817369+0000) 2019-10-13T09:01:29.818 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:29.817+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.818300+0000) 2019-10-13T09:01:29.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:29.817+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.818729+0000) 2019-10-13T09:01:29.819 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:29.817+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.819350+0000) 2019-10-13T09:01:29.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:29.823+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:29.827791+0000) 2019-10-13T09:01:31.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:31.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.639157+0000) 2019-10-13T09:01:31.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:31.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:31.640431+0000) 2019-10-13T09:01:33.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:33.953+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:33.953288+0000) 2019-10-13T09:01:34.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:34.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.364016+0000) 2019-10-13T09:01:34.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:34.817+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.820665+0000) 2019-10-13T09:01:34.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:34.817+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.821122+0000) 2019-10-13T09:01:34.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:34.819+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.821750+0000) 2019-10-13T09:01:34.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:34.821+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.821805+0000) 2019-10-13T09:01:34.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:34.821+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.822631+0000) 2019-10-13T09:01:34.824 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:34.821+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.823377+0000) 2019-10-13T09:01:34.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:34.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.823805+0000) 2019-10-13T09:01:34.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:34.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:34.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:34.827+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:34.830705+0000) 2019-10-13T09:01:36.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:36.371+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:36.375974+0000) 2019-10-13T09:01:37.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:37.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:37.493295+0000) 2019-10-13T09:01:37.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:37.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:37.590838+0000) 2019-10-13T09:01:37.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:37.612+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:37.616824+0000) 2019-10-13T09:01:39.828 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:39.824+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.827823+0000) 2019-10-13T09:01:39.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:39.824+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.828173+0000) 2019-10-13T09:01:39.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:39.827+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.830712+0000) 2019-10-13T09:01:39.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:39.828+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.831559+0000) 2019-10-13T09:01:39.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:39.828+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.832045+0000) 2019-10-13T09:01:39.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:39.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:39.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:39.828+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.832625+0000) 2019-10-13T09:01:39.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:39.828+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.833041+0000) 2019-10-13T09:01:39.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:39.835+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:39.837489+0000) 2019-10-13T09:01:41.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:41.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.639358+0000) 2019-10-13T09:01:41.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:41.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:41.640627+0000) 2019-10-13T09:01:43.962 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:43.948+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:43.953513+0000) 2019-10-13T09:01:44.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:44.360+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.364225+0000) 2019-10-13T09:01:44.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:44.832+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.835404+0000) 2019-10-13T09:01:44.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:44.832+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.836057+0000) 2019-10-13T09:01:44.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:44.832+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.836361+0000) 2019-10-13T09:01:44.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:44.832+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.836897+0000) 2019-10-13T09:01:44.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:44.832+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.837562+0000) 2019-10-13T09:01:44.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:44.835+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.837611+0000) 2019-10-13T09:01:44.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:44.836+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.838347+0000) 2019-10-13T09:01:44.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:44.843+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:44.844591+0000) 2019-10-13T09:01:44.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:44.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:46.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:46.379+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:46.382327+0000) 2019-10-13T09:01:47.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:47.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:47.493495+0000) 2019-10-13T09:01:47.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:47.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:47.591073+0000) 2019-10-13T09:01:47.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:47.616+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:47.617055+0000) 2019-10-13T09:01:48.390 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:01:48.391 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:48.395 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:01:48.395 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:01:49.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:49.840+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.842493+0000) 2019-10-13T09:01:49.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:49.840+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.842949+0000) 2019-10-13T09:01:49.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:49.839+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.842979+0000) 2019-10-13T09:01:49.844 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:49.840+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.843296+0000) 2019-10-13T09:01:49.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:49.840+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.843754+0000) 2019-10-13T09:01:49.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:49.840+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.844113+0000) 2019-10-13T09:01:49.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:49.840+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.844724+0000) 2019-10-13T09:01:49.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:49.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:49.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:49.847+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:49.851673+0000) 2019-10-13T09:01:51.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:51.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.639545+0000) 2019-10-13T09:01:51.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:51.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:51.640855+0000) 2019-10-13T09:01:53.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:53.948+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:53.953709+0000) 2019-10-13T09:01:54.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:54.360+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.364421+0000) 2019-10-13T09:01:54.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:54.851+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.852282+0000) 2019-10-13T09:01:54.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:54.848+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.853072+0000) 2019-10-13T09:01:54.854 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:54.848+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.853366+0000) 2019-10-13T09:01:54.854 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:54.852+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.854072+0000) 2019-10-13T09:01:54.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:54.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:54.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:54.852+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.855225+0000) 2019-10-13T09:01:54.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:54.852+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.855538+0000) 2019-10-13T09:01:54.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:54.852+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.855783+0000) 2019-10-13T09:01:54.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:54.859+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:54.860286+0000) 2019-10-13T09:01:56.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:56.387+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:56.388824+0000) 2019-10-13T09:01:57.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:57.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:57.493696+0000) 2019-10-13T09:01:57.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:57.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:57.591290+0000) 2019-10-13T09:01:57.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:57.612+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:57.617323+0000) 2019-10-13T09:01:59.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:01:59.856+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.858590+0000) 2019-10-13T09:01:59.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:01:59.856+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.858961+0000) 2019-10-13T09:01:59.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:01:59.856+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.858930+0000) 2019-10-13T09:01:59.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:01:59.856+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.859512+0000) 2019-10-13T09:01:59.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:01:59.859+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.861329+0000) 2019-10-13T09:01:59.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:01:59.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:01:59.863 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:01:59.860+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.862214+0000) 2019-10-13T09:01:59.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:01:59.860+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.862549+0000) 2019-10-13T09:01:59.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:01:59.863+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:01:59.868045+0000) 2019-10-13T09:02:01.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:01.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.639736+0000) 2019-10-13T09:02:01.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:01.639+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:01.641077+0000) 2019-10-13T09:02:03.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:03.952+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:03.953945+0000) 2019-10-13T09:02:04.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:04.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.364605+0000) 2019-10-13T09:02:04.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:04.860+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.865402+0000) 2019-10-13T09:02:04.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:04.864+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.866343+0000) 2019-10-13T09:02:04.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:04.864+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.868832+0000) 2019-10-13T09:02:04.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:04.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:04.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:04.868+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.870002+0000) 2019-10-13T09:02:04.871 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:04.868+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.870467+0000) 2019-10-13T09:02:04.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:04.866+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.870952+0000) 2019-10-13T09:02:04.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:04.868+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.870857+0000) 2019-10-13T09:02:04.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:04.870+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:04.875015+0000) 2019-10-13T09:02:06.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:06.394+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:06.395320+0000) 2019-10-13T09:02:07.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:07.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:07.493930+0000) 2019-10-13T09:02:07.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:07.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:07.591485+0000) 2019-10-13T09:02:07.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:07.616+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:07.617556+0000) 2019-10-13T09:02:09.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:09.868+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.873180+0000) 2019-10-13T09:02:09.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:09.872+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.874232+0000) 2019-10-13T09:02:09.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:09.872+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.874552+0000) 2019-10-13T09:02:09.875 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:09.872+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.874774+0000) 2019-10-13T09:02:09.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:09.872+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.875813+0000) 2019-10-13T09:02:09.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:09.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:09.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:09.874+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.876472+0000) 2019-10-13T09:02:09.878 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:09.872+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.877499+0000) 2019-10-13T09:02:09.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:09.874+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:09.879696+0000) 2019-10-13T09:02:11.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:11.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.639928+0000) 2019-10-13T09:02:11.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:11.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:11.641274+0000) 2019-10-13T09:02:13.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:13.952+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:13.954143+0000) 2019-10-13T09:02:14.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:14.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.364806+0000) 2019-10-13T09:02:14.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:14.880+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.883138+0000) 2019-10-13T09:02:14.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:14.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:14.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:14.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.883350+0000) 2019-10-13T09:02:14.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:14.880+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.884168+0000) 2019-10-13T09:02:14.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:14.880+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.885400+0000) 2019-10-13T09:02:14.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:14.884+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.885733+0000) 2019-10-13T09:02:14.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:14.884+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.886909+0000) 2019-10-13T09:02:14.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:14.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.887245+0000) 2019-10-13T09:02:15.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:14.894+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:14.896499+0000) 2019-10-13T09:02:16.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:16.398+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:16.401802+0000) 2019-10-13T09:02:17.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:17.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:17.494136+0000) 2019-10-13T09:02:17.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:17.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:17.591690+0000) 2019-10-13T09:02:17.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:17.616+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:17.617773+0000) 2019-10-13T09:02:18.539 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:02:18.539 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:18.543 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:02:18.543 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:19.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:19.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.886795+0000) 2019-10-13T09:02:19.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:19.884+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.888076+0000) 2019-10-13T09:02:19.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:19.886+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.888347+0000) 2019-10-13T09:02:19.889 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:19.884+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.889026+0000) 2019-10-13T09:02:19.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:19.888+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.889795+0000) 2019-10-13T09:02:19.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:19.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:19.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:19.888+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.890175+0000) 2019-10-13T09:02:19.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:19.888+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.891107+0000) 2019-10-13T09:02:19.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:19.890+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:19.893358+0000) 2019-10-13T09:02:21.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:21.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.640142+0000) 2019-10-13T09:02:21.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:21.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:21.641468+0000) 2019-10-13T09:02:23.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:23.952+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:23.954347+0000) 2019-10-13T09:02:24.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:24.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.365037+0000) 2019-10-13T09:02:24.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:24.888+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.893593+0000) 2019-10-13T09:02:24.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:24.892+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.893879+0000) 2019-10-13T09:02:24.895 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:24.892+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.895343+0000) 2019-10-13T09:02:24.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:24.890+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.895471+0000) 2019-10-13T09:02:24.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:24.892+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.896342+0000) 2019-10-13T09:02:24.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:24.892+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.896735+0000) 2019-10-13T09:02:24.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:24.892+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.897385+0000) 2019-10-13T09:02:24.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:24.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:24.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:24.898+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:24.902419+0000) 2019-10-13T09:02:26.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:26.402+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:26.408009+0000) 2019-10-13T09:02:27.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:27.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:27.494369+0000) 2019-10-13T09:02:27.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:27.588+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:27.591931+0000) 2019-10-13T09:02:27.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:27.616+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:27.618021+0000) 2019-10-13T09:02:29.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:29.898+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.901085+0000) 2019-10-13T09:02:29.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:29.900+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.902116+0000) 2019-10-13T09:02:29.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:29.900+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.902637+0000) 2019-10-13T09:02:29.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:29.900+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.903225+0000) 2019-10-13T09:02:29.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:29.900+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.903781+0000) 2019-10-13T09:02:29.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:29.900+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.904590+0000) 2019-10-13T09:02:29.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:29.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:29.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:29.900+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.905396+0000) 2019-10-13T09:02:29.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:29.906+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:29.908540+0000) 2019-10-13T09:02:31.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:31.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.640335+0000) 2019-10-13T09:02:31.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:31.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:31.641666+0000) 2019-10-13T09:02:33.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:33.951+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:33.954543+0000) 2019-10-13T09:02:34.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:34.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.365238+0000) 2019-10-13T09:02:34.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:34.903+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.908798+0000) 2019-10-13T09:02:34.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:34.903+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.909022+0000) 2019-10-13T09:02:34.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:34.903+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.909520+0000) 2019-10-13T09:02:34.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:34.907+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.910324+0000) 2019-10-13T09:02:34.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:34.906+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.910184+0000) 2019-10-13T09:02:34.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:34.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86051 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:34.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:34.907+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.911366+0000) 2019-10-13T09:02:34.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:34.907+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.911962+0000) 2019-10-13T09:02:34.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:34.910+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:34.914248+0000) 2019-10-13T09:02:36.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:36.414+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:36.414669+0000) 2019-10-13T09:02:37.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:37.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:37.494581+0000) 2019-10-13T09:02:37.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:37.591+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:37.592132+0000) 2019-10-13T09:02:37.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:37.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:37.618221+0000) 2019-10-13T09:02:39.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:39.910+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.914147+0000) 2019-10-13T09:02:39.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:39.911+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.915805+0000) 2019-10-13T09:02:39.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:39.911+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.915868+0000) 2019-10-13T09:02:39.917 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:39.911+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.916518+0000) 2019-10-13T09:02:39.917 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:39.911+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.916940+0000) 2019-10-13T09:02:39.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:39.911+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.917161+0000) 2019-10-13T09:02:39.918 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:39.911+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.917439+0000) 2019-10-13T09:02:39.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:39.915+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:39.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:39.918+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:39.921744+0000) 2019-10-13T09:02:41.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:41.635+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.640534+0000) 2019-10-13T09:02:41.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:41.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:41.641895+0000) 2019-10-13T09:02:43.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:43.951+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:43.954750+0000) 2019-10-13T09:02:44.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:44.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.365437+0000) 2019-10-13T09:02:44.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:44.919+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.922564+0000) 2019-10-13T09:02:44.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:44.919+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.923191+0000) 2019-10-13T09:02:44.924 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:44.919+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.923831+0000) 2019-10-13T09:02:44.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:44.918+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.923953+0000) 2019-10-13T09:02:44.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:44.919+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.925243+0000) 2019-10-13T09:02:44.926 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:44.919+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.924925+0000) 2019-10-13T09:02:44.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:44.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.924988+0000) 2019-10-13T09:02:44.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:44.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:44.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:44.926+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:44.928135+0000) 2019-10-13T09:02:46.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:46.418+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:46.421279+0000) 2019-10-13T09:02:47.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:47.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:47.494777+0000) 2019-10-13T09:02:47.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:47.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:47.592324+0000) 2019-10-13T09:02:47.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:47.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:47.618412+0000) 2019-10-13T09:02:48.678 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:02:48.679 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:48.682 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:02:48.683 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:02:49.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:49.923+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.929330+0000) 2019-10-13T09:02:49.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:49.927+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.931561+0000) 2019-10-13T09:02:49.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:49.927+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:49.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:49.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.933511+0000) 2019-10-13T09:02:49.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:49.931+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.933859+0000) 2019-10-13T09:02:49.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:49.931+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.934543+0000) 2019-10-13T09:02:49.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:49.930+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.934637+0000) 2019-10-13T09:02:49.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:49.931+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.935256+0000) 2019-10-13T09:02:49.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:49.934+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:49.939326+0000) 2019-10-13T09:02:51.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:51.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.640765+0000) 2019-10-13T09:02:51.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:51.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:51.642118+0000) 2019-10-13T09:02:53.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:53.951+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:53.954952+0000) 2019-10-13T09:02:54.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:54.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.365644+0000) 2019-10-13T09:02:54.936 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:54.931+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.936261+0000) 2019-10-13T09:02:54.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:54.931+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.937186+0000) 2019-10-13T09:02:54.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:54.931+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.937373+0000) 2019-10-13T09:02:54.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:54.935+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.938219+0000) 2019-10-13T09:02:54.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:54.935+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.939974+0000) 2019-10-13T09:02:54.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:54.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:54.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:54.938+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.941406+0000) 2019-10-13T09:02:54.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:54.939+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.942356+0000) 2019-10-13T09:02:54.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:54.942+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:54.946973+0000) 2019-10-13T09:02:56.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:56.426+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:56.427789+0000) 2019-10-13T09:02:57.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:57.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:57.494983+0000) 2019-10-13T09:02:57.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:57.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:57.592541+0000) 2019-10-13T09:02:57.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:57.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:57.618609+0000) 2019-10-13T09:02:59.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:02:59.939+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.944406+0000) 2019-10-13T09:02:59.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:02:59.943+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.946961+0000) 2019-10-13T09:02:59.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:02:59.942+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.946676+0000) 2019-10-13T09:02:59.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:02:59.943+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.947796+0000) 2019-10-13T09:02:59.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:02:59.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:02:59.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:02:59.943+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.949334+0000) 2019-10-13T09:02:59.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:02:59.947+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.949833+0000) 2019-10-13T09:02:59.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:02:59.947+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.950803+0000) 2019-10-13T09:02:59.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:02:59.954+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:02:59.957098+0000) 2019-10-13T09:03:01.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:01.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.640964+0000) 2019-10-13T09:03:01.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:01.638+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:01.642319+0000) 2019-10-13T09:03:03.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:03.951+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:03.955146+0000) 2019-10-13T09:03:04.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:04.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.365856+0000) 2019-10-13T09:03:04.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:04.951+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.953875+0000) 2019-10-13T09:03:04.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:04.951+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.954314+0000) 2019-10-13T09:03:04.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:04.951+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.955007+0000) 2019-10-13T09:03:04.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:04.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:04.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:04.951+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.955376+0000) 2019-10-13T09:03:04.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:04.951+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.955722+0000) 2019-10-13T09:03:04.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:04.950+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.955818+0000) 2019-10-13T09:03:04.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:04.951+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.956698+0000) 2019-10-13T09:03:04.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:04.958+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:04.962294+0000) 2019-10-13T09:03:06.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:06.430+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:06.433980+0000) 2019-10-13T09:03:07.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:07.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:07.495205+0000) 2019-10-13T09:03:07.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:07.587+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:07.592766+0000) 2019-10-13T09:03:07.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:07.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:07.618824+0000) 2019-10-13T09:03:09.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:09.955+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.958440+0000) 2019-10-13T09:03:09.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:09.955+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.958908+0000) 2019-10-13T09:03:09.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:09.955+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.959561+0000) 2019-10-13T09:03:09.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:09.955+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.959756+0000) 2019-10-13T09:03:09.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:09.955+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.960012+0000) 2019-10-13T09:03:09.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:09.957+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.960409+0000) 2019-10-13T09:03:09.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:09.955+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.961159+0000) 2019-10-13T09:03:09.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:09.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:09.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:09.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:09.969792+0000) 2019-10-13T09:03:11.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:11.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.641163+0000) 2019-10-13T09:03:11.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:11.641+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:11.642509+0000) 2019-10-13T09:03:13.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:13.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:13.955341+0000) 2019-10-13T09:03:14.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:14.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.366029+0000) 2019-10-13T09:03:14.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:14.959+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.964631+0000) 2019-10-13T09:03:14.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:14.959+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.964899+0000) 2019-10-13T09:03:14.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:14.963+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.965951+0000) 2019-10-13T09:03:14.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:14.963+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.966355+0000) 2019-10-13T09:03:14.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:14.963+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.966639+0000) 2019-10-13T09:03:14.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:14.963+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.967722+0000) 2019-10-13T09:03:14.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:14.961+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.967556+0000) 2019-10-13T09:03:14.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:14.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:14.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:14.969+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:14.974723+0000) 2019-10-13T09:03:16.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:16.437+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:16.440641+0000) 2019-10-13T09:03:17.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:17.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:17.495427+0000) 2019-10-13T09:03:17.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:17.591+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:17.592972+0000) 2019-10-13T09:03:17.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:17.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:17.619016+0000) 2019-10-13T09:03:18.763 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:03:18.763 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:18.767 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:03:18.767 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:19.972 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:19.967+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.971552+0000) 2019-10-13T09:03:19.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:19.967+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.972381+0000) 2019-10-13T09:03:19.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:19.971+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.974354+0000) 2019-10-13T09:03:19.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:19.969+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.974152+0000) 2019-10-13T09:03:19.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:19.971+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.975368+0000) 2019-10-13T09:03:19.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:19.971+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.975957+0000) 2019-10-13T09:03:19.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:19.971+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.976315+0000) 2019-10-13T09:03:19.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:19.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:19.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:19.977+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:19.983904+0000) 2019-10-13T09:03:21.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:21.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.641386+0000) 2019-10-13T09:03:21.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:21.641+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:21.642736+0000) 2019-10-13T09:03:23.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:23.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:23.955546+0000) 2019-10-13T09:03:24.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:24.363+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.366184+0000) 2019-10-13T09:03:24.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:24.975+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.979996+0000) 2019-10-13T09:03:24.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:24.977+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.982222+0000) 2019-10-13T09:03:24.983 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:24.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.982982+0000) 2019-10-13T09:03:24.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:24.979+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.983399+0000) 2019-10-13T09:03:24.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:24.979+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.983733+0000) 2019-10-13T09:03:24.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:24.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:24.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:24.979+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.984060+0000) 2019-10-13T09:03:24.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:24.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.984676+0000) 2019-10-13T09:03:24.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:24.989+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:24.992165+0000) 2019-10-13T09:03:26.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:26.445+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:26.447190+0000) 2019-10-13T09:03:27.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:27.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:27.495633+0000) 2019-10-13T09:03:27.593 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:27.591+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:27.593196+0000) 2019-10-13T09:03:27.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:27.615+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:27.619222+0000) 2019-10-13T09:03:29.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:29.982+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.987511+0000) 2019-10-13T09:03:29.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:29.986+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.990052+0000) 2019-10-13T09:03:29.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:29.986+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.990389+0000) 2019-10-13T09:03:29.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:29.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:29.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:29.986+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.990857+0000) 2019-10-13T09:03:29.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:29.986+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.990990+0000) 2019-10-13T09:03:29.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:29.985+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.991766+0000) 2019-10-13T09:03:29.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:29.986+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.991777+0000) 2019-10-13T09:03:29.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:29.993+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:29.997586+0000) 2019-10-13T09:03:31.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:31.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.641609+0000) 2019-10-13T09:03:31.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:31.637+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:31.642935+0000) 2019-10-13T09:03:33.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:33.950+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:33.955770+0000) 2019-10-13T09:03:34.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:34.362+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.366366+0000) 2019-10-13T09:03:34.995 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:34.990+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.994514+0000) 2019-10-13T09:03:34.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:34.990+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.996120+0000) 2019-10-13T09:03:34.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:34.990+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.996583+0000) 2019-10-13T09:03:34.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:34.990+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.997292+0000) 2019-10-13T09:03:34.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:34.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:34.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:34.994+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.998335+0000) 2019-10-13T09:03:34.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:34.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.999197+0000) 2019-10-13T09:03:35.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:34.993+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:34.999277+0000) 2019-10-13T09:03:35.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:35.001+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:35.004345+0000) 2019-10-13T09:03:36.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:36.449+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:36.453741+0000) 2019-10-13T09:03:37.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:37.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:37.495896+0000) 2019-10-13T09:03:37.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:37.586+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:37.593409+0000) 2019-10-13T09:03:37.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:37.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:37.619421+0000) 2019-10-13T09:03:40.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:39.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.002099+0000) 2019-10-13T09:03:40.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:39.998+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.003027+0000) 2019-10-13T09:03:40.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:39.998+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.003260+0000) 2019-10-13T09:03:40.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:39.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.004721+0000) 2019-10-13T09:03:40.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:39.998+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.005164+0000) 2019-10-13T09:03:40.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:39.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:40.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:39.998+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.005553+0000) 2019-10-13T09:03:40.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:40.002+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.006281+0000) 2019-10-13T09:03:40.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:40.005+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:40.008355+0000) 2019-10-13T09:03:41.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:41.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.641799+0000) 2019-10-13T09:03:41.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:41.641+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:41.643137+0000) 2019-10-13T09:03:43.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:43.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:43.955974+0000) 2019-10-13T09:03:44.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:44.362+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:44.366572+0000) 2019-10-13T09:03:45.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:45.002+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.008811+0000) 2019-10-13T09:03:45.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:45.006+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.009841+0000) 2019-10-13T09:03:45.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:45.005+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.009744+0000) 2019-10-13T09:03:45.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:45.006+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.011728+0000) 2019-10-13T09:03:45.014 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:45.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:45.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:45.006+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.012458+0000) 2019-10-13T09:03:45.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:45.006+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.012613+0000) 2019-10-13T09:03:45.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:45.006+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.013049+0000) 2019-10-13T09:03:45.016 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:45.009+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:45.015129+0000) 2019-10-13T09:03:46.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:46.457+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:46.460271+0000) 2019-10-13T09:03:47.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:47.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:47.496158+0000) 2019-10-13T09:03:47.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:47.590+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:47.593624+0000) 2019-10-13T09:03:47.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:47.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:47.619664+0000) 2019-10-13T09:03:48.885 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:03:48.885 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:48.889 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:03:48.889 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:03:50.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:50.010+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.016395+0000) 2019-10-13T09:03:50.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:50.013+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.016882+0000) 2019-10-13T09:03:50.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:50.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.017841+0000) 2019-10-13T09:03:50.019 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:50.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:50.019 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:50.014+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.019063+0000) 2019-10-13T09:03:50.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:50.014+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.020132+0000) 2019-10-13T09:03:50.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:50.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.019903+0000) 2019-10-13T09:03:50.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:50.014+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.020815+0000) 2019-10-13T09:03:50.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:50.017+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:50.022797+0000) 2019-10-13T09:03:51.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:51.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.642027+0000) 2019-10-13T09:03:51.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:51.641+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:51.643343+0000) 2019-10-13T09:03:53.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:53.950+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:53.956181+0000) 2019-10-13T09:03:54.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:54.362+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:54.366759+0000) 2019-10-13T09:03:55.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:03:55.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.023077+0000) 2019-10-13T09:03:55.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:03:55.018+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.023377+0000) 2019-10-13T09:03:55.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:03:55.017+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.023226+0000) 2019-10-13T09:03:55.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:03:55.018+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.024392+0000) 2019-10-13T09:03:55.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:55.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.024409+0000) 2019-10-13T09:03:55.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:55.018+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.025121+0000) 2019-10-13T09:03:55.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:03:55.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:03:55.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:55.022+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.026646+0000) 2019-10-13T09:03:55.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:55.025+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:55.031991+0000) 2019-10-13T09:03:56.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:03:56.465+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:56.467007+0000) 2019-10-13T09:03:57.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:03:57.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:57.496355+0000) 2019-10-13T09:03:57.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:03:57.590+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:57.593859+0000) 2019-10-13T09:03:57.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:03:57.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:03:57.619868+0000) 2019-10-13T09:04:00.030 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:00.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.029841+0000) 2019-10-13T09:04:00.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:00.025+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.029335+0000) 2019-10-13T09:04:00.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:00.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.030728+0000) 2019-10-13T09:04:00.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:00.026+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.030862+0000) 2019-10-13T09:04:00.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:00.026+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.031174+0000) 2019-10-13T09:04:00.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:00.026+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.031573+0000) 2019-10-13T09:04:00.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:00.026+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.032118+0000) 2019-10-13T09:04:00.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:00.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:00.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:00.033+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:00.036314+0000) 2019-10-13T09:04:01.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:01.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.642244+0000) 2019-10-13T09:04:01.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:01.641+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:01.643564+0000) 2019-10-13T09:04:03.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:03.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:03.956432+0000) 2019-10-13T09:04:04.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:04.366+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:04.366956+0000) 2019-10-13T09:04:05.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:05.033+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.037144+0000) 2019-10-13T09:04:05.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:05.034+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.038093+0000) 2019-10-13T09:04:05.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:05.034+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.039071+0000) 2019-10-13T09:04:05.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:05.034+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.039652+0000) 2019-10-13T09:04:05.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:05.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.040320+0000) 2019-10-13T09:04:05.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:05.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:05.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:05.034+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.040888+0000) 2019-10-13T09:04:05.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:05.034+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.041581+0000) 2019-10-13T09:04:05.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:05.041+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:05.045329+0000) 2019-10-13T09:04:06.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:06.473+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:06.473612+0000) 2019-10-13T09:04:07.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:07.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:07.496557+0000) 2019-10-13T09:04:07.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:07.590+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:07.594059+0000) 2019-10-13T09:04:07.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:07.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:07.620068+0000) 2019-10-13T09:04:10.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:10.038+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.043511+0000) 2019-10-13T09:04:10.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:10.038+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.044135+0000) 2019-10-13T09:04:10.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:10.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.044249+0000) 2019-10-13T09:04:10.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:10.041+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.044301+0000) 2019-10-13T09:04:10.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:10.038+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.045351+0000) 2019-10-13T09:04:10.046 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:10.038+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.045690+0000) 2019-10-13T09:04:10.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:10.042+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.047241+0000) 2019-10-13T09:04:10.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:10.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:10.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:10.049+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:10.052402+0000) 2019-10-13T09:04:11.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:11.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.642449+0000) 2019-10-13T09:04:11.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:11.637+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:11.643753+0000) 2019-10-13T09:04:13.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:13.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:13.956589+0000) 2019-10-13T09:04:14.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:14.366+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:14.367161+0000) 2019-10-13T09:04:15.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:15.046+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.051494+0000) 2019-10-13T09:04:15.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:15.046+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.051882+0000) 2019-10-13T09:04:15.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:15.044+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.051932+0000) 2019-10-13T09:04:15.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:15.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.053605+0000) 2019-10-13T09:04:15.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:15.050+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.053985+0000) 2019-10-13T09:04:15.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:15.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:15.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:15.050+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.055349+0000) 2019-10-13T09:04:15.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:15.050+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.056434+0000) 2019-10-13T09:04:15.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:15.056+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:15.061450+0000) 2019-10-13T09:04:16.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:16.476+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:16.480219+0000) 2019-10-13T09:04:17.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:17.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:17.496790+0000) 2019-10-13T09:04:17.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:17.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:17.594284+0000) 2019-10-13T09:04:17.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:17.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:17.620290+0000) 2019-10-13T09:04:18.995 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:04:18.995 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:18.999 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:04:19.000 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:20.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:20.052+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.058356+0000) 2019-10-13T09:04:20.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:20.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.059253+0000) 2019-10-13T09:04:20.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:20.054+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.059474+0000) 2019-10-13T09:04:20.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:20.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.060379+0000) 2019-10-13T09:04:20.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:20.054+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.060627+0000) 2019-10-13T09:04:20.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:20.054+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.060942+0000) 2019-10-13T09:04:20.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:20.058+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.061896+0000) 2019-10-13T09:04:20.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:20.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:20.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:20.056+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:20.063814+0000) 2019-10-13T09:04:21.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:21.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.642662+0000) 2019-10-13T09:04:21.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:21.640+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:21.643976+0000) 2019-10-13T09:04:23.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:23.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:23.956846+0000) 2019-10-13T09:04:24.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:24.362+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:24.367357+0000) 2019-10-13T09:04:25.067 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:25.061+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.066684+0000) 2019-10-13T09:04:25.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:25.061+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.067043+0000) 2019-10-13T09:04:25.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:25.064+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.068493+0000) 2019-10-13T09:04:25.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:25.061+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.069319+0000) 2019-10-13T09:04:25.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:25.061+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.069694+0000) 2019-10-13T09:04:25.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:25.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:25.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:25.065+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.072909+0000) 2019-10-13T09:04:25.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:25.069+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.074083+0000) 2019-10-13T09:04:25.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:25.072+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:25.077232+0000) 2019-10-13T09:04:26.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:26.484+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:26.486631+0000) 2019-10-13T09:04:27.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:27.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:27.496994+0000) 2019-10-13T09:04:27.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:27.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:27.594482+0000) 2019-10-13T09:04:27.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:27.617+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:27.620491+0000) 2019-10-13T09:04:30.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:30.069+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.073722+0000) 2019-10-13T09:04:30.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:30.069+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.074500+0000) 2019-10-13T09:04:30.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:30.069+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.074821+0000) 2019-10-13T09:04:30.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:30.068+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.074759+0000) 2019-10-13T09:04:30.076 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:30.069+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.075995+0000) 2019-10-13T09:04:30.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:30.069+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.076959+0000) 2019-10-13T09:04:30.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:30.069+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.077370+0000) 2019-10-13T09:04:30.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:30.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:30.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:30.076+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:30.082720+0000) 2019-10-13T09:04:31.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:31.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.642852+0000) 2019-10-13T09:04:31.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:31.640+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:31.644182+0000) 2019-10-13T09:04:33.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:33.953+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:33.957069+0000) 2019-10-13T09:04:34.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:34.361+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:34.367568+0000) 2019-10-13T09:04:35.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:35.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.081557+0000) 2019-10-13T09:04:35.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:35.077+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.082560+0000) 2019-10-13T09:04:35.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:35.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.083120+0000) 2019-10-13T09:04:35.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:35.076+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.083486+0000) 2019-10-13T09:04:35.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:35.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:35.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:35.077+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.084614+0000) 2019-10-13T09:04:35.085 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:35.077+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.085187+0000) 2019-10-13T09:04:35.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:35.081+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.085776+0000) 2019-10-13T09:04:35.093 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:35.088+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:35.092718+0000) 2019-10-13T09:04:36.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:36.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:36.493130+0000) 2019-10-13T09:04:37.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:37.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:37.497191+0000) 2019-10-13T09:04:37.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:37.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:37.594678+0000) 2019-10-13T09:04:37.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:37.617+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:37.620693+0000) 2019-10-13T09:04:40.089 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:40.085+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.088274+0000) 2019-10-13T09:04:40.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:40.085+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.088829+0000) 2019-10-13T09:04:40.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:40.085+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.089624+0000) 2019-10-13T09:04:40.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:40.089+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.090261+0000) 2019-10-13T09:04:40.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:40.084+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.090297+0000) 2019-10-13T09:04:40.091 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:40.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86837 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:40.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:40.089+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.091469+0000) 2019-10-13T09:04:40.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:40.089+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.092010+0000) 2019-10-13T09:04:40.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:40.092+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:40.097994+0000) 2019-10-13T09:04:41.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:41.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.643047+0000) 2019-10-13T09:04:41.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:41.640+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:41.644409+0000) 2019-10-13T09:04:43.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:43.953+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:43.957302+0000) 2019-10-13T09:04:44.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:44.365+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:44.367766+0000) 2019-10-13T09:04:45.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:45.088+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.093673+0000) 2019-10-13T09:04:45.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:45.093+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.094873+0000) 2019-10-13T09:04:45.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:45.093+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.096822+0000) 2019-10-13T09:04:45.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:45.093+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.097086+0000) 2019-10-13T09:04:45.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:45.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.097687+0000) 2019-10-13T09:04:45.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:45.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:45.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:45.097+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.098619+0000) 2019-10-13T09:04:45.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:45.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.098980+0000) 2019-10-13T09:04:45.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:45.096+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:45.103793+0000) 2019-10-13T09:04:46.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:46.496+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:46.499612+0000) 2019-10-13T09:04:47.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:47.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:47.497389+0000) 2019-10-13T09:04:47.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:47.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:47.594877+0000) 2019-10-13T09:04:47.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:47.617+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:47.620900+0000) 2019-10-13T09:04:49.075 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:04:49.075 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:49.079 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:04:49.079 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:04:50.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:50.101+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.102518+0000) 2019-10-13T09:04:50.103 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:50.101+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.102768+0000) 2019-10-13T09:04:50.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:50.096+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.102967+0000) 2019-10-13T09:04:50.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:50.101+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.104167+0000) 2019-10-13T09:04:50.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:50.101+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.104423+0000) 2019-10-13T09:04:50.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:50.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:50.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:50.101+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.104665+0000) 2019-10-13T09:04:50.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:50.101+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.104921+0000) 2019-10-13T09:04:50.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:50.104+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:50.108561+0000) 2019-10-13T09:04:51.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:51.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.643240+0000) 2019-10-13T09:04:51.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:51.640+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:51.644574+0000) 2019-10-13T09:04:53.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:53.953+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:53.957465+0000) 2019-10-13T09:04:54.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:54.365+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:54.367993+0000) 2019-10-13T09:04:55.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:04:55.104+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.108233+0000) 2019-10-13T09:04:55.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:04:55.105+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.107930+0000) 2019-10-13T09:04:55.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:04:55.105+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.108444+0000) 2019-10-13T09:04:55.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:55.105+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.109224+0000) 2019-10-13T09:04:55.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:55.109+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.109855+0000) 2019-10-13T09:04:55.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:55.109+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.110213+0000) 2019-10-13T09:04:55.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:04:55.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.110483+0000) 2019-10-13T09:04:55.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:04:55.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:04:55.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:55.112+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:55.116952+0000) 2019-10-13T09:04:56.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:04:56.504+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:56.506245+0000) 2019-10-13T09:04:57.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:04:57.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:57.497586+0000) 2019-10-13T09:04:57.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:04:57.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:57.595069+0000) 2019-10-13T09:04:57.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:04:57.617+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:04:57.621116+0000) 2019-10-13T09:05:00.115 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:00.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.114489+0000) 2019-10-13T09:05:00.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:00.113+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.115837+0000) 2019-10-13T09:05:00.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:00.113+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.116082+0000) 2019-10-13T09:05:00.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:00.113+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.116765+0000) 2019-10-13T09:05:00.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:00.113+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.117506+0000) 2019-10-13T09:05:00.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:00.117+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.118343+0000) 2019-10-13T09:05:00.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:00.117+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.118935+0000) 2019-10-13T09:05:00.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:00.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:00.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:00.120+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:00.125526+0000) 2019-10-13T09:05:01.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:01.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.643441+0000) 2019-10-13T09:05:01.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:01.644+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:01.644784+0000) 2019-10-13T09:05:03.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:03.953+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:03.957681+0000) 2019-10-13T09:05:04.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:04.365+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:04.368211+0000) 2019-10-13T09:05:05.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:05.121+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.122318+0000) 2019-10-13T09:05:05.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:05.121+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.123584+0000) 2019-10-13T09:05:05.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:05.121+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.124061+0000) 2019-10-13T09:05:05.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:05.121+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.124378+0000) 2019-10-13T09:05:05.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:05.120+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.124585+0000) 2019-10-13T09:05:05.125 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:05.121+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.125369+0000) 2019-10-13T09:05:05.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:05.125+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.125798+0000) 2019-10-13T09:05:05.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:05.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:05.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:05.128+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:05.132218+0000) 2019-10-13T09:05:06.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:06.512+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:06.512817+0000) 2019-10-13T09:05:07.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:07.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:07.497787+0000) 2019-10-13T09:05:07.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:07.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:07.595229+0000) 2019-10-13T09:05:07.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:07.617+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:07.621332+0000) 2019-10-13T09:05:10.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:10.129+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.129848+0000) 2019-10-13T09:05:10.131 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:10.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.130313+0000) 2019-10-13T09:05:10.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:10.129+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.130833+0000) 2019-10-13T09:05:10.131 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:10.124+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.130758+0000) 2019-10-13T09:05:10.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:10.129+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.133671+0000) 2019-10-13T09:05:10.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:10.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:10.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:10.133+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.134826+0000) 2019-10-13T09:05:10.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:10.133+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.135243+0000) 2019-10-13T09:05:10.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:10.132+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:10.137623+0000) 2019-10-13T09:05:11.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:11.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.643638+0000) 2019-10-13T09:05:11.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:11.640+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:11.645016+0000) 2019-10-13T09:05:13.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:13.957+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:13.957913+0000) 2019-10-13T09:05:14.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:14.365+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:14.368432+0000) 2019-10-13T09:05:15.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:15.133+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.137198+0000) 2019-10-13T09:05:15.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:15.137+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.138015+0000) 2019-10-13T09:05:15.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:15.137+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.138742+0000) 2019-10-13T09:05:15.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:15.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.139162+0000) 2019-10-13T09:05:15.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:15.136+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.139056+0000) 2019-10-13T09:05:15.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:15.137+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.140111+0000) 2019-10-13T09:05:15.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:15.137+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.140912+0000) 2019-10-13T09:05:15.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:15.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:15.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:15.144+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:15.145902+0000) 2019-10-13T09:05:16.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:16.516+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:16.519297+0000) 2019-10-13T09:05:17.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:17.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:17.497984+0000) 2019-10-13T09:05:17.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:17.593+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:17.595428+0000) 2019-10-13T09:05:17.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:17.621+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:17.621535+0000) 2019-10-13T09:05:19.196 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:05:19.196 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:19.200 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:05:19.200 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:20.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:20.141+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.143964+0000) 2019-10-13T09:05:20.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:20.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.144287+0000) 2019-10-13T09:05:20.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:20.141+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.144914+0000) 2019-10-13T09:05:20.147 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:20.145+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.147270+0000) 2019-10-13T09:05:20.148 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:20.145+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.148238+0000) 2019-10-13T09:05:20.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:20.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:20.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:20.148+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.149766+0000) 2019-10-13T09:05:20.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:20.149+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.150725+0000) 2019-10-13T09:05:20.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:20.156+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:20.159266+0000) 2019-10-13T09:05:21.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:21.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.643849+0000) 2019-10-13T09:05:21.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:21.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:21.645213+0000) 2019-10-13T09:05:23.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:23.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:23.958141+0000) 2019-10-13T09:05:24.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:24.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:24.368665+0000) 2019-10-13T09:05:25.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:25.152+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.154590+0000) 2019-10-13T09:05:25.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:25.151+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.154721+0000) 2019-10-13T09:05:25.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:25.152+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.155620+0000) 2019-10-13T09:05:25.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:25.152+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.155944+0000) 2019-10-13T09:05:25.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:25.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.157027+0000) 2019-10-13T09:05:25.158 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:25.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:25.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:25.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.157382+0000) 2019-10-13T09:05:25.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:25.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.158395+0000) 2019-10-13T09:05:25.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:25.159+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:25.162920+0000) 2019-10-13T09:05:26.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:26.523+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:26.525682+0000) 2019-10-13T09:05:27.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:27.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:27.498199+0000) 2019-10-13T09:05:27.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:27.592+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:27.595623+0000) 2019-10-13T09:05:27.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:27.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:27.621727+0000) 2019-10-13T09:05:30.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:30.156+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.160463+0000) 2019-10-13T09:05:30.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:30.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.161301+0000) 2019-10-13T09:05:30.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:30.156+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.161684+0000) 2019-10-13T09:05:30.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:30.160+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.161956+0000) 2019-10-13T09:05:30.164 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:30.160+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.163494+0000) 2019-10-13T09:05:30.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:30.160+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.163578+0000) 2019-10-13T09:05:30.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:30.159+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.162671+0000) 2019-10-13T09:05:30.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:30.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:30.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:30.167+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:30.170990+0000) 2019-10-13T09:05:31.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:31.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.644057+0000) 2019-10-13T09:05:31.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:31.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:31.645408+0000) 2019-10-13T09:05:33.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:33.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:33.958373+0000) 2019-10-13T09:05:34.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:34.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:34.368895+0000) 2019-10-13T09:05:35.168 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:35.164+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.167625+0000) 2019-10-13T09:05:35.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:35.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.169382+0000) 2019-10-13T09:05:35.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:35.168+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.169893+0000) 2019-10-13T09:05:35.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:35.168+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.171046+0000) 2019-10-13T09:05:35.171 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:35.167+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.170857+0000) 2019-10-13T09:05:35.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:35.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87184 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:35.172 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:35.168+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.171992+0000) 2019-10-13T09:05:35.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:35.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.173105+0000) 2019-10-13T09:05:35.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:35.179+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:35.181610+0000) 2019-10-13T09:05:36.533 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:36.531+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:36.532119+0000) 2019-10-13T09:05:37.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:37.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:37.498398+0000) 2019-10-13T09:05:37.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:37.592+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:37.595851+0000) 2019-10-13T09:05:37.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:37.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:37.621995+0000) 2019-10-13T09:05:40.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:40.171+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.174909+0000) 2019-10-13T09:05:40.180 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:40.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.175058+0000) 2019-10-13T09:05:40.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:40.172+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.175948+0000) 2019-10-13T09:05:40.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:40.172+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.176293+0000) 2019-10-13T09:05:40.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:40.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.178571+0000) 2019-10-13T09:05:40.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:40.176+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.178927+0000) 2019-10-13T09:05:40.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:40.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:40.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:40.176+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.179639+0000) 2019-10-13T09:05:40.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:40.183+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:40.184786+0000) 2019-10-13T09:05:41.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:41.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.644253+0000) 2019-10-13T09:05:41.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:41.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:41.645593+0000) 2019-10-13T09:05:43.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:43.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:43.958572+0000) 2019-10-13T09:05:44.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:44.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:44.369100+0000) 2019-10-13T09:05:45.183 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:45.180+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.182904+0000) 2019-10-13T09:05:45.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:45.180+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.183217+0000) 2019-10-13T09:05:45.184 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:45.179+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.183270+0000) 2019-10-13T09:05:45.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:45.180+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.184241+0000) 2019-10-13T09:05:45.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:45.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.185013+0000) 2019-10-13T09:05:45.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:45.180+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.185124+0000) 2019-10-13T09:05:45.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:45.184+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.186074+0000) 2019-10-13T09:05:45.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:45.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87247 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:45.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:45.191+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:45.192731+0000) 2019-10-13T09:05:46.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:46.535+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:46.538634+0000) 2019-10-13T09:05:47.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:47.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:47.498600+0000) 2019-10-13T09:05:47.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:47.592+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:47.596048+0000) 2019-10-13T09:05:47.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:47.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:47.622192+0000) 2019-10-13T09:05:49.303 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:05:49.303 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:49.307 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:05:49.307 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:05:50.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:50.187+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.189458+0000) 2019-10-13T09:05:50.192 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:50.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.192378+0000) 2019-10-13T09:05:50.193 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:50.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.192737+0000) 2019-10-13T09:05:50.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:50.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:50.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:50.188+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.193629+0000) 2019-10-13T09:05:50.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:50.192+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.194037+0000) 2019-10-13T09:05:50.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:50.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.194598+0000) 2019-10-13T09:05:50.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:50.192+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.195251+0000) 2019-10-13T09:05:50.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:50.195+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:50.197796+0000) 2019-10-13T09:05:51.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:51.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.644494+0000) 2019-10-13T09:05:51.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:51.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:51.645793+0000) 2019-10-13T09:05:53.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:53.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:53.958775+0000) 2019-10-13T09:05:54.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:54.364+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:54.369301+0000) 2019-10-13T09:05:55.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:55.192+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.196825+0000) 2019-10-13T09:05:55.197 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:05:55.192+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.197269+0000) 2019-10-13T09:05:55.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:05:55.192+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.197613+0000) 2019-10-13T09:05:55.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:05:55.196+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.198598+0000) 2019-10-13T09:05:55.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:05:55.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:05:55.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:05:55.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.200169+0000) 2019-10-13T09:05:55.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:55.196+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.200956+0000) 2019-10-13T09:05:55.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:55.200+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.201789+0000) 2019-10-13T09:05:55.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:55.207+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:55.208484+0000) 2019-10-13T09:05:56.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:05:56.543+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:56.545127+0000) 2019-10-13T09:05:57.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:05:57.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:57.498799+0000) 2019-10-13T09:05:57.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:05:57.592+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:57.596270+0000) 2019-10-13T09:05:57.623 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:05:57.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:05:57.622392+0000) 2019-10-13T09:06:00.203 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:00.200+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.203135+0000) 2019-10-13T09:06:00.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:00.200+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.204431+0000) 2019-10-13T09:06:00.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:00.200+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.205249+0000) 2019-10-13T09:06:00.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:00.204+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.205775+0000) 2019-10-13T09:06:00.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:00.203+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.205460+0000) 2019-10-13T09:06:00.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:00.204+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.206346+0000) 2019-10-13T09:06:00.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:00.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.206808+0000) 2019-10-13T09:06:00.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:00.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:00.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:00.211+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:00.213301+0000) 2019-10-13T09:06:01.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:01.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.644706+0000) 2019-10-13T09:06:01.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:01.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:01.645986+0000) 2019-10-13T09:06:03.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:03.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:03.959008+0000) 2019-10-13T09:06:04.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:04.368+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:04.369517+0000) 2019-10-13T09:06:05.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:05.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.210783+0000) 2019-10-13T09:06:05.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:05.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.211306+0000) 2019-10-13T09:06:05.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:05.208+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.211654+0000) 2019-10-13T09:06:05.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:05.208+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.213436+0000) 2019-10-13T09:06:05.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:05.212+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.213949+0000) 2019-10-13T09:06:05.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:05.212+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.214324+0000) 2019-10-13T09:06:05.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:05.211+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.214044+0000) 2019-10-13T09:06:05.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:05.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:05.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:05.219+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:05.221145+0000) 2019-10-13T09:06:06.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:06.551+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:06.551652+0000) 2019-10-13T09:06:07.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:07.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:07.498995+0000) 2019-10-13T09:06:07.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:07.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:07.596423+0000) 2019-10-13T09:06:07.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:07.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:07.622550+0000) 2019-10-13T09:06:10.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:10.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.218284+0000) 2019-10-13T09:06:10.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:10.216+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.218587+0000) 2019-10-13T09:06:10.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:10.216+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.219311+0000) 2019-10-13T09:06:10.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:10.216+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.219055+0000) 2019-10-13T09:06:10.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:10.216+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.220139+0000) 2019-10-13T09:06:10.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:10.219+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.220322+0000) 2019-10-13T09:06:10.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:10.216+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.221370+0000) 2019-10-13T09:06:10.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:10.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:10.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:10.223+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:10.226019+0000) 2019-10-13T09:06:11.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:11.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.644898+0000) 2019-10-13T09:06:11.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:11.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:11.646209+0000) 2019-10-13T09:06:13.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:13.956+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:13.959243+0000) 2019-10-13T09:06:14.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:14.368+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:14.369778+0000) 2019-10-13T09:06:15.226 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:15.220+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.225214+0000) 2019-10-13T09:06:15.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:15.224+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.226008+0000) 2019-10-13T09:06:15.227 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:15.224+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.226462+0000) 2019-10-13T09:06:15.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:15.224+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.226954+0000) 2019-10-13T09:06:15.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:15.224+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.229081+0000) 2019-10-13T09:06:15.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:15.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:15.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:15.227+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.228687+0000) 2019-10-13T09:06:15.232 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:15.228+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.230265+0000) 2019-10-13T09:06:15.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:15.231+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:15.233384+0000) 2019-10-13T09:06:16.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:16.555+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:16.558182+0000) 2019-10-13T09:06:17.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:17.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:17.499231+0000) 2019-10-13T09:06:17.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:17.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:17.596622+0000) 2019-10-13T09:06:17.623 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:17.620+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:17.622764+0000) 2019-10-13T09:06:19.425 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:06:19.426 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:19.450 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:06:19.450 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:20.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:20.227+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.231735+0000) 2019-10-13T09:06:20.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:20.227+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.233418+0000) 2019-10-13T09:06:20.234 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:20.231+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.234212+0000) 2019-10-13T09:06:20.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:20.231+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.234548+0000) 2019-10-13T09:06:20.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:20.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.235159+0000) 2019-10-13T09:06:20.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:20.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.235185+0000) 2019-10-13T09:06:20.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:20.231+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.235718+0000) 2019-10-13T09:06:20.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:20.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:20.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:20.239+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:20.241983+0000) 2019-10-13T09:06:21.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:21.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.645140+0000) 2019-10-13T09:06:21.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:21.643+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:21.646422+0000) 2019-10-13T09:06:23.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:23.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:23.959435+0000) 2019-10-13T09:06:24.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:24.367+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:24.369979+0000) 2019-10-13T09:06:25.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:25.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.239372+0000) 2019-10-13T09:06:25.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:25.235+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.239580+0000) 2019-10-13T09:06:25.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:25.235+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.240349+0000) 2019-10-13T09:06:25.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:25.235+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.241040+0000) 2019-10-13T09:06:25.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:25.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.242265+0000) 2019-10-13T09:06:25.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:25.239+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.242637+0000) 2019-10-13T09:06:25.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:25.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:25.244 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:25.239+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.243570+0000) 2019-10-13T09:06:25.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:25.243+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:25.248008+0000) 2019-10-13T09:06:26.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:26.562+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:26.564505+0000) 2019-10-13T09:06:27.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:27.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:27.499430+0000) 2019-10-13T09:06:27.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:27.591+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:27.596851+0000) 2019-10-13T09:06:27.623 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:27.619+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:27.622964+0000) 2019-10-13T09:06:30.247 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:30.243+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.246982+0000) 2019-10-13T09:06:30.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:30.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.247740+0000) 2019-10-13T09:06:30.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:30.243+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.248863+0000) 2019-10-13T09:06:30.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:30.243+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.249241+0000) 2019-10-13T09:06:30.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:30.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:30.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:30.246+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.250596+0000) 2019-10-13T09:06:30.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:30.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.251148+0000) 2019-10-13T09:06:30.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:30.247+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.250654+0000) 2019-10-13T09:06:30.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:30.247+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:30.253018+0000) 2019-10-13T09:06:31.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:31.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.645366+0000) 2019-10-13T09:06:31.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:31.642+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:31.646577+0000) 2019-10-13T09:06:33.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:33.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:33.959649+0000) 2019-10-13T09:06:34.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:34.367+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:34.370184+0000) 2019-10-13T09:06:35.253 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:35.247+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.253072+0000) 2019-10-13T09:06:35.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:35.250+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.253033+0000) 2019-10-13T09:06:35.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:35.251+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.254425+0000) 2019-10-13T09:06:35.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:35.251+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.255010+0000) 2019-10-13T09:06:35.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:35.251+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.255148+0000) 2019-10-13T09:06:35.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:35.251+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.255512+0000) 2019-10-13T09:06:35.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:35.251+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.255890+0000) 2019-10-13T09:06:35.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:35.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:35.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:35.258+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:35.260376+0000) 2019-10-13T09:06:36.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:36.566+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:36.570997+0000) 2019-10-13T09:06:37.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:37.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:37.499658+0000) 2019-10-13T09:06:37.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:37.591+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:37.597008+0000) 2019-10-13T09:06:37.623 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:37.619+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:37.623157+0000) 2019-10-13T09:06:40.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:40.255+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.260023+0000) 2019-10-13T09:06:40.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:40.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.260626+0000) 2019-10-13T09:06:40.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:40.259+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.262626+0000) 2019-10-13T09:06:40.263 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:40.258+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.262690+0000) 2019-10-13T09:06:40.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:40.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:40.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:40.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.263914+0000) 2019-10-13T09:06:40.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:40.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.264523+0000) 2019-10-13T09:06:40.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:40.259+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.264713+0000) 2019-10-13T09:06:40.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:40.266+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:40.270605+0000) 2019-10-13T09:06:41.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:41.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.645595+0000) 2019-10-13T09:06:41.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:41.642+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:41.646802+0000) 2019-10-13T09:06:43.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:43.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:43.959856+0000) 2019-10-13T09:06:44.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:44.367+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:44.370416+0000) 2019-10-13T09:06:45.267 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:45.263+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.266875+0000) 2019-10-13T09:06:45.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:45.263+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.267249+0000) 2019-10-13T09:06:45.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:45.266+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.269197+0000) 2019-10-13T09:06:45.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:45.267+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.269920+0000) 2019-10-13T09:06:45.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:45.267+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.270569+0000) 2019-10-13T09:06:45.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:45.267+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.271208+0000) 2019-10-13T09:06:45.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:45.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:45.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:45.267+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.271969+0000) 2019-10-13T09:06:45.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:45.270+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:45.273959+0000) 2019-10-13T09:06:46.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:46.574+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:46.576997+0000) 2019-10-13T09:06:47.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:47.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:47.499853+0000) 2019-10-13T09:06:47.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:47.595+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:47.597210+0000) 2019-10-13T09:06:47.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:47.619+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:47.623355+0000) 2019-10-13T09:06:49.529 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:06:49.529 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:49.533 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:06:49.533 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:06:50.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:50.270+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.275298+0000) 2019-10-13T09:06:50.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:50.271+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.277272+0000) 2019-10-13T09:06:50.278 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:50.271+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.277541+0000) 2019-10-13T09:06:50.279 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:50.275+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.278814+0000) 2019-10-13T09:06:50.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:50.275+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.279067+0000) 2019-10-13T09:06:50.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:50.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:50.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:50.275+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.279965+0000) 2019-10-13T09:06:50.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:50.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.280511+0000) 2019-10-13T09:06:50.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:50.282+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:50.285346+0000) 2019-10-13T09:06:51.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:51.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.645817+0000) 2019-10-13T09:06:51.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:51.646+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:51.647018+0000) 2019-10-13T09:06:53.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:53.959+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:53.960116+0000) 2019-10-13T09:06:54.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:54.367+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:54.370618+0000) 2019-10-13T09:06:55.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:06:55.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.283515+0000) 2019-10-13T09:06:55.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:55.279+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.285268+0000) 2019-10-13T09:06:55.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:06:55.283+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.285833+0000) 2019-10-13T09:06:55.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:55.283+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.286491+0000) 2019-10-13T09:06:55.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:06:55.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:06:55.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:55.283+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.286947+0000) 2019-10-13T09:06:55.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:06:55.282+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.287189+0000) 2019-10-13T09:06:55.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:06:55.283+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.288377+0000) 2019-10-13T09:06:55.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:55.290+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:55.293590+0000) 2019-10-13T09:06:56.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:06:56.582+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:56.583617+0000) 2019-10-13T09:06:57.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:06:57.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:57.500055+0000) 2019-10-13T09:06:57.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:06:57.595+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:57.597442+0000) 2019-10-13T09:06:57.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:06:57.623+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:06:57.623558+0000) 2019-10-13T09:07:00.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:00.287+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.290132+0000) 2019-10-13T09:07:00.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:00.287+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.290600+0000) 2019-10-13T09:07:00.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:00.287+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.291272+0000) 2019-10-13T09:07:00.292 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:00.287+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.291987+0000) 2019-10-13T09:07:00.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:00.287+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.292626+0000) 2019-10-13T09:07:00.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:00.290+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.293220+0000) 2019-10-13T09:07:00.294 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:00.291+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.293818+0000) 2019-10-13T09:07:00.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:00.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:00.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:00.294+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:00.298710+0000) 2019-10-13T09:07:01.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:01.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.645993+0000) 2019-10-13T09:07:01.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:01.646+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:01.647181+0000) 2019-10-13T09:07:03.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:03.955+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:03.960345+0000) 2019-10-13T09:07:04.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:04.367+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:04.370821+0000) 2019-10-13T09:07:05.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:05.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.298105+0000) 2019-10-13T09:07:05.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:05.295+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.298646+0000) 2019-10-13T09:07:05.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:05.294+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.298633+0000) 2019-10-13T09:07:05.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:05.295+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.299794+0000) 2019-10-13T09:07:05.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:05.295+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.301667+0000) 2019-10-13T09:07:05.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:05.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:05.303 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:05.299+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.302061+0000) 2019-10-13T09:07:05.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:05.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.303116+0000) 2019-10-13T09:07:05.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:05.302+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:05.305450+0000) 2019-10-13T09:07:06.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:06.586+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:06.590199+0000) 2019-10-13T09:07:07.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:07.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:07.500279+0000) 2019-10-13T09:07:07.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:07.595+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:07.597640+0000) 2019-10-13T09:07:07.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:07.619+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:07.623754+0000) 2019-10-13T09:07:10.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:10.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.304999+0000) 2019-10-13T09:07:10.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:10.299+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.305112+0000) 2019-10-13T09:07:10.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:10.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.306565+0000) 2019-10-13T09:07:10.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:10.303+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.307131+0000) 2019-10-13T09:07:10.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:10.303+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.307416+0000) 2019-10-13T09:07:10.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:10.302+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.307621+0000) 2019-10-13T09:07:10.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:10.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:10.310 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:10.303+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.309492+0000) 2019-10-13T09:07:10.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:10.314+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:10.317821+0000) 2019-10-13T09:07:11.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:11.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.646190+0000) 2019-10-13T09:07:11.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:11.646+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:11.647377+0000) 2019-10-13T09:07:13.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:13.959+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:13.960546+0000) 2019-10-13T09:07:14.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:14.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:14.371079+0000) 2019-10-13T09:07:15.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:15.306+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.312440+0000) 2019-10-13T09:07:15.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:15.306+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.313651+0000) 2019-10-13T09:07:15.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:15.310+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.313949+0000) 2019-10-13T09:07:15.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:15.310+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.315229+0000) 2019-10-13T09:07:15.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:15.310+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.315913+0000) 2019-10-13T09:07:15.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:15.310+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.316281+0000) 2019-10-13T09:07:15.317 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:15.314+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.316126+0000) 2019-10-13T09:07:15.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:15.310+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:15.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:15.314+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:15.319961+0000) 2019-10-13T09:07:16.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:16.594+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:16.596689+0000) 2019-10-13T09:07:17.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:17.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:17.500528+0000) 2019-10-13T09:07:17.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:17.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:17.597842+0000) 2019-10-13T09:07:17.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:17.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:17.623951+0000) 2019-10-13T09:07:19.655 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:07:19.655 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:19.659 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:07:19.659 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:20.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:20.314+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.320307+0000) 2019-10-13T09:07:20.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:20.314+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.320638+0000) 2019-10-13T09:07:20.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:20.314+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.321558+0000) 2019-10-13T09:07:20.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:20.318+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.321815+0000) 2019-10-13T09:07:20.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:20.318+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.322655+0000) 2019-10-13T09:07:20.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:20.318+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.323101+0000) 2019-10-13T09:07:20.324 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:20.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:20.324 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:20.318+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.323934+0000) 2019-10-13T09:07:20.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:20.322+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:20.328055+0000) 2019-10-13T09:07:21.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:21.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.646385+0000) 2019-10-13T09:07:21.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:21.646+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:21.647600+0000) 2019-10-13T09:07:23.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:23.958+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:23.960751+0000) 2019-10-13T09:07:24.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:24.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:24.371295+0000) 2019-10-13T09:07:25.327 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:25.322+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.326091+0000) 2019-10-13T09:07:25.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:25.322+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.327707+0000) 2019-10-13T09:07:25.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:25.322+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.328696+0000) 2019-10-13T09:07:25.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:25.322+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.329180+0000) 2019-10-13T09:07:25.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:25.326+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.330401+0000) 2019-10-13T09:07:25.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:25.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:25.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:25.326+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.330999+0000) 2019-10-13T09:07:25.332 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:25.326+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.331143+0000) 2019-10-13T09:07:25.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:25.330+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:25.333645+0000) 2019-10-13T09:07:26.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:26.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:26.603150+0000) 2019-10-13T09:07:27.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:27.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:27.500746+0000) 2019-10-13T09:07:27.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:27.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:27.598037+0000) 2019-10-13T09:07:27.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:27.622+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:27.624152+0000) 2019-10-13T09:07:30.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:30.330+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.333839+0000) 2019-10-13T09:07:30.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:30.330+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.335112+0000) 2019-10-13T09:07:30.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:30.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.335729+0000) 2019-10-13T09:07:30.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:30.330+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.335512+0000) 2019-10-13T09:07:30.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:30.330+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.336326+0000) 2019-10-13T09:07:30.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:30.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.336928+0000) 2019-10-13T09:07:30.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:30.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:30.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:30.330+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.337567+0000) 2019-10-13T09:07:30.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:30.338+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:30.340336+0000) 2019-10-13T09:07:31.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:31.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.646537+0000) 2019-10-13T09:07:31.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:31.642+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:31.647788+0000) 2019-10-13T09:07:33.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:33.958+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:33.960962+0000) 2019-10-13T09:07:34.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:34.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:34.371473+0000) 2019-10-13T09:07:35.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:35.338+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.342671+0000) 2019-10-13T09:07:35.344 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:35.338+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.342956+0000) 2019-10-13T09:07:35.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:35.338+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.344057+0000) 2019-10-13T09:07:35.345 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:35.337+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.343975+0000) 2019-10-13T09:07:35.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:35.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:35.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:35.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.345312+0000) 2019-10-13T09:07:35.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:35.338+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.345504+0000) 2019-10-13T09:07:35.347 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:35.342+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.345797+0000) 2019-10-13T09:07:35.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:35.345+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:35.348137+0000) 2019-10-13T09:07:36.611 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:36.605+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:36.609860+0000) 2019-10-13T09:07:37.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:37.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:37.500961+0000) 2019-10-13T09:07:37.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:37.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:37.598242+0000) 2019-10-13T09:07:37.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:37.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:37.624352+0000) 2019-10-13T09:07:40.349 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:40.342+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.348447+0000) 2019-10-13T09:07:40.349 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:40.345+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.348642+0000) 2019-10-13T09:07:40.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:40.342+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.349465+0000) 2019-10-13T09:07:40.350 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:40.346+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.350069+0000) 2019-10-13T09:07:40.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:40.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.350724+0000) 2019-10-13T09:07:40.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:40.346+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.350568+0000) 2019-10-13T09:07:40.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:40.346+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.351294+0000) 2019-10-13T09:07:40.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:40.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:40.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:40.349+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:40.353897+0000) 2019-10-13T09:07:41.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:41.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.646780+0000) 2019-10-13T09:07:41.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:41.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:41.647987+0000) 2019-10-13T09:07:43.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:43.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:43.961166+0000) 2019-10-13T09:07:44.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:44.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:44.371670+0000) 2019-10-13T09:07:45.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:45.350+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.355622+0000) 2019-10-13T09:07:45.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:45.350+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.356250+0000) 2019-10-13T09:07:45.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:45.353+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.356310+0000) 2019-10-13T09:07:45.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:45.350+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.357044+0000) 2019-10-13T09:07:45.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:45.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.357411+0000) 2019-10-13T09:07:45.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:45.354+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.357895+0000) 2019-10-13T09:07:45.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:45.354+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.358782+0000) 2019-10-13T09:07:45.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:45.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:45.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:45.357+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:45.361645+0000) 2019-10-13T09:07:46.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:46.613+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:46.616358+0000) 2019-10-13T09:07:47.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:47.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:47.501184+0000) 2019-10-13T09:07:47.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:47.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:47.598442+0000) 2019-10-13T09:07:47.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:47.622+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:47.624548+0000) 2019-10-13T09:07:49.771 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:07:49.771 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:49.778 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:07:49.778 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:07:50.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:50.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.362744+0000) 2019-10-13T09:07:50.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:50.358+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.363457+0000) 2019-10-13T09:07:50.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:50.358+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.364190+0000) 2019-10-13T09:07:50.365 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:50.358+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.364385+0000) 2019-10-13T09:07:50.365 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:50.358+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.364666+0000) 2019-10-13T09:07:50.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:50.358+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.365706+0000) 2019-10-13T09:07:50.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:50.361+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.365499+0000) 2019-10-13T09:07:50.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:50.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:50.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:50.369+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:50.373315+0000) 2019-10-13T09:07:51.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:51.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.646973+0000) 2019-10-13T09:07:51.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:51.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:51.648179+0000) 2019-10-13T09:07:53.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:53.958+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:53.961362+0000) 2019-10-13T09:07:54.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:54.366+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:54.371882+0000) 2019-10-13T09:07:55.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:07:55.366+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.369365+0000) 2019-10-13T09:07:55.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:07:55.366+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.369941+0000) 2019-10-13T09:07:55.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:07:55.366+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.370906+0000) 2019-10-13T09:07:55.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:55.366+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.370331+0000) 2019-10-13T09:07:55.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:07:55.365+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.371227+0000) 2019-10-13T09:07:55.372 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:55.366+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.372464+0000) 2019-10-13T09:07:55.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:55.366+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.372910+0000) 2019-10-13T09:07:55.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:07:55.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:07:55.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:55.373+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:55.377120+0000) 2019-10-13T09:07:56.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:07:56.617+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:56.622858+0000) 2019-10-13T09:07:57.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:07:57.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:57.501378+0000) 2019-10-13T09:07:57.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:07:57.598+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:57.598639+0000) 2019-10-13T09:07:57.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:07:57.618+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:07:57.624751+0000) 2019-10-13T09:08:00.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:00.374+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.378397+0000) 2019-10-13T09:08:00.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:00.374+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.378558+0000) 2019-10-13T09:08:00.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:00.373+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.379095+0000) 2019-10-13T09:08:00.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:00.374+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.380039+0000) 2019-10-13T09:08:00.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:00.374+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.380309+0000) 2019-10-13T09:08:00.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:00.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:00.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:00.374+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.380990+0000) 2019-10-13T09:08:00.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:00.378+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.381892+0000) 2019-10-13T09:08:00.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:00.385+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:00.388502+0000) 2019-10-13T09:08:01.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:01.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.647200+0000) 2019-10-13T09:08:01.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:01.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:01.648398+0000) 2019-10-13T09:08:03.962 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:03.954+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:03.961561+0000) 2019-10-13T09:08:04.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:04.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:04.372087+0000) 2019-10-13T09:08:05.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:05.378+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.383946+0000) 2019-10-13T09:08:05.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:05.382+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.385802+0000) 2019-10-13T09:08:05.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:05.382+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.386221+0000) 2019-10-13T09:08:05.387 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:05.381+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.386669+0000) 2019-10-13T09:08:05.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:05.382+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.387775+0000) 2019-10-13T09:08:05.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:05.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88129 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:05.389 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:05.382+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.388197+0000) 2019-10-13T09:08:05.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:05.382+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.388896+0000) 2019-10-13T09:08:05.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:05.389+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:05.392904+0000) 2019-10-13T09:08:06.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:06.625+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:06.629396+0000) 2019-10-13T09:08:07.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:07.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:07.501588+0000) 2019-10-13T09:08:07.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:07.594+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:07.598859+0000) 2019-10-13T09:08:07.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:07.622+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:07.624973+0000) 2019-10-13T09:08:10.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:10.386+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.391689+0000) 2019-10-13T09:08:10.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:10.385+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.391618+0000) 2019-10-13T09:08:10.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:10.386+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.392448+0000) 2019-10-13T09:08:10.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:10.386+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.392904+0000) 2019-10-13T09:08:10.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:10.386+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.393131+0000) 2019-10-13T09:08:10.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:10.390+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.394234+0000) 2019-10-13T09:08:10.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:10.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.394406+0000) 2019-10-13T09:08:10.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:10.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:10.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:10.397+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:10.403575+0000) 2019-10-13T09:08:11.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:11.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.647401+0000) 2019-10-13T09:08:11.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:11.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:11.648594+0000) 2019-10-13T09:08:13.962 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:13.957+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:13.961784+0000) 2019-10-13T09:08:14.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:14.369+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:14.372311+0000) 2019-10-13T09:08:15.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:15.393+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.398902+0000) 2019-10-13T09:08:15.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:15.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.399541+0000) 2019-10-13T09:08:15.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:15.393+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.401264+0000) 2019-10-13T09:08:15.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:15.393+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.401541+0000) 2019-10-13T09:08:15.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:15.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:15.405 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:15.397+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.402409+0000) 2019-10-13T09:08:15.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:15.397+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.403088+0000) 2019-10-13T09:08:15.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:15.397+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.403614+0000) 2019-10-13T09:08:15.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:15.405+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:15.409464+0000) 2019-10-13T09:08:16.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:16.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:16.635744+0000) 2019-10-13T09:08:17.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:17.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:17.501787+0000) 2019-10-13T09:08:17.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:17.597+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:17.599057+0000) 2019-10-13T09:08:17.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:17.621+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:17.625208+0000) 2019-10-13T09:08:19.905 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:08:19.905 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:19.909 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:08:19.909 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:20.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:20.401+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.405212+0000) 2019-10-13T09:08:20.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:20.401+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.405983+0000) 2019-10-13T09:08:20.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:20.401+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.406576+0000) 2019-10-13T09:08:20.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:20.401+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.406852+0000) 2019-10-13T09:08:20.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:20.401+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.408203+0000) 2019-10-13T09:08:20.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:20.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:20.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:20.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.409384+0000) 2019-10-13T09:08:20.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:20.401+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.409648+0000) 2019-10-13T09:08:20.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:20.413+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:20.416254+0000) 2019-10-13T09:08:21.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:21.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.647597+0000) 2019-10-13T09:08:21.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:21.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:21.648833+0000) 2019-10-13T09:08:23.962 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:23.961+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:23.962017+0000) 2019-10-13T09:08:24.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:24.369+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:24.372512+0000) 2019-10-13T09:08:25.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:25.405+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.412628+0000) 2019-10-13T09:08:25.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:25.405+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.412908+0000) 2019-10-13T09:08:25.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:25.409+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.413976+0000) 2019-10-13T09:08:25.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:25.409+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.414780+0000) 2019-10-13T09:08:25.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:25.409+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.415278+0000) 2019-10-13T09:08:25.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:25.409+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.415554+0000) 2019-10-13T09:08:25.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:25.409+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.415720+0000) 2019-10-13T09:08:25.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:25.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:25.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:25.417+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:25.420826+0000) 2019-10-13T09:08:26.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:26.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:26.642256+0000) 2019-10-13T09:08:27.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:27.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:27.501984+0000) 2019-10-13T09:08:27.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:27.597+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:27.599287+0000) 2019-10-13T09:08:27.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:27.621+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:27.625395+0000) 2019-10-13T09:08:30.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:30.417+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.420170+0000) 2019-10-13T09:08:30.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:30.417+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.420814+0000) 2019-10-13T09:08:30.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:30.421+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.422768+0000) 2019-10-13T09:08:30.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:30.421+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.423027+0000) 2019-10-13T09:08:30.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:30.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:30.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:30.421+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.424243+0000) 2019-10-13T09:08:30.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:30.421+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.425231+0000) 2019-10-13T09:08:30.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:30.425+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.428056+0000) 2019-10-13T09:08:30.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:30.425+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:30.429947+0000) 2019-10-13T09:08:31.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:31.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.647780+0000) 2019-10-13T09:08:31.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:31.645+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:31.649066+0000) 2019-10-13T09:08:33.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:33.961+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:33.962248+0000) 2019-10-13T09:08:34.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:34.369+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:34.372723+0000) 2019-10-13T09:08:35.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:35.425+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.427180+0000) 2019-10-13T09:08:35.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:35.425+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.427600+0000) 2019-10-13T09:08:35.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:35.425+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.429064+0000) 2019-10-13T09:08:35.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:35.425+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.429560+0000) 2019-10-13T09:08:35.430 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:35.425+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.429689+0000) 2019-10-13T09:08:35.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:35.425+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.429929+0000) 2019-10-13T09:08:35.431 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:35.429+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:35.432 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:35.429+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.431037+0000) 2019-10-13T09:08:35.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:35.429+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:35.434656+0000) 2019-10-13T09:08:36.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:36.645+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:36.648646+0000) 2019-10-13T09:08:37.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:37.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:37.502176+0000) 2019-10-13T09:08:37.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:37.597+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:37.599488+0000) 2019-10-13T09:08:37.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:37.625+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:37.625630+0000) 2019-10-13T09:08:40.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:40.433+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.434620+0000) 2019-10-13T09:08:40.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:40.433+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.435275+0000) 2019-10-13T09:08:40.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:40.433+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.435694+0000) 2019-10-13T09:08:40.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:40.433+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.436034+0000) 2019-10-13T09:08:40.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:40.433+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.437089+0000) 2019-10-13T09:08:40.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:40.432+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.437051+0000) 2019-10-13T09:08:40.439 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:40.437+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.437868+0000) 2019-10-13T09:08:40.439 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:40.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:40.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:40.440+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:40.445620+0000) 2019-10-13T09:08:41.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:41.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.647988+0000) 2019-10-13T09:08:41.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:41.644+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:41.649265+0000) 2019-10-13T09:08:43.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:43.961+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:43.962483+0000) 2019-10-13T09:08:44.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:44.369+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:44.372922+0000) 2019-10-13T09:08:45.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:45.437+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.441396+0000) 2019-10-13T09:08:45.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:45.436+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.441799+0000) 2019-10-13T09:08:45.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:45.441+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.441780+0000) 2019-10-13T09:08:45.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:45.441+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.443285+0000) 2019-10-13T09:08:45.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:45.441+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.443699+0000) 2019-10-13T09:08:45.445 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:45.441+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.444228+0000) 2019-10-13T09:08:45.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:45.441+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.444477+0000) 2019-10-13T09:08:45.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:45.441+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:45.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:45.444+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:45.450652+0000) 2019-10-13T09:08:46.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:46.652+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:46.655320+0000) 2019-10-13T09:08:47.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:47.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:47.502376+0000) 2019-10-13T09:08:47.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:47.597+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:47.599685+0000) 2019-10-13T09:08:47.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:47.625+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:47.625854+0000) 2019-10-13T09:08:49.951 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:08:49.951 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:49.955 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:08:49.955 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:08:50.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:50.445+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.448803+0000) 2019-10-13T09:08:50.450 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:50.449+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.449925+0000) 2019-10-13T09:08:50.451 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:50.444+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.450468+0000) 2019-10-13T09:08:50.453 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:50.449+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.452994+0000) 2019-10-13T09:08:50.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:50.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:50.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:50.449+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.453261+0000) 2019-10-13T09:08:50.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:50.453+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.454141+0000) 2019-10-13T09:08:50.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:50.453+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.454030+0000) 2019-10-13T09:08:50.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:50.448+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:50.456077+0000) 2019-10-13T09:08:51.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:51.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.648184+0000) 2019-10-13T09:08:51.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:51.648+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:51.649461+0000) 2019-10-13T09:08:53.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:53.961+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:53.962701+0000) 2019-10-13T09:08:54.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:54.369+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:54.373126+0000) 2019-10-13T09:08:55.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:55.453+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.456340+0000) 2019-10-13T09:08:55.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:08:55.453+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.456895+0000) 2019-10-13T09:08:55.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:55.457+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.458468+0000) 2019-10-13T09:08:55.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:55.457+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.458702+0000) 2019-10-13T09:08:55.460 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:08:55.452+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.458669+0000) 2019-10-13T09:08:55.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:08:55.457+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.458926+0000) 2019-10-13T09:08:55.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:08:55.457+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.459564+0000) 2019-10-13T09:08:55.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:08:55.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:08:55.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:55.460+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:55.465786+0000) 2019-10-13T09:08:56.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:08:56.656+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:56.661965+0000) 2019-10-13T09:08:57.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:08:57.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:57.502600+0000) 2019-10-13T09:08:57.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:08:57.597+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:57.599882+0000) 2019-10-13T09:08:57.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:08:57.625+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:08:57.626088+0000) 2019-10-13T09:09:00.465 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:00.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.464166+0000) 2019-10-13T09:09:00.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:00.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.465182+0000) 2019-10-13T09:09:00.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:00.460+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.465026+0000) 2019-10-13T09:09:00.466 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:00.465+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.466109+0000) 2019-10-13T09:09:00.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:00.465+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.465770+0000) 2019-10-13T09:09:00.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:00.465+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.467125+0000) 2019-10-13T09:09:00.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:00.465+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.467535+0000) 2019-10-13T09:09:00.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:00.465+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:00.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:00.472+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:00.477104+0000) 2019-10-13T09:09:01.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:01.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.648406+0000) 2019-10-13T09:09:01.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:01.644+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:01.649673+0000) 2019-10-13T09:09:03.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:03.961+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:03.962944+0000) 2019-10-13T09:09:04.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:04.373+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:04.373330+0000) 2019-10-13T09:09:05.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:05.469+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.471998+0000) 2019-10-13T09:09:05.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:05.464+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.471880+0000) 2019-10-13T09:09:05.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:05.469+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.472630+0000) 2019-10-13T09:09:05.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:05.473+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.474680+0000) 2019-10-13T09:09:05.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:05.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:05.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:05.473+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.476068+0000) 2019-10-13T09:09:05.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:05.473+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.476943+0000) 2019-10-13T09:09:05.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:05.473+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.477205+0000) 2019-10-13T09:09:05.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:05.480+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:05.484155+0000) 2019-10-13T09:09:06.669 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:06.664+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:06.668508+0000) 2019-10-13T09:09:07.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:07.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:07.502815+0000) 2019-10-13T09:09:07.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:07.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:07.600100+0000) 2019-10-13T09:09:07.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:07.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:07.626305+0000) 2019-10-13T09:09:10.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:10.476+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.478761+0000) 2019-10-13T09:09:10.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:10.472+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.479034+0000) 2019-10-13T09:09:10.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:10.476+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.479863+0000) 2019-10-13T09:09:10.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:10.476+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.480418+0000) 2019-10-13T09:09:10.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:10.476+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.480755+0000) 2019-10-13T09:09:10.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:10.476+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.481647+0000) 2019-10-13T09:09:10.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:10.480+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.482314+0000) 2019-10-13T09:09:10.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:10.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:10.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:10.484+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:10.488425+0000) 2019-10-13T09:09:11.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:11.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.648606+0000) 2019-10-13T09:09:11.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:11.648+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:11.649901+0000) 2019-10-13T09:09:13.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:13.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:13.963147+0000) 2019-10-13T09:09:14.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:14.372+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:14.373538+0000) 2019-10-13T09:09:15.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:15.480+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.486692+0000) 2019-10-13T09:09:15.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:15.484+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.487645+0000) 2019-10-13T09:09:15.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:15.484+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.487942+0000) 2019-10-13T09:09:15.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:15.484+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.489238+0000) 2019-10-13T09:09:15.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:15.484+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.489574+0000) 2019-10-13T09:09:15.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:15.488+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.490418+0000) 2019-10-13T09:09:15.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:15.488+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.490268+0000) 2019-10-13T09:09:15.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:15.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:15.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:15.488+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:15.494048+0000) 2019-10-13T09:09:16.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:16.672+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:16.675109+0000) 2019-10-13T09:09:17.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:17.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:17.503067+0000) 2019-10-13T09:09:17.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:17.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:17.600296+0000) 2019-10-13T09:09:17.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:17.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:17.626514+0000) 2019-10-13T09:09:20.099 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:09:20.099 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:20.103 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:09:20.103 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:20.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:20.488+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.493761+0000) 2019-10-13T09:09:20.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:20.492+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.496287+0000) 2019-10-13T09:09:20.497 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:20.492+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.496541+0000) 2019-10-13T09:09:20.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:20.492+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.497203+0000) 2019-10-13T09:09:20.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:20.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:20.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:20.496+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.498190+0000) 2019-10-13T09:09:20.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:20.496+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.498908+0000) 2019-10-13T09:09:20.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:20.496+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.498659+0000) 2019-10-13T09:09:20.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:20.496+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:20.503571+0000) 2019-10-13T09:09:21.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:21.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.648830+0000) 2019-10-13T09:09:21.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:21.648+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:21.650102+0000) 2019-10-13T09:09:23.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:23.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:23.963343+0000) 2019-10-13T09:09:24.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:24.372+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:24.373768+0000) 2019-10-13T09:09:25.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:25.496+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.501668+0000) 2019-10-13T09:09:25.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:25.500+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.501788+0000) 2019-10-13T09:09:25.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:25.500+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.502521+0000) 2019-10-13T09:09:25.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:25.500+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.502991+0000) 2019-10-13T09:09:25.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:25.500+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.504686+0000) 2019-10-13T09:09:25.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:25.500+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.505234+0000) 2019-10-13T09:09:25.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:25.500+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.505236+0000) 2019-10-13T09:09:25.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:25.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:25.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:25.508+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:25.512418+0000) 2019-10-13T09:09:26.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:26.676+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:26.681776+0000) 2019-10-13T09:09:27.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:27.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:27.503271+0000) 2019-10-13T09:09:27.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:27.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:27.600492+0000) 2019-10-13T09:09:27.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:27.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:27.626723+0000) 2019-10-13T09:09:30.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:30.504+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.508382+0000) 2019-10-13T09:09:30.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:30.504+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.508842+0000) 2019-10-13T09:09:30.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:30.504+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.509562+0000) 2019-10-13T09:09:30.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:30.508+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.509945+0000) 2019-10-13T09:09:30.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:30.508+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.511046+0000) 2019-10-13T09:09:30.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:30.508+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.511380+0000) 2019-10-13T09:09:30.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:30.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:30.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:30.508+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.513184+0000) 2019-10-13T09:09:30.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:30.512+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:30.518386+0000) 2019-10-13T09:09:31.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:31.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.649044+0000) 2019-10-13T09:09:31.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:31.648+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:31.650305+0000) 2019-10-13T09:09:33.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:33.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:33.963563+0000) 2019-10-13T09:09:34.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:34.372+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:34.373940+0000) 2019-10-13T09:09:35.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:35.512+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.516511+0000) 2019-10-13T09:09:35.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:35.512+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.516724+0000) 2019-10-13T09:09:35.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:35.516+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.517857+0000) 2019-10-13T09:09:35.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:35.516+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.518325+0000) 2019-10-13T09:09:35.519 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:35.516+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.518586+0000) 2019-10-13T09:09:35.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:35.516+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.519162+0000) 2019-10-13T09:09:35.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:35.512+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.519360+0000) 2019-10-13T09:09:35.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:35.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:35.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:35.520+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:35.525308+0000) 2019-10-13T09:09:36.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:36.688+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:36.688301+0000) 2019-10-13T09:09:37.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:37.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:37.503492+0000) 2019-10-13T09:09:37.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:37.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:37.600705+0000) 2019-10-13T09:09:37.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:37.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:37.626925+0000) 2019-10-13T09:09:40.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:40.520+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.524044+0000) 2019-10-13T09:09:40.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:40.520+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.524360+0000) 2019-10-13T09:09:40.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:40.520+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.525630+0000) 2019-10-13T09:09:40.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:40.524+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.526310+0000) 2019-10-13T09:09:40.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:40.524+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.526545+0000) 2019-10-13T09:09:40.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:40.524+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.527039+0000) 2019-10-13T09:09:40.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:40.524+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.526971+0000) 2019-10-13T09:09:40.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:40.524+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:40.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:40.532+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:40.533106+0000) 2019-10-13T09:09:41.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:41.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.649269+0000) 2019-10-13T09:09:41.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:41.648+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:41.650528+0000) 2019-10-13T09:09:43.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:43.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:43.963758+0000) 2019-10-13T09:09:44.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:44.372+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:44.374141+0000) 2019-10-13T09:09:45.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:45.528+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.531771+0000) 2019-10-13T09:09:45.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:45.528+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.531961+0000) 2019-10-13T09:09:45.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:45.528+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.531567+0000) 2019-10-13T09:09:45.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:45.528+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.533113+0000) 2019-10-13T09:09:45.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:45.532+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.533985+0000) 2019-10-13T09:09:45.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:45.532+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.534191+0000) 2019-10-13T09:09:45.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:45.532+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88757 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:45.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:45.531+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.535781+0000) 2019-10-13T09:09:45.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:45.539+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:45.541433+0000) 2019-10-13T09:09:46.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:46.691+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:46.694912+0000) 2019-10-13T09:09:47.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:47.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:47.503692+0000) 2019-10-13T09:09:47.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:47.596+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:47.600902+0000) 2019-10-13T09:09:47.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:47.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:47.627137+0000) 2019-10-13T09:09:50.205 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:09:50.205 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:50.209 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:09:50.209 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:09:50.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:50.536+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.539152+0000) 2019-10-13T09:09:50.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:50.536+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.540004+0000) 2019-10-13T09:09:50.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:50.536+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.540582+0000) 2019-10-13T09:09:50.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:50.536+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.540740+0000) 2019-10-13T09:09:50.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:50.536+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.541349+0000) 2019-10-13T09:09:50.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:50.539+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.541205+0000) 2019-10-13T09:09:50.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:50.540+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.541995+0000) 2019-10-13T09:09:50.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:50.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:50.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:50.543+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:50.546114+0000) 2019-10-13T09:09:51.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:51.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.649454+0000) 2019-10-13T09:09:51.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:51.647+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:51.650746+0000) 2019-10-13T09:09:53.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:53.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:53.963956+0000) 2019-10-13T09:09:54.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:54.372+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:54.374347+0000) 2019-10-13T09:09:55.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:09:55.543+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.545853+0000) 2019-10-13T09:09:55.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:09:55.544+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.546857+0000) 2019-10-13T09:09:55.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:55.544+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.547900+0000) 2019-10-13T09:09:55.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:09:55.544+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.548235+0000) 2019-10-13T09:09:55.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:09:55.544+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.549070+0000) 2019-10-13T09:09:55.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:09:55.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:09:55.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:55.548+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.550607+0000) 2019-10-13T09:09:55.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:55.548+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.551309+0000) 2019-10-13T09:09:55.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:55.551+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:55.553729+0000) 2019-10-13T09:09:56.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:09:56.699+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:56.701058+0000) 2019-10-13T09:09:57.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:09:57.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:57.503881+0000) 2019-10-13T09:09:57.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:09:57.600+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:57.601104+0000) 2019-10-13T09:09:57.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:09:57.624+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:09:57.627363+0000) 2019-10-13T09:10:00.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:00.551+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.553139+0000) 2019-10-13T09:10:00.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:00.552+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.554617+0000) 2019-10-13T09:10:00.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:00.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.555300+0000) 2019-10-13T09:10:00.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:00.552+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.555423+0000) 2019-10-13T09:10:00.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:00.552+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.555738+0000) 2019-10-13T09:10:00.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:00.552+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.556609+0000) 2019-10-13T09:10:00.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:00.552+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.557277+0000) 2019-10-13T09:10:00.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:00.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:00.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:00.559+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:00.562123+0000) 2019-10-13T09:10:01.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:01.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.649659+0000) 2019-10-13T09:10:01.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:01.647+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:01.650966+0000) 2019-10-13T09:10:03.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:03.960+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:03.964156+0000) 2019-10-13T09:10:04.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:04.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:04.374608+0000) 2019-10-13T09:10:05.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:05.555+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.561275+0000) 2019-10-13T09:10:05.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:05.555+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.561420+0000) 2019-10-13T09:10:05.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:05.559+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.561893+0000) 2019-10-13T09:10:05.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:05.559+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.562810+0000) 2019-10-13T09:10:05.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:05.559+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.562707+0000) 2019-10-13T09:10:05.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:05.559+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.563477+0000) 2019-10-13T09:10:05.565 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:05.559+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.564944+0000) 2019-10-13T09:10:05.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:05.559+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:05.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:05.567+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:05.570052+0000) 2019-10-13T09:10:06.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:06.703+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:06.707575+0000) 2019-10-13T09:10:07.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:07.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:07.504087+0000) 2019-10-13T09:10:07.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:07.595+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:07.601339+0000) 2019-10-13T09:10:07.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:07.623+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:07.627573+0000) 2019-10-13T09:10:10.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:10.563+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.568758+0000) 2019-10-13T09:10:10.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:10.563+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.569361+0000) 2019-10-13T09:10:10.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:10.567+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.570017+0000) 2019-10-13T09:10:10.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:10.567+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.570460+0000) 2019-10-13T09:10:10.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:10.567+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.570244+0000) 2019-10-13T09:10:10.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:10.567+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.571023+0000) 2019-10-13T09:10:10.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:10.567+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.571910+0000) 2019-10-13T09:10:10.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:10.567+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:10.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:10.575+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:10.577141+0000) 2019-10-13T09:10:11.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:11.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.649892+0000) 2019-10-13T09:10:11.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:11.647+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:11.651155+0000) 2019-10-13T09:10:13.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:13.959+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:13.964384+0000) 2019-10-13T09:10:14.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:14.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:14.374845+0000) 2019-10-13T09:10:15.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:15.571+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.576627+0000) 2019-10-13T09:10:15.577 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:15.571+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.577190+0000) 2019-10-13T09:10:15.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:15.575+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.578717+0000) 2019-10-13T09:10:15.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:15.575+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.579558+0000) 2019-10-13T09:10:15.581 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:15.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:15.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:15.579+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.580353+0000) 2019-10-13T09:10:15.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:15.579+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.581872+0000) 2019-10-13T09:10:15.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:15.575+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.581342+0000) 2019-10-13T09:10:15.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:15.587+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:15.589739+0000) 2019-10-13T09:10:16.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:16.711+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:16.714121+0000) 2019-10-13T09:10:17.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:17.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:17.504327+0000) 2019-10-13T09:10:17.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:17.599+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:17.601541+0000) 2019-10-13T09:10:17.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:17.623+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:17.627809+0000) 2019-10-13T09:10:20.346 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:10:20.346 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:20.350 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:10:20.350 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:20.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:20.579+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.584564+0000) 2019-10-13T09:10:20.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:20.579+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.584936+0000) 2019-10-13T09:10:20.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:20.579+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.583908+0000) 2019-10-13T09:10:20.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:20.579+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.585645+0000) 2019-10-13T09:10:20.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:20.583+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.585984+0000) 2019-10-13T09:10:20.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:20.583+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.586456+0000) 2019-10-13T09:10:20.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:20.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:20.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:20.583+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.587943+0000) 2019-10-13T09:10:20.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:20.591+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:20.592645+0000) 2019-10-13T09:10:21.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:21.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.650086+0000) 2019-10-13T09:10:21.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:21.647+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:21.651353+0000) 2019-10-13T09:10:23.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:23.963+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:23.964580+0000) 2019-10-13T09:10:24.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:24.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:24.375059+0000) 2019-10-13T09:10:25.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:25.587+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.591374+0000) 2019-10-13T09:10:25.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:25.587+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.592189+0000) 2019-10-13T09:10:25.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:25.587+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.592730+0000) 2019-10-13T09:10:25.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:25.587+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.593151+0000) 2019-10-13T09:10:25.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:25.591+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.593055+0000) 2019-10-13T09:10:25.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:25.587+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.593688+0000) 2019-10-13T09:10:25.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:25.591+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.594044+0000) 2019-10-13T09:10:25.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:25.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:25.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:25.599+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:25.600284+0000) 2019-10-13T09:10:26.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:26.719+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:26.720746+0000) 2019-10-13T09:10:27.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:27.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:27.504554+0000) 2019-10-13T09:10:27.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:27.599+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:27.601761+0000) 2019-10-13T09:10:27.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:27.623+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:27.628012+0000) 2019-10-13T09:10:30.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:30.591+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.597682+0000) 2019-10-13T09:10:30.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:30.595+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.598427+0000) 2019-10-13T09:10:30.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:30.595+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.599077+0000) 2019-10-13T09:10:30.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:30.595+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.599475+0000) 2019-10-13T09:10:30.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:30.595+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.599634+0000) 2019-10-13T09:10:30.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:30.595+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.600830+0000) 2019-10-13T09:10:30.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:30.595+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.601316+0000) 2019-10-13T09:10:30.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:30.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89040 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:30.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:30.603+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:30.606608+0000) 2019-10-13T09:10:31.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:31.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.650313+0000) 2019-10-13T09:10:31.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:31.647+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:31.651555+0000) 2019-10-13T09:10:33.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:33.963+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:33.964780+0000) 2019-10-13T09:10:34.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:34.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:34.375264+0000) 2019-10-13T09:10:35.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:35.603+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.605961+0000) 2019-10-13T09:10:35.607 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:35.599+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.605593+0000) 2019-10-13T09:10:35.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:35.603+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.606373+0000) 2019-10-13T09:10:35.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:35.603+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.606778+0000) 2019-10-13T09:10:35.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:35.603+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.607437+0000) 2019-10-13T09:10:35.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:35.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.608448+0000) 2019-10-13T09:10:35.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:35.603+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.609040+0000) 2019-10-13T09:10:35.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:35.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:35.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:35.611+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:35.614026+0000) 2019-10-13T09:10:36.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:36.723+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:36.727269+0000) 2019-10-13T09:10:37.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:37.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:37.504758+0000) 2019-10-13T09:10:37.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:37.599+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:37.601967+0000) 2019-10-13T09:10:37.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:37.627+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:37.628261+0000) 2019-10-13T09:10:40.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:40.607+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.613401+0000) 2019-10-13T09:10:40.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:40.611+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.614014+0000) 2019-10-13T09:10:40.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:40.611+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.614418+0000) 2019-10-13T09:10:40.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:40.611+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.614526+0000) 2019-10-13T09:10:40.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:40.611+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.616637+0000) 2019-10-13T09:10:40.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:40.611+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.616375+0000) 2019-10-13T09:10:40.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:40.611+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:40.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:40.615+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.617996+0000) 2019-10-13T09:10:40.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:40.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:40.624342+0000) 2019-10-13T09:10:41.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:41.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.650527+0000) 2019-10-13T09:10:41.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:41.651+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:41.651744+0000) 2019-10-13T09:10:43.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:43.959+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:43.965046+0000) 2019-10-13T09:10:44.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:44.371+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:44.375453+0000) 2019-10-13T09:10:45.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:45.615+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.620074+0000) 2019-10-13T09:10:45.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:45.615+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.621310+0000) 2019-10-13T09:10:45.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:45.619+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.622873+0000) 2019-10-13T09:10:45.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:45.619+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.623948+0000) 2019-10-13T09:10:45.626 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:45.619+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.625530+0000) 2019-10-13T09:10:45.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:45.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:45.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:45.623+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.626085+0000) 2019-10-13T09:10:45.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:45.623+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.626866+0000) 2019-10-13T09:10:45.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:45.627+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:45.629509+0000) 2019-10-13T09:10:46.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:46.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:46.733867+0000) 2019-10-13T09:10:47.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:47.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:47.504964+0000) 2019-10-13T09:10:47.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:47.599+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:47.602169+0000) 2019-10-13T09:10:47.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:47.623+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:47.628453+0000) 2019-10-13T09:10:50.484 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:10:50.484 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:50.488 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:10:50.488 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:10:50.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:50.623+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.628050+0000) 2019-10-13T09:10:50.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:50.623+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.629039+0000) 2019-10-13T09:10:50.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:50.623+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.628748+0000) 2019-10-13T09:10:50.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:50.626+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.628905+0000) 2019-10-13T09:10:50.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:50.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.631334+0000) 2019-10-13T09:10:50.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:50.627+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.632253+0000) 2019-10-13T09:10:50.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:50.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:50.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:50.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.633922+0000) 2019-10-13T09:10:50.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:50.634+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:50.639349+0000) 2019-10-13T09:10:51.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:51.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.650752+0000) 2019-10-13T09:10:51.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:51.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:51.651932+0000) 2019-10-13T09:10:53.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:53.963+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:53.965278+0000) 2019-10-13T09:10:54.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:54.375+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:54.375670+0000) 2019-10-13T09:10:55.637 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:10:55.631+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.636617+0000) 2019-10-13T09:10:55.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:55.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.637136+0000) 2019-10-13T09:10:55.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:10:55.631+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.637670+0000) 2019-10-13T09:10:55.638 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:55.635+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.638071+0000) 2019-10-13T09:10:55.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:10:55.634+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.639887+0000) 2019-10-13T09:10:55.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:55.635+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.638841+0000) 2019-10-13T09:10:55.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:10:55.635+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.639723+0000) 2019-10-13T09:10:55.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:10:55.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89197 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:10:55.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:55.642+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:55.646838+0000) 2019-10-13T09:10:56.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:10:56.738+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:56.739701+0000) 2019-10-13T09:10:57.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:10:57.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:57.505184+0000) 2019-10-13T09:10:57.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:10:57.599+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:57.602423+0000) 2019-10-13T09:10:57.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:10:57.627+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:10:57.628660+0000) 2019-10-13T09:11:00.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:00.642+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.645979+0000) 2019-10-13T09:11:00.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:00.642+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.646520+0000) 2019-10-13T09:11:00.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:00.642+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.646827+0000) 2019-10-13T09:11:00.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:00.642+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.647372+0000) 2019-10-13T09:11:00.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:00.642+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.647349+0000) 2019-10-13T09:11:00.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:00.642+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:00.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:00.642+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.648290+0000) 2019-10-13T09:11:00.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:00.642+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.648366+0000) 2019-10-13T09:11:00.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:00.646+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:00.651259+0000) 2019-10-13T09:11:01.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:01.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.650977+0000) 2019-10-13T09:11:01.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:01.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:01.652136+0000) 2019-10-13T09:11:03.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:03.962+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:03.965488+0000) 2019-10-13T09:11:04.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:04.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:04.375902+0000) 2019-10-13T09:11:05.652 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:05.646+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.651538+0000) 2019-10-13T09:11:05.652 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:05.646+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.652403+0000) 2019-10-13T09:11:05.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:05.646+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.652881+0000) 2019-10-13T09:11:05.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:05.646+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.653241+0000) 2019-10-13T09:11:05.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:05.650+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.653884+0000) 2019-10-13T09:11:05.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:05.650+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.654018+0000) 2019-10-13T09:11:05.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:05.650+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.654683+0000) 2019-10-13T09:11:05.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:05.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:05.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:05.654+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:05.659673+0000) 2019-10-13T09:11:06.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:06.746+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:06.749340+0000) 2019-10-13T09:11:07.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:07.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:07.505368+0000) 2019-10-13T09:11:07.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:07.598+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:07.602621+0000) 2019-10-13T09:11:07.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:07.626+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:07.628870+0000) 2019-10-13T09:11:10.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:10.654+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.659627+0000) 2019-10-13T09:11:10.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:10.654+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.660055+0000) 2019-10-13T09:11:10.662 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:10.658+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.662208+0000) 2019-10-13T09:11:10.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:10.658+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.662533+0000) 2019-10-13T09:11:10.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:10.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:10.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:10.662+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.664217+0000) 2019-10-13T09:11:10.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:10.658+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.664504+0000) 2019-10-13T09:11:10.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:10.658+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.664357+0000) 2019-10-13T09:11:10.669 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:10.666+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:10.668952+0000) 2019-10-13T09:11:11.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:11.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.651173+0000) 2019-10-13T09:11:11.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:11.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:11.652330+0000) 2019-10-13T09:11:13.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:13.958+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:13.965692+0000) 2019-10-13T09:11:14.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:14.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:14.376102+0000) 2019-10-13T09:11:15.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:15.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.666875+0000) 2019-10-13T09:11:15.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:15.662+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.668606+0000) 2019-10-13T09:11:15.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:15.662+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.668769+0000) 2019-10-13T09:11:15.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:15.662+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.669260+0000) 2019-10-13T09:11:15.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:15.666+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.669235+0000) 2019-10-13T09:11:15.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:15.666+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.670051+0000) 2019-10-13T09:11:15.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:15.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:15.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:15.666+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.670752+0000) 2019-10-13T09:11:15.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:15.670+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:15.674075+0000) 2019-10-13T09:11:16.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:16.758+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:16.760674+0000) 2019-10-13T09:11:17.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:17.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:17.505597+0000) 2019-10-13T09:11:17.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:17.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:17.602855+0000) 2019-10-13T09:11:17.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:17.626+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:17.629094+0000) 2019-10-13T09:11:20.591 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:11:20.591 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:20.595 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:11:20.595 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:20.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:20.670+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.673726+0000) 2019-10-13T09:11:20.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:20.670+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.674472+0000) 2019-10-13T09:11:20.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:20.670+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.674525+0000) 2019-10-13T09:11:20.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:20.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.674441+0000) 2019-10-13T09:11:20.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:20.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.675199+0000) 2019-10-13T09:11:20.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:20.670+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.675493+0000) 2019-10-13T09:11:20.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:20.670+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.676413+0000) 2019-10-13T09:11:20.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:20.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:20.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:20.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:20.684064+0000) 2019-10-13T09:11:21.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:21.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.651351+0000) 2019-10-13T09:11:21.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:21.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:21.652520+0000) 2019-10-13T09:11:23.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:23.962+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:23.965930+0000) 2019-10-13T09:11:24.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:24.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:24.376294+0000) 2019-10-13T09:11:25.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:25.674+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.681240+0000) 2019-10-13T09:11:25.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:25.678+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.682191+0000) 2019-10-13T09:11:25.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:25.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:25.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:25.678+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.685451+0000) 2019-10-13T09:11:25.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:25.682+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.685831+0000) 2019-10-13T09:11:25.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:25.682+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.685911+0000) 2019-10-13T09:11:25.690 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:25.682+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.686866+0000) 2019-10-13T09:11:25.690 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:25.682+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.687220+0000) 2019-10-13T09:11:25.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:25.686+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:25.689041+0000) 2019-10-13T09:11:26.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:26.766+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:26.767365+0000) 2019-10-13T09:11:27.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:27.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:27.505791+0000) 2019-10-13T09:11:27.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:27.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:27.603096+0000) 2019-10-13T09:11:27.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:27.622+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:27.629287+0000) 2019-10-13T09:11:30.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:30.682+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.688103+0000) 2019-10-13T09:11:30.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:30.682+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.688475+0000) 2019-10-13T09:11:30.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:30.682+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.688674+0000) 2019-10-13T09:11:30.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:30.686+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.690753+0000) 2019-10-13T09:11:30.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:30.686+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.691136+0000) 2019-10-13T09:11:30.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:30.686+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.691785+0000) 2019-10-13T09:11:30.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:30.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:30.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:30.686+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.693225+0000) 2019-10-13T09:11:30.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:30.694+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:30.699262+0000) 2019-10-13T09:11:31.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:31.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.651524+0000) 2019-10-13T09:11:31.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:31.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:31.652733+0000) 2019-10-13T09:11:33.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:33.962+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:33.966140+0000) 2019-10-13T09:11:34.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:34.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:34.376500+0000) 2019-10-13T09:11:35.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:35.690+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.695632+0000) 2019-10-13T09:11:35.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:35.690+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.695928+0000) 2019-10-13T09:11:35.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:35.690+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.695981+0000) 2019-10-13T09:11:35.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:35.690+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.696922+0000) 2019-10-13T09:11:35.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:35.690+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.696920+0000) 2019-10-13T09:11:35.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:35.690+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.697167+0000) 2019-10-13T09:11:35.700 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:35.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:35.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:35.694+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.700054+0000) 2019-10-13T09:11:35.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:35.702+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:35.704845+0000) 2019-10-13T09:11:36.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:36.770+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:36.773910+0000) 2019-10-13T09:11:37.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:37.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:37.505969+0000) 2019-10-13T09:11:37.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:37.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:37.603300+0000) 2019-10-13T09:11:37.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:37.626+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:37.629484+0000) 2019-10-13T09:11:40.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:40.698+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.703628+0000) 2019-10-13T09:11:40.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:40.698+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.704226+0000) 2019-10-13T09:11:40.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:40.702+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.704820+0000) 2019-10-13T09:11:40.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:40.698+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.705637+0000) 2019-10-13T09:11:40.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:40.702+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.705970+0000) 2019-10-13T09:11:40.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:40.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.706070+0000) 2019-10-13T09:11:40.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:40.702+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.706616+0000) 2019-10-13T09:11:40.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:40.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:40.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:40.706+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:40.709841+0000) 2019-10-13T09:11:41.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:41.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.651725+0000) 2019-10-13T09:11:41.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:41.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:41.652921+0000) 2019-10-13T09:11:43.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:43.966+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:43.966346+0000) 2019-10-13T09:11:44.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:44.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:44.376700+0000) 2019-10-13T09:11:45.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:45.706+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.709522+0000) 2019-10-13T09:11:45.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:45.706+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.709869+0000) 2019-10-13T09:11:45.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:45.706+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.710442+0000) 2019-10-13T09:11:45.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:45.706+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.711369+0000) 2019-10-13T09:11:45.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:45.706+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.711889+0000) 2019-10-13T09:11:45.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:45.706+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.712311+0000) 2019-10-13T09:11:45.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:45.706+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.712886+0000) 2019-10-13T09:11:45.714 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:45.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89513 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:45.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:45.714+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:45.716535+0000) 2019-10-13T09:11:46.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:46.778+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:46.780398+0000) 2019-10-13T09:11:47.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:47.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:47.506214+0000) 2019-10-13T09:11:47.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:47.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:47.603519+0000) 2019-10-13T09:11:47.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:47.626+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:47.629712+0000) 2019-10-13T09:11:50.663 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:11:50.664 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:50.668 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:11:50.668 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:11:50.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:50.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.716471+0000) 2019-10-13T09:11:50.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:50.714+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.718224+0000) 2019-10-13T09:11:50.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:50.714+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.718615+0000) 2019-10-13T09:11:50.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:50.714+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.719272+0000) 2019-10-13T09:11:50.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:50.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.719707+0000) 2019-10-13T09:11:50.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:50.714+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.719855+0000) 2019-10-13T09:11:50.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:50.714+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.720585+0000) 2019-10-13T09:11:50.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:50.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:50.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:50.718+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:50.723533+0000) 2019-10-13T09:11:51.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:51.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.651914+0000) 2019-10-13T09:11:51.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:51.650+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:51.653096+0000) 2019-10-13T09:11:53.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:53.966+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:53.966570+0000) 2019-10-13T09:11:54.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:54.370+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:54.376920+0000) 2019-10-13T09:11:55.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:55.718+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.723881+0000) 2019-10-13T09:11:55.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:11:55.718+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.723899+0000) 2019-10-13T09:11:55.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:11:55.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.724668+0000) 2019-10-13T09:11:55.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:55.718+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.724837+0000) 2019-10-13T09:11:55.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:55.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.725197+0000) 2019-10-13T09:11:55.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:11:55.722+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.726194+0000) 2019-10-13T09:11:55.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:11:55.722+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.726413+0000) 2019-10-13T09:11:55.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:11:55.722+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89575 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:11:55.732 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:55.726+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:55.731581+0000) 2019-10-13T09:11:56.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:11:56.781+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:56.786654+0000) 2019-10-13T09:11:57.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:11:57.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:57.506403+0000) 2019-10-13T09:11:57.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:11:57.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:57.603720+0000) 2019-10-13T09:11:57.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:11:57.625+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:11:57.629940+0000) 2019-10-13T09:12:00.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:00.725+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.730528+0000) 2019-10-13T09:12:00.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:00.725+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.731093+0000) 2019-10-13T09:12:00.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:00.725+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.731584+0000) 2019-10-13T09:12:00.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:00.725+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.733239+0000) 2019-10-13T09:12:00.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:00.725+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.733690+0000) 2019-10-13T09:12:00.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:00.729+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.734030+0000) 2019-10-13T09:12:00.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:00.729+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.734525+0000) 2019-10-13T09:12:00.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:00.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:00.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:00.733+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:00.737802+0000) 2019-10-13T09:12:01.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:01.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.652115+0000) 2019-10-13T09:12:01.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:01.649+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:01.653292+0000) 2019-10-13T09:12:03.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:03.965+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:03.966808+0000) 2019-10-13T09:12:04.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:04.373+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:04.377149+0000) 2019-10-13T09:12:05.739 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:05.733+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.738836+0000) 2019-10-13T09:12:05.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:05.733+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.739354+0000) 2019-10-13T09:12:05.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:05.733+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.739354+0000) 2019-10-13T09:12:05.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:05.737+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.740433+0000) 2019-10-13T09:12:05.742 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:05.733+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.741006+0000) 2019-10-13T09:12:05.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:05.733+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.741257+0000) 2019-10-13T09:12:05.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:05.733+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.740638+0000) 2019-10-13T09:12:05.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:05.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:05.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:05.745+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:05.749569+0000) 2019-10-13T09:12:06.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:06.789+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:06.793250+0000) 2019-10-13T09:12:07.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:07.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:07.506626+0000) 2019-10-13T09:12:07.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:07.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:07.603950+0000) 2019-10-13T09:12:07.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:07.625+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:07.630082+0000) 2019-10-13T09:12:10.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:10.741+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.746065+0000) 2019-10-13T09:12:10.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:10.741+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.747057+0000) 2019-10-13T09:12:10.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:10.741+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.746888+0000) 2019-10-13T09:12:10.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:10.741+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.747891+0000) 2019-10-13T09:12:10.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:10.741+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.748317+0000) 2019-10-13T09:12:10.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:10.741+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.748868+0000) 2019-10-13T09:12:10.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:10.741+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.748924+0000) 2019-10-13T09:12:10.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:10.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:10.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:10.749+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:10.754891+0000) 2019-10-13T09:12:11.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:11.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.652318+0000) 2019-10-13T09:12:11.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:11.649+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:11.653469+0000) 2019-10-13T09:12:13.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:13.965+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:13.967044+0000) 2019-10-13T09:12:14.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:14.373+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:14.377353+0000) 2019-10-13T09:12:15.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:15.753+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.754294+0000) 2019-10-13T09:12:15.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:15.753+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.755000+0000) 2019-10-13T09:12:15.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:15.753+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.755897+0000) 2019-10-13T09:12:15.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:15.749+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.756037+0000) 2019-10-13T09:12:15.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:15.753+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.756890+0000) 2019-10-13T09:12:15.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:15.753+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.756828+0000) 2019-10-13T09:12:15.758 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:15.753+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.757608+0000) 2019-10-13T09:12:15.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:15.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:15.764 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:15.757+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:15.763252+0000) 2019-10-13T09:12:16.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:16.801+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:16.804864+0000) 2019-10-13T09:12:17.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:17.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:17.506839+0000) 2019-10-13T09:12:17.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:17.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:17.604154+0000) 2019-10-13T09:12:17.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:17.629+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:17.630291+0000) 2019-10-13T09:12:20.722 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:12:20.722 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:20.726 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:12:20.726 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:20.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:20.757+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.762654+0000) 2019-10-13T09:12:20.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:20.769+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.772806+0000) 2019-10-13T09:12:20.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:20.757+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.760152+0000) 2019-10-13T09:12:20.788 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:20.761+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.761811+0000) 2019-10-13T09:12:20.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:20.757+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.761610+0000) 2019-10-13T09:12:20.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:20.757+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.761048+0000) 2019-10-13T09:12:20.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:20.761+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.762168+0000) 2019-10-13T09:12:20.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:20.757+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:20.760915+0000) 2019-10-13T09:12:20.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:20.761+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:21.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:21.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.652488+0000) 2019-10-13T09:12:21.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:21.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:21.653658+0000) 2019-10-13T09:12:23.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:23.965+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:23.967272+0000) 2019-10-13T09:12:24.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:24.373+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:24.377547+0000) 2019-10-13T09:12:25.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:25.765+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.765965+0000) 2019-10-13T09:12:25.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:25.765+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.766239+0000) 2019-10-13T09:12:25.768 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:25.765+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.767618+0000) 2019-10-13T09:12:25.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:25.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.767802+0000) 2019-10-13T09:12:25.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:25.765+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.767831+0000) 2019-10-13T09:12:25.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:25.765+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.768196+0000) 2019-10-13T09:12:25.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:25.765+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.768548+0000) 2019-10-13T09:12:25.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:25.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89764 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:25.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:25.769+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:25.775158+0000) 2019-10-13T09:12:26.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:26.813+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:26.816377+0000) 2019-10-13T09:12:27.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:27.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:27.507041+0000) 2019-10-13T09:12:27.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:27.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:27.604357+0000) 2019-10-13T09:12:27.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:27.629+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:27.630526+0000) 2019-10-13T09:12:30.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:30.773+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.774416+0000) 2019-10-13T09:12:30.775 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:30.773+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.774613+0000) 2019-10-13T09:12:30.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:30.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.775677+0000) 2019-10-13T09:12:30.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:30.769+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.775394+0000) 2019-10-13T09:12:30.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:30.773+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.776208+0000) 2019-10-13T09:12:30.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:30.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:30.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:30.773+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.776727+0000) 2019-10-13T09:12:30.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:30.773+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.776953+0000) 2019-10-13T09:12:30.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:30.773+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:30.777780+0000) 2019-10-13T09:12:31.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:31.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.652662+0000) 2019-10-13T09:12:31.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:31.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:31.653833+0000) 2019-10-13T09:12:33.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:33.965+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:33.967432+0000) 2019-10-13T09:12:34.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:34.377+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:34.377771+0000) 2019-10-13T09:12:35.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:35.773+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.779790+0000) 2019-10-13T09:12:35.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:35.777+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.780863+0000) 2019-10-13T09:12:35.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:35.777+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.781408+0000) 2019-10-13T09:12:35.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:35.781+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.781972+0000) 2019-10-13T09:12:35.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:35.781+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.782453+0000) 2019-10-13T09:12:35.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:35.781+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.782491+0000) 2019-10-13T09:12:35.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:35.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:35.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:35.781+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.783180+0000) 2019-10-13T09:12:35.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:35.785+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:35.789280+0000) 2019-10-13T09:12:36.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:36.817+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:36.823056+0000) 2019-10-13T09:12:37.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:37.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:37.507245+0000) 2019-10-13T09:12:37.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:37.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:37.604552+0000) 2019-10-13T09:12:37.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:37.629+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:37.630728+0000) 2019-10-13T09:12:40.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:40.781+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.785215+0000) 2019-10-13T09:12:40.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:40.785+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.785719+0000) 2019-10-13T09:12:40.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:40.785+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.786103+0000) 2019-10-13T09:12:40.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:40.785+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.786365+0000) 2019-10-13T09:12:40.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:40.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.786420+0000) 2019-10-13T09:12:40.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:40.785+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.786815+0000) 2019-10-13T09:12:40.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:40.785+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.787371+0000) 2019-10-13T09:12:40.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:40.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:40.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:40.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:40.793084+0000) 2019-10-13T09:12:41.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:41.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.652839+0000) 2019-10-13T09:12:41.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:41.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:41.654008+0000) 2019-10-13T09:12:43.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:43.965+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:43.967642+0000) 2019-10-13T09:12:44.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:44.377+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:44.377984+0000) 2019-10-13T09:12:45.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:45.789+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.791577+0000) 2019-10-13T09:12:45.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:45.789+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.792210+0000) 2019-10-13T09:12:45.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:45.789+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.792503+0000) 2019-10-13T09:12:45.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:45.789+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.793042+0000) 2019-10-13T09:12:45.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:45.789+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.793316+0000) 2019-10-13T09:12:45.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:45.789+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.795031+0000) 2019-10-13T09:12:45.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:45.793+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.796591+0000) 2019-10-13T09:12:45.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:45.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89890 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:45.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:45.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:45.801687+0000) 2019-10-13T09:12:46.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:46.829+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:46.829659+0000) 2019-10-13T09:12:47.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:47.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:47.507475+0000) 2019-10-13T09:12:47.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:47.601+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:47.604753+0000) 2019-10-13T09:12:47.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:47.629+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:47.630939+0000) 2019-10-13T09:12:50.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:50.797+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.799442+0000) 2019-10-13T09:12:50.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:50.797+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.800074+0000) 2019-10-13T09:12:50.801 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:50.797+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.800395+0000) 2019-10-13T09:12:50.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:50.797+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.800165+0000) 2019-10-13T09:12:50.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:50.801+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.802950+0000) 2019-10-13T09:12:50.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:50.801+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.804183+0000) 2019-10-13T09:12:50.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:50.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:50.806 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:50.805+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.806124+0000) 2019-10-13T09:12:50.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:50.809+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:50.812694+0000) 2019-10-13T09:12:50.852 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:12:50.852 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:50.856 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:12:50.856 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:12:51.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:51.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.653008+0000) 2019-10-13T09:12:51.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:51.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:51.654197+0000) 2019-10-13T09:12:53.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:53.964+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:53.967840+0000) 2019-10-13T09:12:54.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:54.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:54.378220+0000) 2019-10-13T09:12:55.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:55.808+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.809972+0000) 2019-10-13T09:12:55.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:12:55.808+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.810366+0000) 2019-10-13T09:12:55.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:12:55.808+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.810967+0000) 2019-10-13T09:12:55.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:12:55.809+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.812398+0000) 2019-10-13T09:12:55.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:55.808+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.813470+0000) 2019-10-13T09:12:55.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:12:55.808+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:12:55.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:12:55.808+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.813176+0000) 2019-10-13T09:12:55.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:55.812+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.814078+0000) 2019-10-13T09:12:55.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:55.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:55.821234+0000) 2019-10-13T09:12:56.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:12:56.833+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:56.836218+0000) 2019-10-13T09:12:57.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:12:57.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:57.507704+0000) 2019-10-13T09:12:57.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:12:57.600+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:57.604905+0000) 2019-10-13T09:12:57.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:12:57.628+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:12:57.631138+0000) 2019-10-13T09:13:00.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:00.812+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.816641+0000) 2019-10-13T09:13:00.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:00.812+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.817548+0000) 2019-10-13T09:13:00.819 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:00.816+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.818316+0000) 2019-10-13T09:13:00.819 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:00.816+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.818737+0000) 2019-10-13T09:13:00.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:00.816+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.819077+0000) 2019-10-13T09:13:00.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:00.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:00.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:00.817+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.820876+0000) 2019-10-13T09:13:00.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:00.816+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.821696+0000) 2019-10-13T09:13:00.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:00.821+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:00.825906+0000) 2019-10-13T09:13:01.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:01.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.653182+0000) 2019-10-13T09:13:01.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:01.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:01.654372+0000) 2019-10-13T09:13:03.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:03.964+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:03.968070+0000) 2019-10-13T09:13:04.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:04.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:04.378423+0000) 2019-10-13T09:13:05.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:05.820+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.824009+0000) 2019-10-13T09:13:05.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:05.820+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.824474+0000) 2019-10-13T09:13:05.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:05.824+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.826400+0000) 2019-10-13T09:13:05.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:05.820+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.826066+0000) 2019-10-13T09:13:05.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:05.824+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.826926+0000) 2019-10-13T09:13:05.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:05.824+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.827806+0000) 2019-10-13T09:13:05.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:05.824+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:05.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:05.824+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.828334+0000) 2019-10-13T09:13:05.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:05.828+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:05.833308+0000) 2019-10-13T09:13:06.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:06.840+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:06.842763+0000) 2019-10-13T09:13:07.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:07.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:07.507899+0000) 2019-10-13T09:13:07.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:07.604+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:07.605102+0000) 2019-10-13T09:13:07.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:07.628+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:07.631372+0000) 2019-10-13T09:13:10.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:10.828+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.831637+0000) 2019-10-13T09:13:10.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:10.828+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.831996+0000) 2019-10-13T09:13:10.834 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:10.828+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.833697+0000) 2019-10-13T09:13:10.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:10.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.834447+0000) 2019-10-13T09:13:10.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:10.832+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.835114+0000) 2019-10-13T09:13:10.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:10.832+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.835573+0000) 2019-10-13T09:13:10.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:10.832+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:10.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:10.832+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.836087+0000) 2019-10-13T09:13:10.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:10.836+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:10.840903+0000) 2019-10-13T09:13:11.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:11.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.653374+0000) 2019-10-13T09:13:11.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:11.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:11.654554+0000) 2019-10-13T09:13:13.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:13.964+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:13.968275+0000) 2019-10-13T09:13:14.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:14.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:14.378638+0000) 2019-10-13T09:13:15.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:15.832+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.838908+0000) 2019-10-13T09:13:15.840 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:15.836+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.839993+0000) 2019-10-13T09:13:15.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:15.836+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.840865+0000) 2019-10-13T09:13:15.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:15.836+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.841582+0000) 2019-10-13T09:13:15.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:15.840+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.842038+0000) 2019-10-13T09:13:15.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:15.840+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.842573+0000) 2019-10-13T09:13:15.843 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:15.840+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.843162+0000) 2019-10-13T09:13:15.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:15.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:15.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:15.840+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:15.846214+0000) 2019-10-13T09:13:16.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:16.848+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:16.849289+0000) 2019-10-13T09:13:17.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:17.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:17.508120+0000) 2019-10-13T09:13:17.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:17.600+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:17.605311+0000) 2019-10-13T09:13:17.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:17.628+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:17.631573+0000) 2019-10-13T09:13:20.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:20.844+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.846857+0000) 2019-10-13T09:13:20.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:20.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.847570+0000) 2019-10-13T09:13:20.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:20.844+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.848354+0000) 2019-10-13T09:13:20.849 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:20.844+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.849143+0000) 2019-10-13T09:13:20.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:20.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:20.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:20.848+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.851291+0000) 2019-10-13T09:13:20.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:20.848+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.851950+0000) 2019-10-13T09:13:20.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:20.848+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.852263+0000) 2019-10-13T09:13:20.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:20.848+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:20.854197+0000) 2019-10-13T09:13:20.892 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:13:20.893 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:20.896 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:13:20.896 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:21.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:21.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.653543+0000) 2019-10-13T09:13:21.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:21.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:21.654741+0000) 2019-10-13T09:13:23.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:23.968+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:23.968506+0000) 2019-10-13T09:13:24.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:24.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:24.378857+0000) 2019-10-13T09:13:25.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:25.848+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.853546+0000) 2019-10-13T09:13:25.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:25.848+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.853741+0000) 2019-10-13T09:13:25.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:25.852+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.855499+0000) 2019-10-13T09:13:25.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:25.852+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.855798+0000) 2019-10-13T09:13:25.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:25.852+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.856194+0000) 2019-10-13T09:13:25.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:25.852+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.856591+0000) 2019-10-13T09:13:25.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:25.852+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.857133+0000) 2019-10-13T09:13:25.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:25.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:25.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:25.856+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:25.861537+0000) 2019-10-13T09:13:26.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:26.852+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:26.855783+0000) 2019-10-13T09:13:27.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:27.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:27.508336+0000) 2019-10-13T09:13:27.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:27.604+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:27.605513+0000) 2019-10-13T09:13:27.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:27.628+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:27.631801+0000) 2019-10-13T09:13:30.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:30.856+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.860560+0000) 2019-10-13T09:13:30.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:30.856+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.861428+0000) 2019-10-13T09:13:30.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:30.860+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.863435+0000) 2019-10-13T09:13:30.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:30.860+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.864412+0000) 2019-10-13T09:13:30.866 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:30.860+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.865484+0000) 2019-10-13T09:13:30.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:30.864+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.865854+0000) 2019-10-13T09:13:30.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:30.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:30.867 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:30.864+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.866450+0000) 2019-10-13T09:13:30.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:30.868+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:30.872819+0000) 2019-10-13T09:13:31.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:31.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.653720+0000) 2019-10-13T09:13:31.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:31.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:31.654904+0000) 2019-10-13T09:13:33.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:33.964+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:33.968712+0000) 2019-10-13T09:13:34.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:34.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:34.379059+0000) 2019-10-13T09:13:35.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:35.864+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.869196+0000) 2019-10-13T09:13:35.870 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:35.868+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.869845+0000) 2019-10-13T09:13:35.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:35.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.869856+0000) 2019-10-13T09:13:35.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:35.868+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.871847+0000) 2019-10-13T09:13:35.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:35.868+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.871374+0000) 2019-10-13T09:13:35.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:35.868+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.872340+0000) 2019-10-13T09:13:35.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:35.868+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.871981+0000) 2019-10-13T09:13:35.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:35.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90197 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:35.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:35.872+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:35.876659+0000) 2019-10-13T09:13:36.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:36.860+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:36.862152+0000) 2019-10-13T09:13:37.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:37.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:37.508556+0000) 2019-10-13T09:13:37.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:37.604+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:37.605712+0000) 2019-10-13T09:13:37.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:37.628+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:37.632001+0000) 2019-10-13T09:13:40.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:40.872+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.876836+0000) 2019-10-13T09:13:40.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:40.872+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.876995+0000) 2019-10-13T09:13:40.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:40.872+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.877680+0000) 2019-10-13T09:13:40.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:40.872+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.877741+0000) 2019-10-13T09:13:40.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:40.876+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.878785+0000) 2019-10-13T09:13:40.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:40.876+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.880114+0000) 2019-10-13T09:13:40.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:40.876+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.880824+0000) 2019-10-13T09:13:40.881 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:40.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90232 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:40.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:40.880+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:40.884562+0000) 2019-10-13T09:13:41.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:41.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.653933+0000) 2019-10-13T09:13:41.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:41.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:41.655096+0000) 2019-10-13T09:13:43.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:43.968+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:43.968945+0000) 2019-10-13T09:13:44.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:44.376+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:44.379254+0000) 2019-10-13T09:13:45.885 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:45.880+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.884105+0000) 2019-10-13T09:13:45.885 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:45.880+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.884579+0000) 2019-10-13T09:13:45.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:45.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.885930+0000) 2019-10-13T09:13:45.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:45.884+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.886226+0000) 2019-10-13T09:13:45.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:45.880+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.886617+0000) 2019-10-13T09:13:45.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:45.884+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.887406+0000) 2019-10-13T09:13:45.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:45.884+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.887738+0000) 2019-10-13T09:13:45.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:45.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90259 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:45.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:45.888+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:45.895569+0000) 2019-10-13T09:13:46.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:46.864+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:46.868742+0000) 2019-10-13T09:13:47.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:47.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:47.508775+0000) 2019-10-13T09:13:47.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:47.604+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:47.605943+0000) 2019-10-13T09:13:47.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:47.632+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:47.632198+0000) 2019-10-13T09:13:50.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:50.887+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.891174+0000) 2019-10-13T09:13:50.892 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:50.887+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.891726+0000) 2019-10-13T09:13:50.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:50.888+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.891856+0000) 2019-10-13T09:13:50.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:50.887+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.893241+0000) 2019-10-13T09:13:50.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:50.891+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.894662+0000) 2019-10-13T09:13:50.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:50.891+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.894468+0000) 2019-10-13T09:13:50.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:50.891+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.894045+0000) 2019-10-13T09:13:50.897 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:50.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:50.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:50.896+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:50.901114+0000) 2019-10-13T09:13:50.999 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:13:50.999 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:51.003 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:13:51.003 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:13:51.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:51.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.654112+0000) 2019-10-13T09:13:51.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:51.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:51.655272+0000) 2019-10-13T09:13:53.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:53.963+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:53.969150+0000) 2019-10-13T09:13:54.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:54.375+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:54.379456+0000) 2019-10-13T09:13:55.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:55.895+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.900582+0000) 2019-10-13T09:13:55.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:55.895+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.900980+0000) 2019-10-13T09:13:55.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:13:55.895+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.901485+0000) 2019-10-13T09:13:55.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:55.899+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.901718+0000) 2019-10-13T09:13:55.903 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:13:55.899+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.902882+0000) 2019-10-13T09:13:55.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:13:55.896+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.902755+0000) 2019-10-13T09:13:55.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:13:55.899+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.903989+0000) 2019-10-13T09:13:55.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:13:55.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:13:55.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:55.904+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:55.908559+0000) 2019-10-13T09:13:56.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:13:56.872+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:56.875167+0000) 2019-10-13T09:13:57.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:13:57.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:57.508978+0000) 2019-10-13T09:13:57.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:13:57.603+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:57.606178+0000) 2019-10-13T09:13:57.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:13:57.627+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:13:57.632411+0000) 2019-10-13T09:14:00.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:00.903+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.907588+0000) 2019-10-13T09:14:00.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:00.903+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.908947+0000) 2019-10-13T09:14:00.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:00.903+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.909278+0000) 2019-10-13T09:14:00.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:00.907+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.910248+0000) 2019-10-13T09:14:00.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:00.904+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.910506+0000) 2019-10-13T09:14:00.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:00.907+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.910784+0000) 2019-10-13T09:14:00.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:00.907+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.911598+0000) 2019-10-13T09:14:00.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:00.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:00.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:00.908+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:00.914389+0000) 2019-10-13T09:14:01.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:01.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.654297+0000) 2019-10-13T09:14:01.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:01.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:01.655447+0000) 2019-10-13T09:14:03.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:03.967+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:03.969352+0000) 2019-10-13T09:14:04.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:04.375+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:04.379654+0000) 2019-10-13T09:14:05.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:05.915+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.917881+0000) 2019-10-13T09:14:05.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:05.915+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.918208+0000) 2019-10-13T09:14:05.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:05.915+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.918560+0000) 2019-10-13T09:14:05.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:05.916+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.918939+0000) 2019-10-13T09:14:05.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:05.915+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:05.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:05.915+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.920487+0000) 2019-10-13T09:14:05.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:05.915+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.921073+0000) 2019-10-13T09:14:05.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:05.915+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.921564+0000) 2019-10-13T09:14:05.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:05.924+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:05.927165+0000) 2019-10-13T09:14:06.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:06.884+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:06.885347+0000) 2019-10-13T09:14:07.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:07.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:07.509180+0000) 2019-10-13T09:14:07.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:07.603+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:07.606375+0000) 2019-10-13T09:14:07.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:07.627+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:07.632604+0000) 2019-10-13T09:14:10.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:10.919+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.922968+0000) 2019-10-13T09:14:10.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:10.919+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.922922+0000) 2019-10-13T09:14:10.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:10.919+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.924055+0000) 2019-10-13T09:14:10.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:10.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.924436+0000) 2019-10-13T09:14:10.925 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:10.919+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.924701+0000) 2019-10-13T09:14:10.926 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:10.919+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.925096+0000) 2019-10-13T09:14:10.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:10.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:10.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:10.923+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.927383+0000) 2019-10-13T09:14:10.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:10.931+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:10.932357+0000) 2019-10-13T09:14:11.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:11.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.654493+0000) 2019-10-13T09:14:11.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:11.651+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:11.655620+0000) 2019-10-13T09:14:13.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:13.967+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:13.969610+0000) 2019-10-13T09:14:14.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:14.375+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:14.379861+0000) 2019-10-13T09:14:15.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:15.927+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.930618+0000) 2019-10-13T09:14:15.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:15.927+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.931058+0000) 2019-10-13T09:14:15.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:15.927+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.931318+0000) 2019-10-13T09:14:15.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:15.927+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.932407+0000) 2019-10-13T09:14:15.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:15.931+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.932512+0000) 2019-10-13T09:14:15.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:15.931+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.934387+0000) 2019-10-13T09:14:15.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:15.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:15.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:15.931+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.935459+0000) 2019-10-13T09:14:15.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:15.935+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:15.938118+0000) 2019-10-13T09:14:16.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:16.887+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:16.891724+0000) 2019-10-13T09:14:17.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:17.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:17.509384+0000) 2019-10-13T09:14:17.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:17.603+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:17.606613+0000) 2019-10-13T09:14:17.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:17.631+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:17.632819+0000) 2019-10-13T09:14:20.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:20.935+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.937877+0000) 2019-10-13T09:14:20.939 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:20.935+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.938887+0000) 2019-10-13T09:14:20.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:20.935+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.939543+0000) 2019-10-13T09:14:20.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:20.935+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.939589+0000) 2019-10-13T09:14:20.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:20.935+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.939943+0000) 2019-10-13T09:14:20.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:20.935+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.941127+0000) 2019-10-13T09:14:20.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:20.939+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.941184+0000) 2019-10-13T09:14:20.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:20.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:20.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:20.943+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:20.946753+0000) 2019-10-13T09:14:21.110 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:14:21.111 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:21.114 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:14:21.115 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:21.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:21.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.654668+0000) 2019-10-13T09:14:21.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:21.651+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:21.655823+0000) 2019-10-13T09:14:23.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:23.967+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:23.969811+0000) 2019-10-13T09:14:24.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:24.379+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:24.380108+0000) 2019-10-13T09:14:25.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:25.943+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.945855+0000) 2019-10-13T09:14:25.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:25.943+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.946295+0000) 2019-10-13T09:14:25.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:25.943+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.948267+0000) 2019-10-13T09:14:25.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:25.943+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.948754+0000) 2019-10-13T09:14:25.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:25.943+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.949137+0000) 2019-10-13T09:14:25.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:25.947+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.950129+0000) 2019-10-13T09:14:25.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:25.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:25.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:25.947+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.950698+0000) 2019-10-13T09:14:25.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:25.951+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:25.955483+0000) 2019-10-13T09:14:26.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:26.895+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:26.898151+0000) 2019-10-13T09:14:27.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:27.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:27.509618+0000) 2019-10-13T09:14:27.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:27.603+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:27.606829+0000) 2019-10-13T09:14:27.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:27.627+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:27.633015+0000) 2019-10-13T09:14:30.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:30.951+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.953920+0000) 2019-10-13T09:14:30.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:30.951+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.954508+0000) 2019-10-13T09:14:30.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:30.951+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.955184+0000) 2019-10-13T09:14:30.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:30.951+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.955839+0000) 2019-10-13T09:14:30.958 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:30.955+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.957795+0000) 2019-10-13T09:14:30.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:30.955+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.957925+0000) 2019-10-13T09:14:30.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:30.955+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.958032+0000) 2019-10-13T09:14:30.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:30.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:30.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:30.963+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:30.964374+0000) 2019-10-13T09:14:31.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:31.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.654869+0000) 2019-10-13T09:14:31.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:31.651+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:31.655997+0000) 2019-10-13T09:14:33.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:33.967+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:33.969975+0000) 2019-10-13T09:14:34.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:34.375+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:34.380307+0000) 2019-10-13T09:14:35.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:35.959+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.962327+0000) 2019-10-13T09:14:35.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:35.959+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.962810+0000) 2019-10-13T09:14:35.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:35.959+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.964332+0000) 2019-10-13T09:14:35.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:35.963+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.964278+0000) 2019-10-13T09:14:35.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:35.959+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.965252+0000) 2019-10-13T09:14:35.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:35.959+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.965455+0000) 2019-10-13T09:14:35.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:35.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:35.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:35.963+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.966623+0000) 2019-10-13T09:14:35.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:35.967+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:35.971757+0000) 2019-10-13T09:14:36.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:36.903+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:36.904605+0000) 2019-10-13T09:14:37.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:37.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:37.509811+0000) 2019-10-13T09:14:37.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:37.603+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:37.607027+0000) 2019-10-13T09:14:37.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:37.631+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:37.633237+0000) 2019-10-13T09:14:40.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:40.967+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.969784+0000) 2019-10-13T09:14:40.972 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:40.967+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.971485+0000) 2019-10-13T09:14:40.973 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:40.967+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.972420+0000) 2019-10-13T09:14:40.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:40.971+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.972520+0000) 2019-10-13T09:14:40.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:40.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:40.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:40.971+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.974542+0000) 2019-10-13T09:14:40.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:40.971+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.974894+0000) 2019-10-13T09:14:40.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:40.971+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.975435+0000) 2019-10-13T09:14:40.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:40.979+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:40.983351+0000) 2019-10-13T09:14:41.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:41.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.655033+0000) 2019-10-13T09:14:41.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:41.655+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:41.656168+0000) 2019-10-13T09:14:43.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:43.967+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:43.970137+0000) 2019-10-13T09:14:44.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:44.379+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:44.380526+0000) 2019-10-13T09:14:45.979 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:45.975+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.977869+0000) 2019-10-13T09:14:45.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:45.975+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.978303+0000) 2019-10-13T09:14:45.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:45.975+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.979304+0000) 2019-10-13T09:14:45.981 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:45.975+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.978638+0000) 2019-10-13T09:14:45.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:45.975+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.979436+0000) 2019-10-13T09:14:45.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:45.975+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.980433+0000) 2019-10-13T09:14:45.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:45.975+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.981019+0000) 2019-10-13T09:14:45.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:45.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:45.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:45.983+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:45.986741+0000) 2019-10-13T09:14:46.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:46.907+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:46.911201+0000) 2019-10-13T09:14:47.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:47.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:47.510008+0000) 2019-10-13T09:14:47.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:47.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:47.607244+0000) 2019-10-13T09:14:47.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:47.630+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:47.633437+0000) 2019-10-13T09:14:50.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:50.978+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.984902+0000) 2019-10-13T09:14:50.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:50.982+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.985920+0000) 2019-10-13T09:14:50.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:50.983+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.986005+0000) 2019-10-13T09:14:50.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:50.982+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.987664+0000) 2019-10-13T09:14:50.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:50.982+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.988203+0000) 2019-10-13T09:14:50.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:50.982+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.988244+0000) 2019-10-13T09:14:50.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:50.982+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.988579+0000) 2019-10-13T09:14:50.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:50.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:50.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:50.991+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:50.992865+0000) 2019-10-13T09:14:51.233 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:14:51.234 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:51.238 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:14:51.238 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:14:51.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:51.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.655230+0000) 2019-10-13T09:14:51.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:51.655+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:51.656387+0000) 2019-10-13T09:14:53.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:53.966+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:53.970342+0000) 2019-10-13T09:14:54.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:54.378+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:54.380759+0000) 2019-10-13T09:14:55.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:14:55.987+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.991987+0000) 2019-10-13T09:14:55.994 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:14:55.990+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.993846+0000) 2019-10-13T09:14:55.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:55.990+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.995674+0000) 2019-10-13T09:14:55.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:14:55.990+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.996255+0000) 2019-10-13T09:14:55.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:14:55.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:14:55.997 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:55.990+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.997218+0000) 2019-10-13T09:14:55.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:14:55.994+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.997807+0000) 2019-10-13T09:14:55.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:55.994+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:55.998150+0000) 2019-10-13T09:14:56.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:56.003+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.004470+0000) 2019-10-13T09:14:56.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:14:56.915+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:56.917768+0000) 2019-10-13T09:14:57.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:14:57.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:57.510162+0000) 2019-10-13T09:14:57.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:14:57.602+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:57.607436+0000) 2019-10-13T09:14:57.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:14:57.630+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:14:57.633640+0000) 2019-10-13T09:15:01.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:00.994+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.000578+0000) 2019-10-13T09:15:01.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:00.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.001523+0000) 2019-10-13T09:15:01.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:00.998+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.002099+0000) 2019-10-13T09:15:01.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:00.999+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.001937+0000) 2019-10-13T09:15:01.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:00.998+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.002934+0000) 2019-10-13T09:15:01.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:00.998+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.003343+0000) 2019-10-13T09:15:01.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:00.998+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.003626+0000) 2019-10-13T09:15:01.005 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:00.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:01.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:01.007+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.008545+0000) 2019-10-13T09:15:01.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:01.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.655405+0000) 2019-10-13T09:15:01.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:01.655+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:01.656564+0000) 2019-10-13T09:15:03.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:03.966+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:03.970551+0000) 2019-10-13T09:15:04.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:04.378+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:04.381005+0000) 2019-10-13T09:15:06.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:06.002+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.007764+0000) 2019-10-13T09:15:06.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:06.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.008836+0000) 2019-10-13T09:15:06.010 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:06.002+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.009058+0000) 2019-10-13T09:15:06.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:06.006+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.010908+0000) 2019-10-13T09:15:06.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:06.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.010839+0000) 2019-10-13T09:15:06.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:06.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:06.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:06.006+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.012532+0000) 2019-10-13T09:15:06.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:06.006+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.012661+0000) 2019-10-13T09:15:06.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:06.015+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.017971+0000) 2019-10-13T09:15:06.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:06.923+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:06.924378+0000) 2019-10-13T09:15:07.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:07.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:07.510359+0000) 2019-10-13T09:15:07.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:07.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:07.607637+0000) 2019-10-13T09:15:07.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:07.630+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:07.633862+0000) 2019-10-13T09:15:11.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:11.010+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.015172+0000) 2019-10-13T09:15:11.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:11.011+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.015995+0000) 2019-10-13T09:15:11.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:11.014+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.018502+0000) 2019-10-13T09:15:11.019 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:11.014+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.019122+0000) 2019-10-13T09:15:11.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:11.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:11.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:11.014+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.018748+0000) 2019-10-13T09:15:11.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:11.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.020552+0000) 2019-10-13T09:15:11.021 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:11.014+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.021203+0000) 2019-10-13T09:15:11.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:11.023+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.026183+0000) 2019-10-13T09:15:11.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:11.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.655575+0000) 2019-10-13T09:15:11.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:11.655+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:11.656778+0000) 2019-10-13T09:15:13.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:13.966+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:13.970729+0000) 2019-10-13T09:15:14.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:14.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:14.381241+0000) 2019-10-13T09:15:16.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:16.018+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.022642+0000) 2019-10-13T09:15:16.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:16.018+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.023906+0000) 2019-10-13T09:15:16.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:16.018+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.023706+0000) 2019-10-13T09:15:16.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:16.018+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.024566+0000) 2019-10-13T09:15:16.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:16.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.025186+0000) 2019-10-13T09:15:16.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:16.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.025543+0000) 2019-10-13T09:15:16.027 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:16.022+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.026426+0000) 2019-10-13T09:15:16.028 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:16.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:16.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:16.026+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.029225+0000) 2019-10-13T09:15:16.932 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:16.930+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:16.931000+0000) 2019-10-13T09:15:17.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:17.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:17.510565+0000) 2019-10-13T09:15:17.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:17.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:17.607834+0000) 2019-10-13T09:15:17.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:17.630+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:17.634062+0000) 2019-10-13T09:15:21.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:21.030+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.032295+0000) 2019-10-13T09:15:21.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:21.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.030808+0000) 2019-10-13T09:15:21.036 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:21.026+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.031189+0000) 2019-10-13T09:15:21.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:21.026+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.031704+0000) 2019-10-13T09:15:21.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:21.026+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.031952+0000) 2019-10-13T09:15:21.037 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:21.026+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.032992+0000) 2019-10-13T09:15:21.037 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:21.030+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.033812+0000) 2019-10-13T09:15:21.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:21.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:21.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:21.034+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.037422+0000) 2019-10-13T09:15:21.338 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:15:21.339 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:21.342 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:15:21.343 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:21.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:21.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.655759+0000) 2019-10-13T09:15:21.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:21.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:21.656969+0000) 2019-10-13T09:15:23.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:23.970+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:23.970941+0000) 2019-10-13T09:15:24.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:24.378+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:24.381450+0000) 2019-10-13T09:15:26.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:26.034+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.037964+0000) 2019-10-13T09:15:26.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:26.034+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.038604+0000) 2019-10-13T09:15:26.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:26.034+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.039477+0000) 2019-10-13T09:15:26.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:26.034+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.040039+0000) 2019-10-13T09:15:26.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:26.034+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.040568+0000) 2019-10-13T09:15:26.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:26.034+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.040844+0000) 2019-10-13T09:15:26.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:26.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.041695+0000) 2019-10-13T09:15:26.042 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:26.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:26.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:26.042+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.044403+0000) 2019-10-13T09:15:26.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:26.934+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:26.937595+0000) 2019-10-13T09:15:27.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:27.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:27.510791+0000) 2019-10-13T09:15:27.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:27.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:27.608066+0000) 2019-10-13T09:15:27.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:27.630+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:27.634287+0000) 2019-10-13T09:15:31.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:31.042+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.045260+0000) 2019-10-13T09:15:31.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:31.042+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.047050+0000) 2019-10-13T09:15:31.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:31.042+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.047354+0000) 2019-10-13T09:15:31.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:31.042+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.047815+0000) 2019-10-13T09:15:31.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:31.042+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.047997+0000) 2019-10-13T09:15:31.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:31.042+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.048392+0000) 2019-10-13T09:15:31.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:31.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:31.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:31.042+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.049029+0000) 2019-10-13T09:15:31.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:31.050+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.053330+0000) 2019-10-13T09:15:31.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:31.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.655932+0000) 2019-10-13T09:15:31.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:31.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:31.657170+0000) 2019-10-13T09:15:33.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:33.970+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:33.971160+0000) 2019-10-13T09:15:34.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:34.374+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:34.381679+0000) 2019-10-13T09:15:36.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:36.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.052370+0000) 2019-10-13T09:15:36.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:36.050+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.052412+0000) 2019-10-13T09:15:36.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:36.046+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.053329+0000) 2019-10-13T09:15:36.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:36.050+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.055055+0000) 2019-10-13T09:15:36.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:36.050+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.056648+0000) 2019-10-13T09:15:36.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:36.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:36.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:36.050+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.057247+0000) 2019-10-13T09:15:36.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:36.050+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.057678+0000) 2019-10-13T09:15:36.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:36.058+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.061562+0000) 2019-10-13T09:15:36.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:36.938+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:36.944004+0000) 2019-10-13T09:15:37.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:37.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:37.510992+0000) 2019-10-13T09:15:37.609 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:37.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:37.608595+0000) 2019-10-13T09:15:37.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:37.634+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:37.634489+0000) 2019-10-13T09:15:41.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:41.054+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.058925+0000) 2019-10-13T09:15:41.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:41.054+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.059831+0000) 2019-10-13T09:15:41.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:41.054+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.060525+0000) 2019-10-13T09:15:41.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:41.054+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.061171+0000) 2019-10-13T09:15:41.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:41.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.061428+0000) 2019-10-13T09:15:41.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:41.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.062058+0000) 2019-10-13T09:15:41.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:41.058+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.062277+0000) 2019-10-13T09:15:41.065 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:41.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:41.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:41.066+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.068664+0000) 2019-10-13T09:15:41.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:41.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.656104+0000) 2019-10-13T09:15:41.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:41.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:41.657345+0000) 2019-10-13T09:15:43.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:43.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:43.971358+0000) 2019-10-13T09:15:44.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:44.377+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:44.381906+0000) 2019-10-13T09:15:46.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:46.061+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.067396+0000) 2019-10-13T09:15:46.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:46.061+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.068899+0000) 2019-10-13T09:15:46.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:46.061+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.068924+0000) 2019-10-13T09:15:46.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:46.061+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.068355+0000) 2019-10-13T09:15:46.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:46.065+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.069813+0000) 2019-10-13T09:15:46.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:46.066+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.069705+0000) 2019-10-13T09:15:46.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:46.065+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.070632+0000) 2019-10-13T09:15:46.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:46.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:46.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:46.070+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.075314+0000) 2019-10-13T09:15:46.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:46.954+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:46.954984+0000) 2019-10-13T09:15:47.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:47.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:47.511191+0000) 2019-10-13T09:15:47.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:47.605+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:47.608992+0000) 2019-10-13T09:15:47.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:47.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:47.634711+0000) 2019-10-13T09:15:51.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:51.069+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.074554+0000) 2019-10-13T09:15:51.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:51.070+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.074562+0000) 2019-10-13T09:15:51.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:51.069+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.075882+0000) 2019-10-13T09:15:51.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:51.069+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.076300+0000) 2019-10-13T09:15:51.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:51.069+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.076845+0000) 2019-10-13T09:15:51.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:51.069+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.077474+0000) 2019-10-13T09:15:51.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:51.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.078747+0000) 2019-10-13T09:15:51.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:51.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:51.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:51.078+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.082606+0000) 2019-10-13T09:15:51.436 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:15:51.436 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:51.440 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:15:51.440 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:15:51.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:51.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.656314+0000) 2019-10-13T09:15:51.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:51.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:51.657528+0000) 2019-10-13T09:15:53.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:53.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:53.971561+0000) 2019-10-13T09:15:54.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:54.381+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:54.382114+0000) 2019-10-13T09:15:56.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:56.077+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.083173+0000) 2019-10-13T09:15:56.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:15:56.077+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.083667+0000) 2019-10-13T09:15:56.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:15:56.077+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.083857+0000) 2019-10-13T09:15:56.085 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:56.077+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.084817+0000) 2019-10-13T09:15:56.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:15:56.082+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.084868+0000) 2019-10-13T09:15:56.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:56.081+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.085902+0000) 2019-10-13T09:15:56.087 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:15:56.081+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.086538+0000) 2019-10-13T09:15:56.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:15:56.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:15:56.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:56.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.090503+0000) 2019-10-13T09:15:56.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:15:56.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:56.961547+0000) 2019-10-13T09:15:57.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:15:57.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:57.511391+0000) 2019-10-13T09:15:57.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:15:57.605+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:57.609142+0000) 2019-10-13T09:15:57.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:15:57.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:15:57.634909+0000) 2019-10-13T09:16:01.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:01.089+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.091163+0000) 2019-10-13T09:16:01.092 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:01.089+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.091591+0000) 2019-10-13T09:16:01.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:01.089+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.092146+0000) 2019-10-13T09:16:01.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:01.089+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.092422+0000) 2019-10-13T09:16:01.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:01.090+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.093402+0000) 2019-10-13T09:16:01.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:01.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.094181+0000) 2019-10-13T09:16:01.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:01.093+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.094256+0000) 2019-10-13T09:16:01.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:01.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:01.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:01.094+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.099524+0000) 2019-10-13T09:16:01.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:01.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.656490+0000) 2019-10-13T09:16:01.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:01.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:01.657764+0000) 2019-10-13T09:16:03.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:03.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:03.971758+0000) 2019-10-13T09:16:04.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:04.381+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:04.382364+0000) 2019-10-13T09:16:06.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:06.097+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.101306+0000) 2019-10-13T09:16:06.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:06.101+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.101730+0000) 2019-10-13T09:16:06.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:06.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.102196+0000) 2019-10-13T09:16:06.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:06.101+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.102819+0000) 2019-10-13T09:16:06.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:06.101+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.103871+0000) 2019-10-13T09:16:06.105 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:06.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:06.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:06.101+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.104164+0000) 2019-10-13T09:16:06.106 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:06.102+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.104752+0000) 2019-10-13T09:16:06.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:06.102+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.106197+0000) 2019-10-13T09:16:06.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:06.966+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:06.968136+0000) 2019-10-13T09:16:07.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:07.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:07.511597+0000) 2019-10-13T09:16:07.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:07.605+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:07.609340+0000) 2019-10-13T09:16:07.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:07.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:07.635104+0000) 2019-10-13T09:16:11.108 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:11.105+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.106873+0000) 2019-10-13T09:16:11.108 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:11.105+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.107528+0000) 2019-10-13T09:16:11.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:11.105+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.107709+0000) 2019-10-13T09:16:11.109 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:11.106+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.108196+0000) 2019-10-13T09:16:11.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:11.105+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.109176+0000) 2019-10-13T09:16:11.110 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:11.105+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.109551+0000) 2019-10-13T09:16:11.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:11.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.110416+0000) 2019-10-13T09:16:11.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:11.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:11.115 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:11.110+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.114031+0000) 2019-10-13T09:16:11.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:11.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.656684+0000) 2019-10-13T09:16:11.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:11.654+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:11.657930+0000) 2019-10-13T09:16:13.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:13.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:13.971960+0000) 2019-10-13T09:16:14.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:14.381+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:14.382622+0000) 2019-10-13T09:16:16.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:16.113+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.114692+0000) 2019-10-13T09:16:16.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:16.113+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.115047+0000) 2019-10-13T09:16:16.117 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:16.114+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.116655+0000) 2019-10-13T09:16:16.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:16.113+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.117316+0000) 2019-10-13T09:16:16.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:16.117+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.118220+0000) 2019-10-13T09:16:16.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:16.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:16.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:16.117+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.119104+0000) 2019-10-13T09:16:16.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:16.117+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.119583+0000) 2019-10-13T09:16:16.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:16.118+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.123643+0000) 2019-10-13T09:16:16.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:16.974+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:16.974609+0000) 2019-10-13T09:16:17.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:17.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:17.511833+0000) 2019-10-13T09:16:17.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:17.605+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:17.609537+0000) 2019-10-13T09:16:17.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:17.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:17.635302+0000) 2019-10-13T09:16:21.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:21.117+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.121496+0000) 2019-10-13T09:16:21.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:21.121+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.122360+0000) 2019-10-13T09:16:21.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:21.121+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.122771+0000) 2019-10-13T09:16:21.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:21.121+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.123469+0000) 2019-10-13T09:16:21.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:21.121+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.123857+0000) 2019-10-13T09:16:21.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:21.121+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.124311+0000) 2019-10-13T09:16:21.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:21.121+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.124373+0000) 2019-10-13T09:16:21.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:21.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91200 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:21.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:21.125+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.130444+0000) 2019-10-13T09:16:21.549 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:16:21.550 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:21.554 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:16:21.554 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:21.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:21.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.656860+0000) 2019-10-13T09:16:21.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:21.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:21.658131+0000) 2019-10-13T09:16:23.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:23.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:23.972229+0000) 2019-10-13T09:16:24.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:24.381+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:24.382869+0000) 2019-10-13T09:16:26.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:26.129+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.130834+0000) 2019-10-13T09:16:26.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:26.129+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.131023+0000) 2019-10-13T09:16:26.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:26.129+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.131815+0000) 2019-10-13T09:16:26.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:26.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:26.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:26.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.132634+0000) 2019-10-13T09:16:26.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:26.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.133301+0000) 2019-10-13T09:16:26.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:26.133+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.133845+0000) 2019-10-13T09:16:26.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:26.133+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.133986+0000) 2019-10-13T09:16:26.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:26.133+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.138620+0000) 2019-10-13T09:16:26.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:26.977+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:26.981082+0000) 2019-10-13T09:16:27.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:27.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:27.512093+0000) 2019-10-13T09:16:27.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:27.609+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:27.609782+0000) 2019-10-13T09:16:27.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:27.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:27.635501+0000) 2019-10-13T09:16:31.136 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:31.133+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.136014+0000) 2019-10-13T09:16:31.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:31.133+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.136476+0000) 2019-10-13T09:16:31.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:31.133+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.137610+0000) 2019-10-13T09:16:31.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:31.137+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.138351+0000) 2019-10-13T09:16:31.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:31.137+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.138531+0000) 2019-10-13T09:16:31.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:31.133+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.138729+0000) 2019-10-13T09:16:31.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:31.137+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.139171+0000) 2019-10-13T09:16:31.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:31.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91259 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:31.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:31.141+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.146029+0000) 2019-10-13T09:16:31.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:31.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.657038+0000) 2019-10-13T09:16:31.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:31.657+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:31.658336+0000) 2019-10-13T09:16:33.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:33.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:33.972431+0000) 2019-10-13T09:16:34.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:34.381+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:34.383074+0000) 2019-10-13T09:16:36.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:36.137+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.143187+0000) 2019-10-13T09:16:36.145 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:36.141+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.144500+0000) 2019-10-13T09:16:36.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:36.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.144130+0000) 2019-10-13T09:16:36.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:36.141+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.144803+0000) 2019-10-13T09:16:36.146 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:36.141+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.145325+0000) 2019-10-13T09:16:36.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:36.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.145500+0000) 2019-10-13T09:16:36.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:36.145+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.146963+0000) 2019-10-13T09:16:36.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:36.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:36.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:36.149+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.152276+0000) 2019-10-13T09:16:36.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:36.981+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:36.987602+0000) 2019-10-13T09:16:37.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:37.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:37.512328+0000) 2019-10-13T09:16:37.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:37.609+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:37.609940+0000) 2019-10-13T09:16:37.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:37.633+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:37.635744+0000) 2019-10-13T09:16:41.152 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:41.148+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.151861+0000) 2019-10-13T09:16:41.153 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:41.149+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.152182+0000) 2019-10-13T09:16:41.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:41.148+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.152861+0000) 2019-10-13T09:16:41.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:41.148+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.153534+0000) 2019-10-13T09:16:41.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:41.152+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.154125+0000) 2019-10-13T09:16:41.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:41.152+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.154233+0000) 2019-10-13T09:16:41.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:41.152+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.154831+0000) 2019-10-13T09:16:41.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:41.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:41.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:41.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.160470+0000) 2019-10-13T09:16:41.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:41.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.657213+0000) 2019-10-13T09:16:41.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:41.657+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:41.658510+0000) 2019-10-13T09:16:43.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:43.968+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:43.972625+0000) 2019-10-13T09:16:44.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:44.380+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:44.383299+0000) 2019-10-13T09:16:46.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:46.156+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.159633+0000) 2019-10-13T09:16:46.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:46.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.160566+0000) 2019-10-13T09:16:46.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:46.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.161582+0000) 2019-10-13T09:16:46.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:46.160+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.162249+0000) 2019-10-13T09:16:46.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:46.160+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.162919+0000) 2019-10-13T09:16:46.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:46.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.162908+0000) 2019-10-13T09:16:46.164 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:46.160+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.163769+0000) 2019-10-13T09:16:46.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:46.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91348 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:46.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:46.165+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.170116+0000) 2019-10-13T09:16:47.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:46.993+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:46.998995+0000) 2019-10-13T09:16:47.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:47.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:47.512536+0000) 2019-10-13T09:16:47.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:47.608+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:47.610162+0000) 2019-10-13T09:16:47.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:47.632+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:47.635939+0000) 2019-10-13T09:16:51.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:51.164+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.167065+0000) 2019-10-13T09:16:51.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:51.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.168060+0000) 2019-10-13T09:16:51.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:51.161+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.168038+0000) 2019-10-13T09:16:51.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:51.164+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.168686+0000) 2019-10-13T09:16:51.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:51.164+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.169026+0000) 2019-10-13T09:16:51.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:51.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.170156+0000) 2019-10-13T09:16:51.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:51.168+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.170849+0000) 2019-10-13T09:16:51.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:51.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:51.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:51.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.175316+0000) 2019-10-13T09:16:51.621 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:16:51.621 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:51.626 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:16:51.626 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:16:51.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:51.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.657387+0000) 2019-10-13T09:16:51.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:51.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:51.658716+0000) 2019-10-13T09:16:53.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:53.972+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:53.972856+0000) 2019-10-13T09:16:54.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:54.380+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:54.383554+0000) 2019-10-13T09:16:56.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:56.172+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.174537+0000) 2019-10-13T09:16:56.176 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:56.172+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.175184+0000) 2019-10-13T09:16:56.176 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:16:56.169+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.175083+0000) 2019-10-13T09:16:56.179 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:16:56.172+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.177026+0000) 2019-10-13T09:16:56.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:16:56.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.177830+0000) 2019-10-13T09:16:56.180 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:56.176+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.178186+0000) 2019-10-13T09:16:56.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:16:56.176+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.178594+0000) 2019-10-13T09:16:56.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:16:56.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:16:56.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:56.177+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:56.182597+0000) 2019-10-13T09:16:57.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:16:57.001+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:57.005657+0000) 2019-10-13T09:16:57.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:16:57.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:57.512767+0000) 2019-10-13T09:16:57.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:16:57.608+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:57.610356+0000) 2019-10-13T09:16:57.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:16:57.632+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:16:57.636143+0000) 2019-10-13T09:17:01.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:01.180+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.183477+0000) 2019-10-13T09:17:01.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:01.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.184326+0000) 2019-10-13T09:17:01.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:01.180+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.184865+0000) 2019-10-13T09:17:01.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:01.180+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.185516+0000) 2019-10-13T09:17:01.187 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:01.181+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.186154+0000) 2019-10-13T09:17:01.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:01.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91440 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:01.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:01.184+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.188316+0000) 2019-10-13T09:17:01.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:01.184+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.188613+0000) 2019-10-13T09:17:01.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:01.185+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.191134+0000) 2019-10-13T09:17:01.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:01.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.657559+0000) 2019-10-13T09:17:01.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:01.657+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:01.658911+0000) 2019-10-13T09:17:03.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:03.972+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:03.973053+0000) 2019-10-13T09:17:04.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:04.380+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:04.383789+0000) 2019-10-13T09:17:06.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:06.188+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.190852+0000) 2019-10-13T09:17:06.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:06.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.191607+0000) 2019-10-13T09:17:06.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:06.188+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.191956+0000) 2019-10-13T09:17:06.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:06.189+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.192167+0000) 2019-10-13T09:17:06.193 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:06.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.192848+0000) 2019-10-13T09:17:06.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:06.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.193788+0000) 2019-10-13T09:17:06.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:06.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.194346+0000) 2019-10-13T09:17:06.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:06.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:06.200 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:06.193+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:06.199463+0000) 2019-10-13T09:17:07.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:07.009+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:07.012783+0000) 2019-10-13T09:17:07.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:07.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:07.512970+0000) 2019-10-13T09:17:07.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:07.608+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:07.610558+0000) 2019-10-13T09:17:07.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:07.632+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:07.636345+0000) 2019-10-13T09:17:11.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:11.196+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.200483+0000) 2019-10-13T09:17:11.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:11.196+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.201250+0000) 2019-10-13T09:17:11.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:11.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.202982+0000) 2019-10-13T09:17:11.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:11.200+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.203073+0000) 2019-10-13T09:17:11.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:11.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:11.205 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:11.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.204617+0000) 2019-10-13T09:17:11.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:11.201+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.204526+0000) 2019-10-13T09:17:11.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:11.204+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.205712+0000) 2019-10-13T09:17:11.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:11.201+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.208002+0000) 2019-10-13T09:17:11.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:11.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.657735+0000) 2019-10-13T09:17:11.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:11.657+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:11.659085+0000) 2019-10-13T09:17:13.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:13.968+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:13.973255+0000) 2019-10-13T09:17:14.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:14.380+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:14.383989+0000) 2019-10-13T09:17:16.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:16.204+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.207358+0000) 2019-10-13T09:17:16.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:16.204+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.208543+0000) 2019-10-13T09:17:16.210 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:16.205+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.208811+0000) 2019-10-13T09:17:16.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:16.204+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.209078+0000) 2019-10-13T09:17:16.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:16.208+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.209960+0000) 2019-10-13T09:17:16.211 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:16.208+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.209733+0000) 2019-10-13T09:17:16.212 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:16.208+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.212162+0000) 2019-10-13T09:17:16.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:16.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91527 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:16.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:16.213+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:16.218837+0000) 2019-10-13T09:17:17.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:17.013+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:17.019839+0000) 2019-10-13T09:17:17.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:17.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:17.513214+0000) 2019-10-13T09:17:17.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:17.608+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:17.610782+0000) 2019-10-13T09:17:17.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:17.636+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:17.636546+0000) 2019-10-13T09:17:21.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:21.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.215432+0000) 2019-10-13T09:17:21.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:21.212+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.216196+0000) 2019-10-13T09:17:21.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:21.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.216699+0000) 2019-10-13T09:17:21.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:21.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.217079+0000) 2019-10-13T09:17:21.218 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:21.216+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.218239+0000) 2019-10-13T09:17:21.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:21.216+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.218968+0000) 2019-10-13T09:17:21.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:21.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:21.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:21.216+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.219811+0000) 2019-10-13T09:17:21.225 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:21.221+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.224366+0000) 2019-10-13T09:17:21.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:21.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.657939+0000) 2019-10-13T09:17:21.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:21.653+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:21.659224+0000) 2019-10-13T09:17:21.743 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:17:21.744 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:21.747 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:17:21.748 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:23.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:23.968+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:23.973485+0000) 2019-10-13T09:17:24.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:24.380+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:24.384190+0000) 2019-10-13T09:17:26.223 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:26.216+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.222732+0000) 2019-10-13T09:17:26.224 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:26.220+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.223754+0000) 2019-10-13T09:17:26.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:26.220+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.223936+0000) 2019-10-13T09:17:26.225 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:26.220+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.224375+0000) 2019-10-13T09:17:26.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:26.220+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.224678+0000) 2019-10-13T09:17:26.226 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:26.220+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.224588+0000) 2019-10-13T09:17:26.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:26.220+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.225570+0000) 2019-10-13T09:17:26.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:26.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:26.232 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:26.224+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:26.231030+0000) 2019-10-13T09:17:27.028 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:27.020+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:27.026876+0000) 2019-10-13T09:17:27.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:27.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:27.513443+0000) 2019-10-13T09:17:27.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:27.608+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:27.610983+0000) 2019-10-13T09:17:27.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:27.636+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:27.636759+0000) 2019-10-13T09:17:31.230 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:31.228+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.229902+0000) 2019-10-13T09:17:31.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:31.224+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.229869+0000) 2019-10-13T09:17:31.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:31.228+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.230815+0000) 2019-10-13T09:17:31.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:31.228+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.231910+0000) 2019-10-13T09:17:31.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:31.228+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.232199+0000) 2019-10-13T09:17:31.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:31.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.232471+0000) 2019-10-13T09:17:31.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:31.232+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.233893+0000) 2019-10-13T09:17:31.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:31.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:31.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:31.232+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.237744+0000) 2019-10-13T09:17:31.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:31.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.658107+0000) 2019-10-13T09:17:31.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:31.656+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:31.659398+0000) 2019-10-13T09:17:33.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:33.972+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:33.973691+0000) 2019-10-13T09:17:34.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:34.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:34.384396+0000) 2019-10-13T09:17:36.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:36.231+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.237679+0000) 2019-10-13T09:17:36.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:36.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.238310+0000) 2019-10-13T09:17:36.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:36.235+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.238599+0000) 2019-10-13T09:17:36.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:36.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.239518+0000) 2019-10-13T09:17:36.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:36.235+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.240041+0000) 2019-10-13T09:17:36.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:36.236+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.240161+0000) 2019-10-13T09:17:36.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:36.235+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.241309+0000) 2019-10-13T09:17:36.242 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:36.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:36.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:36.244+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:36.249335+0000) 2019-10-13T09:17:37.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:37.032+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:37.034144+0000) 2019-10-13T09:17:37.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:37.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:37.513690+0000) 2019-10-13T09:17:37.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:37.607+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:37.611205+0000) 2019-10-13T09:17:37.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:37.635+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:37.636960+0000) 2019-10-13T09:17:41.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:41.239+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.245177+0000) 2019-10-13T09:17:41.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:41.243+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.247629+0000) 2019-10-13T09:17:41.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:41.240+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.247641+0000) 2019-10-13T09:17:41.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:41.243+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.248602+0000) 2019-10-13T09:17:41.250 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:41.247+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.249721+0000) 2019-10-13T09:17:41.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:41.247+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.250418+0000) 2019-10-13T09:17:41.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:41.247+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.250445+0000) 2019-10-13T09:17:41.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:41.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:41.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:41.248+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.255539+0000) 2019-10-13T09:17:41.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:41.656+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.659528+0000) 2019-10-13T09:17:41.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:41.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:41.658340+0000) 2019-10-13T09:17:43.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:43.971+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:43.973926+0000) 2019-10-13T09:17:44.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:44.383+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:44.384619+0000) 2019-10-13T09:17:46.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:46.251+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.253917+0000) 2019-10-13T09:17:46.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:46.251+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.254887+0000) 2019-10-13T09:17:46.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:46.252+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.256201+0000) 2019-10-13T09:17:46.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:46.251+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.256996+0000) 2019-10-13T09:17:46.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:46.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:46.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:46.255+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.259219+0000) 2019-10-13T09:17:46.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:46.255+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.259467+0000) 2019-10-13T09:17:46.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:46.256+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.263289+0000) 2019-10-13T09:17:46.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:46.259+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:46.261805+0000) 2019-10-13T09:17:47.042 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:47.040+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:47.041252+0000) 2019-10-13T09:17:47.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:47.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:47.513932+0000) 2019-10-13T09:17:47.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:47.607+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:47.611419+0000) 2019-10-13T09:17:47.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:47.631+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:47.637191+0000) 2019-10-13T09:17:51.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:51.259+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.262243+0000) 2019-10-13T09:17:51.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:51.259+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.262473+0000) 2019-10-13T09:17:51.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:51.256+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.262464+0000) 2019-10-13T09:17:51.264 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:51.259+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.263281+0000) 2019-10-13T09:17:51.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:51.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.264635+0000) 2019-10-13T09:17:51.265 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:51.259+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.264951+0000) 2019-10-13T09:17:51.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:51.259+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.265652+0000) 2019-10-13T09:17:51.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:51.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:51.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:51.264+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.270077+0000) 2019-10-13T09:17:51.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:51.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.658558+0000) 2019-10-13T09:17:51.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:51.656+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:51.659695+0000) 2019-10-13T09:17:51.876 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:17:51.876 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:51.880 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:17:51.881 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:17:53.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:53.971+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:53.974151+0000) 2019-10-13T09:17:54.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:54.379+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:54.384838+0000) 2019-10-13T09:17:56.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:56.267+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.271253+0000) 2019-10-13T09:17:56.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:17:56.267+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.271944+0000) 2019-10-13T09:17:56.274 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:56.267+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.273412+0000) 2019-10-13T09:17:56.274 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:17:56.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.273503+0000) 2019-10-13T09:17:56.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:17:56.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:17:56.281 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:56.271+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.274483+0000) 2019-10-13T09:17:56.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:17:56.271+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.275416+0000) 2019-10-13T09:17:56.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:17:56.271+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.277425+0000) 2019-10-13T09:17:56.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:56.276+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:56.282227+0000) 2019-10-13T09:17:57.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:17:57.044+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:57.048325+0000) 2019-10-13T09:17:57.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:17:57.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:57.514162+0000) 2019-10-13T09:17:57.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:17:57.607+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:57.611654+0000) 2019-10-13T09:17:57.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:17:57.631+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:17:57.637405+0000) 2019-10-13T09:18:01.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:01.271+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.277121+0000) 2019-10-13T09:18:01.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:01.271+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.277476+0000) 2019-10-13T09:18:01.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:01.275+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.279177+0000) 2019-10-13T09:18:01.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:01.276+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.280281+0000) 2019-10-13T09:18:01.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:01.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.281071+0000) 2019-10-13T09:18:01.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:01.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:01.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:01.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.281887+0000) 2019-10-13T09:18:01.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:01.279+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.282638+0000) 2019-10-13T09:18:01.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:01.280+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.287471+0000) 2019-10-13T09:18:01.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:01.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.658691+0000) 2019-10-13T09:18:01.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:01.656+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:01.659869+0000) 2019-10-13T09:18:03.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:03.971+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:03.974385+0000) 2019-10-13T09:18:04.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:04.379+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:04.385036+0000) 2019-10-13T09:18:06.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:06.279+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.284574+0000) 2019-10-13T09:18:06.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:06.280+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.284815+0000) 2019-10-13T09:18:06.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:06.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.285667+0000) 2019-10-13T09:18:06.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:06.283+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.285800+0000) 2019-10-13T09:18:06.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:06.283+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.287179+0000) 2019-10-13T09:18:06.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:06.283+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.286551+0000) 2019-10-13T09:18:06.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:06.283+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.287641+0000) 2019-10-13T09:18:06.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:06.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91827 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:06.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:06.288+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:06.294522+0000) 2019-10-13T09:18:07.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:07.056+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:07.059622+0000) 2019-10-13T09:18:07.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:07.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:07.514384+0000) 2019-10-13T09:18:07.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:07.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:07.611861+0000) 2019-10-13T09:18:07.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:07.635+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:07.637601+0000) 2019-10-13T09:18:11.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:11.287+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.292955+0000) 2019-10-13T09:18:11.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:11.287+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.292919+0000) 2019-10-13T09:18:11.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:11.291+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.294562+0000) 2019-10-13T09:18:11.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:11.291+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.294925+0000) 2019-10-13T09:18:11.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:11.288+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.295044+0000) 2019-10-13T09:18:11.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:11.291+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.296221+0000) 2019-10-13T09:18:11.297 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:11.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:11.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:11.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.296910+0000) 2019-10-13T09:18:11.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:11.300+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.304225+0000) 2019-10-13T09:18:11.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:11.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.658819+0000) 2019-10-13T09:18:11.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:11.652+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:11.660044+0000) 2019-10-13T09:18:13.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:13.971+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:13.974589+0000) 2019-10-13T09:18:14.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:14.383+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:14.385256+0000) 2019-10-13T09:18:16.303 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:16.296+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.300938+0000) 2019-10-13T09:18:16.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:16.295+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.300920+0000) 2019-10-13T09:18:16.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:16.299+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.302682+0000) 2019-10-13T09:18:16.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:16.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.303222+0000) 2019-10-13T09:18:16.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:16.299+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.303628+0000) 2019-10-13T09:18:16.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:16.299+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.304142+0000) 2019-10-13T09:18:16.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:16.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:16.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:16.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.304467+0000) 2019-10-13T09:18:16.309 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:16.304+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:16.308085+0000) 2019-10-13T09:18:17.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:17.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:17.066218+0000) 2019-10-13T09:18:17.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:17.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:17.514622+0000) 2019-10-13T09:18:17.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:17.607+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:17.612078+0000) 2019-10-13T09:18:17.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:17.635+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:17.637799+0000) 2019-10-13T09:18:21.310 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:21.304+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.308676+0000) 2019-10-13T09:18:21.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:21.303+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.309389+0000) 2019-10-13T09:18:21.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:21.307+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.309835+0000) 2019-10-13T09:18:21.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:21.303+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.309452+0000) 2019-10-13T09:18:21.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:21.307+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.310232+0000) 2019-10-13T09:18:21.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:21.307+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.310594+0000) 2019-10-13T09:18:21.312 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:21.307+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.311011+0000) 2019-10-13T09:18:21.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:21.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:21.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:21.312+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.318878+0000) 2019-10-13T09:18:21.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:21.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.659030+0000) 2019-10-13T09:18:21.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:21.660+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:21.660245+0000) 2019-10-13T09:18:21.991 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:18:21.992 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:21.996 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:18:21.996 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:23.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:23.971+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:23.974822+0000) 2019-10-13T09:18:24.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:24.383+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:24.385462+0000) 2019-10-13T09:18:26.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:26.311+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.316283+0000) 2019-10-13T09:18:26.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:26.311+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.316907+0000) 2019-10-13T09:18:26.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:26.315+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.319002+0000) 2019-10-13T09:18:26.320 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:26.315+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.319557+0000) 2019-10-13T09:18:26.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:26.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:26.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:26.315+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.321046+0000) 2019-10-13T09:18:26.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:26.320+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.322097+0000) 2019-10-13T09:18:26.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:26.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.322915+0000) 2019-10-13T09:18:26.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:26.328+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:26.329361+0000) 2019-10-13T09:18:27.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:27.072+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:27.072821+0000) 2019-10-13T09:18:27.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:27.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:27.514826+0000) 2019-10-13T09:18:27.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:27.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:27.612303+0000) 2019-10-13T09:18:27.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:27.635+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:27.638005+0000) 2019-10-13T09:18:31.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:31.323+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.326366+0000) 2019-10-13T09:18:31.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:31.323+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.326649+0000) 2019-10-13T09:18:31.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:31.323+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.328654+0000) 2019-10-13T09:18:31.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:31.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:31.330 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:31.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.328987+0000) 2019-10-13T09:18:31.331 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:31.327+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.329388+0000) 2019-10-13T09:18:31.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:31.327+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.330078+0000) 2019-10-13T09:18:31.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:31.327+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.331998+0000) 2019-10-13T09:18:31.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:31.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.336822+0000) 2019-10-13T09:18:31.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:31.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.659243+0000) 2019-10-13T09:18:31.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:31.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:31.660373+0000) 2019-10-13T09:18:33.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:33.970+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:33.975069+0000) 2019-10-13T09:18:34.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:34.378+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:34.385693+0000) 2019-10-13T09:18:36.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:36.326+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.332260+0000) 2019-10-13T09:18:36.335 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:36.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.334529+0000) 2019-10-13T09:18:36.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:36.330+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.334621+0000) 2019-10-13T09:18:36.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:36.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.336071+0000) 2019-10-13T09:18:36.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:36.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:36.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:36.335+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.336668+0000) 2019-10-13T09:18:36.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:36.334+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.337938+0000) 2019-10-13T09:18:36.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:36.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.337808+0000) 2019-10-13T09:18:36.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:36.334+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:36.339170+0000) 2019-10-13T09:18:37.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:37.075+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:37.079258+0000) 2019-10-13T09:18:37.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:37.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:37.515029+0000) 2019-10-13T09:18:37.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:37.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:37.612523+0000) 2019-10-13T09:18:37.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:37.634+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:37.638234+0000) 2019-10-13T09:18:41.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:41.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.339228+0000) 2019-10-13T09:18:41.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:41.334+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.339847+0000) 2019-10-13T09:18:41.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:41.339+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.340533+0000) 2019-10-13T09:18:41.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:41.334+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.341362+0000) 2019-10-13T09:18:41.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:41.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.342464+0000) 2019-10-13T09:18:41.343 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:41.338+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.342690+0000) 2019-10-13T09:18:41.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:41.338+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.343469+0000) 2019-10-13T09:18:41.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:41.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:41.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:41.343+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.348057+0000) 2019-10-13T09:18:41.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:41.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.659439+0000) 2019-10-13T09:18:41.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:41.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:41.660547+0000) 2019-10-13T09:18:43.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:43.970+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:43.975269+0000) 2019-10-13T09:18:44.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:44.382+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:44.385939+0000) 2019-10-13T09:18:46.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:46.342+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.348197+0000) 2019-10-13T09:18:46.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:46.346+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.352141+0000) 2019-10-13T09:18:46.353 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:46.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:46.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:46.346+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.353306+0000) 2019-10-13T09:18:46.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:46.351+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.353209+0000) 2019-10-13T09:18:46.354 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:46.350+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.353927+0000) 2019-10-13T09:18:46.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:46.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.354356+0000) 2019-10-13T09:18:46.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:46.350+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.354498+0000) 2019-10-13T09:18:46.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:46.363+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:46.365030+0000) 2019-10-13T09:18:47.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:47.083+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:47.085782+0000) 2019-10-13T09:18:47.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:47.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:47.515227+0000) 2019-10-13T09:18:47.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:47.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:47.612719+0000) 2019-10-13T09:18:47.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:47.634+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:47.638440+0000) 2019-10-13T09:18:51.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:51.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.356271+0000) 2019-10-13T09:18:51.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:51.350+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.357008+0000) 2019-10-13T09:18:51.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:51.355+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.358091+0000) 2019-10-13T09:18:51.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:51.354+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.358729+0000) 2019-10-13T09:18:51.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:51.354+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.359166+0000) 2019-10-13T09:18:51.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:51.354+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.359705+0000) 2019-10-13T09:18:51.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:51.354+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.360219+0000) 2019-10-13T09:18:51.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:51.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:51.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:51.363+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.364737+0000) 2019-10-13T09:18:51.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:51.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.659648+0000) 2019-10-13T09:18:51.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:51.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:51.660765+0000) 2019-10-13T09:18:52.123 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:18:52.123 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:52.127 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:18:52.128 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:18:53.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:53.974+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:53.975486+0000) 2019-10-13T09:18:54.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:54.382+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:54.386140+0000) 2019-10-13T09:18:56.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:18:56.358+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.363433+0000) 2019-10-13T09:18:56.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:56.358+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.364333+0000) 2019-10-13T09:18:56.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:18:56.362+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.367666+0000) 2019-10-13T09:18:56.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:18:56.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:18:56.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:18:56.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.369650+0000) 2019-10-13T09:18:56.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:18:56.366+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.370474+0000) 2019-10-13T09:18:56.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:56.366+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.370892+0000) 2019-10-13T09:18:56.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:56.366+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.372044+0000) 2019-10-13T09:18:56.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:56.379+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:56.383391+0000) 2019-10-13T09:18:57.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:18:57.095+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:57.097375+0000) 2019-10-13T09:18:57.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:18:57.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:57.515428+0000) 2019-10-13T09:18:57.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:18:57.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:57.612911+0000) 2019-10-13T09:18:57.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:18:57.638+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:18:57.638679+0000) 2019-10-13T09:19:01.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:01.366+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.372570+0000) 2019-10-13T09:19:01.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:01.370+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.374988+0000) 2019-10-13T09:19:01.376 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:01.370+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.375803+0000) 2019-10-13T09:19:01.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:01.371+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.375791+0000) 2019-10-13T09:19:01.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:01.370+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.376742+0000) 2019-10-13T09:19:01.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:01.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:01.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:01.370+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.377184+0000) 2019-10-13T09:19:01.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:01.374+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.377730+0000) 2019-10-13T09:19:01.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:01.379+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.380617+0000) 2019-10-13T09:19:01.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:01.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.659799+0000) 2019-10-13T09:19:01.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:01.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:01.660935+0000) 2019-10-13T09:19:03.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:03.974+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:03.975685+0000) 2019-10-13T09:19:04.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:04.382+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:04.386400+0000) 2019-10-13T09:19:06.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:06.374+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.380217+0000) 2019-10-13T09:19:06.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:06.374+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.380485+0000) 2019-10-13T09:19:06.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:06.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.381136+0000) 2019-10-13T09:19:06.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:06.374+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.381495+0000) 2019-10-13T09:19:06.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:06.378+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.382481+0000) 2019-10-13T09:19:06.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:06.379+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.382688+0000) 2019-10-13T09:19:06.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:06.378+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.383480+0000) 2019-10-13T09:19:06.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:06.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:06.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:06.387+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:06.390674+0000) 2019-10-13T09:19:07.105 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:07.099+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:07.103783+0000) 2019-10-13T09:19:07.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:07.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:07.515636+0000) 2019-10-13T09:19:07.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:07.610+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:07.613111+0000) 2019-10-13T09:19:07.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:07.634+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:07.638876+0000) 2019-10-13T09:19:11.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:11.382+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.388549+0000) 2019-10-13T09:19:11.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:11.382+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.389320+0000) 2019-10-13T09:19:11.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:11.386+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.389866+0000) 2019-10-13T09:19:11.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:11.386+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.391031+0000) 2019-10-13T09:19:11.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:11.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.391592+0000) 2019-10-13T09:19:11.392 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:11.387+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.391578+0000) 2019-10-13T09:19:11.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:11.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:11.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:11.386+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.393158+0000) 2019-10-13T09:19:11.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:11.391+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.394810+0000) 2019-10-13T09:19:11.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:11.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.659942+0000) 2019-10-13T09:19:11.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:11.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:11.661107+0000) 2019-10-13T09:19:13.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:13.970+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:13.975948+0000) 2019-10-13T09:19:14.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:14.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:14.386615+0000) 2019-10-13T09:19:16.397 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:16.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.396440+0000) 2019-10-13T09:19:16.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:16.390+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.396857+0000) 2019-10-13T09:19:16.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:16.395+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.397069+0000) 2019-10-13T09:19:16.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:16.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.398452+0000) 2019-10-13T09:19:16.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:16.394+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.398512+0000) 2019-10-13T09:19:16.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:16.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.399086+0000) 2019-10-13T09:19:16.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:16.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:16.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:16.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.400767+0000) 2019-10-13T09:19:16.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:16.403+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:16.406123+0000) 2019-10-13T09:19:17.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:17.107+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:17.110143+0000) 2019-10-13T09:19:17.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:17.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:17.515837+0000) 2019-10-13T09:19:17.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:17.606+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:17.613324+0000) 2019-10-13T09:19:17.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:17.638+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:17.639094+0000) 2019-10-13T09:19:21.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:21.399+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.402548+0000) 2019-10-13T09:19:21.405 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:21.398+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.404344+0000) 2019-10-13T09:19:21.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:21.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.404970+0000) 2019-10-13T09:19:21.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:21.398+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.405428+0000) 2019-10-13T09:19:21.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:21.398+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.405640+0000) 2019-10-13T09:19:21.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:21.402+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.406648+0000) 2019-10-13T09:19:21.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:21.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.407135+0000) 2019-10-13T09:19:21.408 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:21.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:21.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:21.411+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.414882+0000) 2019-10-13T09:19:21.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:21.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.660154+0000) 2019-10-13T09:19:21.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:21.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:21.661273+0000) 2019-10-13T09:19:22.173 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:19:22.174 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:22.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:19:22.177 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:23.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:23.974+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:23.976112+0000) 2019-10-13T09:19:24.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:24.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:24.386831+0000) 2019-10-13T09:19:26.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:26.407+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.410607+0000) 2019-10-13T09:19:26.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:26.406+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.412103+0000) 2019-10-13T09:19:26.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:26.406+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.413096+0000) 2019-10-13T09:19:26.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:26.410+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.414761+0000) 2019-10-13T09:19:26.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:26.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:26.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:26.410+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.415991+0000) 2019-10-13T09:19:26.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:26.410+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.416349+0000) 2019-10-13T09:19:26.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:26.410+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.416808+0000) 2019-10-13T09:19:26.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:26.419+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:26.422621+0000) 2019-10-13T09:19:27.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:27.115+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:27.116514+0000) 2019-10-13T09:19:27.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:27.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:27.515989+0000) 2019-10-13T09:19:27.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:27.610+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:27.613552+0000) 2019-10-13T09:19:27.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:27.638+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:27.639306+0000) 2019-10-13T09:19:31.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:31.415+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.418627+0000) 2019-10-13T09:19:31.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:31.413+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.419409+0000) 2019-10-13T09:19:31.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:31.413+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.420092+0000) 2019-10-13T09:19:31.422 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:31.413+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.420530+0000) 2019-10-13T09:19:31.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:31.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.421025+0000) 2019-10-13T09:19:31.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:31.413+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.421328+0000) 2019-10-13T09:19:31.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:31.417+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.421930+0000) 2019-10-13T09:19:31.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:31.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:31.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:31.427+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.429948+0000) 2019-10-13T09:19:31.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:31.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.660333+0000) 2019-10-13T09:19:31.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:31.659+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:31.661480+0000) 2019-10-13T09:19:33.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:33.969+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:33.976277+0000) 2019-10-13T09:19:34.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:34.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:34.387030+0000) 2019-10-13T09:19:36.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:36.422+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.426218+0000) 2019-10-13T09:19:36.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:36.421+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.427560+0000) 2019-10-13T09:19:36.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:36.421+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.428397+0000) 2019-10-13T09:19:36.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:36.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:36.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:36.425+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.431996+0000) 2019-10-13T09:19:36.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:36.425+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.432244+0000) 2019-10-13T09:19:36.433 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:36.425+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.432817+0000) 2019-10-13T09:19:36.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:36.425+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.433228+0000) 2019-10-13T09:19:36.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:36.430+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:36.435136+0000) 2019-10-13T09:19:37.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:37.118+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:37.123097+0000) 2019-10-13T09:19:37.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:37.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:37.516186+0000) 2019-10-13T09:19:37.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:37.609+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:37.613771+0000) 2019-10-13T09:19:37.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:37.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:37.639507+0000) 2019-10-13T09:19:41.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:41.429+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.434591+0000) 2019-10-13T09:19:41.435 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:41.430+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.434766+0000) 2019-10-13T09:19:41.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:41.429+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.435874+0000) 2019-10-13T09:19:41.438 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:41.429+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.437084+0000) 2019-10-13T09:19:41.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:41.429+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.437495+0000) 2019-10-13T09:19:41.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:41.433+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.437976+0000) 2019-10-13T09:19:41.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:41.433+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.438287+0000) 2019-10-13T09:19:41.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:41.433+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:41.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:41.438+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.441905+0000) 2019-10-13T09:19:41.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:41.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.660513+0000) 2019-10-13T09:19:41.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:41.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:41.661649+0000) 2019-10-13T09:19:43.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:43.973+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:43.976521+0000) 2019-10-13T09:19:44.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:44.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:44.387260+0000) 2019-10-13T09:19:46.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:46.441+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.442191+0000) 2019-10-13T09:19:46.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:46.441+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.443042+0000) 2019-10-13T09:19:46.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:46.441+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.443294+0000) 2019-10-13T09:19:46.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:46.442+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.445187+0000) 2019-10-13T09:19:46.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:46.445+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.445799+0000) 2019-10-13T09:19:46.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:46.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92427 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:46.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:46.445+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.446720+0000) 2019-10-13T09:19:46.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:46.445+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.446581+0000) 2019-10-13T09:19:46.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:46.450+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:46.454468+0000) 2019-10-13T09:19:47.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:47.126+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:47.129579+0000) 2019-10-13T09:19:47.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:47.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:47.516387+0000) 2019-10-13T09:19:47.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:47.613+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:47.613968+0000) 2019-10-13T09:19:47.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:47.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:47.639738+0000) 2019-10-13T09:19:51.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:51.445+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.449464+0000) 2019-10-13T09:19:51.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:51.445+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.449558+0000) 2019-10-13T09:19:51.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:51.445+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.449416+0000) 2019-10-13T09:19:51.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:51.449+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.449852+0000) 2019-10-13T09:19:51.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:51.446+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.450031+0000) 2019-10-13T09:19:51.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:51.449+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.450488+0000) 2019-10-13T09:19:51.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:51.449+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.451364+0000) 2019-10-13T09:19:51.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:51.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:51.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:51.454+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.456581+0000) 2019-10-13T09:19:51.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:51.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.660725+0000) 2019-10-13T09:19:51.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:51.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:51.661834+0000) 2019-10-13T09:19:52.304 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:19:52.304 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:52.309 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:19:52.309 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:19:53.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:53.973+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:53.976733+0000) 2019-10-13T09:19:54.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:54.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:54.387481+0000) 2019-10-13T09:19:56.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:19:56.453+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.456419+0000) 2019-10-13T09:19:56.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:19:56.454+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.456584+0000) 2019-10-13T09:19:56.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:56.453+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.457378+0000) 2019-10-13T09:19:56.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:19:56.457+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.458863+0000) 2019-10-13T09:19:56.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:56.457+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.459116+0000) 2019-10-13T09:19:56.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:56.457+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.459439+0000) 2019-10-13T09:19:56.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:19:56.457+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.460282+0000) 2019-10-13T09:19:56.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:19:56.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:19:56.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:56.462+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:56.464671+0000) 2019-10-13T09:19:57.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:19:57.130+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:57.136054+0000) 2019-10-13T09:19:57.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:19:57.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:57.516616+0000) 2019-10-13T09:19:57.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:19:57.613+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:57.614175+0000) 2019-10-13T09:19:57.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:19:57.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:19:57.639935+0000) 2019-10-13T09:20:01.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:01.458+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.463216+0000) 2019-10-13T09:20:01.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:01.461+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.464370+0000) 2019-10-13T09:20:01.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:01.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.464543+0000) 2019-10-13T09:20:01.465 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:01.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.465105+0000) 2019-10-13T09:20:01.466 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:01.465+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.466078+0000) 2019-10-13T09:20:01.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:01.465+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.466381+0000) 2019-10-13T09:20:01.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:01.465+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.467038+0000) 2019-10-13T09:20:01.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:01.465+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:01.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:01.470+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.472871+0000) 2019-10-13T09:20:01.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:01.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.660934+0000) 2019-10-13T09:20:01.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:01.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:01.661961+0000) 2019-10-13T09:20:03.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:03.973+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:03.976932+0000) 2019-10-13T09:20:04.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:04.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:04.387697+0000) 2019-10-13T09:20:06.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:06.469+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.471593+0000) 2019-10-13T09:20:06.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:06.469+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.472054+0000) 2019-10-13T09:20:06.473 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:06.469+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.472238+0000) 2019-10-13T09:20:06.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:06.469+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.472561+0000) 2019-10-13T09:20:06.474 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:06.470+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.472981+0000) 2019-10-13T09:20:06.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:06.473+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.473904+0000) 2019-10-13T09:20:06.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:06.473+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.474837+0000) 2019-10-13T09:20:06.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:06.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92546 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:06.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:06.478+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:06.482826+0000) 2019-10-13T09:20:07.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:07.142+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.142708+0000) 2019-10-13T09:20:07.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:07.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.516834+0000) 2019-10-13T09:20:07.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:07.613+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.614395+0000) 2019-10-13T09:20:07.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:07.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:07.640136+0000) 2019-10-13T09:20:11.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:11.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.479192+0000) 2019-10-13T09:20:11.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:11.477+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.479676+0000) 2019-10-13T09:20:11.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:11.474+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.479652+0000) 2019-10-13T09:20:11.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:11.477+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.481250+0000) 2019-10-13T09:20:11.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:11.477+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.481006+0000) 2019-10-13T09:20:11.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:11.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.481489+0000) 2019-10-13T09:20:11.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:11.481+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.481861+0000) 2019-10-13T09:20:11.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:11.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92577 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:11.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:11.486+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.488893+0000) 2019-10-13T09:20:11.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:11.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.661110+0000) 2019-10-13T09:20:11.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:11.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:11.662116+0000) 2019-10-13T09:20:13.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:13.973+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:13.977132+0000) 2019-10-13T09:20:14.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:14.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:14.387897+0000) 2019-10-13T09:20:16.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:16.485+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.488013+0000) 2019-10-13T09:20:16.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:16.485+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.489010+0000) 2019-10-13T09:20:16.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:16.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:16.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:16.489+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.489735+0000) 2019-10-13T09:20:16.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:16.486+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.489625+0000) 2019-10-13T09:20:16.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:16.489+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.490378+0000) 2019-10-13T09:20:16.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:16.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.492233+0000) 2019-10-13T09:20:16.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:16.489+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.492430+0000) 2019-10-13T09:20:16.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:16.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:16.496389+0000) 2019-10-13T09:20:17.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:17.150+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.154048+0000) 2019-10-13T09:20:17.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:17.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.517033+0000) 2019-10-13T09:20:17.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:17.613+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.614616+0000) 2019-10-13T09:20:17.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:17.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:17.640284+0000) 2019-10-13T09:20:21.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:21.490+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.492434+0000) 2019-10-13T09:20:21.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:21.493+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.493908+0000) 2019-10-13T09:20:21.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:21.493+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.494624+0000) 2019-10-13T09:20:21.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:21.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.496321+0000) 2019-10-13T09:20:21.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:21.493+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.496575+0000) 2019-10-13T09:20:21.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:21.493+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.497099+0000) 2019-10-13T09:20:21.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:21.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:21.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:21.497+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.497975+0000) 2019-10-13T09:20:21.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:21.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.499040+0000) 2019-10-13T09:20:21.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:21.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.661304+0000) 2019-10-13T09:20:21.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:21.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:21.662242+0000) 2019-10-13T09:20:22.418 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:20:22.418 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:22.421 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:20:22.422 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:23.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:23.973+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:23.977354+0000) 2019-10-13T09:20:24.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:24.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:24.388049+0000) 2019-10-13T09:20:26.501 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:26.496+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.501029+0000) 2019-10-13T09:20:26.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:26.500+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.502847+0000) 2019-10-13T09:20:26.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:26.500+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.504328+0000) 2019-10-13T09:20:26.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:26.502+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.504212+0000) 2019-10-13T09:20:26.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:26.500+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.505096+0000) 2019-10-13T09:20:26.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:26.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92666 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:26.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:26.504+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.505791+0000) 2019-10-13T09:20:26.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:26.504+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.506121+0000) 2019-10-13T09:20:26.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:26.502+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:26.507701+0000) 2019-10-13T09:20:27.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:27.158+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.160126+0000) 2019-10-13T09:20:27.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:27.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.517244+0000) 2019-10-13T09:20:27.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:27.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.614840+0000) 2019-10-13T09:20:27.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:27.636+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:27.640482+0000) 2019-10-13T09:20:31.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:31.504+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.508719+0000) 2019-10-13T09:20:31.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:31.504+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.509217+0000) 2019-10-13T09:20:31.511 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:31.508+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.510407+0000) 2019-10-13T09:20:31.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:31.508+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.510534+0000) 2019-10-13T09:20:31.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:31.508+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.510816+0000) 2019-10-13T09:20:31.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:31.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.510790+0000) 2019-10-13T09:20:31.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:31.508+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.511775+0000) 2019-10-13T09:20:31.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:31.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:31.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:31.510+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.515923+0000) 2019-10-13T09:20:31.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:31.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.661512+0000) 2019-10-13T09:20:31.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:31.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:31.662388+0000) 2019-10-13T09:20:33.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:33.972+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:33.977507+0000) 2019-10-13T09:20:34.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:34.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:34.388271+0000) 2019-10-13T09:20:36.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:36.512+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.516702+0000) 2019-10-13T09:20:36.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:36.512+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.516904+0000) 2019-10-13T09:20:36.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:36.516+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.520021+0000) 2019-10-13T09:20:36.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:36.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:36.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:36.520+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.521784+0000) 2019-10-13T09:20:36.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:36.516+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.521413+0000) 2019-10-13T09:20:36.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:36.518+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.521803+0000) 2019-10-13T09:20:36.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:36.520+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.522509+0000) 2019-10-13T09:20:36.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:36.522+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:36.525094+0000) 2019-10-13T09:20:37.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:37.166+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.171246+0000) 2019-10-13T09:20:37.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:37.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.517442+0000) 2019-10-13T09:20:37.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:37.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.615028+0000) 2019-10-13T09:20:37.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:37.636+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:37.640713+0000) 2019-10-13T09:20:41.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:41.520+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.524685+0000) 2019-10-13T09:20:41.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:41.524+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.526023+0000) 2019-10-13T09:20:41.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:41.524+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.526611+0000) 2019-10-13T09:20:41.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:41.524+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:41.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:41.524+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.529677+0000) 2019-10-13T09:20:41.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:41.528+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.529814+0000) 2019-10-13T09:20:41.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:41.528+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.530335+0000) 2019-10-13T09:20:41.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:41.526+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.530618+0000) 2019-10-13T09:20:41.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:41.530+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.535443+0000) 2019-10-13T09:20:41.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:41.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.661778+0000) 2019-10-13T09:20:41.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:41.658+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:41.662535+0000) 2019-10-13T09:20:43.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:43.976+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:43.977705+0000) 2019-10-13T09:20:44.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:44.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:44.388473+0000) 2019-10-13T09:20:46.533 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:46.528+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.532623+0000) 2019-10-13T09:20:46.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:46.529+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.533411+0000) 2019-10-13T09:20:46.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:46.532+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.536145+0000) 2019-10-13T09:20:46.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:46.532+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:46.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:46.532+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.537539+0000) 2019-10-13T09:20:46.539 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:46.536+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.538076+0000) 2019-10-13T09:20:46.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:46.536+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.538213+0000) 2019-10-13T09:20:46.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:46.536+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.538385+0000) 2019-10-13T09:20:46.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:46.537+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:46.540114+0000) 2019-10-13T09:20:47.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:47.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.177757+0000) 2019-10-13T09:20:47.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:47.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.517621+0000) 2019-10-13T09:20:47.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:47.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.615225+0000) 2019-10-13T09:20:47.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:47.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:47.640910+0000) 2019-10-13T09:20:51.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:51.536+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.540683+0000) 2019-10-13T09:20:51.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:51.536+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.541397+0000) 2019-10-13T09:20:51.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:51.540+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.542065+0000) 2019-10-13T09:20:51.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:51.540+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.541969+0000) 2019-10-13T09:20:51.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:51.540+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.542617+0000) 2019-10-13T09:20:51.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:51.540+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.542922+0000) 2019-10-13T09:20:51.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:51.537+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.543132+0000) 2019-10-13T09:20:51.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:51.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:51.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:51.545+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.549940+0000) 2019-10-13T09:20:51.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:51.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.661994+0000) 2019-10-13T09:20:51.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:51.661+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:51.662682+0000) 2019-10-13T09:20:52.465 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:20:52.465 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:52.469 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:20:52.469 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:20:53.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:53.976+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:53.977972+0000) 2019-10-13T09:20:54.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:54.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:54.388703+0000) 2019-10-13T09:20:56.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:56.544+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.548483+0000) 2019-10-13T09:20:56.549 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:20:56.544+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.549118+0000) 2019-10-13T09:20:56.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:20:56.545+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.549541+0000) 2019-10-13T09:20:56.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:56.548+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.550487+0000) 2019-10-13T09:20:56.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:56.548+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.550857+0000) 2019-10-13T09:20:56.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:20:56.548+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.551261+0000) 2019-10-13T09:20:56.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:20:56.548+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.551618+0000) 2019-10-13T09:20:56.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:20:56.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:20:56.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:56.549+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:56.555075+0000) 2019-10-13T09:20:57.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:20:57.181+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.184190+0000) 2019-10-13T09:20:57.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:20:57.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.517819+0000) 2019-10-13T09:20:57.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:20:57.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.615426+0000) 2019-10-13T09:20:57.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:20:57.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:20:57.641111+0000) 2019-10-13T09:21:01.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:01.552+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.555765+0000) 2019-10-13T09:21:01.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:01.552+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.556240+0000) 2019-10-13T09:21:01.557 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:01.552+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.556927+0000) 2019-10-13T09:21:01.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:01.552+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.557431+0000) 2019-10-13T09:21:01.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:01.556+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.558109+0000) 2019-10-13T09:21:01.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:01.553+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.557873+0000) 2019-10-13T09:21:01.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:01.556+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.559656+0000) 2019-10-13T09:21:01.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:01.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92878 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:01.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:01.557+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.561901+0000) 2019-10-13T09:21:01.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:01.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.662187+0000) 2019-10-13T09:21:01.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:01.661+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:01.662874+0000) 2019-10-13T09:21:03.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:03.976+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:03.978220+0000) 2019-10-13T09:21:04.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:04.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:04.388911+0000) 2019-10-13T09:21:06.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:06.560+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.563352+0000) 2019-10-13T09:21:06.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:06.560+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.564157+0000) 2019-10-13T09:21:06.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:06.560+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.564788+0000) 2019-10-13T09:21:06.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:06.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.565893+0000) 2019-10-13T09:21:06.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:06.564+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.566262+0000) 2019-10-13T09:21:06.567 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:06.564+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.566776+0000) 2019-10-13T09:21:06.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:06.561+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.566836+0000) 2019-10-13T09:21:06.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:06.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:06.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:06.565+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:06.569662+0000) 2019-10-13T09:21:07.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:07.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.190507+0000) 2019-10-13T09:21:07.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:07.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.518002+0000) 2019-10-13T09:21:07.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:07.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.615616+0000) 2019-10-13T09:21:07.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:07.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:07.641305+0000) 2019-10-13T09:21:11.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:11.568+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.570000+0000) 2019-10-13T09:21:11.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:11.568+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.571647+0000) 2019-10-13T09:21:11.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:11.568+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.572118+0000) 2019-10-13T09:21:11.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:11.572+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.573958+0000) 2019-10-13T09:21:11.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:11.572+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.574409+0000) 2019-10-13T09:21:11.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:11.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92937 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:11.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:11.572+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.575692+0000) 2019-10-13T09:21:11.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:11.569+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.575955+0000) 2019-10-13T09:21:11.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:11.569+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.576074+0000) 2019-10-13T09:21:11.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:11.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.662376+0000) 2019-10-13T09:21:11.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:11.661+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:11.663051+0000) 2019-10-13T09:21:13.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:13.976+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:13.978441+0000) 2019-10-13T09:21:14.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:14.384+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:14.389119+0000) 2019-10-13T09:21:16.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:16.572+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.577546+0000) 2019-10-13T09:21:16.579 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:16.576+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.579086+0000) 2019-10-13T09:21:16.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:16.576+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.580809+0000) 2019-10-13T09:21:16.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:16.576+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.581315+0000) 2019-10-13T09:21:16.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:16.577+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.581420+0000) 2019-10-13T09:21:16.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:16.580+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.582235+0000) 2019-10-13T09:21:16.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:16.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:16.584 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:16.580+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.583461+0000) 2019-10-13T09:21:16.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:16.581+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:16.587609+0000) 2019-10-13T09:21:17.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:17.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.196953+0000) 2019-10-13T09:21:17.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:17.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.518185+0000) 2019-10-13T09:21:17.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:17.612+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.615831+0000) 2019-10-13T09:21:17.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:17.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:17.641499+0000) 2019-10-13T09:21:21.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:21.584+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.586702+0000) 2019-10-13T09:21:21.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:21.584+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.587477+0000) 2019-10-13T09:21:21.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:21.584+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.588686+0000) 2019-10-13T09:21:21.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:21.584+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.589244+0000) 2019-10-13T09:21:21.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:21.585+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.589152+0000) 2019-10-13T09:21:21.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:21.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92998 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:21.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:21.588+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.590045+0000) 2019-10-13T09:21:21.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:21.588+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.590503+0000) 2019-10-13T09:21:21.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:21.589+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.594032+0000) 2019-10-13T09:21:21.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:21.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.662585+0000) 2019-10-13T09:21:21.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:21.657+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:21.663242+0000) 2019-10-13T09:21:22.596 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:21:22.596 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:22.600 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:21:22.601 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:23.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:23.975+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:23.978602+0000) 2019-10-13T09:21:24.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:24.383+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:24.389333+0000) 2019-10-13T09:21:26.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:26.589+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.593311+0000) 2019-10-13T09:21:26.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:26.597+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.600744+0000) 2019-10-13T09:21:26.604 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:26.587+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.593402+0000) 2019-10-13T09:21:26.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:26.591+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.594399+0000) 2019-10-13T09:21:26.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:26.591+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.595519+0000) 2019-10-13T09:21:26.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:26.591+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.595653+0000) 2019-10-13T09:21:26.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:26.591+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.596075+0000) 2019-10-13T09:21:26.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:26.591+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:26.596768+0000) 2019-10-13T09:21:26.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:26.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93026 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:27.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:27.205+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.208347+0000) 2019-10-13T09:21:27.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:27.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.518359+0000) 2019-10-13T09:21:27.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:27.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.616031+0000) 2019-10-13T09:21:27.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:27.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:27.641728+0000) 2019-10-13T09:21:31.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:31.595+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.600354+0000) 2019-10-13T09:21:31.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:31.595+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.600717+0000) 2019-10-13T09:21:31.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:31.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.602406+0000) 2019-10-13T09:21:31.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:31.599+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.603061+0000) 2019-10-13T09:21:31.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:31.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:31.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:31.599+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.605012+0000) 2019-10-13T09:21:31.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:31.599+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.605296+0000) 2019-10-13T09:21:31.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:31.603+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.606554+0000) 2019-10-13T09:21:31.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:31.605+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.608381+0000) 2019-10-13T09:21:31.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:31.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.662766+0000) 2019-10-13T09:21:31.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:31.661+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:31.663413+0000) 2019-10-13T09:21:33.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:33.975+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:33.978813+0000) 2019-10-13T09:21:34.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:34.383+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:34.389536+0000) 2019-10-13T09:21:36.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:36.603+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.608602+0000) 2019-10-13T09:21:36.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:36.603+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.608893+0000) 2019-10-13T09:21:36.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:36.605+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.608540+0000) 2019-10-13T09:21:36.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:36.603+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.609580+0000) 2019-10-13T09:21:36.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:36.607+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.609781+0000) 2019-10-13T09:21:36.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:36.607+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.610846+0000) 2019-10-13T09:21:36.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:36.607+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.611279+0000) 2019-10-13T09:21:36.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:36.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:36.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:36.613+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:36.616753+0000) 2019-10-13T09:21:37.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:37.213+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.214848+0000) 2019-10-13T09:21:37.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:37.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.518539+0000) 2019-10-13T09:21:37.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:37.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.616223+0000) 2019-10-13T09:21:37.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:37.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:37.641977+0000) 2019-10-13T09:21:41.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:41.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.617610+0000) 2019-10-13T09:21:41.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:41.615+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.618420+0000) 2019-10-13T09:21:41.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:41.615+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.619887+0000) 2019-10-13T09:21:41.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:41.615+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.620130+0000) 2019-10-13T09:21:41.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:41.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93118 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:41.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:41.615+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.620521+0000) 2019-10-13T09:21:41.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:41.615+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.621208+0000) 2019-10-13T09:21:41.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:41.615+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.621570+0000) 2019-10-13T09:21:41.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:41.621+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.625601+0000) 2019-10-13T09:21:41.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:41.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.662956+0000) 2019-10-13T09:21:41.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:41.661+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:41.663619+0000) 2019-10-13T09:21:43.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:43.975+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:43.979018+0000) 2019-10-13T09:21:44.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:44.387+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:44.389766+0000) 2019-10-13T09:21:46.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:46.619+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.624267+0000) 2019-10-13T09:21:46.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:46.619+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.624641+0000) 2019-10-13T09:21:46.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:46.621+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.625413+0000) 2019-10-13T09:21:46.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:46.619+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.625049+0000) 2019-10-13T09:21:46.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:46.623+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.626078+0000) 2019-10-13T09:21:46.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:46.623+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.626778+0000) 2019-10-13T09:21:46.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:46.623+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.627456+0000) 2019-10-13T09:21:46.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:46.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:46.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:46.629+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:46.632388+0000) 2019-10-13T09:21:47.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:47.221+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.221363+0000) 2019-10-13T09:21:47.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:47.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.518744+0000) 2019-10-13T09:21:47.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:47.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.616418+0000) 2019-10-13T09:21:47.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:47.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:47.642178+0000) 2019-10-13T09:21:51.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:51.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.631273+0000) 2019-10-13T09:21:51.634 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:51.627+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.633630+0000) 2019-10-13T09:21:51.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:51.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.633890+0000) 2019-10-13T09:21:51.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:51.631+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.634608+0000) 2019-10-13T09:21:51.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:51.628+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.634282+0000) 2019-10-13T09:21:51.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:51.631+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.634973+0000) 2019-10-13T09:21:51.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:51.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:51.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:51.631+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.635871+0000) 2019-10-13T09:21:51.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:51.636+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.641101+0000) 2019-10-13T09:21:51.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:51.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.663135+0000) 2019-10-13T09:21:51.664 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:51.660+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:51.663783+0000) 2019-10-13T09:21:52.668 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:21:52.669 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:52.672 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:21:52.673 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:21:53.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:53.975+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:53.979221+0000) 2019-10-13T09:21:54.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:54.387+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:54.389965+0000) 2019-10-13T09:21:56.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:56.635+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.638441+0000) 2019-10-13T09:21:56.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:21:56.635+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.639794+0000) 2019-10-13T09:21:56.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:21:56.632+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.639891+0000) 2019-10-13T09:21:56.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:56.635+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.640631+0000) 2019-10-13T09:21:56.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:56.635+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.640899+0000) 2019-10-13T09:21:56.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:21:56.635+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.641717+0000) 2019-10-13T09:21:56.648 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:21:56.639+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.642057+0000) 2019-10-13T09:21:56.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:21:56.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:21:56.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:56.640+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:56.647306+0000) 2019-10-13T09:21:57.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:21:57.224+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.228072+0000) 2019-10-13T09:21:57.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:21:57.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.518925+0000) 2019-10-13T09:21:57.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:21:57.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.616595+0000) 2019-10-13T09:21:57.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:21:57.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:21:57.642381+0000) 2019-10-13T09:22:01.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:01.643+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.646854+0000) 2019-10-13T09:22:01.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:01.643+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.647723+0000) 2019-10-13T09:22:01.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:01.643+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.648259+0000) 2019-10-13T09:22:01.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:01.644+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.648147+0000) 2019-10-13T09:22:01.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:01.643+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.648863+0000) 2019-10-13T09:22:01.650 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:01.643+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.649576+0000) 2019-10-13T09:22:01.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:01.647+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.650413+0000) 2019-10-13T09:22:01.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:01.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:01.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:01.652+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.656226+0000) 2019-10-13T09:22:01.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:01.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.663276+0000) 2019-10-13T09:22:01.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:01.656+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:01.663948+0000) 2019-10-13T09:22:03.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:03.979+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:03.979427+0000) 2019-10-13T09:22:04.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:04.387+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:04.390157+0000) 2019-10-13T09:22:06.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:06.651+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.654191+0000) 2019-10-13T09:22:06.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:06.651+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.655150+0000) 2019-10-13T09:22:06.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:06.651+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.655657+0000) 2019-10-13T09:22:06.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:06.651+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.656309+0000) 2019-10-13T09:22:06.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:06.651+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.657000+0000) 2019-10-13T09:22:06.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:06.651+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.657350+0000) 2019-10-13T09:22:06.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:06.652+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.657385+0000) 2019-10-13T09:22:06.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:06.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:06.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:06.656+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:06.662882+0000) 2019-10-13T09:22:07.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:07.232+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.234473+0000) 2019-10-13T09:22:07.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:07.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.519112+0000) 2019-10-13T09:22:07.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:07.611+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.616811+0000) 2019-10-13T09:22:07.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:07.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:07.642545+0000) 2019-10-13T09:22:11.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:11.655+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.661086+0000) 2019-10-13T09:22:11.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:11.659+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.662994+0000) 2019-10-13T09:22:11.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:11.659+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.663536+0000) 2019-10-13T09:22:11.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:11.656+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.663886+0000) 2019-10-13T09:22:11.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:11.660+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.664072+0000) 2019-10-13T09:22:11.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:11.659+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.664603+0000) 2019-10-13T09:22:11.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:11.659+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.665343+0000) 2019-10-13T09:22:11.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:11.659+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.665569+0000) 2019-10-13T09:22:11.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:11.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:11.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:11.664+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:11.668344+0000) 2019-10-13T09:22:13.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:13.979+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:13.979626+0000) 2019-10-13T09:22:14.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:14.387+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:14.390356+0000) 2019-10-13T09:22:16.671 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:16.667+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.670231+0000) 2019-10-13T09:22:16.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:16.667+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.670671+0000) 2019-10-13T09:22:16.673 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:16.667+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.672832+0000) 2019-10-13T09:22:16.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:16.667+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.672867+0000) 2019-10-13T09:22:16.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:16.667+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.673266+0000) 2019-10-13T09:22:16.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:16.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:16.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:16.668+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.674326+0000) 2019-10-13T09:22:16.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:16.671+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.675198+0000) 2019-10-13T09:22:16.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:16.676+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:16.683576+0000) 2019-10-13T09:22:17.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:17.240+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.240977+0000) 2019-10-13T09:22:17.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:17.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.519312+0000) 2019-10-13T09:22:17.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:17.615+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.616991+0000) 2019-10-13T09:22:17.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:17.639+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:17.642737+0000) 2019-10-13T09:22:18.414 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:18.411+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:18.414038+0000) 2019-10-13T09:22:21.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:21.660+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.664329+0000) 2019-10-13T09:22:21.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:21.670+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.677693+0000) 2019-10-13T09:22:21.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:21.674+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.678213+0000) 2019-10-13T09:22:21.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:21.674+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.678333+0000) 2019-10-13T09:22:21.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:21.674+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.679058+0000) 2019-10-13T09:22:21.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:21.674+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.679551+0000) 2019-10-13T09:22:21.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:21.676+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.680231+0000) 2019-10-13T09:22:21.681 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:21.674+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.681061+0000) 2019-10-13T09:22:21.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:21.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93357 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:21.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:21.680+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:21.687942+0000) 2019-10-13T09:22:22.756 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:22:22.756 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:22.760 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:22:22.760 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:23.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:23.978+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:23.979834+0000) 2019-10-13T09:22:24.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:24.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:24.390578+0000) 2019-10-13T09:22:26.686 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:26.678+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.685126+0000) 2019-10-13T09:22:26.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:26.682+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.685847+0000) 2019-10-13T09:22:26.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:26.682+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.686230+0000) 2019-10-13T09:22:26.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:26.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:26.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:26.682+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.689085+0000) 2019-10-13T09:22:26.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:26.684+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.688870+0000) 2019-10-13T09:22:26.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:26.686+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.689930+0000) 2019-10-13T09:22:26.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:26.686+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.690463+0000) 2019-10-13T09:22:26.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:26.688+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:26.692909+0000) 2019-10-13T09:22:27.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:27.244+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.247559+0000) 2019-10-13T09:22:27.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:27.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.519480+0000) 2019-10-13T09:22:27.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:27.614+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.617170+0000) 2019-10-13T09:22:27.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:27.642+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:27.642936+0000) 2019-10-13T09:22:28.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:28.410+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:28.414232+0000) 2019-10-13T09:22:31.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:31.664+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.664531+0000) 2019-10-13T09:22:31.692 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:31.686+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.691895+0000) 2019-10-13T09:22:31.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:31.690+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.694366+0000) 2019-10-13T09:22:31.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:31.690+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.695069+0000) 2019-10-13T09:22:31.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:31.688+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.695343+0000) 2019-10-13T09:22:31.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:31.690+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.695979+0000) 2019-10-13T09:22:31.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:31.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:31.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:31.690+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.696705+0000) 2019-10-13T09:22:31.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:31.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.697925+0000) 2019-10-13T09:22:31.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:31.696+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:31.701667+0000) 2019-10-13T09:22:33.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:33.974+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:33.980048+0000) 2019-10-13T09:22:34.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:34.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:34.390800+0000) 2019-10-13T09:22:36.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:36.694+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.700153+0000) 2019-10-13T09:22:36.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:36.694+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.700510+0000) 2019-10-13T09:22:36.702 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:36.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.701495+0000) 2019-10-13T09:22:36.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:36.698+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.701920+0000) 2019-10-13T09:22:36.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:36.698+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.702552+0000) 2019-10-13T09:22:36.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:36.696+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.702572+0000) 2019-10-13T09:22:36.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:36.698+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.703342+0000) 2019-10-13T09:22:36.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:36.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:36.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:36.700+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:36.707048+0000) 2019-10-13T09:22:37.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:37.252+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.254002+0000) 2019-10-13T09:22:37.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:37.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.519678+0000) 2019-10-13T09:22:37.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:37.614+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.617378+0000) 2019-10-13T09:22:37.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:37.638+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:37.643112+0000) 2019-10-13T09:22:38.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:38.410+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:38.414424+0000) 2019-10-13T09:22:41.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:41.664+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.664731+0000) 2019-10-13T09:22:41.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:41.702+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.707960+0000) 2019-10-13T09:22:41.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:41.702+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.709266+0000) 2019-10-13T09:22:41.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:41.702+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.709689+0000) 2019-10-13T09:22:41.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:41.706+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.710341+0000) 2019-10-13T09:22:41.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:41.704+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.710700+0000) 2019-10-13T09:22:41.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:41.706+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.711499+0000) 2019-10-13T09:22:41.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:41.706+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.710700+0000) 2019-10-13T09:22:41.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:41.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:41.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:41.708+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:41.714820+0000) 2019-10-13T09:22:43.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:43.978+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:43.980253+0000) 2019-10-13T09:22:44.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:44.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:44.391018+0000) 2019-10-13T09:22:46.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:46.710+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.715720+0000) 2019-10-13T09:22:46.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:46.710+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.716125+0000) 2019-10-13T09:22:46.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:46.710+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.716601+0000) 2019-10-13T09:22:46.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:46.710+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.716959+0000) 2019-10-13T09:22:46.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:46.712+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.718914+0000) 2019-10-13T09:22:46.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:46.714+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.719507+0000) 2019-10-13T09:22:46.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:46.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:46.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:46.714+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.720737+0000) 2019-10-13T09:22:46.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:46.724+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:46.728496+0000) 2019-10-13T09:22:47.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:47.260+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.260484+0000) 2019-10-13T09:22:47.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:47.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.519874+0000) 2019-10-13T09:22:47.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:47.614+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.617552+0000) 2019-10-13T09:22:47.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:47.642+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:47.643286+0000) 2019-10-13T09:22:48.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:48.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:48.414624+0000) 2019-10-13T09:22:51.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:51.664+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.664959+0000) 2019-10-13T09:22:51.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:51.718+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.723168+0000) 2019-10-13T09:22:51.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:51.718+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.723915+0000) 2019-10-13T09:22:51.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:51.718+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.724514+0000) 2019-10-13T09:22:51.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:51.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.724943+0000) 2019-10-13T09:22:51.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:51.724+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.725132+0000) 2019-10-13T09:22:51.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:51.722+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.726058+0000) 2019-10-13T09:22:51.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:51.722+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.727506+0000) 2019-10-13T09:22:51.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:51.722+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93534 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:51.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:51.732+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:51.733209+0000) 2019-10-13T09:22:52.887 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:22:52.887 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:52.891 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:22:52.891 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:22:53.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:53.978+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:53.980501+0000) 2019-10-13T09:22:54.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:54.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:54.391264+0000) 2019-10-13T09:22:56.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:56.726+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.731282+0000) 2019-10-13T09:22:56.732 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:22:56.726+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.732135+0000) 2019-10-13T09:22:56.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:22:56.727+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.731861+0000) 2019-10-13T09:22:56.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:22:56.726+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.733338+0000) 2019-10-13T09:22:56.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:56.726+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.733629+0000) 2019-10-13T09:22:56.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:56.730+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.735670+0000) 2019-10-13T09:22:56.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:22:56.730+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:22:56.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:56.730+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.737071+0000) 2019-10-13T09:22:56.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:56.739+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:56.740810+0000) 2019-10-13T09:22:57.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:22:57.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.270866+0000) 2019-10-13T09:22:57.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:22:57.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.520045+0000) 2019-10-13T09:22:57.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:22:57.614+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.617731+0000) 2019-10-13T09:22:57.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:22:57.642+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:57.643462+0000) 2019-10-13T09:22:58.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:22:58.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:22:58.414832+0000) 2019-10-13T09:23:01.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:01.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.665157+0000) 2019-10-13T09:23:01.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:01.735+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.739002+0000) 2019-10-13T09:23:01.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:01.734+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.740418+0000) 2019-10-13T09:23:01.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:01.734+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.741031+0000) 2019-10-13T09:23:01.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:01.738+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.741797+0000) 2019-10-13T09:23:01.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:01.738+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.742064+0000) 2019-10-13T09:23:01.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:01.738+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.742560+0000) 2019-10-13T09:23:01.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:01.738+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.743269+0000) 2019-10-13T09:23:01.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:01.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:01.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:01.743+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:01.746932+0000) 2019-10-13T09:23:03.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:03.974+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:03.980742+0000) 2019-10-13T09:23:04.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:04.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:04.391483+0000) 2019-10-13T09:23:06.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:06.743+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.746604+0000) 2019-10-13T09:23:06.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:06.742+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.747409+0000) 2019-10-13T09:23:06.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:06.742+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.748100+0000) 2019-10-13T09:23:06.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:06.742+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.748591+0000) 2019-10-13T09:23:06.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:06.742+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.748948+0000) 2019-10-13T09:23:06.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:06.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.749864+0000) 2019-10-13T09:23:06.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:06.746+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.749963+0000) 2019-10-13T09:23:06.752 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:06.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:06.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:06.755+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:06.756711+0000) 2019-10-13T09:23:07.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:07.275+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.277298+0000) 2019-10-13T09:23:07.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:07.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.520250+0000) 2019-10-13T09:23:07.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:07.614+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.617941+0000) 2019-10-13T09:23:07.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:07.638+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:07.643677+0000) 2019-10-13T09:23:08.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:08.410+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:08.415013+0000) 2019-10-13T09:23:11.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:11.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.665355+0000) 2019-10-13T09:23:11.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:11.750+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.754996+0000) 2019-10-13T09:23:11.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:11.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.755546+0000) 2019-10-13T09:23:11.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:11.751+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.755835+0000) 2019-10-13T09:23:11.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:11.750+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.756733+0000) 2019-10-13T09:23:11.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:11.754+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.758718+0000) 2019-10-13T09:23:11.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:11.754+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.758934+0000) 2019-10-13T09:23:11.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:11.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:11.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:11.754+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.759740+0000) 2019-10-13T09:23:11.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:11.763+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:11.765480+0000) 2019-10-13T09:23:13.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:13.978+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:13.980941+0000) 2019-10-13T09:23:14.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:14.386+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:14.391683+0000) 2019-10-13T09:23:16.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:16.757+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.762930+0000) 2019-10-13T09:23:16.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:16.757+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.763274+0000) 2019-10-13T09:23:16.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:16.759+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.763103+0000) 2019-10-13T09:23:16.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:16.757+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.764447+0000) 2019-10-13T09:23:16.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:16.757+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.764648+0000) 2019-10-13T09:23:16.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:16.757+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.765614+0000) 2019-10-13T09:23:16.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:16.761+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.766387+0000) 2019-10-13T09:23:16.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:16.761+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:16.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:16.767+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:16.769945+0000) 2019-10-13T09:23:17.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:17.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.283807+0000) 2019-10-13T09:23:17.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:17.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.520430+0000) 2019-10-13T09:23:17.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:17.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.618145+0000) 2019-10-13T09:23:17.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:17.637+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:17.643857+0000) 2019-10-13T09:23:18.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:18.409+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:18.415214+0000) 2019-10-13T09:23:21.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:21.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.665555+0000) 2019-10-13T09:23:21.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:21.765+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.772250+0000) 2019-10-13T09:23:21.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:21.765+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.772690+0000) 2019-10-13T09:23:21.774 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:21.765+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.773475+0000) 2019-10-13T09:23:21.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:21.765+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.773676+0000) 2019-10-13T09:23:21.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:21.771+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.773673+0000) 2019-10-13T09:23:21.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:21.769+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.774423+0000) 2019-10-13T09:23:21.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:21.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:21.777 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:21.769+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.776374+0000) 2019-10-13T09:23:21.782 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:21.779+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:21.780791+0000) 2019-10-13T09:23:23.000 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:23:23.000 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:23.005 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:23:23.005 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:23.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:23.977+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:23.981148+0000) 2019-10-13T09:23:24.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:24.385+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:24.391899+0000) 2019-10-13T09:23:26.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:26.773+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.778404+0000) 2019-10-13T09:23:26.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:26.775+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.778540+0000) 2019-10-13T09:23:26.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:26.773+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.779724+0000) 2019-10-13T09:23:26.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:26.773+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.780125+0000) 2019-10-13T09:23:26.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:26.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.780607+0000) 2019-10-13T09:23:26.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:26.773+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.781007+0000) 2019-10-13T09:23:26.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:26.773+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.781712+0000) 2019-10-13T09:23:26.783 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:26.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:26.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:26.787+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:26.789776+0000) 2019-10-13T09:23:27.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:27.287+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.290285+0000) 2019-10-13T09:23:27.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:27.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.520612+0000) 2019-10-13T09:23:27.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:27.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.618356+0000) 2019-10-13T09:23:27.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:27.641+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:27.644062+0000) 2019-10-13T09:23:28.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:28.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:28.415432+0000) 2019-10-13T09:23:31.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:31.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.665808+0000) 2019-10-13T09:23:31.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:31.783+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.786080+0000) 2019-10-13T09:23:31.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:31.785+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.788432+0000) 2019-10-13T09:23:31.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:31.785+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.788949+0000) 2019-10-13T09:23:31.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:31.785+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.789384+0000) 2019-10-13T09:23:31.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:31.789+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:31.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:31.789+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.790690+0000) 2019-10-13T09:23:31.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:31.789+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.791531+0000) 2019-10-13T09:23:31.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:31.789+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.792309+0000) 2019-10-13T09:23:31.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:31.795+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:31.797435+0000) 2019-10-13T09:23:33.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:33.977+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:33.981340+0000) 2019-10-13T09:23:34.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:34.389+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:34.392101+0000) 2019-10-13T09:23:36.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:36.793+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.793963+0000) 2019-10-13T09:23:36.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:36.793+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.794162+0000) 2019-10-13T09:23:36.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:36.793+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.795807+0000) 2019-10-13T09:23:36.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:36.793+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.796148+0000) 2019-10-13T09:23:36.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:36.793+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.796751+0000) 2019-10-13T09:23:36.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:36.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:36.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:36.795+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.797492+0000) 2019-10-13T09:23:36.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:36.797+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.798720+0000) 2019-10-13T09:23:36.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:36.803+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:36.805241+0000) 2019-10-13T09:23:37.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:37.299+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.301333+0000) 2019-10-13T09:23:37.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:37.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.520790+0000) 2019-10-13T09:23:37.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:37.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.618536+0000) 2019-10-13T09:23:37.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:37.641+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:37.644237+0000) 2019-10-13T09:23:38.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:38.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:38.415655+0000) 2019-10-13T09:23:41.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:41.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.666029+0000) 2019-10-13T09:23:41.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:41.797+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.800022+0000) 2019-10-13T09:23:41.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:41.801+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.802673+0000) 2019-10-13T09:23:41.803 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:41.801+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.802960+0000) 2019-10-13T09:23:41.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:41.801+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.803295+0000) 2019-10-13T09:23:41.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:41.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:41.805 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:41.801+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.804217+0000) 2019-10-13T09:23:41.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:41.799+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.803912+0000) 2019-10-13T09:23:41.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:41.801+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.804870+0000) 2019-10-13T09:23:41.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:41.807+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:41.810829+0000) 2019-10-13T09:23:43.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:43.977+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:43.981535+0000) 2019-10-13T09:23:44.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:44.389+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:44.392309+0000) 2019-10-13T09:23:46.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:46.803+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.806947+0000) 2019-10-13T09:23:46.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:46.805+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.808160+0000) 2019-10-13T09:23:46.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:46.805+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.808412+0000) 2019-10-13T09:23:46.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:46.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.809109+0000) 2019-10-13T09:23:46.811 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:46.809+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.811168+0000) 2019-10-13T09:23:46.812 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:46.809+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.811506+0000) 2019-10-13T09:23:46.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:46.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:46.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:46.809+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.812668+0000) 2019-10-13T09:23:46.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:46.815+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:46.819038+0000) 2019-10-13T09:23:47.309 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:47.307+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.307984+0000) 2019-10-13T09:23:47.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:47.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.520969+0000) 2019-10-13T09:23:47.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:47.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.618734+0000) 2019-10-13T09:23:47.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:47.641+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:47.644414+0000) 2019-10-13T09:23:48.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:48.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:48.415854+0000) 2019-10-13T09:23:51.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:51.663+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.666252+0000) 2019-10-13T09:23:51.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:51.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.815947+0000) 2019-10-13T09:23:51.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:51.813+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.816054+0000) 2019-10-13T09:23:51.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:51.815+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.816337+0000) 2019-10-13T09:23:51.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:51.813+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.817564+0000) 2019-10-13T09:23:51.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:51.817+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.817799+0000) 2019-10-13T09:23:51.819 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:51.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.818675+0000) 2019-10-13T09:23:51.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:51.817+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.819165+0000) 2019-10-13T09:23:51.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:51.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:51.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:51.819+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:51.823102+0000) 2019-10-13T09:23:53.077 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:23:53.077 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:53.081 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:23:53.081 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:23:53.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:53.981+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:53.981792+0000) 2019-10-13T09:23:54.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:54.389+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:54.392503+0000) 2019-10-13T09:23:56.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:56.821+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.823351+0000) 2019-10-13T09:23:56.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:23:56.821+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.823806+0000) 2019-10-13T09:23:56.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:56.821+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.824262+0000) 2019-10-13T09:23:56.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:23:56.823+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.824282+0000) 2019-10-13T09:23:56.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:23:56.825+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.825836+0000) 2019-10-13T09:23:56.826 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:56.825+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.825954+0000) 2019-10-13T09:23:56.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:56.825+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.826631+0000) 2019-10-13T09:23:56.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:23:56.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:23:56.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:56.831+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:56.832186+0000) 2019-10-13T09:23:57.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:23:57.311+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.314004+0000) 2019-10-13T09:23:57.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:23:57.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.521150+0000) 2019-10-13T09:23:57.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:23:57.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.618915+0000) 2019-10-13T09:23:57.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:23:57.641+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:57.644588+0000) 2019-10-13T09:23:58.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:23:58.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:23:58.416050+0000) 2019-10-13T09:24:01.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:01.662+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.666488+0000) 2019-10-13T09:24:01.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:01.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.831093+0000) 2019-10-13T09:24:01.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:01.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.832126+0000) 2019-10-13T09:24:01.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:01.829+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.832532+0000) 2019-10-13T09:24:01.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:01.830+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.832647+0000) 2019-10-13T09:24:01.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:01.834+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.837583+0000) 2019-10-13T09:24:01.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:01.829+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.833619+0000) 2019-10-13T09:24:01.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:01.833+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.833801+0000) 2019-10-13T09:24:01.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:01.833+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:01.834838+0000) 2019-10-13T09:24:01.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:01.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:03.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:03.981+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:03.981987+0000) 2019-10-13T09:24:04.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:04.389+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:04.392709+0000) 2019-10-13T09:24:06.840 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:06.837+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.839735+0000) 2019-10-13T09:24:06.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:06.837+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.840409+0000) 2019-10-13T09:24:06.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:06.837+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.840902+0000) 2019-10-13T09:24:06.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:06.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:06.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:06.838+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.842502+0000) 2019-10-13T09:24:06.844 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:06.841+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.843108+0000) 2019-10-13T09:24:06.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:06.841+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.843856+0000) 2019-10-13T09:24:06.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:06.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.845333+0000) 2019-10-13T09:24:06.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:06.842+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:06.845610+0000) 2019-10-13T09:24:07.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:07.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.324835+0000) 2019-10-13T09:24:07.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:07.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.521357+0000) 2019-10-13T09:24:07.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:07.617+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.619088+0000) 2019-10-13T09:24:07.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:07.641+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:07.644773+0000) 2019-10-13T09:24:08.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:08.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:08.416277+0000) 2019-10-13T09:24:11.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:11.662+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.666683+0000) 2019-10-13T09:24:11.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:11.844+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.846861+0000) 2019-10-13T09:24:11.849 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:11.844+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.848544+0000) 2019-10-13T09:24:11.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:11.846+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.848173+0000) 2019-10-13T09:24:11.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:11.844+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.848849+0000) 2019-10-13T09:24:11.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:11.844+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.849245+0000) 2019-10-13T09:24:11.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:11.844+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.849674+0000) 2019-10-13T09:24:11.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:11.848+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.850128+0000) 2019-10-13T09:24:11.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:11.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:11.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:11.850+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:11.852778+0000) 2019-10-13T09:24:13.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:13.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:13.982192+0000) 2019-10-13T09:24:14.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:14.388+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:14.392915+0000) 2019-10-13T09:24:16.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:16.852+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.854481+0000) 2019-10-13T09:24:16.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:16.852+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.854749+0000) 2019-10-13T09:24:16.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:16.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.855229+0000) 2019-10-13T09:24:16.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:16.852+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.856704+0000) 2019-10-13T09:24:16.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:16.852+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.857663+0000) 2019-10-13T09:24:16.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:16.856+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.858312+0000) 2019-10-13T09:24:16.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:16.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:16.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:16.856+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.858692+0000) 2019-10-13T09:24:16.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:16.858+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:16.863420+0000) 2019-10-13T09:24:17.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:17.334+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.336191+0000) 2019-10-13T09:24:17.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:17.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.521538+0000) 2019-10-13T09:24:17.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:17.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.619259+0000) 2019-10-13T09:24:17.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:17.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:17.644952+0000) 2019-10-13T09:24:18.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:18.412+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:18.416497+0000) 2019-10-13T09:24:21.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:21.666+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.666879+0000) 2019-10-13T09:24:21.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:21.860+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.862694+0000) 2019-10-13T09:24:21.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:21.860+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.862747+0000) 2019-10-13T09:24:21.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:21.860+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.863295+0000) 2019-10-13T09:24:21.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:21.858+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.863222+0000) 2019-10-13T09:24:21.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:21.860+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.864626+0000) 2019-10-13T09:24:21.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:21.860+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.865458+0000) 2019-10-13T09:24:21.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:21.864+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.865721+0000) 2019-10-13T09:24:21.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:21.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:21.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:21.866+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:21.870297+0000) 2019-10-13T09:24:23.133 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:24:23.133 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:23.137 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:24:23.137 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:23.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:23.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:23.982436+0000) 2019-10-13T09:24:24.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:24.388+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:24.393114+0000) 2019-10-13T09:24:26.870 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:26.864+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.869205+0000) 2019-10-13T09:24:26.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:26.868+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.870285+0000) 2019-10-13T09:24:26.871 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:26.868+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.871145+0000) 2019-10-13T09:24:26.872 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:26.868+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.870871+0000) 2019-10-13T09:24:26.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:26.866+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.871738+0000) 2019-10-13T09:24:26.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:26.868+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.872493+0000) 2019-10-13T09:24:26.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:26.868+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.872867+0000) 2019-10-13T09:24:26.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:26.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:26.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:26.878+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:26.880252+0000) 2019-10-13T09:24:27.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:27.342+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.342783+0000) 2019-10-13T09:24:27.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:27.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.521719+0000) 2019-10-13T09:24:27.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:27.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.619469+0000) 2019-10-13T09:24:27.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:27.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:27.645125+0000) 2019-10-13T09:24:28.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:28.412+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:28.416730+0000) 2019-10-13T09:24:31.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:31.666+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.667105+0000) 2019-10-13T09:24:31.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:31.872+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.876731+0000) 2019-10-13T09:24:31.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:31.876+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.878911+0000) 2019-10-13T09:24:31.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:31.876+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.879348+0000) 2019-10-13T09:24:31.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:31.876+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.880693+0000) 2019-10-13T09:24:31.881 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:31.876+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.881185+0000) 2019-10-13T09:24:31.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:31.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:31.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:31.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.881003+0000) 2019-10-13T09:24:31.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:31.880+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.881879+0000) 2019-10-13T09:24:31.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:31.878+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:31.882249+0000) 2019-10-13T09:24:33.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:33.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:33.982653+0000) 2019-10-13T09:24:34.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:34.392+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:34.393312+0000) 2019-10-13T09:24:36.885 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:36.880+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.884460+0000) 2019-10-13T09:24:36.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:36.880+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.885083+0000) 2019-10-13T09:24:36.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:36.884+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.886380+0000) 2019-10-13T09:24:36.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:36.884+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.888164+0000) 2019-10-13T09:24:36.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:36.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:36.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:36.888+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.889978+0000) 2019-10-13T09:24:36.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:36.886+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.890406+0000) 2019-10-13T09:24:36.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:36.888+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.890884+0000) 2019-10-13T09:24:36.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:36.894+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:36.896498+0000) 2019-10-13T09:24:37.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:37.346+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.349088+0000) 2019-10-13T09:24:37.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:37.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.521926+0000) 2019-10-13T09:24:37.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:37.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.619649+0000) 2019-10-13T09:24:37.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:37.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:37.645304+0000) 2019-10-13T09:24:38.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:38.412+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:38.416955+0000) 2019-10-13T09:24:41.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:41.662+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.667317+0000) 2019-10-13T09:24:41.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:41.888+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.892659+0000) 2019-10-13T09:24:41.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:41.892+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.894132+0000) 2019-10-13T09:24:41.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:41.892+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.894464+0000) 2019-10-13T09:24:41.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:41.892+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.894891+0000) 2019-10-13T09:24:41.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:41.892+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.895320+0000) 2019-10-13T09:24:41.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:41.892+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.895710+0000) 2019-10-13T09:24:41.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:41.894+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.896662+0000) 2019-10-13T09:24:41.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:41.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:41.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:41.898+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:41.900396+0000) 2019-10-13T09:24:43.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:43.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:43.982878+0000) 2019-10-13T09:24:44.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:44.392+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:44.393517+0000) 2019-10-13T09:24:46.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:46.896+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.901407+0000) 2019-10-13T09:24:46.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:46.900+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.902325+0000) 2019-10-13T09:24:46.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:46.900+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.905617+0000) 2019-10-13T09:24:46.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:46.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:46.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:46.904+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.906783+0000) 2019-10-13T09:24:46.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:46.904+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.906487+0000) 2019-10-13T09:24:46.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:46.902+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.907091+0000) 2019-10-13T09:24:46.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:46.904+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.908702+0000) 2019-10-13T09:24:46.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:46.906+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:46.908536+0000) 2019-10-13T09:24:47.356 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:47.354+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.355085+0000) 2019-10-13T09:24:47.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:47.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.522126+0000) 2019-10-13T09:24:47.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:47.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.619830+0000) 2019-10-13T09:24:47.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:47.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:47.645516+0000) 2019-10-13T09:24:48.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:48.412+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:48.417178+0000) 2019-10-13T09:24:51.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:51.662+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.667526+0000) 2019-10-13T09:24:51.910 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:51.904+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.909002+0000) 2019-10-13T09:24:51.910 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:51.908+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.910052+0000) 2019-10-13T09:24:51.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:51.906+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.910213+0000) 2019-10-13T09:24:51.911 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:51.908+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.911188+0000) 2019-10-13T09:24:51.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:51.908+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.911467+0000) 2019-10-13T09:24:51.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:51.908+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.912631+0000) 2019-10-13T09:24:51.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:51.908+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.913047+0000) 2019-10-13T09:24:51.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:51.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:51.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:51.914+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:51.917254+0000) 2019-10-13T09:24:53.190 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:24:53.190 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:53.194 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:24:53.194 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:24:53.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:53.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:53.983087+0000) 2019-10-13T09:24:54.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:54.392+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:54.393765+0000) 2019-10-13T09:24:56.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:56.912+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.916989+0000) 2019-10-13T09:24:56.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:24:56.912+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.917665+0000) 2019-10-13T09:24:56.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:56.916+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.918143+0000) 2019-10-13T09:24:56.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:56.916+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.919086+0000) 2019-10-13T09:24:56.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:24:56.916+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.919103+0000) 2019-10-13T09:24:56.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:56.916+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.919417+0000) 2019-10-13T09:24:56.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:24:56.914+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.920002+0000) 2019-10-13T09:24:56.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:24:56.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:24:56.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:56.918+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:56.921733+0000) 2019-10-13T09:24:57.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:24:57.358+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.361589+0000) 2019-10-13T09:24:57.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:24:57.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.522350+0000) 2019-10-13T09:24:57.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:24:57.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.620006+0000) 2019-10-13T09:24:57.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:24:57.640+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:57.645688+0000) 2019-10-13T09:24:58.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:24:58.412+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:24:58.417371+0000) 2019-10-13T09:25:01.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:01.662+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.667721+0000) 2019-10-13T09:25:01.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:01.920+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.924106+0000) 2019-10-13T09:25:01.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:01.920+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.925070+0000) 2019-10-13T09:25:01.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:01.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.925002+0000) 2019-10-13T09:25:01.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:01.920+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.925662+0000) 2019-10-13T09:25:01.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:01.924+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.926396+0000) 2019-10-13T09:25:01.928 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:01.924+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.926703+0000) 2019-10-13T09:25:01.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:01.924+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.927089+0000) 2019-10-13T09:25:01.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:01.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:01.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:01.926+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:01.930386+0000) 2019-10-13T09:25:03.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:03.980+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:03.983334+0000) 2019-10-13T09:25:04.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:04.392+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:04.394017+0000) 2019-10-13T09:25:06.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:06.928+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.933218+0000) 2019-10-13T09:25:06.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:06.932+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.934121+0000) 2019-10-13T09:25:06.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:06.932+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.934653+0000) 2019-10-13T09:25:06.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:06.929+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.934492+0000) 2019-10-13T09:25:06.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:06.932+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.935405+0000) 2019-10-13T09:25:06.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:06.932+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.936158+0000) 2019-10-13T09:25:06.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:06.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:06.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:06.932+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.937357+0000) 2019-10-13T09:25:06.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:06.937+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:06.942984+0000) 2019-10-13T09:25:07.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:07.365+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.368070+0000) 2019-10-13T09:25:07.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:07.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.522526+0000) 2019-10-13T09:25:07.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:07.616+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.620211+0000) 2019-10-13T09:25:07.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:07.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:07.645927+0000) 2019-10-13T09:25:08.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:08.411+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:08.417559+0000) 2019-10-13T09:25:11.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:11.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.667931+0000) 2019-10-13T09:25:11.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:11.935+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.939656+0000) 2019-10-13T09:25:11.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:11.935+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.941134+0000) 2019-10-13T09:25:11.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:11.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:11.946 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:11.939+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.945417+0000) 2019-10-13T09:25:11.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:11.941+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.945812+0000) 2019-10-13T09:25:11.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:11.943+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.946699+0000) 2019-10-13T09:25:11.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:11.943+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.947087+0000) 2019-10-13T09:25:11.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:11.943+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.948262+0000) 2019-10-13T09:25:11.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:11.949+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:11.955716+0000) 2019-10-13T09:25:13.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:13.979+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:13.983537+0000) 2019-10-13T09:25:14.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:14.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:14.394245+0000) 2019-10-13T09:25:16.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:16.943+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.948660+0000) 2019-10-13T09:25:16.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:16.947+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.950353+0000) 2019-10-13T09:25:16.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:16.947+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.950657+0000) 2019-10-13T09:25:16.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:16.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:16.953 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:16.947+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.952458+0000) 2019-10-13T09:25:16.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:16.945+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.951761+0000) 2019-10-13T09:25:16.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:16.947+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.952886+0000) 2019-10-13T09:25:16.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:16.947+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.953426+0000) 2019-10-13T09:25:16.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:16.953+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:16.959777+0000) 2019-10-13T09:25:17.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:17.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.374545+0000) 2019-10-13T09:25:17.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:17.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.522702+0000) 2019-10-13T09:25:17.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:17.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.620387+0000) 2019-10-13T09:25:17.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:17.643+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:17.646098+0000) 2019-10-13T09:25:18.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:18.415+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:18.417779+0000) 2019-10-13T09:25:21.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:21.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.668155+0000) 2019-10-13T09:25:21.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:21.951+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.956069+0000) 2019-10-13T09:25:21.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:21.953+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.957902+0000) 2019-10-13T09:25:21.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:21.955+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.958483+0000) 2019-10-13T09:25:21.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:21.955+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.959392+0000) 2019-10-13T09:25:21.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:21.955+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.959796+0000) 2019-10-13T09:25:21.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:21.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:21.961 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:21.955+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.960311+0000) 2019-10-13T09:25:21.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:21.955+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.960707+0000) 2019-10-13T09:25:21.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:21.961+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:21.966310+0000) 2019-10-13T09:25:23.328 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:25:23.328 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:23.332 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:25:23.332 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:23.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:23.979+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:23.983769+0000) 2019-10-13T09:25:24.395 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:24.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:24.394467+0000) 2019-10-13T09:25:26.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:26.959+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.963431+0000) 2019-10-13T09:25:26.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:26.959+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.964562+0000) 2019-10-13T09:25:26.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:26.959+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.965091+0000) 2019-10-13T09:25:26.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:26.961+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.964970+0000) 2019-10-13T09:25:26.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:26.963+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.965904+0000) 2019-10-13T09:25:26.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:26.963+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.967061+0000) 2019-10-13T09:25:26.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:26.963+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.966711+0000) 2019-10-13T09:25:26.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:26.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:26.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:26.969+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:26.973860+0000) 2019-10-13T09:25:27.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:27.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.380891+0000) 2019-10-13T09:25:27.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:27.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.522834+0000) 2019-10-13T09:25:27.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:27.615+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.620582+0000) 2019-10-13T09:25:27.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:27.643+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:27.646318+0000) 2019-10-13T09:25:28.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:28.415+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:28.417996+0000) 2019-10-13T09:25:31.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:31.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.668382+0000) 2019-10-13T09:25:31.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:31.965+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.971518+0000) 2019-10-13T09:25:31.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:31.967+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.972397+0000) 2019-10-13T09:25:31.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:31.971+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.974286+0000) 2019-10-13T09:25:31.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:31.971+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.974962+0000) 2019-10-13T09:25:31.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:31.971+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.975194+0000) 2019-10-13T09:25:31.976 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:31.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:31.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:31.971+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.976127+0000) 2019-10-13T09:25:31.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:31.971+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.976900+0000) 2019-10-13T09:25:31.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:31.973+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:31.977314+0000) 2019-10-13T09:25:33.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:33.979+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:33.983997+0000) 2019-10-13T09:25:34.395 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:34.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:34.394677+0000) 2019-10-13T09:25:36.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:36.975+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.979065+0000) 2019-10-13T09:25:36.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:36.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.982026+0000) 2019-10-13T09:25:36.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:36.977+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.981987+0000) 2019-10-13T09:25:36.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:36.979+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.983047+0000) 2019-10-13T09:25:36.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:36.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:36.985 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:36.979+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.983954+0000) 2019-10-13T09:25:36.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:36.979+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.984774+0000) 2019-10-13T09:25:36.986 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:36.979+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.985289+0000) 2019-10-13T09:25:36.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:36.985+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:36.990133+0000) 2019-10-13T09:25:37.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:37.385+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.387462+0000) 2019-10-13T09:25:37.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:37.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.522981+0000) 2019-10-13T09:25:37.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:37.615+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.620751+0000) 2019-10-13T09:25:37.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:37.643+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:37.646509+0000) 2019-10-13T09:25:38.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:38.415+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:38.418185+0000) 2019-10-13T09:25:41.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:41.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.668636+0000) 2019-10-13T09:25:41.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:41.983+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.986728+0000) 2019-10-13T09:25:41.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:41.983+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.988009+0000) 2019-10-13T09:25:41.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:41.983+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.988666+0000) 2019-10-13T09:25:41.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:41.987+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.989883+0000) 2019-10-13T09:25:41.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:41.985+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.989776+0000) 2019-10-13T09:25:41.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:41.987+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.990807+0000) 2019-10-13T09:25:41.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:41.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.991090+0000) 2019-10-13T09:25:41.992 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:41.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:41.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:41.989+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:41.994266+0000) 2019-10-13T09:25:43.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:43.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:43.984223+0000) 2019-10-13T09:25:44.395 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:44.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:44.394904+0000) 2019-10-13T09:25:46.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:46.991+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.994875+0000) 2019-10-13T09:25:46.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:46.989+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.994856+0000) 2019-10-13T09:25:46.996 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:46.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.995556+0000) 2019-10-13T09:25:46.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:46.991+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.996448+0000) 2019-10-13T09:25:46.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:46.991+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.997485+0000) 2019-10-13T09:25:46.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:46.995+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.997805+0000) 2019-10-13T09:25:46.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:46.995+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:46.998166+0000) 2019-10-13T09:25:46.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:46.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:47.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:47.001+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.005224+0000) 2019-10-13T09:25:47.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:47.393+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.393649+0000) 2019-10-13T09:25:47.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:47.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.523150+0000) 2019-10-13T09:25:47.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:47.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.620926+0000) 2019-10-13T09:25:47.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:47.643+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:47.646688+0000) 2019-10-13T09:25:48.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:48.415+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:48.418394+0000) 2019-10-13T09:25:51.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:51.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:51.668849+0000) 2019-10-13T09:25:52.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:51.999+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.001888+0000) 2019-10-13T09:25:52.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:51.999+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.002842+0000) 2019-10-13T09:25:52.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:51.999+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.003788+0000) 2019-10-13T09:25:52.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:51.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.003771+0000) 2019-10-13T09:25:52.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:51.999+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.004371+0000) 2019-10-13T09:25:52.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:51.999+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.005713+0000) 2019-10-13T09:25:52.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:52.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:52.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:52.003+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.006752+0000) 2019-10-13T09:25:52.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:52.005+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:52.009180+0000) 2019-10-13T09:25:53.436 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:25:53.436 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:53.440 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:25:53.440 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:25:53.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:53.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:53.984431+0000) 2019-10-13T09:25:54.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:54.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:54.395108+0000) 2019-10-13T09:25:57.010 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:57.003+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.009591+0000) 2019-10-13T09:25:57.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:25:57.007+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.010011+0000) 2019-10-13T09:25:57.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:57.007+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.012730+0000) 2019-10-13T09:25:57.014 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:25:57.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:25:57.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:25:57.009+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.013969+0000) 2019-10-13T09:25:57.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:57.011+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.015360+0000) 2019-10-13T09:25:57.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:57.011+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.015992+0000) 2019-10-13T09:25:57.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:25:57.011+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.016873+0000) 2019-10-13T09:25:57.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:57.017+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.020324+0000) 2019-10-13T09:25:57.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:25:57.401+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.404442+0000) 2019-10-13T09:25:57.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:25:57.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.523331+0000) 2019-10-13T09:25:57.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:25:57.615+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.621137+0000) 2019-10-13T09:25:57.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:25:57.643+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:57.646866+0000) 2019-10-13T09:25:58.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:25:58.415+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:25:58.418602+0000) 2019-10-13T09:26:01.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:01.665+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:01.669048+0000) 2019-10-13T09:26:02.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:02.011+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.016700+0000) 2019-10-13T09:26:02.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:02.011+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.017058+0000) 2019-10-13T09:26:02.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:02.015+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.018065+0000) 2019-10-13T09:26:02.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:02.015+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.018600+0000) 2019-10-13T09:26:02.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:02.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:02.022 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:02.019+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.021820+0000) 2019-10-13T09:26:02.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:02.017+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.022247+0000) 2019-10-13T09:26:02.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:02.019+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.022181+0000) 2019-10-13T09:26:02.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:02.017+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:02.023366+0000) 2019-10-13T09:26:03.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:03.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:03.984661+0000) 2019-10-13T09:26:04.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:04.391+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:04.395328+0000) 2019-10-13T09:26:07.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:07.018+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.024449+0000) 2019-10-13T09:26:07.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:07.018+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.024926+0000) 2019-10-13T09:26:07.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:07.022+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.027014+0000) 2019-10-13T09:26:07.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:07.022+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.027315+0000) 2019-10-13T09:26:07.029 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:07.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94678 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:07.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:07.022+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.029304+0000) 2019-10-13T09:26:07.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:07.025+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.029905+0000) 2019-10-13T09:26:07.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:07.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.030670+0000) 2019-10-13T09:26:07.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:07.025+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.030847+0000) 2019-10-13T09:26:07.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:07.405+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.411042+0000) 2019-10-13T09:26:07.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:07.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.523529+0000) 2019-10-13T09:26:07.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:07.618+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.621306+0000) 2019-10-13T09:26:07.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:07.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:07.647045+0000) 2019-10-13T09:26:08.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:08.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:08.418828+0000) 2019-10-13T09:26:11.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:11.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:11.669194+0000) 2019-10-13T09:26:12.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:12.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.032857+0000) 2019-10-13T09:26:12.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:12.030+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.034701+0000) 2019-10-13T09:26:12.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:12.030+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.036725+0000) 2019-10-13T09:26:12.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:12.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:12.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:12.032+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.037818+0000) 2019-10-13T09:26:12.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:12.034+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.038150+0000) 2019-10-13T09:26:12.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:12.030+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.037508+0000) 2019-10-13T09:26:12.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:12.034+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.039164+0000) 2019-10-13T09:26:12.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:12.032+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:12.039395+0000) 2019-10-13T09:26:13.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:13.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:13.984858+0000) 2019-10-13T09:26:14.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:14.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:14.395532+0000) 2019-10-13T09:26:17.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:17.034+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.039970+0000) 2019-10-13T09:26:17.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:17.034+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.041099+0000) 2019-10-13T09:26:17.042 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:17.038+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.041811+0000) 2019-10-13T09:26:17.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:17.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.042448+0000) 2019-10-13T09:26:17.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:17.038+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.042773+0000) 2019-10-13T09:26:17.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:17.038+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.043188+0000) 2019-10-13T09:26:17.044 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:17.036+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.043512+0000) 2019-10-13T09:26:17.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:17.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94738 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:17.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:17.040+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.047164+0000) 2019-10-13T09:26:17.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:17.412+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.417468+0000) 2019-10-13T09:26:17.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:17.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.523707+0000) 2019-10-13T09:26:17.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:17.618+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.621517+0000) 2019-10-13T09:26:17.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:17.642+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:17.647238+0000) 2019-10-13T09:26:18.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:18.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:18.419073+0000) 2019-10-13T09:26:21.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:21.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:21.669415+0000) 2019-10-13T09:26:22.048 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:22.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.047282+0000) 2019-10-13T09:26:22.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:22.042+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.048125+0000) 2019-10-13T09:26:22.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:22.042+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.049191+0000) 2019-10-13T09:26:22.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:22.046+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.050141+0000) 2019-10-13T09:26:22.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:22.046+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.051499+0000) 2019-10-13T09:26:22.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:22.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.051908+0000) 2019-10-13T09:26:22.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:22.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:22.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:22.046+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.051872+0000) 2019-10-13T09:26:22.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:22.048+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:22.054154+0000) 2019-10-13T09:26:23.483 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:26:23.484 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:23.488 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:26:23.488 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:23.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:23.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:23.985072+0000) 2019-10-13T09:26:24.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:24.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:24.395769+0000) 2019-10-13T09:26:27.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:27.050+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.055474+0000) 2019-10-13T09:26:27.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:27.050+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.055946+0000) 2019-10-13T09:26:27.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:27.050+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.057281+0000) 2019-10-13T09:26:27.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:27.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.057264+0000) 2019-10-13T09:26:27.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:27.054+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.058505+0000) 2019-10-13T09:26:27.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:27.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:27.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:27.054+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.059234+0000) 2019-10-13T09:26:27.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:27.054+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.059681+0000) 2019-10-13T09:26:27.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:27.060+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.064553+0000) 2019-10-13T09:26:27.425 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:27.416+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.423896+0000) 2019-10-13T09:26:27.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:27.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.523902+0000) 2019-10-13T09:26:27.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:27.618+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.621717+0000) 2019-10-13T09:26:27.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:27.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:27.647444+0000) 2019-10-13T09:26:28.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:28.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:28.419270+0000) 2019-10-13T09:26:31.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:31.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:31.669609+0000) 2019-10-13T09:26:32.062 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:32.056+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.061646+0000) 2019-10-13T09:26:32.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:32.058+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.062489+0000) 2019-10-13T09:26:32.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:32.058+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.063200+0000) 2019-10-13T09:26:32.064 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:32.058+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.063700+0000) 2019-10-13T09:26:32.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:32.058+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.064543+0000) 2019-10-13T09:26:32.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:32.058+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.065159+0000) 2019-10-13T09:26:32.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:32.058+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.065670+0000) 2019-10-13T09:26:32.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:32.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:32.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:32.064+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:32.070030+0000) 2019-10-13T09:26:33.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:33.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:33.985314+0000) 2019-10-13T09:26:34.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:34.390+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:34.395976+0000) 2019-10-13T09:26:37.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:37.066+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.070341+0000) 2019-10-13T09:26:37.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:37.064+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.070429+0000) 2019-10-13T09:26:37.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:37.066+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.071316+0000) 2019-10-13T09:26:37.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:37.066+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.072261+0000) 2019-10-13T09:26:37.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:37.066+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.072477+0000) 2019-10-13T09:26:37.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:37.066+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.073019+0000) 2019-10-13T09:26:37.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:37.066+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.073314+0000) 2019-10-13T09:26:37.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:37.070+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:37.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:37.076+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.080626+0000) 2019-10-13T09:26:37.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:37.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.430595+0000) 2019-10-13T09:26:37.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:37.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.524077+0000) 2019-10-13T09:26:37.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:37.618+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.621940+0000) 2019-10-13T09:26:37.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:37.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:37.647620+0000) 2019-10-13T09:26:38.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:38.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:38.419485+0000) 2019-10-13T09:26:41.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:41.664+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:41.669810+0000) 2019-10-13T09:26:42.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:42.072+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.077652+0000) 2019-10-13T09:26:42.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:42.074+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.078683+0000) 2019-10-13T09:26:42.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:42.074+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.079895+0000) 2019-10-13T09:26:42.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:42.074+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.080457+0000) 2019-10-13T09:26:42.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:42.074+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.079218+0000) 2019-10-13T09:26:42.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:42.074+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.081081+0000) 2019-10-13T09:26:42.082 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:42.074+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.081656+0000) 2019-10-13T09:26:42.083 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:42.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:42.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:42.080+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:42.085212+0000) 2019-10-13T09:26:43.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:43.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:43.985544+0000) 2019-10-13T09:26:44.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:44.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:44.396178+0000) 2019-10-13T09:26:47.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:47.082+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.085998+0000) 2019-10-13T09:26:47.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:47.080+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.086471+0000) 2019-10-13T09:26:47.087 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:47.082+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.087148+0000) 2019-10-13T09:26:47.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:47.082+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.087607+0000) 2019-10-13T09:26:47.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:47.082+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.087813+0000) 2019-10-13T09:26:47.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:47.082+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.088279+0000) 2019-10-13T09:26:47.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:47.082+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.089258+0000) 2019-10-13T09:26:47.091 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:47.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:47.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:47.088+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.094616+0000) 2019-10-13T09:26:47.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:47.432+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.436463+0000) 2019-10-13T09:26:47.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:47.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.524261+0000) 2019-10-13T09:26:47.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:47.618+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.622162+0000) 2019-10-13T09:26:47.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:47.642+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:47.647813+0000) 2019-10-13T09:26:48.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:48.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:48.419682+0000) 2019-10-13T09:26:51.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:51.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:51.670052+0000) 2019-10-13T09:26:52.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:52.090+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.093857+0000) 2019-10-13T09:26:52.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:52.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.094548+0000) 2019-10-13T09:26:52.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:52.090+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.096220+0000) 2019-10-13T09:26:52.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:52.090+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.096492+0000) 2019-10-13T09:26:52.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:52.092+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.097915+0000) 2019-10-13T09:26:52.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:52.094+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.098547+0000) 2019-10-13T09:26:52.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:52.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:52.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:52.094+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.100402+0000) 2019-10-13T09:26:52.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:52.100+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:52.106072+0000) 2019-10-13T09:26:53.555 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:26:53.556 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:53.560 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:26:53.560 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:26:53.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:53.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:53.985781+0000) 2019-10-13T09:26:54.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:54.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:54.396391+0000) 2019-10-13T09:26:57.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:26:57.096+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.102197+0000) 2019-10-13T09:26:57.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:26:57.098+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.103256+0000) 2019-10-13T09:26:57.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:57.098+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.103811+0000) 2019-10-13T09:26:57.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:57.098+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.104131+0000) 2019-10-13T09:26:57.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:57.098+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.104848+0000) 2019-10-13T09:26:57.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:57.098+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.105036+0000) 2019-10-13T09:26:57.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:26:57.102+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.106303+0000) 2019-10-13T09:26:57.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:26:57.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94974 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:26:57.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:57.104+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.110203+0000) 2019-10-13T09:26:57.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:26:57.436+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.442737+0000) 2019-10-13T09:26:57.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:26:57.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.524473+0000) 2019-10-13T09:26:57.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:26:57.622+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.622388+0000) 2019-10-13T09:26:57.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:26:57.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:57.648018+0000) 2019-10-13T09:26:58.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:26:58.414+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:26:58.419869+0000) 2019-10-13T09:27:01.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:01.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:01.670267+0000) 2019-10-13T09:27:02.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:02.105+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.110733+0000) 2019-10-13T09:27:02.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:02.105+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.111260+0000) 2019-10-13T09:27:02.112 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:02.105+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.111504+0000) 2019-10-13T09:27:02.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:02.104+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.111953+0000) 2019-10-13T09:27:02.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:02.105+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.112864+0000) 2019-10-13T09:27:02.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:02.105+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.113018+0000) 2019-10-13T09:27:02.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:02.105+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.113441+0000) 2019-10-13T09:27:02.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:02.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:02.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:02.116+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:02.120602+0000) 2019-10-13T09:27:03.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:03.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:03.986018+0000) 2019-10-13T09:27:04.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:04.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:04.396594+0000) 2019-10-13T09:27:07.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:07.113+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.121362+0000) 2019-10-13T09:27:07.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:07.116+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.121727+0000) 2019-10-13T09:27:07.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:07.117+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.122957+0000) 2019-10-13T09:27:07.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:07.117+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.123674+0000) 2019-10-13T09:27:07.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:07.117+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.124156+0000) 2019-10-13T09:27:07.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:07.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:07.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:07.121+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.126228+0000) 2019-10-13T09:27:07.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:07.121+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.126709+0000) 2019-10-13T09:27:07.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:07.124+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.131492+0000) 2019-10-13T09:27:07.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:07.448+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.449081+0000) 2019-10-13T09:27:07.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:07.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.524679+0000) 2019-10-13T09:27:07.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:07.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.622562+0000) 2019-10-13T09:27:07.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:07.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:07.648152+0000) 2019-10-13T09:27:08.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:08.413+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:08.420060+0000) 2019-10-13T09:27:11.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:11.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:11.670487+0000) 2019-10-13T09:27:12.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:12.121+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.128980+0000) 2019-10-13T09:27:12.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:12.128+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.129260+0000) 2019-10-13T09:27:12.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:12.125+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.131847+0000) 2019-10-13T09:27:12.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:12.125+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.132021+0000) 2019-10-13T09:27:12.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:12.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:12.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:12.129+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.134386+0000) 2019-10-13T09:27:12.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:12.129+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.137205+0000) 2019-10-13T09:27:12.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:12.133+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.137849+0000) 2019-10-13T09:27:12.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:12.136+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:12.138649+0000) 2019-10-13T09:27:13.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:13.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:13.986234+0000) 2019-10-13T09:27:14.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:14.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:14.396800+0000) 2019-10-13T09:27:17.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:17.133+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.136143+0000) 2019-10-13T09:27:17.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:17.133+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.137697+0000) 2019-10-13T09:27:17.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:17.137+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.138234+0000) 2019-10-13T09:27:17.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:17.137+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.138986+0000) 2019-10-13T09:27:17.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:17.137+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.139113+0000) 2019-10-13T09:27:17.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:17.135+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.139263+0000) 2019-10-13T09:27:17.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:17.137+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.140093+0000) 2019-10-13T09:27:17.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:17.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:17.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:17.143+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.147342+0000) 2019-10-13T09:27:17.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:17.451+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.455533+0000) 2019-10-13T09:27:17.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:17.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.524853+0000) 2019-10-13T09:27:17.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:17.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.622744+0000) 2019-10-13T09:27:17.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:17.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:17.648337+0000) 2019-10-13T09:27:18.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:18.417+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:18.420241+0000) 2019-10-13T09:27:21.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:21.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:21.670710+0000) 2019-10-13T09:27:22.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:22.141+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.144194+0000) 2019-10-13T09:27:22.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:22.141+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.144700+0000) 2019-10-13T09:27:22.146 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:22.141+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.145321+0000) 2019-10-13T09:27:22.147 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:22.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.146210+0000) 2019-10-13T09:27:22.148 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:22.145+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.146925+0000) 2019-10-13T09:27:22.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:22.145+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.147196+0000) 2019-10-13T09:27:22.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:22.145+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.147846+0000) 2019-10-13T09:27:22.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:22.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:22.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:22.147+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:22.151844+0000) 2019-10-13T09:27:23.661 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:27:23.661 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:23.665 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:27:23.665 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:23.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:23.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:23.986447+0000) 2019-10-13T09:27:24.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:24.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:24.397012+0000) 2019-10-13T09:27:27.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:27.149+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.151926+0000) 2019-10-13T09:27:27.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:27.149+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.152566+0000) 2019-10-13T09:27:27.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:27.149+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.152691+0000) 2019-10-13T09:27:27.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:27.153+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.154498+0000) 2019-10-13T09:27:27.156 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:27.153+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.156029+0000) 2019-10-13T09:27:27.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:27.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:27.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:27.155+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.156557+0000) 2019-10-13T09:27:27.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:27.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.157326+0000) 2019-10-13T09:27:27.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:27.159+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.162925+0000) 2019-10-13T09:27:27.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:27.459+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.462074+0000) 2019-10-13T09:27:27.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:27.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.525059+0000) 2019-10-13T09:27:27.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:27.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.622873+0000) 2019-10-13T09:27:27.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:27.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:27.648510+0000) 2019-10-13T09:27:28.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:28.417+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:28.420438+0000) 2019-10-13T09:27:31.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:31.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:31.670934+0000) 2019-10-13T09:27:32.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:32.157+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.159628+0000) 2019-10-13T09:27:32.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:32.157+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.160046+0000) 2019-10-13T09:27:32.161 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:32.157+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.160459+0000) 2019-10-13T09:27:32.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:32.159+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.160504+0000) 2019-10-13T09:27:32.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:32.161+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.163066+0000) 2019-10-13T09:27:32.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:32.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:32.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:32.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.163946+0000) 2019-10-13T09:27:32.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:32.161+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.164302+0000) 2019-10-13T09:27:32.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:32.163+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:32.167172+0000) 2019-10-13T09:27:33.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:33.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:33.986665+0000) 2019-10-13T09:27:34.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:34.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:34.397215+0000) 2019-10-13T09:27:37.167 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:37.165+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.166554+0000) 2019-10-13T09:27:37.168 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:37.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.166996+0000) 2019-10-13T09:27:37.168 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:37.165+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.167658+0000) 2019-10-13T09:27:37.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:37.165+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.168297+0000) 2019-10-13T09:27:37.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:37.167+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.168540+0000) 2019-10-13T09:27:37.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:37.165+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.169369+0000) 2019-10-13T09:27:37.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:37.165+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.169579+0000) 2019-10-13T09:27:37.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:37.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95210 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:37.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:37.171+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.175013+0000) 2019-10-13T09:27:37.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:37.467+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.468430+0000) 2019-10-13T09:27:37.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:37.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.525257+0000) 2019-10-13T09:27:37.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:37.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.623065+0000) 2019-10-13T09:27:37.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:37.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:37.648692+0000) 2019-10-13T09:27:38.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:38.417+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:38.420634+0000) 2019-10-13T09:27:41.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:41.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:41.671092+0000) 2019-10-13T09:27:42.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:42.171+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.174455+0000) 2019-10-13T09:27:42.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:42.173+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.175532+0000) 2019-10-13T09:27:42.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:42.173+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.175838+0000) 2019-10-13T09:27:42.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:42.173+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.176112+0000) 2019-10-13T09:27:42.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:42.173+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.176573+0000) 2019-10-13T09:27:42.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:42.173+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.176780+0000) 2019-10-13T09:27:42.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:42.173+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.177235+0000) 2019-10-13T09:27:42.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:42.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:42.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:42.183+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:42.184947+0000) 2019-10-13T09:27:43.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:43.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:43.986875+0000) 2019-10-13T09:27:44.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:44.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:44.397470+0000) 2019-10-13T09:27:47.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:47.181+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.183482+0000) 2019-10-13T09:27:47.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:47.181+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.184083+0000) 2019-10-13T09:27:47.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:47.181+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.184373+0000) 2019-10-13T09:27:47.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:47.181+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.184984+0000) 2019-10-13T09:27:47.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:47.181+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.185264+0000) 2019-10-13T09:27:47.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:47.185+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.185883+0000) 2019-10-13T09:27:47.187 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:47.183+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.185745+0000) 2019-10-13T09:27:47.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:47.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95268 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:47.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:47.191+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.192460+0000) 2019-10-13T09:27:47.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:47.471+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.474962+0000) 2019-10-13T09:27:47.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:47.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.525447+0000) 2019-10-13T09:27:47.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:47.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.623272+0000) 2019-10-13T09:27:47.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:47.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:47.648904+0000) 2019-10-13T09:27:48.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:48.417+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:48.420859+0000) 2019-10-13T09:27:51.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:51.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:51.671316+0000) 2019-10-13T09:27:52.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:52.189+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.191243+0000) 2019-10-13T09:27:52.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:52.189+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.191621+0000) 2019-10-13T09:27:52.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:52.189+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.192108+0000) 2019-10-13T09:27:52.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:52.189+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.192800+0000) 2019-10-13T09:27:52.194 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:52.191+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.192794+0000) 2019-10-13T09:27:52.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:52.189+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.193438+0000) 2019-10-13T09:27:52.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:52.193+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.194360+0000) 2019-10-13T09:27:52.196 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:52.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:52.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:52.195+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:52.196753+0000) 2019-10-13T09:27:53.768 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:27:53.768 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:53.772 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:27:53.772 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:27:53.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:53.985+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:53.987128+0000) 2019-10-13T09:27:54.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:54.393+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:54.397668+0000) 2019-10-13T09:27:57.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:57.197+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.198941+0000) 2019-10-13T09:27:57.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:27:57.197+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.199873+0000) 2019-10-13T09:27:57.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:57.197+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.200205+0000) 2019-10-13T09:27:57.202 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:27:57.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.200553+0000) 2019-10-13T09:27:57.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:57.197+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.200575+0000) 2019-10-13T09:27:57.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:57.197+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.201662+0000) 2019-10-13T09:27:57.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:27:57.201+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.203038+0000) 2019-10-13T09:27:57.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:27:57.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:27:57.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:57.203+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.207390+0000) 2019-10-13T09:27:57.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:27:57.479+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.481654+0000) 2019-10-13T09:27:57.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:27:57.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.525625+0000) 2019-10-13T09:27:57.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:27:57.621+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.623463+0000) 2019-10-13T09:27:57.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:27:57.649+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:57.649083+0000) 2019-10-13T09:27:58.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:27:58.420+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:27:58.421078+0000) 2019-10-13T09:28:01.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:01.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:01.671509+0000) 2019-10-13T09:28:02.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:02.204+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.207313+0000) 2019-10-13T09:28:02.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:02.203+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.207472+0000) 2019-10-13T09:28:02.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:02.204+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.208259+0000) 2019-10-13T09:28:02.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:02.204+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.209156+0000) 2019-10-13T09:28:02.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:02.204+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.209471+0000) 2019-10-13T09:28:02.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:02.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.209854+0000) 2019-10-13T09:28:02.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:02.208+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.210801+0000) 2019-10-13T09:28:02.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:02.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95357 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:02.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:02.215+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:02.216540+0000) 2019-10-13T09:28:03.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:03.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:03.987354+0000) 2019-10-13T09:28:04.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:04.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:04.397893+0000) 2019-10-13T09:28:07.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:07.212+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.216096+0000) 2019-10-13T09:28:07.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:07.212+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.216788+0000) 2019-10-13T09:28:07.217 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:07.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.217102+0000) 2019-10-13T09:28:07.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:07.212+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.217388+0000) 2019-10-13T09:28:07.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:07.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.218746+0000) 2019-10-13T09:28:07.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:07.215+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.218821+0000) 2019-10-13T09:28:07.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:07.216+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.219629+0000) 2019-10-13T09:28:07.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:07.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:07.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:07.223+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.226906+0000) 2019-10-13T09:28:07.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:07.487+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.488132+0000) 2019-10-13T09:28:07.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:07.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.525804+0000) 2019-10-13T09:28:07.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:07.620+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.623676+0000) 2019-10-13T09:28:07.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:07.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:07.649258+0000) 2019-10-13T09:28:08.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:08.420+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:08.421278+0000) 2019-10-13T09:28:11.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:11.667+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:11.671703+0000) 2019-10-13T09:28:12.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:12.220+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.224154+0000) 2019-10-13T09:28:12.226 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:12.224+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.226024+0000) 2019-10-13T09:28:12.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:12.224+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.226474+0000) 2019-10-13T09:28:12.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:12.224+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.227350+0000) 2019-10-13T09:28:12.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:12.223+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.227211+0000) 2019-10-13T09:28:12.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:12.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:12.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:12.224+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.228533+0000) 2019-10-13T09:28:12.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:12.224+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.229152+0000) 2019-10-13T09:28:12.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:12.235+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:12.236645+0000) 2019-10-13T09:28:13.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:13.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:13.987577+0000) 2019-10-13T09:28:14.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:14.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:14.398102+0000) 2019-10-13T09:28:17.232 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:17.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.231702+0000) 2019-10-13T09:28:17.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:17.228+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.232154+0000) 2019-10-13T09:28:17.233 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:17.228+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.232645+0000) 2019-10-13T09:28:17.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:17.232+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.233821+0000) 2019-10-13T09:28:17.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:17.232+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.234016+0000) 2019-10-13T09:28:17.235 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:17.231+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.234092+0000) 2019-10-13T09:28:17.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:17.232+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.234826+0000) 2019-10-13T09:28:17.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:17.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:17.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:17.235+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.238980+0000) 2019-10-13T09:28:17.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:17.491+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.494501+0000) 2019-10-13T09:28:17.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:17.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.525986+0000) 2019-10-13T09:28:17.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:17.620+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.623852+0000) 2019-10-13T09:28:17.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:17.648+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:17.649499+0000) 2019-10-13T09:28:18.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:18.416+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:18.421488+0000) 2019-10-13T09:28:21.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:21.671+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:21.671942+0000) 2019-10-13T09:28:22.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:22.236+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.238586+0000) 2019-10-13T09:28:22.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:22.238+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.240177+0000) 2019-10-13T09:28:22.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:22.236+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.241229+0000) 2019-10-13T09:28:22.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:22.240+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.241886+0000) 2019-10-13T09:28:22.243 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:22.240+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.242506+0000) 2019-10-13T09:28:22.244 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:22.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:22.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:22.240+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.243042+0000) 2019-10-13T09:28:22.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:22.240+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.243313+0000) 2019-10-13T09:28:22.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:22.246+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:22.249703+0000) 2019-10-13T09:28:23.880 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:28:23.880 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:23.884 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:28:23.884 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:23.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:23.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:23.987781+0000) 2019-10-13T09:28:24.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:24.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:24.398323+0000) 2019-10-13T09:28:27.247 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:27.244+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.246876+0000) 2019-10-13T09:28:27.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:27.242+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.246781+0000) 2019-10-13T09:28:27.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:27.244+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.247868+0000) 2019-10-13T09:28:27.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:27.244+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.247954+0000) 2019-10-13T09:28:27.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:27.244+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.248507+0000) 2019-10-13T09:28:27.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:27.244+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.248714+0000) 2019-10-13T09:28:27.250 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:27.248+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.250113+0000) 2019-10-13T09:28:27.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:27.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:27.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:27.250+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.255863+0000) 2019-10-13T09:28:27.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:27.498+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.500926+0000) 2019-10-13T09:28:27.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:27.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.526164+0000) 2019-10-13T09:28:27.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:27.620+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.623982+0000) 2019-10-13T09:28:27.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:27.644+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:27.649704+0000) 2019-10-13T09:28:28.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:28.420+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:28.421684+0000) 2019-10-13T09:28:31.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:31.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:31.672126+0000) 2019-10-13T09:28:32.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:32.252+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.254118+0000) 2019-10-13T09:28:32.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:32.252+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.255355+0000) 2019-10-13T09:28:32.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:32.252+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.256217+0000) 2019-10-13T09:28:32.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:32.252+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.257005+0000) 2019-10-13T09:28:32.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:32.254+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.257006+0000) 2019-10-13T09:28:32.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:32.252+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.257672+0000) 2019-10-13T09:28:32.259 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:32.256+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.258810+0000) 2019-10-13T09:28:32.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:32.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:32.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:32.258+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:32.261533+0000) 2019-10-13T09:28:33.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:33.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:33.987987+0000) 2019-10-13T09:28:34.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:34.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:34.398536+0000) 2019-10-13T09:28:37.263 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:37.258+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.261666+0000) 2019-10-13T09:28:37.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:37.260+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.263802+0000) 2019-10-13T09:28:37.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:37.260+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.264569+0000) 2019-10-13T09:28:37.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:37.260+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.264996+0000) 2019-10-13T09:28:37.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:37.260+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.265203+0000) 2019-10-13T09:28:37.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:37.264+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.265937+0000) 2019-10-13T09:28:37.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:37.264+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.266275+0000) 2019-10-13T09:28:37.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:37.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:37.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:37.266+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.268960+0000) 2019-10-13T09:28:37.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:37.506+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.507387+0000) 2019-10-13T09:28:37.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:37.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.526362+0000) 2019-10-13T09:28:37.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:37.620+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.624159+0000) 2019-10-13T09:28:37.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:37.648+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:37.649902+0000) 2019-10-13T09:28:38.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:38.420+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:38.421956+0000) 2019-10-13T09:28:41.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:41.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:41.672383+0000) 2019-10-13T09:28:42.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:42.266+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.270405+0000) 2019-10-13T09:28:42.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:42.268+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.271188+0000) 2019-10-13T09:28:42.272 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:42.268+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.271472+0000) 2019-10-13T09:28:42.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:42.268+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.272125+0000) 2019-10-13T09:28:42.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:42.268+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.272608+0000) 2019-10-13T09:28:42.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:42.268+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.273613+0000) 2019-10-13T09:28:42.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:42.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:42.275 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:42.272+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.274788+0000) 2019-10-13T09:28:42.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:42.278+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:42.280738+0000) 2019-10-13T09:28:43.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:43.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:43.988187+0000) 2019-10-13T09:28:44.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:44.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:44.398739+0000) 2019-10-13T09:28:47.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:47.276+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.278172+0000) 2019-10-13T09:28:47.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:47.276+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.278597+0000) 2019-10-13T09:28:47.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:47.276+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.279175+0000) 2019-10-13T09:28:47.281 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:47.274+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.279907+0000) 2019-10-13T09:28:47.281 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:47.276+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.280150+0000) 2019-10-13T09:28:47.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:47.276+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.280552+0000) 2019-10-13T09:28:47.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:47.276+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.281016+0000) 2019-10-13T09:28:47.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:47.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:47.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:47.282+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.286272+0000) 2019-10-13T09:28:47.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:47.510+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.513891+0000) 2019-10-13T09:28:47.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:47.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.526532+0000) 2019-10-13T09:28:47.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:47.620+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.624355+0000) 2019-10-13T09:28:47.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:47.648+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:47.650073+0000) 2019-10-13T09:28:48.423 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:48.420+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:48.422175+0000) 2019-10-13T09:28:51.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:51.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:51.672608+0000) 2019-10-13T09:28:52.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:52.280+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.285715+0000) 2019-10-13T09:28:52.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:52.284+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.286134+0000) 2019-10-13T09:28:52.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:52.284+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.287018+0000) 2019-10-13T09:28:52.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:52.282+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.287233+0000) 2019-10-13T09:28:52.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:52.284+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.288027+0000) 2019-10-13T09:28:52.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:52.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.288523+0000) 2019-10-13T09:28:52.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:52.284+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.289194+0000) 2019-10-13T09:28:52.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:52.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:52.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:52.294+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:52.296541+0000) 2019-10-13T09:28:53.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:53.984+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:53.988385+0000) 2019-10-13T09:28:54.177 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:28:54.177 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:54.180 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:28:54.180 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:28:54.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:54.396+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:54.398958+0000) 2019-10-13T09:28:57.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:28:57.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.293983+0000) 2019-10-13T09:28:57.295 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:57.291+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.294315+0000) 2019-10-13T09:28:57.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:57.291+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.294730+0000) 2019-10-13T09:28:57.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:28:57.290+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.295190+0000) 2019-10-13T09:28:57.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:57.291+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.295840+0000) 2019-10-13T09:28:57.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:57.291+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.296385+0000) 2019-10-13T09:28:57.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:28:57.291+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.296987+0000) 2019-10-13T09:28:57.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:28:57.291+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:28:57.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:57.302+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.304655+0000) 2019-10-13T09:28:57.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:28:57.522+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.524487+0000) 2019-10-13T09:28:57.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:28:57.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.526728+0000) 2019-10-13T09:28:57.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:28:57.623+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.624535+0000) 2019-10-13T09:28:57.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:28:57.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:57.650244+0000) 2019-10-13T09:28:58.423 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:28:58.419+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:28:58.422381+0000) 2019-10-13T09:29:01.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:01.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:01.672792+0000) 2019-10-13T09:29:02.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:02.295+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.300188+0000) 2019-10-13T09:29:02.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:02.295+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.300827+0000) 2019-10-13T09:29:02.303 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:02.299+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.302610+0000) 2019-10-13T09:29:02.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:02.299+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.302911+0000) 2019-10-13T09:29:02.305 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:02.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:02.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:02.303+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.306069+0000) 2019-10-13T09:29:02.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:02.302+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.306261+0000) 2019-10-13T09:29:02.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:02.303+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.307582+0000) 2019-10-13T09:29:02.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:02.310+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:02.313512+0000) 2019-10-13T09:29:03.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:03.987+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:03.988606+0000) 2019-10-13T09:29:04.400 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:04.395+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:04.399195+0000) 2019-10-13T09:29:07.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:07.303+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.308175+0000) 2019-10-13T09:29:07.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:07.306+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.308508+0000) 2019-10-13T09:29:07.310 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:07.307+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.309954+0000) 2019-10-13T09:29:07.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:07.307+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.310527+0000) 2019-10-13T09:29:07.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:07.307+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.311037+0000) 2019-10-13T09:29:07.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:07.307+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.311569+0000) 2019-10-13T09:29:07.312 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:07.307+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.311297+0000) 2019-10-13T09:29:07.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:07.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95738 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:07.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:07.314+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.317696+0000) 2019-10-13T09:29:07.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:07.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.526933+0000) 2019-10-13T09:29:07.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:07.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.530923+0000) 2019-10-13T09:29:07.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:07.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.624713+0000) 2019-10-13T09:29:07.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:07.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:07.650439+0000) 2019-10-13T09:29:08.423 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:08.419+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:08.422569+0000) 2019-10-13T09:29:11.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:11.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:11.673045+0000) 2019-10-13T09:29:12.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:12.311+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.316949+0000) 2019-10-13T09:29:12.318 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:12.315+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.317911+0000) 2019-10-13T09:29:12.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:12.315+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.320078+0000) 2019-10-13T09:29:12.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:12.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:12.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:12.318+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.321789+0000) 2019-10-13T09:29:12.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:12.319+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.322416+0000) 2019-10-13T09:29:12.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:12.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.322816+0000) 2019-10-13T09:29:12.324 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:12.319+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.323920+0000) 2019-10-13T09:29:12.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:12.326+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:12.330338+0000) 2019-10-13T09:29:13.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:13.987+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:13.988824+0000) 2019-10-13T09:29:14.400 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:14.395+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:14.399440+0000) 2019-10-13T09:29:17.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:17.319+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.325070+0000) 2019-10-13T09:29:17.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:17.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.327362+0000) 2019-10-13T09:29:17.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:17.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.327536+0000) 2019-10-13T09:29:17.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:17.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:17.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:17.326+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.329500+0000) 2019-10-13T09:29:17.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:17.327+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.330303+0000) 2019-10-13T09:29:17.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:17.327+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.330825+0000) 2019-10-13T09:29:17.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:17.327+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.331440+0000) 2019-10-13T09:29:17.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:17.330+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.335445+0000) 2019-10-13T09:29:17.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:17.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.527109+0000) 2019-10-13T09:29:17.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:17.534+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.537496+0000) 2019-10-13T09:29:17.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:17.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.624882+0000) 2019-10-13T09:29:17.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:17.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:17.650632+0000) 2019-10-13T09:29:18.423 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:18.419+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:18.422789+0000) 2019-10-13T09:29:21.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:21.670+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:21.673235+0000) 2019-10-13T09:29:22.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:22.327+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.332427+0000) 2019-10-13T09:29:22.334 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:22.331+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.333959+0000) 2019-10-13T09:29:22.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:22.331+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.334391+0000) 2019-10-13T09:29:22.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:22.331+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.334590+0000) 2019-10-13T09:29:22.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:22.330+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.334529+0000) 2019-10-13T09:29:22.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:22.331+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.335701+0000) 2019-10-13T09:29:22.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:22.331+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.336412+0000) 2019-10-13T09:29:22.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:22.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:22.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:22.338+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:22.343856+0000) 2019-10-13T09:29:23.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:23.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:23.989049+0000) 2019-10-13T09:29:24.375 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:29:24.375 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:24.379 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:29:24.379 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:24.400 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:24.395+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:24.399647+0000) 2019-10-13T09:29:27.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:27.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.340356+0000) 2019-10-13T09:29:27.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:27.339+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.342954+0000) 2019-10-13T09:29:27.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:27.338+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.343594+0000) 2019-10-13T09:29:27.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:27.339+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.344255+0000) 2019-10-13T09:29:27.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:27.339+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.344610+0000) 2019-10-13T09:29:27.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:27.339+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.345446+0000) 2019-10-13T09:29:27.346 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:27.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:27.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:27.343+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.345982+0000) 2019-10-13T09:29:27.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:27.346+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.349289+0000) 2019-10-13T09:29:27.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:27.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.527321+0000) 2019-10-13T09:29:27.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:27.546+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.547989+0000) 2019-10-13T09:29:27.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:27.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.625082+0000) 2019-10-13T09:29:27.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:27.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:27.650807+0000) 2019-10-13T09:29:28.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:28.419+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:28.423016+0000) 2019-10-13T09:29:31.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:31.669+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:31.673456+0000) 2019-10-13T09:29:32.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:32.343+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.349186+0000) 2019-10-13T09:29:32.351 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:32.345+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.349490+0000) 2019-10-13T09:29:32.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:32.347+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.349973+0000) 2019-10-13T09:29:32.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:32.347+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.350398+0000) 2019-10-13T09:29:32.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:32.347+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.351086+0000) 2019-10-13T09:29:32.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:32.347+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.351631+0000) 2019-10-13T09:29:32.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:32.347+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.352133+0000) 2019-10-13T09:29:32.353 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:32.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:32.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:32.353+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:32.357378+0000) 2019-10-13T09:29:33.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:33.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:33.989261+0000) 2019-10-13T09:29:34.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:34.395+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:34.399879+0000) 2019-10-13T09:29:37.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:37.353+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.356228+0000) 2019-10-13T09:29:37.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:37.351+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.356864+0000) 2019-10-13T09:29:37.360 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:37.355+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.359512+0000) 2019-10-13T09:29:37.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:37.355+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.359633+0000) 2019-10-13T09:29:37.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:37.355+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.360152+0000) 2019-10-13T09:29:37.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:37.355+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:37.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:37.359+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.361990+0000) 2019-10-13T09:29:37.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:37.359+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.362465+0000) 2019-10-13T09:29:37.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:37.361+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.365576+0000) 2019-10-13T09:29:37.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:37.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.527499+0000) 2019-10-13T09:29:37.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:37.549+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.554573+0000) 2019-10-13T09:29:37.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:37.623+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.625275+0000) 2019-10-13T09:29:37.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:37.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:37.650984+0000) 2019-10-13T09:29:38.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:38.419+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:38.423229+0000) 2019-10-13T09:29:41.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:41.669+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:41.673647+0000) 2019-10-13T09:29:42.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:42.361+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.364368+0000) 2019-10-13T09:29:42.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:42.359+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.364619+0000) 2019-10-13T09:29:42.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:42.363+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.366897+0000) 2019-10-13T09:29:42.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:42.363+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.367527+0000) 2019-10-13T09:29:42.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:42.363+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.367933+0000) 2019-10-13T09:29:42.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:42.363+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.368175+0000) 2019-10-13T09:29:42.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:42.363+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.368422+0000) 2019-10-13T09:29:42.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:42.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:42.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:42.369+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:42.372779+0000) 2019-10-13T09:29:43.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:43.983+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:43.989465+0000) 2019-10-13T09:29:44.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:44.395+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:44.400077+0000) 2019-10-13T09:29:47.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:47.367+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.372764+0000) 2019-10-13T09:29:47.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:47.367+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.373122+0000) 2019-10-13T09:29:47.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:47.371+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.374922+0000) 2019-10-13T09:29:47.376 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:47.371+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.375953+0000) 2019-10-13T09:29:47.377 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:47.369+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.375934+0000) 2019-10-13T09:29:47.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:47.371+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.376783+0000) 2019-10-13T09:29:47.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:47.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:47.378 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:47.371+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.377333+0000) 2019-10-13T09:29:47.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:47.377+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.380924+0000) 2019-10-13T09:29:47.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:47.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.527709+0000) 2019-10-13T09:29:47.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:47.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.560961+0000) 2019-10-13T09:29:47.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:47.619+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.625466+0000) 2019-10-13T09:29:47.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:47.647+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:47.651116+0000) 2019-10-13T09:29:48.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:48.423+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:48.423456+0000) 2019-10-13T09:29:51.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:51.669+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:51.673864+0000) 2019-10-13T09:29:52.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:52.374+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.380048+0000) 2019-10-13T09:29:52.381 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:52.377+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.380516+0000) 2019-10-13T09:29:52.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:52.374+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.380643+0000) 2019-10-13T09:29:52.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:52.378+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.382420+0000) 2019-10-13T09:29:52.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:52.378+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.382791+0000) 2019-10-13T09:29:52.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:52.378+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.383160+0000) 2019-10-13T09:29:52.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:52.378+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.383661+0000) 2019-10-13T09:29:52.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:52.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:52.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:52.381+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:52.385930+0000) 2019-10-13T09:29:53.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:53.982+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:53.989691+0000) 2019-10-13T09:29:54.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:54.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:54.400294+0000) 2019-10-13T09:29:54.483 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:29:54.483 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:54.487 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:29:54.487 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:29:57.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:29:57.382+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.388444+0000) 2019-10-13T09:29:57.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:29:57.385+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.388482+0000) 2019-10-13T09:29:57.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:57.382+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.389123+0000) 2019-10-13T09:29:57.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:29:57.386+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.390443+0000) 2019-10-13T09:29:57.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:57.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.390595+0000) 2019-10-13T09:29:57.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:57.386+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.391145+0000) 2019-10-13T09:29:57.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:57.386+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.391335+0000) 2019-10-13T09:29:57.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:29:57.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:29:57.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:57.389+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.395584+0000) 2019-10-13T09:29:57.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:29:57.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.527930+0000) 2019-10-13T09:29:57.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:29:57.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.567580+0000) 2019-10-13T09:29:57.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:29:57.622+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.625638+0000) 2019-10-13T09:29:57.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:29:57.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:57.651286+0000) 2019-10-13T09:29:58.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:29:58.418+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:29:58.423650+0000) 2019-10-13T09:30:01.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:01.673+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:01.674067+0000) 2019-10-13T09:30:02.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:02.390+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.395551+0000) 2019-10-13T09:30:02.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:02.389+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.395449+0000) 2019-10-13T09:30:02.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:02.390+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.396579+0000) 2019-10-13T09:30:02.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:02.390+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.396998+0000) 2019-10-13T09:30:02.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:02.390+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.397473+0000) 2019-10-13T09:30:02.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:02.394+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.397727+0000) 2019-10-13T09:30:02.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:02.394+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.398229+0000) 2019-10-13T09:30:02.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:02.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:02.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:02.401+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:02.404944+0000) 2019-10-13T09:30:03.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:03.986+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:03.989914+0000) 2019-10-13T09:30:04.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:04.398+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:04.400496+0000) 2019-10-13T09:30:07.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:07.397+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.402685+0000) 2019-10-13T09:30:07.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:07.398+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.403507+0000) 2019-10-13T09:30:07.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:07.398+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.403922+0000) 2019-10-13T09:30:07.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:07.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.405784+0000) 2019-10-13T09:30:07.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:07.402+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.406063+0000) 2019-10-13T09:30:07.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:07.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.406385+0000) 2019-10-13T09:30:07.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:07.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.406709+0000) 2019-10-13T09:30:07.408 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:07.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:07.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:07.405+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.411325+0000) 2019-10-13T09:30:07.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:07.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.528101+0000) 2019-10-13T09:30:07.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:07.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.574111+0000) 2019-10-13T09:30:07.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:07.622+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.625850+0000) 2019-10-13T09:30:07.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:07.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:07.651505+0000) 2019-10-13T09:30:08.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:08.422+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:08.423871+0000) 2019-10-13T09:30:11.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:11.669+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:11.674290+0000) 2019-10-13T09:30:12.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:12.406+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.412069+0000) 2019-10-13T09:30:12.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:12.409+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.412611+0000) 2019-10-13T09:30:12.414 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:12.410+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.413728+0000) 2019-10-13T09:30:12.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:12.410+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.414328+0000) 2019-10-13T09:30:12.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:12.410+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.414958+0000) 2019-10-13T09:30:12.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:12.410+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.415076+0000) 2019-10-13T09:30:12.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:12.410+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.415458+0000) 2019-10-13T09:30:12.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:12.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:12.420 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:12.413+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:12.419191+0000) 2019-10-13T09:30:13.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:13.986+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:13.990117+0000) 2019-10-13T09:30:14.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:14.398+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:14.400710+0000) 2019-10-13T09:30:17.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:17.414+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.420394+0000) 2019-10-13T09:30:17.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:17.414+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.420427+0000) 2019-10-13T09:30:17.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:17.414+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.421223+0000) 2019-10-13T09:30:17.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:17.418+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.422398+0000) 2019-10-13T09:30:17.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:17.417+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.424003+0000) 2019-10-13T09:30:17.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:17.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96151 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:17.426 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:17.422+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.425885+0000) 2019-10-13T09:30:17.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:17.422+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.426289+0000) 2019-10-13T09:30:17.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:17.425+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.431485+0000) 2019-10-13T09:30:17.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:17.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.528351+0000) 2019-10-13T09:30:17.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:17.577+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.583124+0000) 2019-10-13T09:30:17.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:17.622+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.625988+0000) 2019-10-13T09:30:17.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:17.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:17.651698+0000) 2019-10-13T09:30:18.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:18.418+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:18.424021+0000) 2019-10-13T09:30:21.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:21.673+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:21.674549+0000) 2019-10-13T09:30:22.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:22.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.429349+0000) 2019-10-13T09:30:22.431 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:22.422+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.429295+0000) 2019-10-13T09:30:22.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:22.426+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.429980+0000) 2019-10-13T09:30:22.431 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:22.426+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.430502+0000) 2019-10-13T09:30:22.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:22.426+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.431006+0000) 2019-10-13T09:30:22.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:22.426+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.431471+0000) 2019-10-13T09:30:22.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:22.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:22.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:22.429+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.432488+0000) 2019-10-13T09:30:22.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:22.433+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:22.440059+0000) 2019-10-13T09:30:23.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:23.986+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:23.990349+0000) 2019-10-13T09:30:24.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:24.394+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:24.400911+0000) 2019-10-13T09:30:24.560 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:30:24.560 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:24.565 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:30:24.565 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:27.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:27.430+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.436369+0000) 2019-10-13T09:30:27.438 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:27.433+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.436565+0000) 2019-10-13T09:30:27.438 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:27.430+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.436704+0000) 2019-10-13T09:30:27.439 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:27.430+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.437252+0000) 2019-10-13T09:30:27.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:27.430+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.437594+0000) 2019-10-13T09:30:27.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:27.434+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.437891+0000) 2019-10-13T09:30:27.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:27.434+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.438745+0000) 2019-10-13T09:30:27.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:27.441+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.444453+0000) 2019-10-13T09:30:27.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:27.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:27.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:27.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.528555+0000) 2019-10-13T09:30:27.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:27.589+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.589702+0000) 2019-10-13T09:30:27.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:27.622+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.626180+0000) 2019-10-13T09:30:27.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:27.646+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:27.651871+0000) 2019-10-13T09:30:28.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:28.422+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:28.424251+0000) 2019-10-13T09:30:31.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:31.673+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:31.674762+0000) 2019-10-13T09:30:32.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:32.438+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.443726+0000) 2019-10-13T09:30:32.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:32.438+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.444395+0000) 2019-10-13T09:30:32.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:32.442+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.446390+0000) 2019-10-13T09:30:32.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:32.442+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.446558+0000) 2019-10-13T09:30:32.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:32.442+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.446623+0000) 2019-10-13T09:30:32.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:32.441+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.446968+0000) 2019-10-13T09:30:32.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:32.442+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.447655+0000) 2019-10-13T09:30:32.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:32.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:32.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:32.445+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:32.450411+0000) 2019-10-13T09:30:33.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:33.990+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:33.990551+0000) 2019-10-13T09:30:34.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:34.398+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:34.401115+0000) 2019-10-13T09:30:37.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:37.446+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.451744+0000) 2019-10-13T09:30:37.453 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:37.446+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.452129+0000) 2019-10-13T09:30:37.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:37.448+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.452801+0000) 2019-10-13T09:30:37.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:37.450+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.453835+0000) 2019-10-13T09:30:37.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:37.450+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.454268+0000) 2019-10-13T09:30:37.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:37.450+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.454782+0000) 2019-10-13T09:30:37.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:37.450+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.456039+0000) 2019-10-13T09:30:37.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:37.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:37.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:37.456+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.461194+0000) 2019-10-13T09:30:37.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:37.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.528776+0000) 2019-10-13T09:30:37.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:37.588+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.595602+0000) 2019-10-13T09:30:37.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:37.626+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.626358+0000) 2019-10-13T09:30:37.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:37.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:37.652067+0000) 2019-10-13T09:30:38.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:38.418+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:38.424424+0000) 2019-10-13T09:30:41.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:41.672+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:41.674961+0000) 2019-10-13T09:30:42.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:42.454+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.461388+0000) 2019-10-13T09:30:42.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:42.458+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.462699+0000) 2019-10-13T09:30:42.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:42.458+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.463986+0000) 2019-10-13T09:30:42.465 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:42.458+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.464043+0000) 2019-10-13T09:30:42.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:42.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:42.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:42.460+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.464337+0000) 2019-10-13T09:30:42.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:42.458+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.465089+0000) 2019-10-13T09:30:42.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:42.458+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.465546+0000) 2019-10-13T09:30:42.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:42.464+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:42.470367+0000) 2019-10-13T09:30:43.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:43.986+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:43.990772+0000) 2019-10-13T09:30:44.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:44.398+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:44.401349+0000) 2019-10-13T09:30:47.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:47.460+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.466760+0000) 2019-10-13T09:30:47.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:47.462+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.469103+0000) 2019-10-13T09:30:47.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:47.462+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.469324+0000) 2019-10-13T09:30:47.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:47.466+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.471100+0000) 2019-10-13T09:30:47.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:47.466+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.471644+0000) 2019-10-13T09:30:47.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:47.466+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.472019+0000) 2019-10-13T09:30:47.473 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:47.466+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:47.473 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:47.466+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.472685+0000) 2019-10-13T09:30:47.477 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:47.472+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.476578+0000) 2019-10-13T09:30:47.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:47.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.528969+0000) 2019-10-13T09:30:47.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:47.600+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.602204+0000) 2019-10-13T09:30:47.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:47.626+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.626542+0000) 2019-10-13T09:30:47.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:47.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:47.652266+0000) 2019-10-13T09:30:48.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:48.417+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:48.424606+0000) 2019-10-13T09:30:51.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:51.672+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:51.675113+0000) 2019-10-13T09:30:52.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:52.468+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.474947+0000) 2019-10-13T09:30:52.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:52.469+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.475738+0000) 2019-10-13T09:30:52.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:52.469+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.476235+0000) 2019-10-13T09:30:52.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:52.469+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.476723+0000) 2019-10-13T09:30:52.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:52.469+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.477439+0000) 2019-10-13T09:30:52.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:52.473+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.478403+0000) 2019-10-13T09:30:52.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:52.473+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.478999+0000) 2019-10-13T09:30:52.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:52.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:52.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:52.480+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:52.485967+0000) 2019-10-13T09:30:53.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:53.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:53.990951+0000) 2019-10-13T09:30:54.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:54.397+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:54.401549+0000) 2019-10-13T09:30:54.682 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:30:54.682 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:54.686 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:30:54.686 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:30:57.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:30:57.476+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.483282+0000) 2019-10-13T09:30:57.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:57.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.484564+0000) 2019-10-13T09:30:57.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:57.477+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.485328+0000) 2019-10-13T09:30:57.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:30:57.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.485700+0000) 2019-10-13T09:30:57.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:30:57.481+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.486380+0000) 2019-10-13T09:30:57.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:57.481+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.486466+0000) 2019-10-13T09:30:57.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:57.481+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.487292+0000) 2019-10-13T09:30:57.488 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:30:57.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:30:57.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:57.484+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.490613+0000) 2019-10-13T09:30:57.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:30:57.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.529143+0000) 2019-10-13T09:30:57.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:30:57.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.608690+0000) 2019-10-13T09:30:57.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:30:57.625+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.626716+0000) 2019-10-13T09:30:57.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:30:57.645+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:57.652458+0000) 2019-10-13T09:30:58.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:30:58.421+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:30:58.424781+0000) 2019-10-13T09:31:01.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:01.672+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:01.675263+0000) 2019-10-13T09:31:02.492 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:02.489+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.491674+0000) 2019-10-13T09:31:02.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:02.489+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.492951+0000) 2019-10-13T09:31:02.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:02.489+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.493348+0000) 2019-10-13T09:31:02.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:02.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.493553+0000) 2019-10-13T09:31:02.495 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:02.488+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.493479+0000) 2019-10-13T09:31:02.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:02.493+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.494132+0000) 2019-10-13T09:31:02.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:02.493+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.494606+0000) 2019-10-13T09:31:02.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:02.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:02.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:02.496+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:02.500742+0000) 2019-10-13T09:31:03.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:03.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:03.991157+0000) 2019-10-13T09:31:04.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:04.401+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:04.401778+0000) 2019-10-13T09:31:07.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:07.497+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.498972+0000) 2019-10-13T09:31:07.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:07.497+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.499557+0000) 2019-10-13T09:31:07.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:07.496+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.500608+0000) 2019-10-13T09:31:07.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:07.497+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.501309+0000) 2019-10-13T09:31:07.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:07.501+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.502827+0000) 2019-10-13T09:31:07.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:07.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:07.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:07.501+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.503785+0000) 2019-10-13T09:31:07.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:07.501+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.504101+0000) 2019-10-13T09:31:07.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:07.500+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.504560+0000) 2019-10-13T09:31:07.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:07.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.529331+0000) 2019-10-13T09:31:07.616 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:07.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.615210+0000) 2019-10-13T09:31:07.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:07.625+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.626921+0000) 2019-10-13T09:31:07.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:07.649+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:07.652666+0000) 2019-10-13T09:31:08.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:08.421+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:08.424962+0000) 2019-10-13T09:31:11.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:11.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:11.675485+0000) 2019-10-13T09:31:12.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:12.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.507236+0000) 2019-10-13T09:31:12.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:12.500+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.507188+0000) 2019-10-13T09:31:12.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:12.505+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.508608+0000) 2019-10-13T09:31:12.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:12.505+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.509452+0000) 2019-10-13T09:31:12.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:12.505+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.509693+0000) 2019-10-13T09:31:12.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:12.509+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.510604+0000) 2019-10-13T09:31:12.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:12.509+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:12.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:12.509+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.511753+0000) 2019-10-13T09:31:12.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:12.512+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:12.516772+0000) 2019-10-13T09:31:13.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:13.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:13.991361+0000) 2019-10-13T09:31:14.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:14.401+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:14.402012+0000) 2019-10-13T09:31:17.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:17.513+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.514908+0000) 2019-10-13T09:31:17.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:17.513+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.515582+0000) 2019-10-13T09:31:17.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:17.513+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.516300+0000) 2019-10-13T09:31:17.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:17.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.518056+0000) 2019-10-13T09:31:17.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:17.512+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.517983+0000) 2019-10-13T09:31:17.519 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:17.517+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.518943+0000) 2019-10-13T09:31:17.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:17.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:17.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:17.517+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.520519+0000) 2019-10-13T09:31:17.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:17.520+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.525294+0000) 2019-10-13T09:31:17.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:17.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.529463+0000) 2019-10-13T09:31:17.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:17.616+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.621866+0000) 2019-10-13T09:31:17.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:17.625+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.627097+0000) 2019-10-13T09:31:17.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:17.649+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:17.652882+0000) 2019-10-13T09:31:18.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:18.421+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:18.425141+0000) 2019-10-13T09:31:21.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:21.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:21.675681+0000) 2019-10-13T09:31:22.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:22.521+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.524017+0000) 2019-10-13T09:31:22.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:22.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.525606+0000) 2019-10-13T09:31:22.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:22.525+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.526235+0000) 2019-10-13T09:31:22.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:22.525+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.526411+0000) 2019-10-13T09:31:22.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:22.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:22.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:22.525+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.527926+0000) 2019-10-13T09:31:22.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:22.524+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.528082+0000) 2019-10-13T09:31:22.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:22.525+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.528660+0000) 2019-10-13T09:31:22.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:22.528+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:22.533683+0000) 2019-10-13T09:31:23.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:23.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:23.991569+0000) 2019-10-13T09:31:24.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:24.401+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:24.402231+0000) 2019-10-13T09:31:24.826 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:31:24.826 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:24.830 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:31:24.830 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:27.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:27.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.530544+0000) 2019-10-13T09:31:27.533 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:27.529+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.532707+0000) 2019-10-13T09:31:27.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:27.529+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.532975+0000) 2019-10-13T09:31:27.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:27.529+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.533193+0000) 2019-10-13T09:31:27.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:27.528+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.533215+0000) 2019-10-13T09:31:27.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:27.533+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.534042+0000) 2019-10-13T09:31:27.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:27.533+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.534678+0000) 2019-10-13T09:31:27.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:27.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:27.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:27.536+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.542928+0000) 2019-10-13T09:31:27.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:27.625+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.627293+0000) 2019-10-13T09:31:27.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:27.628+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.632231+0000) 2019-10-13T09:31:27.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:27.649+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:27.653055+0000) 2019-10-13T09:31:28.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:28.421+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:28.425356+0000) 2019-10-13T09:31:31.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:31.668+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:31.675904+0000) 2019-10-13T09:31:32.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:32.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.538728+0000) 2019-10-13T09:31:32.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:32.537+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.539575+0000) 2019-10-13T09:31:32.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:32.537+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.541054+0000) 2019-10-13T09:31:32.546 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:32.541+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.543807+0000) 2019-10-13T09:31:32.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:32.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:32.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:32.541+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.544573+0000) 2019-10-13T09:31:32.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:32.541+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.544949+0000) 2019-10-13T09:31:32.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:32.540+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.545830+0000) 2019-10-13T09:31:32.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:32.544+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:32.549024+0000) 2019-10-13T09:31:34.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:33.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:33.991794+0000) 2019-10-13T09:31:34.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:34.277+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:34.280203+0000) 2019-10-13T09:31:34.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:34.401+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:34.402388+0000) 2019-10-13T09:31:37.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:37.545+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.546341+0000) 2019-10-13T09:31:37.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:37.545+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.546781+0000) 2019-10-13T09:31:37.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:37.545+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.547367+0000) 2019-10-13T09:31:37.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:37.545+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.548543+0000) 2019-10-13T09:31:37.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:37.545+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.549304+0000) 2019-10-13T09:31:37.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:37.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.549180+0000) 2019-10-13T09:31:37.550 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:37.549+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.550057+0000) 2019-10-13T09:31:37.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:37.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:37.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:37.552+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.553987+0000) 2019-10-13T09:31:37.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:37.625+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.627493+0000) 2019-10-13T09:31:37.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:37.636+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.636182+0000) 2019-10-13T09:31:37.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:37.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:37.653253+0000) 2019-10-13T09:31:38.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:38.421+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:38.425567+0000) 2019-10-13T09:31:41.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:41.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:41.676130+0000) 2019-10-13T09:31:42.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:42.553+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.555075+0000) 2019-10-13T09:31:42.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:42.553+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.555520+0000) 2019-10-13T09:31:42.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:42.553+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.557410+0000) 2019-10-13T09:31:42.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:42.557+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.559202+0000) 2019-10-13T09:31:42.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:42.555+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.559222+0000) 2019-10-13T09:31:42.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:42.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:42.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:42.557+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.559773+0000) 2019-10-13T09:31:42.561 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:42.557+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.560417+0000) 2019-10-13T09:31:42.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:42.563+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:42.565342+0000) 2019-10-13T09:31:43.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:43.989+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:43.992034+0000) 2019-10-13T09:31:44.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:44.277+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:44.280427+0000) 2019-10-13T09:31:44.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:44.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:44.402589+0000) 2019-10-13T09:31:47.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:47.560+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.562464+0000) 2019-10-13T09:31:47.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:47.560+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.563388+0000) 2019-10-13T09:31:47.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:47.560+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.564048+0000) 2019-10-13T09:31:47.567 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:47.564+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.566937+0000) 2019-10-13T09:31:47.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:47.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:47.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:47.564+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.567189+0000) 2019-10-13T09:31:47.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:47.563+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.567943+0000) 2019-10-13T09:31:47.569 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:47.564+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.568838+0000) 2019-10-13T09:31:47.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:47.571+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.574203+0000) 2019-10-13T09:31:47.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:47.624+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.627698+0000) 2019-10-13T09:31:47.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:47.639+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.642861+0000) 2019-10-13T09:31:47.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:47.648+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:47.653401+0000) 2019-10-13T09:31:48.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:48.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:48.425811+0000) 2019-10-13T09:31:51.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:51.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:51.676359+0000) 2019-10-13T09:31:52.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:52.568+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.570715+0000) 2019-10-13T09:31:52.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:52.568+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.570954+0000) 2019-10-13T09:31:52.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:52.568+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.571577+0000) 2019-10-13T09:31:52.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:52.571+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.572308+0000) 2019-10-13T09:31:52.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:52.568+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.572900+0000) 2019-10-13T09:31:52.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:52.568+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.573336+0000) 2019-10-13T09:31:52.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:52.568+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.573551+0000) 2019-10-13T09:31:52.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:52.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:52.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:52.579+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:52.582197+0000) 2019-10-13T09:31:53.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:53.988+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:53.992233+0000) 2019-10-13T09:31:54.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:54.276+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:54.280626+0000) 2019-10-13T09:31:54.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:54.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:54.402803+0000) 2019-10-13T09:31:55.183 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:31:55.183 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:55.187 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:31:55.187 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:31:57.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:31:57.576+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.578303+0000) 2019-10-13T09:31:57.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:57.576+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.578828+0000) 2019-10-13T09:31:57.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:31:57.575+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.579059+0000) 2019-10-13T09:31:57.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:57.576+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.579749+0000) 2019-10-13T09:31:57.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:31:57.576+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.580443+0000) 2019-10-13T09:31:57.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:31:57.576+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.581487+0000) 2019-10-13T09:31:57.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:57.576+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.581699+0000) 2019-10-13T09:31:57.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:31:57.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:31:57.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:57.587+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.588854+0000) 2019-10-13T09:31:57.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:31:57.624+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.627907+0000) 2019-10-13T09:31:57.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:31:57.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.648864+0000) 2019-10-13T09:31:57.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:31:57.648+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:57.653574+0000) 2019-10-13T09:31:58.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:31:58.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:31:58.425993+0000) 2019-10-13T09:32:01.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:01.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:01.676532+0000) 2019-10-13T09:32:02.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:02.584+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.586122+0000) 2019-10-13T09:32:02.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:02.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.585913+0000) 2019-10-13T09:32:02.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:02.584+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.586928+0000) 2019-10-13T09:32:02.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:02.584+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.589391+0000) 2019-10-13T09:32:02.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:02.584+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.589690+0000) 2019-10-13T09:32:02.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:02.588+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.590469+0000) 2019-10-13T09:32:02.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:02.588+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.591211+0000) 2019-10-13T09:32:02.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:02.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:02.598 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:02.595+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:02.597681+0000) 2019-10-13T09:32:03.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:03.988+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:03.992434+0000) 2019-10-13T09:32:04.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:04.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:04.280829+0000) 2019-10-13T09:32:04.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:04.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:04.403029+0000) 2019-10-13T09:32:07.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:07.592+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.596359+0000) 2019-10-13T09:32:07.597 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:07.592+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.597142+0000) 2019-10-13T09:32:07.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:07.596+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.598630+0000) 2019-10-13T09:32:07.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:07.596+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.598969+0000) 2019-10-13T09:32:07.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:07.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:07.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:07.596+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.599554+0000) 2019-10-13T09:32:07.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:07.596+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.599321+0000) 2019-10-13T09:32:07.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:07.595+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.599572+0000) 2019-10-13T09:32:07.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:07.599+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.601908+0000) 2019-10-13T09:32:07.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:07.624+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.628094+0000) 2019-10-13T09:32:07.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:07.652+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.653778+0000) 2019-10-13T09:32:07.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:07.651+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:07.654951+0000) 2019-10-13T09:32:08.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:08.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:08.426179+0000) 2019-10-13T09:32:11.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:11.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:11.676689+0000) 2019-10-13T09:32:12.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:12.600+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.602425+0000) 2019-10-13T09:32:12.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:12.600+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.602984+0000) 2019-10-13T09:32:12.604 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:12.600+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.603755+0000) 2019-10-13T09:32:12.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:12.600+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.604266+0000) 2019-10-13T09:32:12.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:12.603+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.604869+0000) 2019-10-13T09:32:12.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:12.604+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.606477+0000) 2019-10-13T09:32:12.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:12.604+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.607208+0000) 2019-10-13T09:32:12.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:12.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:12.611 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:12.607+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:12.610443+0000) 2019-10-13T09:32:13.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:13.988+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:13.992638+0000) 2019-10-13T09:32:14.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:14.276+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:14.281029+0000) 2019-10-13T09:32:14.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:14.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:14.403242+0000) 2019-10-13T09:32:17.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:17.608+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.610466+0000) 2019-10-13T09:32:17.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:17.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.611704+0000) 2019-10-13T09:32:17.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:17.608+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.612575+0000) 2019-10-13T09:32:17.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:17.608+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.613616+0000) 2019-10-13T09:32:17.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:17.612+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.614091+0000) 2019-10-13T09:32:17.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:17.612+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.614772+0000) 2019-10-13T09:32:17.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:17.612+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.615447+0000) 2019-10-13T09:32:17.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:17.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:17.620 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:17.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.619457+0000) 2019-10-13T09:32:17.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:17.624+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.628281+0000) 2019-10-13T09:32:17.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:17.652+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.653955+0000) 2019-10-13T09:32:17.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:17.659+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:17.661465+0000) 2019-10-13T09:32:18.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:18.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:18.426389+0000) 2019-10-13T09:32:21.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:21.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:21.676902+0000) 2019-10-13T09:32:22.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:22.616+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.618557+0000) 2019-10-13T09:32:22.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:22.616+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.619079+0000) 2019-10-13T09:32:22.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:22.615+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.619126+0000) 2019-10-13T09:32:22.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:22.616+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.620282+0000) 2019-10-13T09:32:22.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:22.616+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.620749+0000) 2019-10-13T09:32:22.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:22.620+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.622211+0000) 2019-10-13T09:32:22.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:22.620+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.622539+0000) 2019-10-13T09:32:22.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:22.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:22.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:22.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:22.627533+0000) 2019-10-13T09:32:23.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:23.992+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:23.992863+0000) 2019-10-13T09:32:24.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:24.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:24.281254+0000) 2019-10-13T09:32:24.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:24.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:24.403437+0000) 2019-10-13T09:32:25.308 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:32:25.308 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:25.312 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:32:25.312 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:27.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:27.623+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.625763+0000) 2019-10-13T09:32:27.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:27.624+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.628336+0000) 2019-10-13T09:32:27.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:27.624+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.628042+0000) 2019-10-13T09:32:27.630 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:27.624+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.629341+0000) 2019-10-13T09:32:27.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:27.628+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.629952+0000) 2019-10-13T09:32:27.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:27.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:27.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:27.628+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.630859+0000) 2019-10-13T09:32:27.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:27.628+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.631131+0000) 2019-10-13T09:32:27.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:27.631+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.635943+0000) 2019-10-13T09:32:27.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:27.652+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.654092+0000) 2019-10-13T09:32:27.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:27.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:27.670356+0000) 2019-10-13T09:32:28.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:28.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:28.426527+0000) 2019-10-13T09:32:31.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:31.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:31.677101+0000) 2019-10-13T09:32:32.635 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:32.632+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.634343+0000) 2019-10-13T09:32:32.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:32.632+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.636181+0000) 2019-10-13T09:32:32.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:32.632+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.636870+0000) 2019-10-13T09:32:32.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:32.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:32.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:32.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.638313+0000) 2019-10-13T09:32:32.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:32.636+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.638966+0000) 2019-10-13T09:32:32.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:32.636+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.639226+0000) 2019-10-13T09:32:32.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:32.636+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.639551+0000) 2019-10-13T09:32:32.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:32.643+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:32.644499+0000) 2019-10-13T09:32:33.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:33.988+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:33.993058+0000) 2019-10-13T09:32:34.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:34.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:34.281444+0000) 2019-10-13T09:32:34.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:34.376+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:34.379175+0000) 2019-10-13T09:32:34.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:34.400+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:34.403642+0000) 2019-10-13T09:32:37.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:37.636+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.641009+0000) 2019-10-13T09:32:37.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:37.636+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.641527+0000) 2019-10-13T09:32:37.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:37.640+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.641801+0000) 2019-10-13T09:32:37.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:37.639+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.642391+0000) 2019-10-13T09:32:37.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:37.640+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.643477+0000) 2019-10-13T09:32:37.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:37.640+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.643762+0000) 2019-10-13T09:32:37.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:37.640+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.644821+0000) 2019-10-13T09:32:37.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:37.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:37.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:37.647+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.651694+0000) 2019-10-13T09:32:37.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:37.652+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.654211+0000) 2019-10-13T09:32:37.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:37.675+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:37.676843+0000) 2019-10-13T09:32:38.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:38.424+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:38.426735+0000) 2019-10-13T09:32:41.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:41.675+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:41.677322+0000) 2019-10-13T09:32:42.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:42.643+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.648543+0000) 2019-10-13T09:32:42.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:42.643+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.649093+0000) 2019-10-13T09:32:42.651 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:42.647+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.650548+0000) 2019-10-13T09:32:42.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:42.647+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.650443+0000) 2019-10-13T09:32:42.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:42.647+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.653334+0000) 2019-10-13T09:32:42.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:42.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:42.655 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:42.651+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.654893+0000) 2019-10-13T09:32:42.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:42.651+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.655351+0000) 2019-10-13T09:32:42.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:42.651+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:42.655951+0000) 2019-10-13T09:32:43.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:43.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:43.993256+0000) 2019-10-13T09:32:44.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:44.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:44.281647+0000) 2019-10-13T09:32:44.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:44.375+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:44.379379+0000) 2019-10-13T09:32:44.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:44.399+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:44.403886+0000) 2019-10-13T09:32:47.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:47.651+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.654385+0000) 2019-10-13T09:32:47.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:47.655+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.658286+0000) 2019-10-13T09:32:47.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:47.655+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.658579+0000) 2019-10-13T09:32:47.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:47.655+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.659141+0000) 2019-10-13T09:32:47.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:47.655+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.660987+0000) 2019-10-13T09:32:47.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:47.658+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.660840+0000) 2019-10-13T09:32:47.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:47.659+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.662891+0000) 2019-10-13T09:32:47.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:47.659+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.663129+0000) 2019-10-13T09:32:47.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:47.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:47.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:47.666+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.669363+0000) 2019-10-13T09:32:47.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:47.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:47.688168+0000) 2019-10-13T09:32:48.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:48.423+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:48.426914+0000) 2019-10-13T09:32:51.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:51.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:51.677515+0000) 2019-10-13T09:32:52.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:52.663+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.666817+0000) 2019-10-13T09:32:52.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:52.663+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.667248+0000) 2019-10-13T09:32:52.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:52.663+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.667507+0000) 2019-10-13T09:32:52.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:52.663+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.668012+0000) 2019-10-13T09:32:52.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:52.662+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.667980+0000) 2019-10-13T09:32:52.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:52.663+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.668829+0000) 2019-10-13T09:32:52.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:52.667+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.669793+0000) 2019-10-13T09:32:52.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:52.667+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:52.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:52.670+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:52.675747+0000) 2019-10-13T09:32:53.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:53.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:53.993457+0000) 2019-10-13T09:32:54.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:54.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:54.281848+0000) 2019-10-13T09:32:54.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:54.375+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:54.379595+0000) 2019-10-13T09:32:54.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:54.399+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:54.404104+0000) 2019-10-13T09:32:55.452 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:32:55.453 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:55.456 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:32:55.457 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:32:57.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:57.651+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.654588+0000) 2019-10-13T09:32:57.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:32:57.671+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.674289+0000) 2019-10-13T09:32:57.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:32:57.671+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.674554+0000) 2019-10-13T09:32:57.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:57.671+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.675256+0000) 2019-10-13T09:32:57.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:32:57.671+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.676022+0000) 2019-10-13T09:32:57.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:32:57.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.676592+0000) 2019-10-13T09:32:57.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:32:57.671+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.676939+0000) 2019-10-13T09:32:57.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:32:57.674+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.678223+0000) 2019-10-13T09:32:57.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:32:57.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:32:57.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:57.682+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.684789+0000) 2019-10-13T09:32:57.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:32:57.690+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:57.693782+0000) 2019-10-13T09:32:58.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:32:58.423+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:32:58.427108+0000) 2019-10-13T09:33:01.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:01.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:01.677713+0000) 2019-10-13T09:33:02.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:02.679+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.682279+0000) 2019-10-13T09:33:02.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:02.679+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.683061+0000) 2019-10-13T09:33:02.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:02.679+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.683294+0000) 2019-10-13T09:33:02.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:02.679+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.683674+0000) 2019-10-13T09:33:02.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:02.679+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.684387+0000) 2019-10-13T09:33:02.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:02.682+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.684402+0000) 2019-10-13T09:33:02.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:02.679+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.685505+0000) 2019-10-13T09:33:02.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:02.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:02.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:02.686+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:02.690199+0000) 2019-10-13T09:33:03.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:03.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:03.993657+0000) 2019-10-13T09:33:04.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:04.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:04.282074+0000) 2019-10-13T09:33:04.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:04.379+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:04.379820+0000) 2019-10-13T09:33:04.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:04.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:04.404308+0000) 2019-10-13T09:33:07.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:07.651+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.654806+0000) 2019-10-13T09:33:07.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:07.687+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.690193+0000) 2019-10-13T09:33:07.692 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:07.687+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.692010+0000) 2019-10-13T09:33:07.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:07.687+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.692614+0000) 2019-10-13T09:33:07.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:07.691+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.694381+0000) 2019-10-13T09:33:07.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:07.691+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.695079+0000) 2019-10-13T09:33:07.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:07.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:07.696 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:07.691+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.695871+0000) 2019-10-13T09:33:07.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:07.694+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.696349+0000) 2019-10-13T09:33:07.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:07.698+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.700760+0000) 2019-10-13T09:33:07.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:07.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:07.700870+0000) 2019-10-13T09:33:08.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:08.423+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:08.427285+0000) 2019-10-13T09:33:11.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:11.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:11.677914+0000) 2019-10-13T09:33:12.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:12.695+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.697989+0000) 2019-10-13T09:33:12.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:12.695+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.699248+0000) 2019-10-13T09:33:12.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:12.695+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.699594+0000) 2019-10-13T09:33:12.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:12.694+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.700015+0000) 2019-10-13T09:33:12.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:12.695+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.701105+0000) 2019-10-13T09:33:12.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:12.699+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.701810+0000) 2019-10-13T09:33:12.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:12.699+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.702285+0000) 2019-10-13T09:33:12.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:12.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:12.706 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:12.702+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:12.705436+0000) 2019-10-13T09:33:13.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:13.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:13.993906+0000) 2019-10-13T09:33:14.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:14.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:14.282275+0000) 2019-10-13T09:33:14.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:14.375+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:14.380020+0000) 2019-10-13T09:33:14.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:14.399+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:14.404520+0000) 2019-10-13T09:33:17.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:17.651+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.655037+0000) 2019-10-13T09:33:17.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:17.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.701071+0000) 2019-10-13T09:33:17.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:17.702+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.705849+0000) 2019-10-13T09:33:17.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:17.703+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.707191+0000) 2019-10-13T09:33:17.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:17.703+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.707388+0000) 2019-10-13T09:33:17.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:17.703+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.707833+0000) 2019-10-13T09:33:17.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:17.703+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.708173+0000) 2019-10-13T09:33:17.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:17.703+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.709212+0000) 2019-10-13T09:33:17.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:17.707+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.709977+0000) 2019-10-13T09:33:17.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:17.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:17.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:17.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:17.717971+0000) 2019-10-13T09:33:18.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:18.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:18.427484+0000) 2019-10-13T09:33:21.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:21.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:21.678112+0000) 2019-10-13T09:33:22.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:22.711+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.713866+0000) 2019-10-13T09:33:22.716 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:22.711+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.715684+0000) 2019-10-13T09:33:22.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:22.711+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.716312+0000) 2019-10-13T09:33:22.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:22.711+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.716669+0000) 2019-10-13T09:33:22.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:22.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:22.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:22.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.719305+0000) 2019-10-13T09:33:22.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:22.715+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.720006+0000) 2019-10-13T09:33:22.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:22.715+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.720861+0000) 2019-10-13T09:33:22.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:22.722+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:22.726879+0000) 2019-10-13T09:33:23.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:23.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:23.994113+0000) 2019-10-13T09:33:24.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:24.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:24.282491+0000) 2019-10-13T09:33:24.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:24.375+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:24.380212+0000) 2019-10-13T09:33:24.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:24.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:24.404727+0000) 2019-10-13T09:33:25.550 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:33:25.550 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:25.554 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:33:25.554 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:27.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:27.655+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.655243+0000) 2019-10-13T09:33:27.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:27.702+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.707869+0000) 2019-10-13T09:33:27.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:27.719+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.722778+0000) 2019-10-13T09:33:27.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:27.722+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.724159+0000) 2019-10-13T09:33:27.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:27.723+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.725720+0000) 2019-10-13T09:33:27.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:27.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:27.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:27.723+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.727890+0000) 2019-10-13T09:33:27.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:27.723+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.727941+0000) 2019-10-13T09:33:27.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:27.723+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.728855+0000) 2019-10-13T09:33:27.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:27.726+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.730498+0000) 2019-10-13T09:33:27.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:27.727+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:27.730432+0000) 2019-10-13T09:33:28.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:28.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:28.427700+0000) 2019-10-13T09:33:31.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:31.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:31.678306+0000) 2019-10-13T09:33:32.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:32.730+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.732240+0000) 2019-10-13T09:33:32.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:32.727+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.732859+0000) 2019-10-13T09:33:32.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:32.727+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.733341+0000) 2019-10-13T09:33:32.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:32.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:32.748 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:32.731+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.735873+0000) 2019-10-13T09:33:32.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:32.731+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.736719+0000) 2019-10-13T09:33:32.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:32.735+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.737852+0000) 2019-10-13T09:33:32.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:32.738+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.740565+0000) 2019-10-13T09:33:32.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:32.735+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:32.738811+0000) 2019-10-13T09:33:33.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:33.991+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:33.994340+0000) 2019-10-13T09:33:34.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:34.279+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:34.282699+0000) 2019-10-13T09:33:34.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:34.375+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:34.380410+0000) 2019-10-13T09:33:34.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:34.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:34.404932+0000) 2019-10-13T09:33:37.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:37.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.655471+0000) 2019-10-13T09:33:37.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:37.714+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.714404+0000) 2019-10-13T09:33:37.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:37.734+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.737949+0000) 2019-10-13T09:33:37.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:37.734+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.738988+0000) 2019-10-13T09:33:37.740 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:37.734+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.739835+0000) 2019-10-13T09:33:37.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:37.734+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.739894+0000) 2019-10-13T09:33:37.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:37.734+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.740824+0000) 2019-10-13T09:33:37.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:37.734+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.741248+0000) 2019-10-13T09:33:37.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:37.734+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.741695+0000) 2019-10-13T09:33:37.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:37.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:37.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:37.738+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:37.743196+0000) 2019-10-13T09:33:38.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:38.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:38.427913+0000) 2019-10-13T09:33:41.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:41.674+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:41.678499+0000) 2019-10-13T09:33:42.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:42.742+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.745817+0000) 2019-10-13T09:33:42.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:42.742+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.747698+0000) 2019-10-13T09:33:42.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:42.742+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.748168+0000) 2019-10-13T09:33:42.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:42.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:42.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:42.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.750752+0000) 2019-10-13T09:33:42.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:42.746+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.751734+0000) 2019-10-13T09:33:42.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:42.746+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.751899+0000) 2019-10-13T09:33:42.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:42.746+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.753002+0000) 2019-10-13T09:33:42.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:42.750+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:42.754502+0000) 2019-10-13T09:33:43.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:43.990+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:43.994565+0000) 2019-10-13T09:33:44.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:44.278+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:44.282902+0000) 2019-10-13T09:33:44.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:44.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:44.380644+0000) 2019-10-13T09:33:44.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:44.402+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:44.405135+0000) 2019-10-13T09:33:47.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:47.654+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.655703+0000) 2019-10-13T09:33:47.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:47.718+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.720776+0000) 2019-10-13T09:33:47.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:47.750+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.754408+0000) 2019-10-13T09:33:47.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:47.750+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.754622+0000) 2019-10-13T09:33:47.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:47.750+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.754967+0000) 2019-10-13T09:33:47.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:47.750+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.755803+0000) 2019-10-13T09:33:47.757 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:47.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.756867+0000) 2019-10-13T09:33:47.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:47.750+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.757348+0000) 2019-10-13T09:33:47.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:47.750+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.757593+0000) 2019-10-13T09:33:47.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:47.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:47.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:47.758+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:47.762487+0000) 2019-10-13T09:33:48.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:48.422+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:48.428090+0000) 2019-10-13T09:33:51.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:51.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:51.678725+0000) 2019-10-13T09:33:52.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:52.754+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.761678+0000) 2019-10-13T09:33:52.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:52.758+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.762979+0000) 2019-10-13T09:33:52.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:52.761+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.764547+0000) 2019-10-13T09:33:52.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:52.758+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.765012+0000) 2019-10-13T09:33:52.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:52.762+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.765827+0000) 2019-10-13T09:33:52.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:52.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:52.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:52.762+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.767300+0000) 2019-10-13T09:33:52.768 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:52.762+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.767589+0000) 2019-10-13T09:33:52.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:52.769+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:52.772904+0000) 2019-10-13T09:33:53.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:53.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:53.994781+0000) 2019-10-13T09:33:54.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:54.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:54.283137+0000) 2019-10-13T09:33:54.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:54.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:54.380847+0000) 2019-10-13T09:33:54.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:54.398+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:54.405337+0000) 2019-10-13T09:33:55.614 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:33:55.615 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:55.618 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:33:55.618 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:33:57.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:57.654+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.655909+0000) 2019-10-13T09:33:57.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:57.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.727251+0000) 2019-10-13T09:33:57.771 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:33:57.766+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.770668+0000) 2019-10-13T09:33:57.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:57.766+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.771200+0000) 2019-10-13T09:33:57.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:33:57.766+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.771686+0000) 2019-10-13T09:33:57.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:33:57.765+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.771674+0000) 2019-10-13T09:33:57.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:33:57.766+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.772517+0000) 2019-10-13T09:33:57.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:33:57.766+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.771857+0000) 2019-10-13T09:33:57.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:33:57.770+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.773888+0000) 2019-10-13T09:33:57.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:33:57.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:33:57.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:33:57.773+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:57.778842+0000) 2019-10-13T09:33:58.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:33:58.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:33:58.428263+0000) 2019-10-13T09:34:01.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:01.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:01.678943+0000) 2019-10-13T09:34:02.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:02.774+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.778460+0000) 2019-10-13T09:34:02.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:02.773+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.778472+0000) 2019-10-13T09:34:02.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:02.774+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.779479+0000) 2019-10-13T09:34:02.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:02.774+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.780462+0000) 2019-10-13T09:34:02.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:02.774+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.780134+0000) 2019-10-13T09:34:02.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:02.774+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.781285+0000) 2019-10-13T09:34:02.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:02.774+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.781693+0000) 2019-10-13T09:34:02.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:02.778+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:02.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:02.781+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:02.786121+0000) 2019-10-13T09:34:04.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:03.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:03.994987+0000) 2019-10-13T09:34:04.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:04.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:04.283313+0000) 2019-10-13T09:34:04.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:04.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:04.381060+0000) 2019-10-13T09:34:04.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:04.402+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:04.405536+0000) 2019-10-13T09:34:07.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:07.650+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.656103+0000) 2019-10-13T09:34:07.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:07.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.733697+0000) 2019-10-13T09:34:07.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:07.781+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.787608+0000) 2019-10-13T09:34:07.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:07.782+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.786027+0000) 2019-10-13T09:34:07.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:07.782+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.786790+0000) 2019-10-13T09:34:07.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:07.782+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.787603+0000) 2019-10-13T09:34:07.791 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:07.782+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.787670+0000) 2019-10-13T09:34:07.791 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:07.782+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.788463+0000) 2019-10-13T09:34:07.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:07.782+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.789018+0000) 2019-10-13T09:34:07.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:07.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:07.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:07.789+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:07.792659+0000) 2019-10-13T09:34:08.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:08.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:08.428437+0000) 2019-10-13T09:34:11.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:11.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:11.679097+0000) 2019-10-13T09:34:12.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:12.790+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.794510+0000) 2019-10-13T09:34:12.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:12.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.794900+0000) 2019-10-13T09:34:12.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:12.790+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.795875+0000) 2019-10-13T09:34:12.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:12.790+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.796374+0000) 2019-10-13T09:34:12.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:12.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.796713+0000) 2019-10-13T09:34:12.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:12.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.798179+0000) 2019-10-13T09:34:12.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:12.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:12.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:12.794+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.799062+0000) 2019-10-13T09:34:12.807 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:12.801+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:12.806070+0000) 2019-10-13T09:34:13.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:13.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:13.995226+0000) 2019-10-13T09:34:14.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:14.278+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:14.283551+0000) 2019-10-13T09:34:14.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:14.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:14.381258+0000) 2019-10-13T09:34:14.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:14.402+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:14.405786+0000) 2019-10-13T09:34:17.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:17.654+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.656329+0000) 2019-10-13T09:34:17.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:17.741+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.744656+0000) 2019-10-13T09:34:17.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:17.798+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.803121+0000) 2019-10-13T09:34:17.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:17.798+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.803972+0000) 2019-10-13T09:34:17.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:17.798+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.803993+0000) 2019-10-13T09:34:17.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:17.798+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.804746+0000) 2019-10-13T09:34:17.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:17.801+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.804830+0000) 2019-10-13T09:34:17.807 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:17.802+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.806405+0000) 2019-10-13T09:34:17.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:17.802+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.807101+0000) 2019-10-13T09:34:17.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:17.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:17.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:17.805+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:17.810372+0000) 2019-10-13T09:34:18.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:18.422+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:18.428630+0000) 2019-10-13T09:34:21.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:21.673+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:21.679316+0000) 2019-10-13T09:34:22.810 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:22.806+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.809925+0000) 2019-10-13T09:34:22.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:22.805+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.810385+0000) 2019-10-13T09:34:22.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:22.806+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.811930+0000) 2019-10-13T09:34:22.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:22.806+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.812289+0000) 2019-10-13T09:34:22.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:22.806+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.813247+0000) 2019-10-13T09:34:22.814 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:22.806+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.813672+0000) 2019-10-13T09:34:22.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:22.810+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.814041+0000) 2019-10-13T09:34:22.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:22.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:22.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:22.813+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:22.816634+0000) 2019-10-13T09:34:23.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:23.990+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:23.995406+0000) 2019-10-13T09:34:24.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:24.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:24.283754+0000) 2019-10-13T09:34:24.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:24.374+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:24.381465+0000) 2019-10-13T09:34:24.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:24.402+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:24.406009+0000) 2019-10-13T09:34:25.814 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:34:25.814 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:25.817 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:34:25.818 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:27.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:27.654+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.656547+0000) 2019-10-13T09:34:27.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:27.749+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.751175+0000) 2019-10-13T09:34:27.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:27.813+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.818191+0000) 2019-10-13T09:34:27.820 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:27.814+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.819050+0000) 2019-10-13T09:34:27.820 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:27.814+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.819273+0000) 2019-10-13T09:34:27.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:27.814+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.819448+0000) 2019-10-13T09:34:27.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:27.814+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.819887+0000) 2019-10-13T09:34:27.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:27.814+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.820881+0000) 2019-10-13T09:34:27.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:27.814+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.821265+0000) 2019-10-13T09:34:27.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:27.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:27.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:27.825+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:27.828310+0000) 2019-10-13T09:34:28.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:28.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:28.428808+0000) 2019-10-13T09:34:31.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:31.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:31.679512+0000) 2019-10-13T09:34:32.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:32.822+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.826332+0000) 2019-10-13T09:34:32.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:32.821+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.826452+0000) 2019-10-13T09:34:32.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:32.822+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.828239+0000) 2019-10-13T09:34:32.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:32.822+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.828713+0000) 2019-10-13T09:34:32.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:32.822+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.829423+0000) 2019-10-13T09:34:32.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:32.826+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.829802+0000) 2019-10-13T09:34:32.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:32.826+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.830303+0000) 2019-10-13T09:34:32.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:32.826+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:32.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:32.829+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:32.834623+0000) 2019-10-13T09:34:33.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:33.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:33.995611+0000) 2019-10-13T09:34:34.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:34.277+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:34.283953+0000) 2019-10-13T09:34:34.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:34.377+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:34.381689+0000) 2019-10-13T09:34:34.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:34.405+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:34.406213+0000) 2019-10-13T09:34:37.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:37.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.656757+0000) 2019-10-13T09:34:37.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:37.753+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.757744+0000) 2019-10-13T09:34:37.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:37.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.833699+0000) 2019-10-13T09:34:37.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:37.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.834521+0000) 2019-10-13T09:34:37.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:37.829+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.834907+0000) 2019-10-13T09:34:37.836 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:37.829+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.835080+0000) 2019-10-13T09:34:37.836 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:37.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.835501+0000) 2019-10-13T09:34:37.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:37.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.835819+0000) 2019-10-13T09:34:37.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:37.829+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.837253+0000) 2019-10-13T09:34:37.840 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:37.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:37.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:37.837+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:37.842060+0000) 2019-10-13T09:34:38.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:38.425+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:38.429024+0000) 2019-10-13T09:34:41.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:41.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:41.679739+0000) 2019-10-13T09:34:42.844 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:42.837+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.843219+0000) 2019-10-13T09:34:42.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:42.837+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.843397+0000) 2019-10-13T09:34:42.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:42.837+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.843749+0000) 2019-10-13T09:34:42.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:42.841+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.845900+0000) 2019-10-13T09:34:42.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:42.841+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.845803+0000) 2019-10-13T09:34:42.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:42.841+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.848419+0000) 2019-10-13T09:34:42.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:42.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:42.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:42.841+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.849293+0000) 2019-10-13T09:34:42.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:42.849+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:42.853351+0000) 2019-10-13T09:34:43.996 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:43.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:43.995823+0000) 2019-10-13T09:34:44.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:44.281+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:44.284155+0000) 2019-10-13T09:34:44.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:44.377+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:44.381919+0000) 2019-10-13T09:34:44.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:44.401+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:44.406429+0000) 2019-10-13T09:34:47.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:47.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.656956+0000) 2019-10-13T09:34:47.764 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:47.761+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.763384+0000) 2019-10-13T09:34:47.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:47.845+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.851599+0000) 2019-10-13T09:34:47.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:47.845+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.851705+0000) 2019-10-13T09:34:47.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:47.845+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.852704+0000) 2019-10-13T09:34:47.854 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:47.845+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.853226+0000) 2019-10-13T09:34:47.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:47.849+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.855110+0000) 2019-10-13T09:34:47.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:47.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:47.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:47.849+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.856994+0000) 2019-10-13T09:34:47.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:47.849+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.857345+0000) 2019-10-13T09:34:47.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:47.857+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:47.862925+0000) 2019-10-13T09:34:48.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:48.425+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:48.429233+0000) 2019-10-13T09:34:51.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:51.677+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:51.679966+0000) 2019-10-13T09:34:52.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:52.857+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.860897+0000) 2019-10-13T09:34:52.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:52.857+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.861310+0000) 2019-10-13T09:34:52.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:52.857+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.862016+0000) 2019-10-13T09:34:52.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:52.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.862937+0000) 2019-10-13T09:34:52.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:52.861+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.863014+0000) 2019-10-13T09:34:52.864 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:52.861+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.863356+0000) 2019-10-13T09:34:52.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:52.861+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.863911+0000) 2019-10-13T09:34:52.865 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:52.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:52.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:52.865+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:52.870751+0000) 2019-10-13T09:34:53.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:53.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:53.996062+0000) 2019-10-13T09:34:54.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:54.281+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:54.284349+0000) 2019-10-13T09:34:54.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:54.381+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:54.382171+0000) 2019-10-13T09:34:54.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:54.405+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:54.406636+0000) 2019-10-13T09:34:55.861 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:34:55.861 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:55.865 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:34:55.865 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:34:57.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:57.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.657166+0000) 2019-10-13T09:34:57.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:57.764+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.769898+0000) 2019-10-13T09:34:57.869 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:34:57.865+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.868067+0000) 2019-10-13T09:34:57.869 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:57.865+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.868765+0000) 2019-10-13T09:34:57.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:34:57.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.868950+0000) 2019-10-13T09:34:57.870 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:34:57.865+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.869391+0000) 2019-10-13T09:34:57.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:34:57.869+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.869805+0000) 2019-10-13T09:34:57.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:34:57.864+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.870451+0000) 2019-10-13T09:34:57.872 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:34:57.869+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.871297+0000) 2019-10-13T09:34:57.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:34:57.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:34:57.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:34:57.868+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:57.873734+0000) 2019-10-13T09:34:58.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:34:58.425+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:34:58.429407+0000) 2019-10-13T09:35:01.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:01.676+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:01.680166+0000) 2019-10-13T09:35:02.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:02.873+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.875931+0000) 2019-10-13T09:35:02.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:02.872+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.876305+0000) 2019-10-13T09:35:02.878 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:02.873+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.876517+0000) 2019-10-13T09:35:02.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:02.873+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.876966+0000) 2019-10-13T09:35:02.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:02.873+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.877147+0000) 2019-10-13T09:35:02.879 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:02.877+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.879187+0000) 2019-10-13T09:35:02.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:02.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:02.880 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:02.877+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.879795+0000) 2019-10-13T09:35:02.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:02.880+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:02.887462+0000) 2019-10-13T09:35:03.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:03.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:03.996291+0000) 2019-10-13T09:35:04.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:04.281+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:04.284556+0000) 2019-10-13T09:35:04.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:04.381+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:04.382403+0000) 2019-10-13T09:35:04.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:04.405+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:04.406835+0000) 2019-10-13T09:35:07.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:07.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.657362+0000) 2019-10-13T09:35:07.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:07.772+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.776389+0000) 2019-10-13T09:35:07.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:07.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.883283+0000) 2019-10-13T09:35:07.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:07.881+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.883801+0000) 2019-10-13T09:35:07.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:07.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.884311+0000) 2019-10-13T09:35:07.885 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:07.881+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.884568+0000) 2019-10-13T09:35:07.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:07.880+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.885233+0000) 2019-10-13T09:35:07.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:07.885+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.886194+0000) 2019-10-13T09:35:07.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:07.885+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.886496+0000) 2019-10-13T09:35:07.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:07.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:07.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:07.892+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:07.896788+0000) 2019-10-13T09:35:08.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:08.425+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:08.429587+0000) 2019-10-13T09:35:11.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:11.676+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:11.680361+0000) 2019-10-13T09:35:12.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:12.889+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.891078+0000) 2019-10-13T09:35:12.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:12.889+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.893104+0000) 2019-10-13T09:35:12.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:12.889+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.893601+0000) 2019-10-13T09:35:12.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:12.893+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.894241+0000) 2019-10-13T09:35:12.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:12.888+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.894051+0000) 2019-10-13T09:35:12.896 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:12.893+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.895312+0000) 2019-10-13T09:35:12.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:12.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:12.897 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:12.893+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.895976+0000) 2019-10-13T09:35:12.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:12.896+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:12.900992+0000) 2019-10-13T09:35:13.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:13.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:13.996502+0000) 2019-10-13T09:35:14.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:14.281+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:14.284767+0000) 2019-10-13T09:35:14.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:14.381+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:14.382596+0000) 2019-10-13T09:35:14.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:14.405+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:14.407065+0000) 2019-10-13T09:35:17.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:17.653+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.657560+0000) 2019-10-13T09:35:17.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:17.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.782928+0000) 2019-10-13T09:35:17.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:17.897+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.899431+0000) 2019-10-13T09:35:17.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:17.897+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.900460+0000) 2019-10-13T09:35:17.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:17.897+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.900915+0000) 2019-10-13T09:35:17.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:17.896+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.900838+0000) 2019-10-13T09:35:17.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:17.897+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.901705+0000) 2019-10-13T09:35:17.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:17.901+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.902346+0000) 2019-10-13T09:35:17.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:17.901+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.902135+0000) 2019-10-13T09:35:17.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:17.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:17.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:17.904+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:17.908454+0000) 2019-10-13T09:35:18.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:18.429+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:18.429775+0000) 2019-10-13T09:35:21.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:21.676+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:21.680558+0000) 2019-10-13T09:35:22.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:22.900+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.906853+0000) 2019-10-13T09:35:22.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:22.905+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.907805+0000) 2019-10-13T09:35:22.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:22.905+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.908699+0000) 2019-10-13T09:35:22.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:22.905+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.909400+0000) 2019-10-13T09:35:22.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:22.909+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.910180+0000) 2019-10-13T09:35:22.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:22.909+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.910906+0000) 2019-10-13T09:35:22.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:22.909+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.911123+0000) 2019-10-13T09:35:22.912 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:22.909+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:22.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:22.908+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:22.914168+0000) 2019-10-13T09:35:23.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:23.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:23.996771+0000) 2019-10-13T09:35:24.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:24.281+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:24.284975+0000) 2019-10-13T09:35:24.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:24.381+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:24.382796+0000) 2019-10-13T09:35:24.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:24.405+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:24.407286+0000) 2019-10-13T09:35:25.999 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:35:25.999 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:26.002 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:35:26.002 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:27.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:27.657+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.657792+0000) 2019-10-13T09:35:27.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:27.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.789432+0000) 2019-10-13T09:35:27.917 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:27.913+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.915789+0000) 2019-10-13T09:35:27.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:27.908+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.915928+0000) 2019-10-13T09:35:27.918 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:27.913+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.916817+0000) 2019-10-13T09:35:27.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:27.913+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.917636+0000) 2019-10-13T09:35:27.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:27.917+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.918253+0000) 2019-10-13T09:35:27.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:27.917+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.919124+0000) 2019-10-13T09:35:27.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:27.917+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.918766+0000) 2019-10-13T09:35:27.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:27.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:27.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:27.920+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:27.924469+0000) 2019-10-13T09:35:28.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:28.429+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:28.430028+0000) 2019-10-13T09:35:31.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:31.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:31.680791+0000) 2019-10-13T09:35:32.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:32.920+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.923632+0000) 2019-10-13T09:35:32.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:32.916+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.923681+0000) 2019-10-13T09:35:32.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:32.920+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.924518+0000) 2019-10-13T09:35:32.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:32.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.925121+0000) 2019-10-13T09:35:32.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:32.920+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.925665+0000) 2019-10-13T09:35:32.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:32.924+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.925952+0000) 2019-10-13T09:35:32.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:32.924+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.926112+0000) 2019-10-13T09:35:32.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:32.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:32.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:32.928+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:32.933036+0000) 2019-10-13T09:35:33.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:33.992+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:33.996976+0000) 2019-10-13T09:35:34.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:34.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:34.285197+0000) 2019-10-13T09:35:34.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:34.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:34.382988+0000) 2019-10-13T09:35:34.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:34.404+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:34.407483+0000) 2019-10-13T09:35:37.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:37.656+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.657994+0000) 2019-10-13T09:35:37.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:37.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.796023+0000) 2019-10-13T09:35:37.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:37.928+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.931948+0000) 2019-10-13T09:35:37.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:37.928+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.932612+0000) 2019-10-13T09:35:37.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:37.928+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.933471+0000) 2019-10-13T09:35:37.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:37.928+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.933259+0000) 2019-10-13T09:35:37.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:37.932+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.934054+0000) 2019-10-13T09:35:37.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:37.932+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.934424+0000) 2019-10-13T09:35:37.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:37.932+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.935253+0000) 2019-10-13T09:35:37.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:37.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:37.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:37.936+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:37.941514+0000) 2019-10-13T09:35:38.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:38.428+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:38.430240+0000) 2019-10-13T09:35:41.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:41.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:41.680990+0000) 2019-10-13T09:35:42.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:42.936+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.939846+0000) 2019-10-13T09:35:42.941 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:42.936+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.940598+0000) 2019-10-13T09:35:42.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:42.936+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.941039+0000) 2019-10-13T09:35:42.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:42.936+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.941349+0000) 2019-10-13T09:35:42.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:42.940+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.941971+0000) 2019-10-13T09:35:42.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:42.940+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.942754+0000) 2019-10-13T09:35:42.944 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:42.940+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.943508+0000) 2019-10-13T09:35:42.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:42.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:42.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:42.944+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:42.948347+0000) 2019-10-13T09:35:43.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:43.996+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:43.997179+0000) 2019-10-13T09:35:44.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:44.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:44.285402+0000) 2019-10-13T09:35:44.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:44.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:44.383186+0000) 2019-10-13T09:35:44.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:44.404+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:44.407698+0000) 2019-10-13T09:35:47.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:47.656+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.658190+0000) 2019-10-13T09:35:47.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:47.800+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.802473+0000) 2019-10-13T09:35:47.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:47.944+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.946782+0000) 2019-10-13T09:35:47.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:47.944+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.947772+0000) 2019-10-13T09:35:47.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:47.944+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.948213+0000) 2019-10-13T09:35:47.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:47.944+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.949132+0000) 2019-10-13T09:35:47.950 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:47.948+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.949920+0000) 2019-10-13T09:35:47.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:47.948+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.950531+0000) 2019-10-13T09:35:47.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:47.948+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.951405+0000) 2019-10-13T09:35:47.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:47.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:47.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:47.952+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:47.956366+0000) 2019-10-13T09:35:48.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:48.428+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:48.430418+0000) 2019-10-13T09:35:51.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:51.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:51.681193+0000) 2019-10-13T09:35:52.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:52.952+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.955422+0000) 2019-10-13T09:35:52.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:52.952+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.955406+0000) 2019-10-13T09:35:52.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:52.952+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.956267+0000) 2019-10-13T09:35:52.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:52.952+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.956688+0000) 2019-10-13T09:35:52.958 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:52.952+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.957677+0000) 2019-10-13T09:35:52.959 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:52.956+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.958332+0000) 2019-10-13T09:35:52.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:52.956+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.958615+0000) 2019-10-13T09:35:52.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:52.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:52.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:52.960+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:52.965173+0000) 2019-10-13T09:35:53.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:53.992+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:53.997397+0000) 2019-10-13T09:35:54.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:54.280+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:54.285621+0000) 2019-10-13T09:35:54.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:54.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:54.383420+0000) 2019-10-13T09:35:54.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:54.404+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:54.407906+0000) 2019-10-13T09:35:56.062 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:35:56.062 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:56.066 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:35:56.066 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:35:57.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:57.656+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.658410+0000) 2019-10-13T09:35:57.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:57.808+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.808927+0000) 2019-10-13T09:35:57.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:35:57.960+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.963570+0000) 2019-10-13T09:35:57.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:35:57.960+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.963960+0000) 2019-10-13T09:35:57.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:35:57.964+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.966361+0000) 2019-10-13T09:35:57.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:57.964+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.966800+0000) 2019-10-13T09:35:57.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:35:57.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:35:57.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:35:57.964+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.969476+0000) 2019-10-13T09:35:57.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:35:57.968+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.969668+0000) 2019-10-13T09:35:57.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:35:57.968+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.971076+0000) 2019-10-13T09:35:57.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:35:57.972+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:57.973399+0000) 2019-10-13T09:35:58.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:35:58.428+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:35:58.430589+0000) 2019-10-13T09:36:01.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:01.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:01.681394+0000) 2019-10-13T09:36:02.973 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:02.968+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.972300+0000) 2019-10-13T09:36:02.973 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:02.968+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.972767+0000) 2019-10-13T09:36:02.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:02.971+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.972734+0000) 2019-10-13T09:36:02.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:02.968+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.973539+0000) 2019-10-13T09:36:02.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:02.972+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.973905+0000) 2019-10-13T09:36:02.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:02.972+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.975226+0000) 2019-10-13T09:36:02.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:02.972+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.976132+0000) 2019-10-13T09:36:02.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:02.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:02.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:02.979+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:02.981130+0000) 2019-10-13T09:36:03.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:03.992+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:03.997593+0000) 2019-10-13T09:36:04.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:04.284+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:04.285850+0000) 2019-10-13T09:36:04.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:04.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:04.383627+0000) 2019-10-13T09:36:04.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:04.404+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:04.408106+0000) 2019-10-13T09:36:07.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:07.656+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.658618+0000) 2019-10-13T09:36:07.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:07.811+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.815354+0000) 2019-10-13T09:36:07.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:07.980+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.982285+0000) 2019-10-13T09:36:07.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:07.980+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.981967+0000) 2019-10-13T09:36:07.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:07.979+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.982165+0000) 2019-10-13T09:36:07.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:07.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.983177+0000) 2019-10-13T09:36:07.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:07.980+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.984395+0000) 2019-10-13T09:36:07.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:07.980+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.984866+0000) 2019-10-13T09:36:07.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:07.980+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.985192+0000) 2019-10-13T09:36:07.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:07.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:07.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:07.987+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:07.989328+0000) 2019-10-13T09:36:08.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:08.428+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:08.430756+0000) 2019-10-13T09:36:11.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:11.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:11.681594+0000) 2019-10-13T09:36:12.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:12.984+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.989212+0000) 2019-10-13T09:36:12.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:12.988+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.990618+0000) 2019-10-13T09:36:12.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:12.988+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.992379+0000) 2019-10-13T09:36:12.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:12.988+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.992730+0000) 2019-10-13T09:36:12.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:12.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98259 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:12.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:12.992+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.993807+0000) 2019-10-13T09:36:12.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:12.992+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.994180+0000) 2019-10-13T09:36:12.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:12.991+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:12.994065+0000) 2019-10-13T09:36:13.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:12.999+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.000554+0000) 2019-10-13T09:36:13.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:13.996+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:13.997795+0000) 2019-10-13T09:36:14.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:14.284+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:14.286045+0000) 2019-10-13T09:36:14.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:14.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:14.383824+0000) 2019-10-13T09:36:14.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:14.408+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:14.408301+0000) 2019-10-13T09:36:17.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:17.656+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.658823+0000) 2019-10-13T09:36:17.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:17.819+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.821716+0000) 2019-10-13T09:36:17.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:17.992+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.996752+0000) 2019-10-13T09:36:17.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:17.992+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.997315+0000) 2019-10-13T09:36:17.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:17.996+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.998039+0000) 2019-10-13T09:36:17.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:17.996+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.998381+0000) 2019-10-13T09:36:18.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:17.996+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.999619+0000) 2019-10-13T09:36:18.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:17.995+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.999530+0000) 2019-10-13T09:36:18.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:17.996+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:17.999662+0000) 2019-10-13T09:36:18.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:17.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:18.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:17.999+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.003772+0000) 2019-10-13T09:36:18.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:18.428+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:18.430931+0000) 2019-10-13T09:36:21.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:21.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:21.681807+0000) 2019-10-13T09:36:23.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:23.004+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.006378+0000) 2019-10-13T09:36:23.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:23.004+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.006917+0000) 2019-10-13T09:36:23.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:23.004+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.009581+0000) 2019-10-13T09:36:23.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:23.008+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.009859+0000) 2019-10-13T09:36:23.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:23.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:23.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:23.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.010396+0000) 2019-10-13T09:36:23.012 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:23.008+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.010421+0000) 2019-10-13T09:36:23.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:23.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.011459+0000) 2019-10-13T09:36:23.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:23.015+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.016708+0000) 2019-10-13T09:36:23.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:23.996+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:23.998035+0000) 2019-10-13T09:36:24.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:24.284+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:24.286270+0000) 2019-10-13T09:36:24.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:24.380+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:24.384024+0000) 2019-10-13T09:36:24.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:24.408+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:24.408529+0000) 2019-10-13T09:36:26.193 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:36:26.193 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:26.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:36:26.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:27.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:27.655+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:27.659025+0000) 2019-10-13T09:36:27.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:27.827+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:27.828146+0000) 2019-10-13T09:36:28.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:28.007+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.013413+0000) 2019-10-13T09:36:28.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:28.007+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.013066+0000) 2019-10-13T09:36:28.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:28.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.014328+0000) 2019-10-13T09:36:28.015 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:28.011+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.014668+0000) 2019-10-13T09:36:28.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:28.011+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.015028+0000) 2019-10-13T09:36:28.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:28.011+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.015098+0000) 2019-10-13T09:36:28.017 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:28.011+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.016040+0000) 2019-10-13T09:36:28.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:28.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:28.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:28.019+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.022660+0000) 2019-10-13T09:36:28.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:28.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:28.431106+0000) 2019-10-13T09:36:31.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:31.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:31.682002+0000) 2019-10-13T09:36:33.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:33.019+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.022989+0000) 2019-10-13T09:36:33.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:33.019+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.023222+0000) 2019-10-13T09:36:33.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:33.019+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.023552+0000) 2019-10-13T09:36:33.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:33.019+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.023679+0000) 2019-10-13T09:36:33.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:33.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:33.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:33.023+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.026217+0000) 2019-10-13T09:36:33.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:33.023+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.026909+0000) 2019-10-13T09:36:33.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:33.023+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.026559+0000) 2019-10-13T09:36:33.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:33.027+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.030790+0000) 2019-10-13T09:36:33.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:33.995+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:33.998229+0000) 2019-10-13T09:36:34.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:34.283+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:34.286487+0000) 2019-10-13T09:36:34.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:34.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:34.384220+0000) 2019-10-13T09:36:34.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:34.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:34.408725+0000) 2019-10-13T09:36:37.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:37.655+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:37.659239+0000) 2019-10-13T09:36:37.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:37.839+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:37.839694+0000) 2019-10-13T09:36:38.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:38.023+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.028973+0000) 2019-10-13T09:36:38.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:38.027+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.029930+0000) 2019-10-13T09:36:38.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:38.027+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.031425+0000) 2019-10-13T09:36:38.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:38.027+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.031766+0000) 2019-10-13T09:36:38.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:38.027+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.032137+0000) 2019-10-13T09:36:38.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:38.027+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.031998+0000) 2019-10-13T09:36:38.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:38.027+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.033556+0000) 2019-10-13T09:36:38.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:38.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:38.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:38.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.039197+0000) 2019-10-13T09:36:38.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:38.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:38.431312+0000) 2019-10-13T09:36:41.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:41.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:41.682209+0000) 2019-10-13T09:36:43.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:43.031+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.037457+0000) 2019-10-13T09:36:43.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:43.035+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.037808+0000) 2019-10-13T09:36:43.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:43.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.039560+0000) 2019-10-13T09:36:43.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:43.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.040026+0000) 2019-10-13T09:36:43.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:43.035+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.040856+0000) 2019-10-13T09:36:43.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:43.039+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.040701+0000) 2019-10-13T09:36:43.042 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:43.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:43.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:43.039+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.042486+0000) 2019-10-13T09:36:43.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:43.043+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.048050+0000) 2019-10-13T09:36:43.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:43.995+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:43.998429+0000) 2019-10-13T09:36:44.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:44.283+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:44.286687+0000) 2019-10-13T09:36:44.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:44.379+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:44.384439+0000) 2019-10-13T09:36:44.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:44.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:44.408919+0000) 2019-10-13T09:36:47.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:47.655+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:47.659436+0000) 2019-10-13T09:36:47.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:47.843+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:47.846083+0000) 2019-10-13T09:36:48.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:48.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.045177+0000) 2019-10-13T09:36:48.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:48.039+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.045449+0000) 2019-10-13T09:36:48.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:48.043+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.047185+0000) 2019-10-13T09:36:48.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:48.043+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.045895+0000) 2019-10-13T09:36:48.048 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:48.043+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.048210+0000) 2019-10-13T09:36:48.049 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:48.047+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.048433+0000) 2019-10-13T09:36:48.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:48.043+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.049169+0000) 2019-10-13T09:36:48.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:48.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:48.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:48.051+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.053758+0000) 2019-10-13T09:36:48.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:48.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:48.431492+0000) 2019-10-13T09:36:51.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:51.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:51.682409+0000) 2019-10-13T09:36:53.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:53.047+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.053499+0000) 2019-10-13T09:36:53.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:53.047+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.053703+0000) 2019-10-13T09:36:53.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:53.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.055958+0000) 2019-10-13T09:36:53.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:53.051+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.056668+0000) 2019-10-13T09:36:53.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:53.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:53.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:53.055+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.058348+0000) 2019-10-13T09:36:53.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:53.055+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.058810+0000) 2019-10-13T09:36:53.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:53.055+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.059434+0000) 2019-10-13T09:36:53.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:53.063+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.064587+0000) 2019-10-13T09:36:53.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:53.995+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:53.998632+0000) 2019-10-13T09:36:54.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:54.283+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:54.286915+0000) 2019-10-13T09:36:54.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:54.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:54.384639+0000) 2019-10-13T09:36:54.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:54.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:54.409121+0000) 2019-10-13T09:36:56.447 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:36:56.447 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:56.451 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:36:56.451 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:36:57.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:57.655+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:57.659647+0000) 2019-10-13T09:36:57.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:57.851+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:57.852486+0000) 2019-10-13T09:36:58.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:36:58.055+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.061210+0000) 2019-10-13T09:36:58.062 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:58.055+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.061705+0000) 2019-10-13T09:36:58.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:36:58.059+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.062365+0000) 2019-10-13T09:36:58.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:36:58.059+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.062843+0000) 2019-10-13T09:36:58.064 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:36:58.059+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.062776+0000) 2019-10-13T09:36:58.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:36:58.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.064004+0000) 2019-10-13T09:36:58.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:36:58.059+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.064219+0000) 2019-10-13T09:36:58.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:36:58.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:36:58.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:36:58.067+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.069215+0000) 2019-10-13T09:36:58.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:36:58.427+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:36:58.431683+0000) 2019-10-13T09:37:01.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:01.679+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:01.682607+0000) 2019-10-13T09:37:03.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:03.067+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.069882+0000) 2019-10-13T09:37:03.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:03.067+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.070266+0000) 2019-10-13T09:37:03.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:03.067+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.070965+0000) 2019-10-13T09:37:03.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:03.071+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.072875+0000) 2019-10-13T09:37:03.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:03.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.073581+0000) 2019-10-13T09:37:03.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:03.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:03.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:03.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.075344+0000) 2019-10-13T09:37:03.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:03.071+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.074806+0000) 2019-10-13T09:37:03.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:03.075+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.077135+0000) 2019-10-13T09:37:03.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:03.995+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:03.998832+0000) 2019-10-13T09:37:04.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:04.283+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:04.287116+0000) 2019-10-13T09:37:04.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:04.379+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:04.384854+0000) 2019-10-13T09:37:04.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:04.407+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:04.409318+0000) 2019-10-13T09:37:07.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:07.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:07.659868+0000) 2019-10-13T09:37:07.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:07.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:07.858912+0000) 2019-10-13T09:37:08.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:08.075+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.078557+0000) 2019-10-13T09:37:08.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:08.074+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.078463+0000) 2019-10-13T09:37:08.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:08.075+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.079508+0000) 2019-10-13T09:37:08.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:08.075+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.080207+0000) 2019-10-13T09:37:08.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:08.075+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.080805+0000) 2019-10-13T09:37:08.082 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:08.075+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.081282+0000) 2019-10-13T09:37:08.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:08.075+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.081681+0000) 2019-10-13T09:37:08.083 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:08.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:08.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:08.082+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.085950+0000) 2019-10-13T09:37:08.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:08.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:08.431859+0000) 2019-10-13T09:37:11.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:11.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:11.682843+0000) 2019-10-13T09:37:13.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:13.082+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.085803+0000) 2019-10-13T09:37:13.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:13.083+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.086753+0000) 2019-10-13T09:37:13.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:13.083+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.088389+0000) 2019-10-13T09:37:13.090 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:13.083+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.089289+0000) 2019-10-13T09:37:13.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:13.083+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.089651+0000) 2019-10-13T09:37:13.091 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:13.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:13.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:13.087+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.091445+0000) 2019-10-13T09:37:13.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:13.087+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.092822+0000) 2019-10-13T09:37:13.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:13.094+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.097283+0000) 2019-10-13T09:37:14.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:13.995+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:13.999038+0000) 2019-10-13T09:37:14.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:14.283+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:14.287315+0000) 2019-10-13T09:37:14.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:14.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:14.385052+0000) 2019-10-13T09:37:14.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:14.403+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:14.409522+0000) 2019-10-13T09:37:17.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:17.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:17.660069+0000) 2019-10-13T09:37:17.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:17.866+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:17.868382+0000) 2019-10-13T09:37:18.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:18.091+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.093972+0000) 2019-10-13T09:37:18.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:18.091+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.095194+0000) 2019-10-13T09:37:18.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:18.091+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.095418+0000) 2019-10-13T09:37:18.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:18.091+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.096000+0000) 2019-10-13T09:37:18.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:18.091+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.097303+0000) 2019-10-13T09:37:18.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:18.094+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.097042+0000) 2019-10-13T09:37:18.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:18.095+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.098041+0000) 2019-10-13T09:37:18.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:18.095+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:18.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:18.098+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.102270+0000) 2019-10-13T09:37:18.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:18.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:18.432045+0000) 2019-10-13T09:37:21.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:21.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:21.683034+0000) 2019-10-13T09:37:23.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:23.095+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.101651+0000) 2019-10-13T09:37:23.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:23.099+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.102821+0000) 2019-10-13T09:37:23.106 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:23.099+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.103130+0000) 2019-10-13T09:37:23.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:23.099+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.103564+0000) 2019-10-13T09:37:23.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:23.099+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.104550+0000) 2019-10-13T09:37:23.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:23.099+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.104922+0000) 2019-10-13T09:37:23.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:23.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:23.108 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:23.102+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.106427+0000) 2019-10-13T09:37:23.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:23.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.112556+0000) 2019-10-13T09:37:24.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:23.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:23.999248+0000) 2019-10-13T09:37:24.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:24.286+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:24.287517+0000) 2019-10-13T09:37:24.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:24.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:24.385251+0000) 2019-10-13T09:37:24.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:24.406+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:24.409723+0000) 2019-10-13T09:37:26.564 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:37:26.564 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:26.568 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:37:26.568 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:27.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:27.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:27.660305+0000) 2019-10-13T09:37:27.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:27.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:27.874410+0000) 2019-10-13T09:37:28.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:28.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.110045+0000) 2019-10-13T09:37:28.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:28.106+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.112342+0000) 2019-10-13T09:37:28.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:28.106+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.112667+0000) 2019-10-13T09:37:28.114 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:28.106+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.113319+0000) 2019-10-13T09:37:28.115 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:28.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98726 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:28.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:28.110+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.114676+0000) 2019-10-13T09:37:28.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:28.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.114897+0000) 2019-10-13T09:37:28.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:28.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.115682+0000) 2019-10-13T09:37:28.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:28.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.115969+0000) 2019-10-13T09:37:28.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:28.430+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:28.432237+0000) 2019-10-13T09:37:31.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:31.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:31.683223+0000) 2019-10-13T09:37:33.120 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:33.114+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.119349+0000) 2019-10-13T09:37:33.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:33.114+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.119891+0000) 2019-10-13T09:37:33.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:33.114+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.120107+0000) 2019-10-13T09:37:33.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:33.114+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.121121+0000) 2019-10-13T09:37:33.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:33.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:33.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:33.118+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.122194+0000) 2019-10-13T09:37:33.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:33.118+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.122184+0000) 2019-10-13T09:37:33.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:33.118+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.122915+0000) 2019-10-13T09:37:33.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:33.122+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.126756+0000) 2019-10-13T09:37:34.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:33.998+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:33.999434+0000) 2019-10-13T09:37:34.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:34.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:34.287732+0000) 2019-10-13T09:37:34.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:34.382+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:34.385457+0000) 2019-10-13T09:37:34.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:34.406+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:34.409906+0000) 2019-10-13T09:37:37.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:37.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:37.660504+0000) 2019-10-13T09:37:37.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:37.878+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:37.880302+0000) 2019-10-13T09:37:38.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:38.118+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.125011+0000) 2019-10-13T09:37:38.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:38.118+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.125462+0000) 2019-10-13T09:37:38.126 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:38.122+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.125991+0000) 2019-10-13T09:37:38.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:38.122+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.126638+0000) 2019-10-13T09:37:38.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:38.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.126871+0000) 2019-10-13T09:37:38.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:38.122+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.127769+0000) 2019-10-13T09:37:38.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:38.122+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.127923+0000) 2019-10-13T09:37:38.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:38.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:38.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:38.130+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.133167+0000) 2019-10-13T09:37:38.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:38.430+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:38.432479+0000) 2019-10-13T09:37:41.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:41.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:41.683414+0000) 2019-10-13T09:37:43.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:43.126+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.132987+0000) 2019-10-13T09:37:43.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:43.130+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.135443+0000) 2019-10-13T09:37:43.137 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:43.130+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.135605+0000) 2019-10-13T09:37:43.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:43.130+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.136475+0000) 2019-10-13T09:37:43.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:43.130+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.137268+0000) 2019-10-13T09:37:43.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:43.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:43.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:43.134+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.137895+0000) 2019-10-13T09:37:43.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:43.134+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.138297+0000) 2019-10-13T09:37:43.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:43.138+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.143730+0000) 2019-10-13T09:37:44.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:43.998+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:43.999619+0000) 2019-10-13T09:37:44.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:44.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:44.287932+0000) 2019-10-13T09:37:44.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:44.378+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:44.385658+0000) 2019-10-13T09:37:44.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:44.406+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:44.410065+0000) 2019-10-13T09:37:47.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:47.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:47.660705+0000) 2019-10-13T09:37:47.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:47.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:47.886777+0000) 2019-10-13T09:37:48.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:48.138+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.140524+0000) 2019-10-13T09:37:48.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:48.138+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.141803+0000) 2019-10-13T09:37:48.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:48.138+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.142066+0000) 2019-10-13T09:37:48.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:48.138+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.142259+0000) 2019-10-13T09:37:48.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:48.138+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.143438+0000) 2019-10-13T09:37:48.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:48.138+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.145057+0000) 2019-10-13T09:37:48.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:48.138+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.145447+0000) 2019-10-13T09:37:48.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:48.142+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:48.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:48.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.148417+0000) 2019-10-13T09:37:48.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:48.430+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:48.432648+0000) 2019-10-13T09:37:51.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:51.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:51.683606+0000) 2019-10-13T09:37:53.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:53.146+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.148766+0000) 2019-10-13T09:37:53.150 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:53.146+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.149842+0000) 2019-10-13T09:37:53.151 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:53.146+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.150300+0000) 2019-10-13T09:37:53.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:53.146+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.152034+0000) 2019-10-13T09:37:53.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:53.146+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.152031+0000) 2019-10-13T09:37:53.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:53.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:53.155 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:53.150+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.155121+0000) 2019-10-13T09:37:53.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:53.150+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.154563+0000) 2019-10-13T09:37:53.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:53.154+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.158696+0000) 2019-10-13T09:37:54.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:53.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:53.999807+0000) 2019-10-13T09:37:54.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:54.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:54.288159+0000) 2019-10-13T09:37:54.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:54.382+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:54.385858+0000) 2019-10-13T09:37:54.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:54.406+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:54.410267+0000) 2019-10-13T09:37:56.762 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:37:56.762 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:56.766 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:37:56.766 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:37:57.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:57.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:57.660941+0000) 2019-10-13T09:37:57.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:57.890+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:57.893354+0000) 2019-10-13T09:37:58.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:58.154+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.157672+0000) 2019-10-13T09:37:58.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:37:58.154+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.157645+0000) 2019-10-13T09:37:58.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:37:58.154+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.158375+0000) 2019-10-13T09:37:58.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:37:58.154+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.158709+0000) 2019-10-13T09:37:58.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:37:58.154+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.159265+0000) 2019-10-13T09:37:58.160 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:37:58.154+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.159875+0000) 2019-10-13T09:37:58.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:37:58.154+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.160417+0000) 2019-10-13T09:37:58.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:37:58.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:37:58.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:37:58.162+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.165952+0000) 2019-10-13T09:37:58.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:37:58.430+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:37:58.432832+0000) 2019-10-13T09:38:01.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:01.678+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:01.683804+0000) 2019-10-13T09:38:03.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:03.162+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.165970+0000) 2019-10-13T09:38:03.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:03.162+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.166818+0000) 2019-10-13T09:38:03.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:03.162+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.168105+0000) 2019-10-13T09:38:03.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:03.162+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.167747+0000) 2019-10-13T09:38:03.169 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:03.162+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.168853+0000) 2019-10-13T09:38:03.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:03.162+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.169558+0000) 2019-10-13T09:38:03.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:03.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.170236+0000) 2019-10-13T09:38:03.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:03.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:03.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:03.170+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.173457+0000) 2019-10-13T09:38:04.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:03.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:03.999990+0000) 2019-10-13T09:38:04.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:04.286+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:04.288354+0000) 2019-10-13T09:38:04.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:04.382+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:04.386072+0000) 2019-10-13T09:38:04.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:04.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:04.410520+0000) 2019-10-13T09:38:07.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:07.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:07.661140+0000) 2019-10-13T09:38:07.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:07.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:07.899694+0000) 2019-10-13T09:38:08.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:08.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.174428+0000) 2019-10-13T09:38:08.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:08.170+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.175207+0000) 2019-10-13T09:38:08.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:08.170+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.177478+0000) 2019-10-13T09:38:08.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:08.174+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.177855+0000) 2019-10-13T09:38:08.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:08.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:08.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:08.174+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.179031+0000) 2019-10-13T09:38:08.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:08.174+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.179507+0000) 2019-10-13T09:38:08.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:08.174+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.179513+0000) 2019-10-13T09:38:08.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:08.178+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.182418+0000) 2019-10-13T09:38:08.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:08.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:08.433043+0000) 2019-10-13T09:38:11.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:11.682+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:11.684051+0000) 2019-10-13T09:38:13.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:13.178+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.182801+0000) 2019-10-13T09:38:13.184 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:13.178+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.183098+0000) 2019-10-13T09:38:13.185 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:13.178+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.184747+0000) 2019-10-13T09:38:13.185 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:13.181+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.184716+0000) 2019-10-13T09:38:13.186 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:13.178+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.185540+0000) 2019-10-13T09:38:13.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:13.182+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:13.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:13.182+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.186368+0000) 2019-10-13T09:38:13.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:13.182+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.186856+0000) 2019-10-13T09:38:13.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:13.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:13.192383+0000) 2019-10-13T09:38:14.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:13.994+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:14.000179+0000) 2019-10-13T09:38:14.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:14.282+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:14.288553+0000) 2019-10-13T09:38:14.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:14.382+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:14.386276+0000) 2019-10-13T09:38:14.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:14.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:14.410751+0000) 2019-10-13T09:38:17.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:17.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:17.661333+0000) 2019-10-13T09:38:17.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:17.909+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:17.910696+0000) 2019-10-13T09:38:18.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:18.182+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.189505+0000) 2019-10-13T09:38:18.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:18.186+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.190226+0000) 2019-10-13T09:38:18.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:18.182+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.189606+0000) 2019-10-13T09:38:18.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:18.186+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.190970+0000) 2019-10-13T09:38:18.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:18.186+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.191390+0000) 2019-10-13T09:38:18.194 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:18.189+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.193016+0000) 2019-10-13T09:38:18.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:18.190+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.193779+0000) 2019-10-13T09:38:18.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:18.190+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:18.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:18.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.200437+0000) 2019-10-13T09:38:18.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:18.426+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:18.433236+0000) 2019-10-13T09:38:21.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:21.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:21.684302+0000) 2019-10-13T09:38:23.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:23.193+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.197600+0000) 2019-10-13T09:38:23.199 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:23.193+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.198214+0000) 2019-10-13T09:38:23.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:23.193+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.198843+0000) 2019-10-13T09:38:23.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:23.197+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.201863+0000) 2019-10-13T09:38:23.203 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:23.197+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:23.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:23.197+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.203380+0000) 2019-10-13T09:38:23.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:23.197+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.203355+0000) 2019-10-13T09:38:23.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:23.197+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.204818+0000) 2019-10-13T09:38:23.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:23.201+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:23.205813+0000) 2019-10-13T09:38:24.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:23.993+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:24.000387+0000) 2019-10-13T09:38:24.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:24.285+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:24.288780+0000) 2019-10-13T09:38:24.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:24.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:24.386501+0000) 2019-10-13T09:38:24.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:24.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:24.410949+0000) 2019-10-13T09:38:26.873 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:38:26.873 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:26.876 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:38:26.876 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:27.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:27.657+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:27.661543+0000) 2019-10-13T09:38:27.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:27.917+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:27.921286+0000) 2019-10-13T09:38:28.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:28.201+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.206361+0000) 2019-10-13T09:38:28.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:28.201+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.206585+0000) 2019-10-13T09:38:28.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:28.201+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.208304+0000) 2019-10-13T09:38:28.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:28.201+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.207929+0000) 2019-10-13T09:38:28.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:28.205+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.208413+0000) 2019-10-13T09:38:28.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:28.201+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.209218+0000) 2019-10-13T09:38:28.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:28.205+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.209793+0000) 2019-10-13T09:38:28.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:28.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:28.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:28.209+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.214064+0000) 2019-10-13T09:38:28.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:28.429+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:28.433410+0000) 2019-10-13T09:38:31.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:31.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:31.684499+0000) 2019-10-13T09:38:33.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:33.209+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.214068+0000) 2019-10-13T09:38:33.215 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:33.209+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.214780+0000) 2019-10-13T09:38:33.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:33.209+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.215671+0000) 2019-10-13T09:38:33.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:33.209+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.216014+0000) 2019-10-13T09:38:33.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:33.209+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.216500+0000) 2019-10-13T09:38:33.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:33.209+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.216203+0000) 2019-10-13T09:38:33.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:33.213+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.216316+0000) 2019-10-13T09:38:33.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:33.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:33.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:33.217+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:33.223714+0000) 2019-10-13T09:38:34.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:33.997+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:34.000532+0000) 2019-10-13T09:38:34.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:34.285+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:34.289020+0000) 2019-10-13T09:38:34.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:34.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:34.386694+0000) 2019-10-13T09:38:34.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:34.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:34.411186+0000) 2019-10-13T09:38:37.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:37.661+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:37.661769+0000) 2019-10-13T09:38:37.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:37.921+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:37.927121+0000) 2019-10-13T09:38:38.223 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:38.221+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.222015+0000) 2019-10-13T09:38:38.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:38.221+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.222514+0000) 2019-10-13T09:38:38.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:38.221+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.223134+0000) 2019-10-13T09:38:38.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:38.221+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.224298+0000) 2019-10-13T09:38:38.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:38.221+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.224772+0000) 2019-10-13T09:38:38.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:38.221+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.224673+0000) 2019-10-13T09:38:38.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:38.225+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.226136+0000) 2019-10-13T09:38:38.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:38.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99158 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:38.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:38.225+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.229289+0000) 2019-10-13T09:38:38.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:38.429+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:38.433602+0000) 2019-10-13T09:38:41.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:41.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:41.684698+0000) 2019-10-13T09:38:43.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:43.229+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.230475+0000) 2019-10-13T09:38:43.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:43.229+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.230712+0000) 2019-10-13T09:38:43.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:43.225+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.230600+0000) 2019-10-13T09:38:43.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:43.229+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.231475+0000) 2019-10-13T09:38:43.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:43.229+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.232207+0000) 2019-10-13T09:38:43.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:43.229+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.233475+0000) 2019-10-13T09:38:43.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:43.233+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.233852+0000) 2019-10-13T09:38:43.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:43.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:43.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:43.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:43.239265+0000) 2019-10-13T09:38:44.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:43.997+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:44.000708+0000) 2019-10-13T09:38:44.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:44.285+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:44.289229+0000) 2019-10-13T09:38:44.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:44.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:44.386890+0000) 2019-10-13T09:38:44.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:44.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:44.411384+0000) 2019-10-13T09:38:47.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:47.661+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:47.661969+0000) 2019-10-13T09:38:47.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:47.933+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:47.933612+0000) 2019-10-13T09:38:48.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:48.237+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.239071+0000) 2019-10-13T09:38:48.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:48.233+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.239038+0000) 2019-10-13T09:38:48.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:48.237+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.239814+0000) 2019-10-13T09:38:48.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:48.237+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.240134+0000) 2019-10-13T09:38:48.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:48.237+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.240176+0000) 2019-10-13T09:38:48.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:48.237+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.241137+0000) 2019-10-13T09:38:48.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:48.241+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.242344+0000) 2019-10-13T09:38:48.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:48.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:48.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:48.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.245333+0000) 2019-10-13T09:38:48.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:48.433+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:48.433813+0000) 2019-10-13T09:38:51.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:51.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:51.684890+0000) 2019-10-13T09:38:53.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:53.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.246959+0000) 2019-10-13T09:38:53.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:53.245+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.247211+0000) 2019-10-13T09:38:53.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:53.245+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.247887+0000) 2019-10-13T09:38:53.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:53.245+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.248731+0000) 2019-10-13T09:38:53.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:53.245+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.249103+0000) 2019-10-13T09:38:53.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:53.245+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.249089+0000) 2019-10-13T09:38:53.250 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:53.245+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.249094+0000) 2019-10-13T09:38:53.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:53.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99247 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:53.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:53.249+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:53.255842+0000) 2019-10-13T09:38:54.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:53.997+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:54.000894+0000) 2019-10-13T09:38:54.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:54.285+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:54.289459+0000) 2019-10-13T09:38:54.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:54.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:54.387151+0000) 2019-10-13T09:38:54.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:54.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:54.411630+0000) 2019-10-13T09:38:56.982 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:38:56.982 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:56.986 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:38:56.986 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:38:57.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:57.661+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:57.662146+0000) 2019-10-13T09:38:57.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:57.941+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:57.944724+0000) 2019-10-13T09:38:58.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:38:58.253+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.254393+0000) 2019-10-13T09:38:58.256 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:38:58.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.255222+0000) 2019-10-13T09:38:58.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:38:58.253+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.255034+0000) 2019-10-13T09:38:58.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:38:58.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.255199+0000) 2019-10-13T09:38:58.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:38:58.253+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.256208+0000) 2019-10-13T09:38:58.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:58.253+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.256800+0000) 2019-10-13T09:38:58.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:38:58.253+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.257616+0000) 2019-10-13T09:38:58.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:38:58.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:38:58.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:38:58.257+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.263610+0000) 2019-10-13T09:38:58.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:38:58.433+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:38:58.434034+0000) 2019-10-13T09:39:01.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:01.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:01.685113+0000) 2019-10-13T09:39:03.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:03.257+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.261676+0000) 2019-10-13T09:39:03.264 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:03.261+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.263830+0000) 2019-10-13T09:39:03.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:03.257+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.263980+0000) 2019-10-13T09:39:03.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:03.261+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.264799+0000) 2019-10-13T09:39:03.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:03.261+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.265332+0000) 2019-10-13T09:39:03.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:03.261+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.265459+0000) 2019-10-13T09:39:03.267 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:03.265+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.266227+0000) 2019-10-13T09:39:03.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:03.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:03.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:03.269+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:03.272639+0000) 2019-10-13T09:39:04.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:03.997+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:04.001114+0000) 2019-10-13T09:39:04.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:04.285+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:04.289656+0000) 2019-10-13T09:39:04.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:04.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:04.387352+0000) 2019-10-13T09:39:04.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:04.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:04.411824+0000) 2019-10-13T09:39:07.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:07.661+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:07.662381+0000) 2019-10-13T09:39:07.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:07.953+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:07.956100+0000) 2019-10-13T09:39:08.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:08.269+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.269919+0000) 2019-10-13T09:39:08.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:08.269+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.270719+0000) 2019-10-13T09:39:08.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:08.269+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.270973+0000) 2019-10-13T09:39:08.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:08.269+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.272534+0000) 2019-10-13T09:39:08.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:08.269+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.272841+0000) 2019-10-13T09:39:08.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:08.269+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.274410+0000) 2019-10-13T09:39:08.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:08.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:08.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:08.273+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.275504+0000) 2019-10-13T09:39:08.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:08.269+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.275806+0000) 2019-10-13T09:39:08.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:08.433+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:08.434214+0000) 2019-10-13T09:39:11.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:11.681+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:11.685313+0000) 2019-10-13T09:39:13.281 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:13.277+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.280342+0000) 2019-10-13T09:39:13.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:13.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.280661+0000) 2019-10-13T09:39:13.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:13.277+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.281538+0000) 2019-10-13T09:39:13.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:13.277+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.281352+0000) 2019-10-13T09:39:13.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:13.281+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.283309+0000) 2019-10-13T09:39:13.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:13.277+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.283092+0000) 2019-10-13T09:39:13.285 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:13.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:13.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:13.281+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.282153+0000) 2019-10-13T09:39:13.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:13.281+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:13.284916+0000) 2019-10-13T09:39:14.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:13.997+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:14.001295+0000) 2019-10-13T09:39:14.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:14.289+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:14.289858+0000) 2019-10-13T09:39:14.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:14.385+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:14.387546+0000) 2019-10-13T09:39:14.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:14.409+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:14.412028+0000) 2019-10-13T09:39:17.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:17.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:17.662581+0000) 2019-10-13T09:39:17.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:17.956+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:17.962036+0000) 2019-10-13T09:39:18.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:18.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.287108+0000) 2019-10-13T09:39:18.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:18.284+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.287467+0000) 2019-10-13T09:39:18.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:18.284+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.287780+0000) 2019-10-13T09:39:18.290 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:18.284+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.288417+0000) 2019-10-13T09:39:18.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:18.284+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.289148+0000) 2019-10-13T09:39:18.291 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:18.284+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.289622+0000) 2019-10-13T09:39:18.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:18.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:18.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:18.288+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.291899+0000) 2019-10-13T09:39:18.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:18.292+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.297794+0000) 2019-10-13T09:39:18.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:18.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:18.434384+0000) 2019-10-13T09:39:21.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:21.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:21.685503+0000) 2019-10-13T09:39:23.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:23.292+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.295361+0000) 2019-10-13T09:39:23.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:23.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.296031+0000) 2019-10-13T09:39:23.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:23.292+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.296162+0000) 2019-10-13T09:39:23.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:23.292+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.296480+0000) 2019-10-13T09:39:23.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:23.292+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.296584+0000) 2019-10-13T09:39:23.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:23.292+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.297718+0000) 2019-10-13T09:39:23.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:23.296+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.298725+0000) 2019-10-13T09:39:23.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:23.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:23.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:23.300+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:23.304502+0000) 2019-10-13T09:39:24.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:23.996+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:24.001479+0000) 2019-10-13T09:39:24.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:24.288+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:24.290056+0000) 2019-10-13T09:39:24.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:24.384+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:24.387787+0000) 2019-10-13T09:39:24.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:24.408+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:24.412258+0000) 2019-10-13T09:39:27.113 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:39:27.113 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:27.117 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:39:27.117 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:27.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:27.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:27.662822+0000) 2019-10-13T09:39:27.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:27.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:27.968689+0000) 2019-10-13T09:39:28.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:28.300+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.304683+0000) 2019-10-13T09:39:28.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:28.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.305161+0000) 2019-10-13T09:39:28.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:28.300+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.305653+0000) 2019-10-13T09:39:28.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:28.304+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.306229+0000) 2019-10-13T09:39:28.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:28.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:28.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:28.300+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.307791+0000) 2019-10-13T09:39:28.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:28.304+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.309667+0000) 2019-10-13T09:39:28.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:28.308+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.311171+0000) 2019-10-13T09:39:28.316 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:28.308+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.315168+0000) 2019-10-13T09:39:28.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:28.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:28.434560+0000) 2019-10-13T09:39:31.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:31.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:31.685710+0000) 2019-10-13T09:39:33.311 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:33.304+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.310457+0000) 2019-10-13T09:39:33.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:33.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.311625+0000) 2019-10-13T09:39:33.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:33.308+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.312426+0000) 2019-10-13T09:39:33.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:33.308+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.312951+0000) 2019-10-13T09:39:33.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:33.308+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.312561+0000) 2019-10-13T09:39:33.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:33.312+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.313724+0000) 2019-10-13T09:39:33.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:33.312+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.314045+0000) 2019-10-13T09:39:33.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:33.312+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99490 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:33.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:33.312+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:33.318708+0000) 2019-10-13T09:39:34.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:33.996+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.001659+0000) 2019-10-13T09:39:34.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:34.288+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.290232+0000) 2019-10-13T09:39:34.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:34.384+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.387990+0000) 2019-10-13T09:39:34.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:34.408+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:34.412461+0000) 2019-10-13T09:39:37.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:37.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:37.663064+0000) 2019-10-13T09:39:37.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:37.972+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:37.975209+0000) 2019-10-13T09:39:38.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:38.316+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.319704+0000) 2019-10-13T09:39:38.321 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:38.312+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.319989+0000) 2019-10-13T09:39:38.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:38.316+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.320092+0000) 2019-10-13T09:39:38.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:38.316+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.320786+0000) 2019-10-13T09:39:38.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:38.316+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.321026+0000) 2019-10-13T09:39:38.322 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:38.320+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.321960+0000) 2019-10-13T09:39:38.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:38.320+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.322665+0000) 2019-10-13T09:39:38.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:38.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:38.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:38.320+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.327912+0000) 2019-10-13T09:39:38.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:38.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:38.434738+0000) 2019-10-13T09:39:41.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:41.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:41.685911+0000) 2019-10-13T09:39:43.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:43.324+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.326525+0000) 2019-10-13T09:39:43.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:43.324+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.327388+0000) 2019-10-13T09:39:43.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:43.324+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.327573+0000) 2019-10-13T09:39:43.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:43.320+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.327455+0000) 2019-10-13T09:39:43.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:43.324+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.328388+0000) 2019-10-13T09:39:43.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:43.324+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.328625+0000) 2019-10-13T09:39:43.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:43.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.328985+0000) 2019-10-13T09:39:43.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:43.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:43.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:43.328+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:43.335463+0000) 2019-10-13T09:39:44.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:44.000+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.001874+0000) 2019-10-13T09:39:44.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:44.288+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.290436+0000) 2019-10-13T09:39:44.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:44.384+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.388189+0000) 2019-10-13T09:39:44.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:44.408+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:44.412723+0000) 2019-10-13T09:39:47.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:47.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:47.663271+0000) 2019-10-13T09:39:47.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:47.980+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:47.985920+0000) 2019-10-13T09:39:48.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:48.332+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.334834+0000) 2019-10-13T09:39:48.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:48.332+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.335512+0000) 2019-10-13T09:39:48.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:48.332+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.335839+0000) 2019-10-13T09:39:48.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:48.332+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.337206+0000) 2019-10-13T09:39:48.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:48.332+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.337577+0000) 2019-10-13T09:39:48.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:48.336+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.338218+0000) 2019-10-13T09:39:48.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:48.336+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:48.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:48.336+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.338835+0000) 2019-10-13T09:39:48.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:48.336+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.341150+0000) 2019-10-13T09:39:48.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:48.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:48.434930+0000) 2019-10-13T09:39:51.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:51.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:51.686117+0000) 2019-10-13T09:39:53.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:53.340+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.342506+0000) 2019-10-13T09:39:53.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:53.336+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.342843+0000) 2019-10-13T09:39:53.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:53.340+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.343683+0000) 2019-10-13T09:39:53.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:53.344+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.345780+0000) 2019-10-13T09:39:53.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:53.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99612 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:53.348 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:53.344+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.347734+0000) 2019-10-13T09:39:53.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:53.344+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.347308+0000) 2019-10-13T09:39:53.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:53.344+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.348134+0000) 2019-10-13T09:39:53.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:53.344+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:53.348779+0000) 2019-10-13T09:39:54.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:54.000+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.002084+0000) 2019-10-13T09:39:54.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:54.288+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.290615+0000) 2019-10-13T09:39:54.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:54.388+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.388417+0000) 2019-10-13T09:39:54.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:54.412+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:54.412948+0000) 2019-10-13T09:39:57.227 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:39:57.227 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:57.231 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:39:57.231 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:39:57.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:57.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:57.663464+0000) 2019-10-13T09:39:57.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:57.988+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:57.992599+0000) 2019-10-13T09:39:58.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:39:58.344+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.349009+0000) 2019-10-13T09:39:58.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:39:58.348+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.350165+0000) 2019-10-13T09:39:58.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:39:58.348+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.350699+0000) 2019-10-13T09:39:58.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:39:58.348+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.351163+0000) 2019-10-13T09:39:58.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:39:58.348+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.351540+0000) 2019-10-13T09:39:58.352 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:58.348+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.351877+0000) 2019-10-13T09:39:58.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:39:58.348+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.352791+0000) 2019-10-13T09:39:58.355 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:39:58.352+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:39:58.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:39:58.352+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.356823+0000) 2019-10-13T09:39:58.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:39:58.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:39:58.435106+0000) 2019-10-13T09:40:01.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:01.680+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:01.686299+0000) 2019-10-13T09:40:03.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:03.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.358020+0000) 2019-10-13T09:40:03.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:03.356+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.358447+0000) 2019-10-13T09:40:03.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:03.356+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.359094+0000) 2019-10-13T09:40:03.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:03.356+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.359340+0000) 2019-10-13T09:40:03.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:03.356+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.359608+0000) 2019-10-13T09:40:03.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:03.352+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.359755+0000) 2019-10-13T09:40:03.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:03.356+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.360735+0000) 2019-10-13T09:40:03.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:03.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:03.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:03.364+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:03.368141+0000) 2019-10-13T09:40:04.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:04.000+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.002274+0000) 2019-10-13T09:40:04.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:04.288+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.290748+0000) 2019-10-13T09:40:04.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:04.384+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.388610+0000) 2019-10-13T09:40:04.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:04.412+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:04.413146+0000) 2019-10-13T09:40:07.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:07.660+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:07.663701+0000) 2019-10-13T09:40:08.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:07.996+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:07.999188+0000) 2019-10-13T09:40:08.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:08.360+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.365551+0000) 2019-10-13T09:40:08.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:08.364+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.366383+0000) 2019-10-13T09:40:08.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:08.360+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.366118+0000) 2019-10-13T09:40:08.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:08.364+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.367019+0000) 2019-10-13T09:40:08.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:08.364+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.367759+0000) 2019-10-13T09:40:08.370 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:08.364+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.368159+0000) 2019-10-13T09:40:08.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:08.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.368617+0000) 2019-10-13T09:40:08.371 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:08.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:08.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:08.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.375002+0000) 2019-10-13T09:40:08.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:08.432+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:08.435296+0000) 2019-10-13T09:40:11.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:11.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:11.686465+0000) 2019-10-13T09:40:13.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:13.372+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.374791+0000) 2019-10-13T09:40:13.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:13.372+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.375145+0000) 2019-10-13T09:40:13.376 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:13.372+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.375565+0000) 2019-10-13T09:40:13.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:13.372+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.376197+0000) 2019-10-13T09:40:13.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:13.372+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.376670+0000) 2019-10-13T09:40:13.377 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:13.372+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.376702+0000) 2019-10-13T09:40:13.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:13.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.377154+0000) 2019-10-13T09:40:13.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:13.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:13.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:13.380+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:13.384683+0000) 2019-10-13T09:40:14.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:13.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.002472+0000) 2019-10-13T09:40:14.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:14.287+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.290949+0000) 2019-10-13T09:40:14.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:14.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.388811+0000) 2019-10-13T09:40:14.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:14.411+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:14.413344+0000) 2019-10-13T09:40:17.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:17.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:17.663861+0000) 2019-10-13T09:40:18.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:18.008+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.010821+0000) 2019-10-13T09:40:18.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:18.379+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.382841+0000) 2019-10-13T09:40:18.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:18.379+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.385573+0000) 2019-10-13T09:40:18.388 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:18.383+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.388007+0000) 2019-10-13T09:40:18.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:18.383+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.388552+0000) 2019-10-13T09:40:18.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:18.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:18.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:18.383+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.389386+0000) 2019-10-13T09:40:18.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:18.388+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.389194+0000) 2019-10-13T09:40:18.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:18.387+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.389990+0000) 2019-10-13T09:40:18.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:18.388+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.391722+0000) 2019-10-13T09:40:18.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:18.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:18.435467+0000) 2019-10-13T09:40:21.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:21.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:21.686661+0000) 2019-10-13T09:40:23.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:23.387+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.391166+0000) 2019-10-13T09:40:23.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:23.387+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.392265+0000) 2019-10-13T09:40:23.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:23.387+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.393270+0000) 2019-10-13T09:40:23.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:23.387+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.393577+0000) 2019-10-13T09:40:23.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:23.391+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.394120+0000) 2019-10-13T09:40:23.395 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:23.391+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.393904+0000) 2019-10-13T09:40:23.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:23.391+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.394778+0000) 2019-10-13T09:40:23.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:23.391+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:23.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:23.395+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:23.397520+0000) 2019-10-13T09:40:24.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:23.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.002613+0000) 2019-10-13T09:40:24.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:24.287+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.291127+0000) 2019-10-13T09:40:24.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:24.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.389033+0000) 2019-10-13T09:40:24.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:24.407+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:24.413544+0000) 2019-10-13T09:40:27.366 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:40:27.367 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:27.370 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:40:27.371 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:27.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:27.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:27.664026+0000) 2019-10-13T09:40:28.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:28.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.017478+0000) 2019-10-13T09:40:28.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:28.395+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.400408+0000) 2019-10-13T09:40:28.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:28.395+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.400670+0000) 2019-10-13T09:40:28.402 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:28.399+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.401011+0000) 2019-10-13T09:40:28.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:28.395+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.401088+0000) 2019-10-13T09:40:28.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:28.399+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.401847+0000) 2019-10-13T09:40:28.403 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:28.399+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.402271+0000) 2019-10-13T09:40:28.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:28.399+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.403244+0000) 2019-10-13T09:40:28.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:28.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:28.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:28.407+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.408715+0000) 2019-10-13T09:40:28.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:28.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:28.435638+0000) 2019-10-13T09:40:31.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:31.683+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:31.686873+0000) 2019-10-13T09:40:33.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:33.403+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.407983+0000) 2019-10-13T09:40:33.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:33.407+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.410181+0000) 2019-10-13T09:40:33.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:33.407+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.411114+0000) 2019-10-13T09:40:33.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:33.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:33.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:33.407+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.412764+0000) 2019-10-13T09:40:33.414 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:33.407+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.413650+0000) 2019-10-13T09:40:33.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:33.407+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.413096+0000) 2019-10-13T09:40:33.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:33.411+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.414587+0000) 2019-10-13T09:40:33.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:33.411+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:33.414886+0000) 2019-10-13T09:40:34.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:33.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.002789+0000) 2019-10-13T09:40:34.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:34.287+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.291300+0000) 2019-10-13T09:40:34.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:34.387+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.389212+0000) 2019-10-13T09:40:34.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:34.411+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:34.413768+0000) 2019-10-13T09:40:37.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:37.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:37.664255+0000) 2019-10-13T09:40:38.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:38.023+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.023896+0000) 2019-10-13T09:40:38.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:38.411+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.416122+0000) 2019-10-13T09:40:38.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:38.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.415695+0000) 2019-10-13T09:40:38.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:38.411+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.416562+0000) 2019-10-13T09:40:38.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:38.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.416897+0000) 2019-10-13T09:40:38.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:38.415+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.419007+0000) 2019-10-13T09:40:38.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:38.415+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.419510+0000) 2019-10-13T09:40:38.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:38.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:38.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:38.415+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.420816+0000) 2019-10-13T09:40:38.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:38.423+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.426727+0000) 2019-10-13T09:40:38.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:38.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:38.435780+0000) 2019-10-13T09:40:41.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:41.683+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:41.687085+0000) 2019-10-13T09:40:43.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:43.419+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.423573+0000) 2019-10-13T09:40:43.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:43.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.424145+0000) 2019-10-13T09:40:43.425 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:43.419+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.424393+0000) 2019-10-13T09:40:43.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:43.423+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.425296+0000) 2019-10-13T09:40:43.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:43.423+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.426684+0000) 2019-10-13T09:40:43.428 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:43.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:43.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:43.423+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.429321+0000) 2019-10-13T09:40:43.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:43.427+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.431034+0000) 2019-10-13T09:40:43.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:43.431+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:43.433953+0000) 2019-10-13T09:40:44.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:43.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.002973+0000) 2019-10-13T09:40:44.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:44.287+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.291496+0000) 2019-10-13T09:40:44.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:44.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.389391+0000) 2019-10-13T09:40:44.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:44.411+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:44.413973+0000) 2019-10-13T09:40:47.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:47.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:47.664462+0000) 2019-10-13T09:40:48.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:48.027+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.030489+0000) 2019-10-13T09:40:48.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:48.427+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.431128+0000) 2019-10-13T09:40:48.433 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:48.427+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.433134+0000) 2019-10-13T09:40:48.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:48.431+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.434575+0000) 2019-10-13T09:40:48.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:48.431+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.434858+0000) 2019-10-13T09:40:48.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:48.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.435894+0000) 2019-10-13T09:40:48.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:48.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.436474+0000) 2019-10-13T09:40:48.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:48.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:48.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:48.431+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.437048+0000) 2019-10-13T09:40:48.438 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:48.435+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.436973+0000) 2019-10-13T09:40:48.438 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:48.435+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:48.437848+0000) 2019-10-13T09:40:51.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:51.683+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:51.687281+0000) 2019-10-13T09:40:53.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:53.435+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.439676+0000) 2019-10-13T09:40:53.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:53.435+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.440357+0000) 2019-10-13T09:40:53.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:53.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.441044+0000) 2019-10-13T09:40:53.442 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:53.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.441174+0000) 2019-10-13T09:40:53.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:53.439+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.441950+0000) 2019-10-13T09:40:53.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:53.439+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.442829+0000) 2019-10-13T09:40:53.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:53.439+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.443317+0000) 2019-10-13T09:40:53.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:53.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:53.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:53.443+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:53.447502+0000) 2019-10-13T09:40:54.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:53.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.003153+0000) 2019-10-13T09:40:54.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:54.291+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.291670+0000) 2019-10-13T09:40:54.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:54.383+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.389564+0000) 2019-10-13T09:40:54.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:54.411+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:54.414148+0000) 2019-10-13T09:40:57.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:40:57.493 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:57.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:40:57.497 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:40:57.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:57.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:57.664693+0000) 2019-10-13T09:40:58.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:58.035+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.037129+0000) 2019-10-13T09:40:58.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:58.431+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.436073+0000) 2019-10-13T09:40:58.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:40:58.443+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.448321+0000) 2019-10-13T09:40:58.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:40:58.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.449418+0000) 2019-10-13T09:40:58.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:40:58.447+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.449338+0000) 2019-10-13T09:40:58.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:40:58.447+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.450209+0000) 2019-10-13T09:40:58.451 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:40:58.447+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.450952+0000) 2019-10-13T09:40:58.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:40:58.447+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.451492+0000) 2019-10-13T09:40:58.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:40:58.447+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.451993+0000) 2019-10-13T09:40:58.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:40:58.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:40:58.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:40:58.455+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:40:58.456638+0000) 2019-10-13T09:41:01.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:01.683+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:01.687479+0000) 2019-10-13T09:41:03.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:03.451+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.457585+0000) 2019-10-13T09:41:03.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:03.455+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.458358+0000) 2019-10-13T09:41:03.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:03.455+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.458542+0000) 2019-10-13T09:41:03.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:03.455+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.460141+0000) 2019-10-13T09:41:03.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:03.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.461980+0000) 2019-10-13T09:41:03.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:03.459+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.462109+0000) 2019-10-13T09:41:03.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:03.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:03.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:03.459+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.462567+0000) 2019-10-13T09:41:03.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:03.463+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:03.465174+0000) 2019-10-13T09:41:04.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:03.999+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.003350+0000) 2019-10-13T09:41:04.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:04.291+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.291869+0000) 2019-10-13T09:41:04.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:04.387+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.389780+0000) 2019-10-13T09:41:04.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:04.411+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:04.414337+0000) 2019-10-13T09:41:07.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:07.659+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:07.664906+0000) 2019-10-13T09:41:08.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:08.039+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.043703+0000) 2019-10-13T09:41:08.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:08.435+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.436272+0000) 2019-10-13T09:41:08.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:08.459+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.465635+0000) 2019-10-13T09:41:08.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:08.463+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.466064+0000) 2019-10-13T09:41:08.467 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:08.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.466636+0000) 2019-10-13T09:41:08.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:08.463+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.466806+0000) 2019-10-13T09:41:08.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:08.463+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.468238+0000) 2019-10-13T09:41:08.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:08.463+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.468434+0000) 2019-10-13T09:41:08.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:08.463+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.468648+0000) 2019-10-13T09:41:08.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:08.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:08.477 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:08.475+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:08.476668+0000) 2019-10-13T09:41:11.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:11.683+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:11.687689+0000) 2019-10-13T09:41:13.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:13.466+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.473296+0000) 2019-10-13T09:41:13.474 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:13.466+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.473078+0000) 2019-10-13T09:41:13.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:13.470+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.473882+0000) 2019-10-13T09:41:13.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:13.471+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.475522+0000) 2019-10-13T09:41:13.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:13.470+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.476389+0000) 2019-10-13T09:41:13.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:13.474+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.477828+0000) 2019-10-13T09:41:13.479 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:13.474+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:13.479 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:13.474+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.479007+0000) 2019-10-13T09:41:13.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:13.479+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:13.481306+0000) 2019-10-13T09:41:14.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:14.002+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.003564+0000) 2019-10-13T09:41:14.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:14.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.292054+0000) 2019-10-13T09:41:14.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:14.386+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.389963+0000) 2019-10-13T09:41:14.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:14.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:14.414514+0000) 2019-10-13T09:41:17.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:17.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:17.665100+0000) 2019-10-13T09:41:18.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:18.047+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.050117+0000) 2019-10-13T09:41:18.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:18.430+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.436505+0000) 2019-10-13T09:41:18.482 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:18.474+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.481150+0000) 2019-10-13T09:41:18.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:18.478+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.482005+0000) 2019-10-13T09:41:18.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:18.478+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.483005+0000) 2019-10-13T09:41:18.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:18.478+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.483470+0000) 2019-10-13T09:41:18.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:18.478+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.485324+0000) 2019-10-13T09:41:18.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:18.478+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.484896+0000) 2019-10-13T09:41:18.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:18.483+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.484942+0000) 2019-10-13T09:41:18.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:18.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:18.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:18.491+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:18.494356+0000) 2019-10-13T09:41:21.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:21.687+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:21.687904+0000) 2019-10-13T09:41:23.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:23.486+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.491823+0000) 2019-10-13T09:41:23.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:23.486+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.492271+0000) 2019-10-13T09:41:23.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:23.486+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.492620+0000) 2019-10-13T09:41:23.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:23.491+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.492643+0000) 2019-10-13T09:41:23.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:23.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:23.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:23.490+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.495307+0000) 2019-10-13T09:41:23.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:23.490+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.496391+0000) 2019-10-13T09:41:23.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:23.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.496266+0000) 2019-10-13T09:41:23.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:23.499+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:23.502900+0000) 2019-10-13T09:41:24.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:24.002+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.003754+0000) 2019-10-13T09:41:24.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:24.286+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.292249+0000) 2019-10-13T09:41:24.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:24.386+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.390136+0000) 2019-10-13T09:41:24.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:24.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:24.414687+0000) 2019-10-13T09:41:27.620 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:41:27.620 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:27.623 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:41:27.624 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:27.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:27.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:27.665305+0000) 2019-10-13T09:41:28.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:28.055+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.056717+0000) 2019-10-13T09:41:28.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:28.434+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.436745+0000) 2019-10-13T09:41:28.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:28.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.499167+0000) 2019-10-13T09:41:28.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:28.495+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.499190+0000) 2019-10-13T09:41:28.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:28.494+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.499993+0000) 2019-10-13T09:41:28.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:28.494+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.500285+0000) 2019-10-13T09:41:28.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:28.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.502484+0000) 2019-10-13T09:41:28.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:28.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:28.504 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:28.498+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.504125+0000) 2019-10-13T09:41:28.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:28.498+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.504623+0000) 2019-10-13T09:41:28.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:28.507+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:28.510813+0000) 2019-10-13T09:41:31.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:31.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:31.688138+0000) 2019-10-13T09:41:33.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:33.502+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.506706+0000) 2019-10-13T09:41:33.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:33.502+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.507351+0000) 2019-10-13T09:41:33.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:33.502+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.508593+0000) 2019-10-13T09:41:33.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:33.502+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.509086+0000) 2019-10-13T09:41:33.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:33.502+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.509384+0000) 2019-10-13T09:41:33.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:33.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.509263+0000) 2019-10-13T09:41:33.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:33.506+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.510325+0000) 2019-10-13T09:41:33.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:33.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:33.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:33.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:33.512529+0000) 2019-10-13T09:41:34.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:34.002+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.003941+0000) 2019-10-13T09:41:34.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:34.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.292463+0000) 2019-10-13T09:41:34.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:34.386+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.390346+0000) 2019-10-13T09:41:34.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:34.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:34.414898+0000) 2019-10-13T09:41:37.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:37.662+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:37.665493+0000) 2019-10-13T09:41:38.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:38.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.067774+0000) 2019-10-13T09:41:38.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:38.434+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.436937+0000) 2019-10-13T09:41:38.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:38.510+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.515274+0000) 2019-10-13T09:41:38.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:38.510+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.514925+0000) 2019-10-13T09:41:38.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:38.510+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.515931+0000) 2019-10-13T09:41:38.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:38.510+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.515655+0000) 2019-10-13T09:41:38.527 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:38.514+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.517912+0000) 2019-10-13T09:41:38.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:38.514+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.518561+0000) 2019-10-13T09:41:38.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:38.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:38.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:38.514+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.520685+0000) 2019-10-13T09:41:38.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:38.522+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:38.524153+0000) 2019-10-13T09:41:41.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:41.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:41.688295+0000) 2019-10-13T09:41:43.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:43.518+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.522682+0000) 2019-10-13T09:41:43.524 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:43.518+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.523905+0000) 2019-10-13T09:41:43.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:43.518+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.524268+0000) 2019-10-13T09:41:43.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:43.518+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.525130+0000) 2019-10-13T09:41:43.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:43.518+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.525381+0000) 2019-10-13T09:41:43.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:43.518+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.525717+0000) 2019-10-13T09:41:43.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:43.522+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.527854+0000) 2019-10-13T09:41:43.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:43.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:43.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:43.526+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:43.531306+0000) 2019-10-13T09:41:44.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:44.002+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.004142+0000) 2019-10-13T09:41:44.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:44.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.292639+0000) 2019-10-13T09:41:44.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:44.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.390521+0000) 2019-10-13T09:41:44.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:44.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:44.415103+0000) 2019-10-13T09:41:47.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:47.658+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:47.665692+0000) 2019-10-13T09:41:48.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:48.074+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.076784+0000) 2019-10-13T09:41:48.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:48.434+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.437160+0000) 2019-10-13T09:41:48.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:48.526+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.531396+0000) 2019-10-13T09:41:48.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:48.526+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.532396+0000) 2019-10-13T09:41:48.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:48.526+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.532727+0000) 2019-10-13T09:41:48.535 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:48.530+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.534767+0000) 2019-10-13T09:41:48.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:48.530+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.534508+0000) 2019-10-13T09:41:48.536 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:48.530+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.535683+0000) 2019-10-13T09:41:48.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:48.530+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.536144+0000) 2019-10-13T09:41:48.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:48.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:48.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:48.538+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:48.543529+0000) 2019-10-13T09:41:51.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:51.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:51.688488+0000) 2019-10-13T09:41:53.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:53.534+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.539834+0000) 2019-10-13T09:41:53.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:53.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.540028+0000) 2019-10-13T09:41:53.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:53.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.540840+0000) 2019-10-13T09:41:53.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:53.534+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.541705+0000) 2019-10-13T09:41:53.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:53.538+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.542578+0000) 2019-10-13T09:41:53.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:53.538+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.542731+0000) 2019-10-13T09:41:53.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:53.538+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.543222+0000) 2019-10-13T09:41:53.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:53.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:53.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:53.546+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:53.549668+0000) 2019-10-13T09:41:54.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:54.002+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.004346+0000) 2019-10-13T09:41:54.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:54.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.292818+0000) 2019-10-13T09:41:54.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:54.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.390697+0000) 2019-10-13T09:41:54.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:54.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:54.415278+0000) 2019-10-13T09:41:57.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:57.662+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:57.665945+0000) 2019-10-13T09:41:57.858 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:41:57.858 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:57.862 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:41:57.862 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:41:58.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:58.082+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.083229+0000) 2019-10-13T09:41:58.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:58.434+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.437365+0000) 2019-10-13T09:41:58.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:41:58.546+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.548328+0000) 2019-10-13T09:41:58.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:41:58.542+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.549110+0000) 2019-10-13T09:41:58.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:41:58.546+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.551642+0000) 2019-10-13T09:41:58.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:41:58.546+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.551963+0000) 2019-10-13T09:41:58.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:41:58.546+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.552815+0000) 2019-10-13T09:41:58.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:41:58.546+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:41:58.554 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:41:58.546+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.553701+0000) 2019-10-13T09:41:58.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:41:58.550+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.554053+0000) 2019-10-13T09:41:58.561 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:41:58.554+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:41:58.559751+0000) 2019-10-13T09:42:01.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:01.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:01.688678+0000) 2019-10-13T09:42:03.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:03.550+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.555938+0000) 2019-10-13T09:42:03.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:03.550+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.557333+0000) 2019-10-13T09:42:03.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:03.554+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.557892+0000) 2019-10-13T09:42:03.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:03.554+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.558317+0000) 2019-10-13T09:42:03.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:03.554+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.558842+0000) 2019-10-13T09:42:03.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:03.554+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.559664+0000) 2019-10-13T09:42:03.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:03.554+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.560052+0000) 2019-10-13T09:42:03.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:03.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:03.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:03.558+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:03.562872+0000) 2019-10-13T09:42:04.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:03.998+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.004527+0000) 2019-10-13T09:42:04.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:04.286+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.292996+0000) 2019-10-13T09:42:04.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:04.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.390876+0000) 2019-10-13T09:42:04.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:04.410+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:04.415401+0000) 2019-10-13T09:42:07.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:07.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:07.666179+0000) 2019-10-13T09:42:08.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:08.086+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.089474+0000) 2019-10-13T09:42:08.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:08.433+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.437581+0000) 2019-10-13T09:42:08.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:08.557+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.565093+0000) 2019-10-13T09:42:08.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:08.562+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.565012+0000) 2019-10-13T09:42:08.567 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:08.561+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.566273+0000) 2019-10-13T09:42:08.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:08.561+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.566697+0000) 2019-10-13T09:42:08.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:08.561+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.566471+0000) 2019-10-13T09:42:08.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:08.561+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.567583+0000) 2019-10-13T09:42:08.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:08.561+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.567855+0000) 2019-10-13T09:42:08.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:08.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:08.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:08.570+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:08.573853+0000) 2019-10-13T09:42:11.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:11.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:11.688888+0000) 2019-10-13T09:42:13.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:13.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.573442+0000) 2019-10-13T09:42:13.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:13.570+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.574423+0000) 2019-10-13T09:42:13.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:13.569+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.575206+0000) 2019-10-13T09:42:13.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:13.569+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.576228+0000) 2019-10-13T09:42:13.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:13.569+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.576463+0000) 2019-10-13T09:42:13.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:13.569+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.577184+0000) 2019-10-13T09:42:13.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:13.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:13.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:13.573+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.577805+0000) 2019-10-13T09:42:13.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:13.578+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:13.581915+0000) 2019-10-13T09:42:14.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:14.001+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.004710+0000) 2019-10-13T09:42:14.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:14.289+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.293176+0000) 2019-10-13T09:42:14.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:14.389+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.391084+0000) 2019-10-13T09:42:14.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:14.413+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:14.415574+0000) 2019-10-13T09:42:17.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:17.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:17.666385+0000) 2019-10-13T09:42:18.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:18.094+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.095961+0000) 2019-10-13T09:42:18.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:18.433+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.437772+0000) 2019-10-13T09:42:18.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:18.573+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.580592+0000) 2019-10-13T09:42:18.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:18.578+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.580725+0000) 2019-10-13T09:42:18.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:18.577+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.581836+0000) 2019-10-13T09:42:18.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:18.577+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.582212+0000) 2019-10-13T09:42:18.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:18.577+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.582441+0000) 2019-10-13T09:42:18.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:18.577+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.583305+0000) 2019-10-13T09:42:18.584 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:18.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.583307+0000) 2019-10-13T09:42:18.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:18.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:18.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:18.586+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:18.590091+0000) 2019-10-13T09:42:21.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:21.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:21.689082+0000) 2019-10-13T09:42:23.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:23.585+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.589252+0000) 2019-10-13T09:42:23.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:23.585+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.589317+0000) 2019-10-13T09:42:23.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:23.589+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.590631+0000) 2019-10-13T09:42:23.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:23.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.591487+0000) 2019-10-13T09:42:23.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:23.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.591498+0000) 2019-10-13T09:42:23.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:23.589+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.592675+0000) 2019-10-13T09:42:23.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:23.589+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.593566+0000) 2019-10-13T09:42:23.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:23.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:23.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:23.594+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:23.598702+0000) 2019-10-13T09:42:24.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:24.001+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.004926+0000) 2019-10-13T09:42:24.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:24.289+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.293390+0000) 2019-10-13T09:42:24.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:24.389+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.391261+0000) 2019-10-13T09:42:24.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:24.413+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:24.415749+0000) 2019-10-13T09:42:27.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:27.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:27.666582+0000) 2019-10-13T09:42:28.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:28.102+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.102537+0000) 2019-10-13T09:42:28.191 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:42:28.191 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:28.195 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:42:28.195 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:28.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:28.437+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.437925+0000) 2019-10-13T09:42:28.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:28.593+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.596683+0000) 2019-10-13T09:42:28.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:28.593+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.597524+0000) 2019-10-13T09:42:28.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:28.597+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.598239+0000) 2019-10-13T09:42:28.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:28.597+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.598637+0000) 2019-10-13T09:42:28.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:28.597+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.599657+0000) 2019-10-13T09:42:28.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:28.594+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.599685+0000) 2019-10-13T09:42:28.603 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:28.601+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.602356+0000) 2019-10-13T09:42:28.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:28.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:28.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:28.606+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:28.608384+0000) 2019-10-13T09:42:31.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:31.686+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:31.689318+0000) 2019-10-13T09:42:33.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:33.605+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.607698+0000) 2019-10-13T09:42:33.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:33.605+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.608296+0000) 2019-10-13T09:42:33.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:33.606+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.608287+0000) 2019-10-13T09:42:33.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:33.605+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.609367+0000) 2019-10-13T09:42:33.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:33.609+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.609885+0000) 2019-10-13T09:42:33.611 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:33.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:33.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:33.609+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.611411+0000) 2019-10-13T09:42:33.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:33.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.612789+0000) 2019-10-13T09:42:33.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:33.618+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:33.622723+0000) 2019-10-13T09:42:34.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:34.001+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.005074+0000) 2019-10-13T09:42:34.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:34.289+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.293568+0000) 2019-10-13T09:42:34.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:34.389+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.391439+0000) 2019-10-13T09:42:34.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:34.413+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:34.415930+0000) 2019-10-13T09:42:37.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:37.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:37.666814+0000) 2019-10-13T09:42:38.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:38.105+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.111943+0000) 2019-10-13T09:42:38.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:38.437+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.438113+0000) 2019-10-13T09:42:38.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:38.613+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.614055+0000) 2019-10-13T09:42:38.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:38.613+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.614739+0000) 2019-10-13T09:42:38.617 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:38.613+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.616069+0000) 2019-10-13T09:42:38.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:38.613+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.616139+0000) 2019-10-13T09:42:38.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:38.613+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.616404+0000) 2019-10-13T09:42:38.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:38.613+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.617154+0000) 2019-10-13T09:42:38.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:38.617+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.617884+0000) 2019-10-13T09:42:38.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:38.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:38.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:38.617+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:38.620951+0000) 2019-10-13T09:42:41.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:41.685+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:41.689524+0000) 2019-10-13T09:42:43.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:43.617+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.621941+0000) 2019-10-13T09:42:43.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:43.621+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.623155+0000) 2019-10-13T09:42:43.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:43.621+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.623495+0000) 2019-10-13T09:42:43.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:43.621+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.624015+0000) 2019-10-13T09:42:43.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:43.621+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.625008+0000) 2019-10-13T09:42:43.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:43.621+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.625313+0000) 2019-10-13T09:42:43.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:43.625+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.626055+0000) 2019-10-13T09:42:43.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:43.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100637 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:43.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:43.625+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:43.631474+0000) 2019-10-13T09:42:44.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:44.001+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.005253+0000) 2019-10-13T09:42:44.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:44.293+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.293772+0000) 2019-10-13T09:42:44.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:44.389+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.391612+0000) 2019-10-13T09:42:44.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:44.413+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:44.416135+0000) 2019-10-13T09:42:47.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:47.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:47.667010+0000) 2019-10-13T09:42:48.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:48.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.118577+0000) 2019-10-13T09:42:48.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:48.437+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.438307+0000) 2019-10-13T09:42:48.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:48.625+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.630482+0000) 2019-10-13T09:42:48.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:48.629+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.631946+0000) 2019-10-13T09:42:48.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:48.629+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.632445+0000) 2019-10-13T09:42:48.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:48.629+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.632705+0000) 2019-10-13T09:42:48.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:48.629+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.632077+0000) 2019-10-13T09:42:48.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:48.629+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.633152+0000) 2019-10-13T09:42:48.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:48.629+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.633484+0000) 2019-10-13T09:42:48.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:48.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100669 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:48.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:48.637+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:48.640393+0000) 2019-10-13T09:42:51.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:51.685+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:51.689752+0000) 2019-10-13T09:42:53.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:53.633+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.639787+0000) 2019-10-13T09:42:53.641 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:53.637+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.640731+0000) 2019-10-13T09:42:53.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:53.637+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.641180+0000) 2019-10-13T09:42:53.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:53.641+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.643370+0000) 2019-10-13T09:42:53.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:53.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:53.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:53.641+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.644905+0000) 2019-10-13T09:42:53.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:53.641+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.645128+0000) 2019-10-13T09:42:53.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:53.641+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.645494+0000) 2019-10-13T09:42:53.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:53.641+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:53.647668+0000) 2019-10-13T09:42:54.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:54.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.005436+0000) 2019-10-13T09:42:54.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:54.293+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.293981+0000) 2019-10-13T09:42:54.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:54.389+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.391826+0000) 2019-10-13T09:42:54.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:54.413+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:54.416343+0000) 2019-10-13T09:42:57.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:57.665+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:57.667206+0000) 2019-10-13T09:42:58.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:58.121+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.125188+0000) 2019-10-13T09:42:58.248 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:42:58.249 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:58.252 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:42:58.252 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:42:58.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:58.437+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.438523+0000) 2019-10-13T09:42:58.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:42:58.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.647950+0000) 2019-10-13T09:42:58.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:42:58.645+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.648289+0000) 2019-10-13T09:42:58.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:42:58.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.648173+0000) 2019-10-13T09:42:58.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:42:58.645+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.648959+0000) 2019-10-13T09:42:58.650 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:42:58.645+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.649187+0000) 2019-10-13T09:42:58.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:42:58.645+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.649577+0000) 2019-10-13T09:42:58.651 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:42:58.649+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.650955+0000) 2019-10-13T09:42:58.653 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:42:58.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:42:58.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:42:58.649+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:42:58.655864+0000) 2019-10-13T09:43:01.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:01.685+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:01.689953+0000) 2019-10-13T09:43:03.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:03.653+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.656153+0000) 2019-10-13T09:43:03.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:03.653+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.656964+0000) 2019-10-13T09:43:03.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:03.657+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.658865+0000) 2019-10-13T09:43:03.660 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:03.657+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.659432+0000) 2019-10-13T09:43:03.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:03.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.659841+0000) 2019-10-13T09:43:03.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:03.657+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.660737+0000) 2019-10-13T09:43:03.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:03.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:03.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:03.657+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.661607+0000) 2019-10-13T09:43:03.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:03.661+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:03.666305+0000) 2019-10-13T09:43:04.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:04.000+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.005616+0000) 2019-10-13T09:43:04.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:04.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.294177+0000) 2019-10-13T09:43:04.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:04.388+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.392016+0000) 2019-10-13T09:43:04.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:04.412+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:04.416540+0000) 2019-10-13T09:43:07.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:07.664+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:07.667406+0000) 2019-10-13T09:43:08.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:08.129+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.131641+0000) 2019-10-13T09:43:08.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:08.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.438727+0000) 2019-10-13T09:43:08.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:08.660+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.664339+0000) 2019-10-13T09:43:08.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:08.660+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.665145+0000) 2019-10-13T09:43:08.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:08.660+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.665559+0000) 2019-10-13T09:43:08.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:08.664+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.666217+0000) 2019-10-13T09:43:08.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:08.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.666966+0000) 2019-10-13T09:43:08.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:08.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100787 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:08.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:08.665+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.668989+0000) 2019-10-13T09:43:08.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:08.668+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.670374+0000) 2019-10-13T09:43:08.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:08.669+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:08.674165+0000) 2019-10-13T09:43:11.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:11.689+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:11.690146+0000) 2019-10-13T09:43:13.673 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:13.668+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.672345+0000) 2019-10-13T09:43:13.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:13.668+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.672874+0000) 2019-10-13T09:43:13.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:13.668+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.673643+0000) 2019-10-13T09:43:13.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:13.672+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.674281+0000) 2019-10-13T09:43:13.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:13.669+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.674275+0000) 2019-10-13T09:43:13.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:13.672+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.675053+0000) 2019-10-13T09:43:13.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:13.672+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.675472+0000) 2019-10-13T09:43:13.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:13.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:13.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:13.673+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:13.679256+0000) 2019-10-13T09:43:14.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:14.004+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.005800+0000) 2019-10-13T09:43:14.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:14.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.294380+0000) 2019-10-13T09:43:14.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:14.388+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.392141+0000) 2019-10-13T09:43:14.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:14.412+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:14.416720+0000) 2019-10-13T09:43:17.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:17.664+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:17.667628+0000) 2019-10-13T09:43:18.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:18.137+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.137585+0000) 2019-10-13T09:43:18.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:18.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.438923+0000) 2019-10-13T09:43:18.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:18.676+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.679671+0000) 2019-10-13T09:43:18.681 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:18.676+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.680735+0000) 2019-10-13T09:43:18.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:18.676+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.681226+0000) 2019-10-13T09:43:18.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:18.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.681221+0000) 2019-10-13T09:43:18.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:18.680+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.682924+0000) 2019-10-13T09:43:18.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:18.680+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.683406+0000) 2019-10-13T09:43:18.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:18.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:18.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:18.681+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.684723+0000) 2019-10-13T09:43:18.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:18.689+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:18.692424+0000) 2019-10-13T09:43:21.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:21.685+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:21.690339+0000) 2019-10-13T09:43:23.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:23.684+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.688790+0000) 2019-10-13T09:43:23.690 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:23.684+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.689181+0000) 2019-10-13T09:43:23.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:23.684+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.689685+0000) 2019-10-13T09:43:23.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:23.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.691128+0000) 2019-10-13T09:43:23.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:23.685+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.691012+0000) 2019-10-13T09:43:23.692 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:23.688+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.691964+0000) 2019-10-13T09:43:23.693 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:23.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.692804+0000) 2019-10-13T09:43:23.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:23.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:23.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:23.693+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:23.697598+0000) 2019-10-13T09:43:24.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:24.004+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.005989+0000) 2019-10-13T09:43:24.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:24.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.294595+0000) 2019-10-13T09:43:24.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:24.388+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.392303+0000) 2019-10-13T09:43:24.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:24.412+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:24.416889+0000) 2019-10-13T09:43:27.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:27.664+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:27.667826+0000) 2019-10-13T09:43:28.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:28.141+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.144431+0000) 2019-10-13T09:43:28.319 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:43:28.319 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:28.323 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:43:28.323 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:28.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:28.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.439115+0000) 2019-10-13T09:43:28.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:28.689+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.695985+0000) 2019-10-13T09:43:28.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:28.692+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.696937+0000) 2019-10-13T09:43:28.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:28.692+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.697082+0000) 2019-10-13T09:43:28.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:28.692+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.697436+0000) 2019-10-13T09:43:28.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:28.696+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.698073+0000) 2019-10-13T09:43:28.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:28.696+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.698184+0000) 2019-10-13T09:43:28.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:28.696+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.699575+0000) 2019-10-13T09:43:28.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:28.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:28.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:28.701+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:28.705902+0000) 2019-10-13T09:43:31.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:31.689+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:31.690533+0000) 2019-10-13T09:43:33.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:33.700+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.704284+0000) 2019-10-13T09:43:33.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:33.700+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.704710+0000) 2019-10-13T09:43:33.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:33.700+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.705158+0000) 2019-10-13T09:43:33.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:33.701+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.707525+0000) 2019-10-13T09:43:33.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:33.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:33.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:33.704+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.709325+0000) 2019-10-13T09:43:33.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:33.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.709948+0000) 2019-10-13T09:43:33.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:33.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.710314+0000) 2019-10-13T09:43:33.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:33.705+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:33.711706+0000) 2019-10-13T09:43:34.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:34.004+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.006167+0000) 2019-10-13T09:43:34.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:34.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.294771+0000) 2019-10-13T09:43:34.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:34.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.392513+0000) 2019-10-13T09:43:34.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:34.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:34.417064+0000) 2019-10-13T09:43:37.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:37.664+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:37.668022+0000) 2019-10-13T09:43:38.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:38.149+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.150475+0000) 2019-10-13T09:43:38.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:38.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.439319+0000) 2019-10-13T09:43:38.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:38.708+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.712224+0000) 2019-10-13T09:43:38.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:38.708+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.712646+0000) 2019-10-13T09:43:38.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:38.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.713141+0000) 2019-10-13T09:43:38.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:38.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.713634+0000) 2019-10-13T09:43:38.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:38.712+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.714520+0000) 2019-10-13T09:43:38.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:38.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.715255+0000) 2019-10-13T09:43:38.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:38.709+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.715289+0000) 2019-10-13T09:43:38.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:38.712+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:38.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:38.717+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:38.720107+0000) 2019-10-13T09:43:41.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:41.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:41.690728+0000) 2019-10-13T09:43:43.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:43.716+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.719804+0000) 2019-10-13T09:43:43.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:43.716+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.720626+0000) 2019-10-13T09:43:43.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:43.716+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.722211+0000) 2019-10-13T09:43:43.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:43.720+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.723971+0000) 2019-10-13T09:43:43.725 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:43.720+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:43.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:43.720+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.724732+0000) 2019-10-13T09:43:43.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:43.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.725178+0000) 2019-10-13T09:43:43.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:43.720+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.724940+0000) 2019-10-13T09:43:43.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:43.724+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:43.728968+0000) 2019-10-13T09:43:44.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:44.004+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.006354+0000) 2019-10-13T09:43:44.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:44.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.294944+0000) 2019-10-13T09:43:44.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:44.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.392688+0000) 2019-10-13T09:43:44.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:44.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:44.417254+0000) 2019-10-13T09:43:47.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:47.664+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:47.668249+0000) 2019-10-13T09:43:48.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:48.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.157082+0000) 2019-10-13T09:43:48.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:48.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.439528+0000) 2019-10-13T09:43:48.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:48.724+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.728745+0000) 2019-10-13T09:43:48.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:48.724+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.730458+0000) 2019-10-13T09:43:48.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:48.728+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.731134+0000) 2019-10-13T09:43:48.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:48.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101023 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:48.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:48.728+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.732899+0000) 2019-10-13T09:43:48.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:48.728+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.733167+0000) 2019-10-13T09:43:48.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:48.728+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.733420+0000) 2019-10-13T09:43:48.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:48.728+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.733742+0000) 2019-10-13T09:43:48.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:48.732+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:48.734010+0000) 2019-10-13T09:43:51.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:51.688+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:51.690939+0000) 2019-10-13T09:43:53.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:53.732+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.736183+0000) 2019-10-13T09:43:53.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:53.732+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.736838+0000) 2019-10-13T09:43:53.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:53.732+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.737504+0000) 2019-10-13T09:43:53.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:53.736+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.738020+0000) 2019-10-13T09:43:53.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:53.732+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.737971+0000) 2019-10-13T09:43:53.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:53.736+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.738769+0000) 2019-10-13T09:43:53.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:53.736+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.739228+0000) 2019-10-13T09:43:53.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:53.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:53.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:53.740+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:53.745223+0000) 2019-10-13T09:43:54.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:54.004+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.006560+0000) 2019-10-13T09:43:54.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:54.292+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.295139+0000) 2019-10-13T09:43:54.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:54.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.392873+0000) 2019-10-13T09:43:54.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:54.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:54.417431+0000) 2019-10-13T09:43:57.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:57.668+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:57.668479+0000) 2019-10-13T09:43:58.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:58.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.163715+0000) 2019-10-13T09:43:58.434 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:43:58.434 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:58.437 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:43:58.438 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:43:58.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:58.436+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.439722+0000) 2019-10-13T09:43:58.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:43:58.740+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.743918+0000) 2019-10-13T09:43:58.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:43:58.740+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.745413+0000) 2019-10-13T09:43:58.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:43:58.740+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.745579+0000) 2019-10-13T09:43:58.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:43:58.740+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.745973+0000) 2019-10-13T09:43:58.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:43:58.744+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.746730+0000) 2019-10-13T09:43:58.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:43:58.744+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.747139+0000) 2019-10-13T09:43:58.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:43:58.744+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.748059+0000) 2019-10-13T09:43:58.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:43:58.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:43:58.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:43:58.744+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:43:58.750690+0000) 2019-10-13T09:44:01.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:01.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:01.691159+0000) 2019-10-13T09:44:03.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:03.747+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.752883+0000) 2019-10-13T09:44:03.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:03.751+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.753786+0000) 2019-10-13T09:44:03.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:03.751+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.754049+0000) 2019-10-13T09:44:03.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:03.751+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.754422+0000) 2019-10-13T09:44:03.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:03.748+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.754873+0000) 2019-10-13T09:44:03.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:03.751+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.755748+0000) 2019-10-13T09:44:03.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:03.751+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.756245+0000) 2019-10-13T09:44:03.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:03.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:03.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:03.760+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:03.764707+0000) 2019-10-13T09:44:04.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:04.003+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.006762+0000) 2019-10-13T09:44:04.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:04.291+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.295349+0000) 2019-10-13T09:44:04.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:04.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.393050+0000) 2019-10-13T09:44:04.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:04.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:04.417653+0000) 2019-10-13T09:44:07.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:07.667+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:07.668685+0000) 2019-10-13T09:44:08.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:08.168+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.170265+0000) 2019-10-13T09:44:08.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:08.435+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.439888+0000) 2019-10-13T09:44:08.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:08.755+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.760950+0000) 2019-10-13T09:44:08.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:08.755+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.761656+0000) 2019-10-13T09:44:08.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:08.759+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.762295+0000) 2019-10-13T09:44:08.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:08.759+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.762704+0000) 2019-10-13T09:44:08.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:08.759+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.763492+0000) 2019-10-13T09:44:08.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:08.756+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.763670+0000) 2019-10-13T09:44:08.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:08.759+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.764430+0000) 2019-10-13T09:44:08.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:08.759+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:08.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:08.760+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:08.766429+0000) 2019-10-13T09:44:11.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:11.688+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:11.691358+0000) 2019-10-13T09:44:13.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:13.763+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.769127+0000) 2019-10-13T09:44:13.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:13.763+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.769329+0000) 2019-10-13T09:44:13.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:13.767+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.769864+0000) 2019-10-13T09:44:13.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:13.767+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.770240+0000) 2019-10-13T09:44:13.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:13.767+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.770948+0000) 2019-10-13T09:44:13.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:13.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.771406+0000) 2019-10-13T09:44:13.773 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:13.767+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.772406+0000) 2019-10-13T09:44:13.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:13.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:13.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:13.772+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:13.776813+0000) 2019-10-13T09:44:14.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:14.003+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.006936+0000) 2019-10-13T09:44:14.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:14.291+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.295528+0000) 2019-10-13T09:44:14.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:14.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.393244+0000) 2019-10-13T09:44:14.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:14.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:14.417847+0000) 2019-10-13T09:44:17.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:17.663+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:17.668881+0000) 2019-10-13T09:44:18.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:18.176+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.176763+0000) 2019-10-13T09:44:18.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:18.435+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.440119+0000) 2019-10-13T09:44:18.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:18.771+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.777402+0000) 2019-10-13T09:44:18.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:18.771+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.777657+0000) 2019-10-13T09:44:18.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:18.775+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.778112+0000) 2019-10-13T09:44:18.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:18.775+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.778751+0000) 2019-10-13T09:44:18.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:18.775+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.779574+0000) 2019-10-13T09:44:18.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:18.775+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.779989+0000) 2019-10-13T09:44:18.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:18.776+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.780468+0000) 2019-10-13T09:44:18.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:18.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:18.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:18.780+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:18.785029+0000) 2019-10-13T09:44:21.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:21.684+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:21.691570+0000) 2019-10-13T09:44:23.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:23.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.784573+0000) 2019-10-13T09:44:23.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:23.780+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.785168+0000) 2019-10-13T09:44:23.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:23.783+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.786532+0000) 2019-10-13T09:44:23.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:23.783+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.787199+0000) 2019-10-13T09:44:23.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:23.783+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.787542+0000) 2019-10-13T09:44:23.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:23.783+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.787968+0000) 2019-10-13T09:44:23.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:23.783+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.788484+0000) 2019-10-13T09:44:23.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:23.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:23.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:23.784+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:23.791365+0000) 2019-10-13T09:44:24.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:24.003+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.007122+0000) 2019-10-13T09:44:24.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:24.291+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.295709+0000) 2019-10-13T09:44:24.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:24.387+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.393449+0000) 2019-10-13T09:44:24.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:24.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:24.418054+0000) 2019-10-13T09:44:27.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:27.663+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:27.669078+0000) 2019-10-13T09:44:28.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:28.180+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.183236+0000) 2019-10-13T09:44:28.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:28.435+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.440309+0000) 2019-10-13T09:44:28.540 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:44:28.540 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:28.567 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:44:28.567 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:28.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:28.787+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.793074+0000) 2019-10-13T09:44:28.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:28.787+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.793553+0000) 2019-10-13T09:44:28.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:28.791+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.794334+0000) 2019-10-13T09:44:28.828 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:28.791+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.794515+0000) 2019-10-13T09:44:28.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:28.791+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.794717+0000) 2019-10-13T09:44:28.828 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:28.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.796046+0000) 2019-10-13T09:44:28.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:28.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:28.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:28.788+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.793449+0000) 2019-10-13T09:44:28.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:28.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:28.800763+0000) 2019-10-13T09:44:31.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:31.688+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:31.691799+0000) 2019-10-13T09:44:33.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:33.795+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.800729+0000) 2019-10-13T09:44:33.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:33.795+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.801407+0000) 2019-10-13T09:44:33.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:33.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.801922+0000) 2019-10-13T09:44:33.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:33.799+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.802356+0000) 2019-10-13T09:44:33.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:33.796+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.802051+0000) 2019-10-13T09:44:33.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:33.799+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.803027+0000) 2019-10-13T09:44:33.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:33.799+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.804220+0000) 2019-10-13T09:44:33.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:33.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:33.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:33.800+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:33.806841+0000) 2019-10-13T09:44:34.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:34.003+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.007338+0000) 2019-10-13T09:44:34.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:34.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.295884+0000) 2019-10-13T09:44:34.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:34.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.393635+0000) 2019-10-13T09:44:34.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:34.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:34.418228+0000) 2019-10-13T09:44:37.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:37.667+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:37.669279+0000) 2019-10-13T09:44:38.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:38.188+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.189720+0000) 2019-10-13T09:44:38.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:38.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.440499+0000) 2019-10-13T09:44:38.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:38.804+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.807923+0000) 2019-10-13T09:44:38.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:38.803+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.808949+0000) 2019-10-13T09:44:38.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:38.807+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.810430+0000) 2019-10-13T09:44:38.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:38.807+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.810692+0000) 2019-10-13T09:44:38.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:38.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:38.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:38.807+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.812403+0000) 2019-10-13T09:44:38.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:38.807+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.812674+0000) 2019-10-13T09:44:38.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:38.811+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.814625+0000) 2019-10-13T09:44:38.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:38.816+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:38.820349+0000) 2019-10-13T09:44:41.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:41.688+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:41.691996+0000) 2019-10-13T09:44:43.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:43.811+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.816296+0000) 2019-10-13T09:44:43.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:43.811+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.816726+0000) 2019-10-13T09:44:43.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:43.811+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.817102+0000) 2019-10-13T09:44:43.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:43.816+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.817844+0000) 2019-10-13T09:44:43.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:43.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:43.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:43.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.820769+0000) 2019-10-13T09:44:43.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:43.815+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.820811+0000) 2019-10-13T09:44:43.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:43.819+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.821846+0000) 2019-10-13T09:44:43.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:43.824+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:43.827558+0000) 2019-10-13T09:44:44.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:44.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.007546+0000) 2019-10-13T09:44:44.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:44.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.296062+0000) 2019-10-13T09:44:44.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:44.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.393882+0000) 2019-10-13T09:44:44.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:44.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:44.418405+0000) 2019-10-13T09:44:47.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:47.667+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:47.669435+0000) 2019-10-13T09:44:48.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:48.195+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.196132+0000) 2019-10-13T09:44:48.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:48.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.440732+0000) 2019-10-13T09:44:48.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:48.819+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.823548+0000) 2019-10-13T09:44:48.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:48.819+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.824657+0000) 2019-10-13T09:44:48.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:48.819+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.824946+0000) 2019-10-13T09:44:48.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:48.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.826077+0000) 2019-10-13T09:44:48.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:48.823+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.826064+0000) 2019-10-13T09:44:48.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:48.823+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.826263+0000) 2019-10-13T09:44:48.828 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:48.823+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.826836+0000) 2019-10-13T09:44:48.828 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:48.823+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:48.827171+0000) 2019-10-13T09:44:48.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:48.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:51.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:51.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:51.692190+0000) 2019-10-13T09:44:53.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:53.827+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.831001+0000) 2019-10-13T09:44:53.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:53.827+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.831530+0000) 2019-10-13T09:44:53.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:53.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.831894+0000) 2019-10-13T09:44:53.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:53.827+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.833556+0000) 2019-10-13T09:44:53.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:53.831+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.833947+0000) 2019-10-13T09:44:53.835 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:53.831+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.834516+0000) 2019-10-13T09:44:53.836 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:53.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.835107+0000) 2019-10-13T09:44:53.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:53.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:53.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:53.839+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:53.840266+0000) 2019-10-13T09:44:54.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:54.003+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.007729+0000) 2019-10-13T09:44:54.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:54.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.296243+0000) 2019-10-13T09:44:54.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:54.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.394062+0000) 2019-10-13T09:44:54.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:54.415+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:54.418542+0000) 2019-10-13T09:44:57.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:57.662+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:57.669654+0000) 2019-10-13T09:44:58.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:58.199+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.202647+0000) 2019-10-13T09:44:58.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:58.434+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.440925+0000) 2019-10-13T09:44:58.694 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:44:58.694 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:58.698 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:44:58.698 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:44:58.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:44:58.834+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.839112+0000) 2019-10-13T09:44:58.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:44:58.839+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.840248+0000) 2019-10-13T09:44:58.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:44:58.834+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.841305+0000) 2019-10-13T09:44:58.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:44:58.838+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.841937+0000) 2019-10-13T09:44:58.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:44:58.838+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.842155+0000) 2019-10-13T09:44:58.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:44:58.838+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.842571+0000) 2019-10-13T09:44:58.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:44:58.838+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.843211+0000) 2019-10-13T09:44:58.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:44:58.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:44:58.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:44:58.843+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:44:58.845667+0000) 2019-10-13T09:45:01.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:01.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:01.692391+0000) 2019-10-13T09:45:03.846 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:03.842+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.845630+0000) 2019-10-13T09:45:03.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:03.842+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.846585+0000) 2019-10-13T09:45:03.847 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:03.842+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.846984+0000) 2019-10-13T09:45:03.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:03.842+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.847892+0000) 2019-10-13T09:45:03.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:03.843+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.848065+0000) 2019-10-13T09:45:03.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:03.842+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.848705+0000) 2019-10-13T09:45:03.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:03.842+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.849235+0000) 2019-10-13T09:45:03.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:03.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:03.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:03.851+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:03.853550+0000) 2019-10-13T09:45:04.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:04.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.007940+0000) 2019-10-13T09:45:04.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:04.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.296412+0000) 2019-10-13T09:45:04.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:04.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.394297+0000) 2019-10-13T09:45:04.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:04.414+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:04.418726+0000) 2019-10-13T09:45:07.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:07.666+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:07.669812+0000) 2019-10-13T09:45:08.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:08.207+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.209342+0000) 2019-10-13T09:45:08.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:08.438+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.441148+0000) 2019-10-13T09:45:08.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:08.850+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.855832+0000) 2019-10-13T09:45:08.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:08.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.857050+0000) 2019-10-13T09:45:08.858 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:08.850+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.857149+0000) 2019-10-13T09:45:08.858 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:08.850+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.857529+0000) 2019-10-13T09:45:08.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:08.854+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.858291+0000) 2019-10-13T09:45:08.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:08.855+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.858014+0000) 2019-10-13T09:45:08.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:08.854+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.858876+0000) 2019-10-13T09:45:08.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:08.854+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:08.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:08.859+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:08.862499+0000) 2019-10-13T09:45:11.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:11.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:11.692546+0000) 2019-10-13T09:45:13.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:13.858+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.863003+0000) 2019-10-13T09:45:13.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:13.858+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.863364+0000) 2019-10-13T09:45:13.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:13.858+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.863700+0000) 2019-10-13T09:45:13.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:13.858+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.864470+0000) 2019-10-13T09:45:13.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:13.863+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.864716+0000) 2019-10-13T09:45:13.866 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:13.858+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.864840+0000) 2019-10-13T09:45:13.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:13.858+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.865331+0000) 2019-10-13T09:45:13.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:13.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:13.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:13.871+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:13.873729+0000) 2019-10-13T09:45:14.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:14.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.008121+0000) 2019-10-13T09:45:14.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:14.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.296592+0000) 2019-10-13T09:45:14.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:14.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.394480+0000) 2019-10-13T09:45:14.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:14.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:14.418900+0000) 2019-10-13T09:45:17.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:17.666+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:17.670016+0000) 2019-10-13T09:45:18.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:18.215+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.215845+0000) 2019-10-13T09:45:18.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:18.438+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.441368+0000) 2019-10-13T09:45:18.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:18.866+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.871365+0000) 2019-10-13T09:45:18.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:18.867+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.871444+0000) 2019-10-13T09:45:18.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:18.866+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.872274+0000) 2019-10-13T09:45:18.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:18.866+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.873497+0000) 2019-10-13T09:45:18.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:18.870+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.874153+0000) 2019-10-13T09:45:18.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:18.870+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.874564+0000) 2019-10-13T09:45:18.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:18.870+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.875681+0000) 2019-10-13T09:45:18.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:18.875+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:18.876761+0000) 2019-10-13T09:45:18.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:18.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:21.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:21.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:21.692779+0000) 2019-10-13T09:45:23.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:23.874+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.879713+0000) 2019-10-13T09:45:23.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:23.874+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.880395+0000) 2019-10-13T09:45:23.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:23.874+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.880633+0000) 2019-10-13T09:45:23.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:23.874+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.881195+0000) 2019-10-13T09:45:23.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:23.878+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.882995+0000) 2019-10-13T09:45:23.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:23.878+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.883219+0000) 2019-10-13T09:45:23.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:23.879+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.883554+0000) 2019-10-13T09:45:23.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:23.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:23.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:23.883+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:23.885644+0000) 2019-10-13T09:45:24.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:24.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.008295+0000) 2019-10-13T09:45:24.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:24.290+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.296804+0000) 2019-10-13T09:45:24.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:24.394+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.394660+0000) 2019-10-13T09:45:24.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:24.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:24.419077+0000) 2019-10-13T09:45:27.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:27.666+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:27.670235+0000) 2019-10-13T09:45:28.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:28.219+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.222303+0000) 2019-10-13T09:45:28.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:28.438+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.441567+0000) 2019-10-13T09:45:28.809 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:45:28.809 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:28.813 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:45:28.813 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:28.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:28.882+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.888119+0000) 2019-10-13T09:45:28.889 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:28.882+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.888651+0000) 2019-10-13T09:45:28.890 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:28.882+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.889387+0000) 2019-10-13T09:45:28.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:28.886+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.891856+0000) 2019-10-13T09:45:28.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:28.886+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.893170+0000) 2019-10-13T09:45:28.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:28.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:28.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:28.891+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.893510+0000) 2019-10-13T09:45:28.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:28.886+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.893397+0000) 2019-10-13T09:45:28.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:28.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:28.899458+0000) 2019-10-13T09:45:31.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:31.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:31.692972+0000) 2019-10-13T09:45:33.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:33.890+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.896963+0000) 2019-10-13T09:45:33.899 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:33.894+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.898386+0000) 2019-10-13T09:45:33.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:33.895+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.898272+0000) 2019-10-13T09:45:33.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:33.894+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.899195+0000) 2019-10-13T09:45:33.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:33.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.899962+0000) 2019-10-13T09:45:33.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:33.894+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.900216+0000) 2019-10-13T09:45:33.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:33.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.900443+0000) 2019-10-13T09:45:33.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:33.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:33.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:33.903+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:33.905067+0000) 2019-10-13T09:45:34.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:34.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.008504+0000) 2019-10-13T09:45:34.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:34.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.296940+0000) 2019-10-13T09:45:34.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:34.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.394857+0000) 2019-10-13T09:45:34.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:34.414+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:34.419252+0000) 2019-10-13T09:45:37.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:37.670+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:37.670436+0000) 2019-10-13T09:45:38.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:38.227+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.228915+0000) 2019-10-13T09:45:38.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:38.438+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.441795+0000) 2019-10-13T09:45:38.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:38.898+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.904794+0000) 2019-10-13T09:45:38.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:38.898+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.905205+0000) 2019-10-13T09:45:38.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:38.902+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.906568+0000) 2019-10-13T09:45:38.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:38.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.906600+0000) 2019-10-13T09:45:38.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:38.903+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.907276+0000) 2019-10-13T09:45:38.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:38.902+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.908071+0000) 2019-10-13T09:45:38.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:38.902+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.908920+0000) 2019-10-13T09:45:38.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:38.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:38.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:38.907+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:38.909083+0000) 2019-10-13T09:45:41.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:41.691+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:41.693170+0000) 2019-10-13T09:45:43.913 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:43.906+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.912301+0000) 2019-10-13T09:45:43.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:43.906+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.912881+0000) 2019-10-13T09:45:43.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:43.906+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.913453+0000) 2019-10-13T09:45:43.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:43.910+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.914393+0000) 2019-10-13T09:45:43.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:43.906+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.913262+0000) 2019-10-13T09:45:43.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:43.910+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.915673+0000) 2019-10-13T09:45:43.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:43.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:43.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:43.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.915955+0000) 2019-10-13T09:45:43.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:43.919+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:43.920457+0000) 2019-10-13T09:45:44.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:44.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.008715+0000) 2019-10-13T09:45:44.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:44.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.297147+0000) 2019-10-13T09:45:44.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:44.390+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.395033+0000) 2019-10-13T09:45:44.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:44.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:44.419471+0000) 2019-10-13T09:45:47.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:47.670+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:47.670632+0000) 2019-10-13T09:45:48.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:48.231+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.235434+0000) 2019-10-13T09:45:48.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:48.438+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.442014+0000) 2019-10-13T09:45:48.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:48.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.918530+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:48.915+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.919480+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:48.923+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.924717+0000) 2019-10-13T09:45:49.206 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:48.914+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.919159+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:48.914+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.920493+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:48.914+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.920928+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:48.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.921196+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:48.914+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:48.921699+0000) 2019-10-13T09:45:49.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:48.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:51.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:51.690+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:51.693400+0000) 2019-10-13T09:45:53.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:53.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.925225+0000) 2019-10-13T09:45:53.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:53.922+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.926485+0000) 2019-10-13T09:45:53.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:53.922+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.927474+0000) 2019-10-13T09:45:53.928 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:53.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.927639+0000) 2019-10-13T09:45:53.929 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:53.922+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.928143+0000) 2019-10-13T09:45:53.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:53.922+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.928475+0000) 2019-10-13T09:45:53.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:53.922+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.928790+0000) 2019-10-13T09:45:53.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:53.922+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:53.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:53.930+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:53.935443+0000) 2019-10-13T09:45:54.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:54.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.008895+0000) 2019-10-13T09:45:54.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:54.293+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.297362+0000) 2019-10-13T09:45:54.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:54.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.395209+0000) 2019-10-13T09:45:54.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:54.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:54.419678+0000) 2019-10-13T09:45:57.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:57.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:57.670835+0000) 2019-10-13T09:45:58.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:58.238+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.241909+0000) 2019-10-13T09:45:58.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:58.437+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.442257+0000) 2019-10-13T09:45:58.915 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:45:58.915 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:58.919 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:45:58.919 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:45:58.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:45:58.925+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.933144+0000) 2019-10-13T09:45:58.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:45:58.925+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.933525+0000) 2019-10-13T09:45:58.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:45:58.930+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.933741+0000) 2019-10-13T09:45:58.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:45:58.929+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.934038+0000) 2019-10-13T09:45:58.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:45:58.929+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.934725+0000) 2019-10-13T09:45:58.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:45:58.929+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.934983+0000) 2019-10-13T09:45:58.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:45:58.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:45:58.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:45:58.929+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.936161+0000) 2019-10-13T09:45:58.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:45:58.934+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:45:58.939306+0000) 2019-10-13T09:46:01.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:01.690+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:01.693623+0000) 2019-10-13T09:46:03.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:03.933+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.940096+0000) 2019-10-13T09:46:03.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:03.934+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.940057+0000) 2019-10-13T09:46:03.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:03.933+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.941531+0000) 2019-10-13T09:46:03.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:03.937+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.941767+0000) 2019-10-13T09:46:03.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:03.937+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.942104+0000) 2019-10-13T09:46:03.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:03.937+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.942509+0000) 2019-10-13T09:46:03.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:03.937+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.943480+0000) 2019-10-13T09:46:03.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:03.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:03.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:03.942+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:03.945706+0000) 2019-10-13T09:46:04.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:04.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.009083+0000) 2019-10-13T09:46:04.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:04.289+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.297540+0000) 2019-10-13T09:46:04.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:04.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.395402+0000) 2019-10-13T09:46:04.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:04.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:04.419861+0000) 2019-10-13T09:46:07.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:07.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:07.671027+0000) 2019-10-13T09:46:08.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:08.246+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.248465+0000) 2019-10-13T09:46:08.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:08.441+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.442470+0000) 2019-10-13T09:46:08.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:08.945+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.948339+0000) 2019-10-13T09:46:08.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:08.945+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.949102+0000) 2019-10-13T09:46:08.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:08.945+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.948917+0000) 2019-10-13T09:46:08.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:08.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.949829+0000) 2019-10-13T09:46:08.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:08.949+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.950140+0000) 2019-10-13T09:46:08.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:08.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.950324+0000) 2019-10-13T09:46:08.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:08.949+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.951562+0000) 2019-10-13T09:46:08.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:08.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:08.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:08.954+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:08.956417+0000) 2019-10-13T09:46:11.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:11.690+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:11.693854+0000) 2019-10-13T09:46:13.958 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:13.953+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.957496+0000) 2019-10-13T09:46:13.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:13.954+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.957359+0000) 2019-10-13T09:46:13.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:13.957+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.958325+0000) 2019-10-13T09:46:13.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:13.957+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.958919+0000) 2019-10-13T09:46:13.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:13.957+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.959649+0000) 2019-10-13T09:46:13.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:13.957+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.960128+0000) 2019-10-13T09:46:13.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:13.957+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:13.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:13.957+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.961070+0000) 2019-10-13T09:46:13.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:13.962+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:13.964400+0000) 2019-10-13T09:46:14.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:14.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.009287+0000) 2019-10-13T09:46:14.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:14.293+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.297711+0000) 2019-10-13T09:46:14.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:14.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.395580+0000) 2019-10-13T09:46:14.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:14.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:14.420037+0000) 2019-10-13T09:46:17.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:17.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:17.671259+0000) 2019-10-13T09:46:18.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:18.254+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.259961+0000) 2019-10-13T09:46:18.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:18.441+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.442680+0000) 2019-10-13T09:46:18.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:18.961+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.964080+0000) 2019-10-13T09:46:18.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:18.965+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.965998+0000) 2019-10-13T09:46:18.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:18.965+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.966296+0000) 2019-10-13T09:46:18.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:18.965+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.966757+0000) 2019-10-13T09:46:18.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:18.965+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.967634+0000) 2019-10-13T09:46:18.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:18.962+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.967557+0000) 2019-10-13T09:46:18.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:18.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:18.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:18.965+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.968732+0000) 2019-10-13T09:46:18.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:18.970+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:18.975409+0000) 2019-10-13T09:46:21.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:21.690+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:21.694043+0000) 2019-10-13T09:46:23.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:23.969+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.972046+0000) 2019-10-13T09:46:23.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:23.970+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.972816+0000) 2019-10-13T09:46:23.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:23.969+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.973667+0000) 2019-10-13T09:46:23.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:23.973+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.973934+0000) 2019-10-13T09:46:23.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:23.973+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.974230+0000) 2019-10-13T09:46:23.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:23.973+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.974702+0000) 2019-10-13T09:46:23.976 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:23.973+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.976039+0000) 2019-10-13T09:46:23.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:23.973+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:23.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:23.974+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:23.978297+0000) 2019-10-13T09:46:24.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:24.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.009463+0000) 2019-10-13T09:46:24.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:24.297+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.297936+0000) 2019-10-13T09:46:24.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:24.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.395785+0000) 2019-10-13T09:46:24.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:24.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:24.420214+0000) 2019-10-13T09:46:27.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:27.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:27.671459+0000) 2019-10-13T09:46:28.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:28.270+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.271486+0000) 2019-10-13T09:46:28.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:28.441+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.442877+0000) 2019-10-13T09:46:28.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:28.974+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.979946+0000) 2019-10-13T09:46:28.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:28.981+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.982631+0000) 2019-10-13T09:46:28.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:28.981+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.983050+0000) 2019-10-13T09:46:28.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:28.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.983522+0000) 2019-10-13T09:46:28.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:28.981+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.984026+0000) 2019-10-13T09:46:28.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:28.981+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.984330+0000) 2019-10-13T09:46:28.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:28.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.984775+0000) 2019-10-13T09:46:28.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:28.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:28.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:28.986+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:28.988332+0000) 2019-10-13T09:46:29.044 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:46:29.044 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:29.048 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:46:29.048 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:31.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:31.690+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:31.694273+0000) 2019-10-13T09:46:33.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:33.985+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.988736+0000) 2019-10-13T09:46:33.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:33.985+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.989235+0000) 2019-10-13T09:46:33.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:33.985+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.989542+0000) 2019-10-13T09:46:33.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:33.986+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.989896+0000) 2019-10-13T09:46:33.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:33.989+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.991514+0000) 2019-10-13T09:46:33.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:33.989+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.991908+0000) 2019-10-13T09:46:33.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:33.989+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.992263+0000) 2019-10-13T09:46:33.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:33.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:33.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:33.990+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:33.995887+0000) 2019-10-13T09:46:34.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:34.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.009633+0000) 2019-10-13T09:46:34.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:34.297+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.298109+0000) 2019-10-13T09:46:34.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:34.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.395958+0000) 2019-10-13T09:46:34.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:34.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:34.420407+0000) 2019-10-13T09:46:37.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:37.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:37.671660+0000) 2019-10-13T09:46:38.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:38.274+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.277943+0000) 2019-10-13T09:46:38.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:38.441+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.443137+0000) 2019-10-13T09:46:38.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:38.993+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.997661+0000) 2019-10-13T09:46:38.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:38.997+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.998006+0000) 2019-10-13T09:46:39.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:38.994+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.997893+0000) 2019-10-13T09:46:39.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:38.997+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.998640+0000) 2019-10-13T09:46:39.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:38.997+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.999424+0000) 2019-10-13T09:46:39.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:38.997+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:38.999598+0000) 2019-10-13T09:46:39.001 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:38.997+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.000369+0000) 2019-10-13T09:46:39.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:38.997+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:39.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:39.002+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:39.006648+0000) 2019-10-13T09:46:41.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:41.694+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:41.694492+0000) 2019-10-13T09:46:44.007 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:44.001+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.005709+0000) 2019-10-13T09:46:44.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:44.005+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.006044+0000) 2019-10-13T09:46:44.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:44.005+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.006357+0000) 2019-10-13T09:46:44.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:44.005+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.007251+0000) 2019-10-13T09:46:44.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:44.002+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.007175+0000) 2019-10-13T09:46:44.008 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:44.005+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.008167+0000) 2019-10-13T09:46:44.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:44.005+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.009089+0000) 2019-10-13T09:46:44.010 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:44.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:44.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:44.009+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.009789+0000) 2019-10-13T09:46:44.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:44.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.011190+0000) 2019-10-13T09:46:44.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:44.297+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.298291+0000) 2019-10-13T09:46:44.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:44.393+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.396142+0000) 2019-10-13T09:46:44.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:44.417+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:44.420582+0000) 2019-10-13T09:46:47.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:47.669+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:47.671889+0000) 2019-10-13T09:46:48.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:48.282+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.284800+0000) 2019-10-13T09:46:48.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:48.441+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:48.443364+0000) 2019-10-13T09:46:49.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:49.009+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.012384+0000) 2019-10-13T09:46:49.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:49.009+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.013638+0000) 2019-10-13T09:46:49.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:49.010+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.013859+0000) 2019-10-13T09:46:49.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:49.013+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.014533+0000) 2019-10-13T09:46:49.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:49.013+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.015025+0000) 2019-10-13T09:46:49.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:49.013+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.017077+0000) 2019-10-13T09:46:49.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:49.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:49.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:49.017+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.018177+0000) 2019-10-13T09:46:49.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:49.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:49.021807+0000) 2019-10-13T09:46:51.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:51.694+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:51.694695+0000) 2019-10-13T09:46:54.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:54.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.009987+0000) 2019-10-13T09:46:54.023 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:54.020+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.022692+0000) 2019-10-13T09:46:54.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:54.020+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.022995+0000) 2019-10-13T09:46:54.025 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:54.020+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.024387+0000) 2019-10-13T09:46:54.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:54.020+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.024949+0000) 2019-10-13T09:46:54.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:54.020+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.025397+0000) 2019-10-13T09:46:54.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:54.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:54.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:54.022+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.027635+0000) 2019-10-13T09:46:54.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:54.024+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.029352+0000) 2019-10-13T09:46:54.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:54.026+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.029466+0000) 2019-10-13T09:46:54.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:54.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.298507+0000) 2019-10-13T09:46:54.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:54.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.396338+0000) 2019-10-13T09:46:54.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:54.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:54.420760+0000) 2019-10-13T09:46:57.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:57.668+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:57.672084+0000) 2019-10-13T09:46:58.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:58.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.296142+0000) 2019-10-13T09:46:58.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:58.440+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:58.443559+0000) 2019-10-13T09:46:59.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:46:59.028+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.030073+0000) 2019-10-13T09:46:59.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:46:59.028+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.030475+0000) 2019-10-13T09:46:59.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:46:59.028+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.030962+0000) 2019-10-13T09:46:59.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:46:59.028+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.031706+0000) 2019-10-13T09:46:59.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:46:59.028+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.031955+0000) 2019-10-13T09:46:59.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:46:59.028+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.033465+0000) 2019-10-13T09:46:59.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:46:59.029+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.033010+0000) 2019-10-13T09:46:59.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:46:59.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:46:59.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:46:59.033+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:46:59.038835+0000) 2019-10-13T09:46:59.170 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:46:59.170 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:46:59.174 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:46:59.174 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:01.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:01.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:01.694945+0000) 2019-10-13T09:47:04.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:04.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.010164+0000) 2019-10-13T09:47:04.038 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:04.032+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.037590+0000) 2019-10-13T09:47:04.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:04.036+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.038566+0000) 2019-10-13T09:47:04.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:04.036+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.038873+0000) 2019-10-13T09:47:04.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:04.036+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.039568+0000) 2019-10-13T09:47:04.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:04.036+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.039683+0000) 2019-10-13T09:47:04.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:04.033+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.039720+0000) 2019-10-13T09:47:04.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:04.036+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.040674+0000) 2019-10-13T09:47:04.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:04.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:04.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:04.045+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.048592+0000) 2019-10-13T09:47:04.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:04.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.298711+0000) 2019-10-13T09:47:04.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:04.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.396547+0000) 2019-10-13T09:47:04.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:04.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:04.420955+0000) 2019-10-13T09:47:07.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:07.668+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:07.672295+0000) 2019-10-13T09:47:08.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:08.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.302688+0000) 2019-10-13T09:47:08.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:08.440+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:08.443768+0000) 2019-10-13T09:47:09.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:09.044+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.046552+0000) 2019-10-13T09:47:09.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:09.044+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.047219+0000) 2019-10-13T09:47:09.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:09.044+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.047407+0000) 2019-10-13T09:47:09.049 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:09.041+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.047372+0000) 2019-10-13T09:47:09.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:09.044+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.048262+0000) 2019-10-13T09:47:09.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:09.044+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.048764+0000) 2019-10-13T09:47:09.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:09.048+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.049921+0000) 2019-10-13T09:47:09.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:09.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:09.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:09.049+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:09.054978+0000) 2019-10-13T09:47:11.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:11.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:11.695147+0000) 2019-10-13T09:47:14.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:14.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.010340+0000) 2019-10-13T09:47:14.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:14.052+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.054400+0000) 2019-10-13T09:47:14.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:14.052+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.055524+0000) 2019-10-13T09:47:14.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:14.052+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.055768+0000) 2019-10-13T09:47:14.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:14.056+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.058073+0000) 2019-10-13T09:47:14.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:14.053+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.057470+0000) 2019-10-13T09:47:14.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:14.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102226 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:14.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:14.056+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.059289+0000) 2019-10-13T09:47:14.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:14.056+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.060554+0000) 2019-10-13T09:47:14.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:14.061+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.064329+0000) 2019-10-13T09:47:14.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:14.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.298920+0000) 2019-10-13T09:47:14.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:14.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.396719+0000) 2019-10-13T09:47:14.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:14.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:14.421138+0000) 2019-10-13T09:47:17.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:17.668+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:17.672495+0000) 2019-10-13T09:47:18.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:18.305+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.309235+0000) 2019-10-13T09:47:18.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:18.440+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:18.443993+0000) 2019-10-13T09:47:19.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:19.060+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.061866+0000) 2019-10-13T09:47:19.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:19.060+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.062564+0000) 2019-10-13T09:47:19.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:19.060+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.063608+0000) 2019-10-13T09:47:19.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:19.060+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.064492+0000) 2019-10-13T09:47:19.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:19.060+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.064773+0000) 2019-10-13T09:47:19.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:19.061+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.064553+0000) 2019-10-13T09:47:19.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:19.060+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.064682+0000) 2019-10-13T09:47:19.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:19.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:19.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:19.065+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:19.069405+0000) 2019-10-13T09:47:21.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:21.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:21.695403+0000) 2019-10-13T09:47:24.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:24.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.010530+0000) 2019-10-13T09:47:24.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:24.068+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.070567+0000) 2019-10-13T09:47:24.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:24.068+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.070947+0000) 2019-10-13T09:47:24.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:24.068+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.071433+0000) 2019-10-13T09:47:24.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:24.068+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.072065+0000) 2019-10-13T09:47:24.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:24.068+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.072518+0000) 2019-10-13T09:47:24.073 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:24.069+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.072603+0000) 2019-10-13T09:47:24.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:24.068+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.073469+0000) 2019-10-13T09:47:24.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:24.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:24.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:24.073+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.078724+0000) 2019-10-13T09:47:24.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:24.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.299073+0000) 2019-10-13T09:47:24.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:24.396+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.396898+0000) 2019-10-13T09:47:24.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:24.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:24.421313+0000) 2019-10-13T09:47:27.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:27.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:27.672695+0000) 2019-10-13T09:47:28.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:28.313+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.315771+0000) 2019-10-13T09:47:28.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:28.440+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:28.444171+0000) 2019-10-13T09:47:29.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:29.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.078831+0000) 2019-10-13T09:47:29.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:29.076+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.081230+0000) 2019-10-13T09:47:29.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:29.076+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.081583+0000) 2019-10-13T09:47:29.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:29.080+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.082485+0000) 2019-10-13T09:47:29.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:29.080+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.082998+0000) 2019-10-13T09:47:29.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:29.080+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.083285+0000) 2019-10-13T09:47:29.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:29.080+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:29.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:29.080+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.084746+0000) 2019-10-13T09:47:29.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:29.081+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:29.087630+0000) 2019-10-13T09:47:29.217 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:47:29.217 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:29.221 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:47:29.221 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:31.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:31.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:31.695626+0000) 2019-10-13T09:47:34.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:34.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.010733+0000) 2019-10-13T09:47:34.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:34.084+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.088229+0000) 2019-10-13T09:47:34.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:34.088+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.090991+0000) 2019-10-13T09:47:34.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:34.085+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.090785+0000) 2019-10-13T09:47:34.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:34.088+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.091568+0000) 2019-10-13T09:47:34.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:34.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:34.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:34.088+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.092901+0000) 2019-10-13T09:47:34.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:34.088+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.093604+0000) 2019-10-13T09:47:34.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:34.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.094027+0000) 2019-10-13T09:47:34.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:34.093+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.097963+0000) 2019-10-13T09:47:34.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:34.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.299270+0000) 2019-10-13T09:47:34.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:34.396+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.397150+0000) 2019-10-13T09:47:34.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:34.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:34.421484+0000) 2019-10-13T09:47:37.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:37.668+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:37.672895+0000) 2019-10-13T09:47:38.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:38.321+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.322276+0000) 2019-10-13T09:47:38.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:38.444+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:38.444398+0000) 2019-10-13T09:47:39.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:39.092+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.095074+0000) 2019-10-13T09:47:39.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:39.092+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.095403+0000) 2019-10-13T09:47:39.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:39.089+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.095676+0000) 2019-10-13T09:47:39.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:39.092+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.096913+0000) 2019-10-13T09:47:39.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:39.092+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.097286+0000) 2019-10-13T09:47:39.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:39.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.097651+0000) 2019-10-13T09:47:39.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:39.096+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.098708+0000) 2019-10-13T09:47:39.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:39.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:39.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:39.101+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:39.105140+0000) 2019-10-13T09:47:41.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:41.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:41.695877+0000) 2019-10-13T09:47:44.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:44.008+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.010932+0000) 2019-10-13T09:47:44.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:44.100+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.103353+0000) 2019-10-13T09:47:44.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:44.100+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.105121+0000) 2019-10-13T09:47:44.106 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:44.100+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.105420+0000) 2019-10-13T09:47:44.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:44.104+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.106239+0000) 2019-10-13T09:47:44.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:44.104+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.106741+0000) 2019-10-13T09:47:44.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:44.101+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.106602+0000) 2019-10-13T09:47:44.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:44.104+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.108447+0000) 2019-10-13T09:47:44.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:44.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:44.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:44.109+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.113422+0000) 2019-10-13T09:47:44.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:44.296+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.299464+0000) 2019-10-13T09:47:44.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:44.392+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.397303+0000) 2019-10-13T09:47:44.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:44.416+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:44.421660+0000) 2019-10-13T09:47:47.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:47.667+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:47.673105+0000) 2019-10-13T09:47:48.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:48.325+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.328767+0000) 2019-10-13T09:47:48.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:48.443+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:48.444593+0000) 2019-10-13T09:47:49.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:49.107+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.112501+0000) 2019-10-13T09:47:49.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:49.109+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.112271+0000) 2019-10-13T09:47:49.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:49.107+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.113202+0000) 2019-10-13T09:47:49.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:49.107+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.113618+0000) 2019-10-13T09:47:49.115 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:49.111+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.115162+0000) 2019-10-13T09:47:49.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:49.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:49.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:49.111+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.117602+0000) 2019-10-13T09:47:49.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:49.115+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.118037+0000) 2019-10-13T09:47:49.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:49.117+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:49.121365+0000) 2019-10-13T09:47:51.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:51.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:51.696072+0000) 2019-10-13T09:47:54.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:54.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.011139+0000) 2019-10-13T09:47:54.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:54.115+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.120181+0000) 2019-10-13T09:47:54.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:54.115+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.120659+0000) 2019-10-13T09:47:54.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:54.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.120502+0000) 2019-10-13T09:47:54.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:54.115+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.121376+0000) 2019-10-13T09:47:54.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:54.119+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.122701+0000) 2019-10-13T09:47:54.124 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:54.119+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.123393+0000) 2019-10-13T09:47:54.124 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:54.119+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.123647+0000) 2019-10-13T09:47:54.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:54.119+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:54.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:54.121+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.127787+0000) 2019-10-13T09:47:54.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:54.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.299639+0000) 2019-10-13T09:47:54.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:54.395+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.397497+0000) 2019-10-13T09:47:54.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:54.419+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:54.421850+0000) 2019-10-13T09:47:57.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:57.671+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:57.673265+0000) 2019-10-13T09:47:58.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:58.333+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.335286+0000) 2019-10-13T09:47:58.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:58.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:58.444788+0000) 2019-10-13T09:47:59.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:47:59.123+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.128374+0000) 2019-10-13T09:47:59.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:47:59.123+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.128685+0000) 2019-10-13T09:47:59.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:47:59.127+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.130786+0000) 2019-10-13T09:47:59.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:47:59.125+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.130571+0000) 2019-10-13T09:47:59.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:47:59.127+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.131484+0000) 2019-10-13T09:47:59.132 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:47:59.127+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.131658+0000) 2019-10-13T09:47:59.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:47:59.127+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.132003+0000) 2019-10-13T09:47:59.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:47:59.127+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:47:59.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:47:59.133+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:47:59.136324+0000) 2019-10-13T09:47:59.343 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:47:59.343 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:47:59.347 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:47:59.347 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:01.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:01.693+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:01.696329+0000) 2019-10-13T09:48:04.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:04.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.011345+0000) 2019-10-13T09:48:04.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:04.135+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.138015+0000) 2019-10-13T09:48:04.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:04.135+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.138599+0000) 2019-10-13T09:48:04.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:04.135+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.138996+0000) 2019-10-13T09:48:04.141 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:04.135+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.140040+0000) 2019-10-13T09:48:04.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:04.135+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.140980+0000) 2019-10-13T09:48:04.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:04.136+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.140694+0000) 2019-10-13T09:48:04.143 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:04.135+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.141417+0000) 2019-10-13T09:48:04.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:04.139+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:04.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:04.140+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.146532+0000) 2019-10-13T09:48:04.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:04.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.299771+0000) 2019-10-13T09:48:04.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:04.391+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.397694+0000) 2019-10-13T09:48:04.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:04.419+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:04.421996+0000) 2019-10-13T09:48:07.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:07.671+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:07.673499+0000) 2019-10-13T09:48:08.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:08.344+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.346698+0000) 2019-10-13T09:48:08.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:08.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:08.444986+0000) 2019-10-13T09:48:09.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:09.139+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.145294+0000) 2019-10-13T09:48:09.148 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:09.143+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.147195+0000) 2019-10-13T09:48:09.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:09.140+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.146091+0000) 2019-10-13T09:48:09.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:09.143+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.147635+0000) 2019-10-13T09:48:09.150 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:09.143+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.148270+0000) 2019-10-13T09:48:09.151 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:09.143+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.148682+0000) 2019-10-13T09:48:09.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:09.143+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.149276+0000) 2019-10-13T09:48:09.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:09.147+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:09.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:09.148+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:09.155274+0000) 2019-10-13T09:48:11.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:11.692+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:11.696530+0000) 2019-10-13T09:48:14.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:14.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.011543+0000) 2019-10-13T09:48:14.155 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:14.151+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.154329+0000) 2019-10-13T09:48:14.156 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:14.148+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.154533+0000) 2019-10-13T09:48:14.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:14.151+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.156787+0000) 2019-10-13T09:48:14.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:14.151+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.156991+0000) 2019-10-13T09:48:14.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:14.151+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.157337+0000) 2019-10-13T09:48:14.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:14.155+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.158071+0000) 2019-10-13T09:48:14.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:14.155+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:14.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:14.155+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.159581+0000) 2019-10-13T09:48:14.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:14.160+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.165232+0000) 2019-10-13T09:48:14.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:14.299+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.299954+0000) 2019-10-13T09:48:14.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:14.395+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.397867+0000) 2019-10-13T09:48:14.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:14.419+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:14.422182+0000) 2019-10-13T09:48:17.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:17.671+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:17.673690+0000) 2019-10-13T09:48:18.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:18.356+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.357794+0000) 2019-10-13T09:48:18.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:18.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:18.445230+0000) 2019-10-13T09:48:19.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:19.155+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.161648+0000) 2019-10-13T09:48:19.163 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:19.159+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.162092+0000) 2019-10-13T09:48:19.164 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:19.159+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.163501+0000) 2019-10-13T09:48:19.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:19.156+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.163774+0000) 2019-10-13T09:48:19.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:19.159+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.164398+0000) 2019-10-13T09:48:19.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:19.159+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.164720+0000) 2019-10-13T09:48:19.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:19.159+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.165330+0000) 2019-10-13T09:48:19.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:19.163+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:19.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:19.164+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:19.168302+0000) 2019-10-13T09:48:21.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:21.692+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:21.696741+0000) 2019-10-13T09:48:24.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:24.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.011785+0000) 2019-10-13T09:48:24.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:24.167+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.171252+0000) 2019-10-13T09:48:24.173 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:24.167+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.172618+0000) 2019-10-13T09:48:24.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:24.167+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.173275+0000) 2019-10-13T09:48:24.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:24.168+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.173080+0000) 2019-10-13T09:48:24.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:24.171+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.174213+0000) 2019-10-13T09:48:24.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:24.171+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.174348+0000) 2019-10-13T09:48:24.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:24.171+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.174835+0000) 2019-10-13T09:48:24.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:24.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102637 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:24.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:24.172+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.179628+0000) 2019-10-13T09:48:24.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:24.299+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.300136+0000) 2019-10-13T09:48:24.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:24.395+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.398053+0000) 2019-10-13T09:48:24.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:24.419+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:24.422362+0000) 2019-10-13T09:48:27.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:27.671+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:27.673948+0000) 2019-10-13T09:48:28.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:28.368+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.369038+0000) 2019-10-13T09:48:28.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:28.439+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:28.445428+0000) 2019-10-13T09:48:29.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:29.175+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.178851+0000) 2019-10-13T09:48:29.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:29.175+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.179101+0000) 2019-10-13T09:48:29.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:29.176+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.181007+0000) 2019-10-13T09:48:29.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:29.179+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.182017+0000) 2019-10-13T09:48:29.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:29.179+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.182109+0000) 2019-10-13T09:48:29.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:29.179+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.184172+0000) 2019-10-13T09:48:29.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:29.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:29.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:29.179+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.184865+0000) 2019-10-13T09:48:29.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:29.180+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:29.187991+0000) 2019-10-13T09:48:29.449 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:48:29.449 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:29.453 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:48:29.453 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:31.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:31.696+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:31.696933+0000) 2019-10-13T09:48:34.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:34.007+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.011999+0000) 2019-10-13T09:48:34.188 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:34.183+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.187582+0000) 2019-10-13T09:48:34.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:34.183+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.188098+0000) 2019-10-13T09:48:34.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:34.183+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.189455+0000) 2019-10-13T09:48:34.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:34.187+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.189819+0000) 2019-10-13T09:48:34.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:34.184+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.189864+0000) 2019-10-13T09:48:34.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:34.187+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.190985+0000) 2019-10-13T09:48:34.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:34.187+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.191652+0000) 2019-10-13T09:48:34.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:34.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:34.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:34.192+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.196418+0000) 2019-10-13T09:48:34.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:34.295+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.300338+0000) 2019-10-13T09:48:34.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:34.395+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.398235+0000) 2019-10-13T09:48:34.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:34.419+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:34.422559+0000) 2019-10-13T09:48:37.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:37.671+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:37.674175+0000) 2019-10-13T09:48:38.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:38.372+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.375671+0000) 2019-10-13T09:48:38.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:38.443+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:38.445637+0000) 2019-10-13T09:48:39.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:39.191+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.195191+0000) 2019-10-13T09:48:39.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:39.191+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.196782+0000) 2019-10-13T09:48:39.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:39.191+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.197197+0000) 2019-10-13T09:48:39.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:39.191+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.197624+0000) 2019-10-13T09:48:39.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:39.192+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.197495+0000) 2019-10-13T09:48:39.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:39.195+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102728 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:39.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:39.195+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.199387+0000) 2019-10-13T09:48:39.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:39.195+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.199984+0000) 2019-10-13T09:48:39.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:39.200+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:39.204977+0000) 2019-10-13T09:48:41.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:41.696+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:41.697171+0000) 2019-10-13T09:48:44.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:44.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.012199+0000) 2019-10-13T09:48:44.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:44.198+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.202566+0000) 2019-10-13T09:48:44.204 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:44.198+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.203297+0000) 2019-10-13T09:48:44.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:44.200+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.204833+0000) 2019-10-13T09:48:44.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:44.202+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.205984+0000) 2019-10-13T09:48:44.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:44.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:44.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:44.202+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.208218+0000) 2019-10-13T09:48:44.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:44.202+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.208461+0000) 2019-10-13T09:48:44.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:44.202+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.208814+0000) 2019-10-13T09:48:44.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:44.208+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.213171+0000) 2019-10-13T09:48:44.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:44.298+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.300554+0000) 2019-10-13T09:48:44.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:44.394+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.398406+0000) 2019-10-13T09:48:44.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:44.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:44.422782+0000) 2019-10-13T09:48:47.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:47.670+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:47.674390+0000) 2019-10-13T09:48:48.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:48.380+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.382239+0000) 2019-10-13T09:48:48.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:48.442+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:48.445829+0000) 2019-10-13T09:48:49.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:49.206+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.210967+0000) 2019-10-13T09:48:49.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:49.206+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.211147+0000) 2019-10-13T09:48:49.213 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:49.204+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.211472+0000) 2019-10-13T09:48:49.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:49.206+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.212284+0000) 2019-10-13T09:48:49.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:49.206+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.212552+0000) 2019-10-13T09:48:49.214 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:49.206+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.213098+0000) 2019-10-13T09:48:49.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:49.210+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.213921+0000) 2019-10-13T09:48:49.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:49.210+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:49.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:49.212+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:49.220008+0000) 2019-10-13T09:48:51.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:51.696+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:51.697365+0000) 2019-10-13T09:48:54.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:54.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.012395+0000) 2019-10-13T09:48:54.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:54.212+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.219579+0000) 2019-10-13T09:48:54.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:54.214+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.219495+0000) 2019-10-13T09:48:54.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:54.214+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.220461+0000) 2019-10-13T09:48:54.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:54.214+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.220806+0000) 2019-10-13T09:48:54.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:54.214+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.221026+0000) 2019-10-13T09:48:54.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:54.218+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.221802+0000) 2019-10-13T09:48:54.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:54.218+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.222860+0000) 2019-10-13T09:48:54.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:54.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:54.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:54.220+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.226788+0000) 2019-10-13T09:48:54.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:54.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.300673+0000) 2019-10-13T09:48:54.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:54.394+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.398593+0000) 2019-10-13T09:48:54.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:54.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:54.422960+0000) 2019-10-13T09:48:57.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:57.670+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:57.674616+0000) 2019-10-13T09:48:58.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:58.388+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.388790+0000) 2019-10-13T09:48:58.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:58.442+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:58.446028+0000) 2019-10-13T09:48:59.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:48:59.222+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.227206+0000) 2019-10-13T09:48:59.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:48:59.222+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.227762+0000) 2019-10-13T09:48:59.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:48:59.222+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.228141+0000) 2019-10-13T09:48:59.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:48:59.222+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.228545+0000) 2019-10-13T09:48:59.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:48:59.224+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.228884+0000) 2019-10-13T09:48:59.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:48:59.226+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.229824+0000) 2019-10-13T09:48:59.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:48:59.226+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.231533+0000) 2019-10-13T09:48:59.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:48:59.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:48:59.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:48:59.232+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:48:59.236282+0000) 2019-10-13T09:48:59.559 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:48:59.560 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:48:59.563 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:48:59.563 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:01.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:01.696+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:01.697566+0000) 2019-10-13T09:49:04.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:04.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.012596+0000) 2019-10-13T09:49:04.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:04.230+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.235403+0000) 2019-10-13T09:49:04.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:04.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.235395+0000) 2019-10-13T09:49:04.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:04.230+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.236786+0000) 2019-10-13T09:49:04.238 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:04.230+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.237006+0000) 2019-10-13T09:49:04.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:04.230+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.237412+0000) 2019-10-13T09:49:04.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:04.234+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.237875+0000) 2019-10-13T09:49:04.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:04.234+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.238463+0000) 2019-10-13T09:49:04.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:04.234+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102875 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:04.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:04.244+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.244443+0000) 2019-10-13T09:49:04.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:04.298+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.300823+0000) 2019-10-13T09:49:04.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:04.394+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.398765+0000) 2019-10-13T09:49:04.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:04.422+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:04.423142+0000) 2019-10-13T09:49:07.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:07.674+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:07.674821+0000) 2019-10-13T09:49:08.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:08.391+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.395412+0000) 2019-10-13T09:49:08.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:08.442+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:08.446258+0000) 2019-10-13T09:49:09.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:09.238+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.242808+0000) 2019-10-13T09:49:09.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:09.238+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.243450+0000) 2019-10-13T09:49:09.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:09.238+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.244302+0000) 2019-10-13T09:49:09.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:09.238+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.244648+0000) 2019-10-13T09:49:09.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:09.243+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.245068+0000) 2019-10-13T09:49:09.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:09.242+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.248393+0000) 2019-10-13T09:49:09.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:09.242+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:09.250 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:09.242+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.248947+0000) 2019-10-13T09:49:09.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:09.251+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:09.253417+0000) 2019-10-13T09:49:11.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:11.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:11.697768+0000) 2019-10-13T09:49:14.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:14.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.012825+0000) 2019-10-13T09:49:14.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:14.246+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.251695+0000) 2019-10-13T09:49:14.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:14.251+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.252804+0000) 2019-10-13T09:49:14.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:14.250+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.253792+0000) 2019-10-13T09:49:14.255 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:14.250+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.254115+0000) 2019-10-13T09:49:14.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:14.250+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.254806+0000) 2019-10-13T09:49:14.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:14.250+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.255257+0000) 2019-10-13T09:49:14.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:14.250+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.256793+0000) 2019-10-13T09:49:14.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:14.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:14.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:14.255+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.259855+0000) 2019-10-13T09:49:14.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:14.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.300971+0000) 2019-10-13T09:49:14.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:14.398+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.398946+0000) 2019-10-13T09:49:14.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:14.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:14.423323+0000) 2019-10-13T09:49:17.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:17.674+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:17.675028+0000) 2019-10-13T09:49:18.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:18.399+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.401925+0000) 2019-10-13T09:49:18.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:18.442+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:18.446439+0000) 2019-10-13T09:49:19.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:19.254+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.261182+0000) 2019-10-13T09:49:19.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:19.254+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.261447+0000) 2019-10-13T09:49:19.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:19.258+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.261844+0000) 2019-10-13T09:49:19.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:19.258+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.262567+0000) 2019-10-13T09:49:19.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:19.259+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.262889+0000) 2019-10-13T09:49:19.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:19.258+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.264168+0000) 2019-10-13T09:49:19.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:19.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.264639+0000) 2019-10-13T09:49:19.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:19.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:19.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:19.267+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:19.268139+0000) 2019-10-13T09:49:21.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:21.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:21.697997+0000) 2019-10-13T09:49:24.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:24.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.013069+0000) 2019-10-13T09:49:24.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:24.266+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.270088+0000) 2019-10-13T09:49:24.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:24.266+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.270411+0000) 2019-10-13T09:49:24.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:24.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.270992+0000) 2019-10-13T09:49:24.272 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:24.266+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.271184+0000) 2019-10-13T09:49:24.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:24.266+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.272344+0000) 2019-10-13T09:49:24.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:24.271+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.272255+0000) 2019-10-13T09:49:24.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:24.266+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.273323+0000) 2019-10-13T09:49:24.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:24.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:24.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:24.271+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.275686+0000) 2019-10-13T09:49:24.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:24.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.301121+0000) 2019-10-13T09:49:24.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:24.394+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.399128+0000) 2019-10-13T09:49:24.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:24.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:24.423504+0000) 2019-10-13T09:49:27.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:27.670+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:27.675255+0000) 2019-10-13T09:49:28.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:28.407+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.408388+0000) 2019-10-13T09:49:28.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:28.446+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:28.446594+0000) 2019-10-13T09:49:29.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:29.275+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.276854+0000) 2019-10-13T09:49:29.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:29.274+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.278030+0000) 2019-10-13T09:49:29.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:29.274+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.278758+0000) 2019-10-13T09:49:29.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:29.274+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.279077+0000) 2019-10-13T09:49:29.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:29.274+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.279367+0000) 2019-10-13T09:49:29.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:29.274+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.280074+0000) 2019-10-13T09:49:29.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:29.274+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.280587+0000) 2019-10-13T09:49:29.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:29.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:29.288 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:29.283+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:29.287651+0000) 2019-10-13T09:49:29.668 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:49:29.669 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:29.672 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:49:29.673 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:31.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:31.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:31.698206+0000) 2019-10-13T09:49:34.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:34.006+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.013269+0000) 2019-10-13T09:49:34.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:34.278+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.285350+0000) 2019-10-13T09:49:34.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:34.283+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.285196+0000) 2019-10-13T09:49:34.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:34.282+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.286437+0000) 2019-10-13T09:49:34.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:34.282+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.287537+0000) 2019-10-13T09:49:34.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:34.282+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.287962+0000) 2019-10-13T09:49:34.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:34.282+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.288651+0000) 2019-10-13T09:49:34.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:34.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:34.291 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:34.282+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.289679+0000) 2019-10-13T09:49:34.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:34.291+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.295655+0000) 2019-10-13T09:49:34.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:34.294+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.301272+0000) 2019-10-13T09:49:34.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:34.398+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.399305+0000) 2019-10-13T09:49:34.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:34.418+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:34.423714+0000) 2019-10-13T09:49:37.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:37.674+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:37.675457+0000) 2019-10-13T09:49:38.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:38.411+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.415005+0000) 2019-10-13T09:49:38.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:38.446+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:38.446834+0000) 2019-10-13T09:49:39.294 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:39.285+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.293707+0000) 2019-10-13T09:49:39.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:39.289+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.294352+0000) 2019-10-13T09:49:39.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:39.291+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.295204+0000) 2019-10-13T09:49:39.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:39.289+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.295830+0000) 2019-10-13T09:49:39.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:39.289+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.296249+0000) 2019-10-13T09:49:39.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:39.289+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.295896+0000) 2019-10-13T09:49:39.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:39.289+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.296858+0000) 2019-10-13T09:49:39.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:39.289+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:39.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:39.295+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:39.300035+0000) 2019-10-13T09:49:41.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:41.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:41.698408+0000) 2019-10-13T09:49:44.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:44.005+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.013505+0000) 2019-10-13T09:49:44.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:44.293+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.300949+0000) 2019-10-13T09:49:44.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:44.293+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.301410+0000) 2019-10-13T09:49:44.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:44.293+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.301293+0000) 2019-10-13T09:49:44.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:44.297+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.302491+0000) 2019-10-13T09:49:44.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:44.297+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.302254+0000) 2019-10-13T09:49:44.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:44.297+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.303585+0000) 2019-10-13T09:49:44.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:44.299+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.303420+0000) 2019-10-13T09:49:44.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:44.297+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.304472+0000) 2019-10-13T09:49:44.305 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:44.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:44.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:44.307+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.310631+0000) 2019-10-13T09:49:44.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:44.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.399483+0000) 2019-10-13T09:49:44.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:44.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:44.423906+0000) 2019-10-13T09:49:47.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:47.673+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:47.675682+0000) 2019-10-13T09:49:48.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:48.419+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.421538+0000) 2019-10-13T09:49:48.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:48.445+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:48.447054+0000) 2019-10-13T09:49:49.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:49.301+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.308636+0000) 2019-10-13T09:49:49.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:49.305+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.309850+0000) 2019-10-13T09:49:49.311 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:49.307+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.309835+0000) 2019-10-13T09:49:49.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:49.305+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.312452+0000) 2019-10-13T09:49:49.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:49.305+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103147 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:49.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:49.305+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.313112+0000) 2019-10-13T09:49:49.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:49.305+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.313399+0000) 2019-10-13T09:49:49.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:49.309+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.314212+0000) 2019-10-13T09:49:49.319 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:49.315+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:49.317782+0000) 2019-10-13T09:49:51.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:51.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:51.698604+0000) 2019-10-13T09:49:54.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:54.009+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.013700+0000) 2019-10-13T09:49:54.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:54.297+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.301636+0000) 2019-10-13T09:49:54.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:54.313+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.316456+0000) 2019-10-13T09:49:54.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:54.313+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.316922+0000) 2019-10-13T09:49:54.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:54.313+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.317567+0000) 2019-10-13T09:49:54.318 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:54.317+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.317880+0000) 2019-10-13T09:49:54.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:54.317+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.318409+0000) 2019-10-13T09:49:54.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:54.317+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.319006+0000) 2019-10-13T09:49:54.320 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:54.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.318992+0000) 2019-10-13T09:49:54.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:54.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:54.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:54.323+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.326887+0000) 2019-10-13T09:49:54.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:54.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.399662+0000) 2019-10-13T09:49:54.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:54.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:54.424084+0000) 2019-10-13T09:49:57.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:57.673+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:57.675891+0000) 2019-10-13T09:49:58.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:58.423+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.428001+0000) 2019-10-13T09:49:58.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:58.445+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:58.447256+0000) 2019-10-13T09:49:59.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:49:59.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.324636+0000) 2019-10-13T09:49:59.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:49:59.321+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.325278+0000) 2019-10-13T09:49:59.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:49:59.321+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.325606+0000) 2019-10-13T09:49:59.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:49:59.325+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.327180+0000) 2019-10-13T09:49:59.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:49:59.325+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.327380+0000) 2019-10-13T09:49:59.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:49:59.325+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.327769+0000) 2019-10-13T09:49:59.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:49:59.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:49:59.331 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:49:59.327+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.329749+0000) 2019-10-13T09:49:59.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:49:59.331+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:49:59.335261+0000) 2019-10-13T09:49:59.788 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:49:59.789 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:49:59.792 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:49:59.792 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:01.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:01.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:01.698802+0000) 2019-10-13T09:50:04.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:04.013+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.013935+0000) 2019-10-13T09:50:04.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:04.301+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.301839+0000) 2019-10-13T09:50:04.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:04.329+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.332742+0000) 2019-10-13T09:50:04.334 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:04.329+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.333661+0000) 2019-10-13T09:50:04.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:04.333+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.334504+0000) 2019-10-13T09:50:04.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:04.333+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.334875+0000) 2019-10-13T09:50:04.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:04.333+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.335227+0000) 2019-10-13T09:50:04.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:04.333+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.336470+0000) 2019-10-13T09:50:04.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:04.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.337068+0000) 2019-10-13T09:50:04.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:04.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:04.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:04.339+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.342310+0000) 2019-10-13T09:50:04.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:04.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.399878+0000) 2019-10-13T09:50:04.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:04.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:04.424260+0000) 2019-10-13T09:50:07.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:07.673+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:07.676132+0000) 2019-10-13T09:50:08.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:08.431+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.435115+0000) 2019-10-13T09:50:08.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:08.445+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:08.447447+0000) 2019-10-13T09:50:09.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:09.341+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.341798+0000) 2019-10-13T09:50:09.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:09.341+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.342066+0000) 2019-10-13T09:50:09.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:09.341+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.342880+0000) 2019-10-13T09:50:09.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:09.341+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.343304+0000) 2019-10-13T09:50:09.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:09.339+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.343504+0000) 2019-10-13T09:50:09.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:09.341+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.344236+0000) 2019-10-13T09:50:09.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:09.341+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.345049+0000) 2019-10-13T09:50:09.346 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:09.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:09.352 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:09.347+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:09.351090+0000) 2019-10-13T09:50:11.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:11.695+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:11.699029+0000) 2019-10-13T09:50:14.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:14.013+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.014134+0000) 2019-10-13T09:50:14.302 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:14.301+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.301992+0000) 2019-10-13T09:50:14.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:14.345+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.349593+0000) 2019-10-13T09:50:14.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:14.349+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.350162+0000) 2019-10-13T09:50:14.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:14.349+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.350734+0000) 2019-10-13T09:50:14.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:14.346+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.351966+0000) 2019-10-13T09:50:14.353 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:14.349+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.352931+0000) 2019-10-13T09:50:14.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:14.349+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.352663+0000) 2019-10-13T09:50:14.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:14.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:14.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:14.353+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.354258+0000) 2019-10-13T09:50:14.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:14.354+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.359572+0000) 2019-10-13T09:50:14.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:14.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.400047+0000) 2019-10-13T09:50:14.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:14.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:14.424442+0000) 2019-10-13T09:50:17.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:17.673+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:17.676315+0000) 2019-10-13T09:50:18.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:18.438+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.441733+0000) 2019-10-13T09:50:18.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:18.445+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:18.447707+0000) 2019-10-13T09:50:19.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:19.357+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.357988+0000) 2019-10-13T09:50:19.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:19.357+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.358750+0000) 2019-10-13T09:50:19.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:19.357+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.358882+0000) 2019-10-13T09:50:19.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:19.357+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.360256+0000) 2019-10-13T09:50:19.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:19.358+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.360314+0000) 2019-10-13T09:50:19.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:19.357+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.360547+0000) 2019-10-13T09:50:19.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:19.357+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.361292+0000) 2019-10-13T09:50:19.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:19.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:19.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:19.362+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:19.367952+0000) 2019-10-13T09:50:21.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:21.698+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:21.699285+0000) 2019-10-13T09:50:24.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:24.013+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.014371+0000) 2019-10-13T09:50:24.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:24.301+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.302213+0000) 2019-10-13T09:50:24.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:24.361+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.365607+0000) 2019-10-13T09:50:24.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:24.365+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.367747+0000) 2019-10-13T09:50:24.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:24.365+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.368479+0000) 2019-10-13T09:50:24.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:24.365+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.368808+0000) 2019-10-13T09:50:24.370 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:24.365+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.369479+0000) 2019-10-13T09:50:24.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:24.369+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.369969+0000) 2019-10-13T09:50:24.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:24.366+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.369778+0000) 2019-10-13T09:50:24.371 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:24.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:24.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:24.374+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.376521+0000) 2019-10-13T09:50:24.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:24.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.400236+0000) 2019-10-13T09:50:24.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:24.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:24.424637+0000) 2019-10-13T09:50:27.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:27.673+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:27.676537+0000) 2019-10-13T09:50:28.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:28.445+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.447903+0000) 2019-10-13T09:50:28.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:28.446+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:28.448180+0000) 2019-10-13T09:50:29.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:29.373+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.373933+0000) 2019-10-13T09:50:29.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:29.373+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.374946+0000) 2019-10-13T09:50:29.377 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:29.373+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.376641+0000) 2019-10-13T09:50:29.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:29.374+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.376753+0000) 2019-10-13T09:50:29.379 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:29.377+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.378139+0000) 2019-10-13T09:50:29.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:29.377+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.378753+0000) 2019-10-13T09:50:29.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:29.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:29.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:29.377+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.379430+0000) 2019-10-13T09:50:29.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:29.378+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:29.383433+0000) 2019-10-13T09:50:29.895 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:50:29.895 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:29.900 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:50:29.900 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:50:31.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:31.694+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:31.699479+0000) 2019-10-13T09:50:34.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:34.013+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.014593+0000) 2019-10-13T09:50:34.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:34.301+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.302418+0000) 2019-10-13T09:50:34.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:34.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.381299+0000) 2019-10-13T09:50:34.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:34.381+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.382874+0000) 2019-10-13T09:50:34.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:34.381+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.383296+0000) 2019-10-13T09:50:34.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:34.381+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.384009+0000) 2019-10-13T09:50:34.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:34.381+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.384287+0000) 2019-10-13T09:50:34.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:34.381+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.385260+0000) 2019-10-13T09:50:34.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:34.381+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.385442+0000) 2019-10-13T09:50:34.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:34.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:34.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:34.386+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.389451+0000) 2019-10-13T09:50:34.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:34.397+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.400445+0000) 2019-10-13T09:50:34.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:34.421+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:34.424789+0000) 2019-10-13T09:50:37.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:37.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:37.676736+0000) 2019-10-13T09:50:38.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:38.444+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.448108+0000) 2019-10-13T09:50:38.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:38.454+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:38.454717+0000) 2019-10-13T09:50:39.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:39.388+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.390516+0000) 2019-10-13T09:50:39.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:39.388+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.391427+0000) 2019-10-13T09:50:39.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:39.388+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.391982+0000) 2019-10-13T09:50:39.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:39.388+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.392486+0000) 2019-10-13T09:50:39.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:39.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.392612+0000) 2019-10-13T09:50:39.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:39.388+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.393331+0000) 2019-10-13T09:50:39.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:39.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.394894+0000) 2019-10-13T09:50:39.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:39.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:39.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:39.394+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:39.397861+0000) 2019-10-13T09:50:41.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:41.694+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:41.699714+0000) 2019-10-13T09:50:44.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:44.012+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.014814+0000) 2019-10-13T09:50:44.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:44.300+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.302630+0000) 2019-10-13T09:50:44.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:44.396+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.398744+0000) 2019-10-13T09:50:44.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:44.396+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.399559+0000) 2019-10-13T09:50:44.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:44.396+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.401496+0000) 2019-10-13T09:50:44.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:44.398+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.402298+0000) 2019-10-13T09:50:44.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:44.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:44.404 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:44.400+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.403704+0000) 2019-10-13T09:50:44.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:44.400+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.403938+0000) 2019-10-13T09:50:44.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:44.400+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.404667+0000) 2019-10-13T09:50:44.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:44.406+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.408995+0000) 2019-10-13T09:50:44.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:44.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:44.424996+0000) 2019-10-13T09:50:47.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:47.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:47.676963+0000) 2019-10-13T09:50:48.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:48.444+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.448339+0000) 2019-10-13T09:50:48.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:48.458+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:48.461459+0000) 2019-10-13T09:50:49.407 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:49.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.406732+0000) 2019-10-13T09:50:49.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:49.404+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.407217+0000) 2019-10-13T09:50:49.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:49.404+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.407536+0000) 2019-10-13T09:50:49.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:49.404+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.407947+0000) 2019-10-13T09:50:49.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:49.402+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.407721+0000) 2019-10-13T09:50:49.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:49.404+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.409237+0000) 2019-10-13T09:50:49.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:49.408+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.409903+0000) 2019-10-13T09:50:49.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:49.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:49.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:49.410+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:49.412138+0000) 2019-10-13T09:50:51.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:51.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:51.151097+0000) 2019-10-13T09:50:51.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:51.694+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:51.699934+0000) 2019-10-13T09:50:54.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:54.012+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.015044+0000) 2019-10-13T09:50:54.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:54.300+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.302817+0000) 2019-10-13T09:50:54.415 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:54.410+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.413918+0000) 2019-10-13T09:50:54.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:54.412+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.415056+0000) 2019-10-13T09:50:54.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:54.412+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.415248+0000) 2019-10-13T09:50:54.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:54.416+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.417984+0000) 2019-10-13T09:50:54.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:54.416+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:54.420 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:54.416+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.419116+0000) 2019-10-13T09:50:54.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:54.416+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.419653+0000) 2019-10-13T09:50:54.422 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:54.416+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.421373+0000) 2019-10-13T09:50:54.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:54.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.425159+0000) 2019-10-13T09:50:54.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:54.422+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:54.424571+0000) 2019-10-13T09:50:57.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:57.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:57.677195+0000) 2019-10-13T09:50:58.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:58.444+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.448567+0000) 2019-10-13T09:50:58.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:58.462+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:58.467931+0000) 2019-10-13T09:50:59.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:50:59.420+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.424015+0000) 2019-10-13T09:50:59.425 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:50:59.420+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.424473+0000) 2019-10-13T09:50:59.426 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:50:59.420+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.424993+0000) 2019-10-13T09:50:59.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:50:59.422+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.424895+0000) 2019-10-13T09:50:59.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:50:59.420+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.425647+0000) 2019-10-13T09:50:59.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:50:59.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:50:59.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:50:59.424+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.426883+0000) 2019-10-13T09:50:59.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:50:59.424+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.427301+0000) 2019-10-13T09:50:59.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:50:59.426+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:50:59.431969+0000) 2019-10-13T09:51:00.125 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:51:00.126 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:00.129 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:51:00.129 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:01.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:01.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:01.151303+0000) 2019-10-13T09:51:01.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:01.698+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:01.700134+0000) 2019-10-13T09:51:04.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:04.012+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.015252+0000) 2019-10-13T09:51:04.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:04.300+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.303046+0000) 2019-10-13T09:51:04.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:04.424+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.425270+0000) 2019-10-13T09:51:04.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:04.428+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.430016+0000) 2019-10-13T09:51:04.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:04.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.430374+0000) 2019-10-13T09:51:04.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:04.428+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.432926+0000) 2019-10-13T09:51:04.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:04.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:04.435 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:04.432+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.434557+0000) 2019-10-13T09:51:04.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:04.432+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.435548+0000) 2019-10-13T09:51:04.437 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:04.432+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.436822+0000) 2019-10-13T09:51:04.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:04.432+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.435152+0000) 2019-10-13T09:51:04.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:04.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:04.442470+0000) 2019-10-13T09:51:07.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:07.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:07.677419+0000) 2019-10-13T09:51:08.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:08.444+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.448756+0000) 2019-10-13T09:51:08.475 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:08.474+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:08.474374+0000) 2019-10-13T09:51:09.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:09.432+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.437270+0000) 2019-10-13T09:51:09.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:09.436+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.440994+0000) 2019-10-13T09:51:09.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:09.436+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.440730+0000) 2019-10-13T09:51:09.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:09.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:09.443 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:09.440+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.442628+0000) 2019-10-13T09:51:09.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:09.440+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.442722+0000) 2019-10-13T09:51:09.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:09.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.442440+0000) 2019-10-13T09:51:09.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:09.440+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.443501+0000) 2019-10-13T09:51:09.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:09.446+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:09.449585+0000) 2019-10-13T09:51:11.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:11.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:11.151522+0000) 2019-10-13T09:51:11.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:11.698+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:11.700289+0000) 2019-10-13T09:51:14.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:14.012+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.015459+0000) 2019-10-13T09:51:14.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:14.300+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.303248+0000) 2019-10-13T09:51:14.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:14.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.425468+0000) 2019-10-13T09:51:14.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:14.444+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.446461+0000) 2019-10-13T09:51:14.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:14.444+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.446682+0000) 2019-10-13T09:51:14.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:14.442+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.447472+0000) 2019-10-13T09:51:14.450 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:14.444+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.449676+0000) 2019-10-13T09:51:14.451 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:14.448+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.450019+0000) 2019-10-13T09:51:14.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:14.448+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.450740+0000) 2019-10-13T09:51:14.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:14.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:14.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:14.448+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.451643+0000) 2019-10-13T09:51:14.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:14.454+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:14.456618+0000) 2019-10-13T09:51:17.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:17.672+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:17.677634+0000) 2019-10-13T09:51:18.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:18.448+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.448981+0000) 2019-10-13T09:51:18.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:18.478+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:18.480832+0000) 2019-10-13T09:51:19.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:19.452+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.454051+0000) 2019-10-13T09:51:19.455 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:19.452+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.454491+0000) 2019-10-13T09:51:19.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:19.452+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.455661+0000) 2019-10-13T09:51:19.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:19.452+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.456894+0000) 2019-10-13T09:51:19.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:19.452+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.456961+0000) 2019-10-13T09:51:19.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:19.453+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.457055+0000) 2019-10-13T09:51:19.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:19.456+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.457929+0000) 2019-10-13T09:51:19.459 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:19.456+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:19.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:19.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:19.460719+0000) 2019-10-13T09:51:21.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:21.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:21.151765+0000) 2019-10-13T09:51:21.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:21.697+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:21.700489+0000) 2019-10-13T09:51:24.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:24.012+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.015679+0000) 2019-10-13T09:51:24.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:24.300+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.303477+0000) 2019-10-13T09:51:24.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:24.420+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.425667+0000) 2019-10-13T09:51:24.463 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:24.457+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.462000+0000) 2019-10-13T09:51:24.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:24.460+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.464109+0000) 2019-10-13T09:51:24.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:24.464+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.466443+0000) 2019-10-13T09:51:24.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:24.464+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:24.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:24.464+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.467806+0000) 2019-10-13T09:51:24.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:24.464+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.468119+0000) 2019-10-13T09:51:24.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:24.464+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.468466+0000) 2019-10-13T09:51:24.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:24.464+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.468957+0000) 2019-10-13T09:51:24.470 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:24.465+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:24.469354+0000) 2019-10-13T09:51:27.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:27.676+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:27.677877+0000) 2019-10-13T09:51:28.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:28.448+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.449208+0000) 2019-10-13T09:51:28.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:28.489+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:28.492476+0000) 2019-10-13T09:51:29.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:29.468+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.471479+0000) 2019-10-13T09:51:29.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:29.468+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.472061+0000) 2019-10-13T09:51:29.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:29.468+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.472170+0000) 2019-10-13T09:51:29.474 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:29.465+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.472042+0000) 2019-10-13T09:51:29.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:29.468+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.472876+0000) 2019-10-13T09:51:29.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:29.468+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.473296+0000) 2019-10-13T09:51:29.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:29.472+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.473880+0000) 2019-10-13T09:51:29.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:29.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:29.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:29.473+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:29.480004+0000) 2019-10-13T09:51:30.256 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:51:30.256 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:30.262 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:51:30.263 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:51:31.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:31.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:31.152031+0000) 2019-10-13T09:51:31.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:31.697+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:31.700725+0000) 2019-10-13T09:51:34.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:34.011+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.015882+0000) 2019-10-13T09:51:34.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:34.299+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.303694+0000) 2019-10-13T09:51:34.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:34.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.425919+0000) 2019-10-13T09:51:34.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:34.475+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.480762+0000) 2019-10-13T09:51:34.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:34.475+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.481144+0000) 2019-10-13T09:51:34.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:34.475+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.481434+0000) 2019-10-13T09:51:34.483 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:34.479+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.482365+0000) 2019-10-13T09:51:34.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:34.479+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.482322+0000) 2019-10-13T09:51:34.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:34.477+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.482158+0000) 2019-10-13T09:51:34.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:34.479+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.482976+0000) 2019-10-13T09:51:34.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:34.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:34.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:34.485+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:34.488771+0000) 2019-10-13T09:51:37.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:37.675+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:37.678106+0000) 2019-10-13T09:51:38.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:38.443+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.449434+0000) 2019-10-13T09:51:38.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:38.497+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:38.499008+0000) 2019-10-13T09:51:39.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:39.483+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.486850+0000) 2019-10-13T09:51:39.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:39.483+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.489460+0000) 2019-10-13T09:51:39.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:39.487+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.489916+0000) 2019-10-13T09:51:39.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:39.487+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.490469+0000) 2019-10-13T09:51:39.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:39.485+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.490624+0000) 2019-10-13T09:51:39.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:39.487+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.491414+0000) 2019-10-13T09:51:39.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:39.487+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.492391+0000) 2019-10-13T09:51:39.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:39.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:39.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:39.489+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:39.494741+0000) 2019-10-13T09:51:41.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:41.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:41.152230+0000) 2019-10-13T09:51:41.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:41.697+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:41.700928+0000) 2019-10-13T09:51:44.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:44.011+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.016088+0000) 2019-10-13T09:51:44.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:44.299+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.303891+0000) 2019-10-13T09:51:44.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:44.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.426151+0000) 2019-10-13T09:51:44.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:44.491+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.495768+0000) 2019-10-13T09:51:44.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:44.491+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.497189+0000) 2019-10-13T09:51:44.498 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:44.493+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.497135+0000) 2019-10-13T09:51:44.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:44.495+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.499522+0000) 2019-10-13T09:51:44.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:44.495+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.499761+0000) 2019-10-13T09:51:44.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:44.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:44.502 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:44.495+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.501643+0000) 2019-10-13T09:51:44.503 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:44.499+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.502118+0000) 2019-10-13T09:51:44.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:44.497+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:44.502509+0000) 2019-10-13T09:51:47.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:47.675+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:47.678298+0000) 2019-10-13T09:51:48.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:48.443+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.449624+0000) 2019-10-13T09:51:48.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:48.501+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:48.505432+0000) 2019-10-13T09:51:49.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:49.501+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.504579+0000) 2019-10-13T09:51:49.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:49.503+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.506004+0000) 2019-10-13T09:51:49.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:49.503+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.506499+0000) 2019-10-13T09:51:49.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:49.503+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.508382+0000) 2019-10-13T09:51:49.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:49.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:49.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:49.507+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.510161+0000) 2019-10-13T09:51:49.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:49.507+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.510966+0000) 2019-10-13T09:51:49.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:49.505+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.511119+0000) 2019-10-13T09:51:49.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:49.507+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:49.511833+0000) 2019-10-13T09:51:51.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:51.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:51.152473+0000) 2019-10-13T09:51:51.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:51.697+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:51.701138+0000) 2019-10-13T09:51:54.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:54.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.016283+0000) 2019-10-13T09:51:54.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:54.303+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.304057+0000) 2019-10-13T09:51:54.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:54.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.426356+0000) 2019-10-13T09:51:54.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:54.511+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.514638+0000) 2019-10-13T09:51:54.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:54.511+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.517028+0000) 2019-10-13T09:51:54.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:54.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:54.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:54.513+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.518327+0000) 2019-10-13T09:51:54.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:54.513+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.519356+0000) 2019-10-13T09:51:54.522 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:54.515+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.519438+0000) 2019-10-13T09:51:54.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:54.515+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.519502+0000) 2019-10-13T09:51:54.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:54.515+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.519966+0000) 2019-10-13T09:51:54.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:54.515+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:54.520889+0000) 2019-10-13T09:51:57.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:57.675+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:57.678477+0000) 2019-10-13T09:51:58.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:58.447+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.449859+0000) 2019-10-13T09:51:58.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:58.509+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:58.512058+0000) 2019-10-13T09:51:59.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:51:59.519+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.522188+0000) 2019-10-13T09:51:59.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:51:59.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.522751+0000) 2019-10-13T09:51:59.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:51:59.519+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.523514+0000) 2019-10-13T09:51:59.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:51:59.519+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.524231+0000) 2019-10-13T09:51:59.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:51:59.519+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.523906+0000) 2019-10-13T09:51:59.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:51:59.521+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.524304+0000) 2019-10-13T09:51:59.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:51:59.519+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.525254+0000) 2019-10-13T09:51:59.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:51:59.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:51:59.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:51:59.525+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:51:59.528770+0000) 2019-10-13T09:52:00.346 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:52:00.346 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:00.350 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:52:00.350 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:01.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:01.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:01.152675+0000) 2019-10-13T09:52:01.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:01.697+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:01.701357+0000) 2019-10-13T09:52:04.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:04.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.016493+0000) 2019-10-13T09:52:04.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:04.299+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.304257+0000) 2019-10-13T09:52:04.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:04.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.426555+0000) 2019-10-13T09:52:04.530 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:04.523+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.529201+0000) 2019-10-13T09:52:04.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:04.527+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.530302+0000) 2019-10-13T09:52:04.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:04.527+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.531448+0000) 2019-10-13T09:52:04.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:04.525+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.531241+0000) 2019-10-13T09:52:04.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:04.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.532456+0000) 2019-10-13T09:52:04.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:04.527+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.533079+0000) 2019-10-13T09:52:04.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:04.527+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.533385+0000) 2019-10-13T09:52:04.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:04.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103958 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:04.540 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:04.533+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:04.538785+0000) 2019-10-13T09:52:07.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:07.675+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:07.678692+0000) 2019-10-13T09:52:08.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:08.447+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.450034+0000) 2019-10-13T09:52:08.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:08.521+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:08.523228+0000) 2019-10-13T09:52:09.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:09.531+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.537264+0000) 2019-10-13T09:52:09.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:09.533+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.538328+0000) 2019-10-13T09:52:09.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:09.535+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.540527+0000) 2019-10-13T09:52:09.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:09.535+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.540778+0000) 2019-10-13T09:52:09.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:09.535+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.541189+0000) 2019-10-13T09:52:09.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:09.539+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.541895+0000) 2019-10-13T09:52:09.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:09.539+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.542210+0000) 2019-10-13T09:52:09.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:09.539+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:09.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:09.545+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:09.548619+0000) 2019-10-13T09:52:11.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:11.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:11.152908+0000) 2019-10-13T09:52:11.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:11.701+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:11.701563+0000) 2019-10-13T09:52:14.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:14.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.016745+0000) 2019-10-13T09:52:14.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:14.303+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.304461+0000) 2019-10-13T09:52:14.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:14.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.426790+0000) 2019-10-13T09:52:14.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:14.543+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.546171+0000) 2019-10-13T09:52:14.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:14.543+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.546836+0000) 2019-10-13T09:52:14.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:14.543+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.548608+0000) 2019-10-13T09:52:14.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:14.545+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.550134+0000) 2019-10-13T09:52:14.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:14.547+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.551253+0000) 2019-10-13T09:52:14.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:14.547+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:14.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:14.547+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.551667+0000) 2019-10-13T09:52:14.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:14.547+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.551902+0000) 2019-10-13T09:52:14.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:14.549+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:14.553488+0000) 2019-10-13T09:52:17.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:17.675+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:17.678910+0000) 2019-10-13T09:52:18.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:18.447+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.450228+0000) 2019-10-13T09:52:18.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:18.529+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:18.529741+0000) 2019-10-13T09:52:19.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:19.551+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.554272+0000) 2019-10-13T09:52:19.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:19.551+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.555971+0000) 2019-10-13T09:52:19.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:19.551+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.556329+0000) 2019-10-13T09:52:19.559 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:19.555+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.558242+0000) 2019-10-13T09:52:19.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:19.555+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.559283+0000) 2019-10-13T09:52:19.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:19.555+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.559611+0000) 2019-10-13T09:52:19.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:19.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:19.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:19.553+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.559449+0000) 2019-10-13T09:52:19.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:19.557+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:19.561708+0000) 2019-10-13T09:52:21.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:21.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:21.153134+0000) 2019-10-13T09:52:21.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:21.701+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:21.701761+0000) 2019-10-13T09:52:24.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:24.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.016984+0000) 2019-10-13T09:52:24.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:24.303+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.304662+0000) 2019-10-13T09:52:24.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:24.423+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.427011+0000) 2019-10-13T09:52:24.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:24.559+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.563548+0000) 2019-10-13T09:52:24.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:24.557+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.563443+0000) 2019-10-13T09:52:24.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:24.559+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.564581+0000) 2019-10-13T09:52:24.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:24.559+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.565343+0000) 2019-10-13T09:52:24.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:24.563+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.565810+0000) 2019-10-13T09:52:24.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:24.563+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.566261+0000) 2019-10-13T09:52:24.567 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:24.563+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.566718+0000) 2019-10-13T09:52:24.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:24.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:24.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:24.565+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:24.569005+0000) 2019-10-13T09:52:27.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:27.674+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:27.679098+0000) 2019-10-13T09:52:28.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:28.446+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.450432+0000) 2019-10-13T09:52:28.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:28.532+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:28.536260+0000) 2019-10-13T09:52:29.571 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:29.566+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.570702+0000) 2019-10-13T09:52:29.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:29.564+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.570708+0000) 2019-10-13T09:52:29.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:29.566+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.572747+0000) 2019-10-13T09:52:29.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:29.566+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.573016+0000) 2019-10-13T09:52:29.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:29.566+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.573447+0000) 2019-10-13T09:52:29.574 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:29.570+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.574009+0000) 2019-10-13T09:52:29.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:29.570+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.574750+0000) 2019-10-13T09:52:29.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:29.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:29.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:29.572+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:29.578345+0000) 2019-10-13T09:52:30.461 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:52:30.461 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:30.466 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:52:30.466 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:52:31.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:31.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:31.153370+0000) 2019-10-13T09:52:31.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:31.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:31.701963+0000) 2019-10-13T09:52:34.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:34.014+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.017192+0000) 2019-10-13T09:52:34.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:34.298+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.304890+0000) 2019-10-13T09:52:34.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:34.426+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.427242+0000) 2019-10-13T09:52:34.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:34.574+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.580241+0000) 2019-10-13T09:52:34.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:34.578+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.582283+0000) 2019-10-13T09:52:34.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:34.576+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.582287+0000) 2019-10-13T09:52:34.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:34.578+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.583350+0000) 2019-10-13T09:52:34.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:34.578+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.583620+0000) 2019-10-13T09:52:34.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:34.578+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.583955+0000) 2019-10-13T09:52:34.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:34.578+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.584264+0000) 2019-10-13T09:52:34.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:34.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:34.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:34.584+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:34.590324+0000) 2019-10-13T09:52:37.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:37.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:37.679291+0000) 2019-10-13T09:52:38.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:38.446+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.450626+0000) 2019-10-13T09:52:38.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:38.540+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:38.542871+0000) 2019-10-13T09:52:39.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:39.582+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.589228+0000) 2019-10-13T09:52:39.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:39.584+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.589206+0000) 2019-10-13T09:52:39.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:39.586+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.590445+0000) 2019-10-13T09:52:39.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:39.586+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.590996+0000) 2019-10-13T09:52:39.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:39.586+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.591979+0000) 2019-10-13T09:52:39.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:39.586+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.592244+0000) 2019-10-13T09:52:39.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:39.586+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.592508+0000) 2019-10-13T09:52:39.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:39.586+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:39.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:39.592+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:39.597821+0000) 2019-10-13T09:52:41.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:41.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:41.153566+0000) 2019-10-13T09:52:41.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:41.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:41.702165+0000) 2019-10-13T09:52:44.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:44.014+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.017400+0000) 2019-10-13T09:52:44.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:44.302+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.305096+0000) 2019-10-13T09:52:44.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:44.426+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.427446+0000) 2019-10-13T09:52:44.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:44.590+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.597675+0000) 2019-10-13T09:52:44.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:44.590+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.597391+0000) 2019-10-13T09:52:44.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:44.592+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.599009+0000) 2019-10-13T09:52:44.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:44.594+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.599069+0000) 2019-10-13T09:52:44.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:44.594+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.599764+0000) 2019-10-13T09:52:44.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:44.594+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.600362+0000) 2019-10-13T09:52:44.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:44.594+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.600895+0000) 2019-10-13T09:52:44.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:44.594+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:44.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:44.604+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:44.608336+0000) 2019-10-13T09:52:47.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:47.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:47.679485+0000) 2019-10-13T09:52:48.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:48.446+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.450843+0000) 2019-10-13T09:52:48.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:48.548+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:48.549507+0000) 2019-10-13T09:52:49.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:49.598+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.605588+0000) 2019-10-13T09:52:49.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:49.602+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.606000+0000) 2019-10-13T09:52:49.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:49.602+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.606434+0000) 2019-10-13T09:52:49.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:49.602+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.606934+0000) 2019-10-13T09:52:49.608 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:49.602+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.607356+0000) 2019-10-13T09:52:49.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:49.602+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.607688+0000) 2019-10-13T09:52:49.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:49.600+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.607982+0000) 2019-10-13T09:52:49.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:49.602+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104232 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:49.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:49.608+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:49.613798+0000) 2019-10-13T09:52:51.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:51.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:51.153791+0000) 2019-10-13T09:52:51.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:51.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:51.702396+0000) 2019-10-13T09:52:54.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:54.014+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.017606+0000) 2019-10-13T09:52:54.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:54.302+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.305304+0000) 2019-10-13T09:52:54.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:54.422+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.427639+0000) 2019-10-13T09:52:54.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:54.606+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.613175+0000) 2019-10-13T09:52:54.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:54.610+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.613874+0000) 2019-10-13T09:52:54.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:54.610+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.614523+0000) 2019-10-13T09:52:54.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:54.610+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.614959+0000) 2019-10-13T09:52:54.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:54.608+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.615646+0000) 2019-10-13T09:52:54.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:54.610+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.615945+0000) 2019-10-13T09:52:54.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:54.610+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.616414+0000) 2019-10-13T09:52:54.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:54.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:54.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:54.616+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:54.621211+0000) 2019-10-13T09:52:57.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:57.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:57.679687+0000) 2019-10-13T09:52:58.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:58.450+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.451033+0000) 2019-10-13T09:52:58.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:58.552+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:58.556004+0000) 2019-10-13T09:52:59.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:52:59.618+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.622328+0000) 2019-10-13T09:52:59.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:52:59.618+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.622761+0000) 2019-10-13T09:52:59.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:52:59.618+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.623411+0000) 2019-10-13T09:52:59.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:52:59.618+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.623972+0000) 2019-10-13T09:52:59.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:52:59.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.627003+0000) 2019-10-13T09:52:59.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:52:59.622+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:52:59.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:52:59.620+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.626406+0000) 2019-10-13T09:52:59.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:52:59.622+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.627476+0000) 2019-10-13T09:52:59.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:52:59.624+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:52:59.631251+0000) 2019-10-13T09:53:00.566 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:53:00.566 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:00.570 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:53:00.570 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:01.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:01.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:01.153997+0000) 2019-10-13T09:53:01.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:01.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:01.702601+0000) 2019-10-13T09:53:04.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:04.014+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.017844+0000) 2019-10-13T09:53:04.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:04.302+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.305533+0000) 2019-10-13T09:53:04.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:04.426+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.427840+0000) 2019-10-13T09:53:04.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:04.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.630195+0000) 2019-10-13T09:53:04.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:04.626+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.631168+0000) 2019-10-13T09:53:04.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:04.626+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.631443+0000) 2019-10-13T09:53:04.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:04.626+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.632338+0000) 2019-10-13T09:53:04.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:04.626+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.633110+0000) 2019-10-13T09:53:04.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:04.626+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.633368+0000) 2019-10-13T09:53:04.634 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:04.626+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.633533+0000) 2019-10-13T09:53:04.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:04.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:04.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:04.632+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:04.637047+0000) 2019-10-13T09:53:07.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:07.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:07.679884+0000) 2019-10-13T09:53:08.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:08.450+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.451264+0000) 2019-10-13T09:53:08.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:08.560+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:08.562541+0000) 2019-10-13T09:53:09.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:09.634+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.638519+0000) 2019-10-13T09:53:09.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:09.632+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.638657+0000) 2019-10-13T09:53:09.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:09.634+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.640686+0000) 2019-10-13T09:53:09.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:09.634+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.640091+0000) 2019-10-13T09:53:09.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:09.634+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.641415+0000) 2019-10-13T09:53:09.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:09.634+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.641497+0000) 2019-10-13T09:53:09.643 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:09.638+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.641846+0000) 2019-10-13T09:53:09.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:09.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104357 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:09.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:09.640+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:09.646042+0000) 2019-10-13T09:53:11.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:11.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:11.154195+0000) 2019-10-13T09:53:11.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:11.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:11.702807+0000) 2019-10-13T09:53:14.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:14.014+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.018046+0000) 2019-10-13T09:53:14.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:14.302+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.305764+0000) 2019-10-13T09:53:14.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:14.426+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.428041+0000) 2019-10-13T09:53:14.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:14.642+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.647188+0000) 2019-10-13T09:53:14.648 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:14.642+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.647452+0000) 2019-10-13T09:53:14.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:14.642+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.647814+0000) 2019-10-13T09:53:14.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:14.642+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.648460+0000) 2019-10-13T09:53:14.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:14.644+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.648383+0000) 2019-10-13T09:53:14.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:14.642+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.649297+0000) 2019-10-13T09:53:14.652 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:14.642+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.649615+0000) 2019-10-13T09:53:14.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:14.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:14.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:14.648+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:14.655066+0000) 2019-10-13T09:53:17.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:17.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:17.680087+0000) 2019-10-13T09:53:18.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:18.450+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.451474+0000) 2019-10-13T09:53:18.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:18.568+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:18.569171+0000) 2019-10-13T09:53:19.656 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:19.650+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.655353+0000) 2019-10-13T09:53:19.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:19.650+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.655634+0000) 2019-10-13T09:53:19.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:19.650+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.656378+0000) 2019-10-13T09:53:19.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:19.654+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.658455+0000) 2019-10-13T09:53:19.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:19.654+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.659387+0000) 2019-10-13T09:53:19.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:19.652+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.659244+0000) 2019-10-13T09:53:19.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:19.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:19.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:19.654+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.660708+0000) 2019-10-13T09:53:19.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:19.660+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:19.665900+0000) 2019-10-13T09:53:21.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:21.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:21.154429+0000) 2019-10-13T09:53:21.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:21.700+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:21.703005+0000) 2019-10-13T09:53:24.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:24.017+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.018245+0000) 2019-10-13T09:53:24.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:24.301+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.305966+0000) 2019-10-13T09:53:24.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:24.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.428244+0000) 2019-10-13T09:53:24.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:24.657+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.663581+0000) 2019-10-13T09:53:24.665 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:24.657+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.663951+0000) 2019-10-13T09:53:24.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:24.661+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.666196+0000) 2019-10-13T09:53:24.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:24.661+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.667165+0000) 2019-10-13T09:53:24.668 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:24.661+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.667506+0000) 2019-10-13T09:53:24.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:24.661+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.667774+0000) 2019-10-13T09:53:24.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:24.664+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.668458+0000) 2019-10-13T09:53:24.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:24.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:24.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:24.668+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:24.675928+0000) 2019-10-13T09:53:27.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:27.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:27.680289+0000) 2019-10-13T09:53:28.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:28.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.451676+0000) 2019-10-13T09:53:28.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:28.572+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:28.575469+0000) 2019-10-13T09:53:29.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:29.665+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.672194+0000) 2019-10-13T09:53:29.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:29.665+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.672697+0000) 2019-10-13T09:53:29.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:29.672+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.672657+0000) 2019-10-13T09:53:29.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:29.669+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.675351+0000) 2019-10-13T09:53:29.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:29.669+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.675724+0000) 2019-10-13T09:53:29.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:29.669+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.675916+0000) 2019-10-13T09:53:29.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:29.669+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.676897+0000) 2019-10-13T09:53:29.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:29.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:29.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:29.680+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:29.682827+0000) 2019-10-13T09:53:30.688 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:53:30.689 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:30.692 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:53:30.693 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:53:31.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:31.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:31.154636+0000) 2019-10-13T09:53:31.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:31.699+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:31.703207+0000) 2019-10-13T09:53:34.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:34.013+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.018501+0000) 2019-10-13T09:53:34.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:34.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.306167+0000) 2019-10-13T09:53:34.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:34.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.428445+0000) 2019-10-13T09:53:34.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:34.677+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.681844+0000) 2019-10-13T09:53:34.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:34.677+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.682666+0000) 2019-10-13T09:53:34.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:34.677+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.682231+0000) 2019-10-13T09:53:34.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:34.679+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.682662+0000) 2019-10-13T09:53:34.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:34.677+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.683785+0000) 2019-10-13T09:53:34.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:34.677+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.684687+0000) 2019-10-13T09:53:34.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:34.677+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.685576+0000) 2019-10-13T09:53:34.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:34.681+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:34.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:34.687+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:34.689772+0000) 2019-10-13T09:53:37.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:37.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:37.680491+0000) 2019-10-13T09:53:38.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:38.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.451879+0000) 2019-10-13T09:53:38.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:38.579+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:38.582014+0000) 2019-10-13T09:53:39.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:39.689+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.690059+0000) 2019-10-13T09:53:39.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:39.689+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.690430+0000) 2019-10-13T09:53:39.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:39.689+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.692796+0000) 2019-10-13T09:53:39.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:39.691+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.693197+0000) 2019-10-13T09:53:39.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:39.693+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.693970+0000) 2019-10-13T09:53:39.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:39.693+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.694463+0000) 2019-10-13T09:53:39.695 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:39.693+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.694928+0000) 2019-10-13T09:53:39.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:39.693+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:39.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:39.695+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:39.698218+0000) 2019-10-13T09:53:41.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:41.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:41.154844+0000) 2019-10-13T09:53:41.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:41.699+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:41.703409+0000) 2019-10-13T09:53:44.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:44.017+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.018740+0000) 2019-10-13T09:53:44.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:44.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.306377+0000) 2019-10-13T09:53:44.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:44.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.428642+0000) 2019-10-13T09:53:44.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:44.695+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.697980+0000) 2019-10-13T09:53:44.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:44.697+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.698908+0000) 2019-10-13T09:53:44.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:44.697+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.699318+0000) 2019-10-13T09:53:44.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:44.697+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.701117+0000) 2019-10-13T09:53:44.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:44.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:44.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:44.701+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.703600+0000) 2019-10-13T09:53:44.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:44.701+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.704239+0000) 2019-10-13T09:53:44.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:44.705+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.705876+0000) 2019-10-13T09:53:44.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:44.707+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:44.709956+0000) 2019-10-13T09:53:47.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:47.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:47.680694+0000) 2019-10-13T09:53:48.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:48.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.452107+0000) 2019-10-13T09:53:48.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:48.583+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:48.587744+0000) 2019-10-13T09:53:49.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:49.705+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.707377+0000) 2019-10-13T09:53:49.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:49.705+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.708035+0000) 2019-10-13T09:53:49.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:49.705+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.708535+0000) 2019-10-13T09:53:49.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:49.705+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.708861+0000) 2019-10-13T09:53:49.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:49.707+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.709159+0000) 2019-10-13T09:53:49.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:49.709+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.709904+0000) 2019-10-13T09:53:49.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:49.709+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.710419+0000) 2019-10-13T09:53:49.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:49.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:49.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:49.715+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:49.717070+0000) 2019-10-13T09:53:51.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:51.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:51.155045+0000) 2019-10-13T09:53:51.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:51.699+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:51.703629+0000) 2019-10-13T09:53:54.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:54.017+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.018942+0000) 2019-10-13T09:53:54.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:54.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.306614+0000) 2019-10-13T09:53:54.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:54.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.428856+0000) 2019-10-13T09:53:54.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:54.713+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.716857+0000) 2019-10-13T09:53:54.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:54.713+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.717258+0000) 2019-10-13T09:53:54.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:54.717+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.717875+0000) 2019-10-13T09:53:54.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:54.717+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.718398+0000) 2019-10-13T09:53:54.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:54.717+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.719135+0000) 2019-10-13T09:53:54.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:54.715+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.718952+0000) 2019-10-13T09:53:54.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:54.717+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:54.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:54.717+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.720574+0000) 2019-10-13T09:53:54.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:54.723+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:54.725995+0000) 2019-10-13T09:53:57.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:57.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:57.680882+0000) 2019-10-13T09:53:58.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:58.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.452329+0000) 2019-10-13T09:53:58.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:58.591+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:58.594389+0000) 2019-10-13T09:53:59.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:53:59.721+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.723203+0000) 2019-10-13T09:53:59.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:53:59.721+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.724848+0000) 2019-10-13T09:53:59.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:53:59.723+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.724832+0000) 2019-10-13T09:53:59.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:53:59.721+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.725615+0000) 2019-10-13T09:53:59.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:53:59.725+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.726184+0000) 2019-10-13T09:53:59.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:53:59.725+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.726673+0000) 2019-10-13T09:53:59.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:53:59.725+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.727298+0000) 2019-10-13T09:53:59.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:53:59.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:53:59.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:53:59.727+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:53:59.730249+0000) 2019-10-13T09:54:00.767 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:54:00.767 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:00.771 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:54:00.771 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:01.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:01.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:01.155262+0000) 2019-10-13T09:54:01.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:01.699+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:01.703856+0000) 2019-10-13T09:54:04.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:04.017+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.019140+0000) 2019-10-13T09:54:04.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:04.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.306807+0000) 2019-10-13T09:54:04.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:04.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.429079+0000) 2019-10-13T09:54:04.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:04.729+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.732381+0000) 2019-10-13T09:54:04.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:04.733+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.734012+0000) 2019-10-13T09:54:04.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:04.731+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.734016+0000) 2019-10-13T09:54:04.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:04.733+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.734891+0000) 2019-10-13T09:54:04.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:04.733+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.735245+0000) 2019-10-13T09:54:04.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:04.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:04.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:04.733+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.736453+0000) 2019-10-13T09:54:04.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:04.733+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.736955+0000) 2019-10-13T09:54:04.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:04.735+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:04.739654+0000) 2019-10-13T09:54:07.682 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:07.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:07.681105+0000) 2019-10-13T09:54:08.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:08.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.452521+0000) 2019-10-13T09:54:08.602 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:08.599+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:08.600986+0000) 2019-10-13T09:54:09.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:09.737+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.739145+0000) 2019-10-13T09:54:09.740 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:09.737+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.739903+0000) 2019-10-13T09:54:09.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:09.741+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.743711+0000) 2019-10-13T09:54:09.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:09.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104733 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:09.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:09.741+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.743927+0000) 2019-10-13T09:54:09.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:09.741+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.744530+0000) 2019-10-13T09:54:09.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:09.743+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.744736+0000) 2019-10-13T09:54:09.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:09.741+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.745495+0000) 2019-10-13T09:54:09.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:09.743+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:09.747267+0000) 2019-10-13T09:54:11.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:11.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:11.155461+0000) 2019-10-13T09:54:11.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:11.699+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:11.704060+0000) 2019-10-13T09:54:14.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:14.017+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.019374+0000) 2019-10-13T09:54:14.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:14.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.307004+0000) 2019-10-13T09:54:14.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:14.425+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.429291+0000) 2019-10-13T09:54:14.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:14.745+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.747324+0000) 2019-10-13T09:54:14.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:14.745+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.747829+0000) 2019-10-13T09:54:14.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:14.745+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.748325+0000) 2019-10-13T09:54:14.750 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:14.745+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.749507+0000) 2019-10-13T09:54:14.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:14.749+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.749817+0000) 2019-10-13T09:54:14.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:14.747+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.749752+0000) 2019-10-13T09:54:14.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:14.749+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.750614+0000) 2019-10-13T09:54:14.752 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:14.749+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:14.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:14.751+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:14.755135+0000) 2019-10-13T09:54:17.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:17.681+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:17.681308+0000) 2019-10-13T09:54:18.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:18.449+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.452730+0000) 2019-10-13T09:54:18.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:18.603+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:18.607502+0000) 2019-10-13T09:54:19.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:19.753+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.755754+0000) 2019-10-13T09:54:19.757 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:19.753+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.756874+0000) 2019-10-13T09:54:19.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:19.753+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.757133+0000) 2019-10-13T09:54:19.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:19.753+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.757523+0000) 2019-10-13T09:54:19.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:19.755+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.757366+0000) 2019-10-13T09:54:19.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:19.757+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.758055+0000) 2019-10-13T09:54:19.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:19.757+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.758679+0000) 2019-10-13T09:54:19.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:19.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:19.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:19.759+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:19.761790+0000) 2019-10-13T09:54:21.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:21.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:21.155665+0000) 2019-10-13T09:54:21.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:21.703+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:21.704330+0000) 2019-10-13T09:54:24.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:24.016+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.019574+0000) 2019-10-13T09:54:24.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:24.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.307236+0000) 2019-10-13T09:54:24.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:24.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.429486+0000) 2019-10-13T09:54:24.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:24.760+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.763679+0000) 2019-10-13T09:54:24.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:24.760+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.764754+0000) 2019-10-13T09:54:24.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:24.760+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.765319+0000) 2019-10-13T09:54:24.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:24.760+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.765026+0000) 2019-10-13T09:54:24.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:24.764+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.766052+0000) 2019-10-13T09:54:24.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:24.763+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.766082+0000) 2019-10-13T09:54:24.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:24.764+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.766690+0000) 2019-10-13T09:54:24.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:24.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:24.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:24.767+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:24.770120+0000) 2019-10-13T09:54:27.682 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:27.680+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:27.681508+0000) 2019-10-13T09:54:28.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:28.452+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.452922+0000) 2019-10-13T09:54:28.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:28.611+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:28.613988+0000) 2019-10-13T09:54:29.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:29.768+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.771327+0000) 2019-10-13T09:54:29.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:29.768+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.773477+0000) 2019-10-13T09:54:29.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:29.772+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.773945+0000) 2019-10-13T09:54:29.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:29.771+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.773871+0000) 2019-10-13T09:54:29.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:29.772+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.774799+0000) 2019-10-13T09:54:29.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:29.772+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.775406+0000) 2019-10-13T09:54:29.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:29.772+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.775715+0000) 2019-10-13T09:54:29.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:29.772+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:29.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:29.775+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:29.778004+0000) 2019-10-13T09:54:30.900 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:54:30.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:30.904 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:54:30.904 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:54:31.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:31.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:31.155870+0000) 2019-10-13T09:54:31.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:31.703+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:31.704557+0000) 2019-10-13T09:54:34.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:34.016+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.019779+0000) 2019-10-13T09:54:34.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:34.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.307452+0000) 2019-10-13T09:54:34.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:34.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.429688+0000) 2019-10-13T09:54:34.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:34.776+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.780247+0000) 2019-10-13T09:54:34.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:34.776+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.780789+0000) 2019-10-13T09:54:34.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:34.776+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.781124+0000) 2019-10-13T09:54:34.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:34.779+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.781198+0000) 2019-10-13T09:54:34.783 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:34.780+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.782104+0000) 2019-10-13T09:54:34.783 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:34.780+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.782352+0000) 2019-10-13T09:54:34.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:34.780+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.783740+0000) 2019-10-13T09:54:34.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:34.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:34.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:34.787+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:34.789522+0000) 2019-10-13T09:54:37.682 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:37.676+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:37.681705+0000) 2019-10-13T09:54:38.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:38.452+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.453077+0000) 2019-10-13T09:54:38.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:38.618+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:38.623357+0000) 2019-10-13T09:54:39.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:39.784+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.787783+0000) 2019-10-13T09:54:39.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:39.784+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.788354+0000) 2019-10-13T09:54:39.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:39.784+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.788933+0000) 2019-10-13T09:54:39.790 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:39.788+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.789768+0000) 2019-10-13T09:54:39.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:39.788+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.790195+0000) 2019-10-13T09:54:39.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:39.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.790614+0000) 2019-10-13T09:54:39.792 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:39.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.791267+0000) 2019-10-13T09:54:39.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:39.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:39.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:39.790+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:39.793433+0000) 2019-10-13T09:54:41.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:41.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:41.156103+0000) 2019-10-13T09:54:41.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:41.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:41.704757+0000) 2019-10-13T09:54:44.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:44.016+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.019986+0000) 2019-10-13T09:54:44.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:44.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.307649+0000) 2019-10-13T09:54:44.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:44.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.429870+0000) 2019-10-13T09:54:44.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:44.792+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.795343+0000) 2019-10-13T09:54:44.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:44.792+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.796213+0000) 2019-10-13T09:54:44.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:44.792+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.796916+0000) 2019-10-13T09:54:44.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:44.792+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.796985+0000) 2019-10-13T09:54:44.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:44.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.797331+0000) 2019-10-13T09:54:44.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:44.794+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.797751+0000) 2019-10-13T09:54:44.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:44.796+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.798433+0000) 2019-10-13T09:54:44.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:44.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:44.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:44.798+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:44.803917+0000) 2019-10-13T09:54:47.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:47.680+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:47.681935+0000) 2019-10-13T09:54:48.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:48.448+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.453270+0000) 2019-10-13T09:54:48.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:48.626+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:48.629869+0000) 2019-10-13T09:54:49.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:49.800+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.803249+0000) 2019-10-13T09:54:49.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:49.800+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.803954+0000) 2019-10-13T09:54:49.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:49.802+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.804352+0000) 2019-10-13T09:54:49.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:49.804+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.806559+0000) 2019-10-13T09:54:49.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:49.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:49.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:49.804+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.807597+0000) 2019-10-13T09:54:49.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:49.804+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.808152+0000) 2019-10-13T09:54:49.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:49.804+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.808007+0000) 2019-10-13T09:54:49.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:49.810+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:49.812749+0000) 2019-10-13T09:54:51.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:51.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:51.156326+0000) 2019-10-13T09:54:51.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:51.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:51.704961+0000) 2019-10-13T09:54:54.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:54.016+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.020217+0000) 2019-10-13T09:54:54.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:54.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.307863+0000) 2019-10-13T09:54:54.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:54.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.430057+0000) 2019-10-13T09:54:54.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:54.808+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.810565+0000) 2019-10-13T09:54:54.812 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:54.808+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.811049+0000) 2019-10-13T09:54:54.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:54.808+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.812019+0000) 2019-10-13T09:54:54.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:54.806+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.812010+0000) 2019-10-13T09:54:54.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:54.808+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.812642+0000) 2019-10-13T09:54:54.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:54.808+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.813089+0000) 2019-10-13T09:54:54.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:54.808+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.813372+0000) 2019-10-13T09:54:54.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:54.812+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:54.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:54.818+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:54.820275+0000) 2019-10-13T09:54:57.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:57.680+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:57.682149+0000) 2019-10-13T09:54:58.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:58.452+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.453466+0000) 2019-10-13T09:54:58.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:58.634+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:58.636508+0000) 2019-10-13T09:54:59.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:54:59.816+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.819015+0000) 2019-10-13T09:54:59.820 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:54:59.816+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.820276+0000) 2019-10-13T09:54:59.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:54:59.816+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.820620+0000) 2019-10-13T09:54:59.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:54:59.816+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.820948+0000) 2019-10-13T09:54:59.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:54:59.820+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.821816+0000) 2019-10-13T09:54:59.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:54:59.818+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.821762+0000) 2019-10-13T09:54:59.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:54:59.820+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.822882+0000) 2019-10-13T09:54:59.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:54:59.820+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:54:59.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:54:59.822+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:54:59.827504+0000) 2019-10-13T09:55:01.003 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:55:01.003 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:01.007 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:55:01.007 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:01.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:01.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:01.156542+0000) 2019-10-13T09:55:01.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:01.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:01.705173+0000) 2019-10-13T09:55:04.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:04.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.020452+0000) 2019-10-13T09:55:04.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:04.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.308086+0000) 2019-10-13T09:55:04.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:04.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.430270+0000) 2019-10-13T09:55:04.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:04.822+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.827959+0000) 2019-10-13T09:55:04.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:04.828+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.830267+0000) 2019-10-13T09:55:04.831 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:04.828+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.830622+0000) 2019-10-13T09:55:04.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:04.828+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.831260+0000) 2019-10-13T09:55:04.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:04.828+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.831452+0000) 2019-10-13T09:55:04.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:04.828+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.832504+0000) 2019-10-13T09:55:04.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:04.828+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.832871+0000) 2019-10-13T09:55:04.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:04.832+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:04.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:04.834+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:04.837331+0000) 2019-10-13T09:55:07.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:07.680+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:07.682383+0000) 2019-10-13T09:55:08.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:08.452+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.453703+0000) 2019-10-13T09:55:08.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:08.638+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:08.643032+0000) 2019-10-13T09:55:09.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:09.832+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.837129+0000) 2019-10-13T09:55:09.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:09.836+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.837825+0000) 2019-10-13T09:55:09.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:09.834+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.838174+0000) 2019-10-13T09:55:09.839 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:09.836+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.838947+0000) 2019-10-13T09:55:09.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:09.836+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.839303+0000) 2019-10-13T09:55:09.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:09.836+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.839764+0000) 2019-10-13T09:55:09.841 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:09.836+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.840317+0000) 2019-10-13T09:55:09.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:09.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:09.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:09.842+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:09.846564+0000) 2019-10-13T09:55:11.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:11.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:11.156760+0000) 2019-10-13T09:55:11.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:11.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:11.705416+0000) 2019-10-13T09:55:14.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:14.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.020607+0000) 2019-10-13T09:55:14.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:14.304+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.308287+0000) 2019-10-13T09:55:14.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:14.428+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.430479+0000) 2019-10-13T09:55:14.846 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:14.840+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.845653+0000) 2019-10-13T09:55:14.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:14.844+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.846395+0000) 2019-10-13T09:55:14.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:14.844+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.846745+0000) 2019-10-13T09:55:14.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:14.844+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.847127+0000) 2019-10-13T09:55:14.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:14.842+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.847810+0000) 2019-10-13T09:55:14.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:14.844+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.848583+0000) 2019-10-13T09:55:14.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:14.844+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.849276+0000) 2019-10-13T09:55:14.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:14.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:14.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:14.850+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:14.854190+0000) 2019-10-13T09:55:17.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:17.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:17.682611+0000) 2019-10-13T09:55:18.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:18.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.453941+0000) 2019-10-13T09:55:18.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:18.646+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:18.649116+0000) 2019-10-13T09:55:19.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:19.850+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.853916+0000) 2019-10-13T09:55:19.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:19.851+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.854673+0000) 2019-10-13T09:55:19.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:19.851+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.855131+0000) 2019-10-13T09:55:19.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:19.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.855742+0000) 2019-10-13T09:55:19.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:19.851+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.855406+0000) 2019-10-13T09:55:19.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:19.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.856525+0000) 2019-10-13T09:55:19.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:19.851+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.856964+0000) 2019-10-13T09:55:19.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:19.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105173 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:19.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:19.858+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:19.862809+0000) 2019-10-13T09:55:21.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:21.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:21.156964+0000) 2019-10-13T09:55:21.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:21.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:21.705614+0000) 2019-10-13T09:55:24.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:24.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.020803+0000) 2019-10-13T09:55:24.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:24.303+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.308490+0000) 2019-10-13T09:55:24.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:24.427+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.430715+0000) 2019-10-13T09:55:24.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:24.859+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.861713+0000) 2019-10-13T09:55:24.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:24.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.863332+0000) 2019-10-13T09:55:24.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:24.859+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.863891+0000) 2019-10-13T09:55:24.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:24.862+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.864264+0000) 2019-10-13T09:55:24.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:24.859+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.865084+0000) 2019-10-13T09:55:24.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:24.859+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.865233+0000) 2019-10-13T09:55:24.866 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:24.863+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.865725+0000) 2019-10-13T09:55:24.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:24.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:24.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:24.866+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:24.872010+0000) 2019-10-13T09:55:27.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:27.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:27.682811+0000) 2019-10-13T09:55:28.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:28.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.454165+0000) 2019-10-13T09:55:28.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:28.654+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:28.655614+0000) 2019-10-13T09:55:29.871 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:29.867+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.870116+0000) 2019-10-13T09:55:29.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:29.866+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.870404+0000) 2019-10-13T09:55:29.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:29.867+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.871249+0000) 2019-10-13T09:55:29.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:29.867+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.873521+0000) 2019-10-13T09:55:29.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:29.867+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.873524+0000) 2019-10-13T09:55:29.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:29.871+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.874281+0000) 2019-10-13T09:55:29.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:29.871+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.873880+0000) 2019-10-13T09:55:29.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:29.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:29.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:29.878+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:29.881041+0000) 2019-10-13T09:55:31.142 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:55:31.142 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:31.146 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:55:31.146 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:55:31.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:31.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:31.157145+0000) 2019-10-13T09:55:31.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:31.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:31.705841+0000) 2019-10-13T09:55:34.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:34.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.021011+0000) 2019-10-13T09:55:34.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:34.303+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.308724+0000) 2019-10-13T09:55:34.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:34.427+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.430908+0000) 2019-10-13T09:55:34.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:34.874+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.877719+0000) 2019-10-13T09:55:34.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:34.875+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.878490+0000) 2019-10-13T09:55:34.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:34.875+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.880289+0000) 2019-10-13T09:55:34.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:34.875+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.880753+0000) 2019-10-13T09:55:34.882 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:34.879+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.881816+0000) 2019-10-13T09:55:34.883 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:34.879+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.882418+0000) 2019-10-13T09:55:34.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:34.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:34.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:34.879+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.883049+0000) 2019-10-13T09:55:34.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:34.886+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:34.889112+0000) 2019-10-13T09:55:37.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:37.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:37.683005+0000) 2019-10-13T09:55:38.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:38.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.454357+0000) 2019-10-13T09:55:38.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:38.658+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:38.662220+0000) 2019-10-13T09:55:39.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:39.879+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.885396+0000) 2019-10-13T09:55:39.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:39.883+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.886251+0000) 2019-10-13T09:55:39.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:39.883+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.886907+0000) 2019-10-13T09:55:39.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:39.882+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.887164+0000) 2019-10-13T09:55:39.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:39.883+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.888192+0000) 2019-10-13T09:55:39.889 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:39.883+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.888608+0000) 2019-10-13T09:55:39.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:39.883+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.889577+0000) 2019-10-13T09:55:39.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:39.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:39.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:39.890+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:39.894439+0000) 2019-10-13T09:55:41.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:41.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:41.157334+0000) 2019-10-13T09:55:41.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:41.702+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:41.706045+0000) 2019-10-13T09:55:44.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:44.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.021213+0000) 2019-10-13T09:55:44.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:44.307+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.308936+0000) 2019-10-13T09:55:44.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:44.427+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.431107+0000) 2019-10-13T09:55:44.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:44.891+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.894593+0000) 2019-10-13T09:55:44.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:44.889+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.894539+0000) 2019-10-13T09:55:44.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:44.891+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.896257+0000) 2019-10-13T09:55:44.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:44.891+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.896335+0000) 2019-10-13T09:55:44.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:44.891+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.896807+0000) 2019-10-13T09:55:44.898 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:44.891+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.897248+0000) 2019-10-13T09:55:44.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:44.895+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.898846+0000) 2019-10-13T09:55:44.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:44.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:44.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:44.897+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:44.901064+0000) 2019-10-13T09:55:47.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:47.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:47.683156+0000) 2019-10-13T09:55:48.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:48.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.454570+0000) 2019-10-13T09:55:48.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:48.665+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:48.668804+0000) 2019-10-13T09:55:49.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:49.899+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.902438+0000) 2019-10-13T09:55:49.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:49.899+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.904162+0000) 2019-10-13T09:55:49.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:49.901+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.904161+0000) 2019-10-13T09:55:49.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:49.903+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.905808+0000) 2019-10-13T09:55:49.907 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:49.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:49.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:49.903+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.907393+0000) 2019-10-13T09:55:49.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:49.903+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.908107+0000) 2019-10-13T09:55:49.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:49.903+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.908388+0000) 2019-10-13T09:55:49.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:49.909+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:49.912389+0000) 2019-10-13T09:55:51.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:51.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:51.157528+0000) 2019-10-13T09:55:51.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:51.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:51.706271+0000) 2019-10-13T09:55:54.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:54.015+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.021382+0000) 2019-10-13T09:55:54.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:54.307+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.309165+0000) 2019-10-13T09:55:54.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:54.427+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.431293+0000) 2019-10-13T09:55:54.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:54.907+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.910238+0000) 2019-10-13T09:55:54.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:54.907+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.910968+0000) 2019-10-13T09:55:54.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:54.907+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.913193+0000) 2019-10-13T09:55:54.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:54.909+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.913178+0000) 2019-10-13T09:55:54.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:54.911+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.914234+0000) 2019-10-13T09:55:54.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:54.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:54.916 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:54.911+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.914888+0000) 2019-10-13T09:55:54.916 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:54.911+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.915532+0000) 2019-10-13T09:55:54.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:54.917+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:54.921237+0000) 2019-10-13T09:55:57.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:57.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:57.683377+0000) 2019-10-13T09:55:58.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:58.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.454798+0000) 2019-10-13T09:55:58.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:58.669+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:58.675462+0000) 2019-10-13T09:55:59.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:55:59.915+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.917691+0000) 2019-10-13T09:55:59.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:55:59.915+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.918343+0000) 2019-10-13T09:55:59.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:55:59.915+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.919425+0000) 2019-10-13T09:55:59.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:55:59.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.919218+0000) 2019-10-13T09:55:59.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:55:59.915+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.920438+0000) 2019-10-13T09:55:59.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:55:59.915+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.920855+0000) 2019-10-13T09:55:59.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:55:59.915+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.921219+0000) 2019-10-13T09:55:59.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:55:59.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:55:59.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:55:59.921+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:55:59.924542+0000) 2019-10-13T09:56:01.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:01.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:01.157789+0000) 2019-10-13T09:56:01.246 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:56:01.247 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:01.250 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:56:01.250 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:01.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:01.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:01.706464+0000) 2019-10-13T09:56:04.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:04.019+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.021593+0000) 2019-10-13T09:56:04.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:04.307+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.309359+0000) 2019-10-13T09:56:04.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:04.427+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.431500+0000) 2019-10-13T09:56:04.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:04.923+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.927643+0000) 2019-10-13T09:56:04.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:04.923+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.927860+0000) 2019-10-13T09:56:04.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:04.921+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.927838+0000) 2019-10-13T09:56:04.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:04.923+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.928582+0000) 2019-10-13T09:56:04.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:04.923+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.928773+0000) 2019-10-13T09:56:04.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:04.923+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.929079+0000) 2019-10-13T09:56:04.931 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:04.923+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.929564+0000) 2019-10-13T09:56:04.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:04.927+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:04.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:04.929+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:04.933345+0000) 2019-10-13T09:56:07.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:07.679+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:07.683572+0000) 2019-10-13T09:56:08.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:08.451+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.454991+0000) 2019-10-13T09:56:08.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:08.677+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:08.682762+0000) 2019-10-13T09:56:09.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:09.931+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.935395+0000) 2019-10-13T09:56:09.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:09.929+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.935838+0000) 2019-10-13T09:56:09.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:09.931+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.935952+0000) 2019-10-13T09:56:09.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:09.931+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.936826+0000) 2019-10-13T09:56:09.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:09.931+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.936657+0000) 2019-10-13T09:56:09.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:09.931+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.937281+0000) 2019-10-13T09:56:09.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:09.935+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.938100+0000) 2019-10-13T09:56:09.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:09.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:09.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:09.937+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:09.943326+0000) 2019-10-13T09:56:11.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:11.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:11.157994+0000) 2019-10-13T09:56:11.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:11.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:11.706661+0000) 2019-10-13T09:56:14.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:14.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.021796+0000) 2019-10-13T09:56:14.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:14.302+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.309583+0000) 2019-10-13T09:56:14.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:14.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.431697+0000) 2019-10-13T09:56:14.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:14.938+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.943615+0000) 2019-10-13T09:56:14.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:14.938+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.943874+0000) 2019-10-13T09:56:14.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:14.937+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.943733+0000) 2019-10-13T09:56:14.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:14.938+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.944771+0000) 2019-10-13T09:56:14.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:14.938+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.945589+0000) 2019-10-13T09:56:14.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:14.942+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.946218+0000) 2019-10-13T09:56:14.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:14.942+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.947050+0000) 2019-10-13T09:56:14.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:14.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:14.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:14.945+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:14.951679+0000) 2019-10-13T09:56:17.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:17.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:17.683805+0000) 2019-10-13T09:56:18.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:18.454+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.455181+0000) 2019-10-13T09:56:18.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:18.685+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:18.689418+0000) 2019-10-13T09:56:19.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:19.946+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.951812+0000) 2019-10-13T09:56:19.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:19.945+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.951835+0000) 2019-10-13T09:56:19.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:19.950+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.954167+0000) 2019-10-13T09:56:19.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:19.950+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.954802+0000) 2019-10-13T09:56:19.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:19.950+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.955105+0000) 2019-10-13T09:56:19.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:19.950+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.955435+0000) 2019-10-13T09:56:19.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:19.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.955866+0000) 2019-10-13T09:56:19.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:19.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:19.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:19.957+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:19.961548+0000) 2019-10-13T09:56:21.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:21.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:21.158226+0000) 2019-10-13T09:56:21.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:21.701+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:21.706854+0000) 2019-10-13T09:56:24.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:24.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.022015+0000) 2019-10-13T09:56:24.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:24.306+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.309825+0000) 2019-10-13T09:56:24.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:24.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.431937+0000) 2019-10-13T09:56:24.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:24.954+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.959928+0000) 2019-10-13T09:56:24.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:24.954+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.960795+0000) 2019-10-13T09:56:24.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:24.954+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.960488+0000) 2019-10-13T09:56:24.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:24.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.961907+0000) 2019-10-13T09:56:24.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:24.958+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.962837+0000) 2019-10-13T09:56:24.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:24.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:24.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:24.958+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.965252+0000) 2019-10-13T09:56:24.966 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:24.958+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.965541+0000) 2019-10-13T09:56:24.973 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:24.965+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:24.972015+0000) 2019-10-13T09:56:27.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:27.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:27.684003+0000) 2019-10-13T09:56:28.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:28.450+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.455375+0000) 2019-10-13T09:56:28.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:28.693+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:28.695932+0000) 2019-10-13T09:56:29.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:29.962+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.967765+0000) 2019-10-13T09:56:29.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:29.962+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.968255+0000) 2019-10-13T09:56:29.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:29.962+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.968894+0000) 2019-10-13T09:56:29.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:29.965+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.969215+0000) 2019-10-13T09:56:29.971 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:29.966+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.970327+0000) 2019-10-13T09:56:29.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:29.966+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.971069+0000) 2019-10-13T09:56:29.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:29.966+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.971441+0000) 2019-10-13T09:56:29.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:29.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:29.977 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:29.973+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:29.976563+0000) 2019-10-13T09:56:31.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:31.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:31.158434+0000) 2019-10-13T09:56:31.358 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:56:31.358 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:31.362 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:56:31.362 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:56:31.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:31.701+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:31.707052+0000) 2019-10-13T09:56:34.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:34.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.022244+0000) 2019-10-13T09:56:34.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:34.306+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.310028+0000) 2019-10-13T09:56:34.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:34.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.432166+0000) 2019-10-13T09:56:34.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:34.969+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.975747+0000) 2019-10-13T09:56:34.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:34.970+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.976944+0000) 2019-10-13T09:56:34.978 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:34.970+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.977392+0000) 2019-10-13T09:56:34.978 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:34.970+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.977506+0000) 2019-10-13T09:56:34.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:34.974+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.978260+0000) 2019-10-13T09:56:34.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:34.974+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.979021+0000) 2019-10-13T09:56:34.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:34.974+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.980162+0000) 2019-10-13T09:56:34.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:34.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:34.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:34.981+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:34.984481+0000) 2019-10-13T09:56:37.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:37.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:37.684203+0000) 2019-10-13T09:56:38.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:38.454+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.455576+0000) 2019-10-13T09:56:38.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:38.697+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:38.702085+0000) 2019-10-13T09:56:39.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:39.978+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.984060+0000) 2019-10-13T09:56:39.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:39.978+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.985210+0000) 2019-10-13T09:56:39.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:39.978+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.985556+0000) 2019-10-13T09:56:39.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:39.981+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.986510+0000) 2019-10-13T09:56:39.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:39.982+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.987428+0000) 2019-10-13T09:56:39.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:39.982+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.987984+0000) 2019-10-13T09:56:39.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:39.982+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.988364+0000) 2019-10-13T09:56:39.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:39.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105676 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:39.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:39.989+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:39.993871+0000) 2019-10-13T09:56:41.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:41.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:41.158666+0000) 2019-10-13T09:56:41.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:41.701+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:41.707259+0000) 2019-10-13T09:56:44.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:44.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.022462+0000) 2019-10-13T09:56:44.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:44.306+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.310231+0000) 2019-10-13T09:56:44.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:44.426+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.432360+0000) 2019-10-13T09:56:44.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:44.986+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.993051+0000) 2019-10-13T09:56:44.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:44.990+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.993843+0000) 2019-10-13T09:56:44.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:44.986+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.993671+0000) 2019-10-13T09:56:44.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:44.990+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.994918+0000) 2019-10-13T09:56:44.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:44.989+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.995124+0000) 2019-10-13T09:56:44.997 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:44.990+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.995593+0000) 2019-10-13T09:56:44.997 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:44.990+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:44.996139+0000) 2019-10-13T09:56:44.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:44.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:45.003 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:44.997+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:45.001845+0000) 2019-10-13T09:56:47.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:47.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:47.684411+0000) 2019-10-13T09:56:48.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:48.454+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.455761+0000) 2019-10-13T09:56:48.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:48.704+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:48.708578+0000) 2019-10-13T09:56:50.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:49.994+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.000948+0000) 2019-10-13T09:56:50.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:49.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.001352+0000) 2019-10-13T09:56:50.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:49.998+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.002665+0000) 2019-10-13T09:56:50.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:49.998+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.003011+0000) 2019-10-13T09:56:50.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:49.996+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.003301+0000) 2019-10-13T09:56:50.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:49.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.003494+0000) 2019-10-13T09:56:50.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:49.998+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.004725+0000) 2019-10-13T09:56:50.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:50.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:50.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:50.008+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:50.014084+0000) 2019-10-13T09:56:51.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:51.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:51.158882+0000) 2019-10-13T09:56:51.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:51.704+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:51.707466+0000) 2019-10-13T09:56:54.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:54.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:54.022677+0000) 2019-10-13T09:56:54.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:54.310+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:54.310443+0000) 2019-10-13T09:56:54.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:54.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:54.432580+0000) 2019-10-13T09:56:55.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:56:55.006+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.010053+0000) 2019-10-13T09:56:55.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:55.006+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.010897+0000) 2019-10-13T09:56:55.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:55.006+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.011412+0000) 2019-10-13T09:56:55.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:56:55.004+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.011204+0000) 2019-10-13T09:56:55.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:56:55.006+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.013038+0000) 2019-10-13T09:56:55.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:56:55.010+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.014561+0000) 2019-10-13T09:56:55.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:56:55.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:56:55.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:56:55.010+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.015062+0000) 2019-10-13T09:56:55.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:55.016+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:55.022657+0000) 2019-10-13T09:56:57.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:56:57.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:57.684606+0000) 2019-10-13T09:56:58.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:56:58.454+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.455958+0000) 2019-10-13T09:56:58.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:56:58.712+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:56:58.715218+0000) 2019-10-13T09:57:00.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:00.010+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.017236+0000) 2019-10-13T09:57:00.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:00.014+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.018444+0000) 2019-10-13T09:57:00.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:00.016+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.020362+0000) 2019-10-13T09:57:00.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:00.014+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.021153+0000) 2019-10-13T09:57:00.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:00.014+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.021549+0000) 2019-10-13T09:57:00.023 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:00.018+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.022326+0000) 2019-10-13T09:57:00.023 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:00.018+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.022379+0000) 2019-10-13T09:57:00.023 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:00.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:00.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:00.024+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:00.028242+0000) 2019-10-13T09:57:01.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:01.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:01.159135+0000) 2019-10-13T09:57:01.431 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:57:01.431 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:01.434 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:57:01.435 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:01.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:01.704+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:01.707695+0000) 2019-10-13T09:57:04.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:04.018+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:04.022910+0000) 2019-10-13T09:57:04.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:04.306+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:04.310663+0000) 2019-10-13T09:57:04.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:04.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:04.432794+0000) 2019-10-13T09:57:05.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:05.022+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.026625+0000) 2019-10-13T09:57:05.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:05.022+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.029074+0000) 2019-10-13T09:57:05.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:05.026+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.031370+0000) 2019-10-13T09:57:05.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:05.024+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.031311+0000) 2019-10-13T09:57:05.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:05.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:05.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:05.026+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.032687+0000) 2019-10-13T09:57:05.035 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:05.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.033630+0000) 2019-10-13T09:57:05.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:05.030+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.034058+0000) 2019-10-13T09:57:05.036 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:05.028+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:05.035514+0000) 2019-10-13T09:57:07.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:07.678+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:07.684836+0000) 2019-10-13T09:57:08.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:08.454+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.456144+0000) 2019-10-13T09:57:08.723 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:08.720+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:08.721798+0000) 2019-10-13T09:57:10.036 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:10.029+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.035411+0000) 2019-10-13T09:57:10.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:10.029+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.036173+0000) 2019-10-13T09:57:10.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:10.032+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.036246+0000) 2019-10-13T09:57:10.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:10.029+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.036875+0000) 2019-10-13T09:57:10.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:10.029+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.037189+0000) 2019-10-13T09:57:10.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:10.033+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.038030+0000) 2019-10-13T09:57:10.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:10.033+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.038212+0000) 2019-10-13T09:57:10.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:10.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:10.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:10.040+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:10.044854+0000) 2019-10-13T09:57:11.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:11.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:11.159337+0000) 2019-10-13T09:57:11.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:11.704+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:11.707895+0000) 2019-10-13T09:57:14.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:14.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:14.023140+0000) 2019-10-13T09:57:14.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:14.305+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:14.310876+0000) 2019-10-13T09:57:14.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:14.429+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:14.432996+0000) 2019-10-13T09:57:15.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:15.037+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.044988+0000) 2019-10-13T09:57:15.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:15.037+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.045640+0000) 2019-10-13T09:57:15.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:15.041+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.047552+0000) 2019-10-13T09:57:15.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:15.041+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.047847+0000) 2019-10-13T09:57:15.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:15.041+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.048583+0000) 2019-10-13T09:57:15.050 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:15.045+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.049860+0000) 2019-10-13T09:57:15.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:15.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:15.051 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:15.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.050339+0000) 2019-10-13T09:57:15.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:15.048+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:15.055674+0000) 2019-10-13T09:57:17.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:17.677+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:17.685028+0000) 2019-10-13T09:57:18.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:18.453+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.456341+0000) 2019-10-13T09:57:18.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:18.724+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:18.728338+0000) 2019-10-13T09:57:20.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:20.049+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.053215+0000) 2019-10-13T09:57:20.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:20.049+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.053636+0000) 2019-10-13T09:57:20.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:20.049+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.055460+0000) 2019-10-13T09:57:20.058 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:20.049+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.057289+0000) 2019-10-13T09:57:20.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:20.053+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.057882+0000) 2019-10-13T09:57:20.059 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:20.053+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.058291+0000) 2019-10-13T09:57:20.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:20.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:20.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:20.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.058650+0000) 2019-10-13T09:57:20.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:20.056+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:20.063699+0000) 2019-10-13T09:57:21.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:21.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:21.159538+0000) 2019-10-13T09:57:21.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:21.704+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:21.708131+0000) 2019-10-13T09:57:24.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:24.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:24.023337+0000) 2019-10-13T09:57:24.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:24.309+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:24.311074+0000) 2019-10-13T09:57:24.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:24.429+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:24.433202+0000) 2019-10-13T09:57:25.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:25.057+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.061612+0000) 2019-10-13T09:57:25.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:25.061+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.061970+0000) 2019-10-13T09:57:25.064 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:25.061+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.063669+0000) 2019-10-13T09:57:25.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:25.061+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.064486+0000) 2019-10-13T09:57:25.067 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:25.065+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.067071+0000) 2019-10-13T09:57:25.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:25.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:25.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:25.065+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.067589+0000) 2019-10-13T09:57:25.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:25.060+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.067503+0000) 2019-10-13T09:57:25.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:25.068+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:25.072646+0000) 2019-10-13T09:57:27.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:27.681+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:27.685223+0000) 2019-10-13T09:57:28.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:28.453+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.456560+0000) 2019-10-13T09:57:28.736 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:28.732+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:28.734799+0000) 2019-10-13T09:57:30.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:30.069+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.070942+0000) 2019-10-13T09:57:30.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:30.069+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.072380+0000) 2019-10-13T09:57:30.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:30.068+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.073968+0000) 2019-10-13T09:57:30.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:30.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.075435+0000) 2019-10-13T09:57:30.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:30.073+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.075644+0000) 2019-10-13T09:57:30.077 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:30.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:30.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:30.073+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.075928+0000) 2019-10-13T09:57:30.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:30.073+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.076406+0000) 2019-10-13T09:57:30.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:30.072+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:30.076202+0000) 2019-10-13T09:57:31.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:31.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:31.159757+0000) 2019-10-13T09:57:31.546 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:57:31.546 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:31.550 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:57:31.551 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:57:31.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:31.704+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:31.708284+0000) 2019-10-13T09:57:34.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:34.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:34.023556+0000) 2019-10-13T09:57:34.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:34.309+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:34.311289+0000) 2019-10-13T09:57:34.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:34.429+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:34.433416+0000) 2019-10-13T09:57:35.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:35.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.079863+0000) 2019-10-13T09:57:35.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:35.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.080339+0000) 2019-10-13T09:57:35.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:35.081+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.081893+0000) 2019-10-13T09:57:35.083 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:35.076+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.082006+0000) 2019-10-13T09:57:35.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:35.081+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.082402+0000) 2019-10-13T09:57:35.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:35.081+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.083849+0000) 2019-10-13T09:57:35.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:35.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:35.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:35.081+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.084291+0000) 2019-10-13T09:57:35.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:35.084+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:35.089007+0000) 2019-10-13T09:57:37.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:37.681+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:37.685417+0000) 2019-10-13T09:57:38.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:38.453+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.456754+0000) 2019-10-13T09:57:38.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:38.740+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:38.740782+0000) 2019-10-13T09:57:40.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:40.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.087814+0000) 2019-10-13T09:57:40.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:40.089+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.090690+0000) 2019-10-13T09:57:40.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:40.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:40.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:40.088+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.092421+0000) 2019-10-13T09:57:40.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:40.089+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.093290+0000) 2019-10-13T09:57:40.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:40.093+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.094164+0000) 2019-10-13T09:57:40.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:40.093+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.094554+0000) 2019-10-13T09:57:40.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:40.089+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.093622+0000) 2019-10-13T09:57:40.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:40.092+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:40.097842+0000) 2019-10-13T09:57:41.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:41.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:41.159991+0000) 2019-10-13T09:57:41.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:41.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:41.708480+0000) 2019-10-13T09:57:44.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:44.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:44.023715+0000) 2019-10-13T09:57:44.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:44.309+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:44.311521+0000) 2019-10-13T09:57:44.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:44.429+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:44.433617+0000) 2019-10-13T09:57:45.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:45.093+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.096647+0000) 2019-10-13T09:57:45.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:45.093+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.096652+0000) 2019-10-13T09:57:45.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:45.093+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.097072+0000) 2019-10-13T09:57:45.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:45.097+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.098052+0000) 2019-10-13T09:57:45.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:45.092+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.098031+0000) 2019-10-13T09:57:45.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:45.097+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.098827+0000) 2019-10-13T09:57:45.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:45.097+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.099554+0000) 2019-10-13T09:57:45.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:45.096+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:45.101652+0000) 2019-10-13T09:57:45.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:45.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:47.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:47.681+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:47.685648+0000) 2019-10-13T09:57:48.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:48.453+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.456951+0000) 2019-10-13T09:57:48.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:48.744+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:48.747308+0000) 2019-10-13T09:57:50.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:50.101+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.103120+0000) 2019-10-13T09:57:50.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:50.101+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.104614+0000) 2019-10-13T09:57:50.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:50.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.104496+0000) 2019-10-13T09:57:50.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:50.101+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.105295+0000) 2019-10-13T09:57:50.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:50.105+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.105952+0000) 2019-10-13T09:57:50.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:50.104+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.105375+0000) 2019-10-13T09:57:50.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:50.105+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.106808+0000) 2019-10-13T09:57:50.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:50.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:50.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:50.108+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:50.111719+0000) 2019-10-13T09:57:51.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:51.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:51.160192+0000) 2019-10-13T09:57:51.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:51.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:51.708720+0000) 2019-10-13T09:57:54.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:54.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:54.023925+0000) 2019-10-13T09:57:54.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:54.309+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:54.311725+0000) 2019-10-13T09:57:54.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:54.433+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:54.433819+0000) 2019-10-13T09:57:55.112 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:55.109+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.111850+0000) 2019-10-13T09:57:55.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:57:55.107+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.111919+0000) 2019-10-13T09:57:55.114 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:55.109+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.112329+0000) 2019-10-13T09:57:55.114 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:57:55.109+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.113057+0000) 2019-10-13T09:57:55.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:57:55.109+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.113712+0000) 2019-10-13T09:57:55.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:57:55.113+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.114099+0000) 2019-10-13T09:57:55.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:57:55.113+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.114531+0000) 2019-10-13T09:57:55.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:57:55.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:57:55.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:55.115+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:55.119260+0000) 2019-10-13T09:57:57.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:57:57.685+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:57.685838+0000) 2019-10-13T09:57:58.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:57:58.453+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.457188+0000) 2019-10-13T09:57:58.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:57:58.751+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:57:58.753800+0000) 2019-10-13T09:58:00.120 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:00.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.118975+0000) 2019-10-13T09:58:00.124 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:00.117+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.121229+0000) 2019-10-13T09:58:00.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:00.121+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.121794+0000) 2019-10-13T09:58:00.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:00.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.122143+0000) 2019-10-13T09:58:00.125 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:00.121+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.122861+0000) 2019-10-13T09:58:00.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:00.121+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.123159+0000) 2019-10-13T09:58:00.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:00.121+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.123567+0000) 2019-10-13T09:58:00.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:00.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:00.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:00.123+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:00.126004+0000) 2019-10-13T09:58:01.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:01.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:01.160392+0000) 2019-10-13T09:58:01.676 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:58:01.676 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:01.680 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:58:01.680 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:01.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:01.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:01.708928+0000) 2019-10-13T09:58:04.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:04.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:04.024144+0000) 2019-10-13T09:58:04.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:04.309+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:04.311928+0000) 2019-10-13T09:58:04.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:04.433+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:04.434041+0000) 2019-10-13T09:58:05.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:05.124+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.127447+0000) 2019-10-13T09:58:05.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:05.124+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.128043+0000) 2019-10-13T09:58:05.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:05.124+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.129422+0000) 2019-10-13T09:58:05.136 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:05.128+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.130326+0000) 2019-10-13T09:58:05.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:05.128+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.130835+0000) 2019-10-13T09:58:05.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:05.128+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.130975+0000) 2019-10-13T09:58:05.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:05.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:05.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:05.127+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.128483+0000) 2019-10-13T09:58:05.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:05.135+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:05.138140+0000) 2019-10-13T09:58:07.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:07.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:07.685997+0000) 2019-10-13T09:58:08.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:08.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.457381+0000) 2019-10-13T09:58:08.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:08.759+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:08.760274+0000) 2019-10-13T09:58:10.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:10.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.136873+0000) 2019-10-13T09:58:10.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:10.135+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.138137+0000) 2019-10-13T09:58:10.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:10.136+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.138921+0000) 2019-10-13T09:58:10.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:10.136+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.140409+0000) 2019-10-13T09:58:10.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:10.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:10.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:10.136+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.140917+0000) 2019-10-13T09:58:10.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:10.136+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.141411+0000) 2019-10-13T09:58:10.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:10.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.142095+0000) 2019-10-13T09:58:10.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:10.143+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:10.147553+0000) 2019-10-13T09:58:11.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:11.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:11.160609+0000) 2019-10-13T09:58:11.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:11.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:11.709119+0000) 2019-10-13T09:58:14.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:14.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:14.024344+0000) 2019-10-13T09:58:14.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:14.308+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:14.312136+0000) 2019-10-13T09:58:14.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:14.432+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:14.434239+0000) 2019-10-13T09:58:15.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:15.140+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.143947+0000) 2019-10-13T09:58:15.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:15.140+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.145135+0000) 2019-10-13T09:58:15.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:15.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.145114+0000) 2019-10-13T09:58:15.146 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:15.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.145143+0000) 2019-10-13T09:58:15.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:15.144+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.146292+0000) 2019-10-13T09:58:15.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:15.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.146503+0000) 2019-10-13T09:58:15.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:15.144+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.147426+0000) 2019-10-13T09:58:15.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:15.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106275 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:15.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:15.151+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:15.152357+0000) 2019-10-13T09:58:17.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:17.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:17.686200+0000) 2019-10-13T09:58:18.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:18.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.457586+0000) 2019-10-13T09:58:18.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:18.763+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:18.766828+0000) 2019-10-13T09:58:20.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:20.148+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.151631+0000) 2019-10-13T09:58:20.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:20.147+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.152001+0000) 2019-10-13T09:58:20.154 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:20.148+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.152894+0000) 2019-10-13T09:58:20.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:20.148+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.153620+0000) 2019-10-13T09:58:20.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:20.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.154188+0000) 2019-10-13T09:58:20.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:20.152+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.155316+0000) 2019-10-13T09:58:20.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:20.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.155559+0000) 2019-10-13T09:58:20.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:20.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106305 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:20.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:20.155+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:20.159180+0000) 2019-10-13T09:58:21.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:21.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:21.160829+0000) 2019-10-13T09:58:21.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:21.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:21.709332+0000) 2019-10-13T09:58:24.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:24.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:24.024560+0000) 2019-10-13T09:58:24.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:24.308+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:24.312397+0000) 2019-10-13T09:58:24.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:24.432+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:24.434451+0000) 2019-10-13T09:58:25.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:25.155+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.160029+0000) 2019-10-13T09:58:25.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:25.156+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.160965+0000) 2019-10-13T09:58:25.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:25.156+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.161704+0000) 2019-10-13T09:58:25.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:25.160+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.162199+0000) 2019-10-13T09:58:25.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:25.160+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.162870+0000) 2019-10-13T09:58:25.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:25.160+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.163238+0000) 2019-10-13T09:58:25.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:25.160+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.163253+0000) 2019-10-13T09:58:25.166 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:25.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:25.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:25.171+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:25.172541+0000) 2019-10-13T09:58:27.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:27.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:27.686400+0000) 2019-10-13T09:58:28.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:28.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.457733+0000) 2019-10-13T09:58:28.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:28.771+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:28.773439+0000) 2019-10-13T09:58:30.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:30.164+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.168391+0000) 2019-10-13T09:58:30.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:30.164+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.168978+0000) 2019-10-13T09:58:30.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:30.168+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.170320+0000) 2019-10-13T09:58:30.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:30.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.170684+0000) 2019-10-13T09:58:30.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:30.168+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.171360+0000) 2019-10-13T09:58:30.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:30.168+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.171293+0000) 2019-10-13T09:58:30.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:30.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:30.176 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:30.167+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.169279+0000) 2019-10-13T09:58:30.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:30.171+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:30.175760+0000) 2019-10-13T09:58:31.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:31.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.161009+0000) 2019-10-13T09:58:31.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:31.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:31.709531+0000) 2019-10-13T09:58:31.824 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:58:31.824 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:31.828 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:58:31.828 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:58:34.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:34.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:34.024778+0000) 2019-10-13T09:58:34.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:34.308+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:34.312601+0000) 2019-10-13T09:58:34.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:34.432+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:34.434651+0000) 2019-10-13T09:58:35.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:35.172+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.177330+0000) 2019-10-13T09:58:35.179 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:35.175+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.177448+0000) 2019-10-13T09:58:35.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:35.176+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.178565+0000) 2019-10-13T09:58:35.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:35.176+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.181092+0000) 2019-10-13T09:58:35.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:35.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:35.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:35.180+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.183032+0000) 2019-10-13T09:58:35.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:35.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.183893+0000) 2019-10-13T09:58:35.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:35.180+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.184172+0000) 2019-10-13T09:58:35.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:35.183+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:35.187922+0000) 2019-10-13T09:58:37.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:37.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:37.686633+0000) 2019-10-13T09:58:38.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:38.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.457965+0000) 2019-10-13T09:58:38.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:38.783+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:38.784616+0000) 2019-10-13T09:58:40.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:40.184+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.186189+0000) 2019-10-13T09:58:40.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:40.184+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.188564+0000) 2019-10-13T09:58:40.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:40.184+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.188923+0000) 2019-10-13T09:58:40.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:40.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.190883+0000) 2019-10-13T09:58:40.192 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:40.187+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.190679+0000) 2019-10-13T09:58:40.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:40.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:40.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:40.188+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.191369+0000) 2019-10-13T09:58:40.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:40.188+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.191675+0000) 2019-10-13T09:58:40.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:40.191+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:40.192413+0000) 2019-10-13T09:58:41.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:41.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.161203+0000) 2019-10-13T09:58:41.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:41.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:41.709765+0000) 2019-10-13T09:58:44.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:44.024+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:44.024992+0000) 2019-10-13T09:58:44.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:44.312+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:44.312833+0000) 2019-10-13T09:58:44.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:44.432+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:44.434854+0000) 2019-10-13T09:58:45.195 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:45.192+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.194385+0000) 2019-10-13T09:58:45.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:45.191+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.194365+0000) 2019-10-13T09:58:45.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:45.192+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.195407+0000) 2019-10-13T09:58:45.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:45.192+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.196541+0000) 2019-10-13T09:58:45.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:45.192+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.196776+0000) 2019-10-13T09:58:45.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:45.196+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.198460+0000) 2019-10-13T09:58:45.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:45.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106465 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:45.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:45.196+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.200750+0000) 2019-10-13T09:58:45.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:45.199+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:45.201697+0000) 2019-10-13T09:58:47.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:47.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:47.686841+0000) 2019-10-13T09:58:48.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:48.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.458197+0000) 2019-10-13T09:58:48.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:48.791+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:48.793816+0000) 2019-10-13T09:58:50.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:50.199+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.203483+0000) 2019-10-13T09:58:50.205 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:50.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.204252+0000) 2019-10-13T09:58:50.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:50.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.204894+0000) 2019-10-13T09:58:50.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:50.200+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.205172+0000) 2019-10-13T09:58:50.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:50.204+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.206052+0000) 2019-10-13T09:58:50.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:50.204+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.206742+0000) 2019-10-13T09:58:50.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:50.204+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.207103+0000) 2019-10-13T09:58:50.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:50.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:50.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:50.211+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:50.213698+0000) 2019-10-13T09:58:51.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:51.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.161381+0000) 2019-10-13T09:58:51.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:51.707+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:51.709982+0000) 2019-10-13T09:58:54.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:54.020+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:54.025241+0000) 2019-10-13T09:58:54.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:54.312+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:54.313056+0000) 2019-10-13T09:58:54.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:54.432+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:54.435073+0000) 2019-10-13T09:58:55.212 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:55.208+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.211263+0000) 2019-10-13T09:58:55.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:58:55.211+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.212421+0000) 2019-10-13T09:58:55.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:58:55.208+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.212562+0000) 2019-10-13T09:58:55.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:58:55.212+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.214827+0000) 2019-10-13T09:58:55.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:55.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.215132+0000) 2019-10-13T09:58:55.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:58:55.212+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.215526+0000) 2019-10-13T09:58:55.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:58:55.212+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.215827+0000) 2019-10-13T09:58:55.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:58:55.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106527 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:58:55.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:55.219+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:55.221889+0000) 2019-10-13T09:58:57.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:58:57.684+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:57.687042+0000) 2019-10-13T09:58:58.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:58:58.456+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.458428+0000) 2019-10-13T09:58:58.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:58:58.799+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:58:58.800432+0000) 2019-10-13T09:59:00.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:00.216+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.219603+0000) 2019-10-13T09:59:00.221 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:00.216+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.219988+0000) 2019-10-13T09:59:00.222 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:00.218+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.220274+0000) 2019-10-13T09:59:00.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:00.216+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.220880+0000) 2019-10-13T09:59:00.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:00.220+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.223231+0000) 2019-10-13T09:59:00.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:00.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:00.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:00.220+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.224970+0000) 2019-10-13T09:59:00.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:00.220+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.225170+0000) 2019-10-13T09:59:00.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:00.222+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:00.227050+0000) 2019-10-13T09:59:01.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:01.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.161577+0000) 2019-10-13T09:59:01.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:01.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:01.710202+0000) 2019-10-13T09:59:01.947 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:59:01.947 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:01.956 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:59:01.956 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:04.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:04.019+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:04.025447+0000) 2019-10-13T09:59:04.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:04.307+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:04.313285+0000) 2019-10-13T09:59:04.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:04.431+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:04.435290+0000) 2019-10-13T09:59:05.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:05.223+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.227938+0000) 2019-10-13T09:59:05.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:05.223+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.228308+0000) 2019-10-13T09:59:05.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:05.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.228912+0000) 2019-10-13T09:59:05.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:05.226+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.229456+0000) 2019-10-13T09:59:05.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:05.223+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.229618+0000) 2019-10-13T09:59:05.231 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:05.227+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.230380+0000) 2019-10-13T09:59:05.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:05.227+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.231055+0000) 2019-10-13T09:59:05.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:05.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:05.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:05.234+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:05.236824+0000) 2019-10-13T09:59:07.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:07.683+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:07.687255+0000) 2019-10-13T09:59:08.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:08.455+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.458663+0000) 2019-10-13T09:59:08.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:08.802+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:08.806825+0000) 2019-10-13T09:59:10.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:10.231+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.236024+0000) 2019-10-13T09:59:10.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:10.235+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.238260+0000) 2019-10-13T09:59:10.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:10.235+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.240080+0000) 2019-10-13T09:59:10.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:10.235+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.240444+0000) 2019-10-13T09:59:10.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:10.235+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.240827+0000) 2019-10-13T09:59:10.242 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:10.235+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:10.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:10.238+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.241075+0000) 2019-10-13T09:59:10.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:10.239+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.242264+0000) 2019-10-13T09:59:10.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:10.242+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:10.246925+0000) 2019-10-13T09:59:11.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:11.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.161799+0000) 2019-10-13T09:59:11.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:11.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:11.710391+0000) 2019-10-13T09:59:14.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:14.019+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:14.025706+0000) 2019-10-13T09:59:14.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:14.311+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:14.313481+0000) 2019-10-13T09:59:14.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:14.431+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:14.435490+0000) 2019-10-13T09:59:15.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:15.239+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.244178+0000) 2019-10-13T09:59:15.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:15.239+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.244797+0000) 2019-10-13T09:59:15.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:15.239+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.245227+0000) 2019-10-13T09:59:15.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:15.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.245830+0000) 2019-10-13T09:59:15.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:15.243+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.246115+0000) 2019-10-13T09:59:15.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:15.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.246857+0000) 2019-10-13T09:59:15.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:15.243+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.247866+0000) 2019-10-13T09:59:15.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:15.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:15.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:15.250+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:15.252588+0000) 2019-10-13T09:59:17.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:17.683+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:17.687454+0000) 2019-10-13T09:59:18.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:18.455+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.458871+0000) 2019-10-13T09:59:18.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:18.810+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:18.813298+0000) 2019-10-13T09:59:20.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:20.247+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.252659+0000) 2019-10-13T09:59:20.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:20.247+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.252999+0000) 2019-10-13T09:59:20.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:20.247+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.253649+0000) 2019-10-13T09:59:20.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:20.251+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.253997+0000) 2019-10-13T09:59:20.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:20.251+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.254396+0000) 2019-10-13T09:59:20.256 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:20.250+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.255502+0000) 2019-10-13T09:59:20.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:20.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:20.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:20.255+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.258072+0000) 2019-10-13T09:59:20.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:20.258+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:20.262186+0000) 2019-10-13T09:59:21.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:21.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.161993+0000) 2019-10-13T09:59:21.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:21.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:21.710616+0000) 2019-10-13T09:59:24.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:24.023+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:24.025965+0000) 2019-10-13T09:59:24.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:24.311+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:24.313715+0000) 2019-10-13T09:59:24.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:24.435+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:24.435709+0000) 2019-10-13T09:59:25.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:25.255+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.261050+0000) 2019-10-13T09:59:25.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:25.255+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.261486+0000) 2019-10-13T09:59:25.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:25.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.262570+0000) 2019-10-13T09:59:25.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:25.259+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.262419+0000) 2019-10-13T09:59:25.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:25.259+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.265206+0000) 2019-10-13T09:59:25.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:25.259+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.265591+0000) 2019-10-13T09:59:25.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:25.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:25.267 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:25.262+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.266307+0000) 2019-10-13T09:59:25.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:25.270+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:25.273172+0000) 2019-10-13T09:59:27.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:27.683+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:27.687603+0000) 2019-10-13T09:59:28.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:28.455+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.459065+0000) 2019-10-13T09:59:28.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:28.818+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:28.819852+0000) 2019-10-13T09:59:30.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:30.263+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.269536+0000) 2019-10-13T09:59:30.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:30.266+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.269958+0000) 2019-10-13T09:59:30.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:30.267+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.270059+0000) 2019-10-13T09:59:30.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:30.267+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.271290+0000) 2019-10-13T09:59:30.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:30.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.273119+0000) 2019-10-13T09:59:30.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:30.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:30.277 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:30.271+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.276359+0000) 2019-10-13T09:59:30.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:30.271+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.276941+0000) 2019-10-13T09:59:30.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:30.278+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:30.281111+0000) 2019-10-13T09:59:31.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:31.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.162172+0000) 2019-10-13T09:59:31.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:31.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:31.710802+0000) 2019-10-13T09:59:32.268 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T09:59:32.268 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:32.273 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T09:59:32.273 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T09:59:34.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:34.023+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:34.026163+0000) 2019-10-13T09:59:34.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:34.311+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:34.313940+0000) 2019-10-13T09:59:34.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:34.431+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:34.435935+0000) 2019-10-13T09:59:35.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:35.271+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.277159+0000) 2019-10-13T09:59:35.280 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:35.275+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.278012+0000) 2019-10-13T09:59:35.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:35.275+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.278293+0000) 2019-10-13T09:59:35.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:35.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.279105+0000) 2019-10-13T09:59:35.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:35.275+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.280991+0000) 2019-10-13T09:59:35.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:35.279+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.283346+0000) 2019-10-13T09:59:35.284 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:35.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:35.285 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:35.282+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.284120+0000) 2019-10-13T09:59:35.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:35.282+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:35.284709+0000) 2019-10-13T09:59:37.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:37.683+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:37.687811+0000) 2019-10-13T09:59:38.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:38.455+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.459267+0000) 2019-10-13T09:59:38.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:38.826+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:38.826449+0000) 2019-10-13T09:59:40.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:40.283+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.287593+0000) 2019-10-13T09:59:40.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:40.283+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.287937+0000) 2019-10-13T09:59:40.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:40.283+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.289679+0000) 2019-10-13T09:59:40.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:40.287+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.290261+0000) 2019-10-13T09:59:40.292 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:40.286+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.291368+0000) 2019-10-13T09:59:40.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:40.287+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.291959+0000) 2019-10-13T09:59:40.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:40.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:40.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:40.287+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.293673+0000) 2019-10-13T09:59:40.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:40.298+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:40.300884+0000) 2019-10-13T09:59:41.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:41.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.162357+0000) 2019-10-13T09:59:41.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:41.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:41.711023+0000) 2019-10-13T09:59:44.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:44.023+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:44.026371+0000) 2019-10-13T09:59:44.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:44.311+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:44.314159+0000) 2019-10-13T09:59:44.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:44.431+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:44.436137+0000) 2019-10-13T09:59:45.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:45.291+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.295909+0000) 2019-10-13T09:59:45.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:45.291+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.296308+0000) 2019-10-13T09:59:45.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:45.294+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.296469+0000) 2019-10-13T09:59:45.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:45.291+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.297342+0000) 2019-10-13T09:59:45.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:45.295+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.297933+0000) 2019-10-13T09:59:45.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:45.295+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.298273+0000) 2019-10-13T09:59:45.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:45.295+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.299396+0000) 2019-10-13T09:59:45.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:45.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106842 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:45.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:45.298+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:45.303313+0000) 2019-10-13T09:59:47.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:47.687+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:47.688047+0000) 2019-10-13T09:59:48.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:48.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.459486+0000) 2019-10-13T09:59:48.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:48.830+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:48.833001+0000) 2019-10-13T09:59:50.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:50.299+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.303841+0000) 2019-10-13T09:59:50.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:50.302+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.304386+0000) 2019-10-13T09:59:50.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:50.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.305256+0000) 2019-10-13T09:59:50.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:50.299+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.305498+0000) 2019-10-13T09:59:50.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:50.303+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.306979+0000) 2019-10-13T09:59:50.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:50.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.307326+0000) 2019-10-13T09:59:50.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:50.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:50.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:50.303+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.309513+0000) 2019-10-13T09:59:50.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:50.314+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:50.317249+0000) 2019-10-13T09:59:51.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:51.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.162538+0000) 2019-10-13T09:59:51.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:51.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:51.711209+0000) 2019-10-13T09:59:54.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:54.023+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:54.026602+0000) 2019-10-13T09:59:54.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:54.311+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:54.314394+0000) 2019-10-13T09:59:54.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:54.431+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:54.436331+0000) 2019-10-13T09:59:55.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T09:59:55.307+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.312380+0000) 2019-10-13T09:59:55.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T09:59:55.307+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.312776+0000) 2019-10-13T09:59:55.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:55.307+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.313600+0000) 2019-10-13T09:59:55.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:55.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.313939+0000) 2019-10-13T09:59:55.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T09:59:55.311+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.314277+0000) 2019-10-13T09:59:55.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T09:59:55.310+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.314215+0000) 2019-10-13T09:59:55.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T09:59:55.311+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.315393+0000) 2019-10-13T09:59:55.318 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T09:59:55.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106904 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T09:59:55.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:55.318+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:55.322970+0000) 2019-10-13T09:59:57.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T09:59:57.686+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:57.688248+0000) 2019-10-13T09:59:58.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T09:59:58.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.459700+0000) 2019-10-13T09:59:58.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T09:59:58.838+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T08:59:58.840049+0000) 2019-10-13T10:00:00.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:00.318+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.322080+0000) 2019-10-13T10:00:00.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:00.318+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.322402+0000) 2019-10-13T10:00:00.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:00.318+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.323177+0000) 2019-10-13T10:00:00.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:00.318+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.323720+0000) 2019-10-13T10:00:00.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:00.318+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.324546+0000) 2019-10-13T10:00:00.327 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:00.322+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:00.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:00.322+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.327001+0000) 2019-10-13T10:00:00.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:00.322+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.328343+0000) 2019-10-13T10:00:00.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:00.330+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:00.332795+0000) 2019-10-13T10:00:01.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:01.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.162719+0000) 2019-10-13T10:00:01.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:01.706+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:01.711407+0000) 2019-10-13T10:00:02.325 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:00:02.325 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:02.328 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:00:02.329 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:04.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:04.022+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:04.026807+0000) 2019-10-13T10:00:04.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:04.310+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:04.314606+0000) 2019-10-13T10:00:04.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:04.434+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:04.436535+0000) 2019-10-13T10:00:05.335 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:05.325+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.331973+0000) 2019-10-13T10:00:05.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:05.326+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.329357+0000) 2019-10-13T10:00:05.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:05.326+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.330607+0000) 2019-10-13T10:00:05.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:05.326+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.330913+0000) 2019-10-13T10:00:05.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:05.326+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.331756+0000) 2019-10-13T10:00:05.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:05.326+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.332149+0000) 2019-10-13T10:00:05.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:05.326+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.333492+0000) 2019-10-13T10:00:05.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:05.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106968 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:05.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:05.333+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:05.339250+0000) 2019-10-13T10:00:07.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:07.686+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:07.688451+0000) 2019-10-13T10:00:08.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:08.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.459902+0000) 2019-10-13T10:00:08.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:08.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:08.846534+0000) 2019-10-13T10:00:10.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:10.334+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.338247+0000) 2019-10-13T10:00:10.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:10.334+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.339177+0000) 2019-10-13T10:00:10.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:10.334+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.339881+0000) 2019-10-13T10:00:10.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:10.333+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.340056+0000) 2019-10-13T10:00:10.342 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:10.334+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.340167+0000) 2019-10-13T10:00:10.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:10.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.340647+0000) 2019-10-13T10:00:10.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:10.334+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.341479+0000) 2019-10-13T10:00:10.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:10.341+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:10.344486+0000) 2019-10-13T10:00:10.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:10.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:11.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:11.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.162904+0000) 2019-10-13T10:00:11.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:11.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:11.711613+0000) 2019-10-13T10:00:14.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:14.026+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:14.027046+0000) 2019-10-13T10:00:14.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:14.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:14.314840+0000) 2019-10-13T10:00:14.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:14.434+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:14.436732+0000) 2019-10-13T10:00:15.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:15.342+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.346740+0000) 2019-10-13T10:00:15.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:15.342+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.347847+0000) 2019-10-13T10:00:15.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:15.342+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.348301+0000) 2019-10-13T10:00:15.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:15.342+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.348917+0000) 2019-10-13T10:00:15.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:15.345+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.348790+0000) 2019-10-13T10:00:15.350 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:15.342+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.349542+0000) 2019-10-13T10:00:15.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:15.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.349879+0000) 2019-10-13T10:00:15.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:15.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:15.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:15.349+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:15.353250+0000) 2019-10-13T10:00:17.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:17.682+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:17.688647+0000) 2019-10-13T10:00:18.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:18.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.460093+0000) 2019-10-13T10:00:18.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:18.849+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:18.853148+0000) 2019-10-13T10:00:20.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:20.350+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.355070+0000) 2019-10-13T10:00:20.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:20.350+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.355348+0000) 2019-10-13T10:00:20.356 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:20.350+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.355855+0000) 2019-10-13T10:00:20.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:20.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.356444+0000) 2019-10-13T10:00:20.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:20.353+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.356616+0000) 2019-10-13T10:00:20.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:20.350+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.357413+0000) 2019-10-13T10:00:20.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:20.354+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.358088+0000) 2019-10-13T10:00:20.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:20.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:20.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:20.357+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:20.363705+0000) 2019-10-13T10:00:21.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:21.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.163120+0000) 2019-10-13T10:00:21.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:21.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:21.711801+0000) 2019-10-13T10:00:24.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:24.022+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:24.027256+0000) 2019-10-13T10:00:24.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:24.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:24.315037+0000) 2019-10-13T10:00:24.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:24.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:24.436934+0000) 2019-10-13T10:00:25.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:25.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.363720+0000) 2019-10-13T10:00:25.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:25.358+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.363948+0000) 2019-10-13T10:00:25.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:25.358+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.364350+0000) 2019-10-13T10:00:25.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:25.361+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.364761+0000) 2019-10-13T10:00:25.366 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:25.358+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.365491+0000) 2019-10-13T10:00:25.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:25.362+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.366406+0000) 2019-10-13T10:00:25.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:25.362+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.366723+0000) 2019-10-13T10:00:25.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:25.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:25.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:25.365+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:25.371580+0000) 2019-10-13T10:00:27.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:27.682+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:27.688872+0000) 2019-10-13T10:00:28.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:28.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.460290+0000) 2019-10-13T10:00:28.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:28.857+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:28.859610+0000) 2019-10-13T10:00:30.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:30.366+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.373159+0000) 2019-10-13T10:00:30.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:30.366+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.373450+0000) 2019-10-13T10:00:30.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:30.369+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.374900+0000) 2019-10-13T10:00:30.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:30.370+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.375639+0000) 2019-10-13T10:00:30.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:30.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:30.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:30.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.377812+0000) 2019-10-13T10:00:30.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:30.374+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.378059+0000) 2019-10-13T10:00:30.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:30.374+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.377953+0000) 2019-10-13T10:00:30.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:30.381+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:30.385094+0000) 2019-10-13T10:00:31.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:31.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.163341+0000) 2019-10-13T10:00:31.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:31.705+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:31.711994+0000) 2019-10-13T10:00:32.431 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:00:32.431 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:32.440 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:00:32.440 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:00:34.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:34.026+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:34.027460+0000) 2019-10-13T10:00:34.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:34.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:34.315239+0000) 2019-10-13T10:00:34.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:34.434+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:34.437137+0000) 2019-10-13T10:00:35.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:35.374+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.381589+0000) 2019-10-13T10:00:35.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:35.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.382486+0000) 2019-10-13T10:00:35.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:35.378+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.382884+0000) 2019-10-13T10:00:35.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:35.378+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.383104+0000) 2019-10-13T10:00:35.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:35.378+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.383583+0000) 2019-10-13T10:00:35.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:35.377+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.383397+0000) 2019-10-13T10:00:35.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:35.378+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.384253+0000) 2019-10-13T10:00:35.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:35.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:35.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:35.385+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:35.389329+0000) 2019-10-13T10:00:37.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:37.686+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:37.689076+0000) 2019-10-13T10:00:38.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:38.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.460479+0000) 2019-10-13T10:00:38.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:38.865+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:38.870872+0000) 2019-10-13T10:00:40.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:40.386+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.389523+0000) 2019-10-13T10:00:40.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:40.386+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.390416+0000) 2019-10-13T10:00:40.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:40.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.392278+0000) 2019-10-13T10:00:40.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:40.386+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.392754+0000) 2019-10-13T10:00:40.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:40.390+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.394503+0000) 2019-10-13T10:00:40.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:40.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:40.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:40.390+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.396023+0000) 2019-10-13T10:00:40.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:40.393+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.396568+0000) 2019-10-13T10:00:40.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:40.393+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:40.396992+0000) 2019-10-13T10:00:41.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:41.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.163516+0000) 2019-10-13T10:00:41.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:41.709+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:41.712200+0000) 2019-10-13T10:00:44.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:44.026+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.027694+0000) 2019-10-13T10:00:44.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:44.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.315447+0000) 2019-10-13T10:00:44.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:44.430+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:44.437368+0000) 2019-10-13T10:00:45.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:45.394+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.399036+0000) 2019-10-13T10:00:45.400 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:45.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.399317+0000) 2019-10-13T10:00:45.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:45.394+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.399998+0000) 2019-10-13T10:00:45.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:45.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.400833+0000) 2019-10-13T10:00:45.402 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:45.397+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.400837+0000) 2019-10-13T10:00:45.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:45.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.401851+0000) 2019-10-13T10:00:45.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:45.398+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.401987+0000) 2019-10-13T10:00:45.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:45.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:45.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:45.405+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:45.408204+0000) 2019-10-13T10:00:47.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:47.686+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:47.689311+0000) 2019-10-13T10:00:48.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:48.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.460674+0000) 2019-10-13T10:00:48.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:48.873+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:48.877320+0000) 2019-10-13T10:00:50.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:50.402+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.407053+0000) 2019-10-13T10:00:50.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:50.402+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.407453+0000) 2019-10-13T10:00:50.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:50.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.407945+0000) 2019-10-13T10:00:50.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:50.402+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.408550+0000) 2019-10-13T10:00:50.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:50.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.408338+0000) 2019-10-13T10:00:50.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:50.406+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.409973+0000) 2019-10-13T10:00:50.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:50.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:50.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:50.405+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.410383+0000) 2019-10-13T10:00:50.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:50.409+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:50.414523+0000) 2019-10-13T10:00:51.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:51.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.163700+0000) 2019-10-13T10:00:51.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:51.709+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:51.712396+0000) 2019-10-13T10:00:54.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:54.025+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.027895+0000) 2019-10-13T10:00:54.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:54.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.315695+0000) 2019-10-13T10:00:54.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:54.433+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:54.437566+0000) 2019-10-13T10:00:55.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:55.409+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.415786+0000) 2019-10-13T10:00:55.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:00:55.409+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.417603+0000) 2019-10-13T10:00:55.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:00:55.413+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.418232+0000) 2019-10-13T10:00:55.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:00:55.413+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.418941+0000) 2019-10-13T10:00:55.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:00:55.413+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.419564+0000) 2019-10-13T10:00:55.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:55.413+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.419977+0000) 2019-10-13T10:00:55.421 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:00:55.413+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.419905+0000) 2019-10-13T10:00:55.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:00:55.413+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:00:55.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:55.421+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:55.425070+0000) 2019-10-13T10:00:57.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:00:57.681+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:57.689522+0000) 2019-10-13T10:00:58.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:00:58.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.460850+0000) 2019-10-13T10:00:58.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:00:58.881+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:00:58.883913+0000) 2019-10-13T10:01:00.424 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:00.417+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.423226+0000) 2019-10-13T10:01:00.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:00.417+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.424021+0000) 2019-10-13T10:01:00.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:00.421+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.426225+0000) 2019-10-13T10:01:00.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:00.421+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.426878+0000) 2019-10-13T10:01:00.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:00.421+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.427143+0000) 2019-10-13T10:01:00.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:00.421+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.427332+0000) 2019-10-13T10:01:00.428 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:00.421+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.427826+0000) 2019-10-13T10:01:00.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:00.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:00.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:00.425+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:00.431574+0000) 2019-10-13T10:01:01.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:01.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.163885+0000) 2019-10-13T10:01:01.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:01.709+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:01.712551+0000) 2019-10-13T10:01:02.545 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:01:02.545 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:02.549 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:01:02.549 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:04.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:04.021+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.028093+0000) 2019-10-13T10:01:04.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:04.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.315891+0000) 2019-10-13T10:01:04.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:04.433+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:04.437785+0000) 2019-10-13T10:01:05.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:05.425+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.431760+0000) 2019-10-13T10:01:05.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:05.425+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.432484+0000) 2019-10-13T10:01:05.433 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:05.425+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.432907+0000) 2019-10-13T10:01:05.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:05.425+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.433465+0000) 2019-10-13T10:01:05.434 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:05.429+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.433373+0000) 2019-10-13T10:01:05.435 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:05.429+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.434017+0000) 2019-10-13T10:01:05.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:05.429+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.434574+0000) 2019-10-13T10:01:05.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:05.429+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:05.442 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:05.437+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:05.440872+0000) 2019-10-13T10:01:07.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:07.685+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:07.689716+0000) 2019-10-13T10:01:08.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:08.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.461060+0000) 2019-10-13T10:01:08.891 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:08.889+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:08.890352+0000) 2019-10-13T10:01:10.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:10.437+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.441183+0000) 2019-10-13T10:01:10.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:10.437+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.441551+0000) 2019-10-13T10:01:10.443 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:10.441+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.443018+0000) 2019-10-13T10:01:10.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:10.441+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.443205+0000) 2019-10-13T10:01:10.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:10.441+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.445039+0000) 2019-10-13T10:01:10.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:10.441+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.445171+0000) 2019-10-13T10:01:10.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:10.441+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:10.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:10.441+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.447210+0000) 2019-10-13T10:01:10.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:10.449+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:10.452289+0000) 2019-10-13T10:01:11.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:11.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.164125+0000) 2019-10-13T10:01:11.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:11.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:11.712740+0000) 2019-10-13T10:01:14.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:14.025+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.028334+0000) 2019-10-13T10:01:14.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:14.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.316090+0000) 2019-10-13T10:01:14.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:14.437+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:14.438009+0000) 2019-10-13T10:01:15.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:15.445+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.448512+0000) 2019-10-13T10:01:15.450 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:15.445+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.449395+0000) 2019-10-13T10:01:15.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:15.449+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.450102+0000) 2019-10-13T10:01:15.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:15.449+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.450085+0000) 2019-10-13T10:01:15.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:15.449+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.450914+0000) 2019-10-13T10:01:15.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:15.449+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.451864+0000) 2019-10-13T10:01:15.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:15.448+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.453018+0000) 2019-10-13T10:01:15.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:15.449+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:15.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:15.456+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:15.460665+0000) 2019-10-13T10:01:17.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:17.689+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:17.689943+0000) 2019-10-13T10:01:18.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:18.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.461256+0000) 2019-10-13T10:01:18.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:18.892+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:18.896721+0000) 2019-10-13T10:01:20.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:20.453+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.457134+0000) 2019-10-13T10:01:20.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:20.453+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.457440+0000) 2019-10-13T10:01:20.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:20.457+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.459214+0000) 2019-10-13T10:01:20.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:20.457+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.459589+0000) 2019-10-13T10:01:20.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:20.457+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.460514+0000) 2019-10-13T10:01:20.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:20.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:20.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:20.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.462761+0000) 2019-10-13T10:01:20.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:20.456+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.462872+0000) 2019-10-13T10:01:20.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:20.460+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:20.467708+0000) 2019-10-13T10:01:21.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:21.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.164302+0000) 2019-10-13T10:01:21.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:21.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:21.712945+0000) 2019-10-13T10:01:24.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:24.025+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.028536+0000) 2019-10-13T10:01:24.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:24.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.316300+0000) 2019-10-13T10:01:24.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:24.437+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:24.438219+0000) 2019-10-13T10:01:25.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:25.465+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.465836+0000) 2019-10-13T10:01:25.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:25.465+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.466237+0000) 2019-10-13T10:01:25.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:25.465+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.466865+0000) 2019-10-13T10:01:25.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:25.460+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.466967+0000) 2019-10-13T10:01:25.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:25.465+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.467747+0000) 2019-10-13T10:01:25.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:25.465+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.469198+0000) 2019-10-13T10:01:25.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:25.465+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.469670+0000) 2019-10-13T10:01:25.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:25.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:25.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:25.472+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:25.476917+0000) 2019-10-13T10:01:27.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:27.689+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:27.690174+0000) 2019-10-13T10:01:28.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:28.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.461464+0000) 2019-10-13T10:01:28.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:28.904+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:28.908260+0000) 2019-10-13T10:01:30.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:30.473+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.473823+0000) 2019-10-13T10:01:30.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:30.468+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.475027+0000) 2019-10-13T10:01:30.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:30.473+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.475922+0000) 2019-10-13T10:01:30.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:30.473+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.475946+0000) 2019-10-13T10:01:30.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:30.473+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.476727+0000) 2019-10-13T10:01:30.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:30.473+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.476615+0000) 2019-10-13T10:01:30.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:30.473+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.477372+0000) 2019-10-13T10:01:30.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:30.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:30.484 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:30.476+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:30.483608+0000) 2019-10-13T10:01:31.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:31.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.164482+0000) 2019-10-13T10:01:31.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:31.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:31.713138+0000) 2019-10-13T10:01:32.668 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:01:32.668 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:32.672 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:01:32.672 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:01:34.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:34.025+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.028762+0000) 2019-10-13T10:01:34.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:34.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.316504+0000) 2019-10-13T10:01:34.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:34.437+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:34.438379+0000) 2019-10-13T10:01:35.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:35.481+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.482839+0000) 2019-10-13T10:01:35.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:35.481+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.485033+0000) 2019-10-13T10:01:35.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:35.480+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.486518+0000) 2019-10-13T10:01:35.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:35.485+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.487062+0000) 2019-10-13T10:01:35.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:35.485+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.487577+0000) 2019-10-13T10:01:35.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:35.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107527 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:35.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:35.485+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.488761+0000) 2019-10-13T10:01:35.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:35.485+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.489180+0000) 2019-10-13T10:01:35.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:35.492+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:35.498790+0000) 2019-10-13T10:01:37.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:37.689+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:37.690374+0000) 2019-10-13T10:01:38.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:38.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.461674+0000) 2019-10-13T10:01:38.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:38.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:38.914813+0000) 2019-10-13T10:01:40.493 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:40.489+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.491534+0000) 2019-10-13T10:01:40.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:40.489+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.491963+0000) 2019-10-13T10:01:40.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:40.489+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.492321+0000) 2019-10-13T10:01:40.495 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:40.488+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.493445+0000) 2019-10-13T10:01:40.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:40.489+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.492879+0000) 2019-10-13T10:01:40.495 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:40.489+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.493353+0000) 2019-10-13T10:01:40.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:40.493+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.494606+0000) 2019-10-13T10:01:40.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:40.492+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:40.498883+0000) 2019-10-13T10:01:40.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:40.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:41.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:41.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.164675+0000) 2019-10-13T10:01:41.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:41.708+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:41.713480+0000) 2019-10-13T10:01:44.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:44.025+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.028979+0000) 2019-10-13T10:01:44.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:44.313+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.316705+0000) 2019-10-13T10:01:44.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:44.437+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:44.438618+0000) 2019-10-13T10:01:45.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:45.497+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.498938+0000) 2019-10-13T10:01:45.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:45.497+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.499350+0000) 2019-10-13T10:01:45.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:45.497+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.500498+0000) 2019-10-13T10:01:45.501 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:45.496+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.500437+0000) 2019-10-13T10:01:45.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:45.501+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.502099+0000) 2019-10-13T10:01:45.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:45.501+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.501887+0000) 2019-10-13T10:01:45.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:45.501+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.502561+0000) 2019-10-13T10:01:45.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:45.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:45.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:45.504+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:45.509289+0000) 2019-10-13T10:01:47.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:47.689+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:47.690574+0000) 2019-10-13T10:01:48.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:48.461+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.461893+0000) 2019-10-13T10:01:48.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:48.924+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:48.925193+0000) 2019-10-13T10:01:50.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:50.504+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.509027+0000) 2019-10-13T10:01:50.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:50.508+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.510066+0000) 2019-10-13T10:01:50.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:50.508+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.510342+0000) 2019-10-13T10:01:50.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:50.508+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.510694+0000) 2019-10-13T10:01:50.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:50.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.511110+0000) 2019-10-13T10:01:50.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:50.508+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.511124+0000) 2019-10-13T10:01:50.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:50.508+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.512562+0000) 2019-10-13T10:01:50.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:50.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:50.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:50.516+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:50.520227+0000) 2019-10-13T10:01:51.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:51.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.164858+0000) 2019-10-13T10:01:51.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:51.712+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:51.713707+0000) 2019-10-13T10:01:54.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:54.024+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.029178+0000) 2019-10-13T10:01:54.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:54.312+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.316915+0000) 2019-10-13T10:01:54.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:54.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:54.438826+0000) 2019-10-13T10:01:55.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:55.512+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.517540+0000) 2019-10-13T10:01:55.519 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:01:55.512+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.517527+0000) 2019-10-13T10:01:55.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:01:55.516+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.518696+0000) 2019-10-13T10:01:55.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:55.516+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.519844+0000) 2019-10-13T10:01:55.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:01:55.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.520241+0000) 2019-10-13T10:01:55.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:01:55.516+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.521444+0000) 2019-10-13T10:01:55.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:01:55.520+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.521911+0000) 2019-10-13T10:01:55.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:01:55.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:01:55.527 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:55.520+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:55.526013+0000) 2019-10-13T10:01:57.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:01:57.688+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:57.690795+0000) 2019-10-13T10:01:58.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:01:58.460+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.462080+0000) 2019-10-13T10:01:58.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:01:58.928+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:01:58.931664+0000) 2019-10-13T10:02:00.527 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:00.524+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.526000+0000) 2019-10-13T10:02:00.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:00.524+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.526728+0000) 2019-10-13T10:02:00.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:00.524+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.527121+0000) 2019-10-13T10:02:00.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:00.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.527027+0000) 2019-10-13T10:02:00.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:00.524+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.527717+0000) 2019-10-13T10:02:00.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:00.524+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.528439+0000) 2019-10-13T10:02:00.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:00.524+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.529153+0000) 2019-10-13T10:02:00.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:00.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:00.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:00.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:00.534553+0000) 2019-10-13T10:02:01.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:01.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.165051+0000) 2019-10-13T10:02:01.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:01.712+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:01.713899+0000) 2019-10-13T10:02:02.748 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:02:02.748 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:02.752 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:02:02.752 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:04.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:04.028+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.029381+0000) 2019-10-13T10:02:04.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:04.312+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.317112+0000) 2019-10-13T10:02:04.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:04.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:04.439023+0000) 2019-10-13T10:02:05.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:05.532+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.534133+0000) 2019-10-13T10:02:05.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:05.532+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.535179+0000) 2019-10-13T10:02:05.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:05.532+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.536055+0000) 2019-10-13T10:02:05.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:05.532+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.537511+0000) 2019-10-13T10:02:05.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:05.536+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.538135+0000) 2019-10-13T10:02:05.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:05.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:05.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:05.536+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.539720+0000) 2019-10-13T10:02:05.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:05.536+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.540176+0000) 2019-10-13T10:02:05.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:05.536+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:05.544045+0000) 2019-10-13T10:02:07.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:07.688+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:07.690996+0000) 2019-10-13T10:02:08.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:08.460+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.462336+0000) 2019-10-13T10:02:08.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:08.936+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:08.938120+0000) 2019-10-13T10:02:10.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:10.540+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.543630+0000) 2019-10-13T10:02:10.545 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:10.540+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.544401+0000) 2019-10-13T10:02:10.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:10.540+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.545108+0000) 2019-10-13T10:02:10.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:10.544+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.545650+0000) 2019-10-13T10:02:10.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:10.544+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.546547+0000) 2019-10-13T10:02:10.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:10.544+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.546883+0000) 2019-10-13T10:02:10.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:10.544+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.547267+0000) 2019-10-13T10:02:10.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:10.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:10.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:10.544+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:10.550482+0000) 2019-10-13T10:02:11.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:11.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.165236+0000) 2019-10-13T10:02:11.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:11.712+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:11.714093+0000) 2019-10-13T10:02:14.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:14.028+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.029611+0000) 2019-10-13T10:02:14.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:14.316+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.317317+0000) 2019-10-13T10:02:14.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:14.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:14.439221+0000) 2019-10-13T10:02:15.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:15.548+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.550883+0000) 2019-10-13T10:02:15.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:15.548+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.551730+0000) 2019-10-13T10:02:15.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:15.548+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.551421+0000) 2019-10-13T10:02:15.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:15.548+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.552327+0000) 2019-10-13T10:02:15.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:15.548+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.552711+0000) 2019-10-13T10:02:15.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:15.548+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.553448+0000) 2019-10-13T10:02:15.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:15.552+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.553949+0000) 2019-10-13T10:02:15.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:15.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:15.561 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:15.560+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:15.560549+0000) 2019-10-13T10:02:17.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:17.688+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:17.691246+0000) 2019-10-13T10:02:18.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:18.460+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.462533+0000) 2019-10-13T10:02:18.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:18.943+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:18.944666+0000) 2019-10-13T10:02:20.561 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:20.556+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.559976+0000) 2019-10-13T10:02:20.561 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:20.556+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.560290+0000) 2019-10-13T10:02:20.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:20.556+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.560697+0000) 2019-10-13T10:02:20.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:20.560+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.561886+0000) 2019-10-13T10:02:20.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:20.559+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.561901+0000) 2019-10-13T10:02:20.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:20.560+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.562833+0000) 2019-10-13T10:02:20.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:20.560+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.563458+0000) 2019-10-13T10:02:20.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:20.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:20.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:20.563+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:20.567388+0000) 2019-10-13T10:02:21.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:21.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.165422+0000) 2019-10-13T10:02:21.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:21.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:21.714290+0000) 2019-10-13T10:02:24.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:24.028+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.029815+0000) 2019-10-13T10:02:24.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:24.312+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.317544+0000) 2019-10-13T10:02:24.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:24.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:24.439449+0000) 2019-10-13T10:02:25.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:25.564+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.567612+0000) 2019-10-13T10:02:25.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:25.563+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.567670+0000) 2019-10-13T10:02:25.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:25.564+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.568637+0000) 2019-10-13T10:02:25.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:25.564+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.569376+0000) 2019-10-13T10:02:25.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:25.568+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.569922+0000) 2019-10-13T10:02:25.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:25.568+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.570381+0000) 2019-10-13T10:02:25.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:25.568+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.570949+0000) 2019-10-13T10:02:25.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:25.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107837 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:25.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:25.571+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:25.574823+0000) 2019-10-13T10:02:27.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:27.688+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:27.691452+0000) 2019-10-13T10:02:28.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:28.460+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.462721+0000) 2019-10-13T10:02:28.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:28.951+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:28.955908+0000) 2019-10-13T10:02:30.576 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:30.572+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.574992+0000) 2019-10-13T10:02:30.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:30.572+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.577150+0000) 2019-10-13T10:02:30.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:30.576+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.577818+0000) 2019-10-13T10:02:30.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:30.576+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.579135+0000) 2019-10-13T10:02:30.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:30.576+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.579404+0000) 2019-10-13T10:02:30.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:30.575+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.580033+0000) 2019-10-13T10:02:30.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:30.576+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.580736+0000) 2019-10-13T10:02:30.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:30.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:30.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:30.583+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:30.584468+0000) 2019-10-13T10:02:31.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:31.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.165599+0000) 2019-10-13T10:02:31.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:31.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:31.714483+0000) 2019-10-13T10:02:32.872 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:02:32.872 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:32.876 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:02:32.876 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:02:34.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:34.028+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.030025+0000) 2019-10-13T10:02:34.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:34.316+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.317786+0000) 2019-10-13T10:02:34.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:34.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:34.439669+0000) 2019-10-13T10:02:35.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:35.580+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.585015+0000) 2019-10-13T10:02:35.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:35.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.585023+0000) 2019-10-13T10:02:35.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:35.584+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.588277+0000) 2019-10-13T10:02:35.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:35.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:35.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:35.588+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.590703+0000) 2019-10-13T10:02:35.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:35.588+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.591324+0000) 2019-10-13T10:02:35.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:35.588+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.591861+0000) 2019-10-13T10:02:35.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:35.588+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.593319+0000) 2019-10-13T10:02:35.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:35.591+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:35.594174+0000) 2019-10-13T10:02:37.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:37.688+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:37.691684+0000) 2019-10-13T10:02:38.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:38.460+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.462924+0000) 2019-10-13T10:02:38.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:38.963+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:38.967448+0000) 2019-10-13T10:02:40.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:40.591+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.593844+0000) 2019-10-13T10:02:40.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:40.592+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.594816+0000) 2019-10-13T10:02:40.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:40.592+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.595186+0000) 2019-10-13T10:02:40.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:40.592+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.597036+0000) 2019-10-13T10:02:40.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:40.596+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.598003+0000) 2019-10-13T10:02:40.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:40.596+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.598431+0000) 2019-10-13T10:02:40.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:40.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:40.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:40.596+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.599890+0000) 2019-10-13T10:02:40.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:40.603+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:40.605572+0000) 2019-10-13T10:02:41.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:41.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.165781+0000) 2019-10-13T10:02:41.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:41.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:41.714673+0000) 2019-10-13T10:02:44.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:44.028+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.030265+0000) 2019-10-13T10:02:44.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:44.316+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.317994+0000) 2019-10-13T10:02:44.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:44.436+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:44.439904+0000) 2019-10-13T10:02:45.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:45.600+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.602892+0000) 2019-10-13T10:02:45.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:45.600+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.603518+0000) 2019-10-13T10:02:45.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:45.600+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.605305+0000) 2019-10-13T10:02:45.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:45.604+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.607986+0000) 2019-10-13T10:02:45.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:45.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:45.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:45.604+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.608848+0000) 2019-10-13T10:02:45.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:45.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.609241+0000) 2019-10-13T10:02:45.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:45.607+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.609336+0000) 2019-10-13T10:02:45.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:45.608+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:45.610674+0000) 2019-10-13T10:02:47.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:47.687+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:47.691914+0000) 2019-10-13T10:02:48.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:48.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.463120+0000) 2019-10-13T10:02:48.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:48.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:48.973855+0000) 2019-10-13T10:02:50.612 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:50.607+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.611825+0000) 2019-10-13T10:02:50.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:50.607+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.612951+0000) 2019-10-13T10:02:50.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:50.607+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.613467+0000) 2019-10-13T10:02:50.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:50.611+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.613792+0000) 2019-10-13T10:02:50.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:50.611+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.614624+0000) 2019-10-13T10:02:50.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:50.611+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.615011+0000) 2019-10-13T10:02:50.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:50.611+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.614963+0000) 2019-10-13T10:02:50.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:50.611+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:50.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:50.619+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:50.620386+0000) 2019-10-13T10:02:51.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:51.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.165995+0000) 2019-10-13T10:02:51.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:51.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:51.714864+0000) 2019-10-13T10:02:54.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:54.027+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.030483+0000) 2019-10-13T10:02:54.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:54.315+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.318202+0000) 2019-10-13T10:02:54.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:54.435+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:54.440115+0000) 2019-10-13T10:02:55.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:55.615+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.620447+0000) 2019-10-13T10:02:55.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:02:55.615+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.621485+0000) 2019-10-13T10:02:55.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:02:55.619+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.623214+0000) 2019-10-13T10:02:55.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:55.619+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.624138+0000) 2019-10-13T10:02:55.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:02:55.619+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.624641+0000) 2019-10-13T10:02:55.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:02:55.619+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.625119+0000) 2019-10-13T10:02:55.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:02:55.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:02:55.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:02:55.623+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.626044+0000) 2019-10-13T10:02:55.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:55.627+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:55.629977+0000) 2019-10-13T10:02:57.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:02:57.687+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:57.692118+0000) 2019-10-13T10:02:58.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:02:58.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.463345+0000) 2019-10-13T10:02:58.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:02:58.979+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:02:58.980458+0000) 2019-10-13T10:03:00.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:00.623+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.629091+0000) 2019-10-13T10:03:00.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:00.623+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.629588+0000) 2019-10-13T10:03:00.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:00.627+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.631413+0000) 2019-10-13T10:03:00.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:00.627+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.631438+0000) 2019-10-13T10:03:00.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:00.627+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.631800+0000) 2019-10-13T10:03:00.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:00.627+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.632234+0000) 2019-10-13T10:03:00.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:00.627+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.632519+0000) 2019-10-13T10:03:00.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:00.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:00.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:00.635+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:00.637264+0000) 2019-10-13T10:03:01.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:01.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.166177+0000) 2019-10-13T10:03:01.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:01.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:01.715072+0000) 2019-10-13T10:03:03.003 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:03:03.003 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:03.007 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:03:03.007 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:04.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:04.027+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.030712+0000) 2019-10-13T10:03:04.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:04.315+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.318400+0000) 2019-10-13T10:03:04.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:04.439+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:04.440326+0000) 2019-10-13T10:03:05.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:05.631+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.637457+0000) 2019-10-13T10:03:05.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:05.635+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.638215+0000) 2019-10-13T10:03:05.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:05.635+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.639120+0000) 2019-10-13T10:03:05.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:05.635+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.638918+0000) 2019-10-13T10:03:05.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:05.635+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.639717+0000) 2019-10-13T10:03:05.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:05.635+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.639880+0000) 2019-10-13T10:03:05.641 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:05.635+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.640802+0000) 2019-10-13T10:03:05.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:05.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:05.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:05.639+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:05.643855+0000) 2019-10-13T10:03:07.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:07.691+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:07.692319+0000) 2019-10-13T10:03:08.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:08.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.463492+0000) 2019-10-13T10:03:08.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:08.987+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:08.991877+0000) 2019-10-13T10:03:10.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:10.643+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.646259+0000) 2019-10-13T10:03:10.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:10.643+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.647090+0000) 2019-10-13T10:03:10.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:10.643+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.647628+0000) 2019-10-13T10:03:10.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:10.643+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.647650+0000) 2019-10-13T10:03:10.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:10.643+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.648090+0000) 2019-10-13T10:03:10.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:10.643+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.648484+0000) 2019-10-13T10:03:10.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:10.647+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.649360+0000) 2019-10-13T10:03:10.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:10.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:10.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:10.655+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:10.657228+0000) 2019-10-13T10:03:11.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:11.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.166365+0000) 2019-10-13T10:03:11.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:11.711+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:11.715267+0000) 2019-10-13T10:03:14.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:14.027+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.030978+0000) 2019-10-13T10:03:14.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:14.315+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.318556+0000) 2019-10-13T10:03:14.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:14.435+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:14.440525+0000) 2019-10-13T10:03:15.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:15.651+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.655219+0000) 2019-10-13T10:03:15.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:15.651+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.656860+0000) 2019-10-13T10:03:15.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:15.651+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.656994+0000) 2019-10-13T10:03:15.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:15.651+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.657355+0000) 2019-10-13T10:03:15.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:15.655+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.658570+0000) 2019-10-13T10:03:15.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:15.655+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.659382+0000) 2019-10-13T10:03:15.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:15.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.659454+0000) 2019-10-13T10:03:15.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:15.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:15.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:15.659+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:15.661620+0000) 2019-10-13T10:03:17.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:17.687+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:17.692533+0000) 2019-10-13T10:03:18.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:18.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.463707+0000) 2019-10-13T10:03:18.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:18.995+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:18.998401+0000) 2019-10-13T10:03:20.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:20.659+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.663378+0000) 2019-10-13T10:03:20.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:20.659+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.663994+0000) 2019-10-13T10:03:20.666 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:20.659+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.664427+0000) 2019-10-13T10:03:20.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:20.659+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.664988+0000) 2019-10-13T10:03:20.666 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:20.659+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.665442+0000) 2019-10-13T10:03:20.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:20.663+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.666509+0000) 2019-10-13T10:03:20.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:20.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.666271+0000) 2019-10-13T10:03:20.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:20.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:20.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:20.671+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:20.672291+0000) 2019-10-13T10:03:21.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:21.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.166575+0000) 2019-10-13T10:03:21.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:21.715+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:21.715464+0000) 2019-10-13T10:03:24.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:24.027+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.031175+0000) 2019-10-13T10:03:24.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:24.315+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.318771+0000) 2019-10-13T10:03:24.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:24.439+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:24.440749+0000) 2019-10-13T10:03:25.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:25.667+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.672224+0000) 2019-10-13T10:03:25.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:25.667+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.673395+0000) 2019-10-13T10:03:25.676 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:25.671+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.674994+0000) 2019-10-13T10:03:25.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:25.671+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.675345+0000) 2019-10-13T10:03:25.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:25.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.675889+0000) 2019-10-13T10:03:25.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:25.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.676697+0000) 2019-10-13T10:03:25.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:25.671+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.676807+0000) 2019-10-13T10:03:25.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:25.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:25.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:25.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:25.681607+0000) 2019-10-13T10:03:27.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:27.687+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:27.692767+0000) 2019-10-13T10:03:28.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:28.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:28.463931+0000) 2019-10-13T10:03:29.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:29.002+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:29.004917+0000) 2019-10-13T10:03:30.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:30.675+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.681116+0000) 2019-10-13T10:03:30.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:30.675+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.681448+0000) 2019-10-13T10:03:30.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:30.679+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.683317+0000) 2019-10-13T10:03:30.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:30.679+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.683868+0000) 2019-10-13T10:03:30.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:30.682+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.684164+0000) 2019-10-13T10:03:30.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:30.679+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.684965+0000) 2019-10-13T10:03:30.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:30.679+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.685243+0000) 2019-10-13T10:03:30.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:30.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:30.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:30.686+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:30.689316+0000) 2019-10-13T10:03:31.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:31.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.166755+0000) 2019-10-13T10:03:31.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:31.710+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:31.715663+0000) 2019-10-13T10:03:33.083 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:03:33.083 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:33.087 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:03:33.087 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:03:34.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:34.027+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.031384+0000) 2019-10-13T10:03:34.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:34.315+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.318968+0000) 2019-10-13T10:03:34.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:34.439+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:34.440949+0000) 2019-10-13T10:03:35.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:35.687+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.689938+0000) 2019-10-13T10:03:35.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:35.687+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.690462+0000) 2019-10-13T10:03:35.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:35.686+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.690575+0000) 2019-10-13T10:03:35.692 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:35.687+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.691401+0000) 2019-10-13T10:03:35.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:35.687+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.691742+0000) 2019-10-13T10:03:35.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:35.687+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.692531+0000) 2019-10-13T10:03:35.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:35.691+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.693831+0000) 2019-10-13T10:03:35.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:35.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:35.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:35.694+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:35.699242+0000) 2019-10-13T10:03:37.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:37.691+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:37.692964+0000) 2019-10-13T10:03:38.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:38.459+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:38.464122+0000) 2019-10-13T10:03:39.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:39.006+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:39.011250+0000) 2019-10-13T10:03:40.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:40.695+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.698948+0000) 2019-10-13T10:03:40.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:40.695+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.699503+0000) 2019-10-13T10:03:40.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:40.695+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.699790+0000) 2019-10-13T10:03:40.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:40.695+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.700230+0000) 2019-10-13T10:03:40.702 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:40.695+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.701346+0000) 2019-10-13T10:03:40.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:40.699+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.701864+0000) 2019-10-13T10:03:40.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:40.698+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.702074+0000) 2019-10-13T10:03:40.705 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:40.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:40.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:40.706+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:40.709637+0000) 2019-10-13T10:03:41.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:41.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.166966+0000) 2019-10-13T10:03:41.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:41.710+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:41.715898+0000) 2019-10-13T10:03:44.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:44.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.031583+0000) 2019-10-13T10:03:44.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:44.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.319204+0000) 2019-10-13T10:03:44.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:44.438+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:44.441152+0000) 2019-10-13T10:03:45.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:45.702+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.708744+0000) 2019-10-13T10:03:45.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:45.702+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.709109+0000) 2019-10-13T10:03:45.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:45.706+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.710171+0000) 2019-10-13T10:03:45.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:45.706+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.711438+0000) 2019-10-13T10:03:45.714 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:45.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:45.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:45.710+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.713902+0000) 2019-10-13T10:03:45.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:45.710+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.714042+0000) 2019-10-13T10:03:45.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:45.710+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.714043+0000) 2019-10-13T10:03:45.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:45.718+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:45.720951+0000) 2019-10-13T10:03:47.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:47.690+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:47.693129+0000) 2019-10-13T10:03:48.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:48.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:48.464317+0000) 2019-10-13T10:03:49.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:49.014+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:49.017875+0000) 2019-10-13T10:03:50.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:50.710+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.717369+0000) 2019-10-13T10:03:50.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:50.714+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.717981+0000) 2019-10-13T10:03:50.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:50.714+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.718934+0000) 2019-10-13T10:03:50.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:50.714+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.719967+0000) 2019-10-13T10:03:50.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:50.714+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.720305+0000) 2019-10-13T10:03:50.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:50.718+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.720275+0000) 2019-10-13T10:03:50.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:50.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.721310+0000) 2019-10-13T10:03:50.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:50.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:50.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:50.718+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:50.724059+0000) 2019-10-13T10:03:51.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:51.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.167179+0000) 2019-10-13T10:03:51.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:51.714+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:51.716115+0000) 2019-10-13T10:03:54.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:54.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.031791+0000) 2019-10-13T10:03:54.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:54.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.319417+0000) 2019-10-13T10:03:54.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:54.434+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:54.441351+0000) 2019-10-13T10:03:55.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:03:55.722+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.726186+0000) 2019-10-13T10:03:55.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:03:55.722+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.726764+0000) 2019-10-13T10:03:55.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:55.722+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.728871+0000) 2019-10-13T10:03:55.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:03:55.722+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.729060+0000) 2019-10-13T10:03:55.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:03:55.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:03:55.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:03:55.726+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.730526+0000) 2019-10-13T10:03:55.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:55.726+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.731526+0000) 2019-10-13T10:03:55.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:03:55.726+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.731737+0000) 2019-10-13T10:03:55.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:55.726+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:55.731148+0000) 2019-10-13T10:03:57.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:03:57.690+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:57.693334+0000) 2019-10-13T10:03:58.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:03:58.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:58.464538+0000) 2019-10-13T10:03:59.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:03:59.022+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:03:59.024323+0000) 2019-10-13T10:04:00.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:00.726+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.733204+0000) 2019-10-13T10:04:00.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:00.726+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.733596+0000) 2019-10-13T10:04:00.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:00.730+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.735203+0000) 2019-10-13T10:04:00.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:00.730+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.735540+0000) 2019-10-13T10:04:00.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:00.730+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.735422+0000) 2019-10-13T10:04:00.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:00.730+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.737220+0000) 2019-10-13T10:04:00.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:00.730+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.737723+0000) 2019-10-13T10:04:00.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:00.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:00.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:00.738+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:00.740745+0000) 2019-10-13T10:04:01.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:01.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.167333+0000) 2019-10-13T10:04:01.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:01.714+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:01.716297+0000) 2019-10-13T10:04:03.205 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:04:03.206 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:03.210 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:04:03.210 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:04.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:04.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.031991+0000) 2019-10-13T10:04:04.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:04.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.319655+0000) 2019-10-13T10:04:04.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:04.434+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:04.441586+0000) 2019-10-13T10:04:05.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:05.738+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.742572+0000) 2019-10-13T10:04:05.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:05.738+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.742866+0000) 2019-10-13T10:04:05.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:05.738+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.743356+0000) 2019-10-13T10:04:05.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:05.738+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.744728+0000) 2019-10-13T10:04:05.747 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:05.738+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.745288+0000) 2019-10-13T10:04:05.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:05.742+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108456 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:05.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:05.742+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.747415+0000) 2019-10-13T10:04:05.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:05.742+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.747503+0000) 2019-10-13T10:04:05.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:05.750+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:05.752765+0000) 2019-10-13T10:04:07.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:07.690+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:07.693538+0000) 2019-10-13T10:04:08.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:08.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:08.464765+0000) 2019-10-13T10:04:09.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:09.030+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:09.030902+0000) 2019-10-13T10:04:10.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:10.746+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.752040+0000) 2019-10-13T10:04:10.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:10.746+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.752830+0000) 2019-10-13T10:04:10.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:10.746+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.753128+0000) 2019-10-13T10:04:10.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:10.750+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.753572+0000) 2019-10-13T10:04:10.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:10.746+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.753449+0000) 2019-10-13T10:04:10.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:10.750+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.754676+0000) 2019-10-13T10:04:10.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:10.750+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.755061+0000) 2019-10-13T10:04:10.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:10.750+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:10.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:10.754+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:10.758360+0000) 2019-10-13T10:04:11.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:11.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.167518+0000) 2019-10-13T10:04:11.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:11.714+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:11.716430+0000) 2019-10-13T10:04:14.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:14.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.032162+0000) 2019-10-13T10:04:14.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:14.314+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.319852+0000) 2019-10-13T10:04:14.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:14.438+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:14.441788+0000) 2019-10-13T10:04:15.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:15.758+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.761293+0000) 2019-10-13T10:04:15.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:15.758+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.762035+0000) 2019-10-13T10:04:15.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:15.758+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.762435+0000) 2019-10-13T10:04:15.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:15.758+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.763350+0000) 2019-10-13T10:04:15.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:15.758+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.763947+0000) 2019-10-13T10:04:15.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:15.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:15.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:15.758+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.764527+0000) 2019-10-13T10:04:15.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:15.758+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.765062+0000) 2019-10-13T10:04:15.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:15.770+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:15.773191+0000) 2019-10-13T10:04:17.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:17.690+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:17.693766+0000) 2019-10-13T10:04:18.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:18.458+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:18.464956+0000) 2019-10-13T10:04:19.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:19.034+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:19.037328+0000) 2019-10-13T10:04:20.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:20.762+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.767794+0000) 2019-10-13T10:04:20.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:20.762+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.768629+0000) 2019-10-13T10:04:20.770 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:20.762+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.769573+0000) 2019-10-13T10:04:20.771 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:20.766+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.770012+0000) 2019-10-13T10:04:20.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:20.766+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.770124+0000) 2019-10-13T10:04:20.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:20.766+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.770533+0000) 2019-10-13T10:04:20.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:20.766+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.770893+0000) 2019-10-13T10:04:20.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:20.766+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:20.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:20.770+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:20.773321+0000) 2019-10-13T10:04:21.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:21.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.167697+0000) 2019-10-13T10:04:21.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:21.714+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:21.716620+0000) 2019-10-13T10:04:24.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:24.026+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.032357+0000) 2019-10-13T10:04:24.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:24.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.320058+0000) 2019-10-13T10:04:24.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:24.438+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:24.441990+0000) 2019-10-13T10:04:25.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:25.770+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.776733+0000) 2019-10-13T10:04:25.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:25.770+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.777237+0000) 2019-10-13T10:04:25.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:25.770+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.777620+0000) 2019-10-13T10:04:25.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:25.774+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.777667+0000) 2019-10-13T10:04:25.779 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:25.774+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.778686+0000) 2019-10-13T10:04:25.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:25.774+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.779123+0000) 2019-10-13T10:04:25.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:25.774+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.780352+0000) 2019-10-13T10:04:25.781 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:25.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:25.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:25.778+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:25.782310+0000) 2019-10-13T10:04:27.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:27.690+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:27.693977+0000) 2019-10-13T10:04:28.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:28.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:28.465150+0000) 2019-10-13T10:04:29.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:29.037+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:29.044029+0000) 2019-10-13T10:04:30.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:30.778+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.784187+0000) 2019-10-13T10:04:30.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:30.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.785489+0000) 2019-10-13T10:04:30.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:30.778+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.784848+0000) 2019-10-13T10:04:30.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:30.778+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.785451+0000) 2019-10-13T10:04:30.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:30.782+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.786432+0000) 2019-10-13T10:04:30.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:30.782+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.786829+0000) 2019-10-13T10:04:30.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:30.782+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.787389+0000) 2019-10-13T10:04:30.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:30.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:30.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:30.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:30.793870+0000) 2019-10-13T10:04:31.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:31.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.167887+0000) 2019-10-13T10:04:31.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:31.713+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:31.716810+0000) 2019-10-13T10:04:33.345 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:04:33.345 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:33.349 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:04:33.349 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:04:34.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:34.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.032578+0000) 2019-10-13T10:04:34.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:34.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.320295+0000) 2019-10-13T10:04:34.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:34.442+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:34.442190+0000) 2019-10-13T10:04:35.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:35.786+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.793166+0000) 2019-10-13T10:04:35.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:35.790+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.793937+0000) 2019-10-13T10:04:35.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:35.790+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.795009+0000) 2019-10-13T10:04:35.796 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:35.790+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.794881+0000) 2019-10-13T10:04:35.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:35.789+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.795524+0000) 2019-10-13T10:04:35.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:35.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.795529+0000) 2019-10-13T10:04:35.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:35.790+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.796613+0000) 2019-10-13T10:04:35.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:35.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:35.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:35.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:35.802170+0000) 2019-10-13T10:04:37.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:37.694+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:37.694177+0000) 2019-10-13T10:04:38.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:38.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:38.465352+0000) 2019-10-13T10:04:39.052 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:39.045+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:39.050619+0000) 2019-10-13T10:04:40.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:40.797+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.804055+0000) 2019-10-13T10:04:40.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:40.797+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.804664+0000) 2019-10-13T10:04:40.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:40.797+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.805013+0000) 2019-10-13T10:04:40.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:40.801+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.804966+0000) 2019-10-13T10:04:40.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:40.801+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.805864+0000) 2019-10-13T10:04:40.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:40.801+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.805988+0000) 2019-10-13T10:04:40.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:40.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:40.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:40.801+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.808148+0000) 2019-10-13T10:04:40.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:40.809+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:40.813483+0000) 2019-10-13T10:04:41.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:41.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.168110+0000) 2019-10-13T10:04:41.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:41.713+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:41.716986+0000) 2019-10-13T10:04:44.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:44.029+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.032813+0000) 2019-10-13T10:04:44.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:44.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.320513+0000) 2019-10-13T10:04:44.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:44.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:44.442408+0000) 2019-10-13T10:04:45.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:45.805+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.811137+0000) 2019-10-13T10:04:45.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:45.805+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.811297+0000) 2019-10-13T10:04:45.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:45.805+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.812041+0000) 2019-10-13T10:04:45.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:45.809+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.814191+0000) 2019-10-13T10:04:45.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:45.809+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.814324+0000) 2019-10-13T10:04:45.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:45.809+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.815774+0000) 2019-10-13T10:04:45.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:45.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:45.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:45.809+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.817218+0000) 2019-10-13T10:04:45.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:45.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:45.821481+0000) 2019-10-13T10:04:47.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:47.693+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:47.694409+0000) 2019-10-13T10:04:48.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:48.457+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:48.465552+0000) 2019-10-13T10:04:49.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:49.053+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:49.057250+0000) 2019-10-13T10:04:50.820 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:50.813+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.819654+0000) 2019-10-13T10:04:50.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:50.813+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.819991+0000) 2019-10-13T10:04:50.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:50.817+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.822069+0000) 2019-10-13T10:04:50.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:50.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.822284+0000) 2019-10-13T10:04:50.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:50.817+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.823007+0000) 2019-10-13T10:04:50.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:50.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:50.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:50.817+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.825608+0000) 2019-10-13T10:04:50.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:50.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.826188+0000) 2019-10-13T10:04:50.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:50.829+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:50.832277+0000) 2019-10-13T10:04:51.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:51.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.168318+0000) 2019-10-13T10:04:51.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:51.713+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:51.717153+0000) 2019-10-13T10:04:54.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:54.029+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.033015+0000) 2019-10-13T10:04:54.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:54.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.320701+0000) 2019-10-13T10:04:54.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:54.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:54.442612+0000) 2019-10-13T10:04:55.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:04:55.825+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.828704+0000) 2019-10-13T10:04:55.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:55.825+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.829110+0000) 2019-10-13T10:04:55.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:04:55.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.830529+0000) 2019-10-13T10:04:55.831 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:04:55.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.831151+0000) 2019-10-13T10:04:55.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:04:55.825+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.831076+0000) 2019-10-13T10:04:55.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:55.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.831614+0000) 2019-10-13T10:04:55.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:04:55.829+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.831983+0000) 2019-10-13T10:04:55.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:04:55.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:04:55.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:55.829+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:55.835218+0000) 2019-10-13T10:04:57.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:04:57.693+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:57.694613+0000) 2019-10-13T10:04:58.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:04:58.465+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:58.465776+0000) 2019-10-13T10:04:59.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:04:59.061+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:04:59.064177+0000) 2019-10-13T10:05:00.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:00.833+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.837353+0000) 2019-10-13T10:05:00.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:00.833+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.837258+0000) 2019-10-13T10:05:00.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:00.837+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.838664+0000) 2019-10-13T10:05:00.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:00.837+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.838927+0000) 2019-10-13T10:05:00.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:00.837+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.839214+0000) 2019-10-13T10:05:00.841 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:00.837+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.840716+0000) 2019-10-13T10:05:00.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:00.837+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.841208+0000) 2019-10-13T10:05:00.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:00.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:00.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:00.841+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:00.844291+0000) 2019-10-13T10:05:01.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:01.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.168495+0000) 2019-10-13T10:05:01.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:01.713+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:01.717324+0000) 2019-10-13T10:05:03.437 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:05:03.438 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:03.441 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:05:03.441 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:04.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:04.029+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.033224+0000) 2019-10-13T10:05:04.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:04.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.320900+0000) 2019-10-13T10:05:04.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:04.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:04.442825+0000) 2019-10-13T10:05:05.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:05.845+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.847028+0000) 2019-10-13T10:05:05.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:05.841+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.846827+0000) 2019-10-13T10:05:05.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:05.845+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.847720+0000) 2019-10-13T10:05:05.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:05.845+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.848310+0000) 2019-10-13T10:05:05.850 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:05.845+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.848800+0000) 2019-10-13T10:05:05.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:05.845+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.849467+0000) 2019-10-13T10:05:05.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:05.849+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.850308+0000) 2019-10-13T10:05:05.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:05.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:05.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:05.853+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:05.857575+0000) 2019-10-13T10:05:07.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:07.693+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:07.694842+0000) 2019-10-13T10:05:08.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:08.465+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:08.465967+0000) 2019-10-13T10:05:09.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:09.069+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:09.070496+0000) 2019-10-13T10:05:10.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:10.853+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.855740+0000) 2019-10-13T10:05:10.858 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:10.857+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.857780+0000) 2019-10-13T10:05:10.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:10.857+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.858064+0000) 2019-10-13T10:05:10.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:10.857+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.859803+0000) 2019-10-13T10:05:10.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:10.853+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.859847+0000) 2019-10-13T10:05:10.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:10.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:10.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:10.857+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.860942+0000) 2019-10-13T10:05:10.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:10.857+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.861531+0000) 2019-10-13T10:05:10.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:10.865+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:10.869464+0000) 2019-10-13T10:05:11.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:11.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.168675+0000) 2019-10-13T10:05:11.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:11.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:11.717498+0000) 2019-10-13T10:05:14.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:14.029+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.033439+0000) 2019-10-13T10:05:14.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:14.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.321144+0000) 2019-10-13T10:05:14.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:14.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:14.443026+0000) 2019-10-13T10:05:15.864 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:15.861+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.863367+0000) 2019-10-13T10:05:15.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:15.861+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.864705+0000) 2019-10-13T10:05:15.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:15.861+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.865124+0000) 2019-10-13T10:05:15.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:15.861+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.865612+0000) 2019-10-13T10:05:15.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:15.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.866519+0000) 2019-10-13T10:05:15.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:15.865+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.866831+0000) 2019-10-13T10:05:15.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:15.865+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.867411+0000) 2019-10-13T10:05:15.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:15.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:15.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:15.869+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:15.872934+0000) 2019-10-13T10:05:17.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:17.693+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:17.695046+0000) 2019-10-13T10:05:18.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:18.465+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:18.466170+0000) 2019-10-13T10:05:19.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:19.073+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:19.076948+0000) 2019-10-13T10:05:20.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:20.869+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.872890+0000) 2019-10-13T10:05:20.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:20.873+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.874098+0000) 2019-10-13T10:05:20.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:20.873+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.874607+0000) 2019-10-13T10:05:20.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:20.873+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.874825+0000) 2019-10-13T10:05:20.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:20.873+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.875173+0000) 2019-10-13T10:05:20.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:20.873+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.876189+0000) 2019-10-13T10:05:20.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:20.873+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.876425+0000) 2019-10-13T10:05:20.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:20.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:20.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:20.877+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:20.880264+0000) 2019-10-13T10:05:21.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:21.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.168888+0000) 2019-10-13T10:05:21.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:21.713+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:21.717665+0000) 2019-10-13T10:05:24.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:24.029+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.033660+0000) 2019-10-13T10:05:24.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:24.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.321301+0000) 2019-10-13T10:05:24.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:24.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:24.443243+0000) 2019-10-13T10:05:25.883 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:25.881+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.882154+0000) 2019-10-13T10:05:25.883 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:25.881+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.882909+0000) 2019-10-13T10:05:25.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:25.881+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.883350+0000) 2019-10-13T10:05:25.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:25.881+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.883805+0000) 2019-10-13T10:05:25.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:25.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.884528+0000) 2019-10-13T10:05:25.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:25.881+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.884405+0000) 2019-10-13T10:05:25.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:25.881+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.885060+0000) 2019-10-13T10:05:25.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:25.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:25.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:25.885+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:25.888510+0000) 2019-10-13T10:05:27.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:27.693+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:27.695238+0000) 2019-10-13T10:05:28.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:28.465+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:28.466365+0000) 2019-10-13T10:05:29.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:29.081+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:29.083277+0000) 2019-10-13T10:05:30.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:30.889+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.890373+0000) 2019-10-13T10:05:30.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:30.889+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.890882+0000) 2019-10-13T10:05:30.892 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:30.889+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.891213+0000) 2019-10-13T10:05:30.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:30.889+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.892700+0000) 2019-10-13T10:05:30.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:30.889+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.893244+0000) 2019-10-13T10:05:30.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:30.889+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.893262+0000) 2019-10-13T10:05:30.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:30.893+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.894363+0000) 2019-10-13T10:05:30.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:30.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:30.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:30.897+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:30.900527+0000) 2019-10-13T10:05:31.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:31.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.169096+0000) 2019-10-13T10:05:31.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:31.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:31.717839+0000) 2019-10-13T10:05:33.544 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:05:33.544 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:33.547 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:05:33.548 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:05:34.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:34.033+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.033867+0000) 2019-10-13T10:05:34.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:34.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.321514+0000) 2019-10-13T10:05:34.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:34.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:34.443441+0000) 2019-10-13T10:05:35.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:35.896+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.899536+0000) 2019-10-13T10:05:35.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:35.896+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.899965+0000) 2019-10-13T10:05:35.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:35.896+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.900197+0000) 2019-10-13T10:05:35.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:35.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.900456+0000) 2019-10-13T10:05:35.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:35.896+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.900637+0000) 2019-10-13T10:05:35.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:35.896+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.901298+0000) 2019-10-13T10:05:35.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:35.896+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.901704+0000) 2019-10-13T10:05:35.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:35.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:35.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:35.904+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:35.909758+0000) 2019-10-13T10:05:37.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:37.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:37.695445+0000) 2019-10-13T10:05:38.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:38.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:38.466574+0000) 2019-10-13T10:05:39.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:39.088+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:39.090117+0000) 2019-10-13T10:05:40.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:40.904+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.908287+0000) 2019-10-13T10:05:40.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:40.908+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.911173+0000) 2019-10-13T10:05:40.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:40.908+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.911362+0000) 2019-10-13T10:05:40.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:40.908+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.912046+0000) 2019-10-13T10:05:40.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:40.908+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.912406+0000) 2019-10-13T10:05:40.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:40.908+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.912476+0000) 2019-10-13T10:05:40.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:40.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:40.914 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:40.908+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.913574+0000) 2019-10-13T10:05:40.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:40.912+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:40.917582+0000) 2019-10-13T10:05:41.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:41.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.169290+0000) 2019-10-13T10:05:41.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:41.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:41.718042+0000) 2019-10-13T10:05:44.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:44.032+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.034079+0000) 2019-10-13T10:05:44.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:44.316+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.321716+0000) 2019-10-13T10:05:44.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:44.440+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:44.443656+0000) 2019-10-13T10:05:45.917 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:45.912+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.916427+0000) 2019-10-13T10:05:45.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:45.912+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.916790+0000) 2019-10-13T10:05:45.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:45.912+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.917488+0000) 2019-10-13T10:05:45.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:45.916+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.918706+0000) 2019-10-13T10:05:45.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:45.916+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.919409+0000) 2019-10-13T10:05:45.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:45.916+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.919831+0000) 2019-10-13T10:05:45.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:45.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.919799+0000) 2019-10-13T10:05:45.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:45.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:45.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:45.920+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:45.927309+0000) 2019-10-13T10:05:47.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:47.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:47.695707+0000) 2019-10-13T10:05:48.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:48.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:48.466720+0000) 2019-10-13T10:05:49.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:49.092+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:49.096790+0000) 2019-10-13T10:05:50.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:50.920+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.924350+0000) 2019-10-13T10:05:50.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:50.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.925072+0000) 2019-10-13T10:05:50.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:50.920+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.925557+0000) 2019-10-13T10:05:50.928 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:50.924+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.927598+0000) 2019-10-13T10:05:50.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:50.924+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.928217+0000) 2019-10-13T10:05:50.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:50.928+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.929730+0000) 2019-10-13T10:05:50.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:50.928+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.930238+0000) 2019-10-13T10:05:50.931 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:50.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:50.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:50.928+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:50.932996+0000) 2019-10-13T10:05:51.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:51.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.169492+0000) 2019-10-13T10:05:51.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:51.712+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:51.718242+0000) 2019-10-13T10:05:54.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:54.032+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.034280+0000) 2019-10-13T10:05:54.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:54.320+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.321936+0000) 2019-10-13T10:05:54.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:54.440+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:54.443853+0000) 2019-10-13T10:05:55.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:55.928+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.933667+0000) 2019-10-13T10:05:55.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:05:55.932+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.934107+0000) 2019-10-13T10:05:55.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:55.932+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.934515+0000) 2019-10-13T10:05:55.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:05:55.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.936530+0000) 2019-10-13T10:05:55.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:05:55.936+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.938518+0000) 2019-10-13T10:05:55.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:05:55.936+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:05:55.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:05:55.932+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.940037+0000) 2019-10-13T10:05:55.941 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:05:55.936+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.940682+0000) 2019-10-13T10:05:55.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:55.944+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:55.948431+0000) 2019-10-13T10:05:57.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:05:57.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:57.695952+0000) 2019-10-13T10:05:58.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:05:58.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:58.466944+0000) 2019-10-13T10:05:59.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:05:59.100+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:05:59.103234+0000) 2019-10-13T10:06:00.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:00.940+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.941923+0000) 2019-10-13T10:06:00.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:00.940+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.942487+0000) 2019-10-13T10:06:00.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:00.940+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.944303+0000) 2019-10-13T10:06:00.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:00.944+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:00.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:00.944+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.948115+0000) 2019-10-13T10:06:00.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:00.944+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.948988+0000) 2019-10-13T10:06:00.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:00.944+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.949079+0000) 2019-10-13T10:06:00.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:00.948+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.949858+0000) 2019-10-13T10:06:00.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:00.952+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:00.956374+0000) 2019-10-13T10:06:01.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:01.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.169673+0000) 2019-10-13T10:06:01.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:01.712+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:01.718427+0000) 2019-10-13T10:06:03.639 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:06:03.639 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:03.643 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:06:03.643 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:04.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:04.032+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.034519+0000) 2019-10-13T10:06:04.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:04.320+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.322141+0000) 2019-10-13T10:06:04.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:04.440+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:04.444080+0000) 2019-10-13T10:06:05.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:05.944+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.950459+0000) 2019-10-13T10:06:05.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:05.948+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.951249+0000) 2019-10-13T10:06:05.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:05.948+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.951633+0000) 2019-10-13T10:06:05.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:05.948+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.952960+0000) 2019-10-13T10:06:05.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:05.948+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.953682+0000) 2019-10-13T10:06:05.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:05.952+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.954260+0000) 2019-10-13T10:06:05.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:05.952+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.954597+0000) 2019-10-13T10:06:05.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:05.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:05.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:05.952+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:05.960000+0000) 2019-10-13T10:06:07.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:07.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:07.696148+0000) 2019-10-13T10:06:08.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:08.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:08.467135+0000) 2019-10-13T10:06:09.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:09.112+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:09.114786+0000) 2019-10-13T10:06:10.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:10.956+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.960111+0000) 2019-10-13T10:06:10.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:10.956+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.960879+0000) 2019-10-13T10:06:10.963 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:10.956+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.961440+0000) 2019-10-13T10:06:10.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:10.960+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.961977+0000) 2019-10-13T10:06:10.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:10.956+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.962102+0000) 2019-10-13T10:06:10.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:10.960+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.962817+0000) 2019-10-13T10:06:10.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:10.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.962622+0000) 2019-10-13T10:06:10.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:10.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109227 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:10.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:10.964+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:10.969393+0000) 2019-10-13T10:06:11.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:11.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.169857+0000) 2019-10-13T10:06:11.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:11.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:11.718656+0000) 2019-10-13T10:06:14.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:14.032+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.034732+0000) 2019-10-13T10:06:14.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:14.320+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.322374+0000) 2019-10-13T10:06:14.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:14.440+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:14.444337+0000) 2019-10-13T10:06:15.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:15.964+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.967412+0000) 2019-10-13T10:06:15.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:15.964+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.968562+0000) 2019-10-13T10:06:15.970 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:15.964+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.969046+0000) 2019-10-13T10:06:15.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:15.964+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.969602+0000) 2019-10-13T10:06:15.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:15.968+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.970191+0000) 2019-10-13T10:06:15.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:15.968+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.970315+0000) 2019-10-13T10:06:15.971 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:15.968+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.970668+0000) 2019-10-13T10:06:15.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:15.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:15.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:15.972+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:15.976892+0000) 2019-10-13T10:06:17.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:17.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:17.696355+0000) 2019-10-13T10:06:18.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:18.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:18.467332+0000) 2019-10-13T10:06:19.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:19.120+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:19.121307+0000) 2019-10-13T10:06:20.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:20.972+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.976674+0000) 2019-10-13T10:06:20.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:20.972+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.977664+0000) 2019-10-13T10:06:20.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:20.976+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.977875+0000) 2019-10-13T10:06:20.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:20.976+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.978375+0000) 2019-10-13T10:06:20.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:20.976+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.979029+0000) 2019-10-13T10:06:20.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:20.976+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.979392+0000) 2019-10-13T10:06:20.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:20.976+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.979824+0000) 2019-10-13T10:06:20.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:20.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:20.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:20.980+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:20.984518+0000) 2019-10-13T10:06:21.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:21.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.169993+0000) 2019-10-13T10:06:21.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:21.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:21.718826+0000) 2019-10-13T10:06:24.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:24.032+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.034963+0000) 2019-10-13T10:06:24.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:24.320+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.322600+0000) 2019-10-13T10:06:24.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:24.444+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:24.444537+0000) 2019-10-13T10:06:25.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:25.984+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.986149+0000) 2019-10-13T10:06:25.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:25.984+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.988140+0000) 2019-10-13T10:06:25.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:25.984+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.988141+0000) 2019-10-13T10:06:25.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:25.984+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.989050+0000) 2019-10-13T10:06:25.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:25.984+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.989649+0000) 2019-10-13T10:06:25.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:25.988+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.990367+0000) 2019-10-13T10:06:25.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:25.988+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.990751+0000) 2019-10-13T10:06:25.992 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:25.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109319 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:25.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:25.988+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:25.993803+0000) 2019-10-13T10:06:27.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:27.692+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:27.696596+0000) 2019-10-13T10:06:28.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:28.464+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:28.467529+0000) 2019-10-13T10:06:29.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:29.124+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:29.127828+0000) 2019-10-13T10:06:30.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:30.992+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.996387+0000) 2019-10-13T10:06:30.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:30.996+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.998446+0000) 2019-10-13T10:06:30.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:30.992+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.998306+0000) 2019-10-13T10:06:31.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:30.996+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.999034+0000) 2019-10-13T10:06:31.000 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:30.996+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:30.999487+0000) 2019-10-13T10:06:31.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:30.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:31.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:30.996+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.000008+0000) 2019-10-13T10:06:31.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:30.996+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.000438+0000) 2019-10-13T10:06:31.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:31.000+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.007808+0000) 2019-10-13T10:06:31.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:31.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.170173+0000) 2019-10-13T10:06:31.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:31.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:31.718995+0000) 2019-10-13T10:06:33.736 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:06:33.737 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:33.741 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:06:33.741 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:06:34.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:34.031+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.035172+0000) 2019-10-13T10:06:34.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:34.319+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.322819+0000) 2019-10-13T10:06:34.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:34.439+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:34.444760+0000) 2019-10-13T10:06:36.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:35.999+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.003234+0000) 2019-10-13T10:06:36.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:35.999+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.004195+0000) 2019-10-13T10:06:36.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:35.999+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.005098+0000) 2019-10-13T10:06:36.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:36.004+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.005166+0000) 2019-10-13T10:06:36.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:36.003+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.005805+0000) 2019-10-13T10:06:36.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:36.003+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.006135+0000) 2019-10-13T10:06:36.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:36.003+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.006710+0000) 2019-10-13T10:06:36.008 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:36.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:36.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:36.008+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:36.010990+0000) 2019-10-13T10:06:37.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:37.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:37.696812+0000) 2019-10-13T10:06:38.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:38.463+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:38.467722+0000) 2019-10-13T10:06:39.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:39.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:39.139259+0000) 2019-10-13T10:06:41.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:41.007+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.010957+0000) 2019-10-13T10:06:41.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:41.007+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.011957+0000) 2019-10-13T10:06:41.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:41.007+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.012714+0000) 2019-10-13T10:06:41.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:41.007+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.012369+0000) 2019-10-13T10:06:41.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:41.007+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.013544+0000) 2019-10-13T10:06:41.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:41.011+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.013558+0000) 2019-10-13T10:06:41.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:41.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.014367+0000) 2019-10-13T10:06:41.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:41.011+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:41.022 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:41.019+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.020983+0000) 2019-10-13T10:06:41.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:41.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.170369+0000) 2019-10-13T10:06:41.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:41.715+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:41.719165+0000) 2019-10-13T10:06:44.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:44.031+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.035412+0000) 2019-10-13T10:06:44.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:44.319+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.323048+0000) 2019-10-13T10:06:44.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:44.439+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:44.444966+0000) 2019-10-13T10:06:46.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:46.015+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.020178+0000) 2019-10-13T10:06:46.021 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:46.015+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.020898+0000) 2019-10-13T10:06:46.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:46.019+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.022860+0000) 2019-10-13T10:06:46.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:46.019+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.022839+0000) 2019-10-13T10:06:46.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:46.019+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.024259+0000) 2019-10-13T10:06:46.025 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:46.019+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.024773+0000) 2019-10-13T10:06:46.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:46.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:46.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:46.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.025161+0000) 2019-10-13T10:06:46.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:46.027+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:46.030112+0000) 2019-10-13T10:06:47.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:47.691+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:47.697015+0000) 2019-10-13T10:06:48.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:48.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:48.467916+0000) 2019-10-13T10:06:49.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:49.143+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:49.145716+0000) 2019-10-13T10:06:51.029 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:51.023+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.028439+0000) 2019-10-13T10:06:51.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:51.023+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.029016+0000) 2019-10-13T10:06:51.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:51.027+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.031439+0000) 2019-10-13T10:06:51.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:51.031+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.033003+0000) 2019-10-13T10:06:51.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:51.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:51.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:51.031+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.033782+0000) 2019-10-13T10:06:51.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:51.031+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.034239+0000) 2019-10-13T10:06:51.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:51.031+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.034642+0000) 2019-10-13T10:06:51.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:51.035+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.037572+0000) 2019-10-13T10:06:51.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:51.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.170551+0000) 2019-10-13T10:06:51.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:51.715+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:51.719361+0000) 2019-10-13T10:06:54.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:54.031+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.035620+0000) 2019-10-13T10:06:54.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:54.319+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.323262+0000) 2019-10-13T10:06:54.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:54.443+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:54.445197+0000) 2019-10-13T10:06:56.038 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:56.031+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.037309+0000) 2019-10-13T10:06:56.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:06:56.035+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.039078+0000) 2019-10-13T10:06:56.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:06:56.039+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.040509+0000) 2019-10-13T10:06:56.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:06:56.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.041148+0000) 2019-10-13T10:06:56.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:06:56.039+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.042383+0000) 2019-10-13T10:06:56.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:06:56.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:06:56.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:06:56.039+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.042959+0000) 2019-10-13T10:06:56.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:56.039+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.043383+0000) 2019-10-13T10:06:56.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:56.047+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:56.049350+0000) 2019-10-13T10:06:57.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:06:57.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:57.697215+0000) 2019-10-13T10:06:58.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:06:58.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:58.468101+0000) 2019-10-13T10:06:59.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:06:59.151+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:06:59.155066+0000) 2019-10-13T10:07:01.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:01.043+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.045323+0000) 2019-10-13T10:07:01.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:01.043+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.046601+0000) 2019-10-13T10:07:01.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:01.043+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.047752+0000) 2019-10-13T10:07:01.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:01.043+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.047349+0000) 2019-10-13T10:07:01.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:01.043+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.048040+0000) 2019-10-13T10:07:01.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:01.043+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.048630+0000) 2019-10-13T10:07:01.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:01.043+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.049051+0000) 2019-10-13T10:07:01.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:01.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:01.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:01.051+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.054691+0000) 2019-10-13T10:07:01.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:01.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.170749+0000) 2019-10-13T10:07:01.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:01.715+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:01.719582+0000) 2019-10-13T10:07:03.809 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:07:03.809 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:03.812 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:07:03.813 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:04.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:04.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.035822+0000) 2019-10-13T10:07:04.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:04.319+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.323464+0000) 2019-10-13T10:07:04.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:04.443+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:04.445396+0000) 2019-10-13T10:07:06.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:06.051+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.054808+0000) 2019-10-13T10:07:06.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:06.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.055486+0000) 2019-10-13T10:07:06.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:06.051+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.055650+0000) 2019-10-13T10:07:06.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:06.051+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.057505+0000) 2019-10-13T10:07:06.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:06.055+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.059108+0000) 2019-10-13T10:07:06.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:06.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:06.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:06.055+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.058956+0000) 2019-10-13T10:07:06.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:06.055+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.059901+0000) 2019-10-13T10:07:06.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:06.063+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:06.067498+0000) 2019-10-13T10:07:07.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:07.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:07.697445+0000) 2019-10-13T10:07:08.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:08.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:08.468316+0000) 2019-10-13T10:07:09.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:09.159+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:09.161555+0000) 2019-10-13T10:07:11.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:11.059+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.062617+0000) 2019-10-13T10:07:11.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:11.059+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.063383+0000) 2019-10-13T10:07:11.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:11.059+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.064158+0000) 2019-10-13T10:07:11.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:11.059+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.064516+0000) 2019-10-13T10:07:11.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:11.059+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.065123+0000) 2019-10-13T10:07:11.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:11.063+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.065782+0000) 2019-10-13T10:07:11.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:11.059+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.065378+0000) 2019-10-13T10:07:11.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:11.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:11.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:11.067+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.069127+0000) 2019-10-13T10:07:11.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:11.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.170944+0000) 2019-10-13T10:07:11.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:11.715+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:11.719779+0000) 2019-10-13T10:07:14.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:14.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.036027+0000) 2019-10-13T10:07:14.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:14.323+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.323700+0000) 2019-10-13T10:07:14.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:14.443+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:14.445601+0000) 2019-10-13T10:07:16.072 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:16.067+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.071527+0000) 2019-10-13T10:07:16.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:16.067+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.071787+0000) 2019-10-13T10:07:16.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:16.067+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.072281+0000) 2019-10-13T10:07:16.074 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:16.067+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.073028+0000) 2019-10-13T10:07:16.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:16.067+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.073668+0000) 2019-10-13T10:07:16.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:16.071+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.074071+0000) 2019-10-13T10:07:16.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:16.071+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.074067+0000) 2019-10-13T10:07:16.077 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:16.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:16.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:16.075+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:16.078493+0000) 2019-10-13T10:07:17.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:17.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:17.697677+0000) 2019-10-13T10:07:18.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:18.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:18.468510+0000) 2019-10-13T10:07:19.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:19.167+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:19.168154+0000) 2019-10-13T10:07:21.080 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:21.075+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.079437+0000) 2019-10-13T10:07:21.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:21.075+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.081584+0000) 2019-10-13T10:07:21.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:21.079+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.081979+0000) 2019-10-13T10:07:21.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:21.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.082601+0000) 2019-10-13T10:07:21.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:21.079+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.083320+0000) 2019-10-13T10:07:21.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:21.079+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.083590+0000) 2019-10-13T10:07:21.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:21.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:21.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:21.083+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.084201+0000) 2019-10-13T10:07:21.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:21.087+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.089729+0000) 2019-10-13T10:07:21.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:21.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.171162+0000) 2019-10-13T10:07:21.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:21.719+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:21.719955+0000) 2019-10-13T10:07:24.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:24.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.036255+0000) 2019-10-13T10:07:24.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:24.323+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.323949+0000) 2019-10-13T10:07:24.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:24.443+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:24.445836+0000) 2019-10-13T10:07:26.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:26.083+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.087206+0000) 2019-10-13T10:07:26.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:26.083+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.087604+0000) 2019-10-13T10:07:26.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:26.083+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.088317+0000) 2019-10-13T10:07:26.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:26.083+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.089338+0000) 2019-10-13T10:07:26.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:26.087+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.090986+0000) 2019-10-13T10:07:26.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:26.087+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.091244+0000) 2019-10-13T10:07:26.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:26.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:26.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:26.087+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.092566+0000) 2019-10-13T10:07:26.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:26.099+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:26.102067+0000) 2019-10-13T10:07:27.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:27.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:27.697897+0000) 2019-10-13T10:07:28.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:28.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:28.468708+0000) 2019-10-13T10:07:29.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:29.171+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:29.174670+0000) 2019-10-13T10:07:31.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:31.090+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.095415+0000) 2019-10-13T10:07:31.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:31.090+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.095827+0000) 2019-10-13T10:07:31.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:31.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.096761+0000) 2019-10-13T10:07:31.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:31.090+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.097286+0000) 2019-10-13T10:07:31.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:31.095+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.097136+0000) 2019-10-13T10:07:31.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:31.094+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.098317+0000) 2019-10-13T10:07:31.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:31.094+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.099092+0000) 2019-10-13T10:07:31.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:31.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:31.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:31.099+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.102405+0000) 2019-10-13T10:07:31.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:31.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.171358+0000) 2019-10-13T10:07:31.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:31.719+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:31.720128+0000) 2019-10-13T10:07:33.931 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:07:33.932 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:33.935 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:07:33.935 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:07:34.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:34.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.036460+0000) 2019-10-13T10:07:34.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:34.322+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.324179+0000) 2019-10-13T10:07:34.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:34.442+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:34.446036+0000) 2019-10-13T10:07:36.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:36.098+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.105151+0000) 2019-10-13T10:07:36.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:36.102+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.105904+0000) 2019-10-13T10:07:36.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:36.102+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.107797+0000) 2019-10-13T10:07:36.109 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:36.102+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.107905+0000) 2019-10-13T10:07:36.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:36.107+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.108950+0000) 2019-10-13T10:07:36.110 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:36.102+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.109532+0000) 2019-10-13T10:07:36.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:36.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:36.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:36.106+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.110774+0000) 2019-10-13T10:07:36.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:36.111+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:36.112480+0000) 2019-10-13T10:07:37.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:37.694+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:37.698134+0000) 2019-10-13T10:07:38.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:38.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:38.468918+0000) 2019-10-13T10:07:39.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:39.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:39.184244+0000) 2019-10-13T10:07:41.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:41.106+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.113533+0000) 2019-10-13T10:07:41.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:41.110+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.114355+0000) 2019-10-13T10:07:41.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:41.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.114735+0000) 2019-10-13T10:07:41.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:41.110+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.116315+0000) 2019-10-13T10:07:41.117 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:41.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.116400+0000) 2019-10-13T10:07:41.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:41.110+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.117689+0000) 2019-10-13T10:07:41.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:41.114+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.118455+0000) 2019-10-13T10:07:41.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:41.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:41.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:41.123+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.124786+0000) 2019-10-13T10:07:41.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:41.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.171543+0000) 2019-10-13T10:07:41.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:41.719+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:41.720334+0000) 2019-10-13T10:07:44.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:44.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.036664+0000) 2019-10-13T10:07:44.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:44.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.324396+0000) 2019-10-13T10:07:44.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:44.442+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:44.446268+0000) 2019-10-13T10:07:46.124 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:46.118+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.122935+0000) 2019-10-13T10:07:46.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:46.118+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.123545+0000) 2019-10-13T10:07:46.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:46.118+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.123813+0000) 2019-10-13T10:07:46.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:46.118+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.124152+0000) 2019-10-13T10:07:46.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:46.118+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.124608+0000) 2019-10-13T10:07:46.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:46.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.124673+0000) 2019-10-13T10:07:46.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:46.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.126194+0000) 2019-10-13T10:07:46.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:46.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:46.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:46.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:46.131776+0000) 2019-10-13T10:07:47.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:47.694+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:47.698310+0000) 2019-10-13T10:07:48.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:48.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:48.469135+0000) 2019-10-13T10:07:49.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:49.190+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:49.191674+0000) 2019-10-13T10:07:51.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:51.126+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.131032+0000) 2019-10-13T10:07:51.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:51.126+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.131414+0000) 2019-10-13T10:07:51.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:51.126+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.132043+0000) 2019-10-13T10:07:51.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:51.130+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.132491+0000) 2019-10-13T10:07:51.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:51.126+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.133176+0000) 2019-10-13T10:07:51.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:51.130+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.134242+0000) 2019-10-13T10:07:51.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:51.130+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.135863+0000) 2019-10-13T10:07:51.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:51.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:51.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:51.134+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.139381+0000) 2019-10-13T10:07:51.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:51.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.171720+0000) 2019-10-13T10:07:51.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:51.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:51.720505+0000) 2019-10-13T10:07:54.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:54.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.036862+0000) 2019-10-13T10:07:54.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:54.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.324598+0000) 2019-10-13T10:07:54.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:54.442+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:54.446463+0000) 2019-10-13T10:07:56.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:07:56.134+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.139453+0000) 2019-10-13T10:07:56.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:56.134+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.139899+0000) 2019-10-13T10:07:56.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:56.134+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.140317+0000) 2019-10-13T10:07:56.142 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:07:56.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.140719+0000) 2019-10-13T10:07:56.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:07:56.134+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.141630+0000) 2019-10-13T10:07:56.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:07:56.138+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.142075+0000) 2019-10-13T10:07:56.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:07:56.138+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.143014+0000) 2019-10-13T10:07:56.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:07:56.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:07:56.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:56.146+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:56.148849+0000) 2019-10-13T10:07:57.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:07:57.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:57.698557+0000) 2019-10-13T10:07:58.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:07:58.466+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:58.469328+0000) 2019-10-13T10:07:59.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:07:59.194+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:07:59.197836+0000) 2019-10-13T10:08:01.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:01.142+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.148285+0000) 2019-10-13T10:08:01.149 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:01.142+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.148811+0000) 2019-10-13T10:08:01.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:01.142+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.149487+0000) 2019-10-13T10:08:01.150 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:01.146+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.150207+0000) 2019-10-13T10:08:01.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:01.146+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.153649+0000) 2019-10-13T10:08:01.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:01.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:01.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:01.150+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.154731+0000) 2019-10-13T10:08:01.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:01.150+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.154287+0000) 2019-10-13T10:08:01.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:01.154+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.156728+0000) 2019-10-13T10:08:01.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:01.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.171925+0000) 2019-10-13T10:08:01.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:01.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:01.720709+0000) 2019-10-13T10:08:04.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:04.034+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.037108+0000) 2019-10-13T10:08:04.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:08:04.061 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:04.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:08:04.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:04.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:04.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.324800+0000) 2019-10-13T10:08:04.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:04.446+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:04.446667+0000) 2019-10-13T10:08:06.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:06.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.158884+0000) 2019-10-13T10:08:06.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:06.150+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.156934+0000) 2019-10-13T10:08:06.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:06.154+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.157885+0000) 2019-10-13T10:08:06.162 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:06.154+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.158164+0000) 2019-10-13T10:08:06.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:06.154+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.158940+0000) 2019-10-13T10:08:06.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:06.154+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.159411+0000) 2019-10-13T10:08:06.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:06.154+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.159922+0000) 2019-10-13T10:08:06.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:06.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:06.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:06.162+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:06.165088+0000) 2019-10-13T10:08:07.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:07.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:07.698739+0000) 2019-10-13T10:08:08.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:08.462+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:08.469527+0000) 2019-10-13T10:08:09.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:09.202+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:09.204281+0000) 2019-10-13T10:08:11.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:11.158+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.165691+0000) 2019-10-13T10:08:11.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:11.162+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.168479+0000) 2019-10-13T10:08:11.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:11.162+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.168519+0000) 2019-10-13T10:08:11.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:11.162+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.168790+0000) 2019-10-13T10:08:11.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:11.166+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.169451+0000) 2019-10-13T10:08:11.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:11.162+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.169673+0000) 2019-10-13T10:08:11.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:11.166+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.170513+0000) 2019-10-13T10:08:11.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:11.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:11.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:11.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.172060+0000) 2019-10-13T10:08:11.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:11.170+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.173428+0000) 2019-10-13T10:08:11.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:11.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:11.720884+0000) 2019-10-13T10:08:14.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:14.030+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.037300+0000) 2019-10-13T10:08:14.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:14.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.325006+0000) 2019-10-13T10:08:14.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:14.446+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:14.446888+0000) 2019-10-13T10:08:16.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:16.170+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.173841+0000) 2019-10-13T10:08:16.176 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:16.170+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.175206+0000) 2019-10-13T10:08:16.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:16.170+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.176659+0000) 2019-10-13T10:08:16.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:16.170+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.177389+0000) 2019-10-13T10:08:16.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:16.174+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.179286+0000) 2019-10-13T10:08:16.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:16.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:16.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:16.174+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.181009+0000) 2019-10-13T10:08:16.183 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:16.174+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.181214+0000) 2019-10-13T10:08:16.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:16.178+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:16.183172+0000) 2019-10-13T10:08:17.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:17.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:17.698965+0000) 2019-10-13T10:08:18.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:18.466+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:18.469767+0000) 2019-10-13T10:08:19.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:19.210+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:19.210674+0000) 2019-10-13T10:08:21.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:21.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.173343+0000) 2019-10-13T10:08:21.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:21.178+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.184348+0000) 2019-10-13T10:08:21.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:21.178+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.184851+0000) 2019-10-13T10:08:21.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:21.178+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.185297+0000) 2019-10-13T10:08:21.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:21.182+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.186130+0000) 2019-10-13T10:08:21.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:21.182+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.185904+0000) 2019-10-13T10:08:21.187 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:21.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.186384+0000) 2019-10-13T10:08:21.188 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:21.182+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.187118+0000) 2019-10-13T10:08:21.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:21.182+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:21.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:21.190+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.192139+0000) 2019-10-13T10:08:21.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:21.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:21.721062+0000) 2019-10-13T10:08:24.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:24.034+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.037538+0000) 2019-10-13T10:08:24.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:24.318+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.325210+0000) 2019-10-13T10:08:24.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:24.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:24.447094+0000) 2019-10-13T10:08:26.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:26.185+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.191263+0000) 2019-10-13T10:08:26.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:26.185+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.192432+0000) 2019-10-13T10:08:26.194 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:26.185+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.193400+0000) 2019-10-13T10:08:26.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:26.189+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.193972+0000) 2019-10-13T10:08:26.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:26.190+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.193916+0000) 2019-10-13T10:08:26.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:26.189+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.195229+0000) 2019-10-13T10:08:26.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:26.189+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.195280+0000) 2019-10-13T10:08:26.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:26.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:26.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:26.198+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:26.200489+0000) 2019-10-13T10:08:27.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:27.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:27.699193+0000) 2019-10-13T10:08:28.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:28.465+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:28.469967+0000) 2019-10-13T10:08:29.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:29.214+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:29.217173+0000) 2019-10-13T10:08:31.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:31.169+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.173606+0000) 2019-10-13T10:08:31.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:31.193+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.200290+0000) 2019-10-13T10:08:31.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:31.193+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.200977+0000) 2019-10-13T10:08:31.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:31.193+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.201280+0000) 2019-10-13T10:08:31.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:31.197+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.203145+0000) 2019-10-13T10:08:31.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:31.198+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.203772+0000) 2019-10-13T10:08:31.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:31.197+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.204760+0000) 2019-10-13T10:08:31.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:31.197+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.204896+0000) 2019-10-13T10:08:31.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:31.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:31.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:31.210+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.213919+0000) 2019-10-13T10:08:31.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:31.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:31.721189+0000) 2019-10-13T10:08:34.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:34.033+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.037773+0000) 2019-10-13T10:08:34.108 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:08:34.108 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:34.111 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:08:34.112 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:08:34.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:34.317+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.325422+0000) 2019-10-13T10:08:34.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:34.441+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:34.447314+0000) 2019-10-13T10:08:36.211 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:36.205+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.209572+0000) 2019-10-13T10:08:36.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:36.205+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.209946+0000) 2019-10-13T10:08:36.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:36.206+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.211289+0000) 2019-10-13T10:08:36.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:36.205+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.213092+0000) 2019-10-13T10:08:36.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:36.205+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.213499+0000) 2019-10-13T10:08:36.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:36.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:36.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:36.209+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.215892+0000) 2019-10-13T10:08:36.217 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:36.209+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.216012+0000) 2019-10-13T10:08:36.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:36.218+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:36.223309+0000) 2019-10-13T10:08:37.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:37.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:37.699395+0000) 2019-10-13T10:08:38.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:38.469+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:38.470183+0000) 2019-10-13T10:08:39.225 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:39.222+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:39.223583+0000) 2019-10-13T10:08:41.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:41.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.173805+0000) 2019-10-13T10:08:41.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:41.217+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.218304+0000) 2019-10-13T10:08:41.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:41.217+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.218948+0000) 2019-10-13T10:08:41.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:41.214+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.219320+0000) 2019-10-13T10:08:41.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:41.217+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.220960+0000) 2019-10-13T10:08:41.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:41.217+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.221386+0000) 2019-10-13T10:08:41.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:41.221+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.221779+0000) 2019-10-13T10:08:41.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:41.221+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.221730+0000) 2019-10-13T10:08:41.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:41.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:41.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:41.222+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.226137+0000) 2019-10-13T10:08:41.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:41.718+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:41.721394+0000) 2019-10-13T10:08:44.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:44.037+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.038010+0000) 2019-10-13T10:08:44.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:44.321+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.325657+0000) 2019-10-13T10:08:44.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:44.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:44.447556+0000) 2019-10-13T10:08:46.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:46.222+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.226699+0000) 2019-10-13T10:08:46.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:46.225+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.227642+0000) 2019-10-13T10:08:46.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:46.225+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.228196+0000) 2019-10-13T10:08:46.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:46.225+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.228946+0000) 2019-10-13T10:08:46.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:46.229+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.229970+0000) 2019-10-13T10:08:46.231 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:46.229+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.230853+0000) 2019-10-13T10:08:46.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:46.229+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.230885+0000) 2019-10-13T10:08:46.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:46.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110172 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:46.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:46.230+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:46.235949+0000) 2019-10-13T10:08:47.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:47.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:47.699608+0000) 2019-10-13T10:08:48.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:48.469+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:48.470395+0000) 2019-10-13T10:08:49.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:49.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:49.230164+0000) 2019-10-13T10:08:51.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:51.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.174016+0000) 2019-10-13T10:08:51.237 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:51.233+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.236291+0000) 2019-10-13T10:08:51.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:51.237+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.238001+0000) 2019-10-13T10:08:51.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:51.237+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.238352+0000) 2019-10-13T10:08:51.239 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:51.237+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.239014+0000) 2019-10-13T10:08:51.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:51.237+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.239414+0000) 2019-10-13T10:08:51.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:51.233+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.239494+0000) 2019-10-13T10:08:51.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:51.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110203 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:51.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:51.237+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.241301+0000) 2019-10-13T10:08:51.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:51.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.247564+0000) 2019-10-13T10:08:51.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:51.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:51.721563+0000) 2019-10-13T10:08:54.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:54.037+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.038227+0000) 2019-10-13T10:08:54.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:54.325+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.325852+0000) 2019-10-13T10:08:54.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:54.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:54.447744+0000) 2019-10-13T10:08:56.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:08:56.241+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.246186+0000) 2019-10-13T10:08:56.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:56.245+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.248260+0000) 2019-10-13T10:08:56.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:08:56.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110232 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:08:56.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:08:56.245+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.249563+0000) 2019-10-13T10:08:56.250 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:56.249+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.249900+0000) 2019-10-13T10:08:56.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:08:56.249+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.250215+0000) 2019-10-13T10:08:56.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:08:56.249+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.250649+0000) 2019-10-13T10:08:56.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:08:56.249+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.251292+0000) 2019-10-13T10:08:56.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:56.245+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:56.251753+0000) 2019-10-13T10:08:57.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:08:57.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:57.699806+0000) 2019-10-13T10:08:58.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:08:58.469+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:58.470614+0000) 2019-10-13T10:08:59.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:08:59.237+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:08:59.241572+0000) 2019-10-13T10:09:01.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:01.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.174234+0000) 2019-10-13T10:09:01.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:01.249+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.252806+0000) 2019-10-13T10:09:01.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:01.249+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.253409+0000) 2019-10-13T10:09:01.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:01.253+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.253810+0000) 2019-10-13T10:09:01.255 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:01.249+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.253924+0000) 2019-10-13T10:09:01.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:01.253+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.254952+0000) 2019-10-13T10:09:01.256 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:01.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.255340+0000) 2019-10-13T10:09:01.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:01.253+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.255937+0000) 2019-10-13T10:09:01.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:01.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:01.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:01.253+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.258492+0000) 2019-10-13T10:09:01.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:01.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:01.721740+0000) 2019-10-13T10:09:04.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:04.037+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.038422+0000) 2019-10-13T10:09:04.222 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:09:04.222 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:04.226 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:09:04.226 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:04.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:04.325+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.326052+0000) 2019-10-13T10:09:04.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:04.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:04.447939+0000) 2019-10-13T10:09:06.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:06.257+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.260842+0000) 2019-10-13T10:09:06.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:06.261+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.262019+0000) 2019-10-13T10:09:06.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:06.261+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.263758+0000) 2019-10-13T10:09:06.265 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:06.261+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.264684+0000) 2019-10-13T10:09:06.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:06.265+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.266576+0000) 2019-10-13T10:09:06.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:06.261+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.266538+0000) 2019-10-13T10:09:06.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:06.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:06.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:06.261+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.267258+0000) 2019-10-13T10:09:06.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:06.265+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:06.268471+0000) 2019-10-13T10:09:07.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:07.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:07.700011+0000) 2019-10-13T10:09:08.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:08.469+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:08.470807+0000) 2019-10-13T10:09:09.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:09.245+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:09.248243+0000) 2019-10-13T10:09:11.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:11.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.174433+0000) 2019-10-13T10:09:11.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:11.269+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.271258+0000) 2019-10-13T10:09:11.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:11.265+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.271808+0000) 2019-10-13T10:09:11.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:11.269+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.272589+0000) 2019-10-13T10:09:11.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:11.269+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.272851+0000) 2019-10-13T10:09:11.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:11.269+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.273256+0000) 2019-10-13T10:09:11.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:11.269+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.273631+0000) 2019-10-13T10:09:11.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:11.273+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.274329+0000) 2019-10-13T10:09:11.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:11.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:11.280 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:11.273+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.278876+0000) 2019-10-13T10:09:11.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:11.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:11.721909+0000) 2019-10-13T10:09:14.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:14.037+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.038629+0000) 2019-10-13T10:09:14.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:14.325+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.326303+0000) 2019-10-13T10:09:14.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:14.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:14.448126+0000) 2019-10-13T10:09:16.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:16.277+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.279417+0000) 2019-10-13T10:09:16.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:16.277+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.280043+0000) 2019-10-13T10:09:16.281 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:16.277+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.280269+0000) 2019-10-13T10:09:16.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:16.277+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.280799+0000) 2019-10-13T10:09:16.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:16.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.281050+0000) 2019-10-13T10:09:16.282 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:16.281+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.281803+0000) 2019-10-13T10:09:16.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:16.281+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.282164+0000) 2019-10-13T10:09:16.285 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:16.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:16.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:16.281+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:16.286345+0000) 2019-10-13T10:09:17.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:17.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:17.700211+0000) 2019-10-13T10:09:18.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:18.469+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:18.471063+0000) 2019-10-13T10:09:19.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:19.249+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:19.254580+0000) 2019-10-13T10:09:21.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:21.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.174650+0000) 2019-10-13T10:09:21.291 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:21.288+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.290376+0000) 2019-10-13T10:09:21.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:21.288+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.290726+0000) 2019-10-13T10:09:21.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:21.289+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.292304+0000) 2019-10-13T10:09:21.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:21.288+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.292828+0000) 2019-10-13T10:09:21.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:21.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:21.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:21.292+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.293932+0000) 2019-10-13T10:09:21.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:21.292+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.293860+0000) 2019-10-13T10:09:21.295 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:21.292+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.294239+0000) 2019-10-13T10:09:21.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:21.293+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.299495+0000) 2019-10-13T10:09:21.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:21.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:21.722107+0000) 2019-10-13T10:09:24.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:24.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.038861+0000) 2019-10-13T10:09:24.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:24.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.326576+0000) 2019-10-13T10:09:24.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:24.444+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:24.448381+0000) 2019-10-13T10:09:26.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:26.289+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.295885+0000) 2019-10-13T10:09:26.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:26.292+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.296518+0000) 2019-10-13T10:09:26.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:26.296+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.298040+0000) 2019-10-13T10:09:26.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:26.296+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.298213+0000) 2019-10-13T10:09:26.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:26.296+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.298614+0000) 2019-10-13T10:09:26.300 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:26.296+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.299360+0000) 2019-10-13T10:09:26.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:26.296+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.299453+0000) 2019-10-13T10:09:26.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:26.300+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:26.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:26.305+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:26.309124+0000) 2019-10-13T10:09:27.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:27.696+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:27.700395+0000) 2019-10-13T10:09:28.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:28.468+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:28.471290+0000) 2019-10-13T10:09:29.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:29.257+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:29.261124+0000) 2019-10-13T10:09:31.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:31.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.174850+0000) 2019-10-13T10:09:31.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:31.304+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.305815+0000) 2019-10-13T10:09:31.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:31.304+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.306373+0000) 2019-10-13T10:09:31.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:31.304+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.306983+0000) 2019-10-13T10:09:31.310 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:31.305+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.309005+0000) 2019-10-13T10:09:31.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:31.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:31.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:31.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.311307+0000) 2019-10-13T10:09:31.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:31.308+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.311524+0000) 2019-10-13T10:09:31.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:31.308+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.312019+0000) 2019-10-13T10:09:31.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:31.309+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.314558+0000) 2019-10-13T10:09:31.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:31.721+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:31.722273+0000) 2019-10-13T10:09:34.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:34.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.039105+0000) 2019-10-13T10:09:34.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:34.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.326795+0000) 2019-10-13T10:09:34.336 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:09:34.336 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:34.340 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:09:34.340 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:09:34.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:34.444+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:34.448601+0000) 2019-10-13T10:09:36.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:36.312+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.315262+0000) 2019-10-13T10:09:36.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:36.312+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.316524+0000) 2019-10-13T10:09:36.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:36.312+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.317374+0000) 2019-10-13T10:09:36.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:36.316+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.318097+0000) 2019-10-13T10:09:36.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:36.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:36.321 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:36.313+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.319474+0000) 2019-10-13T10:09:36.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:36.316+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.320075+0000) 2019-10-13T10:09:36.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:36.316+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.320454+0000) 2019-10-13T10:09:36.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:36.317+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:36.322238+0000) 2019-10-13T10:09:37.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:37.696+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:37.700628+0000) 2019-10-13T10:09:38.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:38.468+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:38.471482+0000) 2019-10-13T10:09:39.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:39.265+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:39.267782+0000) 2019-10-13T10:09:41.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:41.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.175051+0000) 2019-10-13T10:09:41.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:41.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.323403+0000) 2019-10-13T10:09:41.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:41.320+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.323972+0000) 2019-10-13T10:09:41.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:41.320+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.324785+0000) 2019-10-13T10:09:41.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:41.320+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.325319+0000) 2019-10-13T10:09:41.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:41.324+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.326915+0000) 2019-10-13T10:09:41.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:41.324+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.327752+0000) 2019-10-13T10:09:41.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:41.324+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:41.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:41.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.328812+0000) 2019-10-13T10:09:41.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:41.329+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.333396+0000) 2019-10-13T10:09:41.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:41.721+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:41.722457+0000) 2019-10-13T10:09:44.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:44.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.039332+0000) 2019-10-13T10:09:44.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:44.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.326990+0000) 2019-10-13T10:09:44.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:44.448+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:44.448801+0000) 2019-10-13T10:09:46.334 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:46.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.332612+0000) 2019-10-13T10:09:46.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:46.328+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.333307+0000) 2019-10-13T10:09:46.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:46.328+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.333279+0000) 2019-10-13T10:09:46.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:46.332+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.334087+0000) 2019-10-13T10:09:46.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:46.332+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.334575+0000) 2019-10-13T10:09:46.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:46.332+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.334849+0000) 2019-10-13T10:09:46.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:46.332+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.335272+0000) 2019-10-13T10:09:46.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:46.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:46.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:46.337+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:46.340632+0000) 2019-10-13T10:09:47.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:47.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:47.700841+0000) 2019-10-13T10:09:48.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:48.468+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:48.471672+0000) 2019-10-13T10:09:49.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:49.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:49.277476+0000) 2019-10-13T10:09:51.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:51.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.175212+0000) 2019-10-13T10:09:51.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:51.336+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.340774+0000) 2019-10-13T10:09:51.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:51.337+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.340994+0000) 2019-10-13T10:09:51.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:51.340+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.341906+0000) 2019-10-13T10:09:51.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:51.340+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.342249+0000) 2019-10-13T10:09:51.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:51.340+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.342494+0000) 2019-10-13T10:09:51.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:51.340+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.342850+0000) 2019-10-13T10:09:51.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:51.340+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.343531+0000) 2019-10-13T10:09:51.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:51.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:51.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:51.345+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.348797+0000) 2019-10-13T10:09:51.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:51.717+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:51.722627+0000) 2019-10-13T10:09:54.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:54.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.039562+0000) 2019-10-13T10:09:54.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:54.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.327199+0000) 2019-10-13T10:09:54.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:54.448+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:54.449010+0000) 2019-10-13T10:09:56.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:56.348+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.350639+0000) 2019-10-13T10:09:56.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:09:56.348+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.351094+0000) 2019-10-13T10:09:56.352 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:56.348+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.351526+0000) 2019-10-13T10:09:56.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:09:56.348+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.352588+0000) 2019-10-13T10:09:56.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:09:56.348+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.352771+0000) 2019-10-13T10:09:56.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:09:56.349+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.353142+0000) 2019-10-13T10:09:56.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:09:56.352+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.353950+0000) 2019-10-13T10:09:56.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:09:56.352+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:09:56.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:56.353+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:56.358617+0000) 2019-10-13T10:09:57.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:09:57.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:57.701040+0000) 2019-10-13T10:09:58.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:09:58.468+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:58.471862+0000) 2019-10-13T10:09:59.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:09:59.280+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:09:59.283944+0000) 2019-10-13T10:10:01.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:01.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.175394+0000) 2019-10-13T10:10:01.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:01.356+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.359164+0000) 2019-10-13T10:10:01.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:01.356+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.360852+0000) 2019-10-13T10:10:01.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:01.356+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.362686+0000) 2019-10-13T10:10:01.364 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:01.360+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.363515+0000) 2019-10-13T10:10:01.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:01.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:01.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:01.360+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.364107+0000) 2019-10-13T10:10:01.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:01.360+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.364610+0000) 2019-10-13T10:10:01.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:01.360+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.365042+0000) 2019-10-13T10:10:01.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:01.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.364466+0000) 2019-10-13T10:10:01.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:01.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:01.722834+0000) 2019-10-13T10:10:04.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:04.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.039777+0000) 2019-10-13T10:10:04.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:04.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.327399+0000) 2019-10-13T10:10:04.441 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:10:04.441 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:04.445 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:10:04.445 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:04.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:04.444+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:04.449204+0000) 2019-10-13T10:10:06.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:06.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.367097+0000) 2019-10-13T10:10:06.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:06.368+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.369945+0000) 2019-10-13T10:10:06.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:06.368+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.370107+0000) 2019-10-13T10:10:06.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:06.368+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.370575+0000) 2019-10-13T10:10:06.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:06.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:06.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:06.368+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.372415+0000) 2019-10-13T10:10:06.374 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:06.368+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.372604+0000) 2019-10-13T10:10:06.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:06.372+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.373729+0000) 2019-10-13T10:10:06.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:06.368+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:06.373710+0000) 2019-10-13T10:10:07.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:07.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:07.701264+0000) 2019-10-13T10:10:08.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:08.468+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:08.472098+0000) 2019-10-13T10:10:09.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:09.288+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:09.290634+0000) 2019-10-13T10:10:11.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:11.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.175591+0000) 2019-10-13T10:10:11.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:11.372+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.377080+0000) 2019-10-13T10:10:11.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:11.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.377826+0000) 2019-10-13T10:10:11.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:11.376+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.378271+0000) 2019-10-13T10:10:11.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:11.376+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.380539+0000) 2019-10-13T10:10:11.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:11.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:11.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:11.376+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.381605+0000) 2019-10-13T10:10:11.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:11.380+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.382012+0000) 2019-10-13T10:10:11.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:11.376+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.382710+0000) 2019-10-13T10:10:11.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:11.380+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.384735+0000) 2019-10-13T10:10:11.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:11.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:11.723020+0000) 2019-10-13T10:10:14.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:14.036+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.039927+0000) 2019-10-13T10:10:14.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:14.324+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.327635+0000) 2019-10-13T10:10:14.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:14.444+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:14.449420+0000) 2019-10-13T10:10:16.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:16.380+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.384189+0000) 2019-10-13T10:10:16.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:16.380+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.385445+0000) 2019-10-13T10:10:16.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:16.384+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.386087+0000) 2019-10-13T10:10:16.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:16.384+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.386389+0000) 2019-10-13T10:10:16.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:16.384+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.386380+0000) 2019-10-13T10:10:16.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:16.384+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.386901+0000) 2019-10-13T10:10:16.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:16.380+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.387948+0000) 2019-10-13T10:10:16.397 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:16.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:16.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:16.384+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:16.391030+0000) 2019-10-13T10:10:17.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:17.695+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:17.701497+0000) 2019-10-13T10:10:18.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:18.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:18.472289+0000) 2019-10-13T10:10:19.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:19.292+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:19.296601+0000) 2019-10-13T10:10:21.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:21.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.175789+0000) 2019-10-13T10:10:21.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:21.387+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.393390+0000) 2019-10-13T10:10:21.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:21.391+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.394162+0000) 2019-10-13T10:10:21.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:21.391+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.395697+0000) 2019-10-13T10:10:21.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:21.391+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.395876+0000) 2019-10-13T10:10:21.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:21.391+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.397588+0000) 2019-10-13T10:10:21.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:21.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:21.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:21.395+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.398330+0000) 2019-10-13T10:10:21.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:21.392+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.398338+0000) 2019-10-13T10:10:21.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:21.400+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.404351+0000) 2019-10-13T10:10:21.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:21.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:21.723229+0000) 2019-10-13T10:10:24.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:24.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.040086+0000) 2019-10-13T10:10:24.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:24.323+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.327837+0000) 2019-10-13T10:10:24.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:24.443+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:24.449622+0000) 2019-10-13T10:10:26.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:26.399+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.402022+0000) 2019-10-13T10:10:26.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:26.399+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.402460+0000) 2019-10-13T10:10:26.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:26.399+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.403224+0000) 2019-10-13T10:10:26.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:26.396+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.403244+0000) 2019-10-13T10:10:26.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:26.399+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.404108+0000) 2019-10-13T10:10:26.405 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:26.399+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.404622+0000) 2019-10-13T10:10:26.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:26.399+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.405282+0000) 2019-10-13T10:10:26.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:26.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:26.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:26.404+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:26.411791+0000) 2019-10-13T10:10:27.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:27.699+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:27.701702+0000) 2019-10-13T10:10:28.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:28.471+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:28.472434+0000) 2019-10-13T10:10:29.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:29.300+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:29.303114+0000) 2019-10-13T10:10:31.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:31.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.176003+0000) 2019-10-13T10:10:31.411 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:31.407+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.410004+0000) 2019-10-13T10:10:31.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:31.407+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.410491+0000) 2019-10-13T10:10:31.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:31.407+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.411107+0000) 2019-10-13T10:10:31.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:31.407+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.411348+0000) 2019-10-13T10:10:31.412 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:31.407+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.411784+0000) 2019-10-13T10:10:31.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:31.404+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.411909+0000) 2019-10-13T10:10:31.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:31.407+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.412686+0000) 2019-10-13T10:10:31.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:31.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:31.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:31.416+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.421744+0000) 2019-10-13T10:10:31.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:31.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:31.723429+0000) 2019-10-13T10:10:34.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:34.039+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.040307+0000) 2019-10-13T10:10:34.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:34.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.328034+0000) 2019-10-13T10:10:34.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:34.447+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:34.449824+0000) 2019-10-13T10:10:34.564 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:10:34.564 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:34.568 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:10:34.568 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:10:36.419 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:36.415+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.417991+0000) 2019-10-13T10:10:36.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:36.415+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.419359+0000) 2019-10-13T10:10:36.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:36.415+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.420273+0000) 2019-10-13T10:10:36.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:36.415+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.420358+0000) 2019-10-13T10:10:36.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:36.415+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.420730+0000) 2019-10-13T10:10:36.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:36.416+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.421059+0000) 2019-10-13T10:10:36.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:36.419+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.421923+0000) 2019-10-13T10:10:36.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:36.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:36.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:36.420+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:36.425122+0000) 2019-10-13T10:10:37.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:37.699+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:37.701940+0000) 2019-10-13T10:10:38.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:38.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:38.472627+0000) 2019-10-13T10:10:39.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:39.308+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:39.310339+0000) 2019-10-13T10:10:41.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:41.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.176197+0000) 2019-10-13T10:10:41.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:41.423+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.425768+0000) 2019-10-13T10:10:41.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:41.423+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.428226+0000) 2019-10-13T10:10:41.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:41.423+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.428565+0000) 2019-10-13T10:10:41.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:41.423+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.428874+0000) 2019-10-13T10:10:41.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:41.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:41.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:41.424+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.431863+0000) 2019-10-13T10:10:41.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:41.427+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.432463+0000) 2019-10-13T10:10:41.434 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:41.427+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.432992+0000) 2019-10-13T10:10:41.438 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:41.432+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.437371+0000) 2019-10-13T10:10:41.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:41.716+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:41.723598+0000) 2019-10-13T10:10:44.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:44.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.040519+0000) 2019-10-13T10:10:44.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:44.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.328234+0000) 2019-10-13T10:10:44.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:44.447+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:44.450042+0000) 2019-10-13T10:10:46.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:46.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.434701+0000) 2019-10-13T10:10:46.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:46.431+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.435388+0000) 2019-10-13T10:10:46.441 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:46.435+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.440195+0000) 2019-10-13T10:10:46.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:46.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:46.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:46.435+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.441254+0000) 2019-10-13T10:10:46.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:46.439+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.441825+0000) 2019-10-13T10:10:46.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:46.436+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.441940+0000) 2019-10-13T10:10:46.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:46.439+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.442699+0000) 2019-10-13T10:10:46.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:46.448+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:46.452723+0000) 2019-10-13T10:10:47.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:47.699+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:47.702147+0000) 2019-10-13T10:10:48.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:48.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:48.472841+0000) 2019-10-13T10:10:49.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:49.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:49.317416+0000) 2019-10-13T10:10:51.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:51.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.176420+0000) 2019-10-13T10:10:51.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:51.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.443917+0000) 2019-10-13T10:10:51.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:51.439+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.444343+0000) 2019-10-13T10:10:51.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:51.439+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.445220+0000) 2019-10-13T10:10:51.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:51.443+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.445786+0000) 2019-10-13T10:10:51.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:51.443+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.446033+0000) 2019-10-13T10:10:51.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:51.440+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.446434+0000) 2019-10-13T10:10:51.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:51.443+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.446437+0000) 2019-10-13T10:10:51.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:51.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:51.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:51.448+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.453072+0000) 2019-10-13T10:10:51.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:51.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:51.723770+0000) 2019-10-13T10:10:54.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:54.039+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.040716+0000) 2019-10-13T10:10:54.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:54.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.328429+0000) 2019-10-13T10:10:54.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:54.447+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:54.450271+0000) 2019-10-13T10:10:56.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:10:56.448+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.452142+0000) 2019-10-13T10:10:56.455 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:10:56.451+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.454344+0000) 2019-10-13T10:10:56.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:56.451+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.454981+0000) 2019-10-13T10:10:56.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:10:56.451+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.455712+0000) 2019-10-13T10:10:56.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:56.451+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.456052+0000) 2019-10-13T10:10:56.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:10:56.451+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.456549+0000) 2019-10-13T10:10:56.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:10:56.451+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.457197+0000) 2019-10-13T10:10:56.459 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:10:56.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:10:56.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:56.460+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:56.461238+0000) 2019-10-13T10:10:57.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:10:57.699+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:57.702347+0000) 2019-10-13T10:10:58.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:10:58.471+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:58.473067+0000) 2019-10-13T10:10:59.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:10:59.324+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:10:59.324624+0000) 2019-10-13T10:11:01.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:01.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.176634+0000) 2019-10-13T10:11:01.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:01.455+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.460763+0000) 2019-10-13T10:11:01.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:01.455+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.461574+0000) 2019-10-13T10:11:01.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:01.459+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.463553+0000) 2019-10-13T10:11:01.465 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:01.459+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.464179+0000) 2019-10-13T10:11:01.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:01.464+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.464389+0000) 2019-10-13T10:11:01.466 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:01.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.466043+0000) 2019-10-13T10:11:01.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:01.463+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.466198+0000) 2019-10-13T10:11:01.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:01.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:01.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:01.472+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.472835+0000) 2019-10-13T10:11:01.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:01.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:01.723943+0000) 2019-10-13T10:11:04.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:04.035+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.040918+0000) 2019-10-13T10:11:04.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:04.323+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.328637+0000) 2019-10-13T10:11:04.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:04.447+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:04.450493+0000) 2019-10-13T10:11:04.694 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:11:04.694 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:04.698 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:11:04.698 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:06.471 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:06.467+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.470653+0000) 2019-10-13T10:11:06.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:06.467+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.471912+0000) 2019-10-13T10:11:06.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:06.471+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.474510+0000) 2019-10-13T10:11:06.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:06.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.474928+0000) 2019-10-13T10:11:06.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:06.471+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.475223+0000) 2019-10-13T10:11:06.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:06.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:06.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:06.471+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.475878+0000) 2019-10-13T10:11:06.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:06.471+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.476769+0000) 2019-10-13T10:11:06.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:06.479+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:06.480319+0000) 2019-10-13T10:11:07.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:07.699+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:07.702522+0000) 2019-10-13T10:11:08.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:08.467+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:08.473259+0000) 2019-10-13T10:11:09.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:09.327+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:09.331753+0000) 2019-10-13T10:11:11.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:11.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.176838+0000) 2019-10-13T10:11:11.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:11.475+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.479338+0000) 2019-10-13T10:11:11.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:11.475+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.481152+0000) 2019-10-13T10:11:11.483 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:11.479+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.481879+0000) 2019-10-13T10:11:11.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:11.479+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.483881+0000) 2019-10-13T10:11:11.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:11.479+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.484241+0000) 2019-10-13T10:11:11.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:11.479+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.485012+0000) 2019-10-13T10:11:11.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:11.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:11.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:11.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.485924+0000) 2019-10-13T10:11:11.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:11.487+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.491280+0000) 2019-10-13T10:11:11.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:11.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:11.724132+0000) 2019-10-13T10:11:14.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:14.034+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.041186+0000) 2019-10-13T10:11:14.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:14.326+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.328868+0000) 2019-10-13T10:11:14.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:14.446+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:14.450725+0000) 2019-10-13T10:11:16.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:16.482+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.487976+0000) 2019-10-13T10:11:16.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:16.482+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.489424+0000) 2019-10-13T10:11:16.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:16.486+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.491165+0000) 2019-10-13T10:11:16.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:16.486+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.492264+0000) 2019-10-13T10:11:16.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:16.491+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.492623+0000) 2019-10-13T10:11:16.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:16.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:16.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:16.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.494204+0000) 2019-10-13T10:11:16.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:16.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.494542+0000) 2019-10-13T10:11:16.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:16.491+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:16.495893+0000) 2019-10-13T10:11:17.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:17.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:17.702732+0000) 2019-10-13T10:11:18.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:18.470+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:18.473420+0000) 2019-10-13T10:11:19.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:19.339+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:19.343146+0000) 2019-10-13T10:11:21.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:21.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.177033+0000) 2019-10-13T10:11:21.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:21.490+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.497185+0000) 2019-10-13T10:11:21.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:21.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.497711+0000) 2019-10-13T10:11:21.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:21.495+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.497970+0000) 2019-10-13T10:11:21.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:21.494+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.499164+0000) 2019-10-13T10:11:21.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:21.494+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.499661+0000) 2019-10-13T10:11:21.501 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:21.494+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.500361+0000) 2019-10-13T10:11:21.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:21.494+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.500488+0000) 2019-10-13T10:11:21.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:21.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:21.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:21.503+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.504987+0000) 2019-10-13T10:11:21.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:21.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:21.724325+0000) 2019-10-13T10:11:24.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:24.034+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.041429+0000) 2019-10-13T10:11:24.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:24.322+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.329073+0000) 2019-10-13T10:11:24.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:24.450+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:24.450924+0000) 2019-10-13T10:11:26.508 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:26.502+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.507189+0000) 2019-10-13T10:11:26.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:26.502+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.507541+0000) 2019-10-13T10:11:26.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:26.503+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.507575+0000) 2019-10-13T10:11:26.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:26.506+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.510206+0000) 2019-10-13T10:11:26.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:26.506+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.510237+0000) 2019-10-13T10:11:26.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:26.506+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.511043+0000) 2019-10-13T10:11:26.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:26.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:26.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:26.506+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.511807+0000) 2019-10-13T10:11:26.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:26.515+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:26.516662+0000) 2019-10-13T10:11:27.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:27.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:27.702925+0000) 2019-10-13T10:11:28.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:28.470+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:28.473632+0000) 2019-10-13T10:11:29.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:29.347+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:29.349706+0000) 2019-10-13T10:11:31.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:31.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.177232+0000) 2019-10-13T10:11:31.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:31.510+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.515612+0000) 2019-10-13T10:11:31.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:31.510+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.516788+0000) 2019-10-13T10:11:31.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:31.510+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.517395+0000) 2019-10-13T10:11:31.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:31.510+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.517483+0000) 2019-10-13T10:11:31.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:31.515+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.517554+0000) 2019-10-13T10:11:31.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:31.514+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.518248+0000) 2019-10-13T10:11:31.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:31.514+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.519085+0000) 2019-10-13T10:11:31.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:31.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:31.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:31.523+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.525481+0000) 2019-10-13T10:11:31.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:31.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:31.724464+0000) 2019-10-13T10:11:34.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:34.034+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.041671+0000) 2019-10-13T10:11:34.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:34.326+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.329299+0000) 2019-10-13T10:11:34.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:34.450+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:34.451119+0000) 2019-10-13T10:11:34.816 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:11:34.816 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:34.847 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:11:34.848 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:11:36.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:36.518+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.523994+0000) 2019-10-13T10:11:36.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:36.518+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.524472+0000) 2019-10-13T10:11:36.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:36.523+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.524587+0000) 2019-10-13T10:11:36.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:36.518+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.525342+0000) 2019-10-13T10:11:36.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:36.522+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.525954+0000) 2019-10-13T10:11:36.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:36.522+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.526816+0000) 2019-10-13T10:11:36.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:36.522+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.527588+0000) 2019-10-13T10:11:36.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:36.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:36.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:36.527+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:36.529530+0000) 2019-10-13T10:11:37.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:37.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:37.703089+0000) 2019-10-13T10:11:38.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:38.470+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:38.473864+0000) 2019-10-13T10:11:39.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:39.355+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:39.356253+0000) 2019-10-13T10:11:41.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:41.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.177438+0000) 2019-10-13T10:11:41.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:41.527+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.531960+0000) 2019-10-13T10:11:41.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:41.526+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.533304+0000) 2019-10-13T10:11:41.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:41.530+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.533726+0000) 2019-10-13T10:11:41.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:41.530+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.534393+0000) 2019-10-13T10:11:41.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:41.530+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.534513+0000) 2019-10-13T10:11:41.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:41.530+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.536947+0000) 2019-10-13T10:11:41.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:41.530+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.537093+0000) 2019-10-13T10:11:41.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:41.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:41.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:41.543+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.544648+0000) 2019-10-13T10:11:41.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:41.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:41.724638+0000) 2019-10-13T10:11:44.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:44.038+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.041927+0000) 2019-10-13T10:11:44.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:44.326+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.329530+0000) 2019-10-13T10:11:44.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:44.446+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:44.451315+0000) 2019-10-13T10:11:46.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:46.534+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.541183+0000) 2019-10-13T10:11:46.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:46.538+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.542423+0000) 2019-10-13T10:11:46.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:46.538+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.543130+0000) 2019-10-13T10:11:46.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:46.539+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.543150+0000) 2019-10-13T10:11:46.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:46.538+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.543818+0000) 2019-10-13T10:11:46.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:46.538+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.544091+0000) 2019-10-13T10:11:46.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:46.538+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.544580+0000) 2019-10-13T10:11:46.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:46.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:46.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:46.547+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:46.551283+0000) 2019-10-13T10:11:47.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:47.702+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:47.703242+0000) 2019-10-13T10:11:48.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:48.470+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:48.474063+0000) 2019-10-13T10:11:49.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:49.367+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:49.367700+0000) 2019-10-13T10:11:51.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:51.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.177637+0000) 2019-10-13T10:11:51.550 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:51.547+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.549219+0000) 2019-10-13T10:11:51.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:51.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.550035+0000) 2019-10-13T10:11:51.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:51.546+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.550439+0000) 2019-10-13T10:11:51.598 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:51.546+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.552473+0000) 2019-10-13T10:11:51.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:51.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.552585+0000) 2019-10-13T10:11:51.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:51.550+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.554782+0000) 2019-10-13T10:11:51.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:51.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:51.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:51.550+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.555931+0000) 2019-10-13T10:11:51.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:51.559+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.560475+0000) 2019-10-13T10:11:51.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:51.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:51.724813+0000) 2019-10-13T10:11:54.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:54.038+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.042180+0000) 2019-10-13T10:11:54.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:54.326+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.329729+0000) 2019-10-13T10:11:54.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:54.450+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:54.451546+0000) 2019-10-13T10:11:56.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:11:56.554+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.558278+0000) 2019-10-13T10:11:56.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:11:56.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.559373+0000) 2019-10-13T10:11:56.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:11:56.555+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.559340+0000) 2019-10-13T10:11:56.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:56.554+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.561415+0000) 2019-10-13T10:11:56.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:11:56.554+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.561718+0000) 2019-10-13T10:11:56.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:11:56.558+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.562188+0000) 2019-10-13T10:11:56.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:56.558+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.562759+0000) 2019-10-13T10:11:56.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:11:56.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:11:56.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:56.567+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:56.571240+0000) 2019-10-13T10:11:57.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:11:57.702+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:57.703392+0000) 2019-10-13T10:11:58.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:11:58.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:58.474253+0000) 2019-10-13T10:11:59.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:11:59.371+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:11:59.374167+0000) 2019-10-13T10:12:01.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:01.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.177839+0000) 2019-10-13T10:12:01.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:01.562+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.568499+0000) 2019-10-13T10:12:01.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:01.562+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.568879+0000) 2019-10-13T10:12:01.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:01.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.570941+0000) 2019-10-13T10:12:01.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:01.566+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.572819+0000) 2019-10-13T10:12:01.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:01.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:01.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:01.571+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.573276+0000) 2019-10-13T10:12:01.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:01.570+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.573925+0000) 2019-10-13T10:12:01.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:01.570+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.574432+0000) 2019-10-13T10:12:01.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:01.575+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.579093+0000) 2019-10-13T10:12:01.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:01.723+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:01.725055+0000) 2019-10-13T10:12:04.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:04.038+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.042412+0000) 2019-10-13T10:12:04.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:04.326+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.329994+0000) 2019-10-13T10:12:04.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:04.446+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:04.451763+0000) 2019-10-13T10:12:04.935 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:12:04.935 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:04.939 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:12:04.939 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:06.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:06.575+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.576679+0000) 2019-10-13T10:12:06.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:06.570+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.576471+0000) 2019-10-13T10:12:06.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:06.570+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.576757+0000) 2019-10-13T10:12:06.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:06.574+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.578283+0000) 2019-10-13T10:12:06.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:06.574+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.578759+0000) 2019-10-13T10:12:06.582 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:06.574+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.579233+0000) 2019-10-13T10:12:06.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:06.574+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.579759+0000) 2019-10-13T10:12:06.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:06.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:06.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:06.579+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:06.583634+0000) 2019-10-13T10:12:07.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:07.698+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:07.703587+0000) 2019-10-13T10:12:08.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:08.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:08.474452+0000) 2019-10-13T10:12:09.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:09.382+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:09.385393+0000) 2019-10-13T10:12:11.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:11.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.178070+0000) 2019-10-13T10:12:11.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:11.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.584432+0000) 2019-10-13T10:12:11.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:11.577+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.584561+0000) 2019-10-13T10:12:11.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:11.577+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.584988+0000) 2019-10-13T10:12:11.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:11.577+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.585377+0000) 2019-10-13T10:12:11.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:11.581+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.586663+0000) 2019-10-13T10:12:11.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:11.582+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.586980+0000) 2019-10-13T10:12:11.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:11.581+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.588183+0000) 2019-10-13T10:12:11.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:11.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:11.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:11.590+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.592449+0000) 2019-10-13T10:12:11.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:11.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:11.725227+0000) 2019-10-13T10:12:14.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:14.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.042617+0000) 2019-10-13T10:12:14.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:14.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.330196+0000) 2019-10-13T10:12:14.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:14.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:14.451960+0000) 2019-10-13T10:12:16.597 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:16.589+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.593089+0000) 2019-10-13T10:12:16.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:16.590+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.593326+0000) 2019-10-13T10:12:16.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:16.589+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.594288+0000) 2019-10-13T10:12:16.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:16.589+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.594829+0000) 2019-10-13T10:12:16.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:16.589+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.596980+0000) 2019-10-13T10:12:16.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:16.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:16.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:16.593+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.598371+0000) 2019-10-13T10:12:16.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:16.593+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.599113+0000) 2019-10-13T10:12:16.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:16.598+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:16.602888+0000) 2019-10-13T10:12:17.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:17.697+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:17.703792+0000) 2019-10-13T10:12:18.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:18.473+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:18.474668+0000) 2019-10-13T10:12:19.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:19.386+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:19.391932+0000) 2019-10-13T10:12:21.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:21.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.178269+0000) 2019-10-13T10:12:21.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:21.593+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.601164+0000) 2019-10-13T10:12:21.603 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:21.597+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.602166+0000) 2019-10-13T10:12:21.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:21.598+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.603335+0000) 2019-10-13T10:12:21.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:21.597+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.604098+0000) 2019-10-13T10:12:21.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:21.597+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.605595+0000) 2019-10-13T10:12:21.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:21.601+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.606173+0000) 2019-10-13T10:12:21.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:21.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111497 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:21.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:21.601+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.607018+0000) 2019-10-13T10:12:21.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:21.610+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.615800+0000) 2019-10-13T10:12:21.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:21.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:21.725397+0000) 2019-10-13T10:12:24.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:24.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.042840+0000) 2019-10-13T10:12:24.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:24.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.330392+0000) 2019-10-13T10:12:24.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:24.445+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:24.452122+0000) 2019-10-13T10:12:26.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:26.605+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.609300+0000) 2019-10-13T10:12:26.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:26.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.611545+0000) 2019-10-13T10:12:26.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:26.609+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.611797+0000) 2019-10-13T10:12:26.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:26.606+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.612016+0000) 2019-10-13T10:12:26.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:26.609+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.613061+0000) 2019-10-13T10:12:26.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:26.609+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.613399+0000) 2019-10-13T10:12:26.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:26.613+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.614216+0000) 2019-10-13T10:12:26.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:26.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:26.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:26.618+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:26.620250+0000) 2019-10-13T10:12:27.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:27.701+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:27.704028+0000) 2019-10-13T10:12:28.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:28.473+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:28.474904+0000) 2019-10-13T10:12:29.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:29.394+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:29.398416+0000) 2019-10-13T10:12:31.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:31.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.178489+0000) 2019-10-13T10:12:31.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:31.617+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.618800+0000) 2019-10-13T10:12:31.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:31.617+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.619075+0000) 2019-10-13T10:12:31.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:31.614+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.619511+0000) 2019-10-13T10:12:31.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:31.617+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.620211+0000) 2019-10-13T10:12:31.622 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:31.617+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.621339+0000) 2019-10-13T10:12:31.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:31.617+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.621566+0000) 2019-10-13T10:12:31.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:31.621+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.621984+0000) 2019-10-13T10:12:31.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:31.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:31.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:31.622+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.626824+0000) 2019-10-13T10:12:31.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:31.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:31.725572+0000) 2019-10-13T10:12:34.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:34.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.043069+0000) 2019-10-13T10:12:34.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:34.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.330595+0000) 2019-10-13T10:12:34.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:34.449+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:34.452353+0000) 2019-10-13T10:12:35.089 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:12:35.089 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:35.093 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:12:35.093 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:12:36.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:36.625+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.627232+0000) 2019-10-13T10:12:36.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:36.625+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.629162+0000) 2019-10-13T10:12:36.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:36.625+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.629448+0000) 2019-10-13T10:12:36.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:36.629+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.629948+0000) 2019-10-13T10:12:36.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:36.626+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.631043+0000) 2019-10-13T10:12:36.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:36.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.631759+0000) 2019-10-13T10:12:36.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:36.629+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.631960+0000) 2019-10-13T10:12:36.635 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:36.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:36.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:36.634+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:36.636348+0000) 2019-10-13T10:12:37.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:37.701+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:37.704226+0000) 2019-10-13T10:12:38.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:38.473+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:38.475123+0000) 2019-10-13T10:12:39.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:39.402+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:39.405035+0000) 2019-10-13T10:12:41.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:41.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.178688+0000) 2019-10-13T10:12:41.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:41.637+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.638060+0000) 2019-10-13T10:12:41.639 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:41.637+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.638443+0000) 2019-10-13T10:12:41.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:41.637+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.638823+0000) 2019-10-13T10:12:41.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:41.637+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.639891+0000) 2019-10-13T10:12:41.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:41.637+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.640382+0000) 2019-10-13T10:12:41.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:41.638+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.640630+0000) 2019-10-13T10:12:41.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:41.637+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.641562+0000) 2019-10-13T10:12:41.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:41.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:41.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:41.642+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.645046+0000) 2019-10-13T10:12:41.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:41.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:41.725740+0000) 2019-10-13T10:12:44.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:44.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.043292+0000) 2019-10-13T10:12:44.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:44.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.330797+0000) 2019-10-13T10:12:44.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:44.449+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:44.452551+0000) 2019-10-13T10:12:46.651 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:46.645+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.647431+0000) 2019-10-13T10:12:46.653 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:46.645+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.648269+0000) 2019-10-13T10:12:46.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:46.645+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.648687+0000) 2019-10-13T10:12:46.653 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:46.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.649379+0000) 2019-10-13T10:12:46.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:46.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.649163+0000) 2019-10-13T10:12:46.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:46.649+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.650255+0000) 2019-10-13T10:12:46.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:46.649+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.650542+0000) 2019-10-13T10:12:46.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:46.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:46.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:46.654+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:46.656984+0000) 2019-10-13T10:12:47.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:47.701+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:47.704462+0000) 2019-10-13T10:12:48.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:48.473+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:48.475315+0000) 2019-10-13T10:12:49.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:49.410+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:49.415693+0000) 2019-10-13T10:12:51.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:51.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.178921+0000) 2019-10-13T10:12:51.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:51.653+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.654564+0000) 2019-10-13T10:12:51.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:51.653+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.655489+0000) 2019-10-13T10:12:51.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:51.653+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.656254+0000) 2019-10-13T10:12:51.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:51.653+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.656045+0000) 2019-10-13T10:12:51.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:51.653+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.656786+0000) 2019-10-13T10:12:51.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:51.654+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.657020+0000) 2019-10-13T10:12:51.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:51.657+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.658074+0000) 2019-10-13T10:12:51.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:51.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:51.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:51.658+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.660764+0000) 2019-10-13T10:12:51.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:51.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:51.725923+0000) 2019-10-13T10:12:54.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:54.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.043518+0000) 2019-10-13T10:12:54.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:54.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.330999+0000) 2019-10-13T10:12:54.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:54.449+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:54.452750+0000) 2019-10-13T10:12:56.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:56.661+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.663329+0000) 2019-10-13T10:12:56.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:12:56.661+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.665674+0000) 2019-10-13T10:12:56.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:56.665+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.666029+0000) 2019-10-13T10:12:56.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:12:56.665+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.667126+0000) 2019-10-13T10:12:56.669 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:12:56.665+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.668316+0000) 2019-10-13T10:12:56.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:12:56.665+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.668401+0000) 2019-10-13T10:12:56.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:12:56.665+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:12:56.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:12:56.666+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.668665+0000) 2019-10-13T10:12:56.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:56.670+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:56.674431+0000) 2019-10-13T10:12:57.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:12:57.701+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:57.704666+0000) 2019-10-13T10:12:58.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:12:58.473+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:58.475545+0000) 2019-10-13T10:12:59.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:12:59.418+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:12:59.422140+0000) 2019-10-13T10:13:01.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:01.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.179119+0000) 2019-10-13T10:13:01.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:01.669+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.672463+0000) 2019-10-13T10:13:01.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:01.669+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.673111+0000) 2019-10-13T10:13:01.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:01.673+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.673847+0000) 2019-10-13T10:13:01.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:01.673+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.674234+0000) 2019-10-13T10:13:01.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:01.673+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.674880+0000) 2019-10-13T10:13:01.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:01.673+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.675282+0000) 2019-10-13T10:13:01.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:01.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.675618+0000) 2019-10-13T10:13:01.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:01.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:01.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:01.678+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.680591+0000) 2019-10-13T10:13:01.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:01.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:01.726085+0000) 2019-10-13T10:13:04.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:04.041+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.043717+0000) 2019-10-13T10:13:04.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:04.329+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.331206+0000) 2019-10-13T10:13:04.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:04.449+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:04.452947+0000) 2019-10-13T10:13:05.184 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:13:05.185 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:05.188 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:13:05.188 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:06.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:06.676+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.680606+0000) 2019-10-13T10:13:06.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:06.678+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.682472+0000) 2019-10-13T10:13:06.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:06.680+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.684370+0000) 2019-10-13T10:13:06.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:06.684+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.686330+0000) 2019-10-13T10:13:06.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:06.684+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.686612+0000) 2019-10-13T10:13:06.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:06.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:06.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:06.684+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.687123+0000) 2019-10-13T10:13:06.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:06.684+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.687634+0000) 2019-10-13T10:13:06.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:06.686+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:06.690348+0000) 2019-10-13T10:13:07.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:07.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:07.704817+0000) 2019-10-13T10:13:08.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:08.472+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:08.475737+0000) 2019-10-13T10:13:09.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:09.426+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:09.428537+0000) 2019-10-13T10:13:11.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:11.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.179322+0000) 2019-10-13T10:13:11.691 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:11.688+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.689796+0000) 2019-10-13T10:13:11.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:11.686+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.690159+0000) 2019-10-13T10:13:11.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:11.688+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.691424+0000) 2019-10-13T10:13:11.692 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:11.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.691908+0000) 2019-10-13T10:13:11.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:11.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.692858+0000) 2019-10-13T10:13:11.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:11.688+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.693058+0000) 2019-10-13T10:13:11.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:11.692+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.693792+0000) 2019-10-13T10:13:11.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:11.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:11.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:11.694+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.696912+0000) 2019-10-13T10:13:11.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:11.722+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:11.726269+0000) 2019-10-13T10:13:14.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:14.040+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.043916+0000) 2019-10-13T10:13:14.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:14.328+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.331438+0000) 2019-10-13T10:13:14.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:14.452+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:14.453151+0000) 2019-10-13T10:13:16.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:16.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.698669+0000) 2019-10-13T10:13:16.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:16.693+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.699010+0000) 2019-10-13T10:13:16.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:16.696+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.699875+0000) 2019-10-13T10:13:16.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:16.696+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.700425+0000) 2019-10-13T10:13:16.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:16.696+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.700715+0000) 2019-10-13T10:13:16.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:16.696+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.701153+0000) 2019-10-13T10:13:16.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:16.696+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.701707+0000) 2019-10-13T10:13:16.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:16.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:16.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:16.701+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:16.706650+0000) 2019-10-13T10:13:17.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:17.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:17.705015+0000) 2019-10-13T10:13:18.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:18.472+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:18.475954+0000) 2019-10-13T10:13:19.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:19.429+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:19.435143+0000) 2019-10-13T10:13:21.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:21.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.179551+0000) 2019-10-13T10:13:21.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:21.704+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.709033+0000) 2019-10-13T10:13:21.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:21.704+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.709077+0000) 2019-10-13T10:13:21.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:21.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.709816+0000) 2019-10-13T10:13:21.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:21.708+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.710215+0000) 2019-10-13T10:13:21.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:21.708+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.710608+0000) 2019-10-13T10:13:21.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:21.708+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.710967+0000) 2019-10-13T10:13:21.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:21.705+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.710990+0000) 2019-10-13T10:13:21.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:21.708+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:21.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:21.709+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.713510+0000) 2019-10-13T10:13:21.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:21.721+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:21.726434+0000) 2019-10-13T10:13:24.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:24.040+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.044147+0000) 2019-10-13T10:13:24.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:24.328+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.331636+0000) 2019-10-13T10:13:24.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:24.452+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:24.453309+0000) 2019-10-13T10:13:26.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:26.712+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.716139+0000) 2019-10-13T10:13:26.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:26.712+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.716416+0000) 2019-10-13T10:13:26.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:26.712+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.717194+0000) 2019-10-13T10:13:26.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:26.716+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.717795+0000) 2019-10-13T10:13:26.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:26.716+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.718565+0000) 2019-10-13T10:13:26.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:26.716+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.719423+0000) 2019-10-13T10:13:26.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:26.713+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.719642+0000) 2019-10-13T10:13:26.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:26.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111897 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:26.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:26.721+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:26.726323+0000) 2019-10-13T10:13:27.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:27.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:27.705211+0000) 2019-10-13T10:13:28.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:28.472+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:28.476146+0000) 2019-10-13T10:13:29.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:29.441+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:29.446779+0000) 2019-10-13T10:13:31.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:31.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.179740+0000) 2019-10-13T10:13:31.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:31.720+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.724421+0000) 2019-10-13T10:13:31.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:31.721+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.724575+0000) 2019-10-13T10:13:31.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:31.720+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.725574+0000) 2019-10-13T10:13:31.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:31.724+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.726152+0000) 2019-10-13T10:13:31.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:31.724+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.726305+0000) 2019-10-13T10:13:31.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:31.721+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.726562+0000) 2019-10-13T10:13:31.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:31.724+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.727757+0000) 2019-10-13T10:13:31.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:31.724+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.728608+0000) 2019-10-13T10:13:31.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:31.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:31.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:31.729+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:31.734697+0000) 2019-10-13T10:13:34.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:34.040+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.044340+0000) 2019-10-13T10:13:34.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:34.328+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.331827+0000) 2019-10-13T10:13:34.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:34.448+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:34.453510+0000) 2019-10-13T10:13:35.358 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:13:35.358 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:35.362 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:13:35.362 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:13:36.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:36.729+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.732407+0000) 2019-10-13T10:13:36.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:36.732+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.733832+0000) 2019-10-13T10:13:36.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:36.732+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.734398+0000) 2019-10-13T10:13:36.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:36.732+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.734056+0000) 2019-10-13T10:13:36.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:36.732+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.734881+0000) 2019-10-13T10:13:36.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:36.732+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.735196+0000) 2019-10-13T10:13:36.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:36.732+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.735807+0000) 2019-10-13T10:13:36.738 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:36.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:36.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:36.737+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:36.741545+0000) 2019-10-13T10:13:37.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:37.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:37.705362+0000) 2019-10-13T10:13:38.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:38.472+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:38.476355+0000) 2019-10-13T10:13:39.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:39.449+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:39.453426+0000) 2019-10-13T10:13:41.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:41.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.179926+0000) 2019-10-13T10:13:41.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:41.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.726752+0000) 2019-10-13T10:13:41.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:41.736+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.740580+0000) 2019-10-13T10:13:41.742 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:41.736+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.741024+0000) 2019-10-13T10:13:41.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:41.736+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.741525+0000) 2019-10-13T10:13:41.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:41.740+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.742894+0000) 2019-10-13T10:13:41.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:41.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.743275+0000) 2019-10-13T10:13:41.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:41.737+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.743325+0000) 2019-10-13T10:13:41.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:41.740+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.744425+0000) 2019-10-13T10:13:41.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:41.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:41.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:41.749+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:41.753154+0000) 2019-10-13T10:13:44.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:44.040+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.044560+0000) 2019-10-13T10:13:44.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:44.328+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.332029+0000) 2019-10-13T10:13:44.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:44.452+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:44.453763+0000) 2019-10-13T10:13:46.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:46.744+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.748569+0000) 2019-10-13T10:13:46.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:46.745+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.748617+0000) 2019-10-13T10:13:46.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:46.744+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.749533+0000) 2019-10-13T10:13:46.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:46.748+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.751617+0000) 2019-10-13T10:13:46.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:46.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.752275+0000) 2019-10-13T10:13:46.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:46.748+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.752475+0000) 2019-10-13T10:13:46.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:46.748+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.752214+0000) 2019-10-13T10:13:46.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:46.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:46.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:46.757+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:46.761089+0000) 2019-10-13T10:13:47.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:47.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:47.705538+0000) 2019-10-13T10:13:48.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:48.476+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:48.476550+0000) 2019-10-13T10:13:49.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:49.461+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:49.464892+0000) 2019-10-13T10:13:51.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:51.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.180133+0000) 2019-10-13T10:13:51.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:51.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.726943+0000) 2019-10-13T10:13:51.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:51.756+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.757977+0000) 2019-10-13T10:13:51.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:51.756+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.758426+0000) 2019-10-13T10:13:51.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:51.756+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.759358+0000) 2019-10-13T10:13:51.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:51.753+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.759392+0000) 2019-10-13T10:13:51.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:51.756+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.759568+0000) 2019-10-13T10:13:51.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:51.756+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.760147+0000) 2019-10-13T10:13:51.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:51.756+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.760730+0000) 2019-10-13T10:13:51.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:51.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:51.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:51.761+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:51.766166+0000) 2019-10-13T10:13:54.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:54.040+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.044887+0000) 2019-10-13T10:13:54.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:54.332+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.332230+0000) 2019-10-13T10:13:54.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:54.452+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:54.453979+0000) 2019-10-13T10:13:56.767 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:13:56.764+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.766666+0000) 2019-10-13T10:13:56.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:13:56.761+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.766825+0000) 2019-10-13T10:13:56.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:13:56.764+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.767878+0000) 2019-10-13T10:13:56.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:13:56.764+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.768133+0000) 2019-10-13T10:13:56.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:56.764+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.768499+0000) 2019-10-13T10:13:56.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:56.768+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.770391+0000) 2019-10-13T10:13:56.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:56.769+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.772983+0000) 2019-10-13T10:13:56.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:13:56.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:13:56.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:13:56.768+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:56.772641+0000) 2019-10-13T10:13:57.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:13:57.700+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:57.705720+0000) 2019-10-13T10:13:58.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:13:58.472+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:58.476775+0000) 2019-10-13T10:13:59.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:13:59.469+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:13:59.471328+0000) 2019-10-13T10:14:01.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:01.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.180345+0000) 2019-10-13T10:14:01.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:01.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.727148+0000) 2019-10-13T10:14:01.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:01.772+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.777357+0000) 2019-10-13T10:14:01.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:01.773+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.777389+0000) 2019-10-13T10:14:01.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:01.776+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.778057+0000) 2019-10-13T10:14:01.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:01.776+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.778514+0000) 2019-10-13T10:14:01.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:01.776+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.778875+0000) 2019-10-13T10:14:01.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:01.776+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.779163+0000) 2019-10-13T10:14:01.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:01.776+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.781614+0000) 2019-10-13T10:14:01.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:01.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:01.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:01.785+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:01.788201+0000) 2019-10-13T10:14:04.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:04.039+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.045085+0000) 2019-10-13T10:14:04.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:04.331+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.332425+0000) 2019-10-13T10:14:04.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:04.451+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:04.454205+0000) 2019-10-13T10:14:05.489 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:14:05.490 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:05.493 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:14:05.493 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:06.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:06.779+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.785121+0000) 2019-10-13T10:14:06.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:06.783+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.785936+0000) 2019-10-13T10:14:06.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:06.783+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.786726+0000) 2019-10-13T10:14:06.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:06.783+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.787090+0000) 2019-10-13T10:14:06.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:06.783+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.787607+0000) 2019-10-13T10:14:06.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:06.785+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.788153+0000) 2019-10-13T10:14:06.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:06.783+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.789053+0000) 2019-10-13T10:14:06.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:06.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:06.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:06.789+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:06.792801+0000) 2019-10-13T10:14:07.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:07.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:07.705909+0000) 2019-10-13T10:14:08.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:08.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:08.476967+0000) 2019-10-13T10:14:09.484 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:09.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:09.482612+0000) 2019-10-13T10:14:11.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:11.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.180540+0000) 2019-10-13T10:14:11.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:11.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.727373+0000) 2019-10-13T10:14:11.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:11.787+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.792966+0000) 2019-10-13T10:14:11.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:11.787+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.793423+0000) 2019-10-13T10:14:11.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:11.791+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.794072+0000) 2019-10-13T10:14:11.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:11.789+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.794355+0000) 2019-10-13T10:14:11.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:11.791+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.795315+0000) 2019-10-13T10:14:11.796 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:11.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.795733+0000) 2019-10-13T10:14:11.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:11.791+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.796393+0000) 2019-10-13T10:14:11.798 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:11.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:11.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:11.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:11.800363+0000) 2019-10-13T10:14:14.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:14.043+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.045294+0000) 2019-10-13T10:14:14.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:14.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.332626+0000) 2019-10-13T10:14:14.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:14.451+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:14.454411+0000) 2019-10-13T10:14:16.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:16.795+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.800688+0000) 2019-10-13T10:14:16.802 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:16.795+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.801300+0000) 2019-10-13T10:14:16.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:16.797+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.801269+0000) 2019-10-13T10:14:16.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:16.799+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.802211+0000) 2019-10-13T10:14:16.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:16.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.802527+0000) 2019-10-13T10:14:16.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:16.799+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.805401+0000) 2019-10-13T10:14:16.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:16.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:16.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:16.803+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.807330+0000) 2019-10-13T10:14:16.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:16.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:16.813169+0000) 2019-10-13T10:14:17.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:17.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:17.706039+0000) 2019-10-13T10:14:18.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:18.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:18.477160+0000) 2019-10-13T10:14:19.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:19.485+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:19.489090+0000) 2019-10-13T10:14:21.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:21.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.180745+0000) 2019-10-13T10:14:21.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:21.720+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.727566+0000) 2019-10-13T10:14:21.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:21.807+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.810795+0000) 2019-10-13T10:14:21.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:21.807+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.811024+0000) 2019-10-13T10:14:21.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:21.807+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.812633+0000) 2019-10-13T10:14:21.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:21.807+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.813189+0000) 2019-10-13T10:14:21.814 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:21.807+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.813248+0000) 2019-10-13T10:14:21.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:21.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.813582+0000) 2019-10-13T10:14:21.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:21.811+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.814502+0000) 2019-10-13T10:14:21.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:21.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:21.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:21.812+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:21.819890+0000) 2019-10-13T10:14:24.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:24.043+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.045497+0000) 2019-10-13T10:14:24.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:24.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.332829+0000) 2019-10-13T10:14:24.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:24.451+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:24.454621+0000) 2019-10-13T10:14:26.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:26.812+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.818884+0000) 2019-10-13T10:14:26.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:26.815+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.820259+0000) 2019-10-13T10:14:26.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:26.819+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.822340+0000) 2019-10-13T10:14:26.824 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:26.819+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.822624+0000) 2019-10-13T10:14:26.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:26.819+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.823434+0000) 2019-10-13T10:14:26.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:26.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:26.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:26.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.823762+0000) 2019-10-13T10:14:26.825 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:26.819+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.824279+0000) 2019-10-13T10:14:26.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:26.820+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:26.827217+0000) 2019-10-13T10:14:27.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:27.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:27.706203+0000) 2019-10-13T10:14:28.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:28.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:28.477364+0000) 2019-10-13T10:14:29.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:29.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:29.495537+0000) 2019-10-13T10:14:31.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:31.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.180953+0000) 2019-10-13T10:14:31.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:31.724+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.727764+0000) 2019-10-13T10:14:31.828 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:31.823+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.826810+0000) 2019-10-13T10:14:31.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:31.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.827704+0000) 2019-10-13T10:14:31.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:31.823+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.828392+0000) 2019-10-13T10:14:31.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:31.823+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.828808+0000) 2019-10-13T10:14:31.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:31.823+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.829263+0000) 2019-10-13T10:14:31.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:31.824+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.829461+0000) 2019-10-13T10:14:31.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:31.827+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.830410+0000) 2019-10-13T10:14:31.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:31.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:31.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:31.828+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:31.833326+0000) 2019-10-13T10:14:34.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:34.043+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.045714+0000) 2019-10-13T10:14:34.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:34.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.333075+0000) 2019-10-13T10:14:34.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:34.451+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:34.454862+0000) 2019-10-13T10:14:35.607 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:14:35.607 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:35.611 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:14:35.611 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:14:36.837 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:36.831+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.836260+0000) 2019-10-13T10:14:36.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:36.835+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.838455+0000) 2019-10-13T10:14:36.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:36.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.838686+0000) 2019-10-13T10:14:36.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:36.836+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.840277+0000) 2019-10-13T10:14:36.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:36.836+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.841389+0000) 2019-10-13T10:14:36.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:36.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:36.843 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:36.839+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.842289+0000) 2019-10-13T10:14:36.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:36.839+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.842359+0000) 2019-10-13T10:14:36.846 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:36.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:36.843960+0000) 2019-10-13T10:14:37.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:37.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:37.706332+0000) 2019-10-13T10:14:38.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:38.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:38.477557+0000) 2019-10-13T10:14:39.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:39.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:39.501648+0000) 2019-10-13T10:14:41.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:41.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.181181+0000) 2019-10-13T10:14:41.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:41.724+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.727998+0000) 2019-10-13T10:14:41.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:41.843+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.846120+0000) 2019-10-13T10:14:41.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:41.843+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.846530+0000) 2019-10-13T10:14:41.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:41.843+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.846903+0000) 2019-10-13T10:14:41.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:41.843+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.849228+0000) 2019-10-13T10:14:41.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:41.844+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.850719+0000) 2019-10-13T10:14:41.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:41.847+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.851244+0000) 2019-10-13T10:14:41.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:41.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:41.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:41.844+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.851568+0000) 2019-10-13T10:14:41.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:41.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:41.852314+0000) 2019-10-13T10:14:44.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:44.043+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.045985+0000) 2019-10-13T10:14:44.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:44.327+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.333314+0000) 2019-10-13T10:14:44.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:44.451+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:44.455080+0000) 2019-10-13T10:14:46.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:46.851+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.854975+0000) 2019-10-13T10:14:46.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:46.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.855408+0000) 2019-10-13T10:14:46.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:46.851+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.855910+0000) 2019-10-13T10:14:46.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:46.851+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.856552+0000) 2019-10-13T10:14:46.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:46.851+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.856738+0000) 2019-10-13T10:14:46.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:46.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.857684+0000) 2019-10-13T10:14:46.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:46.852+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.857858+0000) 2019-10-13T10:14:46.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:46.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:46.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:46.856+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:46.860719+0000) 2019-10-13T10:14:47.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:47.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:47.706459+0000) 2019-10-13T10:14:48.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:48.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:48.477781+0000) 2019-10-13T10:14:49.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:49.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:49.508004+0000) 2019-10-13T10:14:51.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:51.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.181418+0000) 2019-10-13T10:14:51.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:51.724+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.728200+0000) 2019-10-13T10:14:51.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:51.859+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.863384+0000) 2019-10-13T10:14:51.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:51.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.863733+0000) 2019-10-13T10:14:51.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:51.859+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.864616+0000) 2019-10-13T10:14:51.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:51.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.865032+0000) 2019-10-13T10:14:51.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:51.859+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.864803+0000) 2019-10-13T10:14:51.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:51.860+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.865246+0000) 2019-10-13T10:14:51.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:51.863+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.866006+0000) 2019-10-13T10:14:51.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:51.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:51.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:51.868+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:51.872651+0000) 2019-10-13T10:14:54.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:54.043+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.046218+0000) 2019-10-13T10:14:54.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:54.331+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.333520+0000) 2019-10-13T10:14:54.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:54.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:54.455275+0000) 2019-10-13T10:14:56.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:14:56.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.870626+0000) 2019-10-13T10:14:56.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:14:56.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.872448+0000) 2019-10-13T10:14:56.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:14:56.867+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.872555+0000) 2019-10-13T10:14:56.875 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:56.871+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.874652+0000) 2019-10-13T10:14:56.876 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:14:56.871+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.875122+0000) 2019-10-13T10:14:56.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:56.871+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.875866+0000) 2019-10-13T10:14:56.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:14:56.871+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.876261+0000) 2019-10-13T10:14:56.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:14:56.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:14:56.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:56.876+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:56.881605+0000) 2019-10-13T10:14:57.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:14:57.703+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:57.706596+0000) 2019-10-13T10:14:58.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:14:58.475+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:58.477989+0000) 2019-10-13T10:14:59.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:14:59.512+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:14:59.514558+0000) 2019-10-13T10:15:01.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:01.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.181621+0000) 2019-10-13T10:15:01.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:01.724+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.728397+0000) 2019-10-13T10:15:01.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:01.874+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.880263+0000) 2019-10-13T10:15:01.882 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:01.874+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.880999+0000) 2019-10-13T10:15:01.882 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:01.874+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.881239+0000) 2019-10-13T10:15:01.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:01.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.881648+0000) 2019-10-13T10:15:01.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:01.876+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.883335+0000) 2019-10-13T10:15:01.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:01.878+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.884091+0000) 2019-10-13T10:15:01.885 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:01.878+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.884363+0000) 2019-10-13T10:15:01.886 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:01.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:01.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:01.884+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:01.890706+0000) 2019-10-13T10:15:04.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:04.042+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.046437+0000) 2019-10-13T10:15:04.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:04.330+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.333696+0000) 2019-10-13T10:15:04.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:04.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:04.455504+0000) 2019-10-13T10:15:05.743 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:15:05.743 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:05.747 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:15:05.747 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:06.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:06.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.888901+0000) 2019-10-13T10:15:06.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:06.882+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.889394+0000) 2019-10-13T10:15:06.891 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:06.886+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.889808+0000) 2019-10-13T10:15:06.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:06.886+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.890071+0000) 2019-10-13T10:15:06.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:06.886+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.891218+0000) 2019-10-13T10:15:06.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:06.886+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.892393+0000) 2019-10-13T10:15:06.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:06.888+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.892712+0000) 2019-10-13T10:15:06.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:06.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112516 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:06.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:06.892+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:06.897988+0000) 2019-10-13T10:15:07.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:07.702+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:07.706738+0000) 2019-10-13T10:15:08.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:08.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:08.478184+0000) 2019-10-13T10:15:09.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:09.520+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:09.521114+0000) 2019-10-13T10:15:11.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:11.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.181821+0000) 2019-10-13T10:15:11.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:11.724+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.728606+0000) 2019-10-13T10:15:11.899 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:11.894+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.897960+0000) 2019-10-13T10:15:11.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:11.894+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.898674+0000) 2019-10-13T10:15:11.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:11.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.899148+0000) 2019-10-13T10:15:11.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:11.894+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.899675+0000) 2019-10-13T10:15:11.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:11.894+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.899777+0000) 2019-10-13T10:15:11.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:11.894+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.900364+0000) 2019-10-13T10:15:11.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:11.896+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.901531+0000) 2019-10-13T10:15:11.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:11.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:11.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:11.900+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:11.906787+0000) 2019-10-13T10:15:14.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:14.042+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.046643+0000) 2019-10-13T10:15:14.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:14.330+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.333949+0000) 2019-10-13T10:15:14.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:14.450+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:14.455701+0000) 2019-10-13T10:15:16.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:16.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.907034+0000) 2019-10-13T10:15:16.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:16.902+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.907636+0000) 2019-10-13T10:15:16.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:16.902+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.908346+0000) 2019-10-13T10:15:16.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:16.902+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.908768+0000) 2019-10-13T10:15:16.910 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:16.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.909199+0000) 2019-10-13T10:15:16.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:16.906+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.909810+0000) 2019-10-13T10:15:16.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:16.904+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.910744+0000) 2019-10-13T10:15:16.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:16.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:16.917 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:16.912+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:16.916370+0000) 2019-10-13T10:15:17.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:17.706+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:17.706930+0000) 2019-10-13T10:15:18.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:18.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:18.478395+0000) 2019-10-13T10:15:19.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:19.524+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:19.527571+0000) 2019-10-13T10:15:21.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:21.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.182049+0000) 2019-10-13T10:15:21.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:21.728+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.728822+0000) 2019-10-13T10:15:21.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:21.916+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.916307+0000) 2019-10-13T10:15:21.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:21.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.917210+0000) 2019-10-13T10:15:21.918 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:21.910+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.917383+0000) 2019-10-13T10:15:21.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:21.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.919105+0000) 2019-10-13T10:15:21.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:21.914+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.920628+0000) 2019-10-13T10:15:21.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:21.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:21.922 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:21.914+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.921453+0000) 2019-10-13T10:15:21.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:21.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.922047+0000) 2019-10-13T10:15:21.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:21.924+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:21.924749+0000) 2019-10-13T10:15:24.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:24.042+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.046879+0000) 2019-10-13T10:15:24.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:24.330+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.334142+0000) 2019-10-13T10:15:24.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:24.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:24.455917+0000) 2019-10-13T10:15:26.926 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:26.918+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.924860+0000) 2019-10-13T10:15:26.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:26.922+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.925805+0000) 2019-10-13T10:15:26.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:26.922+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.928782+0000) 2019-10-13T10:15:26.931 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:26.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:26.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:26.927+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.930113+0000) 2019-10-13T10:15:26.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:26.926+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.930727+0000) 2019-10-13T10:15:26.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:26.926+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.931488+0000) 2019-10-13T10:15:26.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:26.926+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.931786+0000) 2019-10-13T10:15:26.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:26.935+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:26.937008+0000) 2019-10-13T10:15:27.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:27.706+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:27.707071+0000) 2019-10-13T10:15:28.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:28.478+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:28.478619+0000) 2019-10-13T10:15:29.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:29.531+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:29.534028+0000) 2019-10-13T10:15:31.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:31.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.182252+0000) 2019-10-13T10:15:31.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:31.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.729017+0000) 2019-10-13T10:15:31.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:31.926+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.933111+0000) 2019-10-13T10:15:31.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:31.930+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.935737+0000) 2019-10-13T10:15:31.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:31.930+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.936023+0000) 2019-10-13T10:15:31.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:31.935+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.936974+0000) 2019-10-13T10:15:31.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:31.930+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.937713+0000) 2019-10-13T10:15:31.939 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:31.934+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.937963+0000) 2019-10-13T10:15:31.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:31.934+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.938324+0000) 2019-10-13T10:15:31.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:31.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:31.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:31.947+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:31.948396+0000) 2019-10-13T10:15:34.048 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:34.046+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.047088+0000) 2019-10-13T10:15:34.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:34.330+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.334341+0000) 2019-10-13T10:15:34.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:34.450+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:34.456147+0000) 2019-10-13T10:15:35.893 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:15:35.893 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:35.897 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:15:35.897 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:15:36.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:36.939+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.941879+0000) 2019-10-13T10:15:36.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:36.938+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.943166+0000) 2019-10-13T10:15:36.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:36.938+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.943619+0000) 2019-10-13T10:15:36.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:36.938+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.944086+0000) 2019-10-13T10:15:36.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:36.938+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.945122+0000) 2019-10-13T10:15:36.946 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:36.938+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.945305+0000) 2019-10-13T10:15:36.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:36.942+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.945809+0000) 2019-10-13T10:15:36.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:36.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:36.954 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:36.951+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:36.952777+0000) 2019-10-13T10:15:37.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:37.706+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:37.707244+0000) 2019-10-13T10:15:38.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:38.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:38.478808+0000) 2019-10-13T10:15:39.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:39.539+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:39.540412+0000) 2019-10-13T10:15:41.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:41.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.182456+0000) 2019-10-13T10:15:41.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:41.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.729220+0000) 2019-10-13T10:15:41.953 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:41.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.952203+0000) 2019-10-13T10:15:41.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:41.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.952193+0000) 2019-10-13T10:15:41.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:41.946+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.952671+0000) 2019-10-13T10:15:41.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:41.946+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.953168+0000) 2019-10-13T10:15:41.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:41.946+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.953405+0000) 2019-10-13T10:15:41.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:41.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.954221+0000) 2019-10-13T10:15:41.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:41.950+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.954975+0000) 2019-10-13T10:15:41.956 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:41.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:41.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:41.955+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:41.956839+0000) 2019-10-13T10:15:44.048 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:44.046+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.047323+0000) 2019-10-13T10:15:44.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:44.334+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.334570+0000) 2019-10-13T10:15:44.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:44.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:44.456356+0000) 2019-10-13T10:15:46.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:46.954+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.960739+0000) 2019-10-13T10:15:46.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:46.954+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.961275+0000) 2019-10-13T10:15:46.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:46.959+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.962336+0000) 2019-10-13T10:15:46.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:46.958+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.962997+0000) 2019-10-13T10:15:46.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:46.958+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.963378+0000) 2019-10-13T10:15:46.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:46.958+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.963788+0000) 2019-10-13T10:15:46.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:46.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112763 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:46.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:46.958+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.964739+0000) 2019-10-13T10:15:46.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:46.963+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:46.967476+0000) 2019-10-13T10:15:47.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:47.706+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:47.707434+0000) 2019-10-13T10:15:48.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:48.474+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:48.478998+0000) 2019-10-13T10:15:49.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:49.547+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:49.551986+0000) 2019-10-13T10:15:51.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:51.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.182653+0000) 2019-10-13T10:15:51.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:51.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.729412+0000) 2019-10-13T10:15:51.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:51.962+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.968541+0000) 2019-10-13T10:15:51.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:51.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.968846+0000) 2019-10-13T10:15:51.971 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:51.962+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.969396+0000) 2019-10-13T10:15:51.971 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:51.962+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.969334+0000) 2019-10-13T10:15:51.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:51.962+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.969682+0000) 2019-10-13T10:15:51.971 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:51.967+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.970211+0000) 2019-10-13T10:15:51.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:51.966+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.971878+0000) 2019-10-13T10:15:51.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:51.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:51.977 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:51.975+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:51.976652+0000) 2019-10-13T10:15:54.048 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:54.046+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.047549+0000) 2019-10-13T10:15:54.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:54.330+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.334763+0000) 2019-10-13T10:15:54.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:54.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:54.456615+0000) 2019-10-13T10:15:56.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:15:56.973+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.979776+0000) 2019-10-13T10:15:56.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:15:56.973+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:15:56.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:56.977+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.982017+0000) 2019-10-13T10:15:56.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:56.977+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.982499+0000) 2019-10-13T10:15:56.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:15:56.977+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.983054+0000) 2019-10-13T10:15:56.984 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:15:56.977+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.983420+0000) 2019-10-13T10:15:56.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:15:56.979+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.983636+0000) 2019-10-13T10:15:56.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:15:56.977+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.985011+0000) 2019-10-13T10:15:56.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:56.983+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:56.985962+0000) 2019-10-13T10:15:57.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:15:57.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:57.707643+0000) 2019-10-13T10:15:58.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:15:58.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:58.479206+0000) 2019-10-13T10:15:59.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:15:59.555+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:15:59.558460+0000) 2019-10-13T10:16:01.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:01.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.182874+0000) 2019-10-13T10:16:01.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:01.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.729629+0000) 2019-10-13T10:16:01.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:01.983+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.985970+0000) 2019-10-13T10:16:01.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:01.981+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.987963+0000) 2019-10-13T10:16:01.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:01.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.988393+0000) 2019-10-13T10:16:01.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:01.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:01.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:01.985+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.991154+0000) 2019-10-13T10:16:01.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:01.985+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.992028+0000) 2019-10-13T10:16:01.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:01.985+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.992481+0000) 2019-10-13T10:16:01.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:01.985+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.993219+0000) 2019-10-13T10:16:01.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:01.995+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:01.997375+0000) 2019-10-13T10:16:04.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:04.045+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.047784+0000) 2019-10-13T10:16:04.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:04.333+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.334975+0000) 2019-10-13T10:16:04.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:04.453+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:04.456784+0000) 2019-10-13T10:16:05.963 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:16:05.963 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:05.967 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:16:05.967 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:06.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:06.989+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.994262+0000) 2019-10-13T10:16:06.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:06.989+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.994668+0000) 2019-10-13T10:16:06.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:06.989+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.996056+0000) 2019-10-13T10:16:06.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:06.989+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.996160+0000) 2019-10-13T10:16:06.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:06.989+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.996723+0000) 2019-10-13T10:16:07.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:06.989+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.996987+0000) 2019-10-13T10:16:07.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:06.991+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:06.995177+0000) 2019-10-13T10:16:07.000 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:06.993+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:07.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:07.003+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.004576+0000) 2019-10-13T10:16:07.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:07.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:07.707836+0000) 2019-10-13T10:16:08.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:08.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:08.479399+0000) 2019-10-13T10:16:09.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:09.567+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:09.569936+0000) 2019-10-13T10:16:11.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:11.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.183103+0000) 2019-10-13T10:16:11.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:11.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:11.729825+0000) 2019-10-13T10:16:12.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:12.001+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.004287+0000) 2019-10-13T10:16:12.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:12.001+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.005593+0000) 2019-10-13T10:16:12.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:12.005+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.005976+0000) 2019-10-13T10:16:12.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:12.003+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.006311+0000) 2019-10-13T10:16:12.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:12.005+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.006851+0000) 2019-10-13T10:16:12.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:12.005+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.007427+0000) 2019-10-13T10:16:12.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:12.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:12.009 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:12.005+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.008009+0000) 2019-10-13T10:16:12.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:12.007+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:12.011192+0000) 2019-10-13T10:16:14.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:14.045+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.048019+0000) 2019-10-13T10:16:14.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:14.333+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.335177+0000) 2019-10-13T10:16:14.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:14.453+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:14.456995+0000) 2019-10-13T10:16:17.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:17.009+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.011914+0000) 2019-10-13T10:16:17.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:17.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.012229+0000) 2019-10-13T10:16:17.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:17.009+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.012562+0000) 2019-10-13T10:16:17.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:17.009+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.012909+0000) 2019-10-13T10:16:17.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:17.013+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.013955+0000) 2019-10-13T10:16:17.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:17.013+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.014249+0000) 2019-10-13T10:16:17.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:17.011+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.014752+0000) 2019-10-13T10:16:17.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:17.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:17.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:17.019+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.020537+0000) 2019-10-13T10:16:17.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:17.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:17.708049+0000) 2019-10-13T10:16:18.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:18.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:18.479633+0000) 2019-10-13T10:16:19.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:19.575+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:19.576395+0000) 2019-10-13T10:16:21.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:21.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.183306+0000) 2019-10-13T10:16:21.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:21.727+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:21.730039+0000) 2019-10-13T10:16:22.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:22.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.019712+0000) 2019-10-13T10:16:22.022 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:22.017+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.021312+0000) 2019-10-13T10:16:22.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:22.021+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.023188+0000) 2019-10-13T10:16:22.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:22.021+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.023517+0000) 2019-10-13T10:16:22.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:22.021+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.024213+0000) 2019-10-13T10:16:22.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:22.021+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.025049+0000) 2019-10-13T10:16:22.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:22.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:22.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:22.021+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.025417+0000) 2019-10-13T10:16:22.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:22.027+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:22.030528+0000) 2019-10-13T10:16:24.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:24.045+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.048229+0000) 2019-10-13T10:16:24.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:24.333+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.335379+0000) 2019-10-13T10:16:24.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:24.453+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:24.457216+0000) 2019-10-13T10:16:27.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:27.025+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.029350+0000) 2019-10-13T10:16:27.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:27.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.029490+0000) 2019-10-13T10:16:27.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:27.029+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.031124+0000) 2019-10-13T10:16:27.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:27.027+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.031082+0000) 2019-10-13T10:16:27.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:27.029+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.031688+0000) 2019-10-13T10:16:27.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:27.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.032172+0000) 2019-10-13T10:16:27.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:27.029+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.032755+0000) 2019-10-13T10:16:27.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:27.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:27.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:27.035+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.036589+0000) 2019-10-13T10:16:27.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:27.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:27.708258+0000) 2019-10-13T10:16:28.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:28.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:28.479858+0000) 2019-10-13T10:16:29.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:29.579+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:29.582964+0000) 2019-10-13T10:16:31.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:31.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.183511+0000) 2019-10-13T10:16:31.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:31.726+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:31.730226+0000) 2019-10-13T10:16:32.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:32.034+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.037255+0000) 2019-10-13T10:16:32.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:32.037+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.037913+0000) 2019-10-13T10:16:32.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:32.037+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.038191+0000) 2019-10-13T10:16:32.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:32.037+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.039053+0000) 2019-10-13T10:16:32.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:32.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.039890+0000) 2019-10-13T10:16:32.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:32.037+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.039968+0000) 2019-10-13T10:16:32.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:32.037+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.040402+0000) 2019-10-13T10:16:32.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:32.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:32.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:32.042+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:32.047958+0000) 2019-10-13T10:16:34.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:34.045+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.048467+0000) 2019-10-13T10:16:34.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:34.333+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.335581+0000) 2019-10-13T10:16:34.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:34.453+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:34.457415+0000) 2019-10-13T10:16:36.121 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:16:36.122 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:36.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:16:36.125 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:16:37.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:37.045+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.046157+0000) 2019-10-13T10:16:37.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:37.045+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.046609+0000) 2019-10-13T10:16:37.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:37.045+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.048776+0000) 2019-10-13T10:16:37.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:37.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:37.052 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:37.049+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.051053+0000) 2019-10-13T10:16:37.052 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:37.046+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.051182+0000) 2019-10-13T10:16:37.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:37.049+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.051969+0000) 2019-10-13T10:16:37.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:37.049+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.052860+0000) 2019-10-13T10:16:37.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:37.058+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.060695+0000) 2019-10-13T10:16:37.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:37.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:37.708428+0000) 2019-10-13T10:16:38.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:38.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:38.480059+0000) 2019-10-13T10:16:39.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:39.586+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:39.589479+0000) 2019-10-13T10:16:41.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:41.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.183719+0000) 2019-10-13T10:16:41.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:41.726+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:41.730423+0000) 2019-10-13T10:16:42.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:42.053+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.054374+0000) 2019-10-13T10:16:42.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:42.053+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.054674+0000) 2019-10-13T10:16:42.057 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:42.050+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.055306+0000) 2019-10-13T10:16:42.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:42.053+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.054997+0000) 2019-10-13T10:16:42.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:42.053+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.056266+0000) 2019-10-13T10:16:42.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:42.053+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.056529+0000) 2019-10-13T10:16:42.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:42.053+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.057039+0000) 2019-10-13T10:16:42.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:42.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:42.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:42.062+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:42.064173+0000) 2019-10-13T10:16:44.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:44.045+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.048672+0000) 2019-10-13T10:16:44.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:44.333+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.335779+0000) 2019-10-13T10:16:44.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:44.457+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:44.457587+0000) 2019-10-13T10:16:47.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:47.061+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.062549+0000) 2019-10-13T10:16:47.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:47.061+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.064235+0000) 2019-10-13T10:16:47.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:47.061+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.063769+0000) 2019-10-13T10:16:47.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:47.061+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.065694+0000) 2019-10-13T10:16:47.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:47.062+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.065738+0000) 2019-10-13T10:16:47.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:47.065+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.068163+0000) 2019-10-13T10:16:47.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:47.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:47.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:47.065+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.068668+0000) 2019-10-13T10:16:47.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:47.066+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.069198+0000) 2019-10-13T10:16:47.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:47.705+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:47.708612+0000) 2019-10-13T10:16:48.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:48.477+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:48.480281+0000) 2019-10-13T10:16:49.602 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:49.598+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:49.600849+0000) 2019-10-13T10:16:51.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:51.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.183953+0000) 2019-10-13T10:16:51.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:51.726+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:51.730617+0000) 2019-10-13T10:16:52.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:52.068+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.072321+0000) 2019-10-13T10:16:52.074 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:52.068+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.072585+0000) 2019-10-13T10:16:52.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:52.072+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.074487+0000) 2019-10-13T10:16:52.076 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:52.070+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.075004+0000) 2019-10-13T10:16:52.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:52.072+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.075484+0000) 2019-10-13T10:16:52.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:52.072+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.075898+0000) 2019-10-13T10:16:52.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:52.072+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.076472+0000) 2019-10-13T10:16:52.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:52.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:52.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:52.078+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:52.080338+0000) 2019-10-13T10:16:54.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:54.044+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.048905+0000) 2019-10-13T10:16:54.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:54.332+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.336011+0000) 2019-10-13T10:16:54.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:54.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:54.457809+0000) 2019-10-13T10:16:57.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:16:57.076+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.080256+0000) 2019-10-13T10:16:57.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:16:57.076+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.081182+0000) 2019-10-13T10:16:57.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:16:57.080+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.082090+0000) 2019-10-13T10:16:57.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:57.080+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.082546+0000) 2019-10-13T10:16:57.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:57.080+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.083029+0000) 2019-10-13T10:16:57.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:16:57.078+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.083454+0000) 2019-10-13T10:16:57.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:16:57.080+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.084638+0000) 2019-10-13T10:16:57.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:16:57.080+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113197 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:16:57.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:57.082+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.087575+0000) 2019-10-13T10:16:57.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:16:57.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:57.708790+0000) 2019-10-13T10:16:58.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:16:58.476+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:58.480475+0000) 2019-10-13T10:16:59.611 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:16:59.606+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:16:59.609701+0000) 2019-10-13T10:17:01.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:01.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.184172+0000) 2019-10-13T10:17:01.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:01.726+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:01.730806+0000) 2019-10-13T10:17:02.090 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:02.084+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.089597+0000) 2019-10-13T10:17:02.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:02.088+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.090074+0000) 2019-10-13T10:17:02.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:02.088+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.090399+0000) 2019-10-13T10:17:02.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:02.088+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.090790+0000) 2019-10-13T10:17:02.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:02.088+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.091788+0000) 2019-10-13T10:17:02.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:02.090+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.092226+0000) 2019-10-13T10:17:02.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:02.088+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.093257+0000) 2019-10-13T10:17:02.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:02.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:02.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:02.094+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:02.097658+0000) 2019-10-13T10:17:04.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:04.044+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.049107+0000) 2019-10-13T10:17:04.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:04.332+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.336243+0000) 2019-10-13T10:17:04.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:04.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:04.458009+0000) 2019-10-13T10:17:06.351 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:17:06.351 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:06.355 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:17:06.355 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:07.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:07.092+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.096542+0000) 2019-10-13T10:17:07.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:07.092+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.097177+0000) 2019-10-13T10:17:07.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:07.094+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.097860+0000) 2019-10-13T10:17:07.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:07.096+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.098577+0000) 2019-10-13T10:17:07.100 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:07.096+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.099680+0000) 2019-10-13T10:17:07.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:07.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.099174+0000) 2019-10-13T10:17:07.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:07.096+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.100230+0000) 2019-10-13T10:17:07.104 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:07.100+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:07.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:07.102+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.105509+0000) 2019-10-13T10:17:07.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:07.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:07.708975+0000) 2019-10-13T10:17:08.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:08.476+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:08.480704+0000) 2019-10-13T10:17:09.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:09.614+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:09.616277+0000) 2019-10-13T10:17:11.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:11.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.184375+0000) 2019-10-13T10:17:11.734 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:11.730+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:11.731004+0000) 2019-10-13T10:17:12.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:12.104+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.107042+0000) 2019-10-13T10:17:12.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:12.104+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.107588+0000) 2019-10-13T10:17:12.109 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:12.102+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.107887+0000) 2019-10-13T10:17:12.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:12.104+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.108677+0000) 2019-10-13T10:17:12.110 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:12.104+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.109019+0000) 2019-10-13T10:17:12.110 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:12.104+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.109474+0000) 2019-10-13T10:17:12.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:12.108+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.109927+0000) 2019-10-13T10:17:12.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:12.108+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:12.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:12.114+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:12.116658+0000) 2019-10-13T10:17:14.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:14.044+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.049304+0000) 2019-10-13T10:17:14.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:14.332+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.336440+0000) 2019-10-13T10:17:14.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:14.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:14.458227+0000) 2019-10-13T10:17:17.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:17.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.114582+0000) 2019-10-13T10:17:17.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:17.112+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.115775+0000) 2019-10-13T10:17:17.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:17.112+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.116696+0000) 2019-10-13T10:17:17.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:17.112+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.117337+0000) 2019-10-13T10:17:17.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:17.112+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.117701+0000) 2019-10-13T10:17:17.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:17.112+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.116979+0000) 2019-10-13T10:17:17.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:17.116+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.119065+0000) 2019-10-13T10:17:17.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:17.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:17.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:17.122+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.125126+0000) 2019-10-13T10:17:17.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:17.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:17.709178+0000) 2019-10-13T10:17:18.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:18.476+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:18.480942+0000) 2019-10-13T10:17:19.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:19.618+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:19.622877+0000) 2019-10-13T10:17:21.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:21.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.184578+0000) 2019-10-13T10:17:21.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:21.726+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:21.731211+0000) 2019-10-13T10:17:22.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:22.120+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.123310+0000) 2019-10-13T10:17:22.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:22.120+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.124328+0000) 2019-10-13T10:17:22.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:22.120+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.125191+0000) 2019-10-13T10:17:22.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:22.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.125427+0000) 2019-10-13T10:17:22.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:22.124+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.125996+0000) 2019-10-13T10:17:22.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:22.124+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.126706+0000) 2019-10-13T10:17:22.127 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:22.124+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.126796+0000) 2019-10-13T10:17:22.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:22.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:22.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:22.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:22.131135+0000) 2019-10-13T10:17:24.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:24.048+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.049515+0000) 2019-10-13T10:17:24.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:24.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.336669+0000) 2019-10-13T10:17:24.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:24.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:24.458428+0000) 2019-10-13T10:17:27.132 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:27.128+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.131453+0000) 2019-10-13T10:17:27.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:27.128+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.131906+0000) 2019-10-13T10:17:27.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:27.130+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.132166+0000) 2019-10-13T10:17:27.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:27.132+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.133861+0000) 2019-10-13T10:17:27.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:27.132+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.134514+0000) 2019-10-13T10:17:27.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:27.132+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.134793+0000) 2019-10-13T10:17:27.136 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:27.132+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.135514+0000) 2019-10-13T10:17:27.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:27.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:27.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:27.134+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.139389+0000) 2019-10-13T10:17:27.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:27.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:27.709362+0000) 2019-10-13T10:17:28.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:28.480+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:28.481143+0000) 2019-10-13T10:17:29.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:29.626+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:29.629514+0000) 2019-10-13T10:17:31.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:31.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.184781+0000) 2019-10-13T10:17:31.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:31.730+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:31.731465+0000) 2019-10-13T10:17:32.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:32.136+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.140496+0000) 2019-10-13T10:17:32.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:32.140+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.142176+0000) 2019-10-13T10:17:32.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:32.140+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.144116+0000) 2019-10-13T10:17:32.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:32.144+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.146012+0000) 2019-10-13T10:17:32.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:32.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113414 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:32.147 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:32.142+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.146349+0000) 2019-10-13T10:17:32.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:32.142+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.147160+0000) 2019-10-13T10:17:32.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:32.144+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.147329+0000) 2019-10-13T10:17:32.149 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:32.144+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:32.148606+0000) 2019-10-13T10:17:34.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:34.048+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.049735+0000) 2019-10-13T10:17:34.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:34.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.336868+0000) 2019-10-13T10:17:34.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:34.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:34.458665+0000) 2019-10-13T10:17:36.456 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:17:36.456 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:36.460 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:17:36.460 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:17:37.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:37.148+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.151704+0000) 2019-10-13T10:17:37.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:37.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113445 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:37.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:37.152+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.154620+0000) 2019-10-13T10:17:37.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:37.152+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.155017+0000) 2019-10-13T10:17:37.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:37.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.155258+0000) 2019-10-13T10:17:37.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:37.152+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.155478+0000) 2019-10-13T10:17:37.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:37.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.156243+0000) 2019-10-13T10:17:37.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:37.152+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.156479+0000) 2019-10-13T10:17:37.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:37.157+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.161118+0000) 2019-10-13T10:17:37.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:37.704+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:37.709582+0000) 2019-10-13T10:17:38.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:38.476+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:38.481338+0000) 2019-10-13T10:17:39.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:39.629+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:39.636048+0000) 2019-10-13T10:17:41.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:41.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:41.184984+0000) 2019-10-13T10:17:41.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:41.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:41.731684+0000) 2019-10-13T10:17:42.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:42.156+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.157943+0000) 2019-10-13T10:17:42.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:42.156+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.158521+0000) 2019-10-13T10:17:42.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:42.156+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.158871+0000) 2019-10-13T10:17:42.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:42.153+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.159983+0000) 2019-10-13T10:17:42.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:42.156+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.160512+0000) 2019-10-13T10:17:42.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:42.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.160628+0000) 2019-10-13T10:17:42.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:42.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.161308+0000) 2019-10-13T10:17:42.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:42.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:42.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:42.161+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:42.165453+0000) 2019-10-13T10:17:44.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:44.048+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.049968+0000) 2019-10-13T10:17:44.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:44.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.337066+0000) 2019-10-13T10:17:44.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:44.456+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:44.458871+0000) 2019-10-13T10:17:47.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:47.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.166292+0000) 2019-10-13T10:17:47.168 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:47.164+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.166931+0000) 2019-10-13T10:17:47.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:47.164+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.167224+0000) 2019-10-13T10:17:47.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:47.164+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.168050+0000) 2019-10-13T10:17:47.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:47.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.168466+0000) 2019-10-13T10:17:47.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:47.165+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.168517+0000) 2019-10-13T10:17:47.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:47.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.169881+0000) 2019-10-13T10:17:47.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:47.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:47.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:47.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.175007+0000) 2019-10-13T10:17:47.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:47.708+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:47.709822+0000) 2019-10-13T10:17:48.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:48.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:48.481571+0000) 2019-10-13T10:17:49.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:49.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:49.642557+0000) 2019-10-13T10:17:51.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:51.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:51.185187+0000) 2019-10-13T10:17:51.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:51.725+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:51.731918+0000) 2019-10-13T10:17:52.176 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:52.171+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.174610+0000) 2019-10-13T10:17:52.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:52.171+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.174922+0000) 2019-10-13T10:17:52.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:52.173+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.176443+0000) 2019-10-13T10:17:52.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:52.171+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.177008+0000) 2019-10-13T10:17:52.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:52.171+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.177305+0000) 2019-10-13T10:17:52.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:52.171+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.177696+0000) 2019-10-13T10:17:52.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:52.175+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.178642+0000) 2019-10-13T10:17:52.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:52.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:52.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:52.181+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:52.185181+0000) 2019-10-13T10:17:54.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:54.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.050133+0000) 2019-10-13T10:17:54.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:54.331+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.337267+0000) 2019-10-13T10:17:54.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:54.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:54.459088+0000) 2019-10-13T10:17:57.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:57.179+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.182745+0000) 2019-10-13T10:17:57.184 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:17:57.177+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.182651+0000) 2019-10-13T10:17:57.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:57.179+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.183424+0000) 2019-10-13T10:17:57.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:17:57.179+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.185193+0000) 2019-10-13T10:17:57.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:17:57.183+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.186209+0000) 2019-10-13T10:17:57.187 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:17:57.183+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.186687+0000) 2019-10-13T10:17:57.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:17:57.183+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.188676+0000) 2019-10-13T10:17:57.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:17:57.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:17:57.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:57.189+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.194818+0000) 2019-10-13T10:17:57.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:17:57.707+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:57.709996+0000) 2019-10-13T10:17:58.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:17:58.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:58.481785+0000) 2019-10-13T10:17:59.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:17:59.645+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:17:59.648782+0000) 2019-10-13T10:18:01.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:01.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:01.185391+0000) 2019-10-13T10:18:01.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:01.729+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:01.732081+0000) 2019-10-13T10:18:02.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:02.185+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.191552+0000) 2019-10-13T10:18:02.193 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:02.187+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.192416+0000) 2019-10-13T10:18:02.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:02.187+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.192806+0000) 2019-10-13T10:18:02.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:02.187+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.193344+0000) 2019-10-13T10:18:02.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:02.191+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.195686+0000) 2019-10-13T10:18:02.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:02.191+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.197536+0000) 2019-10-13T10:18:02.199 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:02.195+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:02.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:02.195+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.198796+0000) 2019-10-13T10:18:02.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:02.197+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:02.201381+0000) 2019-10-13T10:18:04.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:04.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.050386+0000) 2019-10-13T10:18:04.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:04.331+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.337494+0000) 2019-10-13T10:18:04.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:04.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:04.459291+0000) 2019-10-13T10:18:06.570 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:18:06.570 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:06.574 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:18:06.574 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:07.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:07.195+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.201025+0000) 2019-10-13T10:18:07.203 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:07.199+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.202603+0000) 2019-10-13T10:18:07.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:07.197+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.202759+0000) 2019-10-13T10:18:07.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:07.199+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.203609+0000) 2019-10-13T10:18:07.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:07.199+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.203999+0000) 2019-10-13T10:18:07.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:07.199+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.204444+0000) 2019-10-13T10:18:07.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:07.199+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.204628+0000) 2019-10-13T10:18:07.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:07.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:07.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:07.205+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.210323+0000) 2019-10-13T10:18:07.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:07.707+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:07.710178+0000) 2019-10-13T10:18:08.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:08.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:08.481981+0000) 2019-10-13T10:18:09.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:09.649+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:09.655409+0000) 2019-10-13T10:18:11.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:11.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:11.185610+0000) 2019-10-13T10:18:11.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:11.729+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:11.732307+0000) 2019-10-13T10:18:12.212 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:12.207+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.210805+0000) 2019-10-13T10:18:12.212 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:12.207+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.211256+0000) 2019-10-13T10:18:12.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:12.207+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.212050+0000) 2019-10-13T10:18:12.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:12.207+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.212136+0000) 2019-10-13T10:18:12.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:12.211+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:12.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:12.211+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.216469+0000) 2019-10-13T10:18:12.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:12.213+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.216473+0000) 2019-10-13T10:18:12.218 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:12.211+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.217248+0000) 2019-10-13T10:18:12.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:12.213+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:12.218528+0000) 2019-10-13T10:18:14.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:14.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.050591+0000) 2019-10-13T10:18:14.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:14.335+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.337702+0000) 2019-10-13T10:18:14.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:14.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:14.459480+0000) 2019-10-13T10:18:17.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:17.215+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.220307+0000) 2019-10-13T10:18:17.222 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:17.215+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.220671+0000) 2019-10-13T10:18:17.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:17.215+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.220930+0000) 2019-10-13T10:18:17.223 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:17.215+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.221356+0000) 2019-10-13T10:18:17.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:17.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.221932+0000) 2019-10-13T10:18:17.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:17.219+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.222187+0000) 2019-10-13T10:18:17.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:17.217+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.223052+0000) 2019-10-13T10:18:17.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:17.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:17.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:17.221+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.226225+0000) 2019-10-13T10:18:17.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:17.707+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:17.710368+0000) 2019-10-13T10:18:18.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:18.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:18.482172+0000) 2019-10-13T10:18:19.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:19.657+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:19.661999+0000) 2019-10-13T10:18:21.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:21.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:21.185808+0000) 2019-10-13T10:18:21.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:21.729+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:21.732500+0000) 2019-10-13T10:18:22.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:22.223+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.228212+0000) 2019-10-13T10:18:22.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:22.223+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.228495+0000) 2019-10-13T10:18:22.230 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:22.223+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.229459+0000) 2019-10-13T10:18:22.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:22.227+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.231733+0000) 2019-10-13T10:18:22.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:22.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:22.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:22.231+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.234240+0000) 2019-10-13T10:18:22.236 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:22.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.234861+0000) 2019-10-13T10:18:22.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:22.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.234794+0000) 2019-10-13T10:18:22.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:22.233+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:22.238795+0000) 2019-10-13T10:18:24.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:24.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.050793+0000) 2019-10-13T10:18:24.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:24.335+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.337907+0000) 2019-10-13T10:18:24.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:24.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:24.459648+0000) 2019-10-13T10:18:27.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:27.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.237006+0000) 2019-10-13T10:18:27.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:27.231+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.237379+0000) 2019-10-13T10:18:27.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:27.235+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.237943+0000) 2019-10-13T10:18:27.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:27.235+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.238089+0000) 2019-10-13T10:18:27.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:27.233+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.238755+0000) 2019-10-13T10:18:27.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:27.235+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.239267+0000) 2019-10-13T10:18:27.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:27.235+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.239923+0000) 2019-10-13T10:18:27.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:27.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:27.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:27.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.247160+0000) 2019-10-13T10:18:27.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:27.707+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:27.710542+0000) 2019-10-13T10:18:28.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:28.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:28.482363+0000) 2019-10-13T10:18:29.669 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:29.661+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:29.667866+0000) 2019-10-13T10:18:31.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:31.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:31.185967+0000) 2019-10-13T10:18:31.734 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:31.729+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:31.732694+0000) 2019-10-13T10:18:32.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:32.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.246003+0000) 2019-10-13T10:18:32.248 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:32.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.247733+0000) 2019-10-13T10:18:32.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:32.243+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.248317+0000) 2019-10-13T10:18:32.249 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:32.245+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.248364+0000) 2019-10-13T10:18:32.250 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:32.243+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.249010+0000) 2019-10-13T10:18:32.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:32.243+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.249271+0000) 2019-10-13T10:18:32.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:32.247+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.250572+0000) 2019-10-13T10:18:32.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:32.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:32.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:32.253+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:32.258276+0000) 2019-10-13T10:18:34.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:34.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.051025+0000) 2019-10-13T10:18:34.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:34.335+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.338112+0000) 2019-10-13T10:18:34.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:34.455+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:34.459851+0000) 2019-10-13T10:18:36.702 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:18:36.702 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:36.706 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:18:36.706 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:18:37.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:37.251+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.255095+0000) 2019-10-13T10:18:37.256 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:37.251+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.255545+0000) 2019-10-13T10:18:37.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:37.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.255590+0000) 2019-10-13T10:18:37.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:37.251+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.256612+0000) 2019-10-13T10:18:37.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:37.251+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.256346+0000) 2019-10-13T10:18:37.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:37.251+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.256863+0000) 2019-10-13T10:18:37.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:37.251+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.257642+0000) 2019-10-13T10:18:37.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:37.257+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.260996+0000) 2019-10-13T10:18:37.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:37.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:37.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:37.707+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:37.710725+0000) 2019-10-13T10:18:38.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:38.479+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:38.482589+0000) 2019-10-13T10:18:39.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:39.673+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:39.674491+0000) 2019-10-13T10:18:41.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:41.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:41.186141+0000) 2019-10-13T10:18:41.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:41.729+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:41.732846+0000) 2019-10-13T10:18:42.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:42.261+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.263396+0000) 2019-10-13T10:18:42.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:42.263+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.266841+0000) 2019-10-13T10:18:42.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:42.263+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.267560+0000) 2019-10-13T10:18:42.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:42.263+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.267830+0000) 2019-10-13T10:18:42.269 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:42.263+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.268349+0000) 2019-10-13T10:18:42.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:42.263+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.268876+0000) 2019-10-13T10:18:42.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:42.263+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.269294+0000) 2019-10-13T10:18:42.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:42.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113848 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:42.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:42.269+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:42.272366+0000) 2019-10-13T10:18:44.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:44.047+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.051226+0000) 2019-10-13T10:18:44.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:44.335+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.338343+0000) 2019-10-13T10:18:44.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:44.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:44.460048+0000) 2019-10-13T10:18:47.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:47.266+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.272827+0000) 2019-10-13T10:18:47.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:47.270+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.274018+0000) 2019-10-13T10:18:47.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:47.270+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.274952+0000) 2019-10-13T10:18:47.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:47.270+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.275788+0000) 2019-10-13T10:18:47.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:47.270+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.276117+0000) 2019-10-13T10:18:47.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:47.272+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.276574+0000) 2019-10-13T10:18:47.278 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:47.274+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.277855+0000) 2019-10-13T10:18:47.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:47.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:47.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:47.276+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.281533+0000) 2019-10-13T10:18:47.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:47.706+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:47.710901+0000) 2019-10-13T10:18:48.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:48.478+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:48.482811+0000) 2019-10-13T10:18:49.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:49.680+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:49.680978+0000) 2019-10-13T10:18:51.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:51.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:51.186373+0000) 2019-10-13T10:18:51.734 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:51.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:51.733058+0000) 2019-10-13T10:18:52.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:52.278+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.282387+0000) 2019-10-13T10:18:52.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:52.278+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.282811+0000) 2019-10-13T10:18:52.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:52.278+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.283288+0000) 2019-10-13T10:18:52.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:52.278+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.283982+0000) 2019-10-13T10:18:52.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:52.278+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.284453+0000) 2019-10-13T10:18:52.286 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:52.278+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.285051+0000) 2019-10-13T10:18:52.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:52.280+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.285014+0000) 2019-10-13T10:18:52.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:52.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:52.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:52.284+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:52.290400+0000) 2019-10-13T10:18:54.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:54.050+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.051428+0000) 2019-10-13T10:18:54.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:54.334+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.338557+0000) 2019-10-13T10:18:54.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:54.458+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:54.460274+0000) 2019-10-13T10:18:57.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:18:57.284+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.290425+0000) 2019-10-13T10:18:57.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:57.286+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.293146+0000) 2019-10-13T10:18:57.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:18:57.286+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.293539+0000) 2019-10-13T10:18:57.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:18:57.290+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.295241+0000) 2019-10-13T10:18:57.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:18:57.290+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:18:57.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:18:57.290+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.295397+0000) 2019-10-13T10:18:57.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:57.290+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.295718+0000) 2019-10-13T10:18:57.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:18:57.290+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.296300+0000) 2019-10-13T10:18:57.300 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:57.292+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.296727+0000) 2019-10-13T10:18:57.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:18:57.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:57.711112+0000) 2019-10-13T10:18:58.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:18:58.478+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:58.483000+0000) 2019-10-13T10:18:59.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:18:59.680+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:18:59.687396+0000) 2019-10-13T10:19:01.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:01.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:01.186571+0000) 2019-10-13T10:19:01.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:01.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:01.733255+0000) 2019-10-13T10:19:02.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:02.294+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.299228+0000) 2019-10-13T10:19:02.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:02.294+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.299343+0000) 2019-10-13T10:19:02.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:02.294+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.299765+0000) 2019-10-13T10:19:02.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:02.294+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.300002+0000) 2019-10-13T10:19:02.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:02.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.300060+0000) 2019-10-13T10:19:02.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:02.294+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.301559+0000) 2019-10-13T10:19:02.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:02.298+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.302288+0000) 2019-10-13T10:19:02.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:02.298+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:02.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:02.300+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:02.307507+0000) 2019-10-13T10:19:04.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:04.046+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.051633+0000) 2019-10-13T10:19:04.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:04.334+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.338775+0000) 2019-10-13T10:19:04.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:04.458+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:04.460474+0000) 2019-10-13T10:19:06.757 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:19:06.758 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:06.761 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:19:06.761 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:07.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:07.302+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.307773+0000) 2019-10-13T10:19:07.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:07.302+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.308919+0000) 2019-10-13T10:19:07.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:07.302+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.309433+0000) 2019-10-13T10:19:07.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:07.306+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.310177+0000) 2019-10-13T10:19:07.311 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:07.304+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.310231+0000) 2019-10-13T10:19:07.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:07.306+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.310770+0000) 2019-10-13T10:19:07.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:07.306+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.311019+0000) 2019-10-13T10:19:07.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:07.306+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:07.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:07.308+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.314139+0000) 2019-10-13T10:19:07.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:07.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:07.711289+0000) 2019-10-13T10:19:08.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:08.482+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:08.483193+0000) 2019-10-13T10:19:09.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:09.692+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:09.693911+0000) 2019-10-13T10:19:11.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:11.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:11.186771+0000) 2019-10-13T10:19:11.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:11.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:11.733451+0000) 2019-10-13T10:19:12.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:12.310+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.316987+0000) 2019-10-13T10:19:12.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:12.314+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.319048+0000) 2019-10-13T10:19:12.320 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:12.314+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.318883+0000) 2019-10-13T10:19:12.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:12.314+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.319419+0000) 2019-10-13T10:19:12.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:12.314+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.319851+0000) 2019-10-13T10:19:12.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:12.314+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.320011+0000) 2019-10-13T10:19:12.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:12.316+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.320525+0000) 2019-10-13T10:19:12.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:12.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:12.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:12.320+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:12.324616+0000) 2019-10-13T10:19:14.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:14.050+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.051851+0000) 2019-10-13T10:19:14.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:14.338+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.338974+0000) 2019-10-13T10:19:14.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:14.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:14.460686+0000) 2019-10-13T10:19:17.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:17.322+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.325687+0000) 2019-10-13T10:19:17.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:17.322+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.326436+0000) 2019-10-13T10:19:17.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:17.322+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.328721+0000) 2019-10-13T10:19:17.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:17.326+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.330671+0000) 2019-10-13T10:19:17.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:17.326+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.331032+0000) 2019-10-13T10:19:17.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:17.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:17.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:17.326+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.331562+0000) 2019-10-13T10:19:17.333 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:17.328+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.332377+0000) 2019-10-13T10:19:17.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:17.332+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.336380+0000) 2019-10-13T10:19:17.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:17.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:17.711465+0000) 2019-10-13T10:19:18.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:18.478+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:18.483387+0000) 2019-10-13T10:19:19.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:19.696+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:19.700392+0000) 2019-10-13T10:19:21.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:21.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:21.187000+0000) 2019-10-13T10:19:21.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:21.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:21.733677+0000) 2019-10-13T10:19:22.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:22.330+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.334202+0000) 2019-10-13T10:19:22.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:22.330+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.334910+0000) 2019-10-13T10:19:22.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:22.330+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.335573+0000) 2019-10-13T10:19:22.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:22.328+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.335594+0000) 2019-10-13T10:19:22.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:22.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.337061+0000) 2019-10-13T10:19:22.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:22.330+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.337425+0000) 2019-10-13T10:19:22.338 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:22.330+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.337162+0000) 2019-10-13T10:19:22.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:22.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:22.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:22.336+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:22.343927+0000) 2019-10-13T10:19:24.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:24.050+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.052106+0000) 2019-10-13T10:19:24.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:24.338+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.339183+0000) 2019-10-13T10:19:24.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:24.454+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:24.460909+0000) 2019-10-13T10:19:27.343 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:27.336+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.342131+0000) 2019-10-13T10:19:27.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:27.338+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.342883+0000) 2019-10-13T10:19:27.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:27.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.343491+0000) 2019-10-13T10:19:27.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:27.342+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.345831+0000) 2019-10-13T10:19:27.347 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:27.342+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.346361+0000) 2019-10-13T10:19:27.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:27.342+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.346661+0000) 2019-10-13T10:19:27.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:27.342+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.347147+0000) 2019-10-13T10:19:27.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:27.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:27.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:27.344+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.351894+0000) 2019-10-13T10:19:27.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:27.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:27.711683+0000) 2019-10-13T10:19:28.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:28.478+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:28.483598+0000) 2019-10-13T10:19:29.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:29.704+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:29.706982+0000) 2019-10-13T10:19:31.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:31.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:31.187232+0000) 2019-10-13T10:19:31.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:31.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:31.733867+0000) 2019-10-13T10:19:32.352 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:32.346+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.350936+0000) 2019-10-13T10:19:32.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:32.346+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.351743+0000) 2019-10-13T10:19:32.353 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:32.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.352141+0000) 2019-10-13T10:19:32.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:32.346+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.352879+0000) 2019-10-13T10:19:32.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:32.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.352876+0000) 2019-10-13T10:19:32.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:32.348+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.352832+0000) 2019-10-13T10:19:32.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:32.346+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.353533+0000) 2019-10-13T10:19:32.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:32.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:32.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:32.352+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:32.358355+0000) 2019-10-13T10:19:34.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:34.046+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.052337+0000) 2019-10-13T10:19:34.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:34.338+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.339388+0000) 2019-10-13T10:19:34.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:34.458+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:34.461103+0000) 2019-10-13T10:19:36.803 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:19:36.803 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:36.806 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:19:36.806 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:19:37.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:37.354+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.359543+0000) 2019-10-13T10:19:37.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:37.354+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.359871+0000) 2019-10-13T10:19:37.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:37.354+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.360515+0000) 2019-10-13T10:19:37.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:37.354+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.361300+0000) 2019-10-13T10:19:37.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:37.356+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.361906+0000) 2019-10-13T10:19:37.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:37.358+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.362255+0000) 2019-10-13T10:19:37.364 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:37.358+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.362700+0000) 2019-10-13T10:19:37.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:37.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:37.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:37.364+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.368594+0000) 2019-10-13T10:19:37.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:37.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:37.711875+0000) 2019-10-13T10:19:38.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:38.482+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:38.483822+0000) 2019-10-13T10:19:39.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:39.708+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:39.713468+0000) 2019-10-13T10:19:41.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:41.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:41.187482+0000) 2019-10-13T10:19:41.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:41.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:41.734059+0000) 2019-10-13T10:19:42.370 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:42.361+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.368574+0000) 2019-10-13T10:19:42.370 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:42.361+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.368904+0000) 2019-10-13T10:19:42.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:42.361+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.369612+0000) 2019-10-13T10:19:42.374 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:42.364+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.371356+0000) 2019-10-13T10:19:42.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:42.365+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.371180+0000) 2019-10-13T10:19:42.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:42.365+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.371956+0000) 2019-10-13T10:19:42.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:42.365+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.373041+0000) 2019-10-13T10:19:42.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:42.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:42.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:42.372+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:42.377033+0000) 2019-10-13T10:19:44.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:44.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.052542+0000) 2019-10-13T10:19:44.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:44.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.339609+0000) 2019-10-13T10:19:44.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:44.453+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:44.461303+0000) 2019-10-13T10:19:47.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:47.369+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.376879+0000) 2019-10-13T10:19:47.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:47.369+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.377525+0000) 2019-10-13T10:19:47.379 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:47.369+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.377270+0000) 2019-10-13T10:19:47.379 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:47.376+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.377563+0000) 2019-10-13T10:19:47.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:47.373+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.378242+0000) 2019-10-13T10:19:47.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:47.373+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.379044+0000) 2019-10-13T10:19:47.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:47.373+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.379890+0000) 2019-10-13T10:19:47.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:47.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:47.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:47.384+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.385381+0000) 2019-10-13T10:19:47.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:47.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:47.712059+0000) 2019-10-13T10:19:48.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:48.481+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:48.484019+0000) 2019-10-13T10:19:49.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:49.715+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:49.719934+0000) 2019-10-13T10:19:51.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:51.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:51.187719+0000) 2019-10-13T10:19:51.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:51.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:51.734253+0000) 2019-10-13T10:19:52.388 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:52.381+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.387090+0000) 2019-10-13T10:19:52.388 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:52.383+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.387532+0000) 2019-10-13T10:19:52.389 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:52.381+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.388370+0000) 2019-10-13T10:19:52.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:52.381+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.388791+0000) 2019-10-13T10:19:52.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:52.381+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.389667+0000) 2019-10-13T10:19:52.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:52.385+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.390544+0000) 2019-10-13T10:19:52.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:52.385+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.391233+0000) 2019-10-13T10:19:52.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:52.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:52.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:52.395+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:52.396594+0000) 2019-10-13T10:19:54.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:54.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.052774+0000) 2019-10-13T10:19:54.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:54.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.339824+0000) 2019-10-13T10:19:54.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:54.457+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:54.461508+0000) 2019-10-13T10:19:57.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:19:57.393+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.395727+0000) 2019-10-13T10:19:57.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:19:57.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.396341+0000) 2019-10-13T10:19:57.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:19:57.395+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.396873+0000) 2019-10-13T10:19:57.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:57.393+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.397065+0000) 2019-10-13T10:19:57.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:19:57.397+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.397824+0000) 2019-10-13T10:19:57.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:57.397+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.398143+0000) 2019-10-13T10:19:57.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:19:57.397+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.398975+0000) 2019-10-13T10:19:57.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:19:57.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:19:57.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:57.403+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.407417+0000) 2019-10-13T10:19:57.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:19:57.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:57.712245+0000) 2019-10-13T10:19:58.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:19:58.481+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:58.484216+0000) 2019-10-13T10:19:59.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:19:59.723+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:19:59.726410+0000) 2019-10-13T10:20:01.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:01.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:01.187951+0000) 2019-10-13T10:20:01.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:01.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:01.734421+0000) 2019-10-13T10:20:02.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:02.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.404654+0000) 2019-10-13T10:20:02.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:02.401+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.404956+0000) 2019-10-13T10:20:02.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:02.401+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.405490+0000) 2019-10-13T10:20:02.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:02.405+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.407569+0000) 2019-10-13T10:20:02.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:02.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.408162+0000) 2019-10-13T10:20:02.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:02.405+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.408738+0000) 2019-10-13T10:20:02.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:02.405+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.409081+0000) 2019-10-13T10:20:02.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:02.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:02.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:02.407+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:02.411978+0000) 2019-10-13T10:20:04.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:04.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.052976+0000) 2019-10-13T10:20:04.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:04.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.340067+0000) 2019-10-13T10:20:04.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:04.461+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:04.461767+0000) 2019-10-13T10:20:06.916 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:20:06.916 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:06.920 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:20:06.920 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:07.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:07.409+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.413638+0000) 2019-10-13T10:20:07.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:07.413+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.414583+0000) 2019-10-13T10:20:07.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:07.413+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.414825+0000) 2019-10-13T10:20:07.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:07.413+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.415136+0000) 2019-10-13T10:20:07.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:07.411+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.415793+0000) 2019-10-13T10:20:07.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:07.413+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.415691+0000) 2019-10-13T10:20:07.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:07.413+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.416244+0000) 2019-10-13T10:20:07.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:07.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:07.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:07.419+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.422364+0000) 2019-10-13T10:20:07.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:07.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:07.712427+0000) 2019-10-13T10:20:08.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:08.481+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:08.484442+0000) 2019-10-13T10:20:09.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:09.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:09.732981+0000) 2019-10-13T10:20:11.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:11.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:11.188153+0000) 2019-10-13T10:20:11.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:11.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:11.734625+0000) 2019-10-13T10:20:12.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:12.419+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.421831+0000) 2019-10-13T10:20:12.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:12.421+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.422699+0000) 2019-10-13T10:20:12.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:12.421+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.423283+0000) 2019-10-13T10:20:12.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:12.421+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.423958+0000) 2019-10-13T10:20:12.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:12.421+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.424315+0000) 2019-10-13T10:20:12.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:12.425+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.425957+0000) 2019-10-13T10:20:12.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:12.425+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.427335+0000) 2019-10-13T10:20:12.428 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:12.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:12.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:12.431+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:12.433008+0000) 2019-10-13T10:20:14.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:14.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.053180+0000) 2019-10-13T10:20:14.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:14.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.340298+0000) 2019-10-13T10:20:14.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:14.461+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:14.461980+0000) 2019-10-13T10:20:17.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:17.429+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.430859+0000) 2019-10-13T10:20:17.432 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:17.429+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.431470+0000) 2019-10-13T10:20:17.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:17.429+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.432324+0000) 2019-10-13T10:20:17.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:17.431+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.432412+0000) 2019-10-13T10:20:17.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:17.433+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.435072+0000) 2019-10-13T10:20:17.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:17.433+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:17.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:17.433+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.437678+0000) 2019-10-13T10:20:17.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:17.437+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.438258+0000) 2019-10-13T10:20:17.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:17.439+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.442010+0000) 2019-10-13T10:20:17.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:17.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:17.712609+0000) 2019-10-13T10:20:18.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:18.481+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:18.484667+0000) 2019-10-13T10:20:19.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:19.735+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:19.739460+0000) 2019-10-13T10:20:21.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:21.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:21.188388+0000) 2019-10-13T10:20:21.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:21.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:21.734836+0000) 2019-10-13T10:20:22.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:22.437+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.441191+0000) 2019-10-13T10:20:22.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:22.441+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.441729+0000) 2019-10-13T10:20:22.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:22.441+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.443644+0000) 2019-10-13T10:20:22.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:22.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.443758+0000) 2019-10-13T10:20:22.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:22.445+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.446456+0000) 2019-10-13T10:20:22.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:22.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:22.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:22.445+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.446884+0000) 2019-10-13T10:20:22.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:22.445+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.447408+0000) 2019-10-13T10:20:22.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:22.447+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:22.450119+0000) 2019-10-13T10:20:24.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:24.053+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.053377+0000) 2019-10-13T10:20:24.341 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:24.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.340527+0000) 2019-10-13T10:20:24.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:24.461+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:24.462175+0000) 2019-10-13T10:20:27.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:27.449+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.450195+0000) 2019-10-13T10:20:27.453 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:27.449+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.452138+0000) 2019-10-13T10:20:27.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:27.449+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.451581+0000) 2019-10-13T10:20:27.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:27.449+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.452657+0000) 2019-10-13T10:20:27.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:27.449+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.452389+0000) 2019-10-13T10:20:27.455 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:27.451+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.453814+0000) 2019-10-13T10:20:27.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:27.453+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.454401+0000) 2019-10-13T10:20:27.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:27.453+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:27.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:27.455+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.459952+0000) 2019-10-13T10:20:27.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:27.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:27.712792+0000) 2019-10-13T10:20:28.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:28.481+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:28.484895+0000) 2019-10-13T10:20:29.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:29.743+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:29.745969+0000) 2019-10-13T10:20:31.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:31.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:31.188593+0000) 2019-10-13T10:20:31.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:31.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:31.735051+0000) 2019-10-13T10:20:32.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:32.457+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.459344+0000) 2019-10-13T10:20:32.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:32.457+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.459921+0000) 2019-10-13T10:20:32.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:32.457+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.461508+0000) 2019-10-13T10:20:32.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:32.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.462428+0000) 2019-10-13T10:20:32.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:32.461+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.464436+0000) 2019-10-13T10:20:32.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:32.463+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.464329+0000) 2019-10-13T10:20:32.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:32.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:32.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:32.461+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.465554+0000) 2019-10-13T10:20:32.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:32.467+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:32.468649+0000) 2019-10-13T10:20:34.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:34.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.053581+0000) 2019-10-13T10:20:34.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:34.337+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.340730+0000) 2019-10-13T10:20:34.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:34.461+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:34.462396+0000) 2019-10-13T10:20:36.997 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:20:36.997 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:37.001 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:20:37.002 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:20:37.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:37.464+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.468338+0000) 2019-10-13T10:20:37.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:37.464+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.468828+0000) 2019-10-13T10:20:37.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:37.468+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.470895+0000) 2019-10-13T10:20:37.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:37.468+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.471175+0000) 2019-10-13T10:20:37.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:37.468+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.471773+0000) 2019-10-13T10:20:37.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:37.468+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.473238+0000) 2019-10-13T10:20:37.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:37.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:37.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:37.471+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.474354+0000) 2019-10-13T10:20:37.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:37.471+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.475332+0000) 2019-10-13T10:20:37.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:37.708+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:37.712970+0000) 2019-10-13T10:20:38.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:38.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:38.485087+0000) 2019-10-13T10:20:39.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:39.751+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:39.752608+0000) 2019-10-13T10:20:41.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:41.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:41.188793+0000) 2019-10-13T10:20:41.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:41.735+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:41.735251+0000) 2019-10-13T10:20:42.479 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:42.476+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.477826+0000) 2019-10-13T10:20:42.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:42.476+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.477973+0000) 2019-10-13T10:20:42.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:42.476+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.478494+0000) 2019-10-13T10:20:42.480 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:42.476+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.479190+0000) 2019-10-13T10:20:42.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:42.476+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.478834+0000) 2019-10-13T10:20:42.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:42.476+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.479759+0000) 2019-10-13T10:20:42.482 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:42.479+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.480815+0000) 2019-10-13T10:20:42.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:42.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:42.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:42.483+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:42.484979+0000) 2019-10-13T10:20:44.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:44.052+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.053781+0000) 2019-10-13T10:20:44.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:44.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.340948+0000) 2019-10-13T10:20:44.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:44.460+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:44.462622+0000) 2019-10-13T10:20:47.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:47.484+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.486403+0000) 2019-10-13T10:20:47.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:47.484+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.486661+0000) 2019-10-13T10:20:47.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:47.484+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.487097+0000) 2019-10-13T10:20:47.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:47.484+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.488655+0000) 2019-10-13T10:20:47.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:47.487+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.488492+0000) 2019-10-13T10:20:47.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:47.488+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.489910+0000) 2019-10-13T10:20:47.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:47.488+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.490740+0000) 2019-10-13T10:20:47.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:47.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:47.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:47.495+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.496958+0000) 2019-10-13T10:20:47.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:47.708+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:47.713167+0000) 2019-10-13T10:20:48.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:48.480+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:48.485277+0000) 2019-10-13T10:20:49.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:49.763+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:49.763929+0000) 2019-10-13T10:20:51.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:51.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:51.189002+0000) 2019-10-13T10:20:51.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:51.731+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:51.735440+0000) 2019-10-13T10:20:52.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:52.492+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.494583+0000) 2019-10-13T10:20:52.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:52.492+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.495590+0000) 2019-10-13T10:20:52.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:52.492+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.495827+0000) 2019-10-13T10:20:52.497 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:52.492+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.496277+0000) 2019-10-13T10:20:52.497 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:52.495+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.496208+0000) 2019-10-13T10:20:52.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:52.492+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.496958+0000) 2019-10-13T10:20:52.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:52.492+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.497019+0000) 2019-10-13T10:20:52.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:52.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:52.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:52.499+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:52.503978+0000) 2019-10-13T10:20:54.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:54.052+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.054001+0000) 2019-10-13T10:20:54.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:54.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.341145+0000) 2019-10-13T10:20:54.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:54.460+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:54.462819+0000) 2019-10-13T10:20:57.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:20:57.498+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.503475+0000) 2019-10-13T10:20:57.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:57.500+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.504399+0000) 2019-10-13T10:20:57.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:20:57.500+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.504856+0000) 2019-10-13T10:20:57.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:20:57.500+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.505419+0000) 2019-10-13T10:20:57.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:57.500+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.505714+0000) 2019-10-13T10:20:57.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:20:57.504+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.506261+0000) 2019-10-13T10:20:57.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:20:57.504+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.506617+0000) 2019-10-13T10:20:57.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:20:57.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:20:57.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:57.510+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.513194+0000) 2019-10-13T10:20:57.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:20:57.708+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:57.713313+0000) 2019-10-13T10:20:58.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:20:58.480+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:58.485448+0000) 2019-10-13T10:20:59.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:20:59.766+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:20:59.770519+0000) 2019-10-13T10:21:01.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:01.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:01.189234+0000) 2019-10-13T10:21:01.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:01.734+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:01.735637+0000) 2019-10-13T10:21:02.513 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:02.508+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.512748+0000) 2019-10-13T10:21:02.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:02.508+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.513576+0000) 2019-10-13T10:21:02.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:02.512+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.513957+0000) 2019-10-13T10:21:02.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:02.512+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.515672+0000) 2019-10-13T10:21:02.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:02.512+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.516270+0000) 2019-10-13T10:21:02.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:02.512+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.517068+0000) 2019-10-13T10:21:02.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:02.514+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.517253+0000) 2019-10-13T10:21:02.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:02.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:02.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:02.518+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:02.522347+0000) 2019-10-13T10:21:04.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:04.052+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.054201+0000) 2019-10-13T10:21:04.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:04.336+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.341343+0000) 2019-10-13T10:21:04.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:04.460+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:04.463039+0000) 2019-10-13T10:21:07.057 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:21:07.057 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:07.061 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:21:07.061 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:07.523 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:07.516+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.521620+0000) 2019-10-13T10:21:07.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:07.520+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.522457+0000) 2019-10-13T10:21:07.524 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:07.520+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.522648+0000) 2019-10-13T10:21:07.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:07.520+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.523136+0000) 2019-10-13T10:21:07.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:07.520+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.523565+0000) 2019-10-13T10:21:07.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:07.522+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.524156+0000) 2019-10-13T10:21:07.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:07.520+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.524466+0000) 2019-10-13T10:21:07.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:07.524+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:07.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:07.526+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.530744+0000) 2019-10-13T10:21:07.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:07.708+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:07.713492+0000) 2019-10-13T10:21:08.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:08.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:08.485621+0000) 2019-10-13T10:21:09.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:09.774+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:09.776996+0000) 2019-10-13T10:21:11.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:11.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:11.189437+0000) 2019-10-13T10:21:11.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:11.730+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:11.735852+0000) 2019-10-13T10:21:12.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:12.528+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.532216+0000) 2019-10-13T10:21:12.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:12.528+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.533491+0000) 2019-10-13T10:21:12.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:12.530+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.533917+0000) 2019-10-13T10:21:12.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:12.532+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.536105+0000) 2019-10-13T10:21:12.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:12.532+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:12.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:12.532+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.536983+0000) 2019-10-13T10:21:12.538 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:12.532+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.537433+0000) 2019-10-13T10:21:12.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:12.536+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.538996+0000) 2019-10-13T10:21:12.540 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:12.534+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:12.538915+0000) 2019-10-13T10:21:14.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:14.052+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.054411+0000) 2019-10-13T10:21:14.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:14.340+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.341544+0000) 2019-10-13T10:21:14.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:14.460+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:14.463270+0000) 2019-10-13T10:21:17.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:17.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.539382+0000) 2019-10-13T10:21:17.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:17.536+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.540298+0000) 2019-10-13T10:21:17.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:17.536+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.540622+0000) 2019-10-13T10:21:17.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:17.536+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.541619+0000) 2019-10-13T10:21:17.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:17.540+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.541898+0000) 2019-10-13T10:21:17.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:17.540+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.542760+0000) 2019-10-13T10:21:17.545 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:17.540+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.543928+0000) 2019-10-13T10:21:17.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:17.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:17.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:17.546+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.548629+0000) 2019-10-13T10:21:17.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:17.712+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:17.713650+0000) 2019-10-13T10:21:18.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:18.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:18.485790+0000) 2019-10-13T10:21:19.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:19.778+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:19.782843+0000) 2019-10-13T10:21:21.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:21.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:21.189638+0000) 2019-10-13T10:21:21.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:21.734+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:21.736049+0000) 2019-10-13T10:21:22.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:22.548+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.550811+0000) 2019-10-13T10:21:22.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:22.548+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.551220+0000) 2019-10-13T10:21:22.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:22.546+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.551334+0000) 2019-10-13T10:21:22.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:22.548+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.551991+0000) 2019-10-13T10:21:22.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:22.548+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.552459+0000) 2019-10-13T10:21:22.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:22.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:22.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:22.552+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.555140+0000) 2019-10-13T10:21:22.557 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:22.552+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.556149+0000) 2019-10-13T10:21:22.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:22.558+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:22.561072+0000) 2019-10-13T10:21:24.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:24.052+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.054618+0000) 2019-10-13T10:21:24.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:24.340+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.341789+0000) 2019-10-13T10:21:24.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:24.460+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:24.463462+0000) 2019-10-13T10:21:27.559 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:27.556+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.558455+0000) 2019-10-13T10:21:27.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:27.558+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.560505+0000) 2019-10-13T10:21:27.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:27.556+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.561355+0000) 2019-10-13T10:21:27.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:27.560+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.562226+0000) 2019-10-13T10:21:27.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:27.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:27.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:27.560+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.563478+0000) 2019-10-13T10:21:27.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:27.560+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.563836+0000) 2019-10-13T10:21:27.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:27.560+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.563609+0000) 2019-10-13T10:21:27.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:27.566+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.569618+0000) 2019-10-13T10:21:27.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:27.712+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:27.713853+0000) 2019-10-13T10:21:28.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:28.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:28.485965+0000) 2019-10-13T10:21:29.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:29.786+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:29.789467+0000) 2019-10-13T10:21:31.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:31.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:31.189847+0000) 2019-10-13T10:21:31.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:31.734+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:31.736258+0000) 2019-10-13T10:21:32.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:32.564+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.567302+0000) 2019-10-13T10:21:32.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:32.564+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.567557+0000) 2019-10-13T10:21:32.569 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:32.564+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.568246+0000) 2019-10-13T10:21:32.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:32.564+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.568930+0000) 2019-10-13T10:21:32.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:32.566+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.568755+0000) 2019-10-13T10:21:32.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:32.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.569421+0000) 2019-10-13T10:21:32.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:32.568+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.569913+0000) 2019-10-13T10:21:32.572 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:32.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:32.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:32.574+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:32.576728+0000) 2019-10-13T10:21:34.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:34.051+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.054861+0000) 2019-10-13T10:21:34.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:34.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.341990+0000) 2019-10-13T10:21:34.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:34.459+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:34.463691+0000) 2019-10-13T10:21:37.157 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:21:37.157 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:37.161 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:21:37.161 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:21:37.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:37.571+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.574512+0000) 2019-10-13T10:21:37.576 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:37.571+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.575045+0000) 2019-10-13T10:21:37.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:37.571+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.577292+0000) 2019-10-13T10:21:37.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:37.571+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.577590+0000) 2019-10-13T10:21:37.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:37.575+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.578316+0000) 2019-10-13T10:21:37.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:37.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.578976+0000) 2019-10-13T10:21:37.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:37.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:37.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:37.575+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.579595+0000) 2019-10-13T10:21:37.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:37.582+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.586392+0000) 2019-10-13T10:21:37.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:37.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:37.714052+0000) 2019-10-13T10:21:38.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:38.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:38.486100+0000) 2019-10-13T10:21:39.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:39.794+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:39.796002+0000) 2019-10-13T10:21:41.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:41.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:41.190046+0000) 2019-10-13T10:21:41.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:41.734+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:41.736462+0000) 2019-10-13T10:21:42.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:42.579+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.583805+0000) 2019-10-13T10:21:42.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:42.579+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.584037+0000) 2019-10-13T10:21:42.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:42.579+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.584559+0000) 2019-10-13T10:21:42.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:42.579+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.584802+0000) 2019-10-13T10:21:42.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:42.579+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.585293+0000) 2019-10-13T10:21:42.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:42.579+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.585591+0000) 2019-10-13T10:21:42.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:42.582+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.586322+0000) 2019-10-13T10:21:42.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:42.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:42.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:42.590+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:42.592239+0000) 2019-10-13T10:21:44.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:44.051+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.055134+0000) 2019-10-13T10:21:44.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:44.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.342212+0000) 2019-10-13T10:21:44.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:44.459+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:44.463848+0000) 2019-10-13T10:21:47.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:47.591+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.594135+0000) 2019-10-13T10:21:47.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:47.591+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.597517+0000) 2019-10-13T10:21:47.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:47.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:47.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:47.594+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.597890+0000) 2019-10-13T10:21:47.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:47.594+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.598516+0000) 2019-10-13T10:21:47.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:47.595+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.598744+0000) 2019-10-13T10:21:47.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:47.595+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.599345+0000) 2019-10-13T10:21:47.602 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:47.595+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.599786+0000) 2019-10-13T10:21:47.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:47.595+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.600262+0000) 2019-10-13T10:21:47.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:47.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:47.714266+0000) 2019-10-13T10:21:48.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:48.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:48.486281+0000) 2019-10-13T10:21:49.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:49.802+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:49.802417+0000) 2019-10-13T10:21:51.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:51.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:51.190289+0000) 2019-10-13T10:21:51.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:51.734+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:51.736658+0000) 2019-10-13T10:21:52.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:52.598+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.602096+0000) 2019-10-13T10:21:52.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:52.599+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.603058+0000) 2019-10-13T10:21:52.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:52.599+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.603352+0000) 2019-10-13T10:21:52.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:52.599+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.603684+0000) 2019-10-13T10:21:52.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:52.599+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.605332+0000) 2019-10-13T10:21:52.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:52.603+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.607003+0000) 2019-10-13T10:21:52.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:52.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115026 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:52.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:52.595+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.601663+0000) 2019-10-13T10:21:52.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:52.602+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:52.607697+0000) 2019-10-13T10:21:54.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:54.051+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.055340+0000) 2019-10-13T10:21:54.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:54.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.342428+0000) 2019-10-13T10:21:54.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:54.463+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:54.464048+0000) 2019-10-13T10:21:57.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:21:57.607+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.611070+0000) 2019-10-13T10:21:57.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:21:57.607+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.611451+0000) 2019-10-13T10:21:57.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:21:57.607+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.613058+0000) 2019-10-13T10:21:57.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:21:57.610+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.613467+0000) 2019-10-13T10:21:57.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:57.611+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.616467+0000) 2019-10-13T10:21:57.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:21:57.611+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115055 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:21:57.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:21:57.611+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.617291+0000) 2019-10-13T10:21:57.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:57.615+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.618306+0000) 2019-10-13T10:21:57.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:57.622+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.624521+0000) 2019-10-13T10:21:57.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:21:57.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:57.714475+0000) 2019-10-13T10:21:58.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:21:58.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:58.486506+0000) 2019-10-13T10:21:59.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:21:59.806+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:21:59.808713+0000) 2019-10-13T10:22:01.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:01.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:01.190521+0000) 2019-10-13T10:22:01.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:01.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:01.736859+0000) 2019-10-13T10:22:02.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:02.615+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.619498+0000) 2019-10-13T10:22:02.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:02.615+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.619987+0000) 2019-10-13T10:22:02.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:02.615+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.620268+0000) 2019-10-13T10:22:02.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:02.615+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.620948+0000) 2019-10-13T10:22:02.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:02.619+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.622820+0000) 2019-10-13T10:22:02.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:02.617+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.622633+0000) 2019-10-13T10:22:02.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:02.623+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.625919+0000) 2019-10-13T10:22:02.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:02.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:02.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:02.625+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:02.631076+0000) 2019-10-13T10:22:04.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:04.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.055581+0000) 2019-10-13T10:22:04.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:04.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.342644+0000) 2019-10-13T10:22:04.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:04.459+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:04.464249+0000) 2019-10-13T10:22:07.228 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:22:07.228 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:07.232 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:22:07.232 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:07.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:07.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.629729+0000) 2019-10-13T10:22:07.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:07.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.630170+0000) 2019-10-13T10:22:07.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:07.627+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.630620+0000) 2019-10-13T10:22:07.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:07.627+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.631210+0000) 2019-10-13T10:22:07.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:07.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.632010+0000) 2019-10-13T10:22:07.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:07.627+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.632112+0000) 2019-10-13T10:22:07.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:07.629+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.632794+0000) 2019-10-13T10:22:07.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:07.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:07.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:07.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.638801+0000) 2019-10-13T10:22:07.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:07.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:07.714658+0000) 2019-10-13T10:22:08.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:08.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:08.486684+0000) 2019-10-13T10:22:09.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:09.809+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:09.815205+0000) 2019-10-13T10:22:11.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:11.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:11.190724+0000) 2019-10-13T10:22:11.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:11.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:11.737074+0000) 2019-10-13T10:22:12.639 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:12.635+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.638227+0000) 2019-10-13T10:22:12.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:12.635+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.638685+0000) 2019-10-13T10:22:12.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:12.635+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.639468+0000) 2019-10-13T10:22:12.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:12.635+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.641385+0000) 2019-10-13T10:22:12.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:12.639+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.643405+0000) 2019-10-13T10:22:12.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:12.639+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.643693+0000) 2019-10-13T10:22:12.645 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:12.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:12.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:12.641+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.644803+0000) 2019-10-13T10:22:12.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:12.645+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:12.649553+0000) 2019-10-13T10:22:14.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:14.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.055812+0000) 2019-10-13T10:22:14.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:14.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.342844+0000) 2019-10-13T10:22:14.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:14.463+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:14.464444+0000) 2019-10-13T10:22:17.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:17.643+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.647750+0000) 2019-10-13T10:22:17.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:17.643+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.648088+0000) 2019-10-13T10:22:17.651 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:17.647+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.650025+0000) 2019-10-13T10:22:17.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:17.643+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.649618+0000) 2019-10-13T10:22:17.651 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:17.647+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.650465+0000) 2019-10-13T10:22:17.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:17.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.651135+0000) 2019-10-13T10:22:17.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:17.647+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.650737+0000) 2019-10-13T10:22:17.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:17.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:17.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:17.649+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.654689+0000) 2019-10-13T10:22:17.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:17.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:17.714867+0000) 2019-10-13T10:22:18.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:18.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:18.486873+0000) 2019-10-13T10:22:19.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:19.817+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:19.821267+0000) 2019-10-13T10:22:21.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:21.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:21.190931+0000) 2019-10-13T10:22:21.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:21.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:21.737301+0000) 2019-10-13T10:22:22.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:22.651+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.657432+0000) 2019-10-13T10:22:22.659 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:22.651+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.657699+0000) 2019-10-13T10:22:22.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:22.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.658387+0000) 2019-10-13T10:22:22.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:22.655+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.658437+0000) 2019-10-13T10:22:22.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:22.655+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.658979+0000) 2019-10-13T10:22:22.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:22.655+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.659298+0000) 2019-10-13T10:22:22.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:22.655+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.659556+0000) 2019-10-13T10:22:22.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:22.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115210 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:22.669 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:22.661+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:22.667884+0000) 2019-10-13T10:22:24.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:24.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.056060+0000) 2019-10-13T10:22:24.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:24.339+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.343043+0000) 2019-10-13T10:22:24.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:24.463+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:24.464663+0000) 2019-10-13T10:22:27.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:27.663+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.665730+0000) 2019-10-13T10:22:27.667 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:27.663+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.666321+0000) 2019-10-13T10:22:27.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:27.663+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.666923+0000) 2019-10-13T10:22:27.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:27.663+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.667096+0000) 2019-10-13T10:22:27.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:27.663+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.669204+0000) 2019-10-13T10:22:27.670 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:27.663+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.668078+0000) 2019-10-13T10:22:27.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:27.665+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.670084+0000) 2019-10-13T10:22:27.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:27.667+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:27.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:27.669+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.674801+0000) 2019-10-13T10:22:27.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:27.711+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:27.715056+0000) 2019-10-13T10:22:28.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:28.483+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:28.487043+0000) 2019-10-13T10:22:29.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:29.825+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:29.827825+0000) 2019-10-13T10:22:31.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:31.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:31.191161+0000) 2019-10-13T10:22:31.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:31.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:31.737491+0000) 2019-10-13T10:22:32.675 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:32.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.674692+0000) 2019-10-13T10:22:32.676 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:32.670+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.675319+0000) 2019-10-13T10:22:32.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:32.670+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.675680+0000) 2019-10-13T10:22:32.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:32.670+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.675962+0000) 2019-10-13T10:22:32.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:32.670+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.676748+0000) 2019-10-13T10:22:32.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:32.673+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.677382+0000) 2019-10-13T10:22:32.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:32.670+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.677243+0000) 2019-10-13T10:22:32.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:32.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115273 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:32.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:32.677+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:32.680158+0000) 2019-10-13T10:22:34.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:34.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.056292+0000) 2019-10-13T10:22:34.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:34.338+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.343265+0000) 2019-10-13T10:22:34.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:34.462+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:34.464866+0000) 2019-10-13T10:22:37.283 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:22:37.284 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:37.287 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:22:37.287 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:22:37.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:37.678+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.683602+0000) 2019-10-13T10:22:37.685 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:37.678+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.684696+0000) 2019-10-13T10:22:37.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:37.678+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.684995+0000) 2019-10-13T10:22:37.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:37.678+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.685702+0000) 2019-10-13T10:22:37.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:37.682+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.686404+0000) 2019-10-13T10:22:37.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:37.682+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.686867+0000) 2019-10-13T10:22:37.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:37.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:37.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:37.681+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.687107+0000) 2019-10-13T10:22:37.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:37.685+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.689583+0000) 2019-10-13T10:22:37.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:37.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:37.715261+0000) 2019-10-13T10:22:38.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:38.482+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:38.487219+0000) 2019-10-13T10:22:39.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:39.829+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:39.834102+0000) 2019-10-13T10:22:41.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:41.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:41.191390+0000) 2019-10-13T10:22:41.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:41.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:41.737722+0000) 2019-10-13T10:22:42.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:42.686+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.692254+0000) 2019-10-13T10:22:42.694 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:42.689+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.692257+0000) 2019-10-13T10:22:42.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:42.686+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.693254+0000) 2019-10-13T10:22:42.695 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:42.690+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.694243+0000) 2019-10-13T10:22:42.696 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:42.690+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.694800+0000) 2019-10-13T10:22:42.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:42.690+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.694914+0000) 2019-10-13T10:22:42.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:42.690+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.695387+0000) 2019-10-13T10:22:42.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:42.690+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:42.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:42.697+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:42.702873+0000) 2019-10-13T10:22:44.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:44.050+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.056509+0000) 2019-10-13T10:22:44.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:44.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.343503+0000) 2019-10-13T10:22:44.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:44.458+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:44.465065+0000) 2019-10-13T10:22:47.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:47.694+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.700853+0000) 2019-10-13T10:22:47.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:47.698+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.701881+0000) 2019-10-13T10:22:47.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:47.697+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.702839+0000) 2019-10-13T10:22:47.704 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:47.698+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.703256+0000) 2019-10-13T10:22:47.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:47.698+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.705008+0000) 2019-10-13T10:22:47.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:47.702+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.706321+0000) 2019-10-13T10:22:47.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:47.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:47.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:47.702+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.706583+0000) 2019-10-13T10:22:47.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:47.709+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.713140+0000) 2019-10-13T10:22:47.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:47.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:47.715369+0000) 2019-10-13T10:22:48.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:48.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:48.487418+0000) 2019-10-13T10:22:49.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:49.841+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:49.844954+0000) 2019-10-13T10:22:51.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:51.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:51.191626+0000) 2019-10-13T10:22:51.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:51.733+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:51.737947+0000) 2019-10-13T10:22:52.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:52.706+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.709828+0000) 2019-10-13T10:22:52.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:52.706+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.710089+0000) 2019-10-13T10:22:52.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:52.706+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.711165+0000) 2019-10-13T10:22:52.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:52.706+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.711876+0000) 2019-10-13T10:22:52.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:52.706+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.712279+0000) 2019-10-13T10:22:52.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:52.709+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.712871+0000) 2019-10-13T10:22:52.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:52.706+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.712752+0000) 2019-10-13T10:22:52.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:52.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:52.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:52.713+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:52.719590+0000) 2019-10-13T10:22:54.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:54.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.056711+0000) 2019-10-13T10:22:54.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:54.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.343704+0000) 2019-10-13T10:22:54.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:54.458+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:54.465259+0000) 2019-10-13T10:22:57.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:57.710+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.715553+0000) 2019-10-13T10:22:57.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:22:57.714+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.719142+0000) 2019-10-13T10:22:57.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:22:57.714+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.720876+0000) 2019-10-13T10:22:57.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:22:57.714+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.721397+0000) 2019-10-13T10:22:57.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:22:57.717+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.721531+0000) 2019-10-13T10:22:57.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:57.718+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.723722+0000) 2019-10-13T10:22:57.725 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:22:57.718+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.724085+0000) 2019-10-13T10:22:57.725 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:22:57.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:22:57.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:22:57.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.724720+0000) 2019-10-13T10:22:57.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:57.725+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:57.729480+0000) 2019-10-13T10:22:58.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:22:58.482+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:58.487590+0000) 2019-10-13T10:22:59.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:22:59.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:22:59.851372+0000) 2019-10-13T10:23:01.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:01.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:01.191835+0000) 2019-10-13T10:23:01.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:01.737+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:01.738165+0000) 2019-10-13T10:23:02.729 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:02.722+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.728204+0000) 2019-10-13T10:23:02.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:02.722+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.728711+0000) 2019-10-13T10:23:02.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:02.726+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.730400+0000) 2019-10-13T10:23:02.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:02.726+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.730801+0000) 2019-10-13T10:23:02.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:02.726+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.731847+0000) 2019-10-13T10:23:02.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:02.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115456 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:02.734 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:02.729+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.732850+0000) 2019-10-13T10:23:02.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:02.726+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.733366+0000) 2019-10-13T10:23:02.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:02.733+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:02.738711+0000) 2019-10-13T10:23:04.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:04.050+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.056945+0000) 2019-10-13T10:23:04.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:04.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.343933+0000) 2019-10-13T10:23:04.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:04.462+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:04.465462+0000) 2019-10-13T10:23:07.556 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:23:07.557 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:07.560 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:23:07.561 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:07.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:07.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.715746+0000) 2019-10-13T10:23:07.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:07.730+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.736138+0000) 2019-10-13T10:23:07.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:07.730+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.736665+0000) 2019-10-13T10:23:07.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:07.730+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.737709+0000) 2019-10-13T10:23:07.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:07.734+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.738381+0000) 2019-10-13T10:23:07.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:07.734+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.738867+0000) 2019-10-13T10:23:07.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:07.732+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.738948+0000) 2019-10-13T10:23:07.740 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:07.734+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.739928+0000) 2019-10-13T10:23:07.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:07.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:07.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:07.736+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:07.743073+0000) 2019-10-13T10:23:08.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:08.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:08.487739+0000) 2019-10-13T10:23:09.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:09.856+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:09.857823+0000) 2019-10-13T10:23:11.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:11.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:11.192052+0000) 2019-10-13T10:23:11.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:11.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:11.738362+0000) 2019-10-13T10:23:12.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:12.740+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.747945+0000) 2019-10-13T10:23:12.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:12.742+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.747895+0000) 2019-10-13T10:23:12.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:12.742+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.748443+0000) 2019-10-13T10:23:12.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:12.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:12.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:12.746+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.751189+0000) 2019-10-13T10:23:12.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:12.746+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.751690+0000) 2019-10-13T10:23:12.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:12.746+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.752121+0000) 2019-10-13T10:23:12.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:12.748+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.752282+0000) 2019-10-13T10:23:12.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:12.746+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:12.752712+0000) 2019-10-13T10:23:14.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:14.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.057177+0000) 2019-10-13T10:23:14.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:14.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.344153+0000) 2019-10-13T10:23:14.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:14.462+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:14.465691+0000) 2019-10-13T10:23:17.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:17.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.715990+0000) 2019-10-13T10:23:17.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:17.750+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.753893+0000) 2019-10-13T10:23:17.756 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:17.750+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.755112+0000) 2019-10-13T10:23:17.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:17.750+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.755724+0000) 2019-10-13T10:23:17.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:17.750+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.756062+0000) 2019-10-13T10:23:17.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:17.752+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.756994+0000) 2019-10-13T10:23:17.760 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:17.754+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.758923+0000) 2019-10-13T10:23:17.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:17.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:17.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:17.754+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.760073+0000) 2019-10-13T10:23:17.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:17.756+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:17.760578+0000) 2019-10-13T10:23:18.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:18.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:18.487931+0000) 2019-10-13T10:23:19.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:19.864+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:19.868833+0000) 2019-10-13T10:23:21.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:21.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:21.192251+0000) 2019-10-13T10:23:21.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:21.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:21.738559+0000) 2019-10-13T10:23:22.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:22.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.762256+0000) 2019-10-13T10:23:22.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:22.758+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.763172+0000) 2019-10-13T10:23:22.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:22.758+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.764042+0000) 2019-10-13T10:23:22.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:22.758+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.764709+0000) 2019-10-13T10:23:22.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:22.758+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.765298+0000) 2019-10-13T10:23:22.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:22.758+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.765690+0000) 2019-10-13T10:23:22.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:22.762+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.766150+0000) 2019-10-13T10:23:22.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:22.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:22.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:22.768+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:22.773301+0000) 2019-10-13T10:23:24.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:24.054+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.057331+0000) 2019-10-13T10:23:24.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:24.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.344383+0000) 2019-10-13T10:23:24.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:24.462+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:24.465871+0000) 2019-10-13T10:23:27.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:27.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.716143+0000) 2019-10-13T10:23:27.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:27.765+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.771664+0000) 2019-10-13T10:23:27.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:27.765+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.772370+0000) 2019-10-13T10:23:27.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:27.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.772988+0000) 2019-10-13T10:23:27.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:27.768+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.773035+0000) 2019-10-13T10:23:27.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:27.769+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.773799+0000) 2019-10-13T10:23:27.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:27.769+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.774217+0000) 2019-10-13T10:23:27.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:27.769+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.775141+0000) 2019-10-13T10:23:27.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:27.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:27.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:27.772+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:27.777525+0000) 2019-10-13T10:23:28.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:28.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:28.488106+0000) 2019-10-13T10:23:29.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:29.872+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:29.875422+0000) 2019-10-13T10:23:31.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:31.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:31.192458+0000) 2019-10-13T10:23:31.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:31.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:31.738749+0000) 2019-10-13T10:23:32.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:32.773+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.779917+0000) 2019-10-13T10:23:32.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:32.773+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.780321+0000) 2019-10-13T10:23:32.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:32.776+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.780556+0000) 2019-10-13T10:23:32.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:32.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.780821+0000) 2019-10-13T10:23:32.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:32.777+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.781836+0000) 2019-10-13T10:23:32.783 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:32.777+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.782525+0000) 2019-10-13T10:23:32.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:32.777+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.783533+0000) 2019-10-13T10:23:32.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:32.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:32.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:32.780+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:32.785189+0000) 2019-10-13T10:23:34.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:34.049+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.057541+0000) 2019-10-13T10:23:34.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:34.341+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.344613+0000) 2019-10-13T10:23:34.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:34.465+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:34.466080+0000) 2019-10-13T10:23:37.604 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:23:37.604 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:37.608 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:23:37.608 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:23:37.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:37.709+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.716338+0000) 2019-10-13T10:23:37.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:37.781+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.788518+0000) 2019-10-13T10:23:37.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:37.785+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.791727+0000) 2019-10-13T10:23:37.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:37.785+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.791804+0000) 2019-10-13T10:23:37.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:37.784+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.791847+0000) 2019-10-13T10:23:37.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:37.785+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.792493+0000) 2019-10-13T10:23:37.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:37.789+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.793987+0000) 2019-10-13T10:23:37.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:37.789+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115669 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:37.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:37.789+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.794778+0000) 2019-10-13T10:23:37.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:37.792+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:37.797491+0000) 2019-10-13T10:23:38.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:38.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:38.488279+0000) 2019-10-13T10:23:39.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:39.880+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:39.881949+0000) 2019-10-13T10:23:41.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:41.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:41.192673+0000) 2019-10-13T10:23:41.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:41.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:41.738949+0000) 2019-10-13T10:23:42.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:42.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.796652+0000) 2019-10-13T10:23:42.798 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:42.793+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.797388+0000) 2019-10-13T10:23:42.802 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:42.801+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.801773+0000) 2019-10-13T10:23:42.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:42.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:42.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:42.801+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.803637+0000) 2019-10-13T10:23:42.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:42.796+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.803966+0000) 2019-10-13T10:23:42.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:42.801+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.804089+0000) 2019-10-13T10:23:42.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:42.801+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.804534+0000) 2019-10-13T10:23:42.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:42.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:42.805734+0000) 2019-10-13T10:23:44.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:44.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:44.057785+0000) 2019-10-13T10:23:44.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:44.341+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:44.344815+0000) 2019-10-13T10:23:44.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:44.465+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:44.466310+0000) 2019-10-13T10:23:47.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:47.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.716542+0000) 2019-10-13T10:23:47.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:47.805+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.806097+0000) 2019-10-13T10:23:47.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:47.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.807467+0000) 2019-10-13T10:23:47.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:47.805+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.807827+0000) 2019-10-13T10:23:47.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:47.805+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.808779+0000) 2019-10-13T10:23:47.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:47.805+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.808915+0000) 2019-10-13T10:23:47.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:47.804+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.809121+0000) 2019-10-13T10:23:47.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:47.809+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.809929+0000) 2019-10-13T10:23:47.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:47.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:47.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:47.808+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:47.815588+0000) 2019-10-13T10:23:48.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:48.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:48.488451+0000) 2019-10-13T10:23:49.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:49.888+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:49.888406+0000) 2019-10-13T10:23:51.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:51.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:51.192875+0000) 2019-10-13T10:23:51.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:51.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:51.739145+0000) 2019-10-13T10:23:52.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:52.812+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.815244+0000) 2019-10-13T10:23:52.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:52.813+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.816791+0000) 2019-10-13T10:23:52.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:52.813+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.817082+0000) 2019-10-13T10:23:52.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:52.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.817621+0000) 2019-10-13T10:23:52.819 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:52.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.818012+0000) 2019-10-13T10:23:52.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:52.817+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.818357+0000) 2019-10-13T10:23:52.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:52.817+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.818763+0000) 2019-10-13T10:23:52.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:52.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:52.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:52.820+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:52.825320+0000) 2019-10-13T10:23:54.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:54.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:54.057988+0000) 2019-10-13T10:23:54.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:54.341+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:54.345032+0000) 2019-10-13T10:23:54.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:54.465+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:54.466537+0000) 2019-10-13T10:23:57.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:57.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.716750+0000) 2019-10-13T10:23:57.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:23:57.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.824142+0000) 2019-10-13T10:23:57.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:23:57.821+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.824427+0000) 2019-10-13T10:23:57.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:23:57.825+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.825855+0000) 2019-10-13T10:23:57.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:23:57.820+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.826126+0000) 2019-10-13T10:23:57.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:23:57.825+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.826782+0000) 2019-10-13T10:23:57.828 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:57.825+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.827109+0000) 2019-10-13T10:23:57.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:23:57.825+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.827445+0000) 2019-10-13T10:23:57.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:23:57.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:23:57.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:57.824+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:57.831646+0000) 2019-10-13T10:23:58.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:23:58.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:58.488620+0000) 2019-10-13T10:23:59.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:23:59.892+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:23:59.895062+0000) 2019-10-13T10:24:01.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:01.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:01.193079+0000) 2019-10-13T10:24:01.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:01.732+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:01.739396+0000) 2019-10-13T10:24:02.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:02.828+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.831627+0000) 2019-10-13T10:24:02.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:02.829+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.832324+0000) 2019-10-13T10:24:02.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:02.833+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.833724+0000) 2019-10-13T10:24:02.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:02.833+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.835271+0000) 2019-10-13T10:24:02.836 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:02.833+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.835700+0000) 2019-10-13T10:24:02.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:02.833+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.836680+0000) 2019-10-13T10:24:02.838 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:02.833+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.837111+0000) 2019-10-13T10:24:02.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:02.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:02.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:02.832+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:02.838580+0000) 2019-10-13T10:24:04.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:04.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:04.058236+0000) 2019-10-13T10:24:04.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:04.341+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:04.345231+0000) 2019-10-13T10:24:04.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:04.465+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:04.466707+0000) 2019-10-13T10:24:07.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:07.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.716979+0000) 2019-10-13T10:24:07.720 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:24:07.721 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:07.724 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:24:07.724 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:07.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:07.837+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.840520+0000) 2019-10-13T10:24:07.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:07.837+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.840935+0000) 2019-10-13T10:24:07.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:07.837+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.841512+0000) 2019-10-13T10:24:07.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:07.841+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.842316+0000) 2019-10-13T10:24:07.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:07.840+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.842214+0000) 2019-10-13T10:24:07.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:07.841+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.844478+0000) 2019-10-13T10:24:07.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:07.841+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.844528+0000) 2019-10-13T10:24:07.846 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:07.845+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115851 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:07.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:07.848+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:07.850770+0000) 2019-10-13T10:24:08.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:08.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:08.488752+0000) 2019-10-13T10:24:09.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:09.900+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:09.901700+0000) 2019-10-13T10:24:11.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:11.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:11.193284+0000) 2019-10-13T10:24:11.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:11.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:11.739595+0000) 2019-10-13T10:24:12.850 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:12.845+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.849403+0000) 2019-10-13T10:24:12.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:12.849+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.850091+0000) 2019-10-13T10:24:12.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:12.849+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.850821+0000) 2019-10-13T10:24:12.852 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:12.849+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.850832+0000) 2019-10-13T10:24:12.853 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:12.849+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.852157+0000) 2019-10-13T10:24:12.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:12.849+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.852533+0000) 2019-10-13T10:24:12.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:12.851+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.852785+0000) 2019-10-13T10:24:12.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:12.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:12.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:12.855+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:12.858369+0000) 2019-10-13T10:24:14.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:14.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:14.058489+0000) 2019-10-13T10:24:14.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:14.345+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:14.345463+0000) 2019-10-13T10:24:14.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:14.465+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:14.466923+0000) 2019-10-13T10:24:17.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:17.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.717136+0000) 2019-10-13T10:24:17.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:17.857+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.858743+0000) 2019-10-13T10:24:17.860 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:17.857+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.859400+0000) 2019-10-13T10:24:17.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:17.857+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.859841+0000) 2019-10-13T10:24:17.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:17.857+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.860156+0000) 2019-10-13T10:24:17.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:17.857+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.860490+0000) 2019-10-13T10:24:17.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:17.857+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.861388+0000) 2019-10-13T10:24:17.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:17.859+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.861378+0000) 2019-10-13T10:24:17.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:17.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:17.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:17.863+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:17.865817+0000) 2019-10-13T10:24:18.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:18.485+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:18.488946+0000) 2019-10-13T10:24:19.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:19.907+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:19.908301+0000) 2019-10-13T10:24:21.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:21.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:21.193551+0000) 2019-10-13T10:24:21.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:21.735+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:21.739791+0000) 2019-10-13T10:24:22.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:22.865+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.867310+0000) 2019-10-13T10:24:22.869 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:22.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.867978+0000) 2019-10-13T10:24:22.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:22.865+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.868303+0000) 2019-10-13T10:24:22.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:22.867+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.868702+0000) 2019-10-13T10:24:22.871 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:22.865+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.868834+0000) 2019-10-13T10:24:22.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:22.865+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.869592+0000) 2019-10-13T10:24:22.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:22.869+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.870017+0000) 2019-10-13T10:24:22.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:22.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:22.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:22.875+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:22.877134+0000) 2019-10-13T10:24:24.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:24.057+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:24.058727+0000) 2019-10-13T10:24:24.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:24.345+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:24.345668+0000) 2019-10-13T10:24:24.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:24.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:24.467126+0000) 2019-10-13T10:24:27.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:27.712+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.717342+0000) 2019-10-13T10:24:27.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:27.872+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.875539+0000) 2019-10-13T10:24:27.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:27.872+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.875920+0000) 2019-10-13T10:24:27.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:27.872+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.876290+0000) 2019-10-13T10:24:27.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:27.875+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.876964+0000) 2019-10-13T10:24:27.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:27.876+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.878061+0000) 2019-10-13T10:24:27.879 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:27.876+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.878253+0000) 2019-10-13T10:24:27.880 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:27.876+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.878892+0000) 2019-10-13T10:24:27.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:27.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:27.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:27.883+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:27.885900+0000) 2019-10-13T10:24:28.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:28.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:28.489117+0000) 2019-10-13T10:24:29.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:29.911+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:29.914990+0000) 2019-10-13T10:24:31.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:31.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:31.193805+0000) 2019-10-13T10:24:31.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:31.735+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:31.739979+0000) 2019-10-13T10:24:32.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:32.884+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.886210+0000) 2019-10-13T10:24:32.888 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:32.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.886622+0000) 2019-10-13T10:24:32.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:32.884+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.886992+0000) 2019-10-13T10:24:32.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:32.884+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.887367+0000) 2019-10-13T10:24:32.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:32.883+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.887535+0000) 2019-10-13T10:24:32.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:32.884+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.888875+0000) 2019-10-13T10:24:32.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:32.884+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.889041+0000) 2019-10-13T10:24:32.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:32.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:32.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:32.891+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:32.894170+0000) 2019-10-13T10:24:34.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:34.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:34.058952+0000) 2019-10-13T10:24:34.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:34.344+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:34.345901+0000) 2019-10-13T10:24:34.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:34.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:34.467380+0000) 2019-10-13T10:24:37.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:37.712+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.717571+0000) 2019-10-13T10:24:37.833 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:24:37.833 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:37.837 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:24:37.837 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:24:37.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:37.892+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.894881+0000) 2019-10-13T10:24:37.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:37.892+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.895048+0000) 2019-10-13T10:24:37.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:37.892+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.897519+0000) 2019-10-13T10:24:37.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:37.896+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.897906+0000) 2019-10-13T10:24:37.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:37.895+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.898176+0000) 2019-10-13T10:24:37.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:37.896+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.898859+0000) 2019-10-13T10:24:37.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:37.896+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.900584+0000) 2019-10-13T10:24:37.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:37.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:37.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:37.903+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:37.905693+0000) 2019-10-13T10:24:38.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:38.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:38.489323+0000) 2019-10-13T10:24:39.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:39.919+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:39.921506+0000) 2019-10-13T10:24:41.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:41.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:41.194006+0000) 2019-10-13T10:24:41.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:41.739+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:41.740178+0000) 2019-10-13T10:24:42.905 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:42.900+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.903994+0000) 2019-10-13T10:24:42.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:42.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.904797+0000) 2019-10-13T10:24:42.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:42.900+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.904974+0000) 2019-10-13T10:24:42.906 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:42.900+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.905614+0000) 2019-10-13T10:24:42.907 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:42.904+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.906078+0000) 2019-10-13T10:24:42.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:42.903+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.905900+0000) 2019-10-13T10:24:42.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:42.904+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.906492+0000) 2019-10-13T10:24:42.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:42.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:42.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:42.907+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:42.911391+0000) 2019-10-13T10:24:44.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:44.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:44.059159+0000) 2019-10-13T10:24:44.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:44.344+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:44.346104+0000) 2019-10-13T10:24:44.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:44.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:44.467607+0000) 2019-10-13T10:24:47.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:47.716+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.717790+0000) 2019-10-13T10:24:47.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:47.908+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.913213+0000) 2019-10-13T10:24:47.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:47.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.913844+0000) 2019-10-13T10:24:47.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:47.912+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.914346+0000) 2019-10-13T10:24:47.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:47.912+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.914804+0000) 2019-10-13T10:24:47.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:47.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.915274+0000) 2019-10-13T10:24:47.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:47.912+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.916086+0000) 2019-10-13T10:24:47.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:47.912+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.916561+0000) 2019-10-13T10:24:47.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:47.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116095 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:47.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:47.919+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:47.923909+0000) 2019-10-13T10:24:48.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:48.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:48.489493+0000) 2019-10-13T10:24:49.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:49.927+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:49.928137+0000) 2019-10-13T10:24:51.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:51.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:51.194208+0000) 2019-10-13T10:24:51.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:51.739+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:51.740373+0000) 2019-10-13T10:24:52.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:52.919+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.921841+0000) 2019-10-13T10:24:52.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:52.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.923471+0000) 2019-10-13T10:24:52.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:52.920+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.923844+0000) 2019-10-13T10:24:52.925 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:52.920+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.924258+0000) 2019-10-13T10:24:52.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:52.920+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.925161+0000) 2019-10-13T10:24:52.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:52.920+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.925459+0000) 2019-10-13T10:24:52.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:52.924+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.926265+0000) 2019-10-13T10:24:52.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:52.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:52.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:52.927+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:52.928524+0000) 2019-10-13T10:24:54.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:54.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:54.059377+0000) 2019-10-13T10:24:54.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:54.344+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:54.346306+0000) 2019-10-13T10:24:54.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:54.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:54.467834+0000) 2019-10-13T10:24:57.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:57.716+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.717990+0000) 2019-10-13T10:24:57.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:24:57.928+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.931156+0000) 2019-10-13T10:24:57.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:24:57.928+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.931549+0000) 2019-10-13T10:24:57.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:24:57.928+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.931836+0000) 2019-10-13T10:24:57.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:24:57.928+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.932446+0000) 2019-10-13T10:24:57.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:24:57.931+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.932715+0000) 2019-10-13T10:24:57.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:57.932+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.934000+0000) 2019-10-13T10:24:57.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:24:57.932+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.935089+0000) 2019-10-13T10:24:57.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:24:57.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:24:57.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:57.939+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:57.940293+0000) 2019-10-13T10:24:58.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:24:58.484+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:58.489676+0000) 2019-10-13T10:24:59.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:24:59.931+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:24:59.934661+0000) 2019-10-13T10:25:01.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:01.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:01.194431+0000) 2019-10-13T10:25:01.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:01.739+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:01.740604+0000) 2019-10-13T10:25:02.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:02.936+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.939524+0000) 2019-10-13T10:25:02.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:02.936+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.939668+0000) 2019-10-13T10:25:02.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:02.936+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.940600+0000) 2019-10-13T10:25:02.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:02.936+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.941330+0000) 2019-10-13T10:25:02.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:02.936+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.941724+0000) 2019-10-13T10:25:02.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:02.940+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.942880+0000) 2019-10-13T10:25:02.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:02.939+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.943016+0000) 2019-10-13T10:25:02.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:02.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:02.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:02.947+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:02.948846+0000) 2019-10-13T10:25:04.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:04.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:04.059593+0000) 2019-10-13T10:25:04.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:04.344+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:04.346528+0000) 2019-10-13T10:25:04.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:04.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:04.468044+0000) 2019-10-13T10:25:07.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:07.716+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.718213+0000) 2019-10-13T10:25:07.947 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:25:07.948 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:07.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:07.944+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.947344+0000) 2019-10-13T10:25:07.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:07.944+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.948020+0000) 2019-10-13T10:25:07.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:07.944+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.948395+0000) 2019-10-13T10:25:07.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:07.944+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.949571+0000) 2019-10-13T10:25:07.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:07.948+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.949932+0000) 2019-10-13T10:25:07.954 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:25:07.954 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:07.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:07.948+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.950702+0000) 2019-10-13T10:25:07.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:07.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:07.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:07.947+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.950050+0000) 2019-10-13T10:25:07.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:07.951+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:07.954412+0000) 2019-10-13T10:25:08.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:08.488+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:08.489911+0000) 2019-10-13T10:25:09.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:09.939+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:09.941681+0000) 2019-10-13T10:25:11.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:11.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:11.194663+0000) 2019-10-13T10:25:11.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:11.739+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:11.740823+0000) 2019-10-13T10:25:12.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:12.952+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.955632+0000) 2019-10-13T10:25:12.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:12.952+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.956306+0000) 2019-10-13T10:25:12.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:12.955+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.956395+0000) 2019-10-13T10:25:12.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:12.952+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.957440+0000) 2019-10-13T10:25:12.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:12.956+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.957976+0000) 2019-10-13T10:25:12.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:12.956+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.958819+0000) 2019-10-13T10:25:12.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:12.956+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.959533+0000) 2019-10-13T10:25:12.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:12.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:12.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:12.963+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:12.964398+0000) 2019-10-13T10:25:14.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:14.056+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:14.059800+0000) 2019-10-13T10:25:14.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:14.344+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:14.346789+0000) 2019-10-13T10:25:14.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:14.464+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:14.468264+0000) 2019-10-13T10:25:17.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:17.716+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.718440+0000) 2019-10-13T10:25:17.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:17.960+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.964244+0000) 2019-10-13T10:25:17.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:17.964+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.965873+0000) 2019-10-13T10:25:17.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:17.964+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.966504+0000) 2019-10-13T10:25:17.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:17.964+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.968809+0000) 2019-10-13T10:25:17.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:17.967+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.968740+0000) 2019-10-13T10:25:17.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:17.964+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.969264+0000) 2019-10-13T10:25:17.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:17.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:17.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:17.971+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.974268+0000) 2019-10-13T10:25:17.994 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:17.964+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:17.968356+0000) 2019-10-13T10:25:18.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:18.488+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:18.490102+0000) 2019-10-13T10:25:19.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:19.946+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:19.948910+0000) 2019-10-13T10:25:21.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:21.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:21.194862+0000) 2019-10-13T10:25:21.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:21.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:21.741061+0000) 2019-10-13T10:25:22.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:22.971+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.973975+0000) 2019-10-13T10:25:22.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:22.971+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.974714+0000) 2019-10-13T10:25:22.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:22.971+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.975257+0000) 2019-10-13T10:25:22.977 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:22.971+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.975556+0000) 2019-10-13T10:25:22.977 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:22.971+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.975903+0000) 2019-10-13T10:25:22.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:22.971+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.976825+0000) 2019-10-13T10:25:22.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:22.974+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.977047+0000) 2019-10-13T10:25:22.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:22.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:22.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:22.978+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:22.983367+0000) 2019-10-13T10:25:24.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:24.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.059999+0000) 2019-10-13T10:25:24.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:24.343+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.346987+0000) 2019-10-13T10:25:24.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:24.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:24.468473+0000) 2019-10-13T10:25:27.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:27.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.718672+0000) 2019-10-13T10:25:27.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:27.979+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.981915+0000) 2019-10-13T10:25:27.983 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:27.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.982137+0000) 2019-10-13T10:25:27.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:27.979+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.983287+0000) 2019-10-13T10:25:27.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:27.978+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.983088+0000) 2019-10-13T10:25:27.985 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:27.979+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.983678+0000) 2019-10-13T10:25:27.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:27.979+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.983935+0000) 2019-10-13T10:25:27.985 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:27.979+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.984852+0000) 2019-10-13T10:25:27.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:27.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:27.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:27.990+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:27.993596+0000) 2019-10-13T10:25:28.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:28.487+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:28.490250+0000) 2019-10-13T10:25:29.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:29.954+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:29.956021+0000) 2019-10-13T10:25:31.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:31.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:31.195132+0000) 2019-10-13T10:25:31.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:31.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:31.741254+0000) 2019-10-13T10:25:32.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:32.983+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.988946+0000) 2019-10-13T10:25:32.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:32.987+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.990053+0000) 2019-10-13T10:25:32.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:32.987+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.990754+0000) 2019-10-13T10:25:32.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:32.987+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.990561+0000) 2019-10-13T10:25:32.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:32.990+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.992157+0000) 2019-10-13T10:25:32.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:32.987+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.992701+0000) 2019-10-13T10:25:32.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:32.987+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.993268+0000) 2019-10-13T10:25:32.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:32.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:32.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:32.994+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:32.997710+0000) 2019-10-13T10:25:34.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:34.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.060242+0000) 2019-10-13T10:25:34.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:34.343+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.347203+0000) 2019-10-13T10:25:34.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:34.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:34.468673+0000) 2019-10-13T10:25:37.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:37.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.718882+0000) 2019-10-13T10:25:37.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:37.991+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.997133+0000) 2019-10-13T10:25:37.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:37.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.997086+0000) 2019-10-13T10:25:37.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:37.994+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.998421+0000) 2019-10-13T10:25:38.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:37.995+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.999041+0000) 2019-10-13T10:25:38.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:37.995+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.999349+0000) 2019-10-13T10:25:38.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:37.995+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.999552+0000) 2019-10-13T10:25:38.001 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:37.995+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:37.999858+0000) 2019-10-13T10:25:38.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:37.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:38.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:37.998+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.003858+0000) 2019-10-13T10:25:38.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:25:38.062 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:38.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:25:38.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:25:38.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:38.487+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:38.490438+0000) 2019-10-13T10:25:39.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:39.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:39.963016+0000) 2019-10-13T10:25:41.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:41.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:41.195350+0000) 2019-10-13T10:25:41.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:41.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:41.741449+0000) 2019-10-13T10:25:43.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:42.999+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.004183+0000) 2019-10-13T10:25:43.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:42.999+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.004817+0000) 2019-10-13T10:25:43.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:43.002+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.004741+0000) 2019-10-13T10:25:43.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:43.003+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.006278+0000) 2019-10-13T10:25:43.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:43.003+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.006817+0000) 2019-10-13T10:25:43.008 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:43.003+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.007380+0000) 2019-10-13T10:25:43.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:43.003+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.007722+0000) 2019-10-13T10:25:43.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:43.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:43.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:43.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:43.012036+0000) 2019-10-13T10:25:44.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:44.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.060455+0000) 2019-10-13T10:25:44.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:44.343+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.347399+0000) 2019-10-13T10:25:44.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:44.463+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:44.468877+0000) 2019-10-13T10:25:47.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:47.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:47.719081+0000) 2019-10-13T10:25:48.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:48.011+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.013920+0000) 2019-10-13T10:25:48.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:48.011+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.015037+0000) 2019-10-13T10:25:48.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:48.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.017165+0000) 2019-10-13T10:25:48.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:48.014+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.017311+0000) 2019-10-13T10:25:48.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:48.015+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.018157+0000) 2019-10-13T10:25:48.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:48.015+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.018777+0000) 2019-10-13T10:25:48.020 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:48.015+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.019201+0000) 2019-10-13T10:25:48.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:48.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:48.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:48.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.022905+0000) 2019-10-13T10:25:48.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:48.487+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:48.490641+0000) 2019-10-13T10:25:49.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:49.966+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:49.970109+0000) 2019-10-13T10:25:51.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:51.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:51.195590+0000) 2019-10-13T10:25:51.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:51.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:51.741699+0000) 2019-10-13T10:25:53.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:53.019+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.023405+0000) 2019-10-13T10:25:53.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:53.019+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.025226+0000) 2019-10-13T10:25:53.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:53.019+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.025606+0000) 2019-10-13T10:25:53.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:53.023+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.026036+0000) 2019-10-13T10:25:53.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:53.023+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.028346+0000) 2019-10-13T10:25:53.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:53.026+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.028373+0000) 2019-10-13T10:25:53.030 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:53.023+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:53.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:53.023+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.029567+0000) 2019-10-13T10:25:53.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:53.030+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:53.034696+0000) 2019-10-13T10:25:54.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:54.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.060690+0000) 2019-10-13T10:25:54.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:54.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.347619+0000) 2019-10-13T10:25:54.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:54.463+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:54.469078+0000) 2019-10-13T10:25:57.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:57.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:57.719280+0000) 2019-10-13T10:25:58.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:25:58.027+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.032576+0000) 2019-10-13T10:25:58.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:58.027+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.033029+0000) 2019-10-13T10:25:58.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:25:58.027+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.033293+0000) 2019-10-13T10:25:58.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:25:58.030+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.033749+0000) 2019-10-13T10:25:58.036 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:25:58.031+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.034675+0000) 2019-10-13T10:25:58.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:25:58.031+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.034997+0000) 2019-10-13T10:25:58.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:25:58.031+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.035910+0000) 2019-10-13T10:25:58.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:25:58.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:25:58.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:58.038+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.040434+0000) 2019-10-13T10:25:58.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:25:58.487+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:58.490820+0000) 2019-10-13T10:25:59.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:25:59.974+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:25:59.977173+0000) 2019-10-13T10:26:01.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:01.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:01.195834+0000) 2019-10-13T10:26:01.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:01.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:01.741895+0000) 2019-10-13T10:26:03.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:03.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.041485+0000) 2019-10-13T10:26:03.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:03.039+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.042241+0000) 2019-10-13T10:26:03.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:03.039+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.042651+0000) 2019-10-13T10:26:03.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:03.039+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.043528+0000) 2019-10-13T10:26:03.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:03.039+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.043947+0000) 2019-10-13T10:26:03.045 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:03.042+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.044409+0000) 2019-10-13T10:26:03.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:03.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.045418+0000) 2019-10-13T10:26:03.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:03.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:03.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:03.050+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:03.052341+0000) 2019-10-13T10:26:04.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:04.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.060924+0000) 2019-10-13T10:26:04.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:04.343+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.347814+0000) 2019-10-13T10:26:04.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:04.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:04.469291+0000) 2019-10-13T10:26:07.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:07.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:07.719498+0000) 2019-10-13T10:26:08.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:08.047+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.050126+0000) 2019-10-13T10:26:08.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:08.047+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.051168+0000) 2019-10-13T10:26:08.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:08.047+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.051335+0000) 2019-10-13T10:26:08.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:08.047+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.052165+0000) 2019-10-13T10:26:08.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:08.047+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.052807+0000) 2019-10-13T10:26:08.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:08.050+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.052652+0000) 2019-10-13T10:26:08.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:08.047+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.053629+0000) 2019-10-13T10:26:08.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:08.051+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:08.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:08.058+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.061161+0000) 2019-10-13T10:26:08.158 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:26:08.158 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:08.162 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:26:08.162 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:08.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:08.487+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:08.490995+0000) 2019-10-13T10:26:09.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:09.982+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:09.984279+0000) 2019-10-13T10:26:11.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:11.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:11.196059+0000) 2019-10-13T10:26:11.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:11.738+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:11.742125+0000) 2019-10-13T10:26:13.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:13.055+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.060378+0000) 2019-10-13T10:26:13.062 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:13.058+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.060427+0000) 2019-10-13T10:26:13.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:13.059+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.062647+0000) 2019-10-13T10:26:13.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:13.059+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.063398+0000) 2019-10-13T10:26:13.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:13.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.063803+0000) 2019-10-13T10:26:13.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:13.059+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.064087+0000) 2019-10-13T10:26:13.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:13.059+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.064713+0000) 2019-10-13T10:26:13.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:13.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:13.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:13.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:13.070383+0000) 2019-10-13T10:26:14.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:14.055+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.061160+0000) 2019-10-13T10:26:14.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:14.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.348010+0000) 2019-10-13T10:26:14.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:14.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:14.469522+0000) 2019-10-13T10:26:17.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:17.715+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:17.719731+0000) 2019-10-13T10:26:18.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:18.063+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.068824+0000) 2019-10-13T10:26:18.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:18.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.070147+0000) 2019-10-13T10:26:18.071 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:18.067+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.070501+0000) 2019-10-13T10:26:18.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:18.067+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.071257+0000) 2019-10-13T10:26:18.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:18.066+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.071103+0000) 2019-10-13T10:26:18.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:18.067+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.071963+0000) 2019-10-13T10:26:18.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:18.067+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.072034+0000) 2019-10-13T10:26:18.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:18.070+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:18.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:18.070+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.075412+0000) 2019-10-13T10:26:18.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:18.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:18.491170+0000) 2019-10-13T10:26:19.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:19.986+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:19.991378+0000) 2019-10-13T10:26:21.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:21.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:21.196267+0000) 2019-10-13T10:26:21.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:21.742+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:21.742352+0000) 2019-10-13T10:26:23.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:23.074+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.078101+0000) 2019-10-13T10:26:23.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:23.074+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.077778+0000) 2019-10-13T10:26:23.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:23.074+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.078870+0000) 2019-10-13T10:26:23.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:23.074+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.079248+0000) 2019-10-13T10:26:23.081 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:23.074+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.080445+0000) 2019-10-13T10:26:23.081 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:23.078+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.080250+0000) 2019-10-13T10:26:23.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:23.074+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.081283+0000) 2019-10-13T10:26:23.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:23.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:23.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:23.082+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:23.085419+0000) 2019-10-13T10:26:24.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:24.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.061344+0000) 2019-10-13T10:26:24.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:24.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.348216+0000) 2019-10-13T10:26:24.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:24.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:24.469714+0000) 2019-10-13T10:26:27.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:27.718+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:27.719960+0000) 2019-10-13T10:26:28.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:28.082+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.087783+0000) 2019-10-13T10:26:28.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:28.082+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.088477+0000) 2019-10-13T10:26:28.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:28.086+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.090036+0000) 2019-10-13T10:26:28.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:28.086+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.090496+0000) 2019-10-13T10:26:28.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:28.086+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.091396+0000) 2019-10-13T10:26:28.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:28.086+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.091698+0000) 2019-10-13T10:26:28.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:28.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.091999+0000) 2019-10-13T10:26:28.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:28.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:28.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:28.093+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.099706+0000) 2019-10-13T10:26:28.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:28.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:28.491365+0000) 2019-10-13T10:26:30.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:29.993+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:29.998546+0000) 2019-10-13T10:26:31.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:31.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:31.196466+0000) 2019-10-13T10:26:31.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:31.741+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:31.742547+0000) 2019-10-13T10:26:33.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:33.093+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.096373+0000) 2019-10-13T10:26:33.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:33.090+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.097021+0000) 2019-10-13T10:26:33.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:33.094+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.097855+0000) 2019-10-13T10:26:33.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:33.094+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.098467+0000) 2019-10-13T10:26:33.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:33.094+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.100304+0000) 2019-10-13T10:26:33.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:33.094+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.101378+0000) 2019-10-13T10:26:33.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:33.098+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.102092+0000) 2019-10-13T10:26:33.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:33.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:33.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:33.101+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:33.106975+0000) 2019-10-13T10:26:34.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:34.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.061522+0000) 2019-10-13T10:26:34.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:34.342+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.348410+0000) 2019-10-13T10:26:34.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:34.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:34.469986+0000) 2019-10-13T10:26:37.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:37.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:37.720173+0000) 2019-10-13T10:26:38.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:38.102+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.106369+0000) 2019-10-13T10:26:38.109 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:38.102+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.108280+0000) 2019-10-13T10:26:38.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:38.106+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.109817+0000) 2019-10-13T10:26:38.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:38.106+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.110313+0000) 2019-10-13T10:26:38.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:38.105+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.110743+0000) 2019-10-13T10:26:38.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:38.106+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.111218+0000) 2019-10-13T10:26:38.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:38.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:38.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:38.106+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.111511+0000) 2019-10-13T10:26:38.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:38.113+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.116881+0000) 2019-10-13T10:26:38.269 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:26:38.269 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:38.275 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:26:38.276 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:26:38.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:38.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:38.491551+0000) 2019-10-13T10:26:40.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:40.005+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:40.008712+0000) 2019-10-13T10:26:41.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:41.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:41.196681+0000) 2019-10-13T10:26:41.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:41.741+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:41.742770+0000) 2019-10-13T10:26:43.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:43.110+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.115081+0000) 2019-10-13T10:26:43.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:43.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.115781+0000) 2019-10-13T10:26:43.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:43.110+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.116931+0000) 2019-10-13T10:26:43.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:43.110+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.117263+0000) 2019-10-13T10:26:43.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:43.110+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.117683+0000) 2019-10-13T10:26:43.120 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:43.113+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.116493+0000) 2019-10-13T10:26:43.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:43.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.118265+0000) 2019-10-13T10:26:43.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:43.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:43.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:43.117+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:43.122141+0000) 2019-10-13T10:26:44.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:44.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.061715+0000) 2019-10-13T10:26:44.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:44.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.348628+0000) 2019-10-13T10:26:44.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:44.466+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:44.470226+0000) 2019-10-13T10:26:47.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:47.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:47.720372+0000) 2019-10-13T10:26:48.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:48.118+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.124672+0000) 2019-10-13T10:26:48.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:48.118+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.125533+0000) 2019-10-13T10:26:48.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:48.122+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.126045+0000) 2019-10-13T10:26:48.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:48.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.126508+0000) 2019-10-13T10:26:48.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:48.122+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.127282+0000) 2019-10-13T10:26:48.128 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:48.122+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.127296+0000) 2019-10-13T10:26:48.129 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:48.121+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.127217+0000) 2019-10-13T10:26:48.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:48.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116837 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:48.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:48.129+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.133664+0000) 2019-10-13T10:26:48.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:48.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:48.491740+0000) 2019-10-13T10:26:50.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:50.009+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:50.013830+0000) 2019-10-13T10:26:51.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:51.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:51.196881+0000) 2019-10-13T10:26:51.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:51.737+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:51.743006+0000) 2019-10-13T10:26:53.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:53.126+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.133098+0000) 2019-10-13T10:26:53.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:53.126+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.133520+0000) 2019-10-13T10:26:53.137 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:53.129+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.135674+0000) 2019-10-13T10:26:53.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:53.130+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.136196+0000) 2019-10-13T10:26:53.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:53.130+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.137645+0000) 2019-10-13T10:26:53.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:53.134+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.138618+0000) 2019-10-13T10:26:53.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:53.134+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.138999+0000) 2019-10-13T10:26:53.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:53.134+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:53.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:53.137+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:53.140419+0000) 2019-10-13T10:26:54.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:54.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.061953+0000) 2019-10-13T10:26:54.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:54.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.348825+0000) 2019-10-13T10:26:54.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:54.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:54.470429+0000) 2019-10-13T10:26:57.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:57.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:57.720587+0000) 2019-10-13T10:26:58.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:58.138+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.142657+0000) 2019-10-13T10:26:58.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:26:58.138+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.142758+0000) 2019-10-13T10:26:58.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:26:58.138+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.144009+0000) 2019-10-13T10:26:58.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:26:58.138+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.144532+0000) 2019-10-13T10:26:58.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:26:58.138+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.145449+0000) 2019-10-13T10:26:58.146 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:26:58.141+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.145546+0000) 2019-10-13T10:26:58.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:26:58.142+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.146795+0000) 2019-10-13T10:26:58.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:26:58.142+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:26:58.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:26:58.145+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.151163+0000) 2019-10-13T10:26:58.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:26:58.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:26:58.491939+0000) 2019-10-13T10:27:00.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:00.013+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:00.018761+0000) 2019-10-13T10:27:01.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:01.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:01.197092+0000) 2019-10-13T10:27:01.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:01.737+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:01.743202+0000) 2019-10-13T10:27:03.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:03.150+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.153825+0000) 2019-10-13T10:27:03.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:03.146+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.153500+0000) 2019-10-13T10:27:03.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:03.150+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.154218+0000) 2019-10-13T10:27:03.156 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:03.150+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.155502+0000) 2019-10-13T10:27:03.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:03.150+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.156391+0000) 2019-10-13T10:27:03.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:03.153+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.156396+0000) 2019-10-13T10:27:03.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:03.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.163050+0000) 2019-10-13T10:27:03.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:03.150+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:03.157676+0000) 2019-10-13T10:27:03.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:03.154+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:04.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:04.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.062135+0000) 2019-10-13T10:27:04.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:04.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.349040+0000) 2019-10-13T10:27:04.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:04.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:04.470627+0000) 2019-10-13T10:27:07.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:07.714+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:07.720784+0000) 2019-10-13T10:27:08.162 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:08.154+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.161536+0000) 2019-10-13T10:27:08.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:08.158+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.162093+0000) 2019-10-13T10:27:08.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:08.158+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.164310+0000) 2019-10-13T10:27:08.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:08.158+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.164958+0000) 2019-10-13T10:27:08.166 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:08.158+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.165266+0000) 2019-10-13T10:27:08.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:08.162+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.166354+0000) 2019-10-13T10:27:08.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:08.161+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.166518+0000) 2019-10-13T10:27:08.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:08.162+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:08.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:08.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.173436+0000) 2019-10-13T10:27:08.347 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:27:08.347 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:08.351 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:27:08.351 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:08.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:08.486+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:08.492131+0000) 2019-10-13T10:27:10.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:10.017+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:10.022569+0000) 2019-10-13T10:27:11.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:11.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:11.197292+0000) 2019-10-13T10:27:11.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:11.741+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:11.743396+0000) 2019-10-13T10:27:13.173 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:13.166+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.171726+0000) 2019-10-13T10:27:13.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:13.166+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.172093+0000) 2019-10-13T10:27:13.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:13.166+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.172510+0000) 2019-10-13T10:27:13.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:13.166+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.173073+0000) 2019-10-13T10:27:13.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:13.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.173644+0000) 2019-10-13T10:27:13.176 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:13.169+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.174897+0000) 2019-10-13T10:27:13.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:13.170+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.175575+0000) 2019-10-13T10:27:13.177 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:13.170+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:13.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:13.177+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:13.180783+0000) 2019-10-13T10:27:14.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:14.058+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.062335+0000) 2019-10-13T10:27:14.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:14.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.349248+0000) 2019-10-13T10:27:14.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:14.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:14.470832+0000) 2019-10-13T10:27:17.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:17.713+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:17.720944+0000) 2019-10-13T10:27:18.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:18.173+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.179476+0000) 2019-10-13T10:27:18.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:18.173+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.179821+0000) 2019-10-13T10:27:18.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:18.177+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.180524+0000) 2019-10-13T10:27:18.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:18.173+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.181565+0000) 2019-10-13T10:27:18.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:18.173+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.181660+0000) 2019-10-13T10:27:18.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:18.177+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.182519+0000) 2019-10-13T10:27:18.184 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:18.177+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.183230+0000) 2019-10-13T10:27:18.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:18.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:18.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:18.185+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.188564+0000) 2019-10-13T10:27:18.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:18.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:18.492331+0000) 2019-10-13T10:27:20.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:20.025+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:20.029061+0000) 2019-10-13T10:27:21.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:21.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:21.197492+0000) 2019-10-13T10:27:21.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:21.741+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:21.743605+0000) 2019-10-13T10:27:23.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:23.181+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.188207+0000) 2019-10-13T10:27:23.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:23.181+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.189538+0000) 2019-10-13T10:27:23.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:23.185+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.189502+0000) 2019-10-13T10:27:23.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:23.185+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.191679+0000) 2019-10-13T10:27:23.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:23.185+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.191917+0000) 2019-10-13T10:27:23.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:23.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.192707+0000) 2019-10-13T10:27:23.194 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:23.185+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.193649+0000) 2019-10-13T10:27:23.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:23.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:23.195 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:23.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:23.194490+0000) 2019-10-13T10:27:24.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:24.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.062510+0000) 2019-10-13T10:27:24.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:24.345+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.349455+0000) 2019-10-13T10:27:24.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:24.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:24.471062+0000) 2019-10-13T10:27:27.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:27.717+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:27.721167+0000) 2019-10-13T10:27:28.200 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:28.193+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.198763+0000) 2019-10-13T10:27:28.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:28.193+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.200704+0000) 2019-10-13T10:27:28.203 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:28.197+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.202049+0000) 2019-10-13T10:27:28.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:28.193+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.201578+0000) 2019-10-13T10:27:28.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:28.197+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.202381+0000) 2019-10-13T10:27:28.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:28.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.202427+0000) 2019-10-13T10:27:28.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:28.197+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.203202+0000) 2019-10-13T10:27:28.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:28.197+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:28.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:28.205+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.209958+0000) 2019-10-13T10:27:28.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:28.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:28.492539+0000) 2019-10-13T10:27:30.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:30.029+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:30.035763+0000) 2019-10-13T10:27:31.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:31.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:31.197691+0000) 2019-10-13T10:27:31.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:31.740+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:31.743826+0000) 2019-10-13T10:27:33.207 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:33.205+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.206562+0000) 2019-10-13T10:27:33.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:33.205+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.207614+0000) 2019-10-13T10:27:33.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:33.205+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.207372+0000) 2019-10-13T10:27:33.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:33.205+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.208364+0000) 2019-10-13T10:27:33.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:33.205+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.208680+0000) 2019-10-13T10:27:33.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:33.205+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.209378+0000) 2019-10-13T10:27:33.211 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:33.204+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.209420+0000) 2019-10-13T10:27:33.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:33.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:33.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:33.208+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:33.214178+0000) 2019-10-13T10:27:34.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:34.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.062691+0000) 2019-10-13T10:27:34.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:34.345+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.349651+0000) 2019-10-13T10:27:34.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:34.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:34.471269+0000) 2019-10-13T10:27:37.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:37.717+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:37.721398+0000) 2019-10-13T10:27:38.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:38.213+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.215310+0000) 2019-10-13T10:27:38.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:38.208+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.215786+0000) 2019-10-13T10:27:38.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:38.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.215965+0000) 2019-10-13T10:27:38.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:38.213+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.216645+0000) 2019-10-13T10:27:38.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:38.213+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.216954+0000) 2019-10-13T10:27:38.220 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:38.217+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.219084+0000) 2019-10-13T10:27:38.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:38.217+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.219248+0000) 2019-10-13T10:27:38.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:38.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117148 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:38.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:38.220+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.226499+0000) 2019-10-13T10:27:38.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:38.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:38.492712+0000) 2019-10-13T10:27:38.677 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:27:38.678 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:38.681 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:27:38.682 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:27:40.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:40.040+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:40.042142+0000) 2019-10-13T10:27:41.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:41.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:41.197874+0000) 2019-10-13T10:27:41.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:41.736+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:41.744017+0000) 2019-10-13T10:27:43.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:43.221+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.225656+0000) 2019-10-13T10:27:43.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:43.225+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.226504+0000) 2019-10-13T10:27:43.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:43.225+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.227008+0000) 2019-10-13T10:27:43.229 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:43.220+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.226708+0000) 2019-10-13T10:27:43.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:43.225+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.227461+0000) 2019-10-13T10:27:43.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:43.225+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.228285+0000) 2019-10-13T10:27:43.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:43.225+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.228647+0000) 2019-10-13T10:27:43.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:43.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:43.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:43.224+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:43.231907+0000) 2019-10-13T10:27:44.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:44.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.062869+0000) 2019-10-13T10:27:44.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:44.349+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.349851+0000) 2019-10-13T10:27:44.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:44.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:44.471465+0000) 2019-10-13T10:27:47.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:47.717+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:47.721598+0000) 2019-10-13T10:27:48.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:48.228+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.232642+0000) 2019-10-13T10:27:48.234 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:48.229+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.233573+0000) 2019-10-13T10:27:48.235 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:48.233+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.233987+0000) 2019-10-13T10:27:48.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:48.233+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.235726+0000) 2019-10-13T10:27:48.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:48.233+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.237409+0000) 2019-10-13T10:27:48.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:48.233+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.237646+0000) 2019-10-13T10:27:48.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:48.237+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.238420+0000) 2019-10-13T10:27:48.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:48.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:48.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:48.240+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.244537+0000) 2019-10-13T10:27:48.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:48.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:48.492900+0000) 2019-10-13T10:27:50.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:50.048+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:50.053555+0000) 2019-10-13T10:27:51.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:51.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:51.198067+0000) 2019-10-13T10:27:51.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:51.740+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:51.744248+0000) 2019-10-13T10:27:53.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:53.241+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.242680+0000) 2019-10-13T10:27:53.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:53.241+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.244265+0000) 2019-10-13T10:27:53.245 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:53.241+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.244561+0000) 2019-10-13T10:27:53.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:53.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.246431+0000) 2019-10-13T10:27:53.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:53.245+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.247639+0000) 2019-10-13T10:27:53.249 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:53.240+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.247760+0000) 2019-10-13T10:27:53.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:53.245+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.248941+0000) 2019-10-13T10:27:53.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:53.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:53.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:53.248+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:53.255369+0000) 2019-10-13T10:27:54.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:54.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.063049+0000) 2019-10-13T10:27:54.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:54.349+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.350071+0000) 2019-10-13T10:27:54.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:54.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:54.471672+0000) 2019-10-13T10:27:57.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:57.721+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:57.721810+0000) 2019-10-13T10:27:58.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:27:58.249+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.252172+0000) 2019-10-13T10:27:58.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:27:58.248+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.252611+0000) 2019-10-13T10:27:58.254 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:27:58.249+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.253514+0000) 2019-10-13T10:27:58.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:27:58.253+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.253978+0000) 2019-10-13T10:27:58.256 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:27:58.253+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.254647+0000) 2019-10-13T10:27:58.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:27:58.253+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.255170+0000) 2019-10-13T10:27:58.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:58.253+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.256093+0000) 2019-10-13T10:27:58.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:27:58.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:27:58.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:27:58.256+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.261378+0000) 2019-10-13T10:27:58.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:27:58.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:27:58.493111+0000) 2019-10-13T10:28:00.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:00.052+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:00.058379+0000) 2019-10-13T10:28:01.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:01.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:01.198268+0000) 2019-10-13T10:28:01.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:01.740+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:01.744450+0000) 2019-10-13T10:28:03.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:03.261+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.261986+0000) 2019-10-13T10:28:03.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:03.261+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.265266+0000) 2019-10-13T10:28:03.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:03.265+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.266278+0000) 2019-10-13T10:28:03.267 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:03.260+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.266100+0000) 2019-10-13T10:28:03.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:03.265+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.266639+0000) 2019-10-13T10:28:03.268 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:03.265+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.266887+0000) 2019-10-13T10:28:03.268 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:03.265+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.267282+0000) 2019-10-13T10:28:03.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:03.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:03.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:03.264+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:03.268994+0000) 2019-10-13T10:28:04.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:04.061+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.063237+0000) 2019-10-13T10:28:04.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:04.349+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.350276+0000) 2019-10-13T10:28:04.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:04.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:04.471829+0000) 2019-10-13T10:28:07.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:07.721+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:07.722009+0000) 2019-10-13T10:28:08.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:08.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.271632+0000) 2019-10-13T10:28:08.274 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:08.269+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.272357+0000) 2019-10-13T10:28:08.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:08.269+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.272845+0000) 2019-10-13T10:28:08.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:08.269+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.273358+0000) 2019-10-13T10:28:08.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:08.269+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.273663+0000) 2019-10-13T10:28:08.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:08.273+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.274844+0000) 2019-10-13T10:28:08.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:08.273+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.275039+0000) 2019-10-13T10:28:08.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:08.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117333 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:08.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:08.272+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.280015+0000) 2019-10-13T10:28:08.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:08.489+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:08.493315+0000) 2019-10-13T10:28:08.730 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:28:08.730 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:08.734 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:28:08.734 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:10.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:10.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:10.065054+0000) 2019-10-13T10:28:11.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:11.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:11.198466+0000) 2019-10-13T10:28:11.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:11.744+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:11.744647+0000) 2019-10-13T10:28:13.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:13.276+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.280616+0000) 2019-10-13T10:28:13.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:13.276+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.280453+0000) 2019-10-13T10:28:13.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:13.276+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.281107+0000) 2019-10-13T10:28:13.283 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:13.276+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.282079+0000) 2019-10-13T10:28:13.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:13.280+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.284222+0000) 2019-10-13T10:28:13.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:13.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:13.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:13.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.286179+0000) 2019-10-13T10:28:13.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:13.284+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.286338+0000) 2019-10-13T10:28:13.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:13.284+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:13.288231+0000) 2019-10-13T10:28:14.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:14.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.063455+0000) 2019-10-13T10:28:14.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:14.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.350505+0000) 2019-10-13T10:28:14.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:14.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:14.472026+0000) 2019-10-13T10:28:17.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:17.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:17.722212+0000) 2019-10-13T10:28:18.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:18.284+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.289192+0000) 2019-10-13T10:28:18.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:18.284+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.288613+0000) 2019-10-13T10:28:18.291 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:18.288+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.290181+0000) 2019-10-13T10:28:18.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:18.288+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.293362+0000) 2019-10-13T10:28:18.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:18.292+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:18.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:18.292+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.295089+0000) 2019-10-13T10:28:18.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:18.292+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.296830+0000) 2019-10-13T10:28:18.300 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:18.292+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.296290+0000) 2019-10-13T10:28:18.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:18.296+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.302434+0000) 2019-10-13T10:28:18.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:18.488+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:18.493533+0000) 2019-10-13T10:28:20.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:20.068+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:20.071664+0000) 2019-10-13T10:28:21.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:21.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:21.198684+0000) 2019-10-13T10:28:21.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:21.744+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:21.744879+0000) 2019-10-13T10:28:23.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:23.292+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.298402+0000) 2019-10-13T10:28:23.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:23.296+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.299187+0000) 2019-10-13T10:28:23.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:23.296+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.299544+0000) 2019-10-13T10:28:23.301 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:23.296+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.300110+0000) 2019-10-13T10:28:23.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:23.300+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.301886+0000) 2019-10-13T10:28:23.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:23.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.302096+0000) 2019-10-13T10:28:23.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:23.300+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.302506+0000) 2019-10-13T10:28:23.304 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:23.300+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:23.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:23.304+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:23.309189+0000) 2019-10-13T10:28:24.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:24.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.063645+0000) 2019-10-13T10:28:24.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:24.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.350685+0000) 2019-10-13T10:28:24.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:24.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:24.472230+0000) 2019-10-13T10:28:27.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:27.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:27.722410+0000) 2019-10-13T10:28:28.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:28.304+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.307584+0000) 2019-10-13T10:28:28.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:28.304+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.309310+0000) 2019-10-13T10:28:28.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:28.304+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.309996+0000) 2019-10-13T10:28:28.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:28.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.310735+0000) 2019-10-13T10:28:28.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:28.308+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.310896+0000) 2019-10-13T10:28:28.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:28.308+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.311032+0000) 2019-10-13T10:28:28.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:28.308+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.311564+0000) 2019-10-13T10:28:28.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:28.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:28.322 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:28.312+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.317481+0000) 2019-10-13T10:28:28.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:28.488+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:28.493704+0000) 2019-10-13T10:28:30.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:30.076+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:30.078206+0000) 2019-10-13T10:28:31.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:31.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:31.198889+0000) 2019-10-13T10:28:31.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:31.744+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:31.745090+0000) 2019-10-13T10:28:33.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:33.312+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.315532+0000) 2019-10-13T10:28:33.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:33.312+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.315809+0000) 2019-10-13T10:28:33.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:33.312+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.317115+0000) 2019-10-13T10:28:33.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:33.316+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.318916+0000) 2019-10-13T10:28:33.320 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:33.316+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.318931+0000) 2019-10-13T10:28:33.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:33.316+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.319582+0000) 2019-10-13T10:28:33.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:33.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:33.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:33.316+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.321176+0000) 2019-10-13T10:28:33.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:33.324+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:33.325897+0000) 2019-10-13T10:28:34.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:34.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.063846+0000) 2019-10-13T10:28:34.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:34.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.350890+0000) 2019-10-13T10:28:34.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:34.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:34.472449+0000) 2019-10-13T10:28:37.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:37.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:37.722629+0000) 2019-10-13T10:28:38.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:38.320+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.324343+0000) 2019-10-13T10:28:38.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:38.323+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.325377+0000) 2019-10-13T10:28:38.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:38.324+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.326367+0000) 2019-10-13T10:28:38.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:38.324+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.326611+0000) 2019-10-13T10:28:38.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:38.324+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.326964+0000) 2019-10-13T10:28:38.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:38.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.327803+0000) 2019-10-13T10:28:38.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:38.324+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.327669+0000) 2019-10-13T10:28:38.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:38.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117516 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:38.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:38.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.336140+0000) 2019-10-13T10:28:38.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:38.492+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:38.493877+0000) 2019-10-13T10:28:38.860 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:28:38.860 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:38.864 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:28:38.864 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:28:40.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:40.083+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:40.084882+0000) 2019-10-13T10:28:41.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:41.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:41.199120+0000) 2019-10-13T10:28:41.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:41.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:41.745279+0000) 2019-10-13T10:28:43.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:43.332+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.333885+0000) 2019-10-13T10:28:43.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:43.332+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.334299+0000) 2019-10-13T10:28:43.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:43.332+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.336173+0000) 2019-10-13T10:28:43.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:43.332+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.336683+0000) 2019-10-13T10:28:43.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:43.335+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.338798+0000) 2019-10-13T10:28:43.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:43.336+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:43.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:43.336+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.339487+0000) 2019-10-13T10:28:43.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:43.336+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.340601+0000) 2019-10-13T10:28:43.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:43.343+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:43.346678+0000) 2019-10-13T10:28:44.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:44.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.064037+0000) 2019-10-13T10:28:44.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:44.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.351121+0000) 2019-10-13T10:28:44.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:44.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:44.472644+0000) 2019-10-13T10:28:47.724 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:47.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:47.722846+0000) 2019-10-13T10:28:48.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:48.340+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.343444+0000) 2019-10-13T10:28:48.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:48.340+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.343852+0000) 2019-10-13T10:28:48.345 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:48.340+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.344420+0000) 2019-10-13T10:28:48.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:48.340+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.345377+0000) 2019-10-13T10:28:48.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:48.344+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.345841+0000) 2019-10-13T10:28:48.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:48.343+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.346037+0000) 2019-10-13T10:28:48.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:48.344+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.346001+0000) 2019-10-13T10:28:48.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:48.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:48.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:48.347+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.351952+0000) 2019-10-13T10:28:48.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:48.492+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:48.494080+0000) 2019-10-13T10:28:50.092 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:50.087+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:50.091404+0000) 2019-10-13T10:28:51.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:51.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:51.199323+0000) 2019-10-13T10:28:51.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:51.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:51.745513+0000) 2019-10-13T10:28:53.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:53.348+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.351930+0000) 2019-10-13T10:28:53.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:53.348+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.352758+0000) 2019-10-13T10:28:53.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:53.351+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.354027+0000) 2019-10-13T10:28:53.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:53.352+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.355307+0000) 2019-10-13T10:28:53.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:53.352+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.356288+0000) 2019-10-13T10:28:53.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:53.352+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:53.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:53.352+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.357341+0000) 2019-10-13T10:28:53.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:53.356+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.357924+0000) 2019-10-13T10:28:53.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:53.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:53.360166+0000) 2019-10-13T10:28:54.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:54.060+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.064261+0000) 2019-10-13T10:28:54.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:54.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.351320+0000) 2019-10-13T10:28:54.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:54.468+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:54.472888+0000) 2019-10-13T10:28:57.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:57.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:57.723079+0000) 2019-10-13T10:28:58.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:28:58.356+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.360349+0000) 2019-10-13T10:28:58.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:58.356+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.361009+0000) 2019-10-13T10:28:58.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:28:58.360+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.363413+0000) 2019-10-13T10:28:58.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:28:58.360+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.363501+0000) 2019-10-13T10:28:58.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:28:58.363+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.366065+0000) 2019-10-13T10:28:58.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:28:58.364+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:28:58.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:28:58.364+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.367305+0000) 2019-10-13T10:28:58.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:28:58.364+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.367247+0000) 2019-10-13T10:28:58.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:28:58.367+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.369735+0000) 2019-10-13T10:28:58.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:28:58.492+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:28:58.494284+0000) 2019-10-13T10:29:00.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:00.091+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:00.095839+0000) 2019-10-13T10:29:01.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:01.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:01.199552+0000) 2019-10-13T10:29:01.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:01.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:01.745709+0000) 2019-10-13T10:29:03.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:03.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.369798+0000) 2019-10-13T10:29:03.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:03.368+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.370929+0000) 2019-10-13T10:29:03.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:03.368+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.372824+0000) 2019-10-13T10:29:03.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:03.372+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.373776+0000) 2019-10-13T10:29:03.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:03.372+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.375058+0000) 2019-10-13T10:29:03.376 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:03.372+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.375656+0000) 2019-10-13T10:29:03.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:03.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:03.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:03.375+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.376438+0000) 2019-10-13T10:29:03.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:03.372+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:03.376327+0000) 2019-10-13T10:29:04.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:04.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.064450+0000) 2019-10-13T10:29:04.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:04.348+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.351499+0000) 2019-10-13T10:29:04.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:04.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:04.473086+0000) 2019-10-13T10:29:07.724 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:07.720+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:07.723293+0000) 2019-10-13T10:29:08.381 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:08.375+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.379961+0000) 2019-10-13T10:29:08.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:08.376+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.380675+0000) 2019-10-13T10:29:08.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:08.376+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.381256+0000) 2019-10-13T10:29:08.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:08.376+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.381127+0000) 2019-10-13T10:29:08.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:08.380+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.381841+0000) 2019-10-13T10:29:08.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:08.380+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.382402+0000) 2019-10-13T10:29:08.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:08.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.383134+0000) 2019-10-13T10:29:08.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:08.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:08.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:08.383+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.385615+0000) 2019-10-13T10:29:08.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:08.492+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:08.494479+0000) 2019-10-13T10:29:08.982 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:29:08.982 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:08.986 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:29:08.986 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:10.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:10.099+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:10.101985+0000) 2019-10-13T10:29:11.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:11.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:11.199754+0000) 2019-10-13T10:29:11.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:11.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:11.745902+0000) 2019-10-13T10:29:13.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:13.383+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.389500+0000) 2019-10-13T10:29:13.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:13.387+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.389824+0000) 2019-10-13T10:29:13.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:13.387+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.390653+0000) 2019-10-13T10:29:13.392 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:13.387+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.391132+0000) 2019-10-13T10:29:13.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:13.387+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.391786+0000) 2019-10-13T10:29:13.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:13.387+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.392052+0000) 2019-10-13T10:29:13.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:13.387+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.392461+0000) 2019-10-13T10:29:13.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:13.391+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:13.395387+0000) 2019-10-13T10:29:13.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:13.391+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:14.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:14.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.064666+0000) 2019-10-13T10:29:14.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:14.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.351675+0000) 2019-10-13T10:29:14.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:14.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:14.473320+0000) 2019-10-13T10:29:17.724 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:17.719+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:17.723486+0000) 2019-10-13T10:29:18.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:18.391+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.397569+0000) 2019-10-13T10:29:18.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:18.395+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.398756+0000) 2019-10-13T10:29:18.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:18.395+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.399069+0000) 2019-10-13T10:29:18.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:18.395+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.399800+0000) 2019-10-13T10:29:18.401 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:18.395+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.399901+0000) 2019-10-13T10:29:18.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:18.395+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.400603+0000) 2019-10-13T10:29:18.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:18.395+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.401673+0000) 2019-10-13T10:29:18.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:18.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:18.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:18.403+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.407982+0000) 2019-10-13T10:29:18.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:18.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:18.494692+0000) 2019-10-13T10:29:20.110 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:20.107+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:20.108579+0000) 2019-10-13T10:29:21.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:21.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:21.199950+0000) 2019-10-13T10:29:21.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:21.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:21.746130+0000) 2019-10-13T10:29:23.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:23.403+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.407888+0000) 2019-10-13T10:29:23.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:23.403+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.408675+0000) 2019-10-13T10:29:23.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:23.403+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.409573+0000) 2019-10-13T10:29:23.410 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:23.403+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.409582+0000) 2019-10-13T10:29:23.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:23.407+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.409589+0000) 2019-10-13T10:29:23.411 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:23.407+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.410499+0000) 2019-10-13T10:29:23.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:23.407+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.410835+0000) 2019-10-13T10:29:23.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:23.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117790 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:23.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:23.415+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:23.416147+0000) 2019-10-13T10:29:24.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:24.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.064912+0000) 2019-10-13T10:29:24.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:24.351+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.351881+0000) 2019-10-13T10:29:24.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:24.467+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:24.473515+0000) 2019-10-13T10:29:27.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:27.719+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:27.723697+0000) 2019-10-13T10:29:28.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:28.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.416458+0000) 2019-10-13T10:29:28.419 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:28.411+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.417523+0000) 2019-10-13T10:29:28.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:28.415+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.418468+0000) 2019-10-13T10:29:28.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:28.415+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.418760+0000) 2019-10-13T10:29:28.421 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:28.415+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.419691+0000) 2019-10-13T10:29:28.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:28.415+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.420695+0000) 2019-10-13T10:29:28.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:28.415+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.420773+0000) 2019-10-13T10:29:28.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:28.419+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.422572+0000) 2019-10-13T10:29:28.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:28.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117820 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:28.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:28.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:28.494861+0000) 2019-10-13T10:29:30.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:30.111+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:30.115103+0000) 2019-10-13T10:29:31.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:31.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:31.200165+0000) 2019-10-13T10:29:31.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:31.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:31.746320+0000) 2019-10-13T10:29:33.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:33.423+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.426615+0000) 2019-10-13T10:29:33.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:33.423+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.427156+0000) 2019-10-13T10:29:33.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:33.423+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.427425+0000) 2019-10-13T10:29:33.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:33.423+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.428044+0000) 2019-10-13T10:29:33.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:33.423+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.428685+0000) 2019-10-13T10:29:33.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:33.427+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.430075+0000) 2019-10-13T10:29:33.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:33.427+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.430882+0000) 2019-10-13T10:29:33.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:33.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:33.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:33.431+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:33.433679+0000) 2019-10-13T10:29:34.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:34.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.065059+0000) 2019-10-13T10:29:34.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:34.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.352057+0000) 2019-10-13T10:29:34.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:34.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:34.473735+0000) 2019-10-13T10:29:37.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:37.719+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:37.723924+0000) 2019-10-13T10:29:38.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:38.431+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.436312+0000) 2019-10-13T10:29:38.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:38.435+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.438240+0000) 2019-10-13T10:29:38.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:38.435+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.438758+0000) 2019-10-13T10:29:38.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:38.435+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.439474+0000) 2019-10-13T10:29:38.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:38.435+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.440115+0000) 2019-10-13T10:29:38.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:38.435+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.441479+0000) 2019-10-13T10:29:38.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:38.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:38.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:38.439+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.441795+0000) 2019-10-13T10:29:38.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:38.447+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.448778+0000) 2019-10-13T10:29:38.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:38.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:38.495034+0000) 2019-10-13T10:29:39.026 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:29:39.027 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:39.030 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:29:39.031 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:29:40.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:40.119+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:40.121742+0000) 2019-10-13T10:29:41.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:41.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:41.200324+0000) 2019-10-13T10:29:41.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:41.743+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:41.746552+0000) 2019-10-13T10:29:43.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:43.442+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.444806+0000) 2019-10-13T10:29:43.446 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:43.443+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.445797+0000) 2019-10-13T10:29:43.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:43.443+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.446594+0000) 2019-10-13T10:29:43.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:43.443+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.447067+0000) 2019-10-13T10:29:43.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:43.443+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.447813+0000) 2019-10-13T10:29:43.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:43.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.447896+0000) 2019-10-13T10:29:43.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:43.443+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.448265+0000) 2019-10-13T10:29:43.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:43.447+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:43.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:43.454+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:43.457138+0000) 2019-10-13T10:29:44.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:44.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.065254+0000) 2019-10-13T10:29:44.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:44.351+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.352260+0000) 2019-10-13T10:29:44.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:44.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:44.473975+0000) 2019-10-13T10:29:47.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:47.719+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:47.724125+0000) 2019-10-13T10:29:48.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:48.447+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.453666+0000) 2019-10-13T10:29:48.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:48.451+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.456914+0000) 2019-10-13T10:29:48.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:48.451+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.457204+0000) 2019-10-13T10:29:48.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:48.455+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.458300+0000) 2019-10-13T10:29:48.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:48.455+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.458790+0000) 2019-10-13T10:29:48.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:48.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:48.461 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:48.458+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.460482+0000) 2019-10-13T10:29:48.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:48.455+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.461054+0000) 2019-10-13T10:29:48.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:48.462+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.467131+0000) 2019-10-13T10:29:48.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:48.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:48.495249+0000) 2019-10-13T10:29:50.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:50.126+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:50.128307+0000) 2019-10-13T10:29:51.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:51.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:51.200537+0000) 2019-10-13T10:29:51.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:51.742+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:51.746767+0000) 2019-10-13T10:29:53.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:53.459+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.463318+0000) 2019-10-13T10:29:53.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:53.459+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.463889+0000) 2019-10-13T10:29:53.465 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:53.459+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.464647+0000) 2019-10-13T10:29:53.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:53.459+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.465079+0000) 2019-10-13T10:29:53.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:53.462+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.465358+0000) 2019-10-13T10:29:53.467 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:53.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.466210+0000) 2019-10-13T10:29:53.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:53.463+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.466764+0000) 2019-10-13T10:29:53.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:53.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117974 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:53.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:53.470+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:53.472726+0000) 2019-10-13T10:29:54.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:54.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.065456+0000) 2019-10-13T10:29:54.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:54.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.352442+0000) 2019-10-13T10:29:54.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:54.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:54.474170+0000) 2019-10-13T10:29:57.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:57.723+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:57.724327+0000) 2019-10-13T10:29:58.474 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:29:58.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.472705+0000) 2019-10-13T10:29:58.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:29:58.467+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.473284+0000) 2019-10-13T10:29:58.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:29:58.471+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.473911+0000) 2019-10-13T10:29:58.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:29:58.470+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.473866+0000) 2019-10-13T10:29:58.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:29:58.471+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.476874+0000) 2019-10-13T10:29:58.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:58.475+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.478879+0000) 2019-10-13T10:29:58.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:29:58.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:29:58.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:29:58.475+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.480427+0000) 2019-10-13T10:29:58.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:29:58.478+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.480689+0000) 2019-10-13T10:29:58.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:29:58.491+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:29:58.495422+0000) 2019-10-13T10:30:00.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:00.130+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:00.134553+0000) 2019-10-13T10:30:01.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:01.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:01.200772+0000) 2019-10-13T10:30:01.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:01.742+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:01.746964+0000) 2019-10-13T10:30:03.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:03.479+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.482698+0000) 2019-10-13T10:30:03.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:03.479+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.484328+0000) 2019-10-13T10:30:03.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:03.482+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.484168+0000) 2019-10-13T10:30:03.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:03.479+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.485171+0000) 2019-10-13T10:30:03.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:03.483+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.485615+0000) 2019-10-13T10:30:03.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:03.483+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.486122+0000) 2019-10-13T10:30:03.488 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:03.483+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.487666+0000) 2019-10-13T10:30:03.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:03.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:03.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:03.490+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:03.492227+0000) 2019-10-13T10:30:04.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:04.059+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.065637+0000) 2019-10-13T10:30:04.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:04.347+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.352626+0000) 2019-10-13T10:30:04.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:04.471+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:04.474381+0000) 2019-10-13T10:30:07.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:07.718+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:07.724526+0000) 2019-10-13T10:30:08.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:08.486+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.493055+0000) 2019-10-13T10:30:08.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:08.486+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.493498+0000) 2019-10-13T10:30:08.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:08.490+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.493911+0000) 2019-10-13T10:30:08.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:08.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.494049+0000) 2019-10-13T10:30:08.495 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:08.490+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.494659+0000) 2019-10-13T10:30:08.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:08.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.495264+0000) 2019-10-13T10:30:08.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:08.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.495571+0000) 2019-10-13T10:30:08.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:08.490+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.495682+0000) 2019-10-13T10:30:08.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:08.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:08.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:08.498+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:08.503311+0000) 2019-10-13T10:30:09.160 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:30:09.160 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:09.164 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:30:09.164 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:10.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:10.138+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:10.141073+0000) 2019-10-13T10:30:11.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:11.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:11.200996+0000) 2019-10-13T10:30:11.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:11.746+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:11.747153+0000) 2019-10-13T10:30:13.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:13.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.501897+0000) 2019-10-13T10:30:13.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:13.498+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.502089+0000) 2019-10-13T10:30:13.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:13.498+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.503267+0000) 2019-10-13T10:30:13.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:13.498+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.503245+0000) 2019-10-13T10:30:13.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:13.498+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.504180+0000) 2019-10-13T10:30:13.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:13.502+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.506733+0000) 2019-10-13T10:30:13.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:13.502+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.507356+0000) 2019-10-13T10:30:13.508 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:13.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:13.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:13.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:13.512774+0000) 2019-10-13T10:30:14.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:14.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.065824+0000) 2019-10-13T10:30:14.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:14.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.352804+0000) 2019-10-13T10:30:14.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:14.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:14.474560+0000) 2019-10-13T10:30:17.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:17.722+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:17.724725+0000) 2019-10-13T10:30:18.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:18.494+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.495747+0000) 2019-10-13T10:30:18.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:18.506+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.511366+0000) 2019-10-13T10:30:18.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:18.506+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.511667+0000) 2019-10-13T10:30:18.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:18.506+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.512106+0000) 2019-10-13T10:30:18.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:18.506+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.512730+0000) 2019-10-13T10:30:18.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:18.506+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.513576+0000) 2019-10-13T10:30:18.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:18.510+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.514343+0000) 2019-10-13T10:30:18.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:18.510+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.514832+0000) 2019-10-13T10:30:18.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:18.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:18.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:18.514+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:18.519263+0000) 2019-10-13T10:30:20.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:20.146+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:20.147599+0000) 2019-10-13T10:30:21.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:21.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:21.201221+0000) 2019-10-13T10:30:21.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:21.746+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:21.747358+0000) 2019-10-13T10:30:23.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:23.514+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.519668+0000) 2019-10-13T10:30:23.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:23.514+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.520277+0000) 2019-10-13T10:30:23.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:23.514+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.520641+0000) 2019-10-13T10:30:23.522 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:23.514+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.521116+0000) 2019-10-13T10:30:23.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:23.514+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.521330+0000) 2019-10-13T10:30:23.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:23.518+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.521878+0000) 2019-10-13T10:30:23.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:23.518+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.523059+0000) 2019-10-13T10:30:23.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:23.522+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:23.525673+0000) 2019-10-13T10:30:23.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:23.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:24.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:24.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.066009+0000) 2019-10-13T10:30:24.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:24.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.352999+0000) 2019-10-13T10:30:24.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:24.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:24.474738+0000) 2019-10-13T10:30:27.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:27.718+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:27.724927+0000) 2019-10-13T10:30:28.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:28.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.495905+0000) 2019-10-13T10:30:28.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:28.526+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.530361+0000) 2019-10-13T10:30:28.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:28.526+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.530842+0000) 2019-10-13T10:30:28.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:28.526+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.531866+0000) 2019-10-13T10:30:28.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:28.526+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.532258+0000) 2019-10-13T10:30:28.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:28.530+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.532299+0000) 2019-10-13T10:30:28.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:28.526+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.532991+0000) 2019-10-13T10:30:28.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:28.530+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.533834+0000) 2019-10-13T10:30:28.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:28.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:28.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:28.538+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:28.540533+0000) 2019-10-13T10:30:30.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:30.150+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:30.154067+0000) 2019-10-13T10:30:31.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:31.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:31.201418+0000) 2019-10-13T10:30:31.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:31.746+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:31.747552+0000) 2019-10-13T10:30:33.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:33.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.540010+0000) 2019-10-13T10:30:33.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:33.534+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.540107+0000) 2019-10-13T10:30:33.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:33.534+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.541000+0000) 2019-10-13T10:30:33.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:33.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.541498+0000) 2019-10-13T10:30:33.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:33.538+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.541478+0000) 2019-10-13T10:30:33.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:33.538+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.542283+0000) 2019-10-13T10:30:33.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:33.538+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.542678+0000) 2019-10-13T10:30:33.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:33.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:33.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:33.546+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:33.551170+0000) 2019-10-13T10:30:34.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:34.062+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.066213+0000) 2019-10-13T10:30:34.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:34.350+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.353173+0000) 2019-10-13T10:30:34.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:34.470+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:34.474928+0000) 2019-10-13T10:30:37.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:37.718+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:37.725137+0000) 2019-10-13T10:30:38.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:38.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.496091+0000) 2019-10-13T10:30:38.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:38.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.548205+0000) 2019-10-13T10:30:38.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:38.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.550080+0000) 2019-10-13T10:30:38.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:38.546+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.550149+0000) 2019-10-13T10:30:38.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:38.546+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.551519+0000) 2019-10-13T10:30:38.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:38.546+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.551700+0000) 2019-10-13T10:30:38.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:38.546+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.552372+0000) 2019-10-13T10:30:38.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:38.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.552522+0000) 2019-10-13T10:30:38.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:38.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:38.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:38.554+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:38.557558+0000) 2019-10-13T10:30:39.207 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:30:39.207 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:39.211 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:30:39.211 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:30:40.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:40.158+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:40.160652+0000) 2019-10-13T10:30:41.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:41.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:41.201622+0000) 2019-10-13T10:30:41.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:41.746+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:41.747763+0000) 2019-10-13T10:30:43.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:43.554+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.557975+0000) 2019-10-13T10:30:43.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:43.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.560916+0000) 2019-10-13T10:30:43.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:43.558+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.560682+0000) 2019-10-13T10:30:43.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:43.554+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.561551+0000) 2019-10-13T10:30:43.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:43.558+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.561886+0000) 2019-10-13T10:30:43.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:43.558+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.562177+0000) 2019-10-13T10:30:43.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:43.558+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.562478+0000) 2019-10-13T10:30:43.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:43.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118277 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:43.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:43.566+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:43.570924+0000) 2019-10-13T10:30:44.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:44.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.066395+0000) 2019-10-13T10:30:44.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:44.350+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.353386+0000) 2019-10-13T10:30:44.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:44.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:44.475102+0000) 2019-10-13T10:30:47.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:47.718+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:47.725336+0000) 2019-10-13T10:30:48.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:48.490+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.496283+0000) 2019-10-13T10:30:48.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:48.562+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.566767+0000) 2019-10-13T10:30:48.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:48.562+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.566865+0000) 2019-10-13T10:30:48.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:48.562+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.567364+0000) 2019-10-13T10:30:48.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:48.562+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.567758+0000) 2019-10-13T10:30:48.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:48.566+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.569828+0000) 2019-10-13T10:30:48.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:48.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.570240+0000) 2019-10-13T10:30:48.571 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:48.566+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.570519+0000) 2019-10-13T10:30:48.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:48.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:48.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:48.570+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:48.574015+0000) 2019-10-13T10:30:50.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:50.165+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:50.167176+0000) 2019-10-13T10:30:51.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:51.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:51.201851+0000) 2019-10-13T10:30:51.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:51.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:51.747962+0000) 2019-10-13T10:30:53.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:53.570+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.575962+0000) 2019-10-13T10:30:53.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:53.570+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.576310+0000) 2019-10-13T10:30:53.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:53.570+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.577162+0000) 2019-10-13T10:30:53.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:53.570+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.577655+0000) 2019-10-13T10:30:53.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:53.573+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.578582+0000) 2019-10-13T10:30:53.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:53.574+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.578572+0000) 2019-10-13T10:30:53.582 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:53.574+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.579709+0000) 2019-10-13T10:30:53.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:53.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:30:53.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:53.581+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:53.585624+0000) 2019-10-13T10:30:54.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:54.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.066573+0000) 2019-10-13T10:30:54.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:54.346+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.353577+0000) 2019-10-13T10:30:54.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:54.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:54.475273+0000) 2019-10-13T10:30:57.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:57.722+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:57.725572+0000) 2019-10-13T10:30:58.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:58.494+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.496470+0000) 2019-10-13T10:30:58.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:30:58.582+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.585217+0000) 2019-10-13T10:30:58.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:30:58.582+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.585448+0000) 2019-10-13T10:30:58.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:30:58.582+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.587044+0000) 2019-10-13T10:30:58.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:30:58.582+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.587367+0000) 2019-10-13T10:30:58.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:30:58.585+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.588414+0000) 2019-10-13T10:30:58.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:30:58.582+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.589438+0000) 2019-10-13T10:30:58.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:30:58.582+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.589537+0000) 2019-10-13T10:30:58.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:30:58.585+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:30:58.590115+0000) 2019-10-13T10:30:58.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:30:58.586+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:00.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:00.169+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:00.173801+0000) 2019-10-13T10:31:01.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:01.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:01.202124+0000) 2019-10-13T10:31:01.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:01.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:01.748160+0000) 2019-10-13T10:31:03.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:03.589+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.593599+0000) 2019-10-13T10:31:03.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:03.589+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.595793+0000) 2019-10-13T10:31:03.597 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:03.589+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.596253+0000) 2019-10-13T10:31:03.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:03.589+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.596060+0000) 2019-10-13T10:31:03.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:03.589+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.596915+0000) 2019-10-13T10:31:03.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:03.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.597578+0000) 2019-10-13T10:31:03.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:03.589+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.597124+0000) 2019-10-13T10:31:03.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:03.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:03.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:03.601+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:03.604487+0000) 2019-10-13T10:31:04.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:04.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.066790+0000) 2019-10-13T10:31:04.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:04.349+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.353778+0000) 2019-10-13T10:31:04.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:04.469+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:04.475458+0000) 2019-10-13T10:31:07.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:07.721+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:07.725809+0000) 2019-10-13T10:31:08.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:08.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.496661+0000) 2019-10-13T10:31:08.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:08.597+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.604231+0000) 2019-10-13T10:31:08.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:08.597+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.604495+0000) 2019-10-13T10:31:08.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:08.601+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.604784+0000) 2019-10-13T10:31:08.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:08.597+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.605364+0000) 2019-10-13T10:31:08.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:08.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.606144+0000) 2019-10-13T10:31:08.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:08.601+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.606685+0000) 2019-10-13T10:31:08.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:08.601+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.607274+0000) 2019-10-13T10:31:08.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:08.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:08.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:08.605+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:08.611500+0000) 2019-10-13T10:31:09.362 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:31:09.362 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:09.366 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:31:09.366 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:10.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:10.177+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:10.180335+0000) 2019-10-13T10:31:11.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:11.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:11.202332+0000) 2019-10-13T10:31:11.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:11.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:11.748353+0000) 2019-10-13T10:31:13.614 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:13.605+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.612637+0000) 2019-10-13T10:31:13.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:13.605+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.613642+0000) 2019-10-13T10:31:13.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:13.609+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.614354+0000) 2019-10-13T10:31:13.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:13.609+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.615106+0000) 2019-10-13T10:31:13.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:13.609+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.615523+0000) 2019-10-13T10:31:13.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:13.609+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.615773+0000) 2019-10-13T10:31:13.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:13.609+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.616536+0000) 2019-10-13T10:31:13.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:13.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:13.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:13.617+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:13.620693+0000) 2019-10-13T10:31:14.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:14.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.066971+0000) 2019-10-13T10:31:14.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:14.353+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.353982+0000) 2019-10-13T10:31:14.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:14.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:14.475639+0000) 2019-10-13T10:31:17.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:17.725+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:17.726010+0000) 2019-10-13T10:31:18.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:18.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.496853+0000) 2019-10-13T10:31:18.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:18.621+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.623078+0000) 2019-10-13T10:31:18.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:18.621+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.623694+0000) 2019-10-13T10:31:18.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:18.621+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.624070+0000) 2019-10-13T10:31:18.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:18.621+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.625560+0000) 2019-10-13T10:31:18.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:18.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.625802+0000) 2019-10-13T10:31:18.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:18.621+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.627476+0000) 2019-10-13T10:31:18.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:18.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:18.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:18.625+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.628307+0000) 2019-10-13T10:31:18.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:18.629+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:18.634370+0000) 2019-10-13T10:31:20.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:20.185+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:20.186887+0000) 2019-10-13T10:31:21.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:21.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:21.202568+0000) 2019-10-13T10:31:21.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:21.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:21.748555+0000) 2019-10-13T10:31:23.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:23.629+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.631333+0000) 2019-10-13T10:31:23.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:23.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.632034+0000) 2019-10-13T10:31:23.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:23.629+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.632040+0000) 2019-10-13T10:31:23.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:23.629+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.632706+0000) 2019-10-13T10:31:23.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:23.629+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.633582+0000) 2019-10-13T10:31:23.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:23.629+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.634536+0000) 2019-10-13T10:31:23.636 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:23.633+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.635316+0000) 2019-10-13T10:31:23.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:23.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:23.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:23.641+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:23.644528+0000) 2019-10-13T10:31:24.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:24.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.067155+0000) 2019-10-13T10:31:24.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:24.353+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.354168+0000) 2019-10-13T10:31:24.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:24.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:24.475814+0000) 2019-10-13T10:31:27.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:27.725+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:27.726233+0000) 2019-10-13T10:31:28.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:28.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.497053+0000) 2019-10-13T10:31:28.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:28.637+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.641031+0000) 2019-10-13T10:31:28.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:28.641+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.642218+0000) 2019-10-13T10:31:28.643 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:28.637+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.641425+0000) 2019-10-13T10:31:28.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:28.637+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.643221+0000) 2019-10-13T10:31:28.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:28.641+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.644182+0000) 2019-10-13T10:31:28.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:28.641+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.644307+0000) 2019-10-13T10:31:28.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:28.641+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.644805+0000) 2019-10-13T10:31:28.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:28.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:28.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:28.645+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:28.648940+0000) 2019-10-13T10:31:30.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:30.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:30.193288+0000) 2019-10-13T10:31:31.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:31.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:31.202766+0000) 2019-10-13T10:31:31.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:31.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:31.748753+0000) 2019-10-13T10:31:33.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:33.645+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.650000+0000) 2019-10-13T10:31:33.652 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:33.649+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.650629+0000) 2019-10-13T10:31:33.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:33.649+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.651192+0000) 2019-10-13T10:31:33.652 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:33.649+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.651733+0000) 2019-10-13T10:31:33.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:33.649+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.652213+0000) 2019-10-13T10:31:33.653 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:33.649+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.652535+0000) 2019-10-13T10:31:33.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:33.653+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.653817+0000) 2019-10-13T10:31:33.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:33.653+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:33.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:33.653+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:33.657942+0000) 2019-10-13T10:31:34.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:34.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.067376+0000) 2019-10-13T10:31:34.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:34.353+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.354360+0000) 2019-10-13T10:31:34.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:34.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:34.475991+0000) 2019-10-13T10:31:37.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:37.725+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:37.726440+0000) 2019-10-13T10:31:38.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:38.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.497250+0000) 2019-10-13T10:31:38.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:38.657+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.658392+0000) 2019-10-13T10:31:38.660 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:38.657+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.659339+0000) 2019-10-13T10:31:38.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:38.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.658970+0000) 2019-10-13T10:31:38.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:38.657+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.660494+0000) 2019-10-13T10:31:38.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:38.657+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.660989+0000) 2019-10-13T10:31:38.662 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:38.657+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.660744+0000) 2019-10-13T10:31:38.662 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:38.661+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.661807+0000) 2019-10-13T10:31:38.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:38.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:38.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:38.665+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:38.669434+0000) 2019-10-13T10:31:39.419 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:31:39.419 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:39.423 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:31:39.423 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:31:40.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:40.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:40.199926+0000) 2019-10-13T10:31:41.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:41.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:41.202979+0000) 2019-10-13T10:31:41.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:41.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:41.748957+0000) 2019-10-13T10:31:43.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:43.665+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.667975+0000) 2019-10-13T10:31:43.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:43.669+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.670187+0000) 2019-10-13T10:31:43.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:43.669+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.670810+0000) 2019-10-13T10:31:43.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:43.669+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.671363+0000) 2019-10-13T10:31:43.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:43.669+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.672215+0000) 2019-10-13T10:31:43.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:43.669+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.672905+0000) 2019-10-13T10:31:43.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:43.669+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.673260+0000) 2019-10-13T10:31:43.675 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:43.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:43.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:43.673+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:43.677831+0000) 2019-10-13T10:31:44.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:44.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.067561+0000) 2019-10-13T10:31:44.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:44.353+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.354540+0000) 2019-10-13T10:31:44.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:44.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:44.476167+0000) 2019-10-13T10:31:47.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:47.725+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:47.726639+0000) 2019-10-13T10:31:48.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:48.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.497446+0000) 2019-10-13T10:31:48.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:48.677+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.677901+0000) 2019-10-13T10:31:48.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:48.677+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.678275+0000) 2019-10-13T10:31:48.679 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:48.677+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.678687+0000) 2019-10-13T10:31:48.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:48.677+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.679000+0000) 2019-10-13T10:31:48.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:48.673+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.679049+0000) 2019-10-13T10:31:48.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:48.677+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.679933+0000) 2019-10-13T10:31:48.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:48.681+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.682855+0000) 2019-10-13T10:31:48.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:48.681+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118669 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:48.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:48.685+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:48.688549+0000) 2019-10-13T10:31:50.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:50.201+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:50.206520+0000) 2019-10-13T10:31:51.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:51.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:51.203180+0000) 2019-10-13T10:31:51.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:51.745+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:51.749116+0000) 2019-10-13T10:31:53.688 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:53.685+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.686759+0000) 2019-10-13T10:31:53.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:53.685+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.687591+0000) 2019-10-13T10:31:53.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:53.685+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.688005+0000) 2019-10-13T10:31:53.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:53.685+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.688393+0000) 2019-10-13T10:31:53.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:53.685+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.688654+0000) 2019-10-13T10:31:53.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:53.685+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.688734+0000) 2019-10-13T10:31:53.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:53.689+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.690749+0000) 2019-10-13T10:31:53.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:53.689+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118699 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:53.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:53.689+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:53.695745+0000) 2019-10-13T10:31:54.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:54.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.067701+0000) 2019-10-13T10:31:54.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:54.353+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.354718+0000) 2019-10-13T10:31:54.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:54.473+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:54.476369+0000) 2019-10-13T10:31:57.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:57.725+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:57.726835+0000) 2019-10-13T10:31:58.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:58.493+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.497637+0000) 2019-10-13T10:31:58.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:31:58.697+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.698016+0000) 2019-10-13T10:31:58.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:31:58.697+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.698473+0000) 2019-10-13T10:31:58.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:31:58.697+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.700175+0000) 2019-10-13T10:31:58.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:31:58.697+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.700600+0000) 2019-10-13T10:31:58.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:31:58.697+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.701052+0000) 2019-10-13T10:31:58.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:31:58.696+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.701408+0000) 2019-10-13T10:31:58.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:31:58.701+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.701984+0000) 2019-10-13T10:31:58.704 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:31:58.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118728 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:31:58.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:31:58.704+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:31:58.708485+0000) 2019-10-13T10:32:00.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:00.212+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:00.212967+0000) 2019-10-13T10:32:01.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:01.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:01.203379+0000) 2019-10-13T10:32:01.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:01.744+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:01.749348+0000) 2019-10-13T10:32:03.708 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:03.704+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.707467+0000) 2019-10-13T10:32:03.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:03.700+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.707356+0000) 2019-10-13T10:32:03.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:03.704+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.708455+0000) 2019-10-13T10:32:03.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:03.704+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.708905+0000) 2019-10-13T10:32:03.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:03.704+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.709486+0000) 2019-10-13T10:32:03.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:03.704+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.709714+0000) 2019-10-13T10:32:03.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:03.708+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.710200+0000) 2019-10-13T10:32:03.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:03.708+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:03.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:03.708+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:03.712919+0000) 2019-10-13T10:32:04.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:04.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.067899+0000) 2019-10-13T10:32:04.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:04.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.354935+0000) 2019-10-13T10:32:04.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:04.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:04.476579+0000) 2019-10-13T10:32:07.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:07.724+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:07.727045+0000) 2019-10-13T10:32:08.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:08.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.497871+0000) 2019-10-13T10:32:08.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:08.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.715764+0000) 2019-10-13T10:32:08.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:08.708+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.715971+0000) 2019-10-13T10:32:08.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:08.716+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.718135+0000) 2019-10-13T10:32:08.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:08.716+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.719057+0000) 2019-10-13T10:32:08.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:08.716+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.719945+0000) 2019-10-13T10:32:08.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:08.716+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.720288+0000) 2019-10-13T10:32:08.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:08.716+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.720668+0000) 2019-10-13T10:32:08.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:08.720+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:08.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:08.720+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:08.724537+0000) 2019-10-13T10:32:09.500 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:32:09.500 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:09.504 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:32:09.504 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:10.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:10.216+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:10.219351+0000) 2019-10-13T10:32:11.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:11.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:11.203584+0000) 2019-10-13T10:32:11.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:11.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:11.749572+0000) 2019-10-13T10:32:13.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:13.724+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.726007+0000) 2019-10-13T10:32:13.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:13.720+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.726063+0000) 2019-10-13T10:32:13.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:13.724+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.726897+0000) 2019-10-13T10:32:13.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:13.724+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.727278+0000) 2019-10-13T10:32:13.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:13.724+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.727882+0000) 2019-10-13T10:32:13.729 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:13.724+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.727749+0000) 2019-10-13T10:32:13.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:13.724+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.729240+0000) 2019-10-13T10:32:13.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:13.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118818 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:13.736 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:13.728+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:13.735022+0000) 2019-10-13T10:32:14.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:14.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.068101+0000) 2019-10-13T10:32:14.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:14.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.355131+0000) 2019-10-13T10:32:14.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:14.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:14.476786+0000) 2019-10-13T10:32:17.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:17.724+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:17.727258+0000) 2019-10-13T10:32:18.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:18.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.498062+0000) 2019-10-13T10:32:18.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:18.732+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.736607+0000) 2019-10-13T10:32:18.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:18.732+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.737472+0000) 2019-10-13T10:32:18.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:18.736+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.737942+0000) 2019-10-13T10:32:18.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:18.736+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.740583+0000) 2019-10-13T10:32:18.741 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:18.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:18.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:18.736+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.740916+0000) 2019-10-13T10:32:18.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:18.740+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.741982+0000) 2019-10-13T10:32:18.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:18.736+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.743111+0000) 2019-10-13T10:32:18.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:18.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:18.742350+0000) 2019-10-13T10:32:20.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:20.224+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:20.225898+0000) 2019-10-13T10:32:21.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:21.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:21.203786+0000) 2019-10-13T10:32:21.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:21.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:21.749780+0000) 2019-10-13T10:32:23.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:23.740+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.744836+0000) 2019-10-13T10:32:23.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:23.740+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.745014+0000) 2019-10-13T10:32:23.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:23.740+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.745451+0000) 2019-10-13T10:32:23.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:23.744+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.746253+0000) 2019-10-13T10:32:23.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:23.744+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.746668+0000) 2019-10-13T10:32:23.748 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:23.744+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.747085+0000) 2019-10-13T10:32:23.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:23.744+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.747723+0000) 2019-10-13T10:32:23.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:23.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:23.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:23.744+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:23.750157+0000) 2019-10-13T10:32:24.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:24.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.068286+0000) 2019-10-13T10:32:24.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:24.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.355307+0000) 2019-10-13T10:32:24.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:24.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:24.476959+0000) 2019-10-13T10:32:27.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:27.724+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:27.727490+0000) 2019-10-13T10:32:28.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:28.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.498215+0000) 2019-10-13T10:32:28.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:28.748+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.752268+0000) 2019-10-13T10:32:28.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:28.748+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.753053+0000) 2019-10-13T10:32:28.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:28.748+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.753465+0000) 2019-10-13T10:32:28.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:28.752+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.753824+0000) 2019-10-13T10:32:28.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:28.752+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.754471+0000) 2019-10-13T10:32:28.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:28.752+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.755770+0000) 2019-10-13T10:32:28.758 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:28.752+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.757638+0000) 2019-10-13T10:32:28.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:28.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118907 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:28.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:28.756+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:28.760974+0000) 2019-10-13T10:32:30.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:30.228+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:30.232324+0000) 2019-10-13T10:32:31.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:31.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:31.204001+0000) 2019-10-13T10:32:31.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:31.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:31.749967+0000) 2019-10-13T10:32:33.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:33.760+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.762645+0000) 2019-10-13T10:32:33.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:33.760+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.763946+0000) 2019-10-13T10:32:33.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:33.756+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.763898+0000) 2019-10-13T10:32:33.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:33.760+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.764761+0000) 2019-10-13T10:32:33.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:33.760+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.765477+0000) 2019-10-13T10:32:33.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:33.764+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.765736+0000) 2019-10-13T10:32:33.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:33.764+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.766144+0000) 2019-10-13T10:32:33.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:33.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:33.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:33.764+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:33.771072+0000) 2019-10-13T10:32:34.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:34.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.068480+0000) 2019-10-13T10:32:34.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:34.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.355489+0000) 2019-10-13T10:32:34.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:34.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:34.477132+0000) 2019-10-13T10:32:37.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:37.724+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:37.727749+0000) 2019-10-13T10:32:38.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:38.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.498380+0000) 2019-10-13T10:32:38.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:38.764+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.770780+0000) 2019-10-13T10:32:38.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:38.768+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.772061+0000) 2019-10-13T10:32:38.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:38.768+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.772791+0000) 2019-10-13T10:32:38.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:38.772+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.774878+0000) 2019-10-13T10:32:38.778 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:38.772+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.777031+0000) 2019-10-13T10:32:38.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:38.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:38.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:38.776+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.779077+0000) 2019-10-13T10:32:38.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:38.776+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.779350+0000) 2019-10-13T10:32:38.782 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:38.776+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:38.781360+0000) 2019-10-13T10:32:39.618 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:32:39.619 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:39.622 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:32:39.622 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:32:40.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:40.236+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:40.243945+0000) 2019-10-13T10:32:41.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:41.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:41.204204+0000) 2019-10-13T10:32:41.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:41.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:41.750171+0000) 2019-10-13T10:32:43.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:43.776+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.781435+0000) 2019-10-13T10:32:43.783 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:43.780+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.781872+0000) 2019-10-13T10:32:43.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:43.776+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.782137+0000) 2019-10-13T10:32:43.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:43.780+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.782883+0000) 2019-10-13T10:32:43.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:43.780+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.783035+0000) 2019-10-13T10:32:43.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:43.780+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.783620+0000) 2019-10-13T10:32:43.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:43.780+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.783418+0000) 2019-10-13T10:32:43.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:43.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:43.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:43.784+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:43.790089+0000) 2019-10-13T10:32:44.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:44.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.068696+0000) 2019-10-13T10:32:44.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:44.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.355665+0000) 2019-10-13T10:32:44.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:44.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:44.477304+0000) 2019-10-13T10:32:47.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:47.724+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:47.727984+0000) 2019-10-13T10:32:48.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:48.496+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.498578+0000) 2019-10-13T10:32:48.791 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:48.788+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.790447+0000) 2019-10-13T10:32:48.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:48.788+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.790289+0000) 2019-10-13T10:32:48.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:48.784+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.790930+0000) 2019-10-13T10:32:48.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:48.788+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.791291+0000) 2019-10-13T10:32:48.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:48.788+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.791903+0000) 2019-10-13T10:32:48.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:48.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.794154+0000) 2019-10-13T10:32:48.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:48.792+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.794339+0000) 2019-10-13T10:32:48.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:48.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:48.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:48.792+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:48.798381+0000) 2019-10-13T10:32:50.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:50.248+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:50.250179+0000) 2019-10-13T10:32:51.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:51.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:51.204404+0000) 2019-10-13T10:32:51.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:51.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:51.750368+0000) 2019-10-13T10:32:53.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:53.796+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.798386+0000) 2019-10-13T10:32:53.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:53.796+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.799774+0000) 2019-10-13T10:32:53.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:53.796+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.801373+0000) 2019-10-13T10:32:53.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:53.800+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.801378+0000) 2019-10-13T10:32:53.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:53.800+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.803271+0000) 2019-10-13T10:32:53.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:53.800+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.803762+0000) 2019-10-13T10:32:53.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:53.800+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.804464+0000) 2019-10-13T10:32:53.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:53.800+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:53.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:53.804+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:53.810032+0000) 2019-10-13T10:32:54.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:54.064+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.068882+0000) 2019-10-13T10:32:54.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:54.352+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.355843+0000) 2019-10-13T10:32:54.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:54.472+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:54.477489+0000) 2019-10-13T10:32:57.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:57.723+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:57.728208+0000) 2019-10-13T10:32:58.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:58.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.498775+0000) 2019-10-13T10:32:58.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:32:58.803+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.808135+0000) 2019-10-13T10:32:58.810 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:32:58.803+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.808557+0000) 2019-10-13T10:32:58.810 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:32:58.803+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.809344+0000) 2019-10-13T10:32:58.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:32:58.803+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.809208+0000) 2019-10-13T10:32:58.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:32:58.808+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.810140+0000) 2019-10-13T10:32:58.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:32:58.807+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.810260+0000) 2019-10-13T10:32:58.812 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:32:58.807+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.811044+0000) 2019-10-13T10:32:58.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:32:58.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:32:58.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:32:58.816+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:32:58.816948+0000) 2019-10-13T10:33:00.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:00.256+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:00.256699+0000) 2019-10-13T10:33:01.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:01.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:01.204608+0000) 2019-10-13T10:33:01.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:01.747+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:01.750564+0000) 2019-10-13T10:33:03.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:03.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.816402+0000) 2019-10-13T10:33:03.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:03.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.817098+0000) 2019-10-13T10:33:03.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:03.811+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.817556+0000) 2019-10-13T10:33:03.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:03.815+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.817856+0000) 2019-10-13T10:33:03.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:03.815+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.819643+0000) 2019-10-13T10:33:03.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:03.815+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.820074+0000) 2019-10-13T10:33:03.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:03.819+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.820166+0000) 2019-10-13T10:33:03.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:03.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:03.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:03.823+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:03.827723+0000) 2019-10-13T10:33:04.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:04.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.069088+0000) 2019-10-13T10:33:04.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:04.351+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.356038+0000) 2019-10-13T10:33:04.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:04.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:04.477666+0000) 2019-10-13T10:33:07.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:07.727+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:07.728409+0000) 2019-10-13T10:33:08.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:08.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.499001+0000) 2019-10-13T10:33:08.826 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:08.819+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.825126+0000) 2019-10-13T10:33:08.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:08.823+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.828303+0000) 2019-10-13T10:33:08.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:08.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.828570+0000) 2019-10-13T10:33:08.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:08.827+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.829822+0000) 2019-10-13T10:33:08.831 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:08.827+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.830061+0000) 2019-10-13T10:33:08.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:08.827+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.830689+0000) 2019-10-13T10:33:08.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:08.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119147 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:08.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:08.827+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.831378+0000) 2019-10-13T10:33:08.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:08.831+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:08.833908+0000) 2019-10-13T10:33:09.693 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:33:09.693 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:09.697 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:33:09.697 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:10.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:10.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:10.263264+0000) 2019-10-13T10:33:11.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:11.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:11.204809+0000) 2019-10-13T10:33:11.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:11.747+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:11.750759+0000) 2019-10-13T10:33:13.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:13.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.834428+0000) 2019-10-13T10:33:13.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:13.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.834773+0000) 2019-10-13T10:33:13.838 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:13.831+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.836873+0000) 2019-10-13T10:33:13.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:13.831+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.837055+0000) 2019-10-13T10:33:13.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:13.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.839448+0000) 2019-10-13T10:33:13.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:13.835+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.839827+0000) 2019-10-13T10:33:13.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:13.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:13.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:13.839+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.840257+0000) 2019-10-13T10:33:13.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:13.843+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:13.846416+0000) 2019-10-13T10:33:14.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:14.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.069282+0000) 2019-10-13T10:33:14.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:14.351+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.356213+0000) 2019-10-13T10:33:14.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:14.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:14.477849+0000) 2019-10-13T10:33:17.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:17.723+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:17.728610+0000) 2019-10-13T10:33:18.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:18.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.499167+0000) 2019-10-13T10:33:18.844 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:18.839+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.843205+0000) 2019-10-13T10:33:18.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:18.839+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.844132+0000) 2019-10-13T10:33:18.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:18.843+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.846504+0000) 2019-10-13T10:33:18.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:18.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.847031+0000) 2019-10-13T10:33:18.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:18.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:18.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:18.847+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.848293+0000) 2019-10-13T10:33:18.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:18.843+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.848955+0000) 2019-10-13T10:33:18.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:18.843+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.849669+0000) 2019-10-13T10:33:18.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:18.851+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:18.855777+0000) 2019-10-13T10:33:20.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:20.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:20.269418+0000) 2019-10-13T10:33:21.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:21.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:21.205013+0000) 2019-10-13T10:33:21.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:21.747+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:21.750987+0000) 2019-10-13T10:33:23.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:23.847+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.851881+0000) 2019-10-13T10:33:23.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:23.847+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.852257+0000) 2019-10-13T10:33:23.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:23.847+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.853493+0000) 2019-10-13T10:33:23.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:23.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.854178+0000) 2019-10-13T10:33:23.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:23.851+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.854651+0000) 2019-10-13T10:33:23.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:23.851+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.855351+0000) 2019-10-13T10:33:23.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:23.851+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.855407+0000) 2019-10-13T10:33:23.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:23.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:23.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:23.859+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:23.860451+0000) 2019-10-13T10:33:24.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:24.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.069487+0000) 2019-10-13T10:33:24.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:24.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.356396+0000) 2019-10-13T10:33:24.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:24.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:24.478047+0000) 2019-10-13T10:33:27.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:27.727+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:27.728852+0000) 2019-10-13T10:33:28.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:28.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.499315+0000) 2019-10-13T10:33:28.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:28.855+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.861126+0000) 2019-10-13T10:33:28.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:28.855+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.861513+0000) 2019-10-13T10:33:28.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:28.859+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.861790+0000) 2019-10-13T10:33:28.863 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:28.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.862115+0000) 2019-10-13T10:33:28.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:28.859+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.863428+0000) 2019-10-13T10:33:28.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:28.859+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.863536+0000) 2019-10-13T10:33:28.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:28.859+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.863919+0000) 2019-10-13T10:33:28.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:28.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:28.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:28.863+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:28.866475+0000) 2019-10-13T10:33:30.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:30.275+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:30.275930+0000) 2019-10-13T10:33:31.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:31.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:31.205223+0000) 2019-10-13T10:33:31.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:31.747+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:31.751207+0000) 2019-10-13T10:33:33.871 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:33.867+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.869865+0000) 2019-10-13T10:33:33.871 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:33.867+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.870543+0000) 2019-10-13T10:33:33.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:33.867+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.871005+0000) 2019-10-13T10:33:33.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:33.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.873176+0000) 2019-10-13T10:33:33.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:33.867+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.873518+0000) 2019-10-13T10:33:33.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:33.871+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.874107+0000) 2019-10-13T10:33:33.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:33.871+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.874489+0000) 2019-10-13T10:33:33.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:33.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:33.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:33.875+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:33.878404+0000) 2019-10-13T10:33:34.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:34.063+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.069691+0000) 2019-10-13T10:33:34.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:34.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.356574+0000) 2019-10-13T10:33:34.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:34.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:34.478226+0000) 2019-10-13T10:33:37.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:37.727+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:37.729081+0000) 2019-10-13T10:33:38.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:38.495+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.499521+0000) 2019-10-13T10:33:38.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:38.875+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.879331+0000) 2019-10-13T10:33:38.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:38.875+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.879791+0000) 2019-10-13T10:33:38.881 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:38.875+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.880664+0000) 2019-10-13T10:33:38.882 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:38.875+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.881370+0000) 2019-10-13T10:33:38.883 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:38.875+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.881692+0000) 2019-10-13T10:33:38.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:38.879+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.882077+0000) 2019-10-13T10:33:38.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:38.879+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.882500+0000) 2019-10-13T10:33:38.886 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:38.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:38.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:38.883+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:38.886638+0000) 2019-10-13T10:33:39.805 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:33:39.806 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:39.809 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:33:39.809 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:33:40.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:40.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:40.282462+0000) 2019-10-13T10:33:41.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:41.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:41.205406+0000) 2019-10-13T10:33:41.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:41.747+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:41.751417+0000) 2019-10-13T10:33:43.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:43.883+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.888512+0000) 2019-10-13T10:33:43.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:43.883+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.888885+0000) 2019-10-13T10:33:43.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:43.883+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.889144+0000) 2019-10-13T10:33:43.891 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:43.883+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.889688+0000) 2019-10-13T10:33:43.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:43.887+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.891800+0000) 2019-10-13T10:33:43.893 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:43.887+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.892268+0000) 2019-10-13T10:33:43.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:43.891+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.892324+0000) 2019-10-13T10:33:43.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:43.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:43.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:43.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:43.896217+0000) 2019-10-13T10:33:44.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:44.067+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.069913+0000) 2019-10-13T10:33:44.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:44.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.356756+0000) 2019-10-13T10:33:44.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:44.475+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:44.478418+0000) 2019-10-13T10:33:47.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:47.723+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:47.729280+0000) 2019-10-13T10:33:48.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:48.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.499755+0000) 2019-10-13T10:33:48.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:48.895+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.898126+0000) 2019-10-13T10:33:48.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:48.895+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.898586+0000) 2019-10-13T10:33:48.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:48.895+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.898998+0000) 2019-10-13T10:33:48.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:48.895+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.899797+0000) 2019-10-13T10:33:48.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:48.895+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.900319+0000) 2019-10-13T10:33:48.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:48.895+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.900484+0000) 2019-10-13T10:33:48.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:48.895+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.901519+0000) 2019-10-13T10:33:48.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:48.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:48.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:48.907+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:48.908677+0000) 2019-10-13T10:33:50.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:50.287+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:50.289062+0000) 2019-10-13T10:33:51.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:51.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:51.205640+0000) 2019-10-13T10:33:51.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:51.751+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:51.751650+0000) 2019-10-13T10:33:53.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:53.902+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.906853+0000) 2019-10-13T10:33:53.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:53.902+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.908251+0000) 2019-10-13T10:33:53.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:53.902+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.908839+0000) 2019-10-13T10:33:53.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:53.902+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.909182+0000) 2019-10-13T10:33:53.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:53.907+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.909279+0000) 2019-10-13T10:33:53.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:53.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.910139+0000) 2019-10-13T10:33:53.912 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:53.906+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.911097+0000) 2019-10-13T10:33:53.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:53.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:33:53.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:53.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:53.912738+0000) 2019-10-13T10:33:54.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:54.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.070147+0000) 2019-10-13T10:33:54.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:54.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.356939+0000) 2019-10-13T10:33:54.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:54.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:54.478609+0000) 2019-10-13T10:33:57.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:57.722+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:57.729496+0000) 2019-10-13T10:33:58.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:58.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.499985+0000) 2019-10-13T10:33:58.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:33:58.910+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.916760+0000) 2019-10-13T10:33:58.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:33:58.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.916815+0000) 2019-10-13T10:33:58.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:33:58.910+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.917599+0000) 2019-10-13T10:33:58.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:33:58.914+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.917964+0000) 2019-10-13T10:33:58.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:33:58.914+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.918563+0000) 2019-10-13T10:33:58.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:33:58.915+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.919037+0000) 2019-10-13T10:33:58.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:33:58.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.919580+0000) 2019-10-13T10:33:58.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:33:58.919+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:33:58.921722+0000) 2019-10-13T10:33:58.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:33:58.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:00.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:00.291+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:00.295203+0000) 2019-10-13T10:34:01.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:01.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:01.205853+0000) 2019-10-13T10:34:01.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:01.751+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:01.751874+0000) 2019-10-13T10:34:03.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:03.922+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.926870+0000) 2019-10-13T10:34:03.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:03.923+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.926838+0000) 2019-10-13T10:34:03.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:03.922+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.927481+0000) 2019-10-13T10:34:03.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:03.922+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.927929+0000) 2019-10-13T10:34:03.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:03.922+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.928225+0000) 2019-10-13T10:34:03.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:03.922+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.929118+0000) 2019-10-13T10:34:03.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:03.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.929309+0000) 2019-10-13T10:34:03.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:03.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:03.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:03.931+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:03.932817+0000) 2019-10-13T10:34:04.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:04.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.070368+0000) 2019-10-13T10:34:04.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:04.350+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.357176+0000) 2019-10-13T10:34:04.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:04.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:04.478811+0000) 2019-10-13T10:34:07.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:07.722+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:07.729695+0000) 2019-10-13T10:34:08.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:08.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.500184+0000) 2019-10-13T10:34:08.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:08.930+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.934197+0000) 2019-10-13T10:34:08.935 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:08.930+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.934517+0000) 2019-10-13T10:34:08.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:08.930+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.934688+0000) 2019-10-13T10:34:08.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:08.930+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.936134+0000) 2019-10-13T10:34:08.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:08.930+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.936310+0000) 2019-10-13T10:34:08.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:08.930+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.937206+0000) 2019-10-13T10:34:08.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:08.934+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.938358+0000) 2019-10-13T10:34:08.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:08.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:08.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:08.938+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:08.943729+0000) 2019-10-13T10:34:09.935 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:34:09.936 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:09.939 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:34:09.940 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:10.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:10.298+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:10.301656+0000) 2019-10-13T10:34:11.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:11.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:11.206094+0000) 2019-10-13T10:34:11.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:11.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:11.752071+0000) 2019-10-13T10:34:13.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:13.938+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.944482+0000) 2019-10-13T10:34:13.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:13.938+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.945064+0000) 2019-10-13T10:34:13.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:13.942+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.947028+0000) 2019-10-13T10:34:13.948 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:13.942+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.947396+0000) 2019-10-13T10:34:13.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:13.942+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.947702+0000) 2019-10-13T10:34:13.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:13.946+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.948858+0000) 2019-10-13T10:34:13.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:13.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.950028+0000) 2019-10-13T10:34:13.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:13.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:13.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:13.950+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:13.954239+0000) 2019-10-13T10:34:14.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:14.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.070552+0000) 2019-10-13T10:34:14.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:14.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.357372+0000) 2019-10-13T10:34:14.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:14.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:14.478988+0000) 2019-10-13T10:34:17.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:17.726+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:17.729932+0000) 2019-10-13T10:34:18.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:18.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.500381+0000) 2019-10-13T10:34:18.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:18.946+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.953305+0000) 2019-10-13T10:34:18.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:18.946+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.953657+0000) 2019-10-13T10:34:18.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:18.950+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.954977+0000) 2019-10-13T10:34:18.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:18.950+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.955470+0000) 2019-10-13T10:34:18.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:18.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.955539+0000) 2019-10-13T10:34:18.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:18.950+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.955908+0000) 2019-10-13T10:34:18.957 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:18.950+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.955978+0000) 2019-10-13T10:34:18.960 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:18.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:18.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:18.958+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:18.962635+0000) 2019-10-13T10:34:20.309 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:20.306+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:20.308366+0000) 2019-10-13T10:34:21.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:21.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:21.206285+0000) 2019-10-13T10:34:21.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:21.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:21.752303+0000) 2019-10-13T10:34:23.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:23.958+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.962901+0000) 2019-10-13T10:34:23.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:23.958+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.963214+0000) 2019-10-13T10:34:23.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:23.958+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.963739+0000) 2019-10-13T10:34:23.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:23.958+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.964769+0000) 2019-10-13T10:34:23.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:23.962+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.966456+0000) 2019-10-13T10:34:23.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:23.966+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.968348+0000) 2019-10-13T10:34:23.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:23.962+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.969093+0000) 2019-10-13T10:34:23.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:23.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:23.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:23.970+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:23.974473+0000) 2019-10-13T10:34:24.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:24.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:24.070756+0000) 2019-10-13T10:34:24.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:24.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:24.357553+0000) 2019-10-13T10:34:24.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:24.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:24.479119+0000) 2019-10-13T10:34:27.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:27.726+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:27.730137+0000) 2019-10-13T10:34:28.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:28.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.500576+0000) 2019-10-13T10:34:28.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:28.970+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.972460+0000) 2019-10-13T10:34:28.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:28.970+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.974706+0000) 2019-10-13T10:34:28.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:28.970+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.975391+0000) 2019-10-13T10:34:28.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:28.970+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.975766+0000) 2019-10-13T10:34:28.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:28.970+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.976949+0000) 2019-10-13T10:34:28.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:28.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.977057+0000) 2019-10-13T10:34:28.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:28.974+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.978602+0000) 2019-10-13T10:34:28.980 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:28.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:28.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:28.978+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:28.982856+0000) 2019-10-13T10:34:30.316 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:30.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:30.314799+0000) 2019-10-13T10:34:31.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:31.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:31.206506+0000) 2019-10-13T10:34:31.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:31.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:31.752516+0000) 2019-10-13T10:34:33.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:33.978+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.981999+0000) 2019-10-13T10:34:33.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:33.978+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.984751+0000) 2019-10-13T10:34:33.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:33.978+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.984694+0000) 2019-10-13T10:34:33.987 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:33.982+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.986251+0000) 2019-10-13T10:34:33.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:33.982+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.986366+0000) 2019-10-13T10:34:33.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:33.982+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.987283+0000) 2019-10-13T10:34:33.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:33.982+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.987682+0000) 2019-10-13T10:34:33.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:33.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:33.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:33.986+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:33.988306+0000) 2019-10-13T10:34:34.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:34.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:34.070944+0000) 2019-10-13T10:34:34.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:34.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:34.357733+0000) 2019-10-13T10:34:34.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:34.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:34.479327+0000) 2019-10-13T10:34:37.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:37.726+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:37.730319+0000) 2019-10-13T10:34:38.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:38.494+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.500787+0000) 2019-10-13T10:34:38.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:38.986+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.991217+0000) 2019-10-13T10:34:38.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:38.986+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.992148+0000) 2019-10-13T10:34:38.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:38.986+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.992928+0000) 2019-10-13T10:34:38.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:38.986+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.993568+0000) 2019-10-13T10:34:38.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:38.990+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.995986+0000) 2019-10-13T10:34:38.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:38.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:38.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:38.990+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.997229+0000) 2019-10-13T10:34:38.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:38.994+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:38.997209+0000) 2019-10-13T10:34:39.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:38.998+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:39.002810+0000) 2019-10-13T10:34:40.066 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:34:40.066 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:40.070 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:34:40.070 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:34:40.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:40.318+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:40.321685+0000) 2019-10-13T10:34:41.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:41.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:41.206715+0000) 2019-10-13T10:34:41.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:41.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:41.752734+0000) 2019-10-13T10:34:44.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:43.994+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.000340+0000) 2019-10-13T10:34:44.001 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:43.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.000765+0000) 2019-10-13T10:34:44.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:43.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.001374+0000) 2019-10-13T10:34:44.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:43.998+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.001336+0000) 2019-10-13T10:34:44.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:43.998+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.002624+0000) 2019-10-13T10:34:44.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:43.998+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.003528+0000) 2019-10-13T10:34:44.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:43.998+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.004291+0000) 2019-10-13T10:34:44.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:44.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:44.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:44.006+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.009097+0000) 2019-10-13T10:34:44.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:44.066+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.071143+0000) 2019-10-13T10:34:44.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:44.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.357974+0000) 2019-10-13T10:34:44.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:44.474+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:44.479508+0000) 2019-10-13T10:34:47.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:47.730+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:47.730530+0000) 2019-10-13T10:34:48.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:48.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:48.500987+0000) 2019-10-13T10:34:49.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:49.006+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.008919+0000) 2019-10-13T10:34:49.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:49.006+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.009929+0000) 2019-10-13T10:34:49.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:49.006+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.010411+0000) 2019-10-13T10:34:49.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:49.006+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.010852+0000) 2019-10-13T10:34:49.012 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:49.006+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.011116+0000) 2019-10-13T10:34:49.012 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:49.006+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.011467+0000) 2019-10-13T10:34:49.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:49.006+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.012177+0000) 2019-10-13T10:34:49.016 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:49.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:49.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:49.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:49.020271+0000) 2019-10-13T10:34:50.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:50.326+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:50.329107+0000) 2019-10-13T10:34:51.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:51.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:51.206921+0000) 2019-10-13T10:34:51.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:51.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:51.752936+0000) 2019-10-13T10:34:54.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:54.013+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.018624+0000) 2019-10-13T10:34:54.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:54.014+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.019408+0000) 2019-10-13T10:34:54.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:54.013+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.020222+0000) 2019-10-13T10:34:54.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:54.013+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.020762+0000) 2019-10-13T10:34:54.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:54.013+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.021032+0000) 2019-10-13T10:34:54.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:54.017+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.022136+0000) 2019-10-13T10:34:54.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:54.017+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.022305+0000) 2019-10-13T10:34:54.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:54.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:54.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:54.022+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.024794+0000) 2019-10-13T10:34:54.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:54.065+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.071330+0000) 2019-10-13T10:34:54.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:54.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.358155+0000) 2019-10-13T10:34:54.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:54.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:54.479692+0000) 2019-10-13T10:34:57.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:57.729+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:57.730766+0000) 2019-10-13T10:34:58.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:58.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:58.501209+0000) 2019-10-13T10:34:59.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:34:59.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.028111+0000) 2019-10-13T10:34:59.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:34:59.021+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.028950+0000) 2019-10-13T10:34:59.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:34:59.026+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.028932+0000) 2019-10-13T10:34:59.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:34:59.025+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.031407+0000) 2019-10-13T10:34:59.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:34:59.025+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.031658+0000) 2019-10-13T10:34:59.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:34:59.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:34:59.035 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:34:59.029+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.033727+0000) 2019-10-13T10:34:59.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:34:59.029+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.034256+0000) 2019-10-13T10:34:59.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:34:59.034+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:34:59.039994+0000) 2019-10-13T10:35:00.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:00.334+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:00.336139+0000) 2019-10-13T10:35:01.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:01.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:01.207119+0000) 2019-10-13T10:35:01.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:01.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:01.753159+0000) 2019-10-13T10:35:04.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:04.033+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.037111+0000) 2019-10-13T10:35:04.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:04.033+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.037522+0000) 2019-10-13T10:35:04.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:04.033+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.039547+0000) 2019-10-13T10:35:04.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:04.034+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.039711+0000) 2019-10-13T10:35:04.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:04.033+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.040250+0000) 2019-10-13T10:35:04.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:04.033+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.040713+0000) 2019-10-13T10:35:04.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:04.033+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.041299+0000) 2019-10-13T10:35:04.044 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:04.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:04.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:04.042+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.045102+0000) 2019-10-13T10:35:04.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:04.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.071538+0000) 2019-10-13T10:35:04.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:04.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.358352+0000) 2019-10-13T10:35:04.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:04.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:04.479908+0000) 2019-10-13T10:35:07.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:07.729+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:07.730989+0000) 2019-10-13T10:35:08.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:08.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:08.501416+0000) 2019-10-13T10:35:09.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:09.045+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.048043+0000) 2019-10-13T10:35:09.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:09.045+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.049342+0000) 2019-10-13T10:35:09.050 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:09.049+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.049949+0000) 2019-10-13T10:35:09.051 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:09.046+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.050071+0000) 2019-10-13T10:35:09.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:09.049+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.050724+0000) 2019-10-13T10:35:09.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:09.049+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.051787+0000) 2019-10-13T10:35:09.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:09.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:09.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:09.049+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.052855+0000) 2019-10-13T10:35:09.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:09.058+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:09.061139+0000) 2019-10-13T10:35:10.113 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:35:10.113 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:10.117 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:35:10.117 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:10.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:10.342+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:10.347511+0000) 2019-10-13T10:35:11.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:11.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:11.207357+0000) 2019-10-13T10:35:11.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:11.750+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:11.753357+0000) 2019-10-13T10:35:14.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:14.053+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.055771+0000) 2019-10-13T10:35:14.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:14.053+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.056695+0000) 2019-10-13T10:35:14.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:14.053+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.056850+0000) 2019-10-13T10:35:14.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:14.053+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.057493+0000) 2019-10-13T10:35:14.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:14.057+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.058182+0000) 2019-10-13T10:35:14.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:14.057+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.058641+0000) 2019-10-13T10:35:14.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:14.057+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.058699+0000) 2019-10-13T10:35:14.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:14.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:14.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:14.057+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.062235+0000) 2019-10-13T10:35:14.072 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:14.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.071713+0000) 2019-10-13T10:35:14.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:14.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.358528+0000) 2019-10-13T10:35:14.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:14.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:14.480110+0000) 2019-10-13T10:35:17.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:17.729+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:17.731219+0000) 2019-10-13T10:35:18.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:18.497+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:18.501638+0000) 2019-10-13T10:35:19.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:19.061+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.065413+0000) 2019-10-13T10:35:19.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:19.065+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.065843+0000) 2019-10-13T10:35:19.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:19.061+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.066887+0000) 2019-10-13T10:35:19.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:19.065+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.067688+0000) 2019-10-13T10:35:19.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:19.065+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.068081+0000) 2019-10-13T10:35:19.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:19.065+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.068793+0000) 2019-10-13T10:35:19.070 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:19.065+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.069060+0000) 2019-10-13T10:35:19.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:19.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:19.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:19.069+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:19.072647+0000) 2019-10-13T10:35:20.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:20.357+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:20.359083+0000) 2019-10-13T10:35:21.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:21.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:21.207590+0000) 2019-10-13T10:35:21.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:21.749+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:21.753588+0000) 2019-10-13T10:35:24.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:24.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.071906+0000) 2019-10-13T10:35:24.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:24.073+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.075708+0000) 2019-10-13T10:35:24.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:24.073+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.076125+0000) 2019-10-13T10:35:24.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:24.073+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.076722+0000) 2019-10-13T10:35:24.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:24.073+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.076821+0000) 2019-10-13T10:35:24.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:24.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.077847+0000) 2019-10-13T10:35:24.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:24.077+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.078691+0000) 2019-10-13T10:35:24.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:24.077+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.079053+0000) 2019-10-13T10:35:24.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:24.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:24.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:24.081+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.086473+0000) 2019-10-13T10:35:24.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:24.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.358722+0000) 2019-10-13T10:35:24.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:24.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:24.480307+0000) 2019-10-13T10:35:27.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:27.729+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:27.731418+0000) 2019-10-13T10:35:28.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:28.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:28.501835+0000) 2019-10-13T10:35:29.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:29.081+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.084493+0000) 2019-10-13T10:35:29.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:29.085+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.088109+0000) 2019-10-13T10:35:29.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:29.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.088691+0000) 2019-10-13T10:35:29.090 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:29.085+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.089011+0000) 2019-10-13T10:35:29.090 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:29.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.089453+0000) 2019-10-13T10:35:29.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:29.089+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.090109+0000) 2019-10-13T10:35:29.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:29.089+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.090419+0000) 2019-10-13T10:35:29.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:29.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:29.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:29.089+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:29.094236+0000) 2019-10-13T10:35:30.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:30.365+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:30.365920+0000) 2019-10-13T10:35:31.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:31.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:31.207798+0000) 2019-10-13T10:35:31.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:31.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:31.753789+0000) 2019-10-13T10:35:34.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:34.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.072128+0000) 2019-10-13T10:35:34.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:34.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.094562+0000) 2019-10-13T10:35:34.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:34.093+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.095014+0000) 2019-10-13T10:35:34.096 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:34.093+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.095829+0000) 2019-10-13T10:35:34.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:34.093+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.096154+0000) 2019-10-13T10:35:34.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:34.093+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.096691+0000) 2019-10-13T10:35:34.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:34.093+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.097144+0000) 2019-10-13T10:35:34.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:34.093+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.097440+0000) 2019-10-13T10:35:34.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:34.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120016 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:34.105 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:34.101+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.104289+0000) 2019-10-13T10:35:34.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:34.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.358902+0000) 2019-10-13T10:35:34.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:34.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:34.480499+0000) 2019-10-13T10:35:37.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:37.729+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:37.731646+0000) 2019-10-13T10:35:38.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:38.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:38.502046+0000) 2019-10-13T10:35:39.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:39.101+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.103794+0000) 2019-10-13T10:35:39.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:39.101+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.104489+0000) 2019-10-13T10:35:39.105 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:39.101+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.104467+0000) 2019-10-13T10:35:39.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:39.101+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.105258+0000) 2019-10-13T10:35:39.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:39.105+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.106026+0000) 2019-10-13T10:35:39.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:39.105+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.106199+0000) 2019-10-13T10:35:39.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:39.105+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.107030+0000) 2019-10-13T10:35:39.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:39.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:39.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:39.109+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:39.112697+0000) 2019-10-13T10:35:40.240 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:35:40.240 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:40.244 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:35:40.244 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:35:40.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:40.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:40.372540+0000) 2019-10-13T10:35:41.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:41.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:41.208018+0000) 2019-10-13T10:35:41.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:41.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:41.753984+0000) 2019-10-13T10:35:44.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:44.069+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.072367+0000) 2019-10-13T10:35:44.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:44.109+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.112911+0000) 2019-10-13T10:35:44.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:44.109+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.113480+0000) 2019-10-13T10:35:44.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:44.113+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.114570+0000) 2019-10-13T10:35:44.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:44.113+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.114672+0000) 2019-10-13T10:35:44.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:44.113+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.115110+0000) 2019-10-13T10:35:44.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:44.113+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.115731+0000) 2019-10-13T10:35:44.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:44.113+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.115997+0000) 2019-10-13T10:35:44.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:44.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:44.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:44.121+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.124950+0000) 2019-10-13T10:35:44.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:44.357+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.359084+0000) 2019-10-13T10:35:44.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:44.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:44.480711+0000) 2019-10-13T10:35:47.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:47.728+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:47.731863+0000) 2019-10-13T10:35:48.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:48.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:48.502246+0000) 2019-10-13T10:35:49.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:49.116+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.120436+0000) 2019-10-13T10:35:49.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:49.116+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.120865+0000) 2019-10-13T10:35:49.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:49.117+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.121604+0000) 2019-10-13T10:35:49.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:49.120+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.123357+0000) 2019-10-13T10:35:49.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:49.120+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.123691+0000) 2019-10-13T10:35:49.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:49.120+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.124226+0000) 2019-10-13T10:35:49.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:49.120+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.124815+0000) 2019-10-13T10:35:49.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:49.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:49.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:49.125+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:49.131683+0000) 2019-10-13T10:35:50.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:50.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:50.379051+0000) 2019-10-13T10:35:51.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:51.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:51.208220+0000) 2019-10-13T10:35:51.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:51.749+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:51.754211+0000) 2019-10-13T10:35:54.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:54.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.072567+0000) 2019-10-13T10:35:54.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:54.128+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.132747+0000) 2019-10-13T10:35:54.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:54.128+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.133212+0000) 2019-10-13T10:35:54.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:54.128+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.133623+0000) 2019-10-13T10:35:54.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:54.129+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.135198+0000) 2019-10-13T10:35:54.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:54.132+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.135337+0000) 2019-10-13T10:35:54.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:54.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:54.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:54.132+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.137352+0000) 2019-10-13T10:35:54.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:54.136+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.138185+0000) 2019-10-13T10:35:54.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:54.137+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.141798+0000) 2019-10-13T10:35:54.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:54.356+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.359266+0000) 2019-10-13T10:35:54.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:54.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:54.480907+0000) 2019-10-13T10:35:57.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:57.728+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:57.732096+0000) 2019-10-13T10:35:58.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:58.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:58.502451+0000) 2019-10-13T10:35:59.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:35:59.136+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.139860+0000) 2019-10-13T10:35:59.141 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:35:59.136+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.140547+0000) 2019-10-13T10:35:59.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:35:59.136+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.141117+0000) 2019-10-13T10:35:59.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:35:59.136+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.141211+0000) 2019-10-13T10:35:59.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:35:59.137+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.142054+0000) 2019-10-13T10:35:59.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:35:59.140+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.142701+0000) 2019-10-13T10:35:59.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:35:59.140+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.143140+0000) 2019-10-13T10:35:59.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:35:59.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:35:59.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:35:59.145+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:35:59.150143+0000) 2019-10-13T10:36:00.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:00.385+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:00.385872+0000) 2019-10-13T10:36:01.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:01.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:01.208422+0000) 2019-10-13T10:36:01.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:01.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:01.754442+0000) 2019-10-13T10:36:04.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:04.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.072775+0000) 2019-10-13T10:36:04.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:04.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.148724+0000) 2019-10-13T10:36:04.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:04.145+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.149437+0000) 2019-10-13T10:36:04.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:04.148+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.150450+0000) 2019-10-13T10:36:04.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:04.148+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.152166+0000) 2019-10-13T10:36:04.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:04.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.153835+0000) 2019-10-13T10:36:04.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:04.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:04.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:04.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.155328+0000) 2019-10-13T10:36:04.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:04.152+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.155869+0000) 2019-10-13T10:36:04.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:04.153+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.158489+0000) 2019-10-13T10:36:04.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:04.356+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.359443+0000) 2019-10-13T10:36:04.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:04.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:04.481113+0000) 2019-10-13T10:36:07.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:07.728+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:07.732319+0000) 2019-10-13T10:36:08.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:08.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:08.502657+0000) 2019-10-13T10:36:09.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:09.152+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.157646+0000) 2019-10-13T10:36:09.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:09.156+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.158774+0000) 2019-10-13T10:36:09.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:09.156+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.158978+0000) 2019-10-13T10:36:09.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:09.156+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.159374+0000) 2019-10-13T10:36:09.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:09.156+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.160096+0000) 2019-10-13T10:36:09.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:09.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.160205+0000) 2019-10-13T10:36:09.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:09.156+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.160885+0000) 2019-10-13T10:36:09.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:09.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120226 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:09.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:09.161+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:09.167868+0000) 2019-10-13T10:36:10.382 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:36:10.382 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:10.386 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:36:10.386 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:10.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:10.389+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:10.392467+0000) 2019-10-13T10:36:11.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:11.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:11.208591+0000) 2019-10-13T10:36:11.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:11.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:11.754634+0000) 2019-10-13T10:36:14.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:14.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.073011+0000) 2019-10-13T10:36:14.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:14.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.167328+0000) 2019-10-13T10:36:14.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:14.164+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.169173+0000) 2019-10-13T10:36:14.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:14.168+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.171065+0000) 2019-10-13T10:36:14.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:14.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.171888+0000) 2019-10-13T10:36:14.173 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:14.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.172270+0000) 2019-10-13T10:36:14.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:14.168+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.172920+0000) 2019-10-13T10:36:14.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:14.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:14.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:14.169+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.173319+0000) 2019-10-13T10:36:14.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:14.173+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.177818+0000) 2019-10-13T10:36:14.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:14.356+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.359575+0000) 2019-10-13T10:36:14.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:14.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:14.481287+0000) 2019-10-13T10:36:17.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:17.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:17.732519+0000) 2019-10-13T10:36:18.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:18.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:18.502853+0000) 2019-10-13T10:36:19.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:19.172+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.176401+0000) 2019-10-13T10:36:19.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:19.172+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.177205+0000) 2019-10-13T10:36:19.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:19.172+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.177707+0000) 2019-10-13T10:36:19.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:19.176+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.178352+0000) 2019-10-13T10:36:19.179 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:19.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.179023+0000) 2019-10-13T10:36:19.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:19.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.179678+0000) 2019-10-13T10:36:19.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:19.173+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.179498+0000) 2019-10-13T10:36:19.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:19.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:19.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:19.181+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:19.186119+0000) 2019-10-13T10:36:20.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:20.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:20.399444+0000) 2019-10-13T10:36:21.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:21.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:21.208821+0000) 2019-10-13T10:36:21.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:21.752+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:21.754837+0000) 2019-10-13T10:36:24.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:24.068+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.073240+0000) 2019-10-13T10:36:24.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:24.184+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.186681+0000) 2019-10-13T10:36:24.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:24.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.187406+0000) 2019-10-13T10:36:24.189 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:24.184+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.187866+0000) 2019-10-13T10:36:24.189 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:24.184+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.188103+0000) 2019-10-13T10:36:24.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:24.184+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.189867+0000) 2019-10-13T10:36:24.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:24.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.190601+0000) 2019-10-13T10:36:24.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:24.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:24.194 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:24.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.193717+0000) 2019-10-13T10:36:24.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:24.192+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.197694+0000) 2019-10-13T10:36:24.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:24.356+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.359784+0000) 2019-10-13T10:36:24.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:24.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:24.481464+0000) 2019-10-13T10:36:27.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:27.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:27.732719+0000) 2019-10-13T10:36:28.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:28.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:28.503047+0000) 2019-10-13T10:36:29.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:29.192+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.194787+0000) 2019-10-13T10:36:29.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:29.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.194992+0000) 2019-10-13T10:36:29.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:29.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.197088+0000) 2019-10-13T10:36:29.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:29.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.197287+0000) 2019-10-13T10:36:29.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:29.192+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.197722+0000) 2019-10-13T10:36:29.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:29.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:29.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:29.192+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.199941+0000) 2019-10-13T10:36:29.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:29.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.200691+0000) 2019-10-13T10:36:29.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:29.196+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:29.201303+0000) 2019-10-13T10:36:30.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:30.400+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:30.406446+0000) 2019-10-13T10:36:31.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:31.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:31.209052+0000) 2019-10-13T10:36:31.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:31.752+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:31.755039+0000) 2019-10-13T10:36:34.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:34.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.073446+0000) 2019-10-13T10:36:34.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:34.200+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.203320+0000) 2019-10-13T10:36:34.205 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:34.200+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.204357+0000) 2019-10-13T10:36:34.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:34.200+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.203980+0000) 2019-10-13T10:36:34.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:34.200+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.205092+0000) 2019-10-13T10:36:34.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:34.200+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.205625+0000) 2019-10-13T10:36:34.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:34.204+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.206357+0000) 2019-10-13T10:36:34.208 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:34.204+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.207027+0000) 2019-10-13T10:36:34.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:34.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:34.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:34.208+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.212924+0000) 2019-10-13T10:36:34.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:34.356+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.359992+0000) 2019-10-13T10:36:34.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:34.476+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:34.481640+0000) 2019-10-13T10:36:37.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:37.728+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:37.732942+0000) 2019-10-13T10:36:38.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:38.500+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:38.503245+0000) 2019-10-13T10:36:39.213 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:39.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.212204+0000) 2019-10-13T10:36:39.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:39.208+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.212784+0000) 2019-10-13T10:36:39.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:39.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.213292+0000) 2019-10-13T10:36:39.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:39.212+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.214564+0000) 2019-10-13T10:36:39.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:39.212+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.214411+0000) 2019-10-13T10:36:39.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:39.208+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.215041+0000) 2019-10-13T10:36:39.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:39.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.215808+0000) 2019-10-13T10:36:39.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:39.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:39.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:39.216+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:39.222859+0000) 2019-10-13T10:36:40.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:40.408+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:40.409361+0000) 2019-10-13T10:36:40.509 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:36:40.510 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:40.513 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:36:40.513 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:36:41.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:41.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:41.209281+0000) 2019-10-13T10:36:41.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:41.752+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:41.755234+0000) 2019-10-13T10:36:44.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:44.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.073724+0000) 2019-10-13T10:36:44.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:44.219+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.222164+0000) 2019-10-13T10:36:44.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:44.219+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.222454+0000) 2019-10-13T10:36:44.224 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:44.219+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.222755+0000) 2019-10-13T10:36:44.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:44.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.223119+0000) 2019-10-13T10:36:44.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:44.219+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.223920+0000) 2019-10-13T10:36:44.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:44.220+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.224659+0000) 2019-10-13T10:36:44.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:44.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.225244+0000) 2019-10-13T10:36:44.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:44.223+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:44.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:44.228+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.235247+0000) 2019-10-13T10:36:44.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:44.359+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.360202+0000) 2019-10-13T10:36:44.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:44.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:44.481817+0000) 2019-10-13T10:36:47.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:47.731+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:47.733146+0000) 2019-10-13T10:36:48.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:48.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:48.503439+0000) 2019-10-13T10:36:49.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:49.227+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.230855+0000) 2019-10-13T10:36:49.233 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:49.227+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.232955+0000) 2019-10-13T10:36:49.235 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:49.231+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.234520+0000) 2019-10-13T10:36:49.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:49.231+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.234996+0000) 2019-10-13T10:36:49.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:49.231+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.235515+0000) 2019-10-13T10:36:49.237 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:49.231+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.236105+0000) 2019-10-13T10:36:49.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:49.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:49.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:49.232+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.236694+0000) 2019-10-13T10:36:49.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:49.236+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:49.242945+0000) 2019-10-13T10:36:50.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:50.416+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:50.416577+0000) 2019-10-13T10:36:51.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:51.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:51.209486+0000) 2019-10-13T10:36:51.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:51.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:51.755487+0000) 2019-10-13T10:36:54.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:54.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.074051+0000) 2019-10-13T10:36:54.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:54.235+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.239700+0000) 2019-10-13T10:36:54.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:54.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.240410+0000) 2019-10-13T10:36:54.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:54.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.240965+0000) 2019-10-13T10:36:54.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:54.235+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.241591+0000) 2019-10-13T10:36:54.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:54.239+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.242260+0000) 2019-10-13T10:36:54.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:54.239+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.242600+0000) 2019-10-13T10:36:54.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:54.236+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.242680+0000) 2019-10-13T10:36:54.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:54.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:54.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:54.244+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.249294+0000) 2019-10-13T10:36:54.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:54.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.360381+0000) 2019-10-13T10:36:54.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:54.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:54.482001+0000) 2019-10-13T10:36:57.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:57.727+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:57.733347+0000) 2019-10-13T10:36:58.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:58.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:58.503651+0000) 2019-10-13T10:36:59.250 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:36:59.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.248923+0000) 2019-10-13T10:36:59.251 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:36:59.247+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.249793+0000) 2019-10-13T10:36:59.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:36:59.247+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.250188+0000) 2019-10-13T10:36:59.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:36:59.244+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.250751+0000) 2019-10-13T10:36:59.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:36:59.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.250420+0000) 2019-10-13T10:36:59.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:36:59.247+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.251767+0000) 2019-10-13T10:36:59.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:36:59.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.252223+0000) 2019-10-13T10:36:59.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:36:59.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:36:59.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:36:59.248+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:36:59.255066+0000) 2019-10-13T10:37:00.425 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:00.420+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:00.423683+0000) 2019-10-13T10:37:01.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:01.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:01.209690+0000) 2019-10-13T10:37:01.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:01.748+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:01.755686+0000) 2019-10-13T10:37:04.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:04.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.074277+0000) 2019-10-13T10:37:04.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:04.252+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.257262+0000) 2019-10-13T10:37:04.259 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:04.255+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.258578+0000) 2019-10-13T10:37:04.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:04.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.260247+0000) 2019-10-13T10:37:04.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:04.255+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.261394+0000) 2019-10-13T10:37:04.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:04.255+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.260906+0000) 2019-10-13T10:37:04.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:04.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.261829+0000) 2019-10-13T10:37:04.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:04.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.262698+0000) 2019-10-13T10:37:04.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:04.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:04.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:04.260+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.268074+0000) 2019-10-13T10:37:04.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:04.359+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.360581+0000) 2019-10-13T10:37:04.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:04.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:04.482175+0000) 2019-10-13T10:37:07.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:07.727+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:07.733549+0000) 2019-10-13T10:37:08.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:08.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:08.503843+0000) 2019-10-13T10:37:09.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:09.263+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.266815+0000) 2019-10-13T10:37:09.268 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:09.263+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.267248+0000) 2019-10-13T10:37:09.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:09.267+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.270140+0000) 2019-10-13T10:37:09.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:09.267+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.270521+0000) 2019-10-13T10:37:09.272 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:09.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.270872+0000) 2019-10-13T10:37:09.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:09.267+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.271006+0000) 2019-10-13T10:37:09.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:09.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.269647+0000) 2019-10-13T10:37:09.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:09.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:09.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:09.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:09.275543+0000) 2019-10-13T10:37:10.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:10.432+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:10.435030+0000) 2019-10-13T10:37:10.591 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:37:10.591 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:10.595 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:37:10.595 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:11.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:11.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:11.209935+0000) 2019-10-13T10:37:11.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:11.752+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:11.755880+0000) 2019-10-13T10:37:14.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:14.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.074503+0000) 2019-10-13T10:37:14.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:14.271+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.275524+0000) 2019-10-13T10:37:14.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:14.272+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.275649+0000) 2019-10-13T10:37:14.277 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:14.271+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.276390+0000) 2019-10-13T10:37:14.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:14.271+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.276979+0000) 2019-10-13T10:37:14.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:14.271+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.277366+0000) 2019-10-13T10:37:14.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:14.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.277801+0000) 2019-10-13T10:37:14.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:14.275+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.278640+0000) 2019-10-13T10:37:14.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:14.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:14.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:14.280+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.284667+0000) 2019-10-13T10:37:14.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:14.359+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.360796+0000) 2019-10-13T10:37:14.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:14.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:14.482368+0000) 2019-10-13T10:37:17.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:17.731+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:17.733771+0000) 2019-10-13T10:37:18.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:18.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:18.504043+0000) 2019-10-13T10:37:19.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:19.279+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.285592+0000) 2019-10-13T10:37:19.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:19.283+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.286339+0000) 2019-10-13T10:37:19.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:19.283+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.286767+0000) 2019-10-13T10:37:19.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:19.283+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.287136+0000) 2019-10-13T10:37:19.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:19.280+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.287470+0000) 2019-10-13T10:37:19.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:19.283+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.288062+0000) 2019-10-13T10:37:19.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:19.283+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.288682+0000) 2019-10-13T10:37:19.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:19.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120643 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:19.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:19.292+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:19.296206+0000) 2019-10-13T10:37:20.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:20.440+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:20.441633+0000) 2019-10-13T10:37:21.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:21.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:21.210135+0000) 2019-10-13T10:37:21.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:21.756+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:21.756081+0000) 2019-10-13T10:37:24.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:24.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.074699+0000) 2019-10-13T10:37:24.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:24.291+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.295207+0000) 2019-10-13T10:37:24.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:24.295+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.297931+0000) 2019-10-13T10:37:24.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:24.295+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.298265+0000) 2019-10-13T10:37:24.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:24.295+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.300230+0000) 2019-10-13T10:37:24.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:24.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:24.302 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:24.300+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.301161+0000) 2019-10-13T10:37:24.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:24.299+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.301787+0000) 2019-10-13T10:37:24.304 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:24.299+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.303860+0000) 2019-10-13T10:37:24.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:24.308+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.310689+0000) 2019-10-13T10:37:24.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:24.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.361005+0000) 2019-10-13T10:37:24.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:24.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:24.482538+0000) 2019-10-13T10:37:27.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:27.731+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:27.733988+0000) 2019-10-13T10:37:28.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:28.499+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:28.504242+0000) 2019-10-13T10:37:29.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:29.299+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.303996+0000) 2019-10-13T10:37:29.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:29.303+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.307076+0000) 2019-10-13T10:37:29.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:29.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.307682+0000) 2019-10-13T10:37:29.310 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:29.303+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.309337+0000) 2019-10-13T10:37:29.310 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:29.307+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.309181+0000) 2019-10-13T10:37:29.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:29.307+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.309936+0000) 2019-10-13T10:37:29.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:29.307+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.310408+0000) 2019-10-13T10:37:29.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:29.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:29.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:29.315+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:29.316729+0000) 2019-10-13T10:37:30.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:30.447+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:30.448142+0000) 2019-10-13T10:37:31.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:31.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:31.210365+0000) 2019-10-13T10:37:31.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:31.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:31.756311+0000) 2019-10-13T10:37:34.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:34.071+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.074962+0000) 2019-10-13T10:37:34.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:34.311+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.314279+0000) 2019-10-13T10:37:34.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:34.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.314657+0000) 2019-10-13T10:37:34.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:34.311+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.315171+0000) 2019-10-13T10:37:34.316 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:34.311+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.315385+0000) 2019-10-13T10:37:34.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:34.311+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.316006+0000) 2019-10-13T10:37:34.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:34.311+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.317090+0000) 2019-10-13T10:37:34.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:34.315+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.317146+0000) 2019-10-13T10:37:34.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:34.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:34.322 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:34.319+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.321612+0000) 2019-10-13T10:37:34.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:34.355+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.361208+0000) 2019-10-13T10:37:34.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:34.479+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:34.482717+0000) 2019-10-13T10:37:37.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:37.731+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:37.734183+0000) 2019-10-13T10:37:38.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:38.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:38.504532+0000) 2019-10-13T10:37:39.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:39.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.323607+0000) 2019-10-13T10:37:39.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:39.319+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.323625+0000) 2019-10-13T10:37:39.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:39.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.325807+0000) 2019-10-13T10:37:39.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:39.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.325981+0000) 2019-10-13T10:37:39.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:39.323+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.326831+0000) 2019-10-13T10:37:39.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:39.323+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.327442+0000) 2019-10-13T10:37:39.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:39.323+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.328061+0000) 2019-10-13T10:37:39.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:39.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:39.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:39.331+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:39.334543+0000) 2019-10-13T10:37:40.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:40.451+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:40.454613+0000) 2019-10-13T10:37:40.674 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:37:40.674 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:40.678 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:37:40.678 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:37:41.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:41.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:41.210579+0000) 2019-10-13T10:37:41.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:41.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:41.756499+0000) 2019-10-13T10:37:44.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:44.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.075158+0000) 2019-10-13T10:37:44.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:44.326+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.331772+0000) 2019-10-13T10:37:44.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:44.326+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.332829+0000) 2019-10-13T10:37:44.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:44.326+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.333312+0000) 2019-10-13T10:37:44.334 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:44.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.333126+0000) 2019-10-13T10:37:44.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:44.330+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.334038+0000) 2019-10-13T10:37:44.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:44.330+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.336138+0000) 2019-10-13T10:37:44.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:44.330+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.336386+0000) 2019-10-13T10:37:44.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:44.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:44.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:44.339+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.342811+0000) 2019-10-13T10:37:44.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:44.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.361387+0000) 2019-10-13T10:37:44.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:44.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:44.482892+0000) 2019-10-13T10:37:47.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:47.730+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:47.734409+0000) 2019-10-13T10:37:48.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:48.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:48.504739+0000) 2019-10-13T10:37:49.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:49.338+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.342746+0000) 2019-10-13T10:37:49.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:49.338+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.343125+0000) 2019-10-13T10:37:49.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:49.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.343951+0000) 2019-10-13T10:37:49.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:49.338+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.344064+0000) 2019-10-13T10:37:49.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:49.338+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.344944+0000) 2019-10-13T10:37:49.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:49.343+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.345214+0000) 2019-10-13T10:37:49.346 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:49.342+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.346039+0000) 2019-10-13T10:37:49.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:49.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120827 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:49.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:49.351+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:49.352543+0000) 2019-10-13T10:37:50.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:50.459+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:50.461091+0000) 2019-10-13T10:37:51.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:51.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:51.210805+0000) 2019-10-13T10:37:51.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:51.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:51.756710+0000) 2019-10-13T10:37:54.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:54.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.075356+0000) 2019-10-13T10:37:54.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:54.346+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.350542+0000) 2019-10-13T10:37:54.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:54.346+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.351800+0000) 2019-10-13T10:37:54.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:54.346+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.352251+0000) 2019-10-13T10:37:54.353 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:54.346+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.352515+0000) 2019-10-13T10:37:54.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:54.346+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.352842+0000) 2019-10-13T10:37:54.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:54.346+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.353635+0000) 2019-10-13T10:37:54.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:54.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.354005+0000) 2019-10-13T10:37:54.357 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:54.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:54.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:54.355+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.356889+0000) 2019-10-13T10:37:54.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:54.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.361528+0000) 2019-10-13T10:37:54.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:54.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:54.483078+0000) 2019-10-13T10:37:57.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:57.730+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:57.734641+0000) 2019-10-13T10:37:58.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:58.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:58.504933+0000) 2019-10-13T10:37:59.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:37:59.354+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.360408+0000) 2019-10-13T10:37:59.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:37:59.354+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.360628+0000) 2019-10-13T10:37:59.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:37:59.354+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.360892+0000) 2019-10-13T10:37:59.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:37:59.354+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.361547+0000) 2019-10-13T10:37:59.363 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:37:59.358+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.361930+0000) 2019-10-13T10:37:59.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:37:59.358+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.362756+0000) 2019-10-13T10:37:59.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:37:59.359+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.363726+0000) 2019-10-13T10:37:59.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:37:59.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120887 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:37:59.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:37:59.367+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:37:59.369485+0000) 2019-10-13T10:38:00.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:00.471+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:00.472127+0000) 2019-10-13T10:38:01.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:01.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:01.211038+0000) 2019-10-13T10:38:01.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:01.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:01.756945+0000) 2019-10-13T10:38:04.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:04.070+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.075549+0000) 2019-10-13T10:38:04.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:04.354+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.361706+0000) 2019-10-13T10:38:04.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:04.366+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.370392+0000) 2019-10-13T10:38:04.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:04.366+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.371667+0000) 2019-10-13T10:38:04.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:04.370+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.375108+0000) 2019-10-13T10:38:04.376 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:04.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:04.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:04.370+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.376033+0000) 2019-10-13T10:38:04.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:04.370+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.376451+0000) 2019-10-13T10:38:04.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:04.370+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.376824+0000) 2019-10-13T10:38:04.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:04.375+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.376676+0000) 2019-10-13T10:38:04.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:04.375+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.377124+0000) 2019-10-13T10:38:04.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:04.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:04.483286+0000) 2019-10-13T10:38:07.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:07.730+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:07.734844+0000) 2019-10-13T10:38:08.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:08.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:08.505140+0000) 2019-10-13T10:38:09.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:09.374+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.379359+0000) 2019-10-13T10:38:09.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:09.374+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.379832+0000) 2019-10-13T10:38:09.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:09.374+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.380177+0000) 2019-10-13T10:38:09.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:09.374+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.380423+0000) 2019-10-13T10:38:09.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:09.374+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.380902+0000) 2019-10-13T10:38:09.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:09.379+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.382111+0000) 2019-10-13T10:38:09.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:09.378+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.382939+0000) 2019-10-13T10:38:09.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:09.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:09.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:09.383+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:09.387412+0000) 2019-10-13T10:38:10.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:10.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:10.478512+0000) 2019-10-13T10:38:10.828 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:38:10.828 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:10.831 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:38:10.831 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:11.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:11.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:11.211271+0000) 2019-10-13T10:38:11.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:11.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:11.757149+0000) 2019-10-13T10:38:14.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:14.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.075751+0000) 2019-10-13T10:38:14.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:14.358+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.361936+0000) 2019-10-13T10:38:14.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:14.382+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.387973+0000) 2019-10-13T10:38:14.389 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:14.382+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.388504+0000) 2019-10-13T10:38:14.390 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:14.382+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.389144+0000) 2019-10-13T10:38:14.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:14.387+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.389630+0000) 2019-10-13T10:38:14.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:14.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.390930+0000) 2019-10-13T10:38:14.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:14.386+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.391348+0000) 2019-10-13T10:38:14.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:14.386+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.391981+0000) 2019-10-13T10:38:14.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:14.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:14.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:14.391+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.395887+0000) 2019-10-13T10:38:14.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:14.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:14.483474+0000) 2019-10-13T10:38:17.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:17.734+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:17.735058+0000) 2019-10-13T10:38:18.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:18.498+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:18.505360+0000) 2019-10-13T10:38:19.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:19.394+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.398298+0000) 2019-10-13T10:38:19.400 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:19.395+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.398878+0000) 2019-10-13T10:38:19.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:19.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.399465+0000) 2019-10-13T10:38:19.401 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:19.394+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.400312+0000) 2019-10-13T10:38:19.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:19.394+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.400750+0000) 2019-10-13T10:38:19.402 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:19.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.401024+0000) 2019-10-13T10:38:19.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:19.394+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.401493+0000) 2019-10-13T10:38:19.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:19.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:19.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:19.403+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:19.405889+0000) 2019-10-13T10:38:20.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:20.483+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:20.484989+0000) 2019-10-13T10:38:21.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:21.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:21.211482+0000) 2019-10-13T10:38:21.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:21.755+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:21.757373+0000) 2019-10-13T10:38:24.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:24.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.075955+0000) 2019-10-13T10:38:24.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:24.358+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.362169+0000) 2019-10-13T10:38:24.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:24.402+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.406968+0000) 2019-10-13T10:38:24.409 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:24.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.408325+0000) 2019-10-13T10:38:24.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:24.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.409004+0000) 2019-10-13T10:38:24.410 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:24.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.409190+0000) 2019-10-13T10:38:24.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:24.406+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.409803+0000) 2019-10-13T10:38:24.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:24.406+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.410832+0000) 2019-10-13T10:38:24.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:24.406+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.411022+0000) 2019-10-13T10:38:24.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:24.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121040 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:24.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:24.411+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.414941+0000) 2019-10-13T10:38:24.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:24.482+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:24.483680+0000) 2019-10-13T10:38:27.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:27.734+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:27.735258+0000) 2019-10-13T10:38:28.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:28.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:28.505582+0000) 2019-10-13T10:38:29.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:29.410+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.417713+0000) 2019-10-13T10:38:29.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:29.414+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.418087+0000) 2019-10-13T10:38:29.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:29.414+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.418405+0000) 2019-10-13T10:38:29.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:29.414+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.419628+0000) 2019-10-13T10:38:29.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:29.414+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.421206+0000) 2019-10-13T10:38:29.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:29.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:29.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:29.419+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.422435+0000) 2019-10-13T10:38:29.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:29.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.423303+0000) 2019-10-13T10:38:29.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:29.427+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:29.429997+0000) 2019-10-13T10:38:30.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:30.495+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:30.496560+0000) 2019-10-13T10:38:31.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:31.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:31.211690+0000) 2019-10-13T10:38:31.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:31.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:31.757597+0000) 2019-10-13T10:38:34.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:34.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.076163+0000) 2019-10-13T10:38:34.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:34.358+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.362386+0000) 2019-10-13T10:38:34.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:34.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.425808+0000) 2019-10-13T10:38:34.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:34.422+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.425932+0000) 2019-10-13T10:38:34.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:34.422+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.428491+0000) 2019-10-13T10:38:34.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:34.422+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.428151+0000) 2019-10-13T10:38:34.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:34.422+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.429232+0000) 2019-10-13T10:38:34.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:34.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.429661+0000) 2019-10-13T10:38:34.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:34.426+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.429861+0000) 2019-10-13T10:38:34.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:34.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:34.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:34.430+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.435500+0000) 2019-10-13T10:38:34.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:34.478+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:34.483801+0000) 2019-10-13T10:38:37.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:37.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:37.735459+0000) 2019-10-13T10:38:38.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:38.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:38.505769+0000) 2019-10-13T10:38:39.435 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:39.429+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.434482+0000) 2019-10-13T10:38:39.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:39.429+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.435107+0000) 2019-10-13T10:38:39.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:39.429+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.435955+0000) 2019-10-13T10:38:39.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:39.429+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.437648+0000) 2019-10-13T10:38:39.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:39.433+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.440758+0000) 2019-10-13T10:38:39.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:39.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:39.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:39.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.441809+0000) 2019-10-13T10:38:39.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:39.437+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.442878+0000) 2019-10-13T10:38:39.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:39.442+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:39.444580+0000) 2019-10-13T10:38:40.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:40.502+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:40.503254+0000) 2019-10-13T10:38:40.951 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:38:40.951 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:40.955 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:38:40.955 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:38:41.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:41.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:41.211893+0000) 2019-10-13T10:38:41.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:41.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:41.757792+0000) 2019-10-13T10:38:44.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:44.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.076401+0000) 2019-10-13T10:38:44.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:44.361+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.362613+0000) 2019-10-13T10:38:44.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:44.441+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.446215+0000) 2019-10-13T10:38:44.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:44.441+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.446683+0000) 2019-10-13T10:38:44.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:44.441+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.446937+0000) 2019-10-13T10:38:44.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:44.441+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.447484+0000) 2019-10-13T10:38:44.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:44.441+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.447154+0000) 2019-10-13T10:38:44.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:44.441+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.448364+0000) 2019-10-13T10:38:44.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:44.442+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.447626+0000) 2019-10-13T10:38:44.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:44.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:44.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:44.450+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.452362+0000) 2019-10-13T10:38:44.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:44.477+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:44.483973+0000) 2019-10-13T10:38:47.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:47.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:47.735663+0000) 2019-10-13T10:38:48.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:48.501+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:48.505925+0000) 2019-10-13T10:38:49.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:49.449+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.455039+0000) 2019-10-13T10:38:49.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:49.449+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.455247+0000) 2019-10-13T10:38:49.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:49.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.455136+0000) 2019-10-13T10:38:49.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:49.449+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.456567+0000) 2019-10-13T10:38:49.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:49.453+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.458094+0000) 2019-10-13T10:38:49.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:49.453+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.459848+0000) 2019-10-13T10:38:49.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:49.453+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:49.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:49.453+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.461084+0000) 2019-10-13T10:38:49.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:49.458+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:49.462284+0000) 2019-10-13T10:38:50.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:50.506+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:50.509866+0000) 2019-10-13T10:38:51.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:51.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:51.212075+0000) 2019-10-13T10:38:51.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:51.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:51.757988+0000) 2019-10-13T10:38:54.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:54.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.076597+0000) 2019-10-13T10:38:54.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:54.361+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.362811+0000) 2019-10-13T10:38:54.465 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:54.461+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.464351+0000) 2019-10-13T10:38:54.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:54.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.464188+0000) 2019-10-13T10:38:54.466 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:54.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.464753+0000) 2019-10-13T10:38:54.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:54.461+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.465352+0000) 2019-10-13T10:38:54.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:54.465+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.466390+0000) 2019-10-13T10:38:54.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:54.462+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.466558+0000) 2019-10-13T10:38:54.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:54.465+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.467197+0000) 2019-10-13T10:38:54.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:54.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121224 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:54.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:54.474+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.476594+0000) 2019-10-13T10:38:54.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:54.481+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:54.484130+0000) 2019-10-13T10:38:57.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:57.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:57.735879+0000) 2019-10-13T10:38:58.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:58.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:58.506123+0000) 2019-10-13T10:38:59.474 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:38:59.469+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.473246+0000) 2019-10-13T10:38:59.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:38:59.470+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.473255+0000) 2019-10-13T10:38:59.475 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:38:59.473+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.474328+0000) 2019-10-13T10:38:59.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:38:59.473+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.476479+0000) 2019-10-13T10:38:59.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:38:59.477+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.478073+0000) 2019-10-13T10:38:59.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:38:59.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121253 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:38:59.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:38:59.477+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.479955+0000) 2019-10-13T10:38:59.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:38:59.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.480598+0000) 2019-10-13T10:38:59.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:38:59.478+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:38:59.482867+0000) 2019-10-13T10:39:00.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:00.514+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:00.516195+0000) 2019-10-13T10:39:01.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:01.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:01.212274+0000) 2019-10-13T10:39:01.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:01.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:01.758192+0000) 2019-10-13T10:39:04.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:04.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.076816+0000) 2019-10-13T10:39:04.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:04.361+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.363037+0000) 2019-10-13T10:39:04.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:04.478+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.483009+0000) 2019-10-13T10:39:04.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:04.481+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.483660+0000) 2019-10-13T10:39:04.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:04.481+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.483987+0000) 2019-10-13T10:39:04.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:04.481+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.485510+0000) 2019-10-13T10:39:04.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:04.485+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.485959+0000) 2019-10-13T10:39:04.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:04.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:04.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:04.485+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.487410+0000) 2019-10-13T10:39:04.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:04.485+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.489017+0000) 2019-10-13T10:39:04.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:04.490+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:04.493997+0000) 2019-10-13T10:39:07.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:07.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:07.736083+0000) 2019-10-13T10:39:08.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:08.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:08.506342+0000) 2019-10-13T10:39:09.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:09.489+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.491645+0000) 2019-10-13T10:39:09.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:09.489+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.491846+0000) 2019-10-13T10:39:09.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:09.489+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.492966+0000) 2019-10-13T10:39:09.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:09.490+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.492894+0000) 2019-10-13T10:39:09.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:09.493+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.494014+0000) 2019-10-13T10:39:09.495 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:09.493+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.494678+0000) 2019-10-13T10:39:09.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:09.493+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.495334+0000) 2019-10-13T10:39:09.499 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:09.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121314 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:09.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:09.498+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:09.501339+0000) 2019-10-13T10:39:10.527 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:10.522+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:10.522716+0000) 2019-10-13T10:39:11.065 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:39:11.065 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:11.069 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:39:11.069 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:11.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:11.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:11.212512+0000) 2019-10-13T10:39:11.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:11.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:11.234678+0000) 2019-10-13T10:39:11.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:11.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:11.758382+0000) 2019-10-13T10:39:14.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:14.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.077019+0000) 2019-10-13T10:39:14.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:14.361+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.363241+0000) 2019-10-13T10:39:14.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:14.497+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.501227+0000) 2019-10-13T10:39:14.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:14.498+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.502183+0000) 2019-10-13T10:39:14.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:14.501+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.502937+0000) 2019-10-13T10:39:14.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:14.501+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.503258+0000) 2019-10-13T10:39:14.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:14.501+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.503699+0000) 2019-10-13T10:39:14.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:14.501+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.504350+0000) 2019-10-13T10:39:14.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:14.501+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.505335+0000) 2019-10-13T10:39:14.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:14.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121346 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:14.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:14.506+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:14.509956+0000) 2019-10-13T10:39:17.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:17.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:17.736325+0000) 2019-10-13T10:39:18.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:18.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:18.506542+0000) 2019-10-13T10:39:19.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:19.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.511207+0000) 2019-10-13T10:39:19.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:19.509+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.512229+0000) 2019-10-13T10:39:19.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:19.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.512824+0000) 2019-10-13T10:39:19.518 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:19.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:19.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:19.513+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.517119+0000) 2019-10-13T10:39:19.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:19.513+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.517633+0000) 2019-10-13T10:39:19.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:19.517+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.518042+0000) 2019-10-13T10:39:19.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:19.517+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.518501+0000) 2019-10-13T10:39:19.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:19.514+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:19.519264+0000) 2019-10-13T10:39:20.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:20.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:20.529207+0000) 2019-10-13T10:39:21.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:21.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:21.212709+0000) 2019-10-13T10:39:21.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:21.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:21.234875+0000) 2019-10-13T10:39:21.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:21.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:21.758583+0000) 2019-10-13T10:39:24.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:24.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.077246+0000) 2019-10-13T10:39:24.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:24.361+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.363441+0000) 2019-10-13T10:39:24.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:24.517+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.520547+0000) 2019-10-13T10:39:24.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:24.518+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.523354+0000) 2019-10-13T10:39:24.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:24.521+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.523945+0000) 2019-10-13T10:39:24.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:24.521+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.524186+0000) 2019-10-13T10:39:24.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:24.521+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.524691+0000) 2019-10-13T10:39:24.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:24.521+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.525036+0000) 2019-10-13T10:39:24.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:24.521+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.524955+0000) 2019-10-13T10:39:24.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:24.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:24.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:24.522+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:24.527763+0000) 2019-10-13T10:39:27.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:27.733+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:27.736533+0000) 2019-10-13T10:39:28.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:28.505+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:28.506742+0000) 2019-10-13T10:39:29.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:29.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.529224+0000) 2019-10-13T10:39:29.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:29.525+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.529593+0000) 2019-10-13T10:39:29.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:29.529+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.532359+0000) 2019-10-13T10:39:29.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:29.529+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.532664+0000) 2019-10-13T10:39:29.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:29.530+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.534114+0000) 2019-10-13T10:39:29.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:29.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:29.536 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:29.533+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.534853+0000) 2019-10-13T10:39:29.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:29.533+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.535758+0000) 2019-10-13T10:39:29.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:29.538+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:29.541684+0000) 2019-10-13T10:39:30.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:30.530+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:30.535788+0000) 2019-10-13T10:39:31.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:31.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:31.212943+0000) 2019-10-13T10:39:31.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:31.233+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:31.235076+0000) 2019-10-13T10:39:31.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:31.754+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:31.758797+0000) 2019-10-13T10:39:34.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:34.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.077441+0000) 2019-10-13T10:39:34.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:34.360+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.363653+0000) 2019-10-13T10:39:34.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:34.532+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.537455+0000) 2019-10-13T10:39:34.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:34.536+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.537887+0000) 2019-10-13T10:39:34.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:34.536+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.539373+0000) 2019-10-13T10:39:34.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:34.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.539650+0000) 2019-10-13T10:39:34.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:34.536+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.540459+0000) 2019-10-13T10:39:34.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:34.540+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.542538+0000) 2019-10-13T10:39:34.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:34.540+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.543085+0000) 2019-10-13T10:39:34.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:34.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:34.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:34.546+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:34.550081+0000) 2019-10-13T10:39:37.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:37.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:37.736740+0000) 2019-10-13T10:39:38.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:38.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:38.506965+0000) 2019-10-13T10:39:39.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:39.544+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.547990+0000) 2019-10-13T10:39:39.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:39.545+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.548335+0000) 2019-10-13T10:39:39.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:39.544+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.548836+0000) 2019-10-13T10:39:39.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:39.544+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.549650+0000) 2019-10-13T10:39:39.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:39.548+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.550392+0000) 2019-10-13T10:39:39.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:39.548+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.550524+0000) 2019-10-13T10:39:39.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:39.548+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.550969+0000) 2019-10-13T10:39:39.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:39.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:39.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:39.553+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:39.556697+0000) 2019-10-13T10:39:40.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:40.541+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:40.542507+0000) 2019-10-13T10:39:41.162 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:39:41.162 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:41.166 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:39:41.166 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:39:41.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:41.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:41.213122+0000) 2019-10-13T10:39:41.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:41.232+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:41.235280+0000) 2019-10-13T10:39:41.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:41.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:41.759018+0000) 2019-10-13T10:39:44.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:44.072+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.077637+0000) 2019-10-13T10:39:44.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:44.360+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.363884+0000) 2019-10-13T10:39:44.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:44.552+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.556412+0000) 2019-10-13T10:39:44.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:44.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.556950+0000) 2019-10-13T10:39:44.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:44.552+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.557704+0000) 2019-10-13T10:39:44.559 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:44.556+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.558111+0000) 2019-10-13T10:39:44.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:44.556+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.558954+0000) 2019-10-13T10:39:44.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:44.553+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.559201+0000) 2019-10-13T10:39:44.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:44.556+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.560101+0000) 2019-10-13T10:39:44.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:44.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:44.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:44.561+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:44.564961+0000) 2019-10-13T10:39:47.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:47.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:47.736968+0000) 2019-10-13T10:39:48.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:48.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:48.507192+0000) 2019-10-13T10:39:49.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:49.564+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.566573+0000) 2019-10-13T10:39:49.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:49.564+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.567437+0000) 2019-10-13T10:39:49.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:49.561+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.567541+0000) 2019-10-13T10:39:49.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:49.564+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.568921+0000) 2019-10-13T10:39:49.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:49.564+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.569439+0000) 2019-10-13T10:39:49.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:49.568+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.569933+0000) 2019-10-13T10:39:49.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:49.568+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.570530+0000) 2019-10-13T10:39:49.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:49.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:49.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:49.569+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:49.574522+0000) 2019-10-13T10:39:50.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:50.545+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:50.548993+0000) 2019-10-13T10:39:51.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:51.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:51.213357+0000) 2019-10-13T10:39:51.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:51.232+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:51.235483+0000) 2019-10-13T10:39:51.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:51.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:51.759230+0000) 2019-10-13T10:39:54.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:54.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.077839+0000) 2019-10-13T10:39:54.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:54.360+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.364041+0000) 2019-10-13T10:39:54.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:54.569+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.575077+0000) 2019-10-13T10:39:54.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:54.572+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.575929+0000) 2019-10-13T10:39:54.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:54.572+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.576514+0000) 2019-10-13T10:39:54.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:54.572+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.576906+0000) 2019-10-13T10:39:54.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:54.572+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.577490+0000) 2019-10-13T10:39:54.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:54.576+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.577975+0000) 2019-10-13T10:39:54.580 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:54.576+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.579713+0000) 2019-10-13T10:39:54.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:54.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:54.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:54.581+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:54.584999+0000) 2019-10-13T10:39:57.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:57.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:57.737207+0000) 2019-10-13T10:39:58.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:58.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:58.507384+0000) 2019-10-13T10:39:59.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:39:59.580+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.585666+0000) 2019-10-13T10:39:59.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:39:59.584+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.586156+0000) 2019-10-13T10:39:59.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:39:59.584+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.586871+0000) 2019-10-13T10:39:59.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:39:59.584+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.586951+0000) 2019-10-13T10:39:59.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:39:59.584+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.587166+0000) 2019-10-13T10:39:59.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:39:59.581+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.587187+0000) 2019-10-13T10:39:59.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:39:59.584+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.587988+0000) 2019-10-13T10:39:59.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:39:59.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:39:59.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:39:59.589+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:39:59.594037+0000) 2019-10-13T10:40:00.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:00.549+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:00.555421+0000) 2019-10-13T10:40:01.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:01.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:01.213584+0000) 2019-10-13T10:40:01.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:01.232+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:01.235681+0000) 2019-10-13T10:40:01.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:01.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:01.759424+0000) 2019-10-13T10:40:04.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:04.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.078003+0000) 2019-10-13T10:40:04.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:04.360+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.364267+0000) 2019-10-13T10:40:04.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:04.592+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.596015+0000) 2019-10-13T10:40:04.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:04.592+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.596324+0000) 2019-10-13T10:40:04.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:04.592+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.597372+0000) 2019-10-13T10:40:04.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:04.596+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.598890+0000) 2019-10-13T10:40:04.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:04.596+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.599190+0000) 2019-10-13T10:40:04.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:04.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:04.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:04.596+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.600587+0000) 2019-10-13T10:40:04.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:04.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.600556+0000) 2019-10-13T10:40:04.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:04.601+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:04.605572+0000) 2019-10-13T10:40:07.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:07.732+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:07.737438+0000) 2019-10-13T10:40:08.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:08.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:08.507614+0000) 2019-10-13T10:40:09.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:09.600+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.602705+0000) 2019-10-13T10:40:09.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:09.600+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.603615+0000) 2019-10-13T10:40:09.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:09.600+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.604432+0000) 2019-10-13T10:40:09.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:09.600+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.605113+0000) 2019-10-13T10:40:09.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:09.600+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.605403+0000) 2019-10-13T10:40:09.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:09.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.605332+0000) 2019-10-13T10:40:09.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:09.604+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.606174+0000) 2019-10-13T10:40:09.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:09.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:09.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:09.609+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:09.612697+0000) 2019-10-13T10:40:10.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:10.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:10.561930+0000) 2019-10-13T10:40:11.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:11.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:11.213789+0000) 2019-10-13T10:40:11.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:11.232+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:11.235875+0000) 2019-10-13T10:40:11.301 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:40:11.301 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:11.305 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:40:11.305 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:11.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:11.757+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:11.759597+0000) 2019-10-13T10:40:14.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:14.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.078258+0000) 2019-10-13T10:40:14.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:14.360+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.364478+0000) 2019-10-13T10:40:14.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:14.608+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.612775+0000) 2019-10-13T10:40:14.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:14.609+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.613694+0000) 2019-10-13T10:40:14.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:14.612+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.614707+0000) 2019-10-13T10:40:14.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:14.612+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.615180+0000) 2019-10-13T10:40:14.616 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:14.612+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.615343+0000) 2019-10-13T10:40:14.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:14.612+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.616066+0000) 2019-10-13T10:40:14.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:14.612+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.616572+0000) 2019-10-13T10:40:14.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:14.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:14.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:14.617+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:14.622676+0000) 2019-10-13T10:40:17.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:17.736+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:17.737668+0000) 2019-10-13T10:40:18.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:18.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:18.507804+0000) 2019-10-13T10:40:19.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:19.617+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.622473+0000) 2019-10-13T10:40:19.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:19.620+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.623530+0000) 2019-10-13T10:40:19.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:19.620+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.623992+0000) 2019-10-13T10:40:19.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:19.620+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.624410+0000) 2019-10-13T10:40:19.625 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:19.620+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.624919+0000) 2019-10-13T10:40:19.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:19.620+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.625363+0000) 2019-10-13T10:40:19.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:19.624+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.626049+0000) 2019-10-13T10:40:19.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:19.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:19.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:19.625+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:19.628961+0000) 2019-10-13T10:40:20.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:20.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:20.568515+0000) 2019-10-13T10:40:21.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:21.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:21.214017+0000) 2019-10-13T10:40:21.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:21.232+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:21.236072+0000) 2019-10-13T10:40:21.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:21.753+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:21.759809+0000) 2019-10-13T10:40:24.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:24.076+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.078473+0000) 2019-10-13T10:40:24.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:24.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.364687+0000) 2019-10-13T10:40:24.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:24.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.631692+0000) 2019-10-13T10:40:24.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:24.628+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.632625+0000) 2019-10-13T10:40:24.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:24.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.632338+0000) 2019-10-13T10:40:24.634 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:24.628+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.633008+0000) 2019-10-13T10:40:24.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:24.629+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.633544+0000) 2019-10-13T10:40:24.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:24.632+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.634169+0000) 2019-10-13T10:40:24.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:24.632+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.635873+0000) 2019-10-13T10:40:24.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:24.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:24.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:24.637+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:24.641407+0000) 2019-10-13T10:40:27.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:27.736+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:27.737908+0000) 2019-10-13T10:40:28.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:28.504+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:28.507987+0000) 2019-10-13T10:40:29.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:29.636+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.639939+0000) 2019-10-13T10:40:29.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:29.637+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.640590+0000) 2019-10-13T10:40:29.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:29.636+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.641431+0000) 2019-10-13T10:40:29.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:29.640+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.643825+0000) 2019-10-13T10:40:29.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:29.640+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.644072+0000) 2019-10-13T10:40:29.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:29.644+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.646112+0000) 2019-10-13T10:40:29.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:29.644+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.646552+0000) 2019-10-13T10:40:29.647 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:29.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:29.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:29.645+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:29.652078+0000) 2019-10-13T10:40:30.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:30.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:30.574882+0000) 2019-10-13T10:40:31.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:31.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:31.214219+0000) 2019-10-13T10:40:31.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:31.231+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:31.236303+0000) 2019-10-13T10:40:31.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:31.757+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:31.760022+0000) 2019-10-13T10:40:34.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:34.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.078669+0000) 2019-10-13T10:40:34.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:34.359+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.364917+0000) 2019-10-13T10:40:34.651 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:34.647+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.650369+0000) 2019-10-13T10:40:34.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:34.647+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.651144+0000) 2019-10-13T10:40:34.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:34.645+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.651394+0000) 2019-10-13T10:40:34.654 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:34.647+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.652398+0000) 2019-10-13T10:40:34.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:34.647+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.652378+0000) 2019-10-13T10:40:34.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:34.647+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.652917+0000) 2019-10-13T10:40:34.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:34.647+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.653240+0000) 2019-10-13T10:40:34.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:34.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:34.661 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:34.653+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:34.660047+0000) 2019-10-13T10:40:37.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:37.735+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:37.738106+0000) 2019-10-13T10:40:38.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:38.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:38.508222+0000) 2019-10-13T10:40:39.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:39.655+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.659595+0000) 2019-10-13T10:40:39.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:39.655+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.660331+0000) 2019-10-13T10:40:39.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:39.655+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.661108+0000) 2019-10-13T10:40:39.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:39.659+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.662107+0000) 2019-10-13T10:40:39.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:39.659+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.662400+0000) 2019-10-13T10:40:39.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:39.657+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.662477+0000) 2019-10-13T10:40:39.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:39.659+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.662789+0000) 2019-10-13T10:40:39.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:39.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:39.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:39.661+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:39.665943+0000) 2019-10-13T10:40:40.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:40.581+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:40.586351+0000) 2019-10-13T10:40:41.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:41.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:41.214417+0000) 2019-10-13T10:40:41.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:41.231+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:41.236456+0000) 2019-10-13T10:40:41.416 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:40:41.416 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:41.420 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:40:41.420 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:40:41.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:41.757+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:41.760299+0000) 2019-10-13T10:40:44.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:44.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.078886+0000) 2019-10-13T10:40:44.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:44.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.365115+0000) 2019-10-13T10:40:44.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:44.663+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.668063+0000) 2019-10-13T10:40:44.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:44.663+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.669007+0000) 2019-10-13T10:40:44.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:44.663+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.669351+0000) 2019-10-13T10:40:44.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:44.667+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.669820+0000) 2019-10-13T10:40:44.671 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:44.667+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.670121+0000) 2019-10-13T10:40:44.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:44.664+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.670336+0000) 2019-10-13T10:40:44.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:44.667+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.671829+0000) 2019-10-13T10:40:44.675 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:44.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:44.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:44.672+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:44.679999+0000) 2019-10-13T10:40:47.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:47.735+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:47.738322+0000) 2019-10-13T10:40:48.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:48.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:48.508442+0000) 2019-10-13T10:40:49.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:49.675+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.677897+0000) 2019-10-13T10:40:49.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:49.675+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.678542+0000) 2019-10-13T10:40:49.680 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:49.675+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.679428+0000) 2019-10-13T10:40:49.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:49.675+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.679720+0000) 2019-10-13T10:40:49.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:49.675+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.680059+0000) 2019-10-13T10:40:49.681 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:49.675+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.680770+0000) 2019-10-13T10:40:49.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:49.676+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.681143+0000) 2019-10-13T10:40:49.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:49.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:49.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:49.680+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:49.686579+0000) 2019-10-13T10:40:50.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:50.588+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:50.592919+0000) 2019-10-13T10:40:51.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:51.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:51.214632+0000) 2019-10-13T10:40:51.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:51.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:51.236657+0000) 2019-10-13T10:40:51.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:51.756+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:51.760509+0000) 2019-10-13T10:40:54.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:54.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.079088+0000) 2019-10-13T10:40:54.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:54.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.365342+0000) 2019-10-13T10:40:54.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:54.683+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.688013+0000) 2019-10-13T10:40:54.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:54.683+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.688192+0000) 2019-10-13T10:40:54.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:54.683+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.688535+0000) 2019-10-13T10:40:54.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:54.683+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.689109+0000) 2019-10-13T10:40:54.691 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:54.687+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.689829+0000) 2019-10-13T10:40:54.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:54.684+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.689847+0000) 2019-10-13T10:40:54.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:54.687+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.690522+0000) 2019-10-13T10:40:54.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:54.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:54.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:54.692+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:54.697134+0000) 2019-10-13T10:40:57.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:57.735+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:57.738544+0000) 2019-10-13T10:40:58.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:58.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:58.508660+0000) 2019-10-13T10:40:59.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:40:59.691+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.696429+0000) 2019-10-13T10:40:59.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:40:59.691+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.696448+0000) 2019-10-13T10:40:59.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:40:59.691+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.697180+0000) 2019-10-13T10:40:59.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:40:59.691+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.697547+0000) 2019-10-13T10:40:59.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:40:59.692+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.698549+0000) 2019-10-13T10:40:59.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:40:59.695+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.699519+0000) 2019-10-13T10:40:59.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:40:59.695+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.700125+0000) 2019-10-13T10:40:59.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:40:59.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:40:59.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:40:59.700+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:40:59.707022+0000) 2019-10-13T10:41:00.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:00.600+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:00.604384+0000) 2019-10-13T10:41:01.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:01.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:01.214850+0000) 2019-10-13T10:41:01.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:01.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:01.236881+0000) 2019-10-13T10:41:01.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:01.756+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:01.760709+0000) 2019-10-13T10:41:04.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:04.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.079315+0000) 2019-10-13T10:41:04.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:04.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.365492+0000) 2019-10-13T10:41:04.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:04.700+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.705025+0000) 2019-10-13T10:41:04.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:04.703+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.705993+0000) 2019-10-13T10:41:04.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:04.703+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.706563+0000) 2019-10-13T10:41:04.708 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:04.703+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.707968+0000) 2019-10-13T10:41:04.709 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:04.703+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.708782+0000) 2019-10-13T10:41:04.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:04.707+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.709963+0000) 2019-10-13T10:41:04.711 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:04.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.710352+0000) 2019-10-13T10:41:04.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:04.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:04.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:04.708+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:04.714766+0000) 2019-10-13T10:41:07.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:07.735+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:07.738746+0000) 2019-10-13T10:41:08.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:08.503+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:08.508858+0000) 2019-10-13T10:41:09.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:09.711+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.716887+0000) 2019-10-13T10:41:09.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:09.711+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.717288+0000) 2019-10-13T10:41:09.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:09.712+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.718791+0000) 2019-10-13T10:41:09.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:09.715+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.720698+0000) 2019-10-13T10:41:09.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:09.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:09.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:09.719+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.722005+0000) 2019-10-13T10:41:09.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:09.719+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.723177+0000) 2019-10-13T10:41:09.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:09.719+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.723750+0000) 2019-10-13T10:41:09.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:09.724+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:09.730554+0000) 2019-10-13T10:41:10.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:10.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:10.610898+0000) 2019-10-13T10:41:11.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:11.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:11.215070+0000) 2019-10-13T10:41:11.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:11.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:11.237093+0000) 2019-10-13T10:41:11.547 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:41:11.547 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:11.551 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:41:11.551 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:11.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:11.756+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:11.760901+0000) 2019-10-13T10:41:14.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:14.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.079510+0000) 2019-10-13T10:41:14.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:14.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.365689+0000) 2019-10-13T10:41:14.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:14.720+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.724173+0000) 2019-10-13T10:41:14.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:14.719+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.725014+0000) 2019-10-13T10:41:14.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:14.723+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.727335+0000) 2019-10-13T10:41:14.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:14.723+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.728566+0000) 2019-10-13T10:41:14.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:14.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:14.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:14.727+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.730666+0000) 2019-10-13T10:41:14.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:14.727+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.731019+0000) 2019-10-13T10:41:14.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:14.727+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.731991+0000) 2019-10-13T10:41:14.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:14.728+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:14.732933+0000) 2019-10-13T10:41:17.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:17.735+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:17.738950+0000) 2019-10-13T10:41:18.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:18.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:18.509117+0000) 2019-10-13T10:41:19.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:19.731+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.734352+0000) 2019-10-13T10:41:19.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:19.728+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.734652+0000) 2019-10-13T10:41:19.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:19.731+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.735368+0000) 2019-10-13T10:41:19.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:19.731+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.736075+0000) 2019-10-13T10:41:19.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:19.731+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.736205+0000) 2019-10-13T10:41:19.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:19.731+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.736382+0000) 2019-10-13T10:41:19.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:19.731+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.736788+0000) 2019-10-13T10:41:19.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:19.735+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:19.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:19.740+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:19.745267+0000) 2019-10-13T10:41:20.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:20.616+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:20.617432+0000) 2019-10-13T10:41:21.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:21.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:21.215272+0000) 2019-10-13T10:41:21.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:21.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:21.237292+0000) 2019-10-13T10:41:21.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:21.760+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:21.761109+0000) 2019-10-13T10:41:24.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:24.075+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.079710+0000) 2019-10-13T10:41:24.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:24.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.365921+0000) 2019-10-13T10:41:24.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:24.739+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.742918+0000) 2019-10-13T10:41:24.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:24.739+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.745552+0000) 2019-10-13T10:41:24.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:24.743+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.745965+0000) 2019-10-13T10:41:24.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:24.740+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.745984+0000) 2019-10-13T10:41:24.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:24.743+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.747824+0000) 2019-10-13T10:41:24.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:24.743+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.747288+0000) 2019-10-13T10:41:24.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:24.743+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.748270+0000) 2019-10-13T10:41:24.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:24.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:24.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:24.748+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:24.753694+0000) 2019-10-13T10:41:27.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:27.738+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:27.739173+0000) 2019-10-13T10:41:28.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:28.502+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:28.509326+0000) 2019-10-13T10:41:29.753 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:29.748+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.752078+0000) 2019-10-13T10:41:29.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:29.746+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.752874+0000) 2019-10-13T10:41:29.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:29.746+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.753228+0000) 2019-10-13T10:41:29.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:29.750+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.754417+0000) 2019-10-13T10:41:29.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:29.750+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.754601+0000) 2019-10-13T10:41:29.756 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:29.750+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.754818+0000) 2019-10-13T10:41:29.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:29.750+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.755221+0000) 2019-10-13T10:41:29.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:29.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:29.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:29.752+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:29.760008+0000) 2019-10-13T10:41:30.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:30.616+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:30.623956+0000) 2019-10-13T10:41:31.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:31.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:31.215473+0000) 2019-10-13T10:41:31.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:31.230+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:31.237492+0000) 2019-10-13T10:41:31.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:31.760+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:31.761328+0000) 2019-10-13T10:41:34.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:34.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.079924+0000) 2019-10-13T10:41:34.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:34.362+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.366134+0000) 2019-10-13T10:41:34.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:34.758+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.762301+0000) 2019-10-13T10:41:34.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:34.756+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.762426+0000) 2019-10-13T10:41:34.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:34.758+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.763085+0000) 2019-10-13T10:41:34.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:34.758+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.763385+0000) 2019-10-13T10:41:34.765 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:34.758+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.764204+0000) 2019-10-13T10:41:34.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:34.758+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.764732+0000) 2019-10-13T10:41:34.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:34.758+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.765189+0000) 2019-10-13T10:41:34.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:34.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:34.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:34.764+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:34.771621+0000) 2019-10-13T10:41:37.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:37.738+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:37.739372+0000) 2019-10-13T10:41:38.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:38.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:38.509522+0000) 2019-10-13T10:41:39.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:39.766+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.770879+0000) 2019-10-13T10:41:39.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:39.766+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.772037+0000) 2019-10-13T10:41:39.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:39.766+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.771713+0000) 2019-10-13T10:41:39.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:39.768+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.771913+0000) 2019-10-13T10:41:39.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:39.770+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.774698+0000) 2019-10-13T10:41:39.776 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:39.770+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.775940+0000) 2019-10-13T10:41:39.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:39.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:39.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:39.770+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.777655+0000) 2019-10-13T10:41:39.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:39.776+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:39.781593+0000) 2019-10-13T10:41:40.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:40.628+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:40.630545+0000) 2019-10-13T10:41:41.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:41.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:41.215737+0000) 2019-10-13T10:41:41.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:41.234+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:41.237721+0000) 2019-10-13T10:41:41.694 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:41:41.694 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:41.698 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:41:41.698 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:41:41.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:41.756+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:41.761533+0000) 2019-10-13T10:41:44.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:44.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.080126+0000) 2019-10-13T10:41:44.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:44.362+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.366334+0000) 2019-10-13T10:41:44.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:44.774+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.780343+0000) 2019-10-13T10:41:44.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:44.774+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.781313+0000) 2019-10-13T10:41:44.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:44.778+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.782081+0000) 2019-10-13T10:41:44.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:44.780+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.782705+0000) 2019-10-13T10:41:44.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:44.778+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.783355+0000) 2019-10-13T10:41:44.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:44.778+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.783860+0000) 2019-10-13T10:41:44.785 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:44.778+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.783976+0000) 2019-10-13T10:41:44.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:44.782+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:44.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:44.788+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:44.788594+0000) 2019-10-13T10:41:47.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:47.738+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:47.739572+0000) 2019-10-13T10:41:48.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:48.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:48.509733+0000) 2019-10-13T10:41:49.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:49.786+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.789619+0000) 2019-10-13T10:41:49.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:49.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.791798+0000) 2019-10-13T10:41:49.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:49.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.794161+0000) 2019-10-13T10:41:49.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:49.790+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.794738+0000) 2019-10-13T10:41:49.796 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:49.790+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.795546+0000) 2019-10-13T10:41:49.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:49.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.795841+0000) 2019-10-13T10:41:49.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:49.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:49.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:49.790+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.796238+0000) 2019-10-13T10:41:49.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:49.800+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:49.800591+0000) 2019-10-13T10:41:50.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:50.635+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:50.637042+0000) 2019-10-13T10:41:51.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:51.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:51.215941+0000) 2019-10-13T10:41:51.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:51.234+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:51.238059+0000) 2019-10-13T10:41:51.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:51.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:51.761743+0000) 2019-10-13T10:41:54.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:54.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.080330+0000) 2019-10-13T10:41:54.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:54.362+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.366529+0000) 2019-10-13T10:41:54.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:54.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.798678+0000) 2019-10-13T10:41:54.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:54.794+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.800957+0000) 2019-10-13T10:41:54.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:54.794+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.801244+0000) 2019-10-13T10:41:54.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:54.799+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.801713+0000) 2019-10-13T10:41:54.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:54.798+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.802782+0000) 2019-10-13T10:41:54.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:54.798+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.803175+0000) 2019-10-13T10:41:54.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:54.798+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.805679+0000) 2019-10-13T10:41:54.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:54.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:54.815 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:54.811+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:54.814009+0000) 2019-10-13T10:41:57.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:57.738+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:57.739773+0000) 2019-10-13T10:41:58.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:58.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:58.509966+0000) 2019-10-13T10:41:59.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:41:59.807+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.809847+0000) 2019-10-13T10:41:59.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:41:59.806+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.812379+0000) 2019-10-13T10:41:59.813 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:41:59.806+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.813107+0000) 2019-10-13T10:41:59.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:41:59.810+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.813896+0000) 2019-10-13T10:41:59.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:41:59.810+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.814351+0000) 2019-10-13T10:41:59.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:41:59.810+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.814777+0000) 2019-10-13T10:41:59.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:41:59.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:41:59.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:41:59.810+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.816526+0000) 2019-10-13T10:41:59.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:41:59.819+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:41:59.821644+0000) 2019-10-13T10:42:00.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:00.639+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:00.643537+0000) 2019-10-13T10:42:01.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:01.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:01.216174+0000) 2019-10-13T10:42:01.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:01.234+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:01.238258+0000) 2019-10-13T10:42:01.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:01.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:01.761950+0000) 2019-10-13T10:42:04.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:04.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.080493+0000) 2019-10-13T10:42:04.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:04.366+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.366730+0000) 2019-10-13T10:42:04.820 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:04.814+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.819604+0000) 2019-10-13T10:42:04.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:04.814+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.820038+0000) 2019-10-13T10:42:04.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:04.814+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.821251+0000) 2019-10-13T10:42:04.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:04.818+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.822409+0000) 2019-10-13T10:42:04.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:04.818+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.822748+0000) 2019-10-13T10:42:04.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:04.819+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.823898+0000) 2019-10-13T10:42:04.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:04.822+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.825735+0000) 2019-10-13T10:42:04.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:04.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:04.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:04.823+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:04.828062+0000) 2019-10-13T10:42:07.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:07.738+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:07.739971+0000) 2019-10-13T10:42:08.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:08.506+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:08.510162+0000) 2019-10-13T10:42:09.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:09.826+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.829808+0000) 2019-10-13T10:42:09.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:09.826+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.830660+0000) 2019-10-13T10:42:09.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:09.826+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.831089+0000) 2019-10-13T10:42:09.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:09.826+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.831424+0000) 2019-10-13T10:42:09.832 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:09.826+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.831640+0000) 2019-10-13T10:42:09.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:09.827+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.832017+0000) 2019-10-13T10:42:09.834 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:09.826+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.833300+0000) 2019-10-13T10:42:09.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:09.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:09.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:09.835+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:09.836520+0000) 2019-10-13T10:42:10.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:10.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:10.650034+0000) 2019-10-13T10:42:11.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:11.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:11.216379+0000) 2019-10-13T10:42:11.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:11.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:11.238488+0000) 2019-10-13T10:42:11.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:11.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:11.762165+0000) 2019-10-13T10:42:11.767 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:42:11.767 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:11.770 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:42:11.770 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:14.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:14.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.080716+0000) 2019-10-13T10:42:14.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:14.366+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.366926+0000) 2019-10-13T10:42:14.839 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:14.834+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.838695+0000) 2019-10-13T10:42:14.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:14.834+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.839156+0000) 2019-10-13T10:42:14.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:14.834+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.839690+0000) 2019-10-13T10:42:14.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:14.834+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.840616+0000) 2019-10-13T10:42:14.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:14.834+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.841049+0000) 2019-10-13T10:42:14.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:14.839+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.841216+0000) 2019-10-13T10:42:14.843 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:14.838+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.842302+0000) 2019-10-13T10:42:14.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:14.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:14.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:14.843+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:14.846136+0000) 2019-10-13T10:42:17.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:17.734+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:17.740170+0000) 2019-10-13T10:42:18.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:18.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:18.510405+0000) 2019-10-13T10:42:19.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:19.842+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.848395+0000) 2019-10-13T10:42:19.850 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:19.842+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.848848+0000) 2019-10-13T10:42:19.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:19.842+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.849197+0000) 2019-10-13T10:42:19.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:19.846+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.850122+0000) 2019-10-13T10:42:19.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:19.846+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.850839+0000) 2019-10-13T10:42:19.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:19.847+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.851341+0000) 2019-10-13T10:42:19.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:19.846+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.851307+0000) 2019-10-13T10:42:19.854 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:19.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:19.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:19.855+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:19.857184+0000) 2019-10-13T10:42:20.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:20.655+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:20.656449+0000) 2019-10-13T10:42:21.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:21.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:21.216583+0000) 2019-10-13T10:42:21.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:21.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:21.238651+0000) 2019-10-13T10:42:21.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:21.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:21.762387+0000) 2019-10-13T10:42:24.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:24.074+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.080910+0000) 2019-10-13T10:42:24.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:24.362+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.367126+0000) 2019-10-13T10:42:24.858 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:24.853+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.857369+0000) 2019-10-13T10:42:24.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:24.853+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.857966+0000) 2019-10-13T10:42:24.860 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:24.853+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.858838+0000) 2019-10-13T10:42:24.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:24.853+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.859241+0000) 2019-10-13T10:42:24.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:24.853+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.859650+0000) 2019-10-13T10:42:24.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:24.853+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.860051+0000) 2019-10-13T10:42:24.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:24.859+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.860616+0000) 2019-10-13T10:42:24.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:24.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:24.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:24.863+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:24.864421+0000) 2019-10-13T10:42:27.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:27.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:27.740374+0000) 2019-10-13T10:42:28.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:28.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:28.510604+0000) 2019-10-13T10:42:29.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:29.861+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.868045+0000) 2019-10-13T10:42:29.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:29.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.868645+0000) 2019-10-13T10:42:29.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:29.861+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.869299+0000) 2019-10-13T10:42:29.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:29.865+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.869895+0000) 2019-10-13T10:42:29.871 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:29.865+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.870192+0000) 2019-10-13T10:42:29.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:29.867+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.870700+0000) 2019-10-13T10:42:29.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:29.865+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.871510+0000) 2019-10-13T10:42:29.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:29.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:29.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:29.871+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:29.873309+0000) 2019-10-13T10:42:30.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:30.659+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:30.663018+0000) 2019-10-13T10:42:31.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:31.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:31.216781+0000) 2019-10-13T10:42:31.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:31.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:31.238874+0000) 2019-10-13T10:42:31.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:31.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:31.762576+0000) 2019-10-13T10:42:34.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:34.073+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.081119+0000) 2019-10-13T10:42:34.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:34.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.367342+0000) 2019-10-13T10:42:34.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:34.869+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.875761+0000) 2019-10-13T10:42:34.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:34.869+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.876604+0000) 2019-10-13T10:42:34.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:34.875+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.876469+0000) 2019-10-13T10:42:34.878 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:34.869+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.877290+0000) 2019-10-13T10:42:34.878 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:34.873+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.878047+0000) 2019-10-13T10:42:34.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:34.873+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.878350+0000) 2019-10-13T10:42:34.880 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:34.873+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.879582+0000) 2019-10-13T10:42:34.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:34.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122583 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:34.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:34.883+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:34.884715+0000) 2019-10-13T10:42:37.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:37.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:37.740577+0000) 2019-10-13T10:42:38.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:38.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:38.510802+0000) 2019-10-13T10:42:39.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:39.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.884765+0000) 2019-10-13T10:42:39.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:39.885+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.886587+0000) 2019-10-13T10:42:39.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:39.885+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.889317+0000) 2019-10-13T10:42:39.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:39.885+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.889664+0000) 2019-10-13T10:42:39.891 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:39.889+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.890298+0000) 2019-10-13T10:42:39.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:39.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:39.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:39.887+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.890448+0000) 2019-10-13T10:42:39.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:39.889+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.891411+0000) 2019-10-13T10:42:39.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:39.891+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:39.892918+0000) 2019-10-13T10:42:40.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:40.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:40.669473+0000) 2019-10-13T10:42:41.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:41.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:41.216983+0000) 2019-10-13T10:42:41.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:41.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:41.239077+0000) 2019-10-13T10:42:41.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:41.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:41.762798+0000) 2019-10-13T10:42:41.895 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:42:41.895 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:41.899 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:42:41.899 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:42:44.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:44.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.081313+0000) 2019-10-13T10:42:44.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:44.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.367543+0000) 2019-10-13T10:42:44.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:44.891+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.893470+0000) 2019-10-13T10:42:44.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:44.893+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.894774+0000) 2019-10-13T10:42:44.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:44.893+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.896472+0000) 2019-10-13T10:42:44.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:44.897+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.898378+0000) 2019-10-13T10:42:44.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:44.897+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122645 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:44.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:44.897+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.900209+0000) 2019-10-13T10:42:44.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:44.899+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.900241+0000) 2019-10-13T10:42:44.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:44.897+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.900995+0000) 2019-10-13T10:42:44.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:44.901+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:44.902064+0000) 2019-10-13T10:42:47.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:47.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:47.740806+0000) 2019-10-13T10:42:48.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:48.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:48.511026+0000) 2019-10-13T10:42:49.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:49.899+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.902045+0000) 2019-10-13T10:42:49.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:49.901+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.903030+0000) 2019-10-13T10:42:49.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:49.901+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.903770+0000) 2019-10-13T10:42:49.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:49.905+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.906187+0000) 2019-10-13T10:42:49.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:49.905+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.906586+0000) 2019-10-13T10:42:49.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:49.905+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.907342+0000) 2019-10-13T10:42:49.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:49.905+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.907884+0000) 2019-10-13T10:42:49.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:49.905+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:49.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:49.911+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:49.913661+0000) 2019-10-13T10:42:50.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:50.671+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:50.675974+0000) 2019-10-13T10:42:51.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:51.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:51.217183+0000) 2019-10-13T10:42:51.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:51.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:51.239281+0000) 2019-10-13T10:42:51.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:51.759+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:51.762997+0000) 2019-10-13T10:42:54.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:54.077+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.081539+0000) 2019-10-13T10:42:54.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:54.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.367773+0000) 2019-10-13T10:42:54.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:54.909+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.912256+0000) 2019-10-13T10:42:54.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:54.913+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.914165+0000) 2019-10-13T10:42:54.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:54.913+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.914443+0000) 2019-10-13T10:42:54.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:54.913+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.915080+0000) 2019-10-13T10:42:54.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:54.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.915487+0000) 2019-10-13T10:42:54.917 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:54.913+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.916022+0000) 2019-10-13T10:42:54.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:54.913+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.916330+0000) 2019-10-13T10:42:54.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:54.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:54.924 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:54.919+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:54.923154+0000) 2019-10-13T10:42:57.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:57.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:57.741011+0000) 2019-10-13T10:42:58.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:58.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:58.511222+0000) 2019-10-13T10:42:59.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:42:59.921+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.923620+0000) 2019-10-13T10:42:59.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:42:59.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.924376+0000) 2019-10-13T10:42:59.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:42:59.921+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.924799+0000) 2019-10-13T10:42:59.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:42:59.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.926289+0000) 2019-10-13T10:42:59.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:42:59.925+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:42:59.929 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:42:59.925+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.928571+0000) 2019-10-13T10:42:59.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:42:59.925+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.929049+0000) 2019-10-13T10:42:59.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:42:59.925+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.929295+0000) 2019-10-13T10:42:59.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:42:59.926+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:42:59.929687+0000) 2019-10-13T10:43:00.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:00.678+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:00.681800+0000) 2019-10-13T10:43:01.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:01.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:01.217390+0000) 2019-10-13T10:43:01.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:01.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:01.239487+0000) 2019-10-13T10:43:01.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:01.758+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:01.763241+0000) 2019-10-13T10:43:04.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:04.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.081736+0000) 2019-10-13T10:43:04.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:04.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.367970+0000) 2019-10-13T10:43:04.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:04.926+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.930365+0000) 2019-10-13T10:43:04.932 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:04.929+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.931750+0000) 2019-10-13T10:43:04.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:04.929+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.932252+0000) 2019-10-13T10:43:04.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:04.929+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.933282+0000) 2019-10-13T10:43:04.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:04.929+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.933623+0000) 2019-10-13T10:43:04.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:04.933+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.933878+0000) 2019-10-13T10:43:04.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:04.933+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.934695+0000) 2019-10-13T10:43:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:04.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:04.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:04.938+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:04.940473+0000) 2019-10-13T10:43:07.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:07.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:07.741218+0000) 2019-10-13T10:43:08.512 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:08.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:08.511416+0000) 2019-10-13T10:43:09.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:09.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.941236+0000) 2019-10-13T10:43:09.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:09.937+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.941563+0000) 2019-10-13T10:43:09.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:09.941+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.942642+0000) 2019-10-13T10:43:09.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:09.938+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.943169+0000) 2019-10-13T10:43:09.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:09.941+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.943705+0000) 2019-10-13T10:43:09.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:09.941+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.944273+0000) 2019-10-13T10:43:09.946 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:09.941+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.944837+0000) 2019-10-13T10:43:09.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:09.946+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:09.949233+0000) 2019-10-13T10:43:09.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:09.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:10.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:10.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:10.688358+0000) 2019-10-13T10:43:11.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:11.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.217594+0000) 2019-10-13T10:43:11.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:11.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.239695+0000) 2019-10-13T10:43:11.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:11.758+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:11.763432+0000) 2019-10-13T10:43:12.158 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:43:12.158 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:12.161 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:43:12.161 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:14.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:14.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.081986+0000) 2019-10-13T10:43:14.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:14.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.368166+0000) 2019-10-13T10:43:14.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:14.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.950694+0000) 2019-10-13T10:43:14.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:14.949+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.951329+0000) 2019-10-13T10:43:14.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:14.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.951919+0000) 2019-10-13T10:43:14.953 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:14.949+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.952499+0000) 2019-10-13T10:43:14.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:14.949+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.952821+0000) 2019-10-13T10:43:14.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:14.949+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.953379+0000) 2019-10-13T10:43:14.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:14.953+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.954099+0000) 2019-10-13T10:43:14.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:14.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:14.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:14.954+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:14.957725+0000) 2019-10-13T10:43:17.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:17.737+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:17.741439+0000) 2019-10-13T10:43:18.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:18.509+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:18.511616+0000) 2019-10-13T10:43:19.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:19.957+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.960358+0000) 2019-10-13T10:43:19.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:19.961+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.961878+0000) 2019-10-13T10:43:19.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:19.961+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.962591+0000) 2019-10-13T10:43:19.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:19.961+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.962896+0000) 2019-10-13T10:43:19.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:19.958+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.962932+0000) 2019-10-13T10:43:19.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:19.961+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.963956+0000) 2019-10-13T10:43:19.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:19.961+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.964620+0000) 2019-10-13T10:43:19.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:19.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:19.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:19.962+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:19.967232+0000) 2019-10-13T10:43:20.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:20.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:20.694908+0000) 2019-10-13T10:43:21.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:21.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.217830+0000) 2019-10-13T10:43:21.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:21.237+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.239898+0000) 2019-10-13T10:43:21.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:21.762+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:21.763657+0000) 2019-10-13T10:43:24.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:24.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.082186+0000) 2019-10-13T10:43:24.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:24.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.368367+0000) 2019-10-13T10:43:24.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:24.968+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.972283+0000) 2019-10-13T10:43:24.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:24.968+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.973150+0000) 2019-10-13T10:43:24.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:24.968+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.973507+0000) 2019-10-13T10:43:24.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:24.972+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.974795+0000) 2019-10-13T10:43:24.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:24.972+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.975164+0000) 2019-10-13T10:43:24.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:24.972+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.975452+0000) 2019-10-13T10:43:24.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:24.970+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.975589+0000) 2019-10-13T10:43:24.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:24.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:24.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:24.978+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:24.980535+0000) 2019-10-13T10:43:27.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:27.736+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:27.741636+0000) 2019-10-13T10:43:28.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:28.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:28.511808+0000) 2019-10-13T10:43:29.981 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:29.976+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.979367+0000) 2019-10-13T10:43:29.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:29.974+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.979772+0000) 2019-10-13T10:43:29.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:29.976+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.980573+0000) 2019-10-13T10:43:29.982 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:29.976+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.980939+0000) 2019-10-13T10:43:29.982 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:29.976+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.981331+0000) 2019-10-13T10:43:29.983 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:29.980+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.982364+0000) 2019-10-13T10:43:29.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:29.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.984028+0000) 2019-10-13T10:43:29.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:29.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:29.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:29.986+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:29.988789+0000) 2019-10-13T10:43:30.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:30.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:30.701540+0000) 2019-10-13T10:43:31.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:31.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.217974+0000) 2019-10-13T10:43:31.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:31.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.240093+0000) 2019-10-13T10:43:31.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:31.758+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:31.763849+0000) 2019-10-13T10:43:34.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:34.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.082383+0000) 2019-10-13T10:43:34.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:34.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.368525+0000) 2019-10-13T10:43:34.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:34.984+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.988751+0000) 2019-10-13T10:43:34.990 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:34.984+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.988912+0000) 2019-10-13T10:43:34.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:34.988+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.991525+0000) 2019-10-13T10:43:34.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:34.988+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.992023+0000) 2019-10-13T10:43:34.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:34.988+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.992662+0000) 2019-10-13T10:43:34.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:34.988+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.992910+0000) 2019-10-13T10:43:34.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:34.990+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:34.993798+0000) 2019-10-13T10:43:34.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:34.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:35.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:34.998+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:35.000349+0000) 2019-10-13T10:43:37.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:37.740+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:37.741840+0000) 2019-10-13T10:43:38.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:38.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:38.512038+0000) 2019-10-13T10:43:39.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:39.996+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.998728+0000) 2019-10-13T10:43:40.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:39.994+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:39.999856+0000) 2019-10-13T10:43:40.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:39.996+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.000458+0000) 2019-10-13T10:43:40.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:39.996+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.000898+0000) 2019-10-13T10:43:40.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:39.996+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.001330+0000) 2019-10-13T10:43:40.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:40.000+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.001773+0000) 2019-10-13T10:43:40.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:40.000+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.002434+0000) 2019-10-13T10:43:40.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:40.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:40.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:40.002+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.007475+0000) 2019-10-13T10:43:40.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:40.706+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:40.708137+0000) 2019-10-13T10:43:41.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:41.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.218170+0000) 2019-10-13T10:43:41.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:41.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.240251+0000) 2019-10-13T10:43:41.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:41.758+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:41.764044+0000) 2019-10-13T10:43:42.201 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:43:42.201 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:42.205 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:43:42.205 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:43:44.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:44.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.082611+0000) 2019-10-13T10:43:44.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:44.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:44.368752+0000) 2019-10-13T10:43:45.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:45.004+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.007103+0000) 2019-10-13T10:43:45.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:45.004+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.007352+0000) 2019-10-13T10:43:45.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:45.004+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.009415+0000) 2019-10-13T10:43:45.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:45.006+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.009558+0000) 2019-10-13T10:43:45.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:45.008+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.010366+0000) 2019-10-13T10:43:45.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:45.008+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.009840+0000) 2019-10-13T10:43:45.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:45.008+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.010837+0000) 2019-10-13T10:43:45.014 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:45.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:45.016 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:45.010+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:45.014752+0000) 2019-10-13T10:43:47.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:47.740+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:47.742051+0000) 2019-10-13T10:43:48.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:48.508+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:48.512259+0000) 2019-10-13T10:43:50.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:50.016+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.017816+0000) 2019-10-13T10:43:50.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:50.016+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.018044+0000) 2019-10-13T10:43:50.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:50.016+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.018504+0000) 2019-10-13T10:43:50.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:50.016+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.019490+0000) 2019-10-13T10:43:50.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:50.016+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.019753+0000) 2019-10-13T10:43:50.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:50.014+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.019295+0000) 2019-10-13T10:43:50.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:50.016+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.020643+0000) 2019-10-13T10:43:50.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:50.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:50.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:50.022+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.027538+0000) 2019-10-13T10:43:50.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:50.710+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:50.714404+0000) 2019-10-13T10:43:51.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:51.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.218382+0000) 2019-10-13T10:43:51.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:51.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.240459+0000) 2019-10-13T10:43:51.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:51.762+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:51.764298+0000) 2019-10-13T10:43:54.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:54.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.082771+0000) 2019-10-13T10:43:54.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:54.368+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:54.368967+0000) 2019-10-13T10:43:55.029 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:43:55.024+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.028789+0000) 2019-10-13T10:43:55.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:43:55.024+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.029630+0000) 2019-10-13T10:43:55.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:43:55.028+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.029877+0000) 2019-10-13T10:43:55.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:43:55.028+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.030302+0000) 2019-10-13T10:43:55.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:43:55.026+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.031550+0000) 2019-10-13T10:43:55.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:55.028+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.032504+0000) 2019-10-13T10:43:55.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:43:55.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:43:55.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:55.028+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.032815+0000) 2019-10-13T10:43:55.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:43:55.038+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:55.040319+0000) 2019-10-13T10:43:57.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:43:57.740+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:57.742215+0000) 2019-10-13T10:43:58.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:43:58.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:43:58.512458+0000) 2019-10-13T10:44:00.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:00.032+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.037277+0000) 2019-10-13T10:44:00.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:00.034+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.037361+0000) 2019-10-13T10:44:00.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:00.036+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.038289+0000) 2019-10-13T10:44:00.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:00.036+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.040365+0000) 2019-10-13T10:44:00.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:00.036+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.041475+0000) 2019-10-13T10:44:00.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:00.036+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.041588+0000) 2019-10-13T10:44:00.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:00.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:00.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:00.040+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.042694+0000) 2019-10-13T10:44:00.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:00.042+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.045706+0000) 2019-10-13T10:44:00.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:00.718+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:00.720834+0000) 2019-10-13T10:44:01.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:01.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.218616+0000) 2019-10-13T10:44:01.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:01.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.240689+0000) 2019-10-13T10:44:01.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:01.762+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:01.764496+0000) 2019-10-13T10:44:04.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:04.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.082942+0000) 2019-10-13T10:44:04.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:04.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:04.369135+0000) 2019-10-13T10:44:05.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:05.040+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.045180+0000) 2019-10-13T10:44:05.049 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:05.041+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.047596+0000) 2019-10-13T10:44:05.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:05.044+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.047810+0000) 2019-10-13T10:44:05.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:05.044+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.048695+0000) 2019-10-13T10:44:05.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:05.044+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.049573+0000) 2019-10-13T10:44:05.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:05.048+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.049948+0000) 2019-10-13T10:44:05.051 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:05.048+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.050517+0000) 2019-10-13T10:44:05.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:05.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:05.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:05.049+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:05.055573+0000) 2019-10-13T10:44:07.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:07.740+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:07.742444+0000) 2019-10-13T10:44:08.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:08.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:08.512651+0000) 2019-10-13T10:44:10.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:10.049+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.054942+0000) 2019-10-13T10:44:10.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:10.052+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.054883+0000) 2019-10-13T10:44:10.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:10.052+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.055803+0000) 2019-10-13T10:44:10.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:10.052+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.057223+0000) 2019-10-13T10:44:10.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:10.056+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.057820+0000) 2019-10-13T10:44:10.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:10.056+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.057876+0000) 2019-10-13T10:44:10.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:10.056+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.058664+0000) 2019-10-13T10:44:10.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:10.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123169 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:10.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:10.057+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.062611+0000) 2019-10-13T10:44:10.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:10.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:10.727526+0000) 2019-10-13T10:44:11.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:11.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.218872+0000) 2019-10-13T10:44:11.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:11.236+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.240884+0000) 2019-10-13T10:44:11.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:11.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:11.764763+0000) 2019-10-13T10:44:12.370 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:44:12.370 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:12.376 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:44:12.376 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:14.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:14.080+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.083097+0000) 2019-10-13T10:44:14.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:14.364+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:14.369351+0000) 2019-10-13T10:44:15.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:15.060+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.065601+0000) 2019-10-13T10:44:15.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:15.065+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.068718+0000) 2019-10-13T10:44:15.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:15.068+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.070747+0000) 2019-10-13T10:44:15.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:15.068+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:15.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:15.065+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.071447+0000) 2019-10-13T10:44:15.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:15.068+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.072228+0000) 2019-10-13T10:44:15.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:15.068+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.072739+0000) 2019-10-13T10:44:15.074 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:15.068+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.073182+0000) 2019-10-13T10:44:15.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:15.072+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:15.074237+0000) 2019-10-13T10:44:17.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:17.740+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:17.742666+0000) 2019-10-13T10:44:18.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:18.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:18.512883+0000) 2019-10-13T10:44:20.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:20.071+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.074899+0000) 2019-10-13T10:44:20.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:20.071+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.075455+0000) 2019-10-13T10:44:20.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:20.071+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.075833+0000) 2019-10-13T10:44:20.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:20.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.076536+0000) 2019-10-13T10:44:20.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:20.071+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.076663+0000) 2019-10-13T10:44:20.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:20.073+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.077225+0000) 2019-10-13T10:44:20.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:20.075+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.077902+0000) 2019-10-13T10:44:20.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:20.075+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:20.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:20.077+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.084024+0000) 2019-10-13T10:44:20.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:20.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:20.734054+0000) 2019-10-13T10:44:21.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:21.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.219106+0000) 2019-10-13T10:44:21.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:21.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.241115+0000) 2019-10-13T10:44:21.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:21.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:21.764978+0000) 2019-10-13T10:44:24.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:24.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.083269+0000) 2019-10-13T10:44:24.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:24.363+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:24.369547+0000) 2019-10-13T10:44:25.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:25.079+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.083378+0000) 2019-10-13T10:44:25.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:25.081+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.084167+0000) 2019-10-13T10:44:25.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:25.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.084845+0000) 2019-10-13T10:44:25.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:25.079+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.085402+0000) 2019-10-13T10:44:25.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:25.083+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.086063+0000) 2019-10-13T10:44:25.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:25.083+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.086558+0000) 2019-10-13T10:44:25.088 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:25.083+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.087703+0000) 2019-10-13T10:44:25.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:25.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:25.092 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:25.085+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:25.091513+0000) 2019-10-13T10:44:27.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:27.739+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:27.742898+0000) 2019-10-13T10:44:28.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:28.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:28.513076+0000) 2019-10-13T10:44:30.094 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:30.087+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.093419+0000) 2019-10-13T10:44:30.095 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:30.089+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.094034+0000) 2019-10-13T10:44:30.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:30.091+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.094553+0000) 2019-10-13T10:44:30.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:30.091+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.095051+0000) 2019-10-13T10:44:30.096 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:30.091+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.095346+0000) 2019-10-13T10:44:30.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:30.091+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.097361+0000) 2019-10-13T10:44:30.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:30.095+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.098367+0000) 2019-10-13T10:44:30.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:30.095+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:30.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:30.097+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.103435+0000) 2019-10-13T10:44:30.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:30.737+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:30.740545+0000) 2019-10-13T10:44:31.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:31.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.219341+0000) 2019-10-13T10:44:31.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:31.235+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.241322+0000) 2019-10-13T10:44:31.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:31.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:31.765196+0000) 2019-10-13T10:44:34.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:34.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.083432+0000) 2019-10-13T10:44:34.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:34.367+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:34.369773+0000) 2019-10-13T10:44:35.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:35.095+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.101460+0000) 2019-10-13T10:44:35.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:35.099+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.103253+0000) 2019-10-13T10:44:35.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:35.103+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.106344+0000) 2019-10-13T10:44:35.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:35.101+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.106647+0000) 2019-10-13T10:44:35.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:35.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:35.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:35.103+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.108381+0000) 2019-10-13T10:44:35.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:35.103+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.108902+0000) 2019-10-13T10:44:35.110 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:35.103+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.109158+0000) 2019-10-13T10:44:35.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:35.109+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:35.114880+0000) 2019-10-13T10:44:37.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:37.739+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:37.743116+0000) 2019-10-13T10:44:38.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:38.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:38.513286+0000) 2019-10-13T10:44:40.112 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:40.107+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.110912+0000) 2019-10-13T10:44:40.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:40.107+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.111709+0000) 2019-10-13T10:44:40.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:40.107+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.112218+0000) 2019-10-13T10:44:40.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:40.107+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.112517+0000) 2019-10-13T10:44:40.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:40.109+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.113025+0000) 2019-10-13T10:44:40.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:40.107+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.113519+0000) 2019-10-13T10:44:40.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:40.111+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.114035+0000) 2019-10-13T10:44:40.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:40.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:40.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:40.117+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.120904+0000) 2019-10-13T10:44:40.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:40.745+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:40.747006+0000) 2019-10-13T10:44:41.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:41.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.219564+0000) 2019-10-13T10:44:41.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:41.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.241477+0000) 2019-10-13T10:44:41.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:41.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:41.765345+0000) 2019-10-13T10:44:42.492 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:44:42.492 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:42.496 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:44:42.496 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:44:44.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:44.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.083637+0000) 2019-10-13T10:44:44.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:44.367+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:44.370025+0000) 2019-10-13T10:44:45.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:45.115+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.120595+0000) 2019-10-13T10:44:45.122 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:45.115+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.121400+0000) 2019-10-13T10:44:45.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:45.119+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.121924+0000) 2019-10-13T10:44:45.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:45.117+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.121881+0000) 2019-10-13T10:44:45.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:45.119+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.122659+0000) 2019-10-13T10:44:45.124 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:45.119+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.123518+0000) 2019-10-13T10:44:45.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:45.119+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.124027+0000) 2019-10-13T10:44:45.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:45.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:45.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:45.125+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:45.128709+0000) 2019-10-13T10:44:47.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:47.739+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:47.743318+0000) 2019-10-13T10:44:48.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:48.507+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:48.513505+0000) 2019-10-13T10:44:50.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:50.123+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.129505+0000) 2019-10-13T10:44:50.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:50.127+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.130946+0000) 2019-10-13T10:44:50.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:50.125+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.130897+0000) 2019-10-13T10:44:50.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:50.127+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.131388+0000) 2019-10-13T10:44:50.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:50.127+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.131559+0000) 2019-10-13T10:44:50.133 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:50.127+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.132424+0000) 2019-10-13T10:44:50.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:50.127+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.133381+0000) 2019-10-13T10:44:50.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:50.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:50.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:50.129+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.135921+0000) 2019-10-13T10:44:50.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:50.753+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:50.753403+0000) 2019-10-13T10:44:51.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:51.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.219798+0000) 2019-10-13T10:44:51.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:51.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.241682+0000) 2019-10-13T10:44:51.766 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:51.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:51.765556+0000) 2019-10-13T10:44:54.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:54.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.083871+0000) 2019-10-13T10:44:54.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:54.367+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:54.370262+0000) 2019-10-13T10:44:55.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:55.131+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.137364+0000) 2019-10-13T10:44:55.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:44:55.135+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.138003+0000) 2019-10-13T10:44:55.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:55.135+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.139866+0000) 2019-10-13T10:44:55.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:44:55.137+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.140583+0000) 2019-10-13T10:44:55.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:44:55.139+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:44:55.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:44:55.139+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.142656+0000) 2019-10-13T10:44:55.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:44:55.139+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.142861+0000) 2019-10-13T10:44:55.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:44:55.139+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.142968+0000) 2019-10-13T10:44:55.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:44:55.145+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:55.148240+0000) 2019-10-13T10:44:57.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:44:57.739+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:57.743514+0000) 2019-10-13T10:44:58.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:44:58.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:44:58.513730+0000) 2019-10-13T10:45:00.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:00.139+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.145558+0000) 2019-10-13T10:45:00.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:00.143+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.146837+0000) 2019-10-13T10:45:00.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:00.143+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.147147+0000) 2019-10-13T10:45:00.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:00.143+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.148245+0000) 2019-10-13T10:45:00.149 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:00.143+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.148754+0000) 2019-10-13T10:45:00.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:00.145+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.148921+0000) 2019-10-13T10:45:00.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:00.143+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.149637+0000) 2019-10-13T10:45:00.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:00.147+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:00.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:00.153+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.157145+0000) 2019-10-13T10:45:00.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:00.761+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:00.764800+0000) 2019-10-13T10:45:01.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:01.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.220003+0000) 2019-10-13T10:45:01.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:01.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.241913+0000) 2019-10-13T10:45:01.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:01.761+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:01.765744+0000) 2019-10-13T10:45:04.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:04.079+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.084083+0000) 2019-10-13T10:45:04.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:04.367+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:04.370503+0000) 2019-10-13T10:45:05.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:05.151+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.155276+0000) 2019-10-13T10:45:05.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:05.149+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.155432+0000) 2019-10-13T10:45:05.157 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:05.151+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.156352+0000) 2019-10-13T10:45:05.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:05.151+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.156961+0000) 2019-10-13T10:45:05.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:05.151+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.157205+0000) 2019-10-13T10:45:05.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:05.155+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.158770+0000) 2019-10-13T10:45:05.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:05.155+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.160549+0000) 2019-10-13T10:45:05.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:05.155+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123513 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:05.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:05.161+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:05.164615+0000) 2019-10-13T10:45:07.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:07.739+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:07.743736+0000) 2019-10-13T10:45:08.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:08.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:08.513965+0000) 2019-10-13T10:45:10.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:10.160+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.166264+0000) 2019-10-13T10:45:10.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:10.163+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.167062+0000) 2019-10-13T10:45:10.168 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:10.163+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.167516+0000) 2019-10-13T10:45:10.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:10.163+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.167927+0000) 2019-10-13T10:45:10.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:10.163+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.169514+0000) 2019-10-13T10:45:10.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:10.167+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.170203+0000) 2019-10-13T10:45:10.172 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:10.167+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.171218+0000) 2019-10-13T10:45:10.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:10.167+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:10.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:10.168+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.174654+0000) 2019-10-13T10:45:10.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:10.772+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:10.776274+0000) 2019-10-13T10:45:11.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:11.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.220210+0000) 2019-10-13T10:45:11.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:11.239+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.242117+0000) 2019-10-13T10:45:11.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:11.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:11.765943+0000) 2019-10-13T10:45:12.638 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:45:12.639 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:12.642 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:45:12.642 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:14.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:14.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.084291+0000) 2019-10-13T10:45:14.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:14.367+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:14.370705+0000) 2019-10-13T10:45:15.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:15.168+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.173621+0000) 2019-10-13T10:45:15.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:15.170+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.177265+0000) 2019-10-13T10:45:15.179 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:15.174+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.177884+0000) 2019-10-13T10:45:15.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:15.174+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.178332+0000) 2019-10-13T10:45:15.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:15.174+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.178633+0000) 2019-10-13T10:45:15.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:15.174+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.178415+0000) 2019-10-13T10:45:15.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:15.174+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.180494+0000) 2019-10-13T10:45:15.182 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:15.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:15.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:15.176+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:15.181329+0000) 2019-10-13T10:45:17.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:17.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:17.743941+0000) 2019-10-13T10:45:18.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:18.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:18.514172+0000) 2019-10-13T10:45:20.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:20.178+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.184725+0000) 2019-10-13T10:45:20.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:20.182+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.187144+0000) 2019-10-13T10:45:20.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:20.182+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.187445+0000) 2019-10-13T10:45:20.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:20.180+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.187533+0000) 2019-10-13T10:45:20.189 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:20.182+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.188737+0000) 2019-10-13T10:45:20.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:20.182+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.189027+0000) 2019-10-13T10:45:20.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:20.182+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.189544+0000) 2019-10-13T10:45:20.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:20.186+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:20.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:20.188+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.193307+0000) 2019-10-13T10:45:20.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:20.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:20.782701+0000) 2019-10-13T10:45:21.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:21.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.220473+0000) 2019-10-13T10:45:21.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:21.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.242331+0000) 2019-10-13T10:45:21.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:21.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:21.766140+0000) 2019-10-13T10:45:24.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:24.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.084525+0000) 2019-10-13T10:45:24.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:24.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:24.370909+0000) 2019-10-13T10:45:25.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:25.190+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.193884+0000) 2019-10-13T10:45:25.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:25.190+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.194487+0000) 2019-10-13T10:45:25.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:25.190+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.195875+0000) 2019-10-13T10:45:25.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:25.188+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.195692+0000) 2019-10-13T10:45:25.198 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:25.190+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.197147+0000) 2019-10-13T10:45:25.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:25.194+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.198073+0000) 2019-10-13T10:45:25.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:25.194+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.198422+0000) 2019-10-13T10:45:25.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:25.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:25.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:25.196+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:25.203553+0000) 2019-10-13T10:45:27.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:27.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:27.744145+0000) 2019-10-13T10:45:28.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:28.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:28.514387+0000) 2019-10-13T10:45:30.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:30.198+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.204034+0000) 2019-10-13T10:45:30.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:30.200+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.204170+0000) 2019-10-13T10:45:30.206 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:30.198+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.204606+0000) 2019-10-13T10:45:30.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:30.198+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.205270+0000) 2019-10-13T10:45:30.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:30.202+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.206014+0000) 2019-10-13T10:45:30.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:30.202+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.206384+0000) 2019-10-13T10:45:30.207 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:30.202+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.207062+0000) 2019-10-13T10:45:30.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:30.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123664 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:30.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:30.204+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.209822+0000) 2019-10-13T10:45:30.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:30.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:30.789184+0000) 2019-10-13T10:45:31.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:31.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.220701+0000) 2019-10-13T10:45:31.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:31.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.242585+0000) 2019-10-13T10:45:31.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:31.760+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:31.766328+0000) 2019-10-13T10:45:34.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:34.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.084724+0000) 2019-10-13T10:45:34.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:34.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:34.371135+0000) 2019-10-13T10:45:35.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:35.206+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.213578+0000) 2019-10-13T10:45:35.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:35.210+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.214243+0000) 2019-10-13T10:45:35.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:35.210+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.214511+0000) 2019-10-13T10:45:35.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:35.210+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.214875+0000) 2019-10-13T10:45:35.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:35.210+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.215788+0000) 2019-10-13T10:45:35.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:35.210+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.216126+0000) 2019-10-13T10:45:35.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:35.208+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.216031+0000) 2019-10-13T10:45:35.218 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:35.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:35.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:35.212+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:35.218128+0000) 2019-10-13T10:45:37.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:37.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:37.744324+0000) 2019-10-13T10:45:38.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:38.510+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:38.514576+0000) 2019-10-13T10:45:40.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:40.214+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.220665+0000) 2019-10-13T10:45:40.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:40.214+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.221540+0000) 2019-10-13T10:45:40.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:40.218+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.222034+0000) 2019-10-13T10:45:40.223 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:40.216+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.222414+0000) 2019-10-13T10:45:40.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:40.218+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.223353+0000) 2019-10-13T10:45:40.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:40.218+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.223028+0000) 2019-10-13T10:45:40.225 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:40.218+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.223728+0000) 2019-10-13T10:45:40.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:40.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:40.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:40.224+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.230251+0000) 2019-10-13T10:45:40.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:40.796+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:40.800057+0000) 2019-10-13T10:45:41.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:41.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.220910+0000) 2019-10-13T10:45:41.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:41.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.242792+0000) 2019-10-13T10:45:41.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:41.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:41.766532+0000) 2019-10-13T10:45:42.760 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:45:42.760 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:42.764 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:45:42.764 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:45:44.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:44.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.084929+0000) 2019-10-13T10:45:44.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:44.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:44.371332+0000) 2019-10-13T10:45:45.231 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:45.226+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.230485+0000) 2019-10-13T10:45:45.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:45.226+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.231328+0000) 2019-10-13T10:45:45.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:45.226+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.232106+0000) 2019-10-13T10:45:45.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:45.230+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.233773+0000) 2019-10-13T10:45:45.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:45.228+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.235842+0000) 2019-10-13T10:45:45.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:45.230+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:45.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:45.230+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.237439+0000) 2019-10-13T10:45:45.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:45.232+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.237696+0000) 2019-10-13T10:45:45.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:45.230+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:45.237534+0000) 2019-10-13T10:45:47.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:47.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:47.744525+0000) 2019-10-13T10:45:48.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:48.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:48.514774+0000) 2019-10-13T10:45:50.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:50.234+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.239993+0000) 2019-10-13T10:45:50.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:50.234+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.240355+0000) 2019-10-13T10:45:50.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:50.238+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.241849+0000) 2019-10-13T10:45:50.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:50.238+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.242086+0000) 2019-10-13T10:45:50.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:50.238+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.242355+0000) 2019-10-13T10:45:50.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:50.236+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.242626+0000) 2019-10-13T10:45:50.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:50.238+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.242950+0000) 2019-10-13T10:45:50.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:50.238+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:50.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:50.240+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.247293+0000) 2019-10-13T10:45:50.807 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:50.804+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:50.806459+0000) 2019-10-13T10:45:51.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:51.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.221116+0000) 2019-10-13T10:45:51.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:51.238+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.243006+0000) 2019-10-13T10:45:51.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:51.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:51.766728+0000) 2019-10-13T10:45:54.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:54.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.085138+0000) 2019-10-13T10:45:54.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:54.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:54.371537+0000) 2019-10-13T10:45:55.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:45:55.246+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.249831+0000) 2019-10-13T10:45:55.252 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:55.246+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.250035+0000) 2019-10-13T10:45:55.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:45:55.246+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.250370+0000) 2019-10-13T10:45:55.253 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:45:55.246+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.252002+0000) 2019-10-13T10:45:55.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:55.246+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.252423+0000) 2019-10-13T10:45:55.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:45:55.246+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.253076+0000) 2019-10-13T10:45:55.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:45:55.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:45:55.256 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:45:55.248+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.255526+0000) 2019-10-13T10:45:55.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:45:55.256+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:55.261092+0000) 2019-10-13T10:45:57.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:45:57.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:57.744727+0000) 2019-10-13T10:45:58.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:45:58.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:45:58.514958+0000) 2019-10-13T10:46:00.260 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:00.254+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.258929+0000) 2019-10-13T10:46:00.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:00.254+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.259206+0000) 2019-10-13T10:46:00.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:00.254+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.259988+0000) 2019-10-13T10:46:00.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:00.254+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.260611+0000) 2019-10-13T10:46:00.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:00.258+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.262057+0000) 2019-10-13T10:46:00.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:00.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:00.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:00.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.264111+0000) 2019-10-13T10:46:00.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:00.260+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.264390+0000) 2019-10-13T10:46:00.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:00.264+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.268759+0000) 2019-10-13T10:46:00.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:00.812+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:00.812420+0000) 2019-10-13T10:46:01.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:01.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.221320+0000) 2019-10-13T10:46:01.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:01.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.243212+0000) 2019-10-13T10:46:01.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:01.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:01.766938+0000) 2019-10-13T10:46:04.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:04.078+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.085335+0000) 2019-10-13T10:46:04.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:04.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:04.371747+0000) 2019-10-13T10:46:05.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:05.262+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.267050+0000) 2019-10-13T10:46:05.268 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:05.262+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.267718+0000) 2019-10-13T10:46:05.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:05.266+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.269823+0000) 2019-10-13T10:46:05.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:05.266+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.270132+0000) 2019-10-13T10:46:05.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:05.264+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.271893+0000) 2019-10-13T10:46:05.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:05.266+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.272456+0000) 2019-10-13T10:46:05.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:05.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:05.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:05.266+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.273074+0000) 2019-10-13T10:46:05.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:05.272+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:05.278238+0000) 2019-10-13T10:46:07.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:07.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:07.744926+0000) 2019-10-13T10:46:08.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:08.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:08.515154+0000) 2019-10-13T10:46:10.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:10.272+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.275599+0000) 2019-10-13T10:46:10.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:10.270+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.276581+0000) 2019-10-13T10:46:10.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:10.270+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.277103+0000) 2019-10-13T10:46:10.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:10.270+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.277635+0000) 2019-10-13T10:46:10.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:10.274+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.278442+0000) 2019-10-13T10:46:10.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:10.274+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.278804+0000) 2019-10-13T10:46:10.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:10.274+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.279666+0000) 2019-10-13T10:46:10.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:10.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:10.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:10.280+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.283881+0000) 2019-10-13T10:46:10.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:10.816+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:10.818891+0000) 2019-10-13T10:46:11.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:11.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.221496+0000) 2019-10-13T10:46:11.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:11.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.243426+0000) 2019-10-13T10:46:11.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:11.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:11.767162+0000) 2019-10-13T10:46:12.890 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:46:12.890 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:12.893 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:46:12.893 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:14.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:14.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.085578+0000) 2019-10-13T10:46:14.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:14.365+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:14.371981+0000) 2019-10-13T10:46:15.287 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:15.281+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.286108+0000) 2019-10-13T10:46:15.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:15.281+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.286662+0000) 2019-10-13T10:46:15.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:15.281+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.286785+0000) 2019-10-13T10:46:15.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:15.281+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.288021+0000) 2019-10-13T10:46:15.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:15.288+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.288444+0000) 2019-10-13T10:46:15.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:15.281+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.289123+0000) 2019-10-13T10:46:15.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:15.281+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.289628+0000) 2019-10-13T10:46:15.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:15.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:15.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:15.292+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:15.292985+0000) 2019-10-13T10:46:17.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:17.741+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:17.745128+0000) 2019-10-13T10:46:18.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:18.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:18.515305+0000) 2019-10-13T10:46:20.295 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:20.289+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.294238+0000) 2019-10-13T10:46:20.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:20.289+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.294693+0000) 2019-10-13T10:46:20.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:20.291+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.294747+0000) 2019-10-13T10:46:20.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:20.289+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.296797+0000) 2019-10-13T10:46:20.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:20.289+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.297262+0000) 2019-10-13T10:46:20.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:20.289+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.297550+0000) 2019-10-13T10:46:20.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:20.289+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.297491+0000) 2019-10-13T10:46:20.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:20.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:20.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:20.299+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.301968+0000) 2019-10-13T10:46:20.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:20.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:20.825257+0000) 2019-10-13T10:46:21.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:21.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.221670+0000) 2019-10-13T10:46:21.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:21.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.243563+0000) 2019-10-13T10:46:21.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:21.763+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:21.767353+0000) 2019-10-13T10:46:24.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:24.081+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.085807+0000) 2019-10-13T10:46:24.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:24.369+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:24.372191+0000) 2019-10-13T10:46:25.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:25.297+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.303437+0000) 2019-10-13T10:46:25.304 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:25.297+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.303688+0000) 2019-10-13T10:46:25.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:25.297+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.304672+0000) 2019-10-13T10:46:25.306 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:25.297+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.305672+0000) 2019-10-13T10:46:25.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:25.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.306162+0000) 2019-10-13T10:46:25.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:25.301+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.306477+0000) 2019-10-13T10:46:25.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:25.303+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.306951+0000) 2019-10-13T10:46:25.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:25.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:25.316 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:25.311+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:25.315381+0000) 2019-10-13T10:46:27.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:27.741+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:27.745291+0000) 2019-10-13T10:46:28.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:28.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:28.515521+0000) 2019-10-13T10:46:30.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:30.309+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.312255+0000) 2019-10-13T10:46:30.313 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:30.309+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.312628+0000) 2019-10-13T10:46:30.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:30.313+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.314063+0000) 2019-10-13T10:46:30.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:30.313+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.314750+0000) 2019-10-13T10:46:30.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:30.313+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.315195+0000) 2019-10-13T10:46:30.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:30.313+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.315752+0000) 2019-10-13T10:46:30.317 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:30.311+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.315822+0000) 2019-10-13T10:46:30.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:30.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:30.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:30.319+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.322846+0000) 2019-10-13T10:46:30.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:30.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:30.836554+0000) 2019-10-13T10:46:31.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:31.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.221850+0000) 2019-10-13T10:46:31.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:31.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.243752+0000) 2019-10-13T10:46:31.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:31.763+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:31.767546+0000) 2019-10-13T10:46:34.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:34.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.086018+0000) 2019-10-13T10:46:34.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:34.369+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:34.372418+0000) 2019-10-13T10:46:35.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:35.321+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.321909+0000) 2019-10-13T10:46:35.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:35.321+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.322610+0000) 2019-10-13T10:46:35.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:35.321+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.323655+0000) 2019-10-13T10:46:35.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:35.321+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.323831+0000) 2019-10-13T10:46:35.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:35.323+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.324373+0000) 2019-10-13T10:46:35.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:35.321+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.324548+0000) 2019-10-13T10:46:35.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:35.321+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.325416+0000) 2019-10-13T10:46:35.329 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:35.325+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:35.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:35.327+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:35.329192+0000) 2019-10-13T10:46:37.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:37.741+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:37.745506+0000) 2019-10-13T10:46:38.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:38.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:38.515717+0000) 2019-10-13T10:46:40.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:40.329+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.332401+0000) 2019-10-13T10:46:40.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:40.329+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.332742+0000) 2019-10-13T10:46:40.335 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:40.333+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.335153+0000) 2019-10-13T10:46:40.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:40.333+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.335985+0000) 2019-10-13T10:46:40.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:40.337+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.337721+0000) 2019-10-13T10:46:40.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:40.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:40.339 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:40.335+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.338358+0000) 2019-10-13T10:46:40.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:40.337+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.339151+0000) 2019-10-13T10:46:40.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:40.339+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.341552+0000) 2019-10-13T10:46:40.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:40.839+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:40.842925+0000) 2019-10-13T10:46:41.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:41.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.222025+0000) 2019-10-13T10:46:41.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:41.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.243931+0000) 2019-10-13T10:46:41.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:41.763+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:41.767760+0000) 2019-10-13T10:46:43.004 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:46:43.004 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:43.008 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:46:43.008 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:46:44.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:44.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.086250+0000) 2019-10-13T10:46:44.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:44.369+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:44.372619+0000) 2019-10-13T10:46:45.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:45.339+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.340810+0000) 2019-10-13T10:46:45.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:45.337+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.341700+0000) 2019-10-13T10:46:45.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:45.341+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.342217+0000) 2019-10-13T10:46:45.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:45.341+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.343279+0000) 2019-10-13T10:46:45.344 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:45.341+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.344017+0000) 2019-10-13T10:46:45.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:45.341+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.344776+0000) 2019-10-13T10:46:45.345 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:45.341+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.345214+0000) 2019-10-13T10:46:45.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:45.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:45.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:45.347+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:45.350007+0000) 2019-10-13T10:46:47.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:47.741+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:47.745711+0000) 2019-10-13T10:46:48.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:48.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:48.515916+0000) 2019-10-13T10:46:50.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:50.349+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.350466+0000) 2019-10-13T10:46:50.352 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:50.349+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.351346+0000) 2019-10-13T10:46:50.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:50.349+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.352161+0000) 2019-10-13T10:46:50.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:50.349+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.352290+0000) 2019-10-13T10:46:50.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:50.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.352762+0000) 2019-10-13T10:46:50.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:50.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.353730+0000) 2019-10-13T10:46:50.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:50.353+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.354029+0000) 2019-10-13T10:46:50.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:50.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:50.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:50.355+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.358585+0000) 2019-10-13T10:46:50.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:50.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:50.849474+0000) 2019-10-13T10:46:51.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:51.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.222240+0000) 2019-10-13T10:46:51.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:51.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.244126+0000) 2019-10-13T10:46:51.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:51.763+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:51.767955+0000) 2019-10-13T10:46:54.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:54.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.086450+0000) 2019-10-13T10:46:54.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:54.369+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:54.372850+0000) 2019-10-13T10:46:55.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:55.357+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.359557+0000) 2019-10-13T10:46:55.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:46:55.357+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.360182+0000) 2019-10-13T10:46:55.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:46:55.357+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.361370+0000) 2019-10-13T10:46:55.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:46:55.359+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.361372+0000) 2019-10-13T10:46:55.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:55.361+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.362219+0000) 2019-10-13T10:46:55.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:46:55.361+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.362569+0000) 2019-10-13T10:46:55.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:46:55.361+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.363394+0000) 2019-10-13T10:46:55.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:46:55.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124192 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:46:55.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:46:55.363+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:55.366708+0000) 2019-10-13T10:46:57.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:46:57.745+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:57.745946+0000) 2019-10-13T10:46:58.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:46:58.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:46:58.516099+0000) 2019-10-13T10:47:00.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:00.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.370146+0000) 2019-10-13T10:47:00.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:00.365+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.368902+0000) 2019-10-13T10:47:00.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:00.369+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.369800+0000) 2019-10-13T10:47:00.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:00.369+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.372688+0000) 2019-10-13T10:47:00.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:00.373+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.374310+0000) 2019-10-13T10:47:00.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:00.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:00.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:00.373+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.375470+0000) 2019-10-13T10:47:00.379 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:00.373+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.375821+0000) 2019-10-13T10:47:00.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:00.375+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.378423+0000) 2019-10-13T10:47:00.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:00.855+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:00.855921+0000) 2019-10-13T10:47:01.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:01.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.222419+0000) 2019-10-13T10:47:01.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:01.241+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.244306+0000) 2019-10-13T10:47:01.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:01.767+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:01.768149+0000) 2019-10-13T10:47:04.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:04.085+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.086651+0000) 2019-10-13T10:47:04.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:04.369+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:04.373053+0000) 2019-10-13T10:47:05.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:05.377+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.377995+0000) 2019-10-13T10:47:05.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:05.377+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.378581+0000) 2019-10-13T10:47:05.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:05.375+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.379279+0000) 2019-10-13T10:47:05.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:05.377+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.379899+0000) 2019-10-13T10:47:05.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:05.377+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.380063+0000) 2019-10-13T10:47:05.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:05.377+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.380705+0000) 2019-10-13T10:47:05.382 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:05.377+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.381428+0000) 2019-10-13T10:47:05.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:05.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:05.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:05.383+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:05.387879+0000) 2019-10-13T10:47:07.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:07.745+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:07.746147+0000) 2019-10-13T10:47:08.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:08.513+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:08.516307+0000) 2019-10-13T10:47:10.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:10.383+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.387990+0000) 2019-10-13T10:47:10.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:10.384+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.388780+0000) 2019-10-13T10:47:10.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:10.384+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.388949+0000) 2019-10-13T10:47:10.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:10.388+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.390535+0000) 2019-10-13T10:47:10.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:10.388+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.390867+0000) 2019-10-13T10:47:10.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:10.388+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.391404+0000) 2019-10-13T10:47:10.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:10.388+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.393029+0000) 2019-10-13T10:47:10.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:10.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:10.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:10.399+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.400354+0000) 2019-10-13T10:47:10.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:10.859+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:10.862334+0000) 2019-10-13T10:47:11.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:11.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.222614+0000) 2019-10-13T10:47:11.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:11.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.244502+0000) 2019-10-13T10:47:11.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:11.767+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:11.768341+0000) 2019-10-13T10:47:13.069 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:47:13.070 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:13.074 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:47:13.074 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:14.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:14.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.086854+0000) 2019-10-13T10:47:14.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:14.368+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:14.373246+0000) 2019-10-13T10:47:15.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:15.392+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.397658+0000) 2019-10-13T10:47:15.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:15.396+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.398262+0000) 2019-10-13T10:47:15.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:15.396+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.399343+0000) 2019-10-13T10:47:15.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:15.396+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.399776+0000) 2019-10-13T10:47:15.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:15.396+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.400140+0000) 2019-10-13T10:47:15.402 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:15.399+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.400783+0000) 2019-10-13T10:47:15.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:15.396+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.400627+0000) 2019-10-13T10:47:15.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:15.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:15.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:15.403+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:15.407558+0000) 2019-10-13T10:47:17.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:17.744+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:17.746366+0000) 2019-10-13T10:47:18.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:18.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:18.516535+0000) 2019-10-13T10:47:20.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:20.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.408101+0000) 2019-10-13T10:47:20.409 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:20.404+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.409056+0000) 2019-10-13T10:47:20.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:20.408+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.409779+0000) 2019-10-13T10:47:20.415 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:20.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.411665+0000) 2019-10-13T10:47:20.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:20.408+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.410725+0000) 2019-10-13T10:47:20.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:20.408+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.411130+0000) 2019-10-13T10:47:20.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:20.408+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.412297+0000) 2019-10-13T10:47:20.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:20.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:20.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:20.411+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.415444+0000) 2019-10-13T10:47:20.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:20.867+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:20.868768+0000) 2019-10-13T10:47:21.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:21.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.222791+0000) 2019-10-13T10:47:21.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:21.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.244680+0000) 2019-10-13T10:47:21.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:21.767+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:21.768554+0000) 2019-10-13T10:47:24.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:24.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.087071+0000) 2019-10-13T10:47:24.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:24.368+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:24.373464+0000) 2019-10-13T10:47:25.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:25.416+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.418261+0000) 2019-10-13T10:47:25.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:25.414+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.418031+0000) 2019-10-13T10:47:25.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:25.416+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.419851+0000) 2019-10-13T10:47:25.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:25.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:25.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:25.418+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.423133+0000) 2019-10-13T10:47:25.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:25.420+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.423483+0000) 2019-10-13T10:47:25.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:25.420+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.423776+0000) 2019-10-13T10:47:25.426 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:25.420+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.425093+0000) 2019-10-13T10:47:25.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:25.420+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:25.425415+0000) 2019-10-13T10:47:27.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:27.744+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:27.746596+0000) 2019-10-13T10:47:28.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:28.512+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:28.516747+0000) 2019-10-13T10:47:30.429 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:30.422+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.427728+0000) 2019-10-13T10:47:30.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:30.424+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.428611+0000) 2019-10-13T10:47:30.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:30.424+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.428884+0000) 2019-10-13T10:47:30.431 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:30.428+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.430920+0000) 2019-10-13T10:47:30.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:30.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:30.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:30.428+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.433041+0000) 2019-10-13T10:47:30.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:30.432+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.434078+0000) 2019-10-13T10:47:30.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:30.432+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.434755+0000) 2019-10-13T10:47:30.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:30.434+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.438814+0000) 2019-10-13T10:47:30.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:30.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:30.875367+0000) 2019-10-13T10:47:31.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:31.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.222970+0000) 2019-10-13T10:47:31.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:31.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.244886+0000) 2019-10-13T10:47:31.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:31.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:31.768779+0000) 2019-10-13T10:47:34.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:34.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.087258+0000) 2019-10-13T10:47:34.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:34.368+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:34.373710+0000) 2019-10-13T10:47:35.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:35.432+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.435941+0000) 2019-10-13T10:47:35.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:35.432+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.436545+0000) 2019-10-13T10:47:35.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:35.432+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.436943+0000) 2019-10-13T10:47:35.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:35.436+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.438355+0000) 2019-10-13T10:47:35.440 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:35.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.439466+0000) 2019-10-13T10:47:35.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:35.436+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.440350+0000) 2019-10-13T10:47:35.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:35.436+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.440785+0000) 2019-10-13T10:47:35.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:35.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124440 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:35.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:35.438+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:35.443850+0000) 2019-10-13T10:47:37.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:37.744+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:37.746786+0000) 2019-10-13T10:47:38.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:38.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:38.516946+0000) 2019-10-13T10:47:40.446 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:40.440+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.444982+0000) 2019-10-13T10:47:40.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:40.440+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.445514+0000) 2019-10-13T10:47:40.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:40.444+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.446047+0000) 2019-10-13T10:47:40.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:40.444+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.446897+0000) 2019-10-13T10:47:40.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:40.444+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.447961+0000) 2019-10-13T10:47:40.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:40.444+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.448379+0000) 2019-10-13T10:47:40.451 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:40.446+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.450420+0000) 2019-10-13T10:47:40.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:40.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:40.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:40.454+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.457811+0000) 2019-10-13T10:47:40.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:40.878+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:40.882093+0000) 2019-10-13T10:47:41.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:41.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.223155+0000) 2019-10-13T10:47:41.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:41.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.245090+0000) 2019-10-13T10:47:41.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:41.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:41.768973+0000) 2019-10-13T10:47:43.220 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:47:43.220 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:43.224 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:47:43.224 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:47:44.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:44.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:44.087417+0000) 2019-10-13T10:47:44.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:44.372+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:44.373944+0000) 2019-10-13T10:47:45.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:45.452+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.454511+0000) 2019-10-13T10:47:45.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:45.452+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.456124+0000) 2019-10-13T10:47:45.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:45.452+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.456543+0000) 2019-10-13T10:47:45.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:45.454+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.456579+0000) 2019-10-13T10:47:45.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:45.456+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.459133+0000) 2019-10-13T10:47:45.461 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:45.456+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.460300+0000) 2019-10-13T10:47:45.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:45.456+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.461214+0000) 2019-10-13T10:47:45.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:45.456+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:45.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:45.462+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:45.466408+0000) 2019-10-13T10:47:47.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:47.744+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:47.746989+0000) 2019-10-13T10:47:48.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:48.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:48.517094+0000) 2019-10-13T10:47:50.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:50.460+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.463885+0000) 2019-10-13T10:47:50.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:50.460+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.464119+0000) 2019-10-13T10:47:50.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:50.462+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.464636+0000) 2019-10-13T10:47:50.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:50.460+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.465284+0000) 2019-10-13T10:47:50.466 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:50.460+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.465335+0000) 2019-10-13T10:47:50.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:50.464+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.466336+0000) 2019-10-13T10:47:50.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:50.464+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.467346+0000) 2019-10-13T10:47:50.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:50.468+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:50.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:50.470+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.472511+0000) 2019-10-13T10:47:50.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:50.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:50.887961+0000) 2019-10-13T10:47:51.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:51.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.223338+0000) 2019-10-13T10:47:51.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:51.240+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.245268+0000) 2019-10-13T10:47:51.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:51.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:51.769163+0000) 2019-10-13T10:47:54.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:54.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:54.087639+0000) 2019-10-13T10:47:54.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:54.372+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:54.374152+0000) 2019-10-13T10:47:55.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:47:55.472+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.474045+0000) 2019-10-13T10:47:55.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:55.472+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.474525+0000) 2019-10-13T10:47:55.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:47:55.470+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.474687+0000) 2019-10-13T10:47:55.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:47:55.472+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.475625+0000) 2019-10-13T10:47:55.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:47:55.472+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.477684+0000) 2019-10-13T10:47:55.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:47:55.472+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.477484+0000) 2019-10-13T10:47:55.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:55.476+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.478419+0000) 2019-10-13T10:47:55.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:47:55.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:47:55.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:47:55.482+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:55.484638+0000) 2019-10-13T10:47:57.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:47:57.744+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:57.747186+0000) 2019-10-13T10:47:58.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:47:58.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:47:58.517323+0000) 2019-10-13T10:48:00.486 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:00.482+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.484842+0000) 2019-10-13T10:48:00.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:00.480+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.485339+0000) 2019-10-13T10:48:00.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:00.484+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.486008+0000) 2019-10-13T10:48:00.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:00.484+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.486372+0000) 2019-10-13T10:48:00.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:00.484+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.487007+0000) 2019-10-13T10:48:00.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:00.484+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.488512+0000) 2019-10-13T10:48:00.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:00.484+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.488739+0000) 2019-10-13T10:48:00.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:00.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:00.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:00.490+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.494311+0000) 2019-10-13T10:48:00.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:00.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:00.894651+0000) 2019-10-13T10:48:01.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:01.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.223560+0000) 2019-10-13T10:48:01.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:01.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.245464+0000) 2019-10-13T10:48:01.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:01.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:01.769364+0000) 2019-10-13T10:48:04.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:04.084+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:04.087854+0000) 2019-10-13T10:48:04.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:04.372+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:04.374387+0000) 2019-10-13T10:48:05.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:05.488+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.493191+0000) 2019-10-13T10:48:05.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:05.490+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.493585+0000) 2019-10-13T10:48:05.495 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:05.492+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.494695+0000) 2019-10-13T10:48:05.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:05.492+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.496581+0000) 2019-10-13T10:48:05.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:05.492+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.497154+0000) 2019-10-13T10:48:05.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:05.492+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.497480+0000) 2019-10-13T10:48:05.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:05.496+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.498104+0000) 2019-10-13T10:48:05.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:05.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:05.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:05.502+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:05.505224+0000) 2019-10-13T10:48:07.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:07.743+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:07.747389+0000) 2019-10-13T10:48:08.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:08.511+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:08.517574+0000) 2019-10-13T10:48:10.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:10.499+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.502593+0000) 2019-10-13T10:48:10.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:10.498+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.502743+0000) 2019-10-13T10:48:10.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:10.499+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.503701+0000) 2019-10-13T10:48:10.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:10.499+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.504143+0000) 2019-10-13T10:48:10.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:10.499+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.504414+0000) 2019-10-13T10:48:10.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:10.503+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.506618+0000) 2019-10-13T10:48:10.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:10.503+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.507363+0000) 2019-10-13T10:48:10.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:10.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:10.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:10.510+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.514075+0000) 2019-10-13T10:48:10.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:10.898+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:10.901170+0000) 2019-10-13T10:48:11.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:11.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.223737+0000) 2019-10-13T10:48:11.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:11.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.245644+0000) 2019-10-13T10:48:11.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:11.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:11.769557+0000) 2019-10-13T10:48:13.264 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:48:13.265 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:13.268 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:48:13.268 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:14.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:14.083+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:14.088045+0000) 2019-10-13T10:48:14.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:14.371+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:14.374638+0000) 2019-10-13T10:48:15.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:15.507+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.512763+0000) 2019-10-13T10:48:15.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:15.507+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.513245+0000) 2019-10-13T10:48:15.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:15.511+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.515550+0000) 2019-10-13T10:48:15.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:15.511+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.515970+0000) 2019-10-13T10:48:15.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:15.511+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.516420+0000) 2019-10-13T10:48:15.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:15.514+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.516557+0000) 2019-10-13T10:48:15.518 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:15.515+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.517871+0000) 2019-10-13T10:48:15.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:15.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:15.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:15.522+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:15.524448+0000) 2019-10-13T10:48:17.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:17.743+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:17.747637+0000) 2019-10-13T10:48:18.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:18.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:18.517802+0000) 2019-10-13T10:48:20.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:20.515+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.521296+0000) 2019-10-13T10:48:20.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:20.515+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.521654+0000) 2019-10-13T10:48:20.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:20.519+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.522292+0000) 2019-10-13T10:48:20.524 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:20.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.524144+0000) 2019-10-13T10:48:20.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:20.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.524325+0000) 2019-10-13T10:48:20.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:20.523+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.525822+0000) 2019-10-13T10:48:20.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:20.523+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.526407+0000) 2019-10-13T10:48:20.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:20.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:20.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:20.530+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.533982+0000) 2019-10-13T10:48:20.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:20.902+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:20.907687+0000) 2019-10-13T10:48:21.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:21.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.223920+0000) 2019-10-13T10:48:21.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:21.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.245829+0000) 2019-10-13T10:48:21.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:21.766+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:21.769751+0000) 2019-10-13T10:48:24.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:24.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:24.088258+0000) 2019-10-13T10:48:24.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:24.371+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:24.374839+0000) 2019-10-13T10:48:25.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:25.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.531407+0000) 2019-10-13T10:48:25.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:25.527+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.532873+0000) 2019-10-13T10:48:25.534 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:25.527+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.533658+0000) 2019-10-13T10:48:25.535 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:25.531+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.533910+0000) 2019-10-13T10:48:25.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:25.530+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.535839+0000) 2019-10-13T10:48:25.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:25.531+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.536664+0000) 2019-10-13T10:48:25.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:25.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:25.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:25.535+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.537824+0000) 2019-10-13T10:48:25.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:25.538+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:25.543905+0000) 2019-10-13T10:48:27.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:27.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:27.747836+0000) 2019-10-13T10:48:28.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:28.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:28.517997+0000) 2019-10-13T10:48:30.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:30.535+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.540227+0000) 2019-10-13T10:48:30.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:30.535+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.540887+0000) 2019-10-13T10:48:30.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:30.535+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.541338+0000) 2019-10-13T10:48:30.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:30.535+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.541657+0000) 2019-10-13T10:48:30.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:30.537+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.542309+0000) 2019-10-13T10:48:30.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:30.539+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.543342+0000) 2019-10-13T10:48:30.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:30.539+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.543406+0000) 2019-10-13T10:48:30.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:30.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124771 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:30.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:30.545+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.548986+0000) 2019-10-13T10:48:30.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:30.917+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:30.919062+0000) 2019-10-13T10:48:31.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:31.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.224098+0000) 2019-10-13T10:48:31.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:31.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.246013+0000) 2019-10-13T10:48:31.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:31.765+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:31.769948+0000) 2019-10-13T10:48:34.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:34.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:34.088469+0000) 2019-10-13T10:48:34.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:34.371+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:34.375034+0000) 2019-10-13T10:48:35.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:35.547+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.551614+0000) 2019-10-13T10:48:35.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:35.547+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.552094+0000) 2019-10-13T10:48:35.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:35.547+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.552522+0000) 2019-10-13T10:48:35.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:35.547+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.553345+0000) 2019-10-13T10:48:35.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:35.549+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.553610+0000) 2019-10-13T10:48:35.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:35.551+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.554279+0000) 2019-10-13T10:48:35.555 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:35.551+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.554864+0000) 2019-10-13T10:48:35.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:35.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:35.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:35.557+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:35.564051+0000) 2019-10-13T10:48:37.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:37.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:37.748044+0000) 2019-10-13T10:48:38.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:38.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:38.518224+0000) 2019-10-13T10:48:40.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:40.555+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.559891+0000) 2019-10-13T10:48:40.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:40.555+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.561046+0000) 2019-10-13T10:48:40.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:40.557+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.562119+0000) 2019-10-13T10:48:40.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:40.559+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.562852+0000) 2019-10-13T10:48:40.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:40.559+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.563702+0000) 2019-10-13T10:48:40.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:40.559+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.564258+0000) 2019-10-13T10:48:40.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:40.559+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.565109+0000) 2019-10-13T10:48:40.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:40.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:40.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:40.565+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.568837+0000) 2019-10-13T10:48:40.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:40.921+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:40.925543+0000) 2019-10-13T10:48:41.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:41.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.224283+0000) 2019-10-13T10:48:41.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:41.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.246250+0000) 2019-10-13T10:48:41.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:41.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:41.770144+0000) 2019-10-13T10:48:43.387 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:48:43.388 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:43.391 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:48:43.391 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:48:44.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:44.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:44.088717+0000) 2019-10-13T10:48:44.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:44.371+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:44.375246+0000) 2019-10-13T10:48:45.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:45.567+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.570414+0000) 2019-10-13T10:48:45.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:45.567+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.571478+0000) 2019-10-13T10:48:45.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:45.567+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.571825+0000) 2019-10-13T10:48:45.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:45.565+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.571694+0000) 2019-10-13T10:48:45.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:45.567+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.572248+0000) 2019-10-13T10:48:45.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:45.567+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.572848+0000) 2019-10-13T10:48:45.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:45.567+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.573570+0000) 2019-10-13T10:48:45.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:45.571+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:45.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:45.569+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:45.576030+0000) 2019-10-13T10:48:47.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:47.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:47.748242+0000) 2019-10-13T10:48:48.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:48.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:48.518421+0000) 2019-10-13T10:48:50.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:50.571+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.577443+0000) 2019-10-13T10:48:50.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:50.575+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.578506+0000) 2019-10-13T10:48:50.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:50.575+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.579233+0000) 2019-10-13T10:48:50.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:50.575+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.579668+0000) 2019-10-13T10:48:50.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:50.575+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.580839+0000) 2019-10-13T10:48:50.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:50.577+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.580905+0000) 2019-10-13T10:48:50.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:50.575+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.581481+0000) 2019-10-13T10:48:50.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:50.579+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:50.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:50.581+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.584719+0000) 2019-10-13T10:48:50.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:50.925+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:50.931946+0000) 2019-10-13T10:48:51.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:51.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.224494+0000) 2019-10-13T10:48:51.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:51.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.246429+0000) 2019-10-13T10:48:51.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:51.765+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:51.770336+0000) 2019-10-13T10:48:54.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:54.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:54.088925+0000) 2019-10-13T10:48:54.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:54.375+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:54.375444+0000) 2019-10-13T10:48:55.589 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:48:55.583+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.588309+0000) 2019-10-13T10:48:55.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:55.587+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.590084+0000) 2019-10-13T10:48:55.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:48:55.587+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.590436+0000) 2019-10-13T10:48:55.592 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:48:55.587+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.591393+0000) 2019-10-13T10:48:55.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:55.587+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.593186+0000) 2019-10-13T10:48:55.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:48:55.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:48:55.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:48:55.589+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.594519+0000) 2019-10-13T10:48:55.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:48:55.591+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.595156+0000) 2019-10-13T10:48:55.598 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:48:55.593+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:55.597819+0000) 2019-10-13T10:48:57.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:48:57.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:57.748449+0000) 2019-10-13T10:48:58.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:48:58.515+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:48:58.518572+0000) 2019-10-13T10:49:00.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:00.595+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.597832+0000) 2019-10-13T10:49:00.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:00.595+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.598308+0000) 2019-10-13T10:49:00.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:00.595+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.598895+0000) 2019-10-13T10:49:00.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:00.595+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.599481+0000) 2019-10-13T10:49:00.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:00.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.599359+0000) 2019-10-13T10:49:00.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:00.595+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.600411+0000) 2019-10-13T10:49:00.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:00.595+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.601282+0000) 2019-10-13T10:49:00.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:00.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:00.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:00.601+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.606505+0000) 2019-10-13T10:49:00.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:00.937+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:00.938443+0000) 2019-10-13T10:49:01.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:01.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.224733+0000) 2019-10-13T10:49:01.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:01.243+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.246602+0000) 2019-10-13T10:49:01.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:01.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:01.770593+0000) 2019-10-13T10:49:04.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:04.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:04.089122+0000) 2019-10-13T10:49:04.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:04.370+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:04.375594+0000) 2019-10-13T10:49:05.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:05.602+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.607442+0000) 2019-10-13T10:49:05.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:05.602+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.607775+0000) 2019-10-13T10:49:05.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:05.605+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.608612+0000) 2019-10-13T10:49:05.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:05.602+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.608337+0000) 2019-10-13T10:49:05.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:05.602+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.609382+0000) 2019-10-13T10:49:05.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:05.606+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.610894+0000) 2019-10-13T10:49:05.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:05.606+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.611660+0000) 2019-10-13T10:49:05.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:05.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:05.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:05.613+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:05.617288+0000) 2019-10-13T10:49:07.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:07.746+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:07.748650+0000) 2019-10-13T10:49:08.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:08.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:08.518765+0000) 2019-10-13T10:49:10.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:10.610+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.615780+0000) 2019-10-13T10:49:10.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:10.610+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.616823+0000) 2019-10-13T10:49:10.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:10.610+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.617659+0000) 2019-10-13T10:49:10.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:10.614+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.618987+0000) 2019-10-13T10:49:10.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:10.614+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.619583+0000) 2019-10-13T10:49:10.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:10.610+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.617285+0000) 2019-10-13T10:49:10.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:10.613+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.618488+0000) 2019-10-13T10:49:10.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:10.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:10.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:10.621+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.625436+0000) 2019-10-13T10:49:10.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:10.941+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:10.944953+0000) 2019-10-13T10:49:11.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:11.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.224936+0000) 2019-10-13T10:49:11.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:11.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.246801+0000) 2019-10-13T10:49:11.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:11.765+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:11.770786+0000) 2019-10-13T10:49:13.518 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:49:13.518 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:13.522 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:49:13.522 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:14.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:14.082+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:14.089326+0000) 2019-10-13T10:49:14.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:14.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:14.375745+0000) 2019-10-13T10:49:15.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:15.622+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.626899+0000) 2019-10-13T10:49:15.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:15.622+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.627499+0000) 2019-10-13T10:49:15.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:15.622+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.627924+0000) 2019-10-13T10:49:15.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:15.622+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.629515+0000) 2019-10-13T10:49:15.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:15.625+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.629698+0000) 2019-10-13T10:49:15.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:15.626+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.630320+0000) 2019-10-13T10:49:15.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:15.626+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.631031+0000) 2019-10-13T10:49:15.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:15.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:15.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:15.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:15.639660+0000) 2019-10-13T10:49:17.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:17.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:17.748868+0000) 2019-10-13T10:49:18.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:18.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:18.518956+0000) 2019-10-13T10:49:20.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:20.630+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.636117+0000) 2019-10-13T10:49:20.637 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:20.630+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.636434+0000) 2019-10-13T10:49:20.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:20.630+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.637008+0000) 2019-10-13T10:49:20.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:20.630+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.637354+0000) 2019-10-13T10:49:20.639 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:20.634+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.638625+0000) 2019-10-13T10:49:20.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:20.633+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.638945+0000) 2019-10-13T10:49:20.641 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:20.634+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.640378+0000) 2019-10-13T10:49:20.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:20.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:20.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:20.641+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.645072+0000) 2019-10-13T10:49:20.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:20.949+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:20.951369+0000) 2019-10-13T10:49:21.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:21.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.225115+0000) 2019-10-13T10:49:21.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:21.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.246978+0000) 2019-10-13T10:49:21.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:21.765+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:21.770979+0000) 2019-10-13T10:49:24.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:24.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:24.089530+0000) 2019-10-13T10:49:24.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:24.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:24.375951+0000) 2019-10-13T10:49:25.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:25.638+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.645226+0000) 2019-10-13T10:49:25.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:25.641+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.645534+0000) 2019-10-13T10:49:25.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:25.642+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.646241+0000) 2019-10-13T10:49:25.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:25.642+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.646542+0000) 2019-10-13T10:49:25.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:25.642+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.647202+0000) 2019-10-13T10:49:25.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:25.642+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.648186+0000) 2019-10-13T10:49:25.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:25.642+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.647474+0000) 2019-10-13T10:49:25.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:25.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125103 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:25.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:25.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:25.652640+0000) 2019-10-13T10:49:27.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:27.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:27.749068+0000) 2019-10-13T10:49:28.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:28.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:28.519154+0000) 2019-10-13T10:49:30.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:30.650+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.655475+0000) 2019-10-13T10:49:30.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:30.653+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.658199+0000) 2019-10-13T10:49:30.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:30.650+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.656890+0000) 2019-10-13T10:49:30.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:30.650+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.657230+0000) 2019-10-13T10:49:30.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:30.654+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.659890+0000) 2019-10-13T10:49:30.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:30.654+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.660648+0000) 2019-10-13T10:49:30.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:30.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:30.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:30.654+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.661157+0000) 2019-10-13T10:49:30.668 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:30.661+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.667430+0000) 2019-10-13T10:49:30.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:30.957+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:30.957940+0000) 2019-10-13T10:49:31.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:31.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.225295+0000) 2019-10-13T10:49:31.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:31.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.247194+0000) 2019-10-13T10:49:31.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:31.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:31.771200+0000) 2019-10-13T10:49:34.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:34.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:34.089728+0000) 2019-10-13T10:49:34.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:34.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:34.376098+0000) 2019-10-13T10:49:35.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:35.658+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.664022+0000) 2019-10-13T10:49:35.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:35.660+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.664165+0000) 2019-10-13T10:49:35.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:35.658+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.664803+0000) 2019-10-13T10:49:35.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:35.662+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.665943+0000) 2019-10-13T10:49:35.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:35.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.666300+0000) 2019-10-13T10:49:35.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:35.662+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.667388+0000) 2019-10-13T10:49:35.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:35.662+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.667551+0000) 2019-10-13T10:49:35.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:35.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:35.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:35.668+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:35.673714+0000) 2019-10-13T10:49:37.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:37.742+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:37.749297+0000) 2019-10-13T10:49:38.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:38.514+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:38.519373+0000) 2019-10-13T10:49:40.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:40.666+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.673565+0000) 2019-10-13T10:49:40.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:40.668+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.673627+0000) 2019-10-13T10:49:40.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:40.670+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.676455+0000) 2019-10-13T10:49:40.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:40.670+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.677104+0000) 2019-10-13T10:49:40.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:40.670+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.677387+0000) 2019-10-13T10:49:40.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:40.670+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.677613+0000) 2019-10-13T10:49:40.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:40.674+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.678187+0000) 2019-10-13T10:49:40.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:40.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:40.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:40.676+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.681770+0000) 2019-10-13T10:49:40.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:40.960+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:40.964496+0000) 2019-10-13T10:49:41.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:41.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.225481+0000) 2019-10-13T10:49:41.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:41.242+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.247369+0000) 2019-10-13T10:49:41.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:41.768+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:41.771397+0000) 2019-10-13T10:49:43.644 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:49:43.644 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:43.648 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:49:43.649 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:49:44.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:44.086+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:44.090016+0000) 2019-10-13T10:49:44.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:44.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:44.376290+0000) 2019-10-13T10:49:45.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:45.678+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.682713+0000) 2019-10-13T10:49:45.685 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:45.678+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.683897+0000) 2019-10-13T10:49:45.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:45.678+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.684252+0000) 2019-10-13T10:49:45.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:45.678+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.684484+0000) 2019-10-13T10:49:45.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:45.680+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.684999+0000) 2019-10-13T10:49:45.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:45.678+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.685582+0000) 2019-10-13T10:49:45.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:45.682+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.685933+0000) 2019-10-13T10:49:45.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:45.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:45.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:45.688+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:45.694473+0000) 2019-10-13T10:49:47.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:47.746+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:47.749492+0000) 2019-10-13T10:49:48.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:48.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:48.519563+0000) 2019-10-13T10:49:50.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:50.686+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.692931+0000) 2019-10-13T10:49:50.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:50.686+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.693149+0000) 2019-10-13T10:49:50.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:50.690+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.696063+0000) 2019-10-13T10:49:50.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:50.690+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.696430+0000) 2019-10-13T10:49:50.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:50.690+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.697579+0000) 2019-10-13T10:49:50.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:50.692+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.697518+0000) 2019-10-13T10:49:50.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:50.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.698205+0000) 2019-10-13T10:49:50.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:50.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:50.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:50.696+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.703683+0000) 2019-10-13T10:49:50.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:50.968+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:50.970893+0000) 2019-10-13T10:49:51.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:51.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.225663+0000) 2019-10-13T10:49:51.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:51.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.247551+0000) 2019-10-13T10:49:51.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:51.768+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:51.771605+0000) 2019-10-13T10:49:54.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:54.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:54.090254+0000) 2019-10-13T10:49:54.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:54.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:54.376470+0000) 2019-10-13T10:49:55.704 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:49:55.698+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.703466+0000) 2019-10-13T10:49:55.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:49:55.698+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.705520+0000) 2019-10-13T10:49:55.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:49:55.700+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.705590+0000) 2019-10-13T10:49:55.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:49:55.702+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.707306+0000) 2019-10-13T10:49:55.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:55.702+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.707426+0000) 2019-10-13T10:49:55.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:55.702+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.708616+0000) 2019-10-13T10:49:55.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:49:55.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:49:55.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:49:55.702+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.709350+0000) 2019-10-13T10:49:55.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:49:55.708+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:55.713894+0000) 2019-10-13T10:49:57.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:49:57.746+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:57.749719+0000) 2019-10-13T10:49:58.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:49:58.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:49:58.519754+0000) 2019-10-13T10:50:00.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:00.705+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.712733+0000) 2019-10-13T10:50:00.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:00.705+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.713034+0000) 2019-10-13T10:50:00.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:00.708+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.715445+0000) 2019-10-13T10:50:00.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:00.709+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.716111+0000) 2019-10-13T10:50:00.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:00.713+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.717972+0000) 2019-10-13T10:50:00.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:00.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:00.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:00.713+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.718431+0000) 2019-10-13T10:50:00.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:00.713+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.718789+0000) 2019-10-13T10:50:00.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:00.716+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.723640+0000) 2019-10-13T10:50:00.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:00.976+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:00.977592+0000) 2019-10-13T10:50:01.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:01.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.225799+0000) 2019-10-13T10:50:01.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:01.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.247726+0000) 2019-10-13T10:50:01.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:01.768+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:01.771814+0000) 2019-10-13T10:50:04.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:04.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:04.090462+0000) 2019-10-13T10:50:04.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:04.373+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:04.376628+0000) 2019-10-13T10:50:05.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:05.716+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.722298+0000) 2019-10-13T10:50:05.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:05.717+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.723148+0000) 2019-10-13T10:50:05.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:05.717+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.724798+0000) 2019-10-13T10:50:05.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:05.717+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.725190+0000) 2019-10-13T10:50:05.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:05.717+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.725638+0000) 2019-10-13T10:50:05.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:05.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:05.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:05.721+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.728435+0000) 2019-10-13T10:50:05.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:05.721+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.727995+0000) 2019-10-13T10:50:05.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:05.732+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:05.736581+0000) 2019-10-13T10:50:07.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:07.745+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:07.749907+0000) 2019-10-13T10:50:08.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:08.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:08.519946+0000) 2019-10-13T10:50:10.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:10.724+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.730369+0000) 2019-10-13T10:50:10.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:10.725+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.732661+0000) 2019-10-13T10:50:10.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:10.725+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.733147+0000) 2019-10-13T10:50:10.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:10.725+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.733579+0000) 2019-10-13T10:50:10.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:10.729+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.734476+0000) 2019-10-13T10:50:10.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:10.729+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.736636+0000) 2019-10-13T10:50:10.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:10.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:10.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:10.729+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.737516+0000) 2019-10-13T10:50:10.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:10.736+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.740207+0000) 2019-10-13T10:50:10.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:10.988+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:10.988651+0000) 2019-10-13T10:50:11.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:11.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.225979+0000) 2019-10-13T10:50:11.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:11.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.247935+0000) 2019-10-13T10:50:11.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:11.764+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:11.772042+0000) 2019-10-13T10:50:13.725 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:50:13.725 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:13.729 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:50:13.729 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:14.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:14.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:14.090695+0000) 2019-10-13T10:50:14.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:14.373+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:14.376797+0000) 2019-10-13T10:50:15.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:15.736+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.739842+0000) 2019-10-13T10:50:15.742 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:15.737+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.740915+0000) 2019-10-13T10:50:15.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:15.737+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.741274+0000) 2019-10-13T10:50:15.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:15.741+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.743717+0000) 2019-10-13T10:50:15.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:15.741+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.744115+0000) 2019-10-13T10:50:15.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:15.741+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.744543+0000) 2019-10-13T10:50:15.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:15.741+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.745133+0000) 2019-10-13T10:50:15.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:15.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:15.749 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:15.744+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:15.748492+0000) 2019-10-13T10:50:17.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:17.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:17.750108+0000) 2019-10-13T10:50:18.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:18.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:18.520173+0000) 2019-10-13T10:50:20.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:20.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.749817+0000) 2019-10-13T10:50:20.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:20.749+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.750467+0000) 2019-10-13T10:50:20.752 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:20.749+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.750803+0000) 2019-10-13T10:50:20.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:20.749+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.751079+0000) 2019-10-13T10:50:20.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:20.749+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.752158+0000) 2019-10-13T10:50:20.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:20.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.752702+0000) 2019-10-13T10:50:20.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:20.749+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.752443+0000) 2019-10-13T10:50:20.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:20.748+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.755180+0000) 2019-10-13T10:50:20.756 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:20.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:20.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:20.992+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:20.995192+0000) 2019-10-13T10:50:21.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:21.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.226159+0000) 2019-10-13T10:50:21.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:21.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.248117+0000) 2019-10-13T10:50:21.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:21.768+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:21.772308+0000) 2019-10-13T10:50:24.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:24.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:24.090905+0000) 2019-10-13T10:50:24.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:24.373+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:24.376954+0000) 2019-10-13T10:50:25.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:25.757+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.759556+0000) 2019-10-13T10:50:25.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:25.757+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.760928+0000) 2019-10-13T10:50:25.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:25.757+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.760844+0000) 2019-10-13T10:50:25.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:25.757+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.761608+0000) 2019-10-13T10:50:25.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:25.757+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.761424+0000) 2019-10-13T10:50:25.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:25.761+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.762087+0000) 2019-10-13T10:50:25.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:25.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.762043+0000) 2019-10-13T10:50:25.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:25.761+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:25.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:25.760+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:25.766356+0000) 2019-10-13T10:50:27.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:27.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:27.750338+0000) 2019-10-13T10:50:28.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:28.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:28.520386+0000) 2019-10-13T10:50:30.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:30.765+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.768154+0000) 2019-10-13T10:50:30.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:30.765+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.768771+0000) 2019-10-13T10:50:30.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:30.769+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.769876+0000) 2019-10-13T10:50:30.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:30.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.769939+0000) 2019-10-13T10:50:30.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:30.769+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.770773+0000) 2019-10-13T10:50:30.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:30.769+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.771182+0000) 2019-10-13T10:50:30.772 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:30.769+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.771751+0000) 2019-10-13T10:50:30.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:30.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125490 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:30.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:30.768+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:30.775205+0000) 2019-10-13T10:50:31.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:31.000+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.001611+0000) 2019-10-13T10:50:31.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:31.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.226339+0000) 2019-10-13T10:50:31.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:31.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.248324+0000) 2019-10-13T10:50:31.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:31.768+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:31.772510+0000) 2019-10-13T10:50:34.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:34.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:34.091136+0000) 2019-10-13T10:50:34.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:34.373+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:34.377175+0000) 2019-10-13T10:50:35.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:35.777+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.777974+0000) 2019-10-13T10:50:35.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:35.777+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.778372+0000) 2019-10-13T10:50:35.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:35.777+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.778890+0000) 2019-10-13T10:50:35.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:35.777+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.779087+0000) 2019-10-13T10:50:35.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:35.776+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.779522+0000) 2019-10-13T10:50:35.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:35.777+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.780219+0000) 2019-10-13T10:50:35.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:35.781+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.781802+0000) 2019-10-13T10:50:35.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:35.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125521 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:35.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:35.784+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:35.785601+0000) 2019-10-13T10:50:37.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:37.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:37.750539+0000) 2019-10-13T10:50:38.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:38.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:38.520585+0000) 2019-10-13T10:50:40.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:40.785+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.787013+0000) 2019-10-13T10:50:40.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:40.785+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.787267+0000) 2019-10-13T10:50:40.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:40.784+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.787642+0000) 2019-10-13T10:50:40.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:40.785+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.787723+0000) 2019-10-13T10:50:40.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:40.785+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.788429+0000) 2019-10-13T10:50:40.790 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:40.785+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.788903+0000) 2019-10-13T10:50:40.791 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:40.789+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.790812+0000) 2019-10-13T10:50:40.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:40.789+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:40.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:40.792+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:40.795157+0000) 2019-10-13T10:50:41.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:41.008+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.008060+0000) 2019-10-13T10:50:41.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:41.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.226519+0000) 2019-10-13T10:50:41.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:41.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.248509+0000) 2019-10-13T10:50:41.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:41.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:41.772710+0000) 2019-10-13T10:50:43.779 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:50:43.779 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:43.790 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:50:43.790 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:50:44.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:44.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:44.091342+0000) 2019-10-13T10:50:44.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:44.373+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:44.377330+0000) 2019-10-13T10:50:45.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:45.793+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.796491+0000) 2019-10-13T10:50:45.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:45.793+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.797542+0000) 2019-10-13T10:50:45.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:45.797+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.799658+0000) 2019-10-13T10:50:45.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:45.799+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.802105+0000) 2019-10-13T10:50:45.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:45.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125582 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:45.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:45.801+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.803021+0000) 2019-10-13T10:50:45.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:45.801+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.803186+0000) 2019-10-13T10:50:45.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:45.799+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.803486+0000) 2019-10-13T10:50:45.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:45.801+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:45.803576+0000) 2019-10-13T10:50:47.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:47.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:47.750737+0000) 2019-10-13T10:50:48.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:48.517+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:48.520781+0000) 2019-10-13T10:50:50.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:50.805+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.807001+0000) 2019-10-13T10:50:50.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:50.809+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.810480+0000) 2019-10-13T10:50:50.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:50.809+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.812241+0000) 2019-10-13T10:50:50.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:50.811+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.812182+0000) 2019-10-13T10:50:50.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:50.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:50.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:50.809+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.813174+0000) 2019-10-13T10:50:50.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:50.811+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.813274+0000) 2019-10-13T10:50:50.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:50.813+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.813980+0000) 2019-10-13T10:50:50.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:50.813+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:50.814216+0000) 2019-10-13T10:50:51.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:51.011+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.014132+0000) 2019-10-13T10:50:51.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:51.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.226698+0000) 2019-10-13T10:50:51.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:51.245+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.248709+0000) 2019-10-13T10:50:51.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:51.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:51.772912+0000) 2019-10-13T10:50:54.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:54.089+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:54.091553+0000) 2019-10-13T10:50:54.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:54.377+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:54.377552+0000) 2019-10-13T10:50:55.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:50:55.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.816369+0000) 2019-10-13T10:50:55.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:50:55.813+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.816496+0000) 2019-10-13T10:50:55.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:50:55.813+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.817320+0000) 2019-10-13T10:50:55.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:50:55.815+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.817387+0000) 2019-10-13T10:50:55.819 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:50:55.817+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.818463+0000) 2019-10-13T10:50:55.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:55.817+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.818757+0000) 2019-10-13T10:50:55.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:55.817+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.819127+0000) 2019-10-13T10:50:55.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:50:55.817+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:50:55.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:50:55.819+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:55.821861+0000) 2019-10-13T10:50:57.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:50:57.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:57.750992+0000) 2019-10-13T10:50:58.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:50:58.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:50:58.520972+0000) 2019-10-13T10:51:00.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:00.820+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.824658+0000) 2019-10-13T10:51:00.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:00.820+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.825186+0000) 2019-10-13T10:51:00.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:00.824+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.826227+0000) 2019-10-13T10:51:00.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:00.824+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.828316+0000) 2019-10-13T10:51:00.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:00.827+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.828215+0000) 2019-10-13T10:51:00.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:00.824+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.828791+0000) 2019-10-13T10:51:00.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:00.824+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.829244+0000) 2019-10-13T10:51:00.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:00.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:00.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:00.827+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:00.831183+0000) 2019-10-13T10:51:01.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:01.019+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.020630+0000) 2019-10-13T10:51:01.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:01.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.226880+0000) 2019-10-13T10:51:01.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:01.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.248882+0000) 2019-10-13T10:51:01.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:01.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:01.773085+0000) 2019-10-13T10:51:04.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:04.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:04.091766+0000) 2019-10-13T10:51:04.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:04.376+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:04.377775+0000) 2019-10-13T10:51:05.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:05.832+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.834103+0000) 2019-10-13T10:51:05.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:05.832+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.837279+0000) 2019-10-13T10:51:05.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:05.836+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.837843+0000) 2019-10-13T10:51:05.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:05.836+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.838063+0000) 2019-10-13T10:51:05.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:05.835+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.838137+0000) 2019-10-13T10:51:05.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:05.836+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.838766+0000) 2019-10-13T10:51:05.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:05.836+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.839195+0000) 2019-10-13T10:51:05.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:05.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:05.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:05.839+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:05.842309+0000) 2019-10-13T10:51:07.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:07.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:07.751188+0000) 2019-10-13T10:51:08.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:08.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:08.521197+0000) 2019-10-13T10:51:10.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:10.840+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.844422+0000) 2019-10-13T10:51:10.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:10.843+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.844725+0000) 2019-10-13T10:51:10.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:10.840+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.845211+0000) 2019-10-13T10:51:10.846 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:10.840+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.845485+0000) 2019-10-13T10:51:10.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:10.844+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.845956+0000) 2019-10-13T10:51:10.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:10.844+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.846714+0000) 2019-10-13T10:51:10.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:10.844+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.847031+0000) 2019-10-13T10:51:10.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:10.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:10.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:10.847+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:10.849421+0000) 2019-10-13T10:51:11.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:11.023+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.027751+0000) 2019-10-13T10:51:11.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:11.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.227083+0000) 2019-10-13T10:51:11.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:11.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.249061+0000) 2019-10-13T10:51:11.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:11.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:11.773258+0000) 2019-10-13T10:51:13.903 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:51:13.903 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:13.907 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:51:13.907 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:14.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:14.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:14.091995+0000) 2019-10-13T10:51:14.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:14.376+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:14.378004+0000) 2019-10-13T10:51:15.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:15.848+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.852897+0000) 2019-10-13T10:51:15.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:15.851+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.853048+0000) 2019-10-13T10:51:15.855 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:15.852+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.853886+0000) 2019-10-13T10:51:15.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:15.852+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.854731+0000) 2019-10-13T10:51:15.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:15.852+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.855419+0000) 2019-10-13T10:51:15.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:15.852+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.855975+0000) 2019-10-13T10:51:15.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:15.852+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.856914+0000) 2019-10-13T10:51:15.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:15.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:15.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:15.859+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:15.861821+0000) 2019-10-13T10:51:17.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:17.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:17.751422+0000) 2019-10-13T10:51:18.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:18.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:18.521397+0000) 2019-10-13T10:51:20.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:20.859+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.861922+0000) 2019-10-13T10:51:20.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:20.860+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.862836+0000) 2019-10-13T10:51:20.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:20.860+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.863975+0000) 2019-10-13T10:51:20.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:20.860+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.865535+0000) 2019-10-13T10:51:20.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:20.864+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.865848+0000) 2019-10-13T10:51:20.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:20.864+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.867734+0000) 2019-10-13T10:51:20.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:20.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:20.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:20.864+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.869580+0000) 2019-10-13T10:51:20.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:20.867+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:20.869180+0000) 2019-10-13T10:51:21.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:21.031+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.034257+0000) 2019-10-13T10:51:21.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:21.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.227300+0000) 2019-10-13T10:51:21.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:21.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.249273+0000) 2019-10-13T10:51:21.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:21.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:21.773432+0000) 2019-10-13T10:51:24.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:24.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:24.092225+0000) 2019-10-13T10:51:24.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:24.376+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:24.378204+0000) 2019-10-13T10:51:25.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:25.868+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.873250+0000) 2019-10-13T10:51:25.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:25.872+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.873953+0000) 2019-10-13T10:51:25.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:25.871+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.874367+0000) 2019-10-13T10:51:25.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:25.872+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.874301+0000) 2019-10-13T10:51:25.876 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:25.872+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.875327+0000) 2019-10-13T10:51:25.876 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:25.872+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.875916+0000) 2019-10-13T10:51:25.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:25.872+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.876097+0000) 2019-10-13T10:51:25.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:25.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:25.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:25.875+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:25.878764+0000) 2019-10-13T10:51:27.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:27.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:27.751650+0000) 2019-10-13T10:51:28.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:28.516+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:28.521546+0000) 2019-10-13T10:51:30.885 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:30.880+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.884450+0000) 2019-10-13T10:51:30.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:30.880+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.884936+0000) 2019-10-13T10:51:30.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:30.880+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.885117+0000) 2019-10-13T10:51:30.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:30.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.885815+0000) 2019-10-13T10:51:30.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:30.884+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.886145+0000) 2019-10-13T10:51:30.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:30.884+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.887017+0000) 2019-10-13T10:51:30.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:30.883+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.887623+0000) 2019-10-13T10:51:30.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:30.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:30.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:30.887+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:30.888656+0000) 2019-10-13T10:51:31.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:31.039+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.040598+0000) 2019-10-13T10:51:31.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:31.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.227517+0000) 2019-10-13T10:51:31.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:31.244+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.249473+0000) 2019-10-13T10:51:31.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:31.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:31.773645+0000) 2019-10-13T10:51:34.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:34.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:34.092437+0000) 2019-10-13T10:51:34.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:34.376+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:34.378418+0000) 2019-10-13T10:51:35.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:35.888+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.891566+0000) 2019-10-13T10:51:35.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:35.888+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.893601+0000) 2019-10-13T10:51:35.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:35.888+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.893316+0000) 2019-10-13T10:51:35.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:35.892+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.894824+0000) 2019-10-13T10:51:35.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:35.891+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.895499+0000) 2019-10-13T10:51:35.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:35.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:35.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:35.892+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.897451+0000) 2019-10-13T10:51:35.898 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:35.892+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.897571+0000) 2019-10-13T10:51:35.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:35.899+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:35.902042+0000) 2019-10-13T10:51:37.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:37.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:37.751852+0000) 2019-10-13T10:51:38.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:38.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:38.521769+0000) 2019-10-13T10:51:40.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:40.896+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.901305+0000) 2019-10-13T10:51:40.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:40.899+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.901186+0000) 2019-10-13T10:51:40.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:40.900+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.902277+0000) 2019-10-13T10:51:40.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:40.900+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.902827+0000) 2019-10-13T10:51:40.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:40.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.903256+0000) 2019-10-13T10:51:40.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:40.900+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.903550+0000) 2019-10-13T10:51:40.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:40.900+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.905407+0000) 2019-10-13T10:51:40.907 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:40.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:40.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:40.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:40.915744+0000) 2019-10-13T10:51:41.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:41.047+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.047207+0000) 2019-10-13T10:51:41.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:41.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.227647+0000) 2019-10-13T10:51:41.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:41.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.249612+0000) 2019-10-13T10:51:41.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:41.771+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:41.773854+0000) 2019-10-13T10:51:43.984 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:51:43.984 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:43.988 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:51:43.988 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:51:44.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:44.088+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:44.092642+0000) 2019-10-13T10:51:44.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:44.376+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:44.378651+0000) 2019-10-13T10:51:45.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:45.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.909946+0000) 2019-10-13T10:51:45.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:45.908+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.910535+0000) 2019-10-13T10:51:45.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:45.908+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.910897+0000) 2019-10-13T10:51:45.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:45.908+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.912400+0000) 2019-10-13T10:51:45.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:45.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.917220+0000) 2019-10-13T10:51:45.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:45.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:45.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:45.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.917880+0000) 2019-10-13T10:51:45.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:45.916+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.918156+0000) 2019-10-13T10:51:45.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:45.923+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:45.925393+0000) 2019-10-13T10:51:47.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:47.748+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:47.752050+0000) 2019-10-13T10:51:48.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:48.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:48.521960+0000) 2019-10-13T10:51:50.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:50.918+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.920171+0000) 2019-10-13T10:51:50.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:50.916+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.921002+0000) 2019-10-13T10:51:50.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:50.916+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.921539+0000) 2019-10-13T10:51:50.923 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:50.920+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.922440+0000) 2019-10-13T10:51:50.923 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:50.920+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.922849+0000) 2019-10-13T10:51:50.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:50.920+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.923183+0000) 2019-10-13T10:51:50.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:50.920+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.924037+0000) 2019-10-13T10:51:50.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:50.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:50.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:50.926+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:50.928298+0000) 2019-10-13T10:51:51.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:51.050+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.053898+0000) 2019-10-13T10:51:51.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:51.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.227851+0000) 2019-10-13T10:51:51.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:51.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.249805+0000) 2019-10-13T10:51:51.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:51.770+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:51.774026+0000) 2019-10-13T10:51:54.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:54.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:54.092873+0000) 2019-10-13T10:51:54.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:54.375+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:54.378862+0000) 2019-10-13T10:51:55.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:51:55.926+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.930154+0000) 2019-10-13T10:51:55.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:51:55.927+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.932724+0000) 2019-10-13T10:51:55.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:55.927+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.932779+0000) 2019-10-13T10:51:55.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:51:55.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.933339+0000) 2019-10-13T10:51:55.936 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:55.931+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.935362+0000) 2019-10-13T10:51:55.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:51:55.931+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.935906+0000) 2019-10-13T10:51:55.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:51:55.931+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.936520+0000) 2019-10-13T10:51:55.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:51:55.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:51:55.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:51:55.938+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:55.941899+0000) 2019-10-13T10:51:57.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:51:57.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:57.752256+0000) 2019-10-13T10:51:58.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:51:58.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:51:58.522157+0000) 2019-10-13T10:52:00.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:00.935+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.940201+0000) 2019-10-13T10:52:00.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:00.935+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.941026+0000) 2019-10-13T10:52:00.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:00.938+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.941395+0000) 2019-10-13T10:52:00.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:00.939+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.942578+0000) 2019-10-13T10:52:00.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:00.939+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.943171+0000) 2019-10-13T10:52:00.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:00.939+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.944147+0000) 2019-10-13T10:52:00.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:00.939+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.944494+0000) 2019-10-13T10:52:00.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:00.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:00.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:00.946+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:00.949818+0000) 2019-10-13T10:52:01.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:01.058+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.060614+0000) 2019-10-13T10:52:01.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:01.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.228053+0000) 2019-10-13T10:52:01.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:01.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.249998+0000) 2019-10-13T10:52:01.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:01.770+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:01.774204+0000) 2019-10-13T10:52:04.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:04.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:04.093037+0000) 2019-10-13T10:52:04.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:04.375+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:04.379045+0000) 2019-10-13T10:52:05.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:05.947+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.950387+0000) 2019-10-13T10:52:05.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:05.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.950649+0000) 2019-10-13T10:52:05.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:05.947+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.951350+0000) 2019-10-13T10:52:05.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:05.947+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.953508+0000) 2019-10-13T10:52:05.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:05.951+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.955494+0000) 2019-10-13T10:52:05.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:05.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:05.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:05.951+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.957254+0000) 2019-10-13T10:52:05.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:05.955+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.958720+0000) 2019-10-13T10:52:05.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:05.962+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:05.965066+0000) 2019-10-13T10:52:07.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:07.751+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:07.752458+0000) 2019-10-13T10:52:08.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:08.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:08.522370+0000) 2019-10-13T10:52:10.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:10.959+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.962841+0000) 2019-10-13T10:52:10.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:10.958+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.962855+0000) 2019-10-13T10:52:10.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:10.959+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.963573+0000) 2019-10-13T10:52:10.973 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:10.959+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.964923+0000) 2019-10-13T10:52:10.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:10.959+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.965374+0000) 2019-10-13T10:52:10.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:10.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:10.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:10.963+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.965866+0000) 2019-10-13T10:52:10.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:10.963+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.966639+0000) 2019-10-13T10:52:10.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:10.970+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:10.972933+0000) 2019-10-13T10:52:11.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:11.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.067185+0000) 2019-10-13T10:52:11.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:11.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.228232+0000) 2019-10-13T10:52:11.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:11.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.250170+0000) 2019-10-13T10:52:11.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:11.770+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:11.774408+0000) 2019-10-13T10:52:14.055 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:52:14.055 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:14.059 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:52:14.059 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:14.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:14.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:14.093277+0000) 2019-10-13T10:52:14.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:14.375+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:14.379244+0000) 2019-10-13T10:52:15.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:15.963+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.969229+0000) 2019-10-13T10:52:15.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:15.963+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.969627+0000) 2019-10-13T10:52:15.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:15.967+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.971186+0000) 2019-10-13T10:52:15.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:15.966+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.971311+0000) 2019-10-13T10:52:15.972 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:15.967+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.972086+0000) 2019-10-13T10:52:15.973 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:15.967+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.972760+0000) 2019-10-13T10:52:15.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:15.967+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.973303+0000) 2019-10-13T10:52:15.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:15.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:15.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:15.974+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:15.978375+0000) 2019-10-13T10:52:17.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:17.751+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:17.752663+0000) 2019-10-13T10:52:18.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:18.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:18.522581+0000) 2019-10-13T10:52:20.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:20.975+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.979065+0000) 2019-10-13T10:52:20.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:20.975+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.979310+0000) 2019-10-13T10:52:20.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:20.975+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.979928+0000) 2019-10-13T10:52:20.981 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:20.975+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.980471+0000) 2019-10-13T10:52:20.982 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:20.975+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.979953+0000) 2019-10-13T10:52:20.987 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:20.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.982406+0000) 2019-10-13T10:52:20.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:20.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:20.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:20.978+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.981537+0000) 2019-10-13T10:52:20.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:20.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:20.984715+0000) 2019-10-13T10:52:21.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:21.070+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.073635+0000) 2019-10-13T10:52:21.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:21.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.228437+0000) 2019-10-13T10:52:21.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:21.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.250337+0000) 2019-10-13T10:52:21.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:21.770+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:21.774599+0000) 2019-10-13T10:52:24.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:24.087+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:24.093491+0000) 2019-10-13T10:52:24.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:24.379+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:24.379499+0000) 2019-10-13T10:52:25.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:25.983+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.989000+0000) 2019-10-13T10:52:25.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:25.983+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.989330+0000) 2019-10-13T10:52:25.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:25.987+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.989730+0000) 2019-10-13T10:52:25.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:25.987+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.990082+0000) 2019-10-13T10:52:25.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:25.987+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.990536+0000) 2019-10-13T10:52:25.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:25.987+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.991665+0000) 2019-10-13T10:52:25.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:25.986+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.991870+0000) 2019-10-13T10:52:25.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:25.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:25.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:25.990+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:25.994900+0000) 2019-10-13T10:52:27.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:27.751+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:27.752863+0000) 2019-10-13T10:52:28.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:28.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:28.522781+0000) 2019-10-13T10:52:30.997 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:30.991+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.996696+0000) 2019-10-13T10:52:30.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:30.991+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.997465+0000) 2019-10-13T10:52:30.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:30.995+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.998600+0000) 2019-10-13T10:52:31.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:30.995+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.999170+0000) 2019-10-13T10:52:31.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:30.995+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:30.999712+0000) 2019-10-13T10:52:31.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:30.998+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.000229+0000) 2019-10-13T10:52:31.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:30.995+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.000017+0000) 2019-10-13T10:52:31.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:30.999+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:31.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:31.002+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.007296+0000) 2019-10-13T10:52:31.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:31.078+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.080280+0000) 2019-10-13T10:52:31.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:31.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.228641+0000) 2019-10-13T10:52:31.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:31.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.250521+0000) 2019-10-13T10:52:31.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:31.774+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:31.774778+0000) 2019-10-13T10:52:34.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:34.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:34.093694+0000) 2019-10-13T10:52:34.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:34.379+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:34.379654+0000) 2019-10-13T10:52:36.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:36.003+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.006137+0000) 2019-10-13T10:52:36.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:36.003+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.007009+0000) 2019-10-13T10:52:36.009 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:36.003+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.008743+0000) 2019-10-13T10:52:36.010 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:36.007+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.009989+0000) 2019-10-13T10:52:36.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:36.007+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.012224+0000) 2019-10-13T10:52:36.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:36.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:36.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:36.010+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.012659+0000) 2019-10-13T10:52:36.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:36.007+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.013334+0000) 2019-10-13T10:52:36.016 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:36.010+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:36.015678+0000) 2019-10-13T10:52:37.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:37.747+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:37.753056+0000) 2019-10-13T10:52:38.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:38.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:38.522994+0000) 2019-10-13T10:52:41.017 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:41.011+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.016235+0000) 2019-10-13T10:52:41.017 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:41.011+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.017032+0000) 2019-10-13T10:52:41.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:41.011+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.017555+0000) 2019-10-13T10:52:41.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:41.015+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.018222+0000) 2019-10-13T10:52:41.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:41.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.017689+0000) 2019-10-13T10:52:41.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:41.015+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.018879+0000) 2019-10-13T10:52:41.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:41.014+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.018367+0000) 2019-10-13T10:52:41.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:41.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:41.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:41.022+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.024336+0000) 2019-10-13T10:52:41.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:41.082+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.085391+0000) 2019-10-13T10:52:41.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:41.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.228841+0000) 2019-10-13T10:52:41.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:41.247+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.250699+0000) 2019-10-13T10:52:41.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:41.774+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:41.774959+0000) 2019-10-13T10:52:44.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:44.091+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:44.093920+0000) 2019-10-13T10:52:44.121 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:52:44.122 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:44.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:52:44.125 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:52:44.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:44.379+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:44.379896+0000) 2019-10-13T10:52:46.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:46.023+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.025844+0000) 2019-10-13T10:52:46.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:46.022+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.028006+0000) 2019-10-13T10:52:46.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:46.027+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.030729+0000) 2019-10-13T10:52:46.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:46.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:46.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:46.027+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.032089+0000) 2019-10-13T10:52:46.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:46.027+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.032416+0000) 2019-10-13T10:52:46.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:46.027+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.032749+0000) 2019-10-13T10:52:46.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:46.027+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.033023+0000) 2019-10-13T10:52:46.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:46.030+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:46.035989+0000) 2019-10-13T10:52:47.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:47.751+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:47.753266+0000) 2019-10-13T10:52:48.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:48.519+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:48.523204+0000) 2019-10-13T10:52:51.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:51.030+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.035341+0000) 2019-10-13T10:52:51.036 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:51.030+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.036173+0000) 2019-10-13T10:52:51.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:51.030+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.037529+0000) 2019-10-13T10:52:51.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:51.034+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.037916+0000) 2019-10-13T10:52:51.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:51.034+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.038202+0000) 2019-10-13T10:52:51.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:51.034+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.038149+0000) 2019-10-13T10:52:51.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:51.034+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.038901+0000) 2019-10-13T10:52:51.042 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:51.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:51.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:51.042+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.045355+0000) 2019-10-13T10:52:51.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:51.086+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.090477+0000) 2019-10-13T10:52:51.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:51.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.229035+0000) 2019-10-13T10:52:51.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:51.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.250877+0000) 2019-10-13T10:52:51.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:51.774+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:51.775130+0000) 2019-10-13T10:52:54.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:54.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:54.094124+0000) 2019-10-13T10:52:54.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:54.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:54.380097+0000) 2019-10-13T10:52:56.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:52:56.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.044816+0000) 2019-10-13T10:52:56.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:56.042+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.046081+0000) 2019-10-13T10:52:56.047 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:52:56.042+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.045868+0000) 2019-10-13T10:52:56.048 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:52:56.041+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.046133+0000) 2019-10-13T10:52:56.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:52:56.042+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.046911+0000) 2019-10-13T10:52:56.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:52:56.042+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.047595+0000) 2019-10-13T10:52:56.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:56.042+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.048323+0000) 2019-10-13T10:52:56.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:52:56.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:52:56.057 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:52:56.053+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:56.056681+0000) 2019-10-13T10:52:57.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:52:57.750+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:57.753464+0000) 2019-10-13T10:52:58.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:52:58.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:52:58.523406+0000) 2019-10-13T10:53:01.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:01.046+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.053338+0000) 2019-10-13T10:53:01.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:01.050+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.053926+0000) 2019-10-13T10:53:01.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:01.050+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.054316+0000) 2019-10-13T10:53:01.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:01.050+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.054636+0000) 2019-10-13T10:53:01.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:01.050+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.056382+0000) 2019-10-13T10:53:01.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:01.050+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.056829+0000) 2019-10-13T10:53:01.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:01.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126390 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:01.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:01.057+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.060403+0000) 2019-10-13T10:53:01.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:01.061+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.067199+0000) 2019-10-13T10:53:01.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:01.097+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.100991+0000) 2019-10-13T10:53:01.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:01.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.229234+0000) 2019-10-13T10:53:01.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:01.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.251043+0000) 2019-10-13T10:53:01.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:01.773+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:01.775301+0000) 2019-10-13T10:53:04.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:04.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:04.094325+0000) 2019-10-13T10:53:04.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:04.378+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:04.380309+0000) 2019-10-13T10:53:06.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:06.058+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.063274+0000) 2019-10-13T10:53:06.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:06.058+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.063740+0000) 2019-10-13T10:53:06.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:06.058+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.064156+0000) 2019-10-13T10:53:06.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:06.062+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.066423+0000) 2019-10-13T10:53:06.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:06.062+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.065995+0000) 2019-10-13T10:53:06.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:06.062+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.067455+0000) 2019-10-13T10:53:06.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:06.061+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.067696+0000) 2019-10-13T10:53:06.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:06.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:06.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:06.069+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:06.072575+0000) 2019-10-13T10:53:07.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:07.750+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:07.753694+0000) 2019-10-13T10:53:08.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:08.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:08.523628+0000) 2019-10-13T10:53:11.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:11.066+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.072625+0000) 2019-10-13T10:53:11.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:11.069+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.073491+0000) 2019-10-13T10:53:11.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:11.070+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.074268+0000) 2019-10-13T10:53:11.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:11.070+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.074441+0000) 2019-10-13T10:53:11.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:11.070+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.074960+0000) 2019-10-13T10:53:11.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:11.070+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.075637+0000) 2019-10-13T10:53:11.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:11.070+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.076188+0000) 2019-10-13T10:53:11.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:11.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:11.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:11.077+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.081310+0000) 2019-10-13T10:53:11.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:11.101+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.107273+0000) 2019-10-13T10:53:11.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:11.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.229413+0000) 2019-10-13T10:53:11.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:11.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.251251+0000) 2019-10-13T10:53:11.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:11.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:11.775471+0000) 2019-10-13T10:53:14.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:14.090+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:14.094529+0000) 2019-10-13T10:53:14.240 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:53:14.240 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:14.244 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:53:14.244 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:14.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:14.378+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:14.380509+0000) 2019-10-13T10:53:16.082 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:16.078+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.081717+0000) 2019-10-13T10:53:16.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:16.078+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.083182+0000) 2019-10-13T10:53:16.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:16.077+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.083545+0000) 2019-10-13T10:53:16.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:16.078+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.084096+0000) 2019-10-13T10:53:16.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:16.078+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.084215+0000) 2019-10-13T10:53:16.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:16.078+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.084722+0000) 2019-10-13T10:53:16.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:16.078+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.085645+0000) 2019-10-13T10:53:16.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:16.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:16.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:16.085+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:16.088477+0000) 2019-10-13T10:53:17.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:17.750+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:17.753931+0000) 2019-10-13T10:53:18.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:18.518+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:18.523811+0000) 2019-10-13T10:53:21.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:21.086+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.091048+0000) 2019-10-13T10:53:21.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:21.086+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.091524+0000) 2019-10-13T10:53:21.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:21.086+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.091780+0000) 2019-10-13T10:53:21.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:21.086+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.092583+0000) 2019-10-13T10:53:21.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:21.086+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.092788+0000) 2019-10-13T10:53:21.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:21.089+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.093510+0000) 2019-10-13T10:53:21.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:21.090+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.094805+0000) 2019-10-13T10:53:21.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:21.093+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.100032+0000) 2019-10-13T10:53:21.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:21.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:21.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:21.109+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.112388+0000) 2019-10-13T10:53:21.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:21.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.229593+0000) 2019-10-13T10:53:21.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:21.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.251428+0000) 2019-10-13T10:53:21.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:21.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:21.775644+0000) 2019-10-13T10:53:24.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:24.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:24.094737+0000) 2019-10-13T10:53:24.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:24.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:24.380710+0000) 2019-10-13T10:53:26.102 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:26.094+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.101533+0000) 2019-10-13T10:53:26.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:26.098+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.102000+0000) 2019-10-13T10:53:26.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:26.098+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.102525+0000) 2019-10-13T10:53:26.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:26.098+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.102916+0000) 2019-10-13T10:53:26.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:26.098+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.103627+0000) 2019-10-13T10:53:26.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:26.097+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.103698+0000) 2019-10-13T10:53:26.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:26.098+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.104932+0000) 2019-10-13T10:53:26.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:26.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:26.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:26.105+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:26.110403+0000) 2019-10-13T10:53:27.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:27.750+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:27.754135+0000) 2019-10-13T10:53:28.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:28.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:28.524004+0000) 2019-10-13T10:53:31.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:31.106+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.111239+0000) 2019-10-13T10:53:31.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:31.106+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.111690+0000) 2019-10-13T10:53:31.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:31.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.112723+0000) 2019-10-13T10:53:31.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:31.106+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.112373+0000) 2019-10-13T10:53:31.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:31.109+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.115168+0000) 2019-10-13T10:53:31.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:31.113+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.119629+0000) 2019-10-13T10:53:31.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:31.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.119702+0000) 2019-10-13T10:53:31.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:31.110+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.113828+0000) 2019-10-13T10:53:31.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:31.110+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.114882+0000) 2019-10-13T10:53:31.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:31.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:31.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:31.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.229770+0000) 2019-10-13T10:53:31.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:31.246+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.251604+0000) 2019-10-13T10:53:31.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:31.769+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:31.775814+0000) 2019-10-13T10:53:34.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:34.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:34.094973+0000) 2019-10-13T10:53:34.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:34.378+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:34.380923+0000) 2019-10-13T10:53:36.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:36.114+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.120300+0000) 2019-10-13T10:53:36.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:36.114+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.120785+0000) 2019-10-13T10:53:36.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:36.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.121503+0000) 2019-10-13T10:53:36.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:36.114+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.121556+0000) 2019-10-13T10:53:36.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:36.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.121853+0000) 2019-10-13T10:53:36.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:36.118+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.123352+0000) 2019-10-13T10:53:36.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:36.118+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.124158+0000) 2019-10-13T10:53:36.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:36.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:36.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:36.125+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:36.129147+0000) 2019-10-13T10:53:37.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:37.750+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:37.754349+0000) 2019-10-13T10:53:38.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:38.522+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:38.524235+0000) 2019-10-13T10:53:41.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:41.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.119893+0000) 2019-10-13T10:53:41.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:41.126+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.129447+0000) 2019-10-13T10:53:41.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:41.125+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.130704+0000) 2019-10-13T10:53:41.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:41.126+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.130805+0000) 2019-10-13T10:53:41.132 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:41.126+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.131487+0000) 2019-10-13T10:53:41.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:41.126+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.131893+0000) 2019-10-13T10:53:41.133 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:41.126+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.132331+0000) 2019-10-13T10:53:41.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:41.126+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.132881+0000) 2019-10-13T10:53:41.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:41.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:41.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:41.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.140274+0000) 2019-10-13T10:53:41.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:41.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.229956+0000) 2019-10-13T10:53:41.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:41.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.251782+0000) 2019-10-13T10:53:41.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:41.773+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:41.776006+0000) 2019-10-13T10:53:44.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:44.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:44.095168+0000) 2019-10-13T10:53:44.316 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:53:44.317 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:44.320 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:53:44.321 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:53:44.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:44.374+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:44.381156+0000) 2019-10-13T10:53:46.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:46.133+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.138812+0000) 2019-10-13T10:53:46.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:46.134+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.139618+0000) 2019-10-13T10:53:46.143 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:46.134+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.140840+0000) 2019-10-13T10:53:46.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:46.134+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.141217+0000) 2019-10-13T10:53:46.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:46.134+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.141331+0000) 2019-10-13T10:53:46.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:46.138+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.141835+0000) 2019-10-13T10:53:46.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:46.138+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.142763+0000) 2019-10-13T10:53:46.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:46.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:46.154 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:46.149+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:46.152890+0000) 2019-10-13T10:53:47.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:47.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:47.754504+0000) 2019-10-13T10:53:48.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:48.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:48.524425+0000) 2019-10-13T10:53:51.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:51.125+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.126678+0000) 2019-10-13T10:53:51.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:51.141+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.149220+0000) 2019-10-13T10:53:51.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:51.145+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.149392+0000) 2019-10-13T10:53:51.151 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:51.145+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.150907+0000) 2019-10-13T10:53:51.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:51.145+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.151279+0000) 2019-10-13T10:53:51.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:51.145+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.151601+0000) 2019-10-13T10:53:51.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:51.145+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.152254+0000) 2019-10-13T10:53:51.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:51.145+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.152948+0000) 2019-10-13T10:53:51.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:51.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:51.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:51.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.156372+0000) 2019-10-13T10:53:51.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:51.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.230138+0000) 2019-10-13T10:53:51.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:51.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.251962+0000) 2019-10-13T10:53:51.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:51.773+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:51.776182+0000) 2019-10-13T10:53:54.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:54.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:54.095364+0000) 2019-10-13T10:53:54.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:54.377+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:54.381315+0000) 2019-10-13T10:53:56.159 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:56.153+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.158700+0000) 2019-10-13T10:53:56.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:53:56.153+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.159615+0000) 2019-10-13T10:53:56.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:53:56.153+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.159913+0000) 2019-10-13T10:53:56.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:56.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.160257+0000) 2019-10-13T10:53:56.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:53:56.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.160390+0000) 2019-10-13T10:53:56.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:53:56.153+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.161103+0000) 2019-10-13T10:53:56.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:53:56.157+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.162906+0000) 2019-10-13T10:53:56.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:53:56.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:53:56.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:53:56.165+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:56.168609+0000) 2019-10-13T10:53:57.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:53:57.749+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:57.754717+0000) 2019-10-13T10:53:58.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:53:58.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:53:58.524618+0000) 2019-10-13T10:54:01.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:01.132+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.133348+0000) 2019-10-13T10:54:01.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:01.165+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.168096+0000) 2019-10-13T10:54:01.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:01.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.169815+0000) 2019-10-13T10:54:01.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:01.165+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.170187+0000) 2019-10-13T10:54:01.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:01.165+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.170809+0000) 2019-10-13T10:54:01.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:01.164+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.170597+0000) 2019-10-13T10:54:01.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:01.165+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.171260+0000) 2019-10-13T10:54:01.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:01.165+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.171847+0000) 2019-10-13T10:54:01.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:01.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:01.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:01.172+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.176676+0000) 2019-10-13T10:54:01.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:01.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.230344+0000) 2019-10-13T10:54:01.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:01.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.252143+0000) 2019-10-13T10:54:01.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:01.772+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:01.776385+0000) 2019-10-13T10:54:04.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:04.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:04.095563+0000) 2019-10-13T10:54:04.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:04.377+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:04.381513+0000) 2019-10-13T10:54:06.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:06.173+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.177164+0000) 2019-10-13T10:54:06.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:06.177+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.177815+0000) 2019-10-13T10:54:06.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:06.177+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.177937+0000) 2019-10-13T10:54:06.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:06.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.179035+0000) 2019-10-13T10:54:06.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:06.172+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.178958+0000) 2019-10-13T10:54:06.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:06.177+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.179773+0000) 2019-10-13T10:54:06.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:06.177+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.180531+0000) 2019-10-13T10:54:06.184 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:06.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126771 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:06.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:06.184+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:06.189199+0000) 2019-10-13T10:54:07.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:07.753+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:07.754941+0000) 2019-10-13T10:54:08.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:08.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:08.524809+0000) 2019-10-13T10:54:11.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:11.140+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.144388+0000) 2019-10-13T10:54:11.188 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:11.185+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.187334+0000) 2019-10-13T10:54:11.189 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:11.185+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.188655+0000) 2019-10-13T10:54:11.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:11.185+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.188912+0000) 2019-10-13T10:54:11.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:11.185+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.189325+0000) 2019-10-13T10:54:11.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:11.184+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.189437+0000) 2019-10-13T10:54:11.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:11.189+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.190078+0000) 2019-10-13T10:54:11.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:11.189+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.190830+0000) 2019-10-13T10:54:11.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:11.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:11.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:11.192+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.196448+0000) 2019-10-13T10:54:11.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:11.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.230553+0000) 2019-10-13T10:54:11.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:11.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.252319+0000) 2019-10-13T10:54:11.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:11.772+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:11.776519+0000) 2019-10-13T10:54:14.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:14.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:14.095764+0000) 2019-10-13T10:54:14.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:14.377+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:14.381709+0000) 2019-10-13T10:54:14.443 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:54:14.443 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:14.447 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:54:14.447 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:16.197 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:16.193+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.196863+0000) 2019-10-13T10:54:16.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:16.197+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.199006+0000) 2019-10-13T10:54:16.200 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:16.192+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.199145+0000) 2019-10-13T10:54:16.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:16.197+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.200024+0000) 2019-10-13T10:54:16.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:16.197+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.200579+0000) 2019-10-13T10:54:16.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:16.197+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.201223+0000) 2019-10-13T10:54:16.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:16.197+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.201482+0000) 2019-10-13T10:54:16.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:16.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:16.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:16.200+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:16.206340+0000) 2019-10-13T10:54:17.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:17.753+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:17.755170+0000) 2019-10-13T10:54:18.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:18.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:18.525004+0000) 2019-10-13T10:54:21.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:21.148+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.150485+0000) 2019-10-13T10:54:21.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:21.205+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.207361+0000) 2019-10-13T10:54:21.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:21.205+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.207628+0000) 2019-10-13T10:54:21.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:21.200+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.207869+0000) 2019-10-13T10:54:21.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:21.205+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.208358+0000) 2019-10-13T10:54:21.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:21.205+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.209050+0000) 2019-10-13T10:54:21.210 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:21.205+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.209652+0000) 2019-10-13T10:54:21.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:21.209+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.210150+0000) 2019-10-13T10:54:21.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:21.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:21.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:21.208+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.214805+0000) 2019-10-13T10:54:21.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:21.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.230730+0000) 2019-10-13T10:54:21.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:21.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.252495+0000) 2019-10-13T10:54:21.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:21.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:21.776718+0000) 2019-10-13T10:54:24.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:24.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:24.095955+0000) 2019-10-13T10:54:24.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:24.381+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:24.381940+0000) 2019-10-13T10:54:26.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:26.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.215901+0000) 2019-10-13T10:54:26.217 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:26.213+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.216248+0000) 2019-10-13T10:54:26.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:26.217+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.218003+0000) 2019-10-13T10:54:26.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:26.212+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.217898+0000) 2019-10-13T10:54:26.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:26.217+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.218650+0000) 2019-10-13T10:54:26.220 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:26.217+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.219278+0000) 2019-10-13T10:54:26.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:26.217+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.220156+0000) 2019-10-13T10:54:26.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:26.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:26.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:26.220+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:26.225649+0000) 2019-10-13T10:54:27.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:27.753+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:27.755368+0000) 2019-10-13T10:54:28.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:28.521+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:28.525227+0000) 2019-10-13T10:54:31.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:31.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.157063+0000) 2019-10-13T10:54:31.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:31.225+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.226218+0000) 2019-10-13T10:54:31.227 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:31.220+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.226290+0000) 2019-10-13T10:54:31.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:31.225+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.227480+0000) 2019-10-13T10:54:31.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:31.225+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.227749+0000) 2019-10-13T10:54:31.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:31.225+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.228372+0000) 2019-10-13T10:54:31.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:31.225+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.228602+0000) 2019-10-13T10:54:31.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:31.225+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.229136+0000) 2019-10-13T10:54:31.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:31.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.230864+0000) 2019-10-13T10:54:31.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:31.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:31.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:31.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.235542+0000) 2019-10-13T10:54:31.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:31.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.252652+0000) 2019-10-13T10:54:31.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:31.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:31.776909+0000) 2019-10-13T10:54:34.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:34.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:34.096144+0000) 2019-10-13T10:54:34.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:34.381+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:34.382166+0000) 2019-10-13T10:54:36.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:36.233+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.236150+0000) 2019-10-13T10:54:36.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:36.233+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.236882+0000) 2019-10-13T10:54:36.238 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:36.232+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.237489+0000) 2019-10-13T10:54:36.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:36.237+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.238357+0000) 2019-10-13T10:54:36.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:36.237+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.240013+0000) 2019-10-13T10:54:36.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:36.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:36.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:36.241+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.242813+0000) 2019-10-13T10:54:36.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:36.241+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.243223+0000) 2019-10-13T10:54:36.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:36.240+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:36.244694+0000) 2019-10-13T10:54:37.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:37.753+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:37.755582+0000) 2019-10-13T10:54:38.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:38.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:38.525460+0000) 2019-10-13T10:54:41.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:41.160+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.163563+0000) 2019-10-13T10:54:41.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:41.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.231044+0000) 2019-10-13T10:54:41.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:41.241+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.245032+0000) 2019-10-13T10:54:41.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:41.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.246280+0000) 2019-10-13T10:54:41.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:41.240+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.246668+0000) 2019-10-13T10:54:41.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:41.245+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.247212+0000) 2019-10-13T10:54:41.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:41.245+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.247953+0000) 2019-10-13T10:54:41.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:41.245+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.248821+0000) 2019-10-13T10:54:41.250 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:41.245+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.249460+0000) 2019-10-13T10:54:41.254 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:41.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:41.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:41.249+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.252784+0000) 2019-10-13T10:54:41.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:41.252+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.256103+0000) 2019-10-13T10:54:41.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:41.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:41.777080+0000) 2019-10-13T10:54:44.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:44.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:44.096344+0000) 2019-10-13T10:54:44.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:44.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:44.382365+0000) 2019-10-13T10:54:44.548 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:54:44.548 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:44.551 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:54:44.552 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:54:46.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:46.252+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.254680+0000) 2019-10-13T10:54:46.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:46.252+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.255334+0000) 2019-10-13T10:54:46.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:46.252+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.255975+0000) 2019-10-13T10:54:46.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:46.252+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.256590+0000) 2019-10-13T10:54:46.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:46.252+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.257098+0000) 2019-10-13T10:54:46.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:46.252+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.257626+0000) 2019-10-13T10:54:46.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:46.256+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.258352+0000) 2019-10-13T10:54:46.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:46.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127007 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:46.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:46.260+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:46.264521+0000) 2019-10-13T10:54:47.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:47.752+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:47.755823+0000) 2019-10-13T10:54:48.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:48.520+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:48.525684+0000) 2019-10-13T10:54:51.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:51.168+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.170330+0000) 2019-10-13T10:54:51.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:51.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.231262+0000) 2019-10-13T10:54:51.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:51.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.252994+0000) 2019-10-13T10:54:51.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:51.260+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.264366+0000) 2019-10-13T10:54:51.265 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:51.260+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.264882+0000) 2019-10-13T10:54:51.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:51.260+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.265273+0000) 2019-10-13T10:54:51.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:51.264+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.265873+0000) 2019-10-13T10:54:51.268 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:51.264+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.267598+0000) 2019-10-13T10:54:51.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:51.260+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.267516+0000) 2019-10-13T10:54:51.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:51.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.269901+0000) 2019-10-13T10:54:51.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:51.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:51.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:51.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.275899+0000) 2019-10-13T10:54:51.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:51.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:51.777255+0000) 2019-10-13T10:54:54.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:54.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:54.096542+0000) 2019-10-13T10:54:54.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:54.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:54.382593+0000) 2019-10-13T10:54:56.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:54:56.272+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.274245+0000) 2019-10-13T10:54:56.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:54:56.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.274260+0000) 2019-10-13T10:54:56.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:54:56.272+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.275114+0000) 2019-10-13T10:54:56.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:56.272+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.275578+0000) 2019-10-13T10:54:56.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:54:56.272+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.276714+0000) 2019-10-13T10:54:56.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:56.276+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.278808+0000) 2019-10-13T10:54:56.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:54:56.276+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.280547+0000) 2019-10-13T10:54:56.281 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:54:56.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:54:56.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:54:56.276+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:56.283355+0000) 2019-10-13T10:54:57.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:54:57.752+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:57.756027+0000) 2019-10-13T10:54:58.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:54:58.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:54:58.525949+0000) 2019-10-13T10:55:01.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:01.176+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.177179+0000) 2019-10-13T10:55:01.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:01.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.231467+0000) 2019-10-13T10:55:01.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:01.248+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.253221+0000) 2019-10-13T10:55:01.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:01.280+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.284474+0000) 2019-10-13T10:55:01.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:01.280+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.284843+0000) 2019-10-13T10:55:01.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:01.280+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.285515+0000) 2019-10-13T10:55:01.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:01.284+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.286076+0000) 2019-10-13T10:55:01.287 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:01.284+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.286864+0000) 2019-10-13T10:55:01.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:01.284+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.286970+0000) 2019-10-13T10:55:01.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:01.284+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.287801+0000) 2019-10-13T10:55:01.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:01.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:01.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:01.292+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.292667+0000) 2019-10-13T10:55:01.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:01.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:01.777431+0000) 2019-10-13T10:55:04.097 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:04.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:04.096737+0000) 2019-10-13T10:55:04.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:04.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:04.382811+0000) 2019-10-13T10:55:06.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:06.292+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.294669+0000) 2019-10-13T10:55:06.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:06.292+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.295118+0000) 2019-10-13T10:55:06.296 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:06.292+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.295773+0000) 2019-10-13T10:55:06.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:06.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.295716+0000) 2019-10-13T10:55:06.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:06.292+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.296993+0000) 2019-10-13T10:55:06.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:06.292+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.297461+0000) 2019-10-13T10:55:06.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:06.296+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.298209+0000) 2019-10-13T10:55:06.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:06.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:06.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:06.300+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:06.303760+0000) 2019-10-13T10:55:07.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:07.752+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:07.756227+0000) 2019-10-13T10:55:08.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:08.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:08.526144+0000) 2019-10-13T10:55:11.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:11.179+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.183701+0000) 2019-10-13T10:55:11.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:11.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.231694+0000) 2019-10-13T10:55:11.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:11.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.253449+0000) 2019-10-13T10:55:11.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:11.300+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.303957+0000) 2019-10-13T10:55:11.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:11.300+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.304246+0000) 2019-10-13T10:55:11.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:11.300+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.304565+0000) 2019-10-13T10:55:11.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:11.300+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.305215+0000) 2019-10-13T10:55:11.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:11.304+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.306318+0000) 2019-10-13T10:55:11.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:11.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.306771+0000) 2019-10-13T10:55:11.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:11.304+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.307712+0000) 2019-10-13T10:55:11.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:11.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:11.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:11.307+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.311908+0000) 2019-10-13T10:55:11.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:11.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:11.777660+0000) 2019-10-13T10:55:14.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:14.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:14.096961+0000) 2019-10-13T10:55:14.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:14.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:14.383010+0000) 2019-10-13T10:55:14.671 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:55:14.671 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:14.675 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:55:14.675 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:16.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:16.312+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.314896+0000) 2019-10-13T10:55:16.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:16.312+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.316478+0000) 2019-10-13T10:55:16.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:16.312+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.316755+0000) 2019-10-13T10:55:16.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:16.315+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.316846+0000) 2019-10-13T10:55:16.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:16.316+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.317730+0000) 2019-10-13T10:55:16.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:16.316+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.319358+0000) 2019-10-13T10:55:16.320 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:16.316+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.319684+0000) 2019-10-13T10:55:16.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:16.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:16.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:16.323+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:16.327123+0000) 2019-10-13T10:55:17.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:17.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:17.756453+0000) 2019-10-13T10:55:18.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:18.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:18.526334+0000) 2019-10-13T10:55:21.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:21.187+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.190212+0000) 2019-10-13T10:55:21.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:21.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.231891+0000) 2019-10-13T10:55:21.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:21.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.253648+0000) 2019-10-13T10:55:21.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:21.320+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.324805+0000) 2019-10-13T10:55:21.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:21.320+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.325274+0000) 2019-10-13T10:55:21.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:21.323+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.326965+0000) 2019-10-13T10:55:21.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:21.324+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.327749+0000) 2019-10-13T10:55:21.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:21.324+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.327946+0000) 2019-10-13T10:55:21.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:21.324+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.328855+0000) 2019-10-13T10:55:21.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:21.328+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.330885+0000) 2019-10-13T10:55:21.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:21.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:21.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:21.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.336727+0000) 2019-10-13T10:55:21.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:21.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:21.777866+0000) 2019-10-13T10:55:24.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:24.092+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:24.097161+0000) 2019-10-13T10:55:24.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:24.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:24.383243+0000) 2019-10-13T10:55:26.335 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:26.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.334488+0000) 2019-10-13T10:55:26.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:26.332+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.335360+0000) 2019-10-13T10:55:26.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:26.332+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.336047+0000) 2019-10-13T10:55:26.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:26.332+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.336393+0000) 2019-10-13T10:55:26.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:26.336+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.338096+0000) 2019-10-13T10:55:26.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:26.336+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.338194+0000) 2019-10-13T10:55:26.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:26.336+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.339311+0000) 2019-10-13T10:55:26.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:26.336+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:26.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:26.343+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:26.346956+0000) 2019-10-13T10:55:27.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:27.752+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:27.756651+0000) 2019-10-13T10:55:28.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:28.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:28.526534+0000) 2019-10-13T10:55:31.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:31.195+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.196779+0000) 2019-10-13T10:55:31.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:31.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.232109+0000) 2019-10-13T10:55:31.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:31.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.253870+0000) 2019-10-13T10:55:31.345 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:31.343+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.344436+0000) 2019-10-13T10:55:31.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:31.340+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.345344+0000) 2019-10-13T10:55:31.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:31.344+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.346304+0000) 2019-10-13T10:55:31.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:31.344+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.346517+0000) 2019-10-13T10:55:31.347 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:31.344+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.346989+0000) 2019-10-13T10:55:31.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:31.344+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.347865+0000) 2019-10-13T10:55:31.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:31.344+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.347161+0000) 2019-10-13T10:55:31.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:31.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:31.352 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:31.347+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.351664+0000) 2019-10-13T10:55:31.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:31.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:31.778072+0000) 2019-10-13T10:55:34.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:34.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:34.097340+0000) 2019-10-13T10:55:34.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:34.380+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:34.383441+0000) 2019-10-13T10:55:36.356 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:36.352+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.355108+0000) 2019-10-13T10:55:36.356 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:36.352+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.355527+0000) 2019-10-13T10:55:36.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:36.352+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.356184+0000) 2019-10-13T10:55:36.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:36.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.355953+0000) 2019-10-13T10:55:36.358 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:36.352+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.357491+0000) 2019-10-13T10:55:36.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:36.356+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.357945+0000) 2019-10-13T10:55:36.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:36.356+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.358281+0000) 2019-10-13T10:55:36.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:36.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:36.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:36.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:36.362716+0000) 2019-10-13T10:55:37.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:37.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:37.756863+0000) 2019-10-13T10:55:38.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:38.524+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:38.526726+0000) 2019-10-13T10:55:41.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:41.199+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.202952+0000) 2019-10-13T10:55:41.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:41.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.232338+0000) 2019-10-13T10:55:41.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:41.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.254045+0000) 2019-10-13T10:55:41.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:41.359+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.364225+0000) 2019-10-13T10:55:41.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:41.359+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.364454+0000) 2019-10-13T10:55:41.366 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:41.359+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.365606+0000) 2019-10-13T10:55:41.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:41.363+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.366258+0000) 2019-10-13T10:55:41.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:41.363+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.366460+0000) 2019-10-13T10:55:41.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:41.363+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.366612+0000) 2019-10-13T10:55:41.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:41.363+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.367166+0000) 2019-10-13T10:55:41.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:41.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:41.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:41.371+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.373000+0000) 2019-10-13T10:55:41.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:41.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:41.778250+0000) 2019-10-13T10:55:44.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:44.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:44.097551+0000) 2019-10-13T10:55:44.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:44.379+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:44.383651+0000) 2019-10-13T10:55:44.792 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:55:44.792 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:44.796 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:55:44.796 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:55:46.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:46.367+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.372632+0000) 2019-10-13T10:55:46.374 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:46.371+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.373165+0000) 2019-10-13T10:55:46.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:46.371+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.373848+0000) 2019-10-13T10:55:46.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:46.371+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.374449+0000) 2019-10-13T10:55:46.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:46.371+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.375403+0000) 2019-10-13T10:55:46.376 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:46.371+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.375902+0000) 2019-10-13T10:55:46.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:46.371+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.376555+0000) 2019-10-13T10:55:46.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:46.375+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:46.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:46.375+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:46.380036+0000) 2019-10-13T10:55:47.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:47.755+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:47.757097+0000) 2019-10-13T10:55:48.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:48.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:48.526922+0000) 2019-10-13T10:55:51.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:51.207+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.209172+0000) 2019-10-13T10:55:51.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:51.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.232570+0000) 2019-10-13T10:55:51.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:51.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.254315+0000) 2019-10-13T10:55:51.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:51.379+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.383272+0000) 2019-10-13T10:55:51.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:51.379+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.383716+0000) 2019-10-13T10:55:51.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:51.379+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.384654+0000) 2019-10-13T10:55:51.386 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:51.383+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.384655+0000) 2019-10-13T10:55:51.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:51.379+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.385131+0000) 2019-10-13T10:55:51.387 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:51.383+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.387140+0000) 2019-10-13T10:55:51.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:51.383+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.387795+0000) 2019-10-13T10:55:51.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:51.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:51.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:51.391+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.394006+0000) 2019-10-13T10:55:51.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:51.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:51.778475+0000) 2019-10-13T10:55:54.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:54.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:54.097772+0000) 2019-10-13T10:55:54.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:54.383+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:54.383867+0000) 2019-10-13T10:55:56.394 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:56.387+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.393045+0000) 2019-10-13T10:55:56.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:55:56.387+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.393331+0000) 2019-10-13T10:55:56.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:56.391+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.394668+0000) 2019-10-13T10:55:56.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:55:56.391+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.394996+0000) 2019-10-13T10:55:56.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:55:56.391+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.395439+0000) 2019-10-13T10:55:56.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:55:56.391+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.395554+0000) 2019-10-13T10:55:56.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:55:56.391+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.397218+0000) 2019-10-13T10:55:56.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:55:56.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:55:56.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:55:56.403+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:56.405375+0000) 2019-10-13T10:55:57.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:55:57.751+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:57.757291+0000) 2019-10-13T10:55:58.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:55:58.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:55:58.527113+0000) 2019-10-13T10:56:01.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:01.215+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.215890+0000) 2019-10-13T10:56:01.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:01.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.232764+0000) 2019-10-13T10:56:01.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:01.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.254533+0000) 2019-10-13T10:56:01.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:01.399+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.403299+0000) 2019-10-13T10:56:01.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:01.399+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.403565+0000) 2019-10-13T10:56:01.405 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:01.399+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.404339+0000) 2019-10-13T10:56:01.405 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:01.399+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.404671+0000) 2019-10-13T10:56:01.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:01.399+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.404984+0000) 2019-10-13T10:56:01.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:01.403+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.405568+0000) 2019-10-13T10:56:01.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:01.403+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.406144+0000) 2019-10-13T10:56:01.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:01.407+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.408485+0000) 2019-10-13T10:56:01.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:01.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:01.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:01.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:01.778645+0000) 2019-10-13T10:56:04.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:04.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:04.097953+0000) 2019-10-13T10:56:04.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:04.383+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:04.384079+0000) 2019-10-13T10:56:06.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:06.411+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.412604+0000) 2019-10-13T10:56:06.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:06.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.414459+0000) 2019-10-13T10:56:06.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:06.411+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.417046+0000) 2019-10-13T10:56:06.419 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:06.415+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.418854+0000) 2019-10-13T10:56:06.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:06.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:06.420 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:06.415+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.419294+0000) 2019-10-13T10:56:06.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:06.415+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.420051+0000) 2019-10-13T10:56:06.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:06.415+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.420294+0000) 2019-10-13T10:56:06.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:06.415+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:06.421011+0000) 2019-10-13T10:56:07.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:07.755+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:07.757491+0000) 2019-10-13T10:56:08.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:08.523+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:08.527334+0000) 2019-10-13T10:56:11.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:11.219+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.222322+0000) 2019-10-13T10:56:11.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:11.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.232936+0000) 2019-10-13T10:56:11.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:11.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.254770+0000) 2019-10-13T10:56:11.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:11.419+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.423056+0000) 2019-10-13T10:56:11.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:11.419+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.423577+0000) 2019-10-13T10:56:11.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:11.419+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.424538+0000) 2019-10-13T10:56:11.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:11.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.425099+0000) 2019-10-13T10:56:11.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:11.423+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.425209+0000) 2019-10-13T10:56:11.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:11.423+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.426129+0000) 2019-10-13T10:56:11.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:11.423+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.426704+0000) 2019-10-13T10:56:11.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:11.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:11.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:11.431+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.432303+0000) 2019-10-13T10:56:11.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:11.775+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:11.778819+0000) 2019-10-13T10:56:14.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:14.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:14.098142+0000) 2019-10-13T10:56:14.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:14.379+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:14.384303+0000) 2019-10-13T10:56:14.840 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:56:14.840 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:14.845 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:56:14.845 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:16.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:16.427+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.432402+0000) 2019-10-13T10:56:16.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:16.431+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.435073+0000) 2019-10-13T10:56:16.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:16.431+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.435387+0000) 2019-10-13T10:56:16.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:16.431+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.437422+0000) 2019-10-13T10:56:16.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:16.435+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.437808+0000) 2019-10-13T10:56:16.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:16.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.438120+0000) 2019-10-13T10:56:16.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:16.435+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.438845+0000) 2019-10-13T10:56:16.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:16.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:16.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:16.442+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:16.445054+0000) 2019-10-13T10:56:17.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:17.755+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:17.757727+0000) 2019-10-13T10:56:18.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:18.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:18.527567+0000) 2019-10-13T10:56:21.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:21.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.228925+0000) 2019-10-13T10:56:21.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:21.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.233145+0000) 2019-10-13T10:56:21.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:21.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.254973+0000) 2019-10-13T10:56:21.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:21.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.443149+0000) 2019-10-13T10:56:21.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:21.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.443181+0000) 2019-10-13T10:56:21.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:21.439+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.444281+0000) 2019-10-13T10:56:21.445 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:21.439+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.444782+0000) 2019-10-13T10:56:21.446 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:21.439+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.445429+0000) 2019-10-13T10:56:21.446 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:21.439+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.445588+0000) 2019-10-13T10:56:21.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:21.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.446961+0000) 2019-10-13T10:56:21.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:21.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:21.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:21.450+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.452485+0000) 2019-10-13T10:56:21.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:21.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:21.779013+0000) 2019-10-13T10:56:24.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:24.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:24.098347+0000) 2019-10-13T10:56:24.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:24.383+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:24.384549+0000) 2019-10-13T10:56:26.453 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:26.447+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.452432+0000) 2019-10-13T10:56:26.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:26.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.452719+0000) 2019-10-13T10:56:26.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:26.447+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.453179+0000) 2019-10-13T10:56:26.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:26.451+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.454916+0000) 2019-10-13T10:56:26.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:26.451+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.456796+0000) 2019-10-13T10:56:26.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:26.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:26.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:26.455+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.458267+0000) 2019-10-13T10:56:26.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:26.454+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.458659+0000) 2019-10-13T10:56:26.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:26.455+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:26.459156+0000) 2019-10-13T10:56:27.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:27.755+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:27.757934+0000) 2019-10-13T10:56:28.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:28.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:28.527717+0000) 2019-10-13T10:56:31.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:31.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.233308+0000) 2019-10-13T10:56:31.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:31.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.235452+0000) 2019-10-13T10:56:31.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:31.251+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.255183+0000) 2019-10-13T10:56:31.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:31.455+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.461523+0000) 2019-10-13T10:56:31.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:31.459+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.462337+0000) 2019-10-13T10:56:31.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:31.459+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.462926+0000) 2019-10-13T10:56:31.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:31.459+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.464817+0000) 2019-10-13T10:56:31.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:31.463+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.466511+0000) 2019-10-13T10:56:31.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:31.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:31.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:31.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.467335+0000) 2019-10-13T10:56:31.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:31.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.468209+0000) 2019-10-13T10:56:31.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:31.462+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.467987+0000) 2019-10-13T10:56:31.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:31.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:31.779181+0000) 2019-10-13T10:56:34.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:34.095+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:34.098578+0000) 2019-10-13T10:56:34.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:34.383+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:34.384751+0000) 2019-10-13T10:56:36.471 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:36.467+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.470338+0000) 2019-10-13T10:56:36.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:36.467+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.470883+0000) 2019-10-13T10:56:36.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:36.467+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.471578+0000) 2019-10-13T10:56:36.473 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:36.467+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.472787+0000) 2019-10-13T10:56:36.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:36.471+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.474546+0000) 2019-10-13T10:56:36.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:36.470+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.475167+0000) 2019-10-13T10:56:36.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:36.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.476543+0000) 2019-10-13T10:56:36.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:36.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:36.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:36.478+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:36.480965+0000) 2019-10-13T10:56:37.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:37.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:37.758137+0000) 2019-10-13T10:56:38.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:38.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:38.527916+0000) 2019-10-13T10:56:41.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:41.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.233510+0000) 2019-10-13T10:56:41.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:41.238+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.242038+0000) 2019-10-13T10:56:41.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:41.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.255354+0000) 2019-10-13T10:56:41.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:41.474+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.480806+0000) 2019-10-13T10:56:41.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:41.474+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.481217+0000) 2019-10-13T10:56:41.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:41.482+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.487042+0000) 2019-10-13T10:56:41.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:41.490+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.494420+0000) 2019-10-13T10:56:41.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:41.482+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.486325+0000) 2019-10-13T10:56:41.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:41.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127691 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:41.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:41.482+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.487992+0000) 2019-10-13T10:56:41.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:41.482+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.488513+0000) 2019-10-13T10:56:41.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:41.482+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.488850+0000) 2019-10-13T10:56:41.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:41.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:41.779342+0000) 2019-10-13T10:56:44.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:44.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:44.098783+0000) 2019-10-13T10:56:44.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:44.382+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:44.384953+0000) 2019-10-13T10:56:44.961 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:56:44.961 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:44.965 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:56:44.965 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:56:46.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:46.486+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.490838+0000) 2019-10-13T10:56:46.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:46.486+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.491920+0000) 2019-10-13T10:56:46.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:46.486+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.492233+0000) 2019-10-13T10:56:46.494 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:46.486+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.492895+0000) 2019-10-13T10:56:46.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:46.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.493281+0000) 2019-10-13T10:56:46.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:46.486+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.493720+0000) 2019-10-13T10:56:46.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:46.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.494045+0000) 2019-10-13T10:56:46.497 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:46.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:46.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:46.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:46.497955+0000) 2019-10-13T10:56:47.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:47.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:47.758332+0000) 2019-10-13T10:56:48.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:48.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:48.528104+0000) 2019-10-13T10:56:51.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:51.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.233707+0000) 2019-10-13T10:56:51.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:51.246+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.248546+0000) 2019-10-13T10:56:51.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:51.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.255535+0000) 2019-10-13T10:56:51.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:51.498+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.501918+0000) 2019-10-13T10:56:51.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:51.498+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.503971+0000) 2019-10-13T10:56:51.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:51.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:51.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:51.502+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.507206+0000) 2019-10-13T10:56:51.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:51.502+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.507141+0000) 2019-10-13T10:56:51.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:51.502+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.507471+0000) 2019-10-13T10:56:51.510 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:51.502+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.507883+0000) 2019-10-13T10:56:51.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:51.502+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.508356+0000) 2019-10-13T10:56:51.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:51.510+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.513420+0000) 2019-10-13T10:56:51.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:51.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:51.779550+0000) 2019-10-13T10:56:54.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:54.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:54.099007+0000) 2019-10-13T10:56:54.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:54.382+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:54.385113+0000) 2019-10-13T10:56:56.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:56:56.506+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.510799+0000) 2019-10-13T10:56:56.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:56:56.506+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.512001+0000) 2019-10-13T10:56:56.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:56.506+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.513590+0000) 2019-10-13T10:56:56.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:56.510+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.515709+0000) 2019-10-13T10:56:56.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:56:56.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127780 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:56:56.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:56:56.510+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.516986+0000) 2019-10-13T10:56:56.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:56:56.510+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.517264+0000) 2019-10-13T10:56:56.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:56:56.514+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.518062+0000) 2019-10-13T10:56:56.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:56:56.514+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:56.517761+0000) 2019-10-13T10:56:57.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:56:57.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:57.758508+0000) 2019-10-13T10:56:58.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:56:58.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:56:58.528302+0000) 2019-10-13T10:57:01.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:01.230+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.233963+0000) 2019-10-13T10:57:01.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:01.254+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.255062+0000) 2019-10-13T10:57:01.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:01.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.255694+0000) 2019-10-13T10:57:01.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:01.514+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.521489+0000) 2019-10-13T10:57:01.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:01.518+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.523583+0000) 2019-10-13T10:57:01.525 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:01.518+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.524372+0000) 2019-10-13T10:57:01.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:01.522+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.524417+0000) 2019-10-13T10:57:01.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:01.518+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.525512+0000) 2019-10-13T10:57:01.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:01.522+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.526007+0000) 2019-10-13T10:57:01.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:01.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127815 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:01.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:01.522+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.526610+0000) 2019-10-13T10:57:01.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:01.526+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.529990+0000) 2019-10-13T10:57:01.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:01.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:01.779758+0000) 2019-10-13T10:57:04.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:04.094+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:04.099226+0000) 2019-10-13T10:57:04.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:04.378+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:04.385308+0000) 2019-10-13T10:57:06.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:06.526+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.530763+0000) 2019-10-13T10:57:06.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:06.526+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.532056+0000) 2019-10-13T10:57:06.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:06.526+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.532314+0000) 2019-10-13T10:57:06.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:06.530+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.532635+0000) 2019-10-13T10:57:06.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:06.526+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.533497+0000) 2019-10-13T10:57:06.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:06.526+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.533626+0000) 2019-10-13T10:57:06.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:06.530+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.533945+0000) 2019-10-13T10:57:06.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:06.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:06.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:06.538+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:06.541045+0000) 2019-10-13T10:57:07.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:07.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:07.758667+0000) 2019-10-13T10:57:08.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:08.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:08.528525+0000) 2019-10-13T10:57:11.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:11.230+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.234117+0000) 2019-10-13T10:57:11.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:11.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.255909+0000) 2019-10-13T10:57:11.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:11.258+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.261537+0000) 2019-10-13T10:57:11.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:11.534+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.539759+0000) 2019-10-13T10:57:11.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:11.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.540060+0000) 2019-10-13T10:57:11.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:11.534+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.540576+0000) 2019-10-13T10:57:11.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:11.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.541442+0000) 2019-10-13T10:57:11.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:11.538+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.542207+0000) 2019-10-13T10:57:11.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:11.538+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.543176+0000) 2019-10-13T10:57:11.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:11.538+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.543549+0000) 2019-10-13T10:57:11.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:11.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:11.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:11.542+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.547620+0000) 2019-10-13T10:57:11.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:11.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:11.779958+0000) 2019-10-13T10:57:14.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:14.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:14.099430+0000) 2019-10-13T10:57:14.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:14.378+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:14.385505+0000) 2019-10-13T10:57:15.036 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:57:15.036 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:15.039 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:57:15.040 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:16.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:16.542+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.547676+0000) 2019-10-13T10:57:16.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:16.542+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.548742+0000) 2019-10-13T10:57:16.550 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:16.542+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.549550+0000) 2019-10-13T10:57:16.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:16.546+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.549997+0000) 2019-10-13T10:57:16.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:16.546+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.550596+0000) 2019-10-13T10:57:16.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:16.546+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.551245+0000) 2019-10-13T10:57:16.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:16.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.552026+0000) 2019-10-13T10:57:16.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:16.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:16.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:16.550+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:16.554406+0000) 2019-10-13T10:57:17.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:17.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:17.758867+0000) 2019-10-13T10:57:18.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:18.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:18.528725+0000) 2019-10-13T10:57:21.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:21.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.234316+0000) 2019-10-13T10:57:21.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:21.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.256107+0000) 2019-10-13T10:57:21.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:21.265+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.267946+0000) 2019-10-13T10:57:21.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:21.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.557907+0000) 2019-10-13T10:57:21.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:21.554+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.558241+0000) 2019-10-13T10:57:21.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:21.554+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.558851+0000) 2019-10-13T10:57:21.562 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:21.558+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.562043+0000) 2019-10-13T10:57:21.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:21.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:21.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:21.558+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.564830+0000) 2019-10-13T10:57:21.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:21.561+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.564952+0000) 2019-10-13T10:57:21.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:21.562+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.565784+0000) 2019-10-13T10:57:21.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:21.561+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.565899+0000) 2019-10-13T10:57:21.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:21.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:21.780130+0000) 2019-10-13T10:57:24.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:24.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:24.099630+0000) 2019-10-13T10:57:24.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:24.382+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:24.385687+0000) 2019-10-13T10:57:26.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:26.562+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.567491+0000) 2019-10-13T10:57:26.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:26.562+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.567855+0000) 2019-10-13T10:57:26.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:26.562+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.568934+0000) 2019-10-13T10:57:26.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:26.562+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.569208+0000) 2019-10-13T10:57:26.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:26.562+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.569631+0000) 2019-10-13T10:57:26.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:26.566+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.569946+0000) 2019-10-13T10:57:26.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:26.565+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.571200+0000) 2019-10-13T10:57:26.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:26.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:26.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:26.573+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:26.577041+0000) 2019-10-13T10:57:27.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:27.754+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:27.759091+0000) 2019-10-13T10:57:28.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:28.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:28.528949+0000) 2019-10-13T10:57:31.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:31.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.234523+0000) 2019-10-13T10:57:31.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:31.250+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.256303+0000) 2019-10-13T10:57:31.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:31.273+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.274621+0000) 2019-10-13T10:57:31.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:31.574+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.577791+0000) 2019-10-13T10:57:31.579 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:31.574+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.578812+0000) 2019-10-13T10:57:31.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:31.574+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.579685+0000) 2019-10-13T10:57:31.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:31.574+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.579914+0000) 2019-10-13T10:57:31.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:31.577+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.580374+0000) 2019-10-13T10:57:31.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:31.574+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.581117+0000) 2019-10-13T10:57:31.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:31.574+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.581601+0000) 2019-10-13T10:57:31.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:31.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:31.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:31.581+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.586311+0000) 2019-10-13T10:57:31.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:31.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:31.780322+0000) 2019-10-13T10:57:34.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:34.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:34.099829+0000) 2019-10-13T10:57:34.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:34.381+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:34.385890+0000) 2019-10-13T10:57:36.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:36.581+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.586286+0000) 2019-10-13T10:57:36.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:36.581+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.586961+0000) 2019-10-13T10:57:36.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:36.581+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.588185+0000) 2019-10-13T10:57:36.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:36.581+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.588728+0000) 2019-10-13T10:57:36.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:36.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.589363+0000) 2019-10-13T10:57:36.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:36.585+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.589845+0000) 2019-10-13T10:57:36.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:36.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.590704+0000) 2019-10-13T10:57:36.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:36.589+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:36.594233+0000) 2019-10-13T10:57:36.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:36.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:37.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:37.753+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:37.759285+0000) 2019-10-13T10:57:38.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:38.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:38.529157+0000) 2019-10-13T10:57:41.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:41.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.234717+0000) 2019-10-13T10:57:41.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:41.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.256527+0000) 2019-10-13T10:57:41.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:41.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.281195+0000) 2019-10-13T10:57:41.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:41.593+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.596952+0000) 2019-10-13T10:57:41.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:41.593+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.598641+0000) 2019-10-13T10:57:41.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:41.593+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.600050+0000) 2019-10-13T10:57:41.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:41.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.600567+0000) 2019-10-13T10:57:41.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:41.597+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.603013+0000) 2019-10-13T10:57:41.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:41.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:41.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:41.597+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.603768+0000) 2019-10-13T10:57:41.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:41.597+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.604723+0000) 2019-10-13T10:57:41.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:41.605+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.609571+0000) 2019-10-13T10:57:41.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:41.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:41.780551+0000) 2019-10-13T10:57:44.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:44.093+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:44.100027+0000) 2019-10-13T10:57:44.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:44.385+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:44.386115+0000) 2019-10-13T10:57:45.099 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:57:45.099 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:45.103 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:57:45.103 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:57:46.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:46.601+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.606255+0000) 2019-10-13T10:57:46.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:46.601+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.607398+0000) 2019-10-13T10:57:46.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:46.605+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.608739+0000) 2019-10-13T10:57:46.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:46.601+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.609401+0000) 2019-10-13T10:57:46.611 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:46.605+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.610487+0000) 2019-10-13T10:57:46.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:46.605+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.610757+0000) 2019-10-13T10:57:46.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:46.605+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.611312+0000) 2019-10-13T10:57:46.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:46.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:46.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:46.613+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:46.616305+0000) 2019-10-13T10:57:47.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:47.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:47.759537+0000) 2019-10-13T10:57:48.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:48.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:48.529354+0000) 2019-10-13T10:57:51.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:51.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.234924+0000) 2019-10-13T10:57:51.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:51.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.256729+0000) 2019-10-13T10:57:51.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:51.285+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.287701+0000) 2019-10-13T10:57:51.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:51.613+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.617374+0000) 2019-10-13T10:57:51.619 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:51.613+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.617463+0000) 2019-10-13T10:57:51.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:51.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.618070+0000) 2019-10-13T10:57:51.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:51.617+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.618307+0000) 2019-10-13T10:57:51.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:51.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.618564+0000) 2019-10-13T10:57:51.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:51.617+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.619341+0000) 2019-10-13T10:57:51.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:51.617+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.619799+0000) 2019-10-13T10:57:51.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:51.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:51.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:51.621+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.625890+0000) 2019-10-13T10:57:51.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:51.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:51.780737+0000) 2019-10-13T10:57:54.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:54.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:54.100224+0000) 2019-10-13T10:57:54.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:54.385+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:54.386335+0000) 2019-10-13T10:57:56.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:57:56.625+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.625977+0000) 2019-10-13T10:57:56.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:57:56.625+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.626989+0000) 2019-10-13T10:57:56.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:57:56.625+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.628430+0000) 2019-10-13T10:57:56.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:57:56.625+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.629158+0000) 2019-10-13T10:57:56.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:56.625+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.629117+0000) 2019-10-13T10:57:56.631 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:57:56.629+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.629787+0000) 2019-10-13T10:57:56.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:56.629+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.630251+0000) 2019-10-13T10:57:56.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:57:56.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:57:56.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:57:56.633+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:56.636982+0000) 2019-10-13T10:57:57.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:57:57.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:57.759771+0000) 2019-10-13T10:57:58.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:57:58.525+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:57:58.529547+0000) 2019-10-13T10:58:01.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:01.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.235126+0000) 2019-10-13T10:58:01.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:01.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.256923+0000) 2019-10-13T10:58:01.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:01.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.294253+0000) 2019-10-13T10:58:01.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:01.633+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.636050+0000) 2019-10-13T10:58:01.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:01.633+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.636519+0000) 2019-10-13T10:58:01.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:01.633+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.636895+0000) 2019-10-13T10:58:01.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:01.637+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.638563+0000) 2019-10-13T10:58:01.639 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:01.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.638893+0000) 2019-10-13T10:58:01.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:01.633+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.639343+0000) 2019-10-13T10:58:01.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:01.637+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.639829+0000) 2019-10-13T10:58:01.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:01.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:01.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:01.641+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.647436+0000) 2019-10-13T10:58:01.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:01.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:01.780926+0000) 2019-10-13T10:58:04.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:04.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:04.100447+0000) 2019-10-13T10:58:04.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:04.385+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:04.386564+0000) 2019-10-13T10:58:06.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:06.641+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.645625+0000) 2019-10-13T10:58:06.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:06.645+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.645924+0000) 2019-10-13T10:58:06.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:06.645+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.646235+0000) 2019-10-13T10:58:06.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:06.641+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.646809+0000) 2019-10-13T10:58:06.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:06.645+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.647436+0000) 2019-10-13T10:58:06.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:06.645+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.648094+0000) 2019-10-13T10:58:06.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:06.645+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.648682+0000) 2019-10-13T10:58:06.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:06.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:06.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:06.653+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:06.657241+0000) 2019-10-13T10:58:07.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:07.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:07.759990+0000) 2019-10-13T10:58:08.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:08.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:08.529776+0000) 2019-10-13T10:58:11.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:11.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.235358+0000) 2019-10-13T10:58:11.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:11.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.257115+0000) 2019-10-13T10:58:11.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:11.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.300840+0000) 2019-10-13T10:58:11.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:11.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.656380+0000) 2019-10-13T10:58:11.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:11.653+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.657185+0000) 2019-10-13T10:58:11.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:11.657+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.658114+0000) 2019-10-13T10:58:11.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:11.657+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.658738+0000) 2019-10-13T10:58:11.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:11.657+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.660505+0000) 2019-10-13T10:58:11.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:11.657+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.660901+0000) 2019-10-13T10:58:11.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:11.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128247 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:11.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:11.661+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.662010+0000) 2019-10-13T10:58:11.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:11.657+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.663423+0000) 2019-10-13T10:58:11.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:11.777+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:11.781147+0000) 2019-10-13T10:58:14.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:14.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:14.100643+0000) 2019-10-13T10:58:14.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:14.385+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:14.386761+0000) 2019-10-13T10:58:15.230 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:58:15.230 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:15.233 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:58:15.234 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:16.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:16.661+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.665328+0000) 2019-10-13T10:58:16.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:16.665+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.665836+0000) 2019-10-13T10:58:16.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:16.665+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.666102+0000) 2019-10-13T10:58:16.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:16.665+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.667078+0000) 2019-10-13T10:58:16.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:16.665+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.666359+0000) 2019-10-13T10:58:16.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:16.665+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.667486+0000) 2019-10-13T10:58:16.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:16.661+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.667658+0000) 2019-10-13T10:58:16.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:16.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128277 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:16.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:16.669+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:16.672423+0000) 2019-10-13T10:58:17.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:17.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:17.760186+0000) 2019-10-13T10:58:18.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:18.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:18.529973+0000) 2019-10-13T10:58:21.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:21.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.235553+0000) 2019-10-13T10:58:21.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:21.253+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.257309+0000) 2019-10-13T10:58:21.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:21.301+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.307159+0000) 2019-10-13T10:58:21.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:21.673+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.674698+0000) 2019-10-13T10:58:21.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:21.673+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.675270+0000) 2019-10-13T10:58:21.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:21.673+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.676048+0000) 2019-10-13T10:58:21.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:21.673+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.676771+0000) 2019-10-13T10:58:21.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:21.673+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.677487+0000) 2019-10-13T10:58:21.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:21.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.677726+0000) 2019-10-13T10:58:21.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:21.673+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.677980+0000) 2019-10-13T10:58:21.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:21.677+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:21.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:21.677+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.682371+0000) 2019-10-13T10:58:21.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:21.781+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:21.781323+0000) 2019-10-13T10:58:24.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:24.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:24.100839+0000) 2019-10-13T10:58:24.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:24.385+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:24.386995+0000) 2019-10-13T10:58:26.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:26.681+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.684353+0000) 2019-10-13T10:58:26.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:26.680+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.684972+0000) 2019-10-13T10:58:26.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:26.685+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.685972+0000) 2019-10-13T10:58:26.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:26.685+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.686367+0000) 2019-10-13T10:58:26.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:26.685+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.686592+0000) 2019-10-13T10:58:26.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:26.685+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.687050+0000) 2019-10-13T10:58:26.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:26.685+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.687923+0000) 2019-10-13T10:58:26.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:26.689+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:26.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:26.684+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:26.691080+0000) 2019-10-13T10:58:27.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:27.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:27.760399+0000) 2019-10-13T10:58:28.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:28.529+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:28.530194+0000) 2019-10-13T10:58:31.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:31.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.235768+0000) 2019-10-13T10:58:31.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:31.252+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.257534+0000) 2019-10-13T10:58:31.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:31.312+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.313785+0000) 2019-10-13T10:58:31.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:31.692+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.693793+0000) 2019-10-13T10:58:31.696 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:31.692+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.695507+0000) 2019-10-13T10:58:31.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:31.696+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.697727+0000) 2019-10-13T10:58:31.700 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:31.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128371 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:31.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:31.696+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.700369+0000) 2019-10-13T10:58:31.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:31.696+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.700953+0000) 2019-10-13T10:58:31.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:31.696+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.701459+0000) 2019-10-13T10:58:31.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:31.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.701643+0000) 2019-10-13T10:58:31.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:31.704+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.708991+0000) 2019-10-13T10:58:31.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:31.780+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:31.781507+0000) 2019-10-13T10:58:34.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:34.096+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:34.101047+0000) 2019-10-13T10:58:34.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:34.384+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:34.387230+0000) 2019-10-13T10:58:36.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:36.700+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.703040+0000) 2019-10-13T10:58:36.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:36.700+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.704306+0000) 2019-10-13T10:58:36.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:36.700+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.704726+0000) 2019-10-13T10:58:36.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:36.700+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.705285+0000) 2019-10-13T10:58:36.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:36.704+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.707754+0000) 2019-10-13T10:58:36.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:36.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.710331+0000) 2019-10-13T10:58:36.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:36.708+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:36.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:36.708+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.711685+0000) 2019-10-13T10:58:36.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:36.712+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:36.717583+0000) 2019-10-13T10:58:37.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:37.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:37.760631+0000) 2019-10-13T10:58:38.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:38.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:38.530431+0000) 2019-10-13T10:58:41.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:41.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.235968+0000) 2019-10-13T10:58:41.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:41.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.257731+0000) 2019-10-13T10:58:41.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:41.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.320268+0000) 2019-10-13T10:58:41.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:41.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.714816+0000) 2019-10-13T10:58:41.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:41.712+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.715685+0000) 2019-10-13T10:58:41.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:41.712+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.716507+0000) 2019-10-13T10:58:41.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:41.712+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.716835+0000) 2019-10-13T10:58:41.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:41.712+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.717065+0000) 2019-10-13T10:58:41.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:41.712+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.717501+0000) 2019-10-13T10:58:41.718 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:41.712+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.717855+0000) 2019-10-13T10:58:41.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:41.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:41.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:41.720+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.725546+0000) 2019-10-13T10:58:41.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:41.780+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:41.781684+0000) 2019-10-13T10:58:44.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:44.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:44.101279+0000) 2019-10-13T10:58:44.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:44.384+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:44.387435+0000) 2019-10-13T10:58:45.325 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:58:45.325 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:45.329 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:58:45.329 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:58:46.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:46.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.723543+0000) 2019-10-13T10:58:46.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:46.720+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.724898+0000) 2019-10-13T10:58:46.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:46.720+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.725351+0000) 2019-10-13T10:58:46.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:46.720+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.725618+0000) 2019-10-13T10:58:46.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:46.720+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.725647+0000) 2019-10-13T10:58:46.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:46.724+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.727333+0000) 2019-10-13T10:58:46.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:46.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.727869+0000) 2019-10-13T10:58:46.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:46.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:46.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:46.728+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:46.732389+0000) 2019-10-13T10:58:47.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:47.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:47.760865+0000) 2019-10-13T10:58:48.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:48.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:48.530668+0000) 2019-10-13T10:58:51.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:51.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.236167+0000) 2019-10-13T10:58:51.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:51.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.257968+0000) 2019-10-13T10:58:51.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:51.320+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.326697+0000) 2019-10-13T10:58:51.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:51.728+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.733046+0000) 2019-10-13T10:58:51.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:51.728+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.734353+0000) 2019-10-13T10:58:51.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:51.732+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.734870+0000) 2019-10-13T10:58:51.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:51.732+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.735845+0000) 2019-10-13T10:58:51.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:51.732+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.735714+0000) 2019-10-13T10:58:51.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:51.732+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.735450+0000) 2019-10-13T10:58:51.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:51.732+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.736606+0000) 2019-10-13T10:58:51.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:51.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:51.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:51.736+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.740207+0000) 2019-10-13T10:58:51.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:51.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:51.781847+0000) 2019-10-13T10:58:54.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:54.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:54.101535+0000) 2019-10-13T10:58:54.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:54.384+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:54.387632+0000) 2019-10-13T10:58:56.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:56.740+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.742429+0000) 2019-10-13T10:58:56.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:58:56.736+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.742561+0000) 2019-10-13T10:58:56.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:58:56.740+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.745283+0000) 2019-10-13T10:58:56.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:58:56.740+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.745491+0000) 2019-10-13T10:58:56.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:58:56.744+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.746603+0000) 2019-10-13T10:58:56.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:58:56.744+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.748311+0000) 2019-10-13T10:58:56.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:58:56.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:58:56.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:56.748+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.750217+0000) 2019-10-13T10:58:56.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:58:56.744+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:56.751864+0000) 2019-10-13T10:58:57.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:58:57.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:57.761061+0000) 2019-10-13T10:58:58.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:58:58.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:58:58.530895+0000) 2019-10-13T10:59:01.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:01.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.236367+0000) 2019-10-13T10:59:01.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:01.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.258166+0000) 2019-10-13T10:59:01.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:01.332+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.334067+0000) 2019-10-13T10:59:01.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:01.748+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.753075+0000) 2019-10-13T10:59:01.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:01.752+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.754087+0000) 2019-10-13T10:59:01.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:01.752+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.754741+0000) 2019-10-13T10:59:01.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:01.752+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.755470+0000) 2019-10-13T10:59:01.757 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:01.752+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.755823+0000) 2019-10-13T10:59:01.757 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:01.752+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.756568+0000) 2019-10-13T10:59:01.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:01.752+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.756892+0000) 2019-10-13T10:59:01.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:01.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:01.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:01.756+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.760119+0000) 2019-10-13T10:59:01.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:01.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:01.782026+0000) 2019-10-13T10:59:04.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:04.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:04.101733+0000) 2019-10-13T10:59:04.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:04.384+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:04.387856+0000) 2019-10-13T10:59:06.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:06.760+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.763232+0000) 2019-10-13T10:59:06.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:06.760+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.765098+0000) 2019-10-13T10:59:06.773 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:06.760+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.765510+0000) 2019-10-13T10:59:06.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:06.764+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.765934+0000) 2019-10-13T10:59:06.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:06.764+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.766294+0000) 2019-10-13T10:59:06.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:06.764+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.767114+0000) 2019-10-13T10:59:06.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:06.760+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.767644+0000) 2019-10-13T10:59:06.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:06.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:06.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:06.768+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:06.773496+0000) 2019-10-13T10:59:07.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:07.756+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:07.761259+0000) 2019-10-13T10:59:08.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:08.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:08.531082+0000) 2019-10-13T10:59:11.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:11.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.236597+0000) 2019-10-13T10:59:11.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:11.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.258361+0000) 2019-10-13T10:59:11.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:11.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.340572+0000) 2019-10-13T10:59:11.774 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:11.768+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.773316+0000) 2019-10-13T10:59:11.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:11.772+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.774122+0000) 2019-10-13T10:59:11.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:11.772+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.774444+0000) 2019-10-13T10:59:11.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:11.772+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.775660+0000) 2019-10-13T10:59:11.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:11.772+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.775864+0000) 2019-10-13T10:59:11.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:11.772+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.776280+0000) 2019-10-13T10:59:11.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:11.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:11.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:11.772+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.779537+0000) 2019-10-13T10:59:11.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:11.776+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.782162+0000) 2019-10-13T10:59:11.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:11.780+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:11.784569+0000) 2019-10-13T10:59:14.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:14.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:14.102026+0000) 2019-10-13T10:59:14.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:14.384+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:14.388063+0000) 2019-10-13T10:59:15.398 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:59:15.398 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:15.402 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:59:15.402 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:16.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:16.780+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.783904+0000) 2019-10-13T10:59:16.785 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:16.780+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.783589+0000) 2019-10-13T10:59:16.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:16.780+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.784321+0000) 2019-10-13T10:59:16.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:16.780+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.784898+0000) 2019-10-13T10:59:16.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:16.784+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.787475+0000) 2019-10-13T10:59:16.788 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:16.784+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.787521+0000) 2019-10-13T10:59:16.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:16.784+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.789583+0000) 2019-10-13T10:59:16.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:16.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:16.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:16.788+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:16.795397+0000) 2019-10-13T10:59:17.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:17.760+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:17.761472+0000) 2019-10-13T10:59:18.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:18.528+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:18.531316+0000) 2019-10-13T10:59:21.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:21.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.236836+0000) 2019-10-13T10:59:21.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:21.256+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.258555+0000) 2019-10-13T10:59:21.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:21.344+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.347055+0000) 2019-10-13T10:59:21.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:21.780+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.782357+0000) 2019-10-13T10:59:21.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:21.788+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.793182+0000) 2019-10-13T10:59:21.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:21.792+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.794134+0000) 2019-10-13T10:59:21.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:21.792+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.794321+0000) 2019-10-13T10:59:21.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:21.792+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.794670+0000) 2019-10-13T10:59:21.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:21.788+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.795246+0000) 2019-10-13T10:59:21.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:21.792+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.795830+0000) 2019-10-13T10:59:21.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:21.792+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.796637+0000) 2019-10-13T10:59:21.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:21.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:21.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:21.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:21.802088+0000) 2019-10-13T10:59:24.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:24.100+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:24.102223+0000) 2019-10-13T10:59:24.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:24.388+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:24.388280+0000) 2019-10-13T10:59:26.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:26.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.802731+0000) 2019-10-13T10:59:26.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:26.799+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.803102+0000) 2019-10-13T10:59:26.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:26.799+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.803921+0000) 2019-10-13T10:59:26.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:26.799+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.804236+0000) 2019-10-13T10:59:26.805 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:26.799+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.805108+0000) 2019-10-13T10:59:26.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:26.803+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.806018+0000) 2019-10-13T10:59:26.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:26.804+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.806303+0000) 2019-10-13T10:59:26.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:26.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128709 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:26.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:26.812+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:26.812597+0000) 2019-10-13T10:59:27.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:27.755+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:27.761648+0000) 2019-10-13T10:59:28.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:28.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:28.531511+0000) 2019-10-13T10:59:31.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:31.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.237050+0000) 2019-10-13T10:59:31.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:31.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.258760+0000) 2019-10-13T10:59:31.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:31.352+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.353733+0000) 2019-10-13T10:59:31.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:31.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.782580+0000) 2019-10-13T10:59:31.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:31.807+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.812970+0000) 2019-10-13T10:59:31.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:31.811+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.813799+0000) 2019-10-13T10:59:31.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:31.811+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.814225+0000) 2019-10-13T10:59:31.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:31.811+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.814778+0000) 2019-10-13T10:59:31.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:31.811+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.815804+0000) 2019-10-13T10:59:31.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:31.811+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.816327+0000) 2019-10-13T10:59:31.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:31.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.816833+0000) 2019-10-13T10:59:31.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:31.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:31.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:31.819+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:31.820811+0000) 2019-10-13T10:59:34.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:34.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:34.102421+0000) 2019-10-13T10:59:34.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:34.387+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:34.388500+0000) 2019-10-13T10:59:36.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:36.819+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.822756+0000) 2019-10-13T10:59:36.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:36.819+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.823197+0000) 2019-10-13T10:59:36.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:36.819+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.823590+0000) 2019-10-13T10:59:36.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:36.819+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.823926+0000) 2019-10-13T10:59:36.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:36.819+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.824695+0000) 2019-10-13T10:59:36.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:36.819+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.825449+0000) 2019-10-13T10:59:36.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:36.823+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.825625+0000) 2019-10-13T10:59:36.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:36.827+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:36.828103+0000) 2019-10-13T10:59:36.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:36.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:37.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:37.759+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:37.761832+0000) 2019-10-13T10:59:38.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:38.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:38.531712+0000) 2019-10-13T10:59:41.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:41.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.237255+0000) 2019-10-13T10:59:41.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:41.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.258963+0000) 2019-10-13T10:59:41.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:41.363+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.364948+0000) 2019-10-13T10:59:41.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:41.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.782795+0000) 2019-10-13T10:59:41.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:41.827+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.832192+0000) 2019-10-13T10:59:41.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:41.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.832698+0000) 2019-10-13T10:59:41.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:41.827+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.833384+0000) 2019-10-13T10:59:41.834 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:41.827+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.833120+0000) 2019-10-13T10:59:41.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:41.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.834081+0000) 2019-10-13T10:59:41.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:41.831+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.835385+0000) 2019-10-13T10:59:41.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:41.831+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.836468+0000) 2019-10-13T10:59:41.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:41.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:41.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:41.839+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:41.840599+0000) 2019-10-13T10:59:44.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:44.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:44.102639+0000) 2019-10-13T10:59:44.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:44.387+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:44.388697+0000) 2019-10-13T10:59:45.511 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T10:59:45.511 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:45.515 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T10:59:45.515 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T10:59:46.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:46.839+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.842676+0000) 2019-10-13T10:59:46.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:46.839+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.844480+0000) 2019-10-13T10:59:46.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:46.843+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.844535+0000) 2019-10-13T10:59:46.846 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:46.839+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.845388+0000) 2019-10-13T10:59:46.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:46.843+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.846189+0000) 2019-10-13T10:59:46.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:46.843+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.845955+0000) 2019-10-13T10:59:46.847 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:46.843+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.846744+0000) 2019-10-13T10:59:46.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:46.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:46.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:46.851+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:46.852827+0000) 2019-10-13T10:59:47.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:47.759+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:47.761977+0000) 2019-10-13T10:59:48.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:48.527+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:48.531904+0000) 2019-10-13T10:59:51.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:51.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.237470+0000) 2019-10-13T10:59:51.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:51.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.259157+0000) 2019-10-13T10:59:51.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:51.367+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.371471+0000) 2019-10-13T10:59:51.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:51.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.782990+0000) 2019-10-13T10:59:51.853 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:51.847+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.852155+0000) 2019-10-13T10:59:51.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:51.851+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.852132+0000) 2019-10-13T10:59:51.854 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:51.847+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.852833+0000) 2019-10-13T10:59:51.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:51.851+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.854682+0000) 2019-10-13T10:59:51.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:51.851+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.855094+0000) 2019-10-13T10:59:51.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:51.851+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.855388+0000) 2019-10-13T10:59:51.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:51.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.855797+0000) 2019-10-13T10:59:51.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:51.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:51.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:51.859+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:51.863327+0000) 2019-10-13T10:59:54.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:54.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:54.102903+0000) 2019-10-13T10:59:54.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:54.383+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:54.388895+0000) 2019-10-13T10:59:56.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T10:59:56.859+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.861860+0000) 2019-10-13T10:59:56.863 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T10:59:56.859+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.862366+0000) 2019-10-13T10:59:56.864 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:56.859+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.863551+0000) 2019-10-13T10:59:56.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T10:59:56.859+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.863481+0000) 2019-10-13T10:59:56.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T10:59:56.859+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.865455+0000) 2019-10-13T10:59:56.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:56.859+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.865523+0000) 2019-10-13T10:59:56.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T10:59:56.863+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.865966+0000) 2019-10-13T10:59:56.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T10:59:56.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T10:59:56.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T10:59:56.871+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:56.874818+0000) 2019-10-13T10:59:57.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T10:59:57.759+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:57.762121+0000) 2019-10-13T10:59:58.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T10:59:58.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T09:59:58.532092+0000) 2019-10-13T11:00:01.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:01.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.237666+0000) 2019-10-13T11:00:01.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:01.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.259361+0000) 2019-10-13T11:00:01.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:01.375+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.378060+0000) 2019-10-13T11:00:01.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:01.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.783178+0000) 2019-10-13T11:00:01.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:01.867+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.871579+0000) 2019-10-13T11:00:01.873 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:01.867+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.871880+0000) 2019-10-13T11:00:01.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:01.871+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.872407+0000) 2019-10-13T11:00:01.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:01.867+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.873497+0000) 2019-10-13T11:00:01.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:01.871+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.875208+0000) 2019-10-13T11:00:01.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:01.871+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.875438+0000) 2019-10-13T11:00:01.876 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:01.871+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.876033+0000) 2019-10-13T11:00:01.878 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:01.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:01.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:01.879+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:01.883639+0000) 2019-10-13T11:00:04.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:04.099+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:04.103114+0000) 2019-10-13T11:00:04.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:04.387+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:04.389114+0000) 2019-10-13T11:00:06.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:06.875+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.881210+0000) 2019-10-13T11:00:06.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:06.875+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.881609+0000) 2019-10-13T11:00:06.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:06.879+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.881860+0000) 2019-10-13T11:00:06.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:06.879+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.882385+0000) 2019-10-13T11:00:06.883 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:06.879+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.882781+0000) 2019-10-13T11:00:06.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:06.879+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.883715+0000) 2019-10-13T11:00:06.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:06.879+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.885252+0000) 2019-10-13T11:00:06.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:06.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:06.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:06.887+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:06.891464+0000) 2019-10-13T11:00:07.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:07.759+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:07.762313+0000) 2019-10-13T11:00:08.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:08.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:08.532283+0000) 2019-10-13T11:00:11.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:11.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.237923+0000) 2019-10-13T11:00:11.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:11.255+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.259584+0000) 2019-10-13T11:00:11.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:11.383+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.384523+0000) 2019-10-13T11:00:11.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:11.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.783378+0000) 2019-10-13T11:00:11.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:11.887+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.890154+0000) 2019-10-13T11:00:11.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:11.887+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.890967+0000) 2019-10-13T11:00:11.892 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:11.887+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.891608+0000) 2019-10-13T11:00:11.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:11.887+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.891764+0000) 2019-10-13T11:00:11.893 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:11.887+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.891801+0000) 2019-10-13T11:00:11.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:11.887+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.892366+0000) 2019-10-13T11:00:11.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:11.887+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.892733+0000) 2019-10-13T11:00:11.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:11.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:11.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:11.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:11.899682+0000) 2019-10-13T11:00:14.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:14.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:14.103318+0000) 2019-10-13T11:00:14.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:14.387+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:14.389317+0000) 2019-10-13T11:00:15.642 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:00:15.642 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:15.645 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:00:15.645 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:16.900 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:16.895+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.899009+0000) 2019-10-13T11:00:16.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:16.899+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.901770+0000) 2019-10-13T11:00:16.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:16.899+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.901763+0000) 2019-10-13T11:00:16.904 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:16.899+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.904040+0000) 2019-10-13T11:00:16.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:16.899+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.904549+0000) 2019-10-13T11:00:16.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:16.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:16.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:16.899+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.905156+0000) 2019-10-13T11:00:16.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:16.903+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.906458+0000) 2019-10-13T11:00:16.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:16.907+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:16.909807+0000) 2019-10-13T11:00:17.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:17.759+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:17.762517+0000) 2019-10-13T11:00:18.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:18.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:18.532471+0000) 2019-10-13T11:00:21.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:21.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.238094+0000) 2019-10-13T11:00:21.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:21.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.259842+0000) 2019-10-13T11:00:21.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:21.387+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.390857+0000) 2019-10-13T11:00:21.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:21.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.783600+0000) 2019-10-13T11:00:21.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:21.903+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.908157+0000) 2019-10-13T11:00:21.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:21.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.908296+0000) 2019-10-13T11:00:21.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:21.903+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.908841+0000) 2019-10-13T11:00:21.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:21.903+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.909644+0000) 2019-10-13T11:00:21.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:21.907+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.910144+0000) 2019-10-13T11:00:21.911 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:21.907+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.910435+0000) 2019-10-13T11:00:21.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:21.907+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.912434+0000) 2019-10-13T11:00:21.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:21.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:21.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:21.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:21.915086+0000) 2019-10-13T11:00:24.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:24.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:24.103557+0000) 2019-10-13T11:00:24.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:24.382+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:24.389537+0000) 2019-10-13T11:00:26.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:26.910+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.917554+0000) 2019-10-13T11:00:26.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:26.914+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.918374+0000) 2019-10-13T11:00:26.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:26.914+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.918769+0000) 2019-10-13T11:00:26.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:26.914+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.919066+0000) 2019-10-13T11:00:26.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:26.919+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.920832+0000) 2019-10-13T11:00:26.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:26.914+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.921055+0000) 2019-10-13T11:00:26.923 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:26.918+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.922284+0000) 2019-10-13T11:00:26.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:26.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:26.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:26.927+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:26.928657+0000) 2019-10-13T11:00:27.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:27.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:27.762697+0000) 2019-10-13T11:00:28.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:28.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:28.532666+0000) 2019-10-13T11:00:31.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:31.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.238315+0000) 2019-10-13T11:00:31.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:31.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.260040+0000) 2019-10-13T11:00:31.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:31.395+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.397209+0000) 2019-10-13T11:00:31.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:31.779+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.783787+0000) 2019-10-13T11:00:31.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:31.922+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.927708+0000) 2019-10-13T11:00:31.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:31.922+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.928030+0000) 2019-10-13T11:00:31.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:31.922+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.929070+0000) 2019-10-13T11:00:31.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:31.927+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.928970+0000) 2019-10-13T11:00:31.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:31.922+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.929545+0000) 2019-10-13T11:00:31.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:31.926+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.929880+0000) 2019-10-13T11:00:31.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:31.926+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.930641+0000) 2019-10-13T11:00:31.934 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:31.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:31.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:31.931+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:31.934345+0000) 2019-10-13T11:00:34.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:34.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:34.103756+0000) 2019-10-13T11:00:34.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:34.386+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:34.389772+0000) 2019-10-13T11:00:36.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:36.930+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.936373+0000) 2019-10-13T11:00:36.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:36.930+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.937017+0000) 2019-10-13T11:00:36.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:36.935+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.938517+0000) 2019-10-13T11:00:36.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:36.934+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.939162+0000) 2019-10-13T11:00:36.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:36.934+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.939445+0000) 2019-10-13T11:00:36.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:36.934+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.940862+0000) 2019-10-13T11:00:36.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:36.934+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.941617+0000) 2019-10-13T11:00:36.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:36.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:36.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:36.943+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:36.947344+0000) 2019-10-13T11:00:37.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:37.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:37.762879+0000) 2019-10-13T11:00:38.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:38.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:38.532827+0000) 2019-10-13T11:00:41.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:41.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.238527+0000) 2019-10-13T11:00:41.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:41.254+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.260239+0000) 2019-10-13T11:00:41.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:41.398+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.403748+0000) 2019-10-13T11:00:41.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:41.778+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.784008+0000) 2019-10-13T11:00:41.947 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:41.942+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.946467+0000) 2019-10-13T11:00:41.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:41.942+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.946982+0000) 2019-10-13T11:00:41.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:41.942+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.947832+0000) 2019-10-13T11:00:41.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:41.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.948219+0000) 2019-10-13T11:00:41.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:41.942+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.949031+0000) 2019-10-13T11:00:41.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:41.946+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.949631+0000) 2019-10-13T11:00:41.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:41.946+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.950244+0000) 2019-10-13T11:00:41.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:41.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:41.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:41.954+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:41.956280+0000) 2019-10-13T11:00:44.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:44.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:44.103958+0000) 2019-10-13T11:00:44.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:44.386+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:44.389972+0000) 2019-10-13T11:00:45.760 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:00:45.760 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:45.764 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:00:45.764 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:00:46.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:46.950+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.955971+0000) 2019-10-13T11:00:46.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:46.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.956444+0000) 2019-10-13T11:00:46.959 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:46.954+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.958905+0000) 2019-10-13T11:00:46.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:46.954+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.960686+0000) 2019-10-13T11:00:46.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:46.958+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.960855+0000) 2019-10-13T11:00:46.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:46.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:46.963 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:46.958+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.962428+0000) 2019-10-13T11:00:46.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:46.958+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.963145+0000) 2019-10-13T11:00:46.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:46.958+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:46.963376+0000) 2019-10-13T11:00:47.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:47.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:47.763034+0000) 2019-10-13T11:00:48.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:48.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:48.533015+0000) 2019-10-13T11:00:51.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:51.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.238744+0000) 2019-10-13T11:00:51.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:51.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.260491+0000) 2019-10-13T11:00:51.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:51.406+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.410187+0000) 2019-10-13T11:00:51.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:51.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.784251+0000) 2019-10-13T11:00:51.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:51.958+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.964609+0000) 2019-10-13T11:00:51.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:51.958+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.965124+0000) 2019-10-13T11:00:51.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:51.962+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.967282+0000) 2019-10-13T11:00:51.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:51.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.968902+0000) 2019-10-13T11:00:51.971 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:51.966+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.970482+0000) 2019-10-13T11:00:51.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:51.966+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.971339+0000) 2019-10-13T11:00:51.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:51.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:51.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:51.966+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.972696+0000) 2019-10-13T11:00:51.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:51.978+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:51.981029+0000) 2019-10-13T11:00:54.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:54.098+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:54.104186+0000) 2019-10-13T11:00:54.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:54.386+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:54.390182+0000) 2019-10-13T11:00:56.977 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:00:56.970+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.976979+0000) 2019-10-13T11:00:56.978 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:00:56.970+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.977423+0000) 2019-10-13T11:00:56.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:56.974+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.978058+0000) 2019-10-13T11:00:56.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:00:56.974+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.978341+0000) 2019-10-13T11:00:56.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:56.974+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.979416+0000) 2019-10-13T11:00:56.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:00:56.974+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.979849+0000) 2019-10-13T11:00:56.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:00:56.974+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.980545+0000) 2019-10-13T11:00:56.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:00:56.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129268 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:00:56.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:00:56.982+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:56.988107+0000) 2019-10-13T11:00:57.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:00:57.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:57.763209+0000) 2019-10-13T11:00:58.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:00:58.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:00:58.533229+0000) 2019-10-13T11:01:01.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:01.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.238943+0000) 2019-10-13T11:01:01.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:01.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.260662+0000) 2019-10-13T11:01:01.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:01.414+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.416689+0000) 2019-10-13T11:01:01.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:01.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.784485+0000) 2019-10-13T11:01:01.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:01.982+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.986712+0000) 2019-10-13T11:01:01.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:01.982+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.987366+0000) 2019-10-13T11:01:01.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:01.982+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.988800+0000) 2019-10-13T11:01:01.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:01.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:01.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:01.990+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.992522+0000) 2019-10-13T11:01:01.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:01.986+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.993087+0000) 2019-10-13T11:01:01.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:01.986+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.993402+0000) 2019-10-13T11:01:01.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:01.990+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.994093+0000) 2019-10-13T11:01:01.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:01.994+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:01.996637+0000) 2019-10-13T11:01:04.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:04.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:04.104389+0000) 2019-10-13T11:01:04.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:04.390+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:04.390415+0000) 2019-10-13T11:01:06.996 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:06.990+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.995385+0000) 2019-10-13T11:01:06.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:06.990+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.996538+0000) 2019-10-13T11:01:06.997 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:06.990+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.996937+0000) 2019-10-13T11:01:06.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:06.990+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.997443+0000) 2019-10-13T11:01:06.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:06.994+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.998130+0000) 2019-10-13T11:01:06.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:06.994+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.998768+0000) 2019-10-13T11:01:07.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:06.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:06.999399+0000) 2019-10-13T11:01:07.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:06.998+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:07.001597+0000) 2019-10-13T11:01:07.003 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:06.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:07.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:07.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:07.763383+0000) 2019-10-13T11:01:08.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:08.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:08.533482+0000) 2019-10-13T11:01:11.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:11.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.239144+0000) 2019-10-13T11:01:11.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:11.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.260865+0000) 2019-10-13T11:01:11.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:11.422+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.423400+0000) 2019-10-13T11:01:11.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:11.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:11.784683+0000) 2019-10-13T11:01:12.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:12.002+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.005092+0000) 2019-10-13T11:01:12.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:12.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.006820+0000) 2019-10-13T11:01:12.008 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:12.002+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.007696+0000) 2019-10-13T11:01:12.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:12.002+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.007966+0000) 2019-10-13T11:01:12.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:12.002+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.008445+0000) 2019-10-13T11:01:12.009 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:12.002+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.008884+0000) 2019-10-13T11:01:12.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:12.006+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.008930+0000) 2019-10-13T11:01:12.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:12.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:12.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:12.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:12.012012+0000) 2019-10-13T11:01:14.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:14.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:14.104622+0000) 2019-10-13T11:01:14.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:14.386+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:14.390633+0000) 2019-10-13T11:01:15.893 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:01:15.893 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:15.897 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:01:15.897 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:17.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:17.010+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.015212+0000) 2019-10-13T11:01:17.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:17.010+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.016654+0000) 2019-10-13T11:01:17.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:17.010+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.017339+0000) 2019-10-13T11:01:17.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:17.014+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.018109+0000) 2019-10-13T11:01:17.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:17.014+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.018411+0000) 2019-10-13T11:01:17.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:17.014+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.019476+0000) 2019-10-13T11:01:17.021 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:17.014+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.020364+0000) 2019-10-13T11:01:17.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:17.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:17.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:17.022+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.024181+0000) 2019-10-13T11:01:17.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:17.758+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:17.763587+0000) 2019-10-13T11:01:18.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:18.526+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:18.533679+0000) 2019-10-13T11:01:21.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:21.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.239354+0000) 2019-10-13T11:01:21.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:21.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.261070+0000) 2019-10-13T11:01:21.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:21.434+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.434711+0000) 2019-10-13T11:01:21.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:21.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:21.784900+0000) 2019-10-13T11:01:22.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:22.017+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.024318+0000) 2019-10-13T11:01:22.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:22.017+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.025199+0000) 2019-10-13T11:01:22.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:22.017+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.025596+0000) 2019-10-13T11:01:22.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:22.021+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.026464+0000) 2019-10-13T11:01:22.027 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:22.022+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.026570+0000) 2019-10-13T11:01:22.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:22.021+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.027500+0000) 2019-10-13T11:01:22.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:22.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.027998+0000) 2019-10-13T11:01:22.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:22.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:22.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:22.030+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:22.033802+0000) 2019-10-13T11:01:24.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:24.097+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:24.104818+0000) 2019-10-13T11:01:24.391 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:24.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:24.390834+0000) 2019-10-13T11:01:27.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:27.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.034513+0000) 2019-10-13T11:01:27.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:27.029+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.034996+0000) 2019-10-13T11:01:27.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:27.030+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.035244+0000) 2019-10-13T11:01:27.038 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:27.029+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.037137+0000) 2019-10-13T11:01:27.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:27.033+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.039325+0000) 2019-10-13T11:01:27.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:27.033+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.039806+0000) 2019-10-13T11:01:27.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:27.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:27.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:27.033+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.040939+0000) 2019-10-13T11:01:27.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:27.042+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.045958+0000) 2019-10-13T11:01:27.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:27.757+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:27.763793+0000) 2019-10-13T11:01:28.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:28.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:28.533931+0000) 2019-10-13T11:01:31.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:31.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.239552+0000) 2019-10-13T11:01:31.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:31.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.261276+0000) 2019-10-13T11:01:31.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:31.446+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.446254+0000) 2019-10-13T11:01:31.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:31.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:31.785101+0000) 2019-10-13T11:01:32.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:32.037+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.043336+0000) 2019-10-13T11:01:32.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:32.037+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.043668+0000) 2019-10-13T11:01:32.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:32.037+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.043921+0000) 2019-10-13T11:01:32.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:32.037+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.044326+0000) 2019-10-13T11:01:32.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:32.042+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.044458+0000) 2019-10-13T11:01:32.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:32.037+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.045391+0000) 2019-10-13T11:01:32.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:32.041+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.045981+0000) 2019-10-13T11:01:32.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:32.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129484 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:32.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:32.050+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:32.053182+0000) 2019-10-13T11:01:34.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:34.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:34.105022+0000) 2019-10-13T11:01:34.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:34.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:34.391045+0000) 2019-10-13T11:01:37.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:37.049+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.053458+0000) 2019-10-13T11:01:37.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:37.053+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.053900+0000) 2019-10-13T11:01:37.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:37.053+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.054415+0000) 2019-10-13T11:01:37.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:37.053+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.054771+0000) 2019-10-13T11:01:37.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:37.053+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.055087+0000) 2019-10-13T11:01:37.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:37.053+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.055535+0000) 2019-10-13T11:01:37.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:37.050+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.055732+0000) 2019-10-13T11:01:37.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:37.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:37.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:37.058+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.063592+0000) 2019-10-13T11:01:37.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:37.761+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:37.764001+0000) 2019-10-13T11:01:38.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:38.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:38.534130+0000) 2019-10-13T11:01:41.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:41.237+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.239718+0000) 2019-10-13T11:01:41.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:41.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.261466+0000) 2019-10-13T11:01:41.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:41.450+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.453023+0000) 2019-10-13T11:01:41.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:41.782+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:41.785298+0000) 2019-10-13T11:01:42.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:42.061+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.062505+0000) 2019-10-13T11:01:42.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:42.061+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.062996+0000) 2019-10-13T11:01:42.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:42.061+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.064478+0000) 2019-10-13T11:01:42.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:42.061+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.064916+0000) 2019-10-13T11:01:42.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:42.062+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.064911+0000) 2019-10-13T11:01:42.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:42.061+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.065488+0000) 2019-10-13T11:01:42.067 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:42.065+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.065853+0000) 2019-10-13T11:01:42.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:42.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:42.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:42.066+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:42.071991+0000) 2019-10-13T11:01:44.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:44.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:44.105219+0000) 2019-10-13T11:01:44.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:44.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:44.391278+0000) 2019-10-13T11:01:46.188 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:01:46.188 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:46.192 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:01:46.192 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:01:47.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:47.069+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.071688+0000) 2019-10-13T11:01:47.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:47.069+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.072341+0000) 2019-10-13T11:01:47.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:47.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.074037+0000) 2019-10-13T11:01:47.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:47.073+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.074958+0000) 2019-10-13T11:01:47.076 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:47.069+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.074840+0000) 2019-10-13T11:01:47.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:47.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.075396+0000) 2019-10-13T11:01:47.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:47.073+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.075595+0000) 2019-10-13T11:01:47.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:47.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:47.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:47.077+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.080361+0000) 2019-10-13T11:01:47.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:47.761+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:47.764198+0000) 2019-10-13T11:01:48.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:48.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:48.534354+0000) 2019-10-13T11:01:51.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:51.237+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.239946+0000) 2019-10-13T11:01:51.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:51.257+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.261702+0000) 2019-10-13T11:01:51.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:51.453+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.459496+0000) 2019-10-13T11:01:51.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:51.781+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:51.785488+0000) 2019-10-13T11:01:52.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:52.081+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.082079+0000) 2019-10-13T11:01:52.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:52.081+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.083389+0000) 2019-10-13T11:01:52.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:52.081+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.082930+0000) 2019-10-13T11:01:52.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:52.081+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.084513+0000) 2019-10-13T11:01:52.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:52.081+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.085186+0000) 2019-10-13T11:01:52.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:52.081+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.085503+0000) 2019-10-13T11:01:52.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:52.085+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.085929+0000) 2019-10-13T11:01:52.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:52.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:52.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:52.085+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:52.088448+0000) 2019-10-13T11:01:54.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:54.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:54.105422+0000) 2019-10-13T11:01:54.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:54.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:54.391490+0000) 2019-10-13T11:01:57.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:57.089+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.091937+0000) 2019-10-13T11:01:57.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:01:57.089+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.092153+0000) 2019-10-13T11:01:57.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:57.089+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.092672+0000) 2019-10-13T11:01:57.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:01:57.089+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.093499+0000) 2019-10-13T11:01:57.094 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:01:57.093+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.093811+0000) 2019-10-13T11:01:57.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:01:57.093+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.094271+0000) 2019-10-13T11:01:57.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:01:57.093+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.094554+0000) 2019-10-13T11:01:57.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:01:57.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:01:57.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:01:57.097+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.100885+0000) 2019-10-13T11:01:57.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:01:57.761+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:57.764380+0000) 2019-10-13T11:01:58.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:01:58.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:01:58.534575+0000) 2019-10-13T11:02:01.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:01.237+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.240138+0000) 2019-10-13T11:02:01.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:01.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.261896+0000) 2019-10-13T11:02:01.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:01.465+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.465869+0000) 2019-10-13T11:02:01.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:01.781+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:01.785705+0000) 2019-10-13T11:02:02.102 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:02.097+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.101166+0000) 2019-10-13T11:02:02.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:02.097+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.101276+0000) 2019-10-13T11:02:02.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:02.101+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.102078+0000) 2019-10-13T11:02:02.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:02.097+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.103116+0000) 2019-10-13T11:02:02.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:02.101+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.103839+0000) 2019-10-13T11:02:02.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:02.101+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.103833+0000) 2019-10-13T11:02:02.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:02.101+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.104362+0000) 2019-10-13T11:02:02.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:02.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:02.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:02.105+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:02.110240+0000) 2019-10-13T11:02:04.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:04.101+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:04.105641+0000) 2019-10-13T11:02:04.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:04.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:04.391749+0000) 2019-10-13T11:02:07.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:07.109+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.111525+0000) 2019-10-13T11:02:07.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:07.109+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.111959+0000) 2019-10-13T11:02:07.117 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:07.105+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.111906+0000) 2019-10-13T11:02:07.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:07.109+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.112902+0000) 2019-10-13T11:02:07.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:07.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.113338+0000) 2019-10-13T11:02:07.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:07.113+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.113776+0000) 2019-10-13T11:02:07.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:07.113+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.113964+0000) 2019-10-13T11:02:07.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:07.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129699 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:07.125 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:07.117+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.121057+0000) 2019-10-13T11:02:07.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:07.761+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:07.764572+0000) 2019-10-13T11:02:08.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:08.533+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:08.534768+0000) 2019-10-13T11:02:11.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:11.237+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.240341+0000) 2019-10-13T11:02:11.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:11.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.262129+0000) 2019-10-13T11:02:11.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:11.469+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.472308+0000) 2019-10-13T11:02:11.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:11.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:11.785897+0000) 2019-10-13T11:02:12.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:12.117+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.119783+0000) 2019-10-13T11:02:12.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:12.117+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.120163+0000) 2019-10-13T11:02:12.133 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:12.117+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.120522+0000) 2019-10-13T11:02:12.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:12.117+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.121349+0000) 2019-10-13T11:02:12.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:12.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.121811+0000) 2019-10-13T11:02:12.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:12.121+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.122823+0000) 2019-10-13T11:02:12.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:12.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:12.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:12.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.121914+0000) 2019-10-13T11:02:12.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:12.125+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:12.128974+0000) 2019-10-13T11:02:14.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:14.105+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:14.105867+0000) 2019-10-13T11:02:14.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:14.389+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:14.391965+0000) 2019-10-13T11:02:16.269 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:02:16.269 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:16.273 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:02:16.273 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:17.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:17.124+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.129438+0000) 2019-10-13T11:02:17.130 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:17.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.130003+0000) 2019-10-13T11:02:17.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:17.128+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.131437+0000) 2019-10-13T11:02:17.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:17.129+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.132178+0000) 2019-10-13T11:02:17.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:17.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.132794+0000) 2019-10-13T11:02:17.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:17.132+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.133888+0000) 2019-10-13T11:02:17.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:17.132+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.134174+0000) 2019-10-13T11:02:17.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:17.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:17.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:17.137+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.140865+0000) 2019-10-13T11:02:17.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:17.760+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:17.764771+0000) 2019-10-13T11:02:18.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:18.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:18.534964+0000) 2019-10-13T11:02:21.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:21.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.240533+0000) 2019-10-13T11:02:21.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:21.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.262341+0000) 2019-10-13T11:02:21.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:21.473+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.478660+0000) 2019-10-13T11:02:21.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:21.781+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:21.786124+0000) 2019-10-13T11:02:22.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:22.136+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.140160+0000) 2019-10-13T11:02:22.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:22.136+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.140884+0000) 2019-10-13T11:02:22.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:22.136+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.141408+0000) 2019-10-13T11:02:22.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:22.137+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.142316+0000) 2019-10-13T11:02:22.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:22.140+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.143152+0000) 2019-10-13T11:02:22.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:22.140+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.143549+0000) 2019-10-13T11:02:22.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:22.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.144123+0000) 2019-10-13T11:02:22.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:22.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:22.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:22.145+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:22.151203+0000) 2019-10-13T11:02:24.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:24.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:24.106105+0000) 2019-10-13T11:02:24.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:24.388+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:24.392203+0000) 2019-10-13T11:02:27.150 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:27.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.149622+0000) 2019-10-13T11:02:27.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:27.148+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.150380+0000) 2019-10-13T11:02:27.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:27.148+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.153112+0000) 2019-10-13T11:02:27.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:27.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.153837+0000) 2019-10-13T11:02:27.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:27.152+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.154144+0000) 2019-10-13T11:02:27.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:27.149+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.154405+0000) 2019-10-13T11:02:27.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:27.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.155242+0000) 2019-10-13T11:02:27.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:27.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129823 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:27.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:27.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.158996+0000) 2019-10-13T11:02:27.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:27.760+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:27.764980+0000) 2019-10-13T11:02:28.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:28.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:28.535149+0000) 2019-10-13T11:02:31.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:31.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.240745+0000) 2019-10-13T11:02:31.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:31.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.262559+0000) 2019-10-13T11:02:31.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:31.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.485779+0000) 2019-10-13T11:02:31.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:31.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:31.786269+0000) 2019-10-13T11:02:32.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:32.156+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.159335+0000) 2019-10-13T11:02:32.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:32.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.161598+0000) 2019-10-13T11:02:32.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:32.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.161698+0000) 2019-10-13T11:02:32.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:32.157+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.162224+0000) 2019-10-13T11:02:32.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:32.160+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.162802+0000) 2019-10-13T11:02:32.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:32.160+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.163275+0000) 2019-10-13T11:02:32.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:32.160+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.163776+0000) 2019-10-13T11:02:32.166 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:32.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:32.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:32.165+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:32.171760+0000) 2019-10-13T11:02:34.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:34.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:34.106306+0000) 2019-10-13T11:02:34.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:34.388+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:34.392403+0000) 2019-10-13T11:02:37.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:37.165+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.169489+0000) 2019-10-13T11:02:37.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:37.168+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.171850+0000) 2019-10-13T11:02:37.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:37.168+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.172218+0000) 2019-10-13T11:02:37.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:37.168+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.173005+0000) 2019-10-13T11:02:37.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:37.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.173623+0000) 2019-10-13T11:02:37.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:37.172+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.174953+0000) 2019-10-13T11:02:37.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:37.172+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.175925+0000) 2019-10-13T11:02:37.177 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:37.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:37.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:37.177+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.182419+0000) 2019-10-13T11:02:37.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:37.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:37.765149+0000) 2019-10-13T11:02:38.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:38.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:38.535347+0000) 2019-10-13T11:02:41.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:41.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.240955+0000) 2019-10-13T11:02:41.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:41.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.262772+0000) 2019-10-13T11:02:41.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:41.489+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.492149+0000) 2019-10-13T11:02:41.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:41.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:41.786467+0000) 2019-10-13T11:02:42.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:42.176+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.179552+0000) 2019-10-13T11:02:42.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:42.176+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.180368+0000) 2019-10-13T11:02:42.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:42.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.181021+0000) 2019-10-13T11:02:42.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:42.177+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.181361+0000) 2019-10-13T11:02:42.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:42.180+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.182032+0000) 2019-10-13T11:02:42.183 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:42.180+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.182431+0000) 2019-10-13T11:02:42.183 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:42.180+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.183083+0000) 2019-10-13T11:02:42.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:42.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:42.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:42.181+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:42.187382+0000) 2019-10-13T11:02:44.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:44.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:44.106560+0000) 2019-10-13T11:02:44.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:44.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:44.392609+0000) 2019-10-13T11:02:46.383 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:02:46.384 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:46.387 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:02:46.388 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:02:47.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:47.185+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.190332+0000) 2019-10-13T11:02:47.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:47.188+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.190848+0000) 2019-10-13T11:02:47.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:47.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.191660+0000) 2019-10-13T11:02:47.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:47.188+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.192708+0000) 2019-10-13T11:02:47.193 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:47.188+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.192702+0000) 2019-10-13T11:02:47.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:47.188+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.193282+0000) 2019-10-13T11:02:47.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:47.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.193667+0000) 2019-10-13T11:02:47.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:47.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:47.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:47.197+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.200542+0000) 2019-10-13T11:02:47.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:47.760+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:47.765327+0000) 2019-10-13T11:02:48.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:48.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:48.535547+0000) 2019-10-13T11:02:51.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:51.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.241171+0000) 2019-10-13T11:02:51.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:51.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.262993+0000) 2019-10-13T11:02:51.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:51.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.498672+0000) 2019-10-13T11:02:51.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:51.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:51.786658+0000) 2019-10-13T11:02:52.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:52.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.199850+0000) 2019-10-13T11:02:52.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:52.196+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.200254+0000) 2019-10-13T11:02:52.202 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:52.196+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.201776+0000) 2019-10-13T11:02:52.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:52.200+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.202180+0000) 2019-10-13T11:02:52.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:52.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.202908+0000) 2019-10-13T11:02:52.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:52.200+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.203252+0000) 2019-10-13T11:02:52.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:52.200+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.204348+0000) 2019-10-13T11:02:52.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:52.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:52.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:52.204+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:52.209079+0000) 2019-10-13T11:02:54.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:54.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:54.106799+0000) 2019-10-13T11:02:54.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:54.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:54.392863+0000) 2019-10-13T11:02:57.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:02:57.208+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.210123+0000) 2019-10-13T11:02:57.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:02:57.204+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.210150+0000) 2019-10-13T11:02:57.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:02:57.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.210606+0000) 2019-10-13T11:02:57.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:02:57.208+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.210952+0000) 2019-10-13T11:02:57.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:02:57.208+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.211361+0000) 2019-10-13T11:02:57.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:57.208+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.211829+0000) 2019-10-13T11:02:57.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:57.208+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.211935+0000) 2019-10-13T11:02:57.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:02:57.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:02:57.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:02:57.212+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.218419+0000) 2019-10-13T11:02:57.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:02:57.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:57.765502+0000) 2019-10-13T11:02:58.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:02:58.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:02:58.535736+0000) 2019-10-13T11:03:01.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:01.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.241370+0000) 2019-10-13T11:03:01.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:01.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.263187+0000) 2019-10-13T11:03:01.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:01.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.504647+0000) 2019-10-13T11:03:01.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:01.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:01.786862+0000) 2019-10-13T11:03:02.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:02.212+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.218757+0000) 2019-10-13T11:03:02.221 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:02.216+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.219768+0000) 2019-10-13T11:03:02.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:02.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.220754+0000) 2019-10-13T11:03:02.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:02.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.221299+0000) 2019-10-13T11:03:02.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:02.216+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.221559+0000) 2019-10-13T11:03:02.223 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:02.220+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.221907+0000) 2019-10-13T11:03:02.223 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:02.220+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.223088+0000) 2019-10-13T11:03:02.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:02.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:02.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:02.220+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:02.226107+0000) 2019-10-13T11:03:04.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:04.104+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:04.107001+0000) 2019-10-13T11:03:04.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:04.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:04.393081+0000) 2019-10-13T11:03:07.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:07.224+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.228778+0000) 2019-10-13T11:03:07.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:07.224+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.228652+0000) 2019-10-13T11:03:07.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:07.228+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.230865+0000) 2019-10-13T11:03:07.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:07.228+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.233218+0000) 2019-10-13T11:03:07.234 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:07.228+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.233672+0000) 2019-10-13T11:03:07.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:07.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130071 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:07.236 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:07.228+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.235667+0000) 2019-10-13T11:03:07.237 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:07.232+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.236224+0000) 2019-10-13T11:03:07.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:07.236+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.242994+0000) 2019-10-13T11:03:07.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:07.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:07.765677+0000) 2019-10-13T11:03:08.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:08.532+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:08.535885+0000) 2019-10-13T11:03:11.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:11.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.241571+0000) 2019-10-13T11:03:11.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:11.260+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.263422+0000) 2019-10-13T11:03:11.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:11.508+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.511132+0000) 2019-10-13T11:03:11.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:11.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:11.787073+0000) 2019-10-13T11:03:12.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:12.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.240027+0000) 2019-10-13T11:03:12.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:12.235+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.240323+0000) 2019-10-13T11:03:12.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:12.235+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.241272+0000) 2019-10-13T11:03:12.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:12.235+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.240676+0000) 2019-10-13T11:03:12.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:12.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.241113+0000) 2019-10-13T11:03:12.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:12.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.241668+0000) 2019-10-13T11:03:12.243 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:12.236+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.242483+0000) 2019-10-13T11:03:12.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:12.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:12.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:12.240+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:12.245212+0000) 2019-10-13T11:03:14.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:14.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:14.107204+0000) 2019-10-13T11:03:14.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:14.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:14.393299+0000) 2019-10-13T11:03:16.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:03:16.432 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:16.436 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:03:16.436 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:17.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:17.243+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.248606+0000) 2019-10-13T11:03:17.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:17.243+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.249137+0000) 2019-10-13T11:03:17.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:17.247+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.250281+0000) 2019-10-13T11:03:17.251 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:17.244+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.250626+0000) 2019-10-13T11:03:17.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:17.247+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.251421+0000) 2019-10-13T11:03:17.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:17.247+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.251958+0000) 2019-10-13T11:03:17.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:17.247+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.252627+0000) 2019-10-13T11:03:17.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:17.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:17.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:17.252+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.256668+0000) 2019-10-13T11:03:17.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:17.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:17.765835+0000) 2019-10-13T11:03:18.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:18.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:18.536108+0000) 2019-10-13T11:03:21.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:21.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.241789+0000) 2019-10-13T11:03:21.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:21.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.263660+0000) 2019-10-13T11:03:21.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:21.516+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.517182+0000) 2019-10-13T11:03:21.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:21.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:21.787261+0000) 2019-10-13T11:03:22.258 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:22.255+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.257982+0000) 2019-10-13T11:03:22.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:22.255+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.259633+0000) 2019-10-13T11:03:22.261 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:22.255+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.260054+0000) 2019-10-13T11:03:22.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:22.255+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.260685+0000) 2019-10-13T11:03:22.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:22.256+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.261780+0000) 2019-10-13T11:03:22.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:22.259+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.262809+0000) 2019-10-13T11:03:22.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:22.259+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.263188+0000) 2019-10-13T11:03:22.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:22.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:22.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:22.260+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:22.267097+0000) 2019-10-13T11:03:24.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:24.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:24.107404+0000) 2019-10-13T11:03:24.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:24.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:24.393468+0000) 2019-10-13T11:03:27.269 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:27.263+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.268261+0000) 2019-10-13T11:03:27.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:27.263+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.268915+0000) 2019-10-13T11:03:27.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:27.263+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.269146+0000) 2019-10-13T11:03:27.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:27.267+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.271174+0000) 2019-10-13T11:03:27.272 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:27.267+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.271867+0000) 2019-10-13T11:03:27.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:27.268+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.272223+0000) 2019-10-13T11:03:27.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:27.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.272891+0000) 2019-10-13T11:03:27.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:27.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:27.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:27.272+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.276994+0000) 2019-10-13T11:03:27.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:27.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:27.765971+0000) 2019-10-13T11:03:28.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:28.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:28.536265+0000) 2019-10-13T11:03:31.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:31.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.241996+0000) 2019-10-13T11:03:31.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:31.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.263900+0000) 2019-10-13T11:03:31.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:31.520+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.523721+0000) 2019-10-13T11:03:31.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:31.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:31.787459+0000) 2019-10-13T11:03:32.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:32.271+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.277408+0000) 2019-10-13T11:03:32.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:32.275+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.278126+0000) 2019-10-13T11:03:32.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:32.275+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.278442+0000) 2019-10-13T11:03:32.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:32.272+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.278529+0000) 2019-10-13T11:03:32.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:32.275+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.279308+0000) 2019-10-13T11:03:32.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:32.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.280058+0000) 2019-10-13T11:03:32.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:32.275+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.281121+0000) 2019-10-13T11:03:32.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:32.279+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:32.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:32.280+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:32.286448+0000) 2019-10-13T11:03:34.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:34.103+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:34.107609+0000) 2019-10-13T11:03:34.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:34.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:34.393684+0000) 2019-10-13T11:03:37.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:37.283+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.287082+0000) 2019-10-13T11:03:37.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:37.283+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.287863+0000) 2019-10-13T11:03:37.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:37.283+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.288315+0000) 2019-10-13T11:03:37.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:37.284+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.288186+0000) 2019-10-13T11:03:37.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:37.283+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.288710+0000) 2019-10-13T11:03:37.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:37.283+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.289320+0000) 2019-10-13T11:03:37.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:37.283+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.288978+0000) 2019-10-13T11:03:37.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:37.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:37.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:37.292+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.296420+0000) 2019-10-13T11:03:37.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:37.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:37.766155+0000) 2019-10-13T11:03:38.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:38.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:38.536512+0000) 2019-10-13T11:03:41.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:41.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.242151+0000) 2019-10-13T11:03:41.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:41.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.264115+0000) 2019-10-13T11:03:41.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:41.528+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.529574+0000) 2019-10-13T11:03:41.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:41.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:41.787678+0000) 2019-10-13T11:03:42.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:42.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.297492+0000) 2019-10-13T11:03:42.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:42.295+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.297814+0000) 2019-10-13T11:03:42.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:42.295+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.298645+0000) 2019-10-13T11:03:42.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:42.295+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.299007+0000) 2019-10-13T11:03:42.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:42.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.299128+0000) 2019-10-13T11:03:42.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:42.295+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.299742+0000) 2019-10-13T11:03:42.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:42.296+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.300262+0000) 2019-10-13T11:03:42.304 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:42.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:42.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:42.300+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:42.305666+0000) 2019-10-13T11:03:44.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:44.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:44.107808+0000) 2019-10-13T11:03:44.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:44.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:44.393924+0000) 2019-10-13T11:03:46.553 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:03:46.553 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:46.560 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:03:46.561 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:03:47.307 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:47.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.306846+0000) 2019-10-13T11:03:47.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:47.304+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.309651+0000) 2019-10-13T11:03:47.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:47.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.309629+0000) 2019-10-13T11:03:47.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:47.307+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.310186+0000) 2019-10-13T11:03:47.313 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:47.307+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.310743+0000) 2019-10-13T11:03:47.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:47.307+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.311276+0000) 2019-10-13T11:03:47.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:47.307+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.311877+0000) 2019-10-13T11:03:47.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:47.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:47.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:47.308+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.313483+0000) 2019-10-13T11:03:47.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:47.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:47.766381+0000) 2019-10-13T11:03:48.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:48.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:48.536711+0000) 2019-10-13T11:03:51.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:51.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.242349+0000) 2019-10-13T11:03:51.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:51.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.264347+0000) 2019-10-13T11:03:51.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:51.532+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.535877+0000) 2019-10-13T11:03:51.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:51.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:51.787834+0000) 2019-10-13T11:03:52.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:52.311+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.315466+0000) 2019-10-13T11:03:52.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:52.311+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.316714+0000) 2019-10-13T11:03:52.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:52.311+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.317143+0000) 2019-10-13T11:03:52.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:52.315+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.319687+0000) 2019-10-13T11:03:52.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:52.320+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.321073+0000) 2019-10-13T11:03:52.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:52.315+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.321620+0000) 2019-10-13T11:03:52.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:52.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:52.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:52.319+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.322085+0000) 2019-10-13T11:03:52.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:52.320+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:52.322089+0000) 2019-10-13T11:03:54.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:54.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:54.108006+0000) 2019-10-13T11:03:54.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:54.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:54.394152+0000) 2019-10-13T11:03:57.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:03:57.323+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.326555+0000) 2019-10-13T11:03:57.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:57.323+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.326986+0000) 2019-10-13T11:03:57.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:03:57.327+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.328648+0000) 2019-10-13T11:03:57.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:03:57.323+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.329499+0000) 2019-10-13T11:03:57.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:57.327+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.331242+0000) 2019-10-13T11:03:57.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:03:57.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130381 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:03:57.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:03:57.327+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.332780+0000) 2019-10-13T11:03:57.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:03:57.327+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.333097+0000) 2019-10-13T11:03:57.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:03:57.335+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.337852+0000) 2019-10-13T11:03:57.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:03:57.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:57.766557+0000) 2019-10-13T11:03:58.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:03:58.531+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:03:58.536908+0000) 2019-10-13T11:04:01.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:01.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.242546+0000) 2019-10-13T11:04:01.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:01.259+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.264579+0000) 2019-10-13T11:04:01.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:01.539+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.541519+0000) 2019-10-13T11:04:01.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:01.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:01.787986+0000) 2019-10-13T11:04:02.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:02.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.334833+0000) 2019-10-13T11:04:02.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:02.331+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.336432+0000) 2019-10-13T11:04:02.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:02.331+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.336770+0000) 2019-10-13T11:04:02.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:02.331+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.337500+0000) 2019-10-13T11:04:02.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:02.335+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.338157+0000) 2019-10-13T11:04:02.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:02.335+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.338544+0000) 2019-10-13T11:04:02.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:02.335+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.338944+0000) 2019-10-13T11:04:02.343 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:02.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:02.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:02.343+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:02.346865+0000) 2019-10-13T11:04:04.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:04.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:04.108204+0000) 2019-10-13T11:04:04.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:04.391+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:04.394353+0000) 2019-10-13T11:04:07.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:07.343+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.346087+0000) 2019-10-13T11:04:07.347 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:07.343+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.346879+0000) 2019-10-13T11:04:07.348 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:07.343+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.347421+0000) 2019-10-13T11:04:07.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:07.343+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.347289+0000) 2019-10-13T11:04:07.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:07.347+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.350331+0000) 2019-10-13T11:04:07.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:07.343+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.348010+0000) 2019-10-13T11:04:07.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:07.347+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.350865+0000) 2019-10-13T11:04:07.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:07.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:07.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:07.351+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.354839+0000) 2019-10-13T11:04:07.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:07.763+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:07.766750+0000) 2019-10-13T11:04:08.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:08.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:08.537116+0000) 2019-10-13T11:04:11.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:11.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.242750+0000) 2019-10-13T11:04:11.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:11.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.264805+0000) 2019-10-13T11:04:11.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:11.547+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.547957+0000) 2019-10-13T11:04:11.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:11.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:11.788163+0000) 2019-10-13T11:04:12.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:12.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.355168+0000) 2019-10-13T11:04:12.357 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:12.350+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.356552+0000) 2019-10-13T11:04:12.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:12.350+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.356991+0000) 2019-10-13T11:04:12.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:12.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.357409+0000) 2019-10-13T11:04:12.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:12.354+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.357885+0000) 2019-10-13T11:04:12.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:12.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.358511+0000) 2019-10-13T11:04:12.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:12.354+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.359615+0000) 2019-10-13T11:04:12.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:12.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:12.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:12.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:12.362532+0000) 2019-10-13T11:04:14.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:14.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:14.108424+0000) 2019-10-13T11:04:14.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:14.390+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:14.394551+0000) 2019-10-13T11:04:16.659 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:04:16.660 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:16.663 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:04:16.664 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:17.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:17.362+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.366548+0000) 2019-10-13T11:04:17.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:17.363+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.366721+0000) 2019-10-13T11:04:17.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:17.362+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.367736+0000) 2019-10-13T11:04:17.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:17.362+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.368088+0000) 2019-10-13T11:04:17.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:17.366+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.370543+0000) 2019-10-13T11:04:17.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:17.366+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.370678+0000) 2019-10-13T11:04:17.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:17.367+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.370832+0000) 2019-10-13T11:04:17.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:17.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:17.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:17.366+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.372022+0000) 2019-10-13T11:04:17.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:17.762+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:17.766929+0000) 2019-10-13T11:04:18.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:18.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:18.537305+0000) 2019-10-13T11:04:21.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:21.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.242957+0000) 2019-10-13T11:04:21.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:21.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.264997+0000) 2019-10-13T11:04:21.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:21.551+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.554300+0000) 2019-10-13T11:04:21.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:21.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:21.788361+0000) 2019-10-13T11:04:22.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:22.370+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.374773+0000) 2019-10-13T11:04:22.376 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:22.370+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.375412+0000) 2019-10-13T11:04:22.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:22.371+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.375550+0000) 2019-10-13T11:04:22.377 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:22.370+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.376309+0000) 2019-10-13T11:04:22.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:22.370+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.376178+0000) 2019-10-13T11:04:22.378 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:22.370+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.377636+0000) 2019-10-13T11:04:22.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:22.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.378358+0000) 2019-10-13T11:04:22.382 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:22.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:22.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:22.383+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:22.385299+0000) 2019-10-13T11:04:24.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:24.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:24.108653+0000) 2019-10-13T11:04:24.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:24.390+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:24.394752+0000) 2019-10-13T11:04:27.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:27.378+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.385720+0000) 2019-10-13T11:04:27.387 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:27.382+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.386160+0000) 2019-10-13T11:04:27.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:27.382+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.387934+0000) 2019-10-13T11:04:27.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:27.382+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.388640+0000) 2019-10-13T11:04:27.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:27.387+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.389237+0000) 2019-10-13T11:04:27.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:27.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:27.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:27.386+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.391850+0000) 2019-10-13T11:04:27.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:27.386+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.392292+0000) 2019-10-13T11:04:27.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:27.391+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.395781+0000) 2019-10-13T11:04:27.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:27.762+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:27.767107+0000) 2019-10-13T11:04:28.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:28.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:28.537499+0000) 2019-10-13T11:04:31.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:31.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.243157+0000) 2019-10-13T11:04:31.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:31.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.265224+0000) 2019-10-13T11:04:31.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:31.559+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.560783+0000) 2019-10-13T11:04:31.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:31.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:31.788571+0000) 2019-10-13T11:04:32.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:32.390+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.394852+0000) 2019-10-13T11:04:32.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:32.391+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.395151+0000) 2019-10-13T11:04:32.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:32.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.397256+0000) 2019-10-13T11:04:32.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:32.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.397544+0000) 2019-10-13T11:04:32.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:32.394+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.398464+0000) 2019-10-13T11:04:32.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:32.394+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.399721+0000) 2019-10-13T11:04:32.401 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:32.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:32.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:32.394+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.401075+0000) 2019-10-13T11:04:32.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:32.403+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:32.406002+0000) 2019-10-13T11:04:34.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:34.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:34.108854+0000) 2019-10-13T11:04:34.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:34.394+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:34.394957+0000) 2019-10-13T11:04:37.405 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:37.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.403926+0000) 2019-10-13T11:04:37.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:37.398+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.405197+0000) 2019-10-13T11:04:37.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:37.398+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.405171+0000) 2019-10-13T11:04:37.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:37.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.405938+0000) 2019-10-13T11:04:37.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:37.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.406358+0000) 2019-10-13T11:04:37.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:37.402+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.407061+0000) 2019-10-13T11:04:37.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:37.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.407548+0000) 2019-10-13T11:04:37.410 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:37.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:37.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:37.407+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.410354+0000) 2019-10-13T11:04:37.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:37.762+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:37.767299+0000) 2019-10-13T11:04:38.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:38.530+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:38.537646+0000) 2019-10-13T11:04:41.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:41.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.243355+0000) 2019-10-13T11:04:41.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:41.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.265476+0000) 2019-10-13T11:04:41.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:41.563+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.567224+0000) 2019-10-13T11:04:41.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:41.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:41.788773+0000) 2019-10-13T11:04:42.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:42.411+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.413009+0000) 2019-10-13T11:04:42.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:42.410+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.413869+0000) 2019-10-13T11:04:42.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:42.410+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.414155+0000) 2019-10-13T11:04:42.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:42.410+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.415081+0000) 2019-10-13T11:04:42.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:42.410+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.415351+0000) 2019-10-13T11:04:42.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:42.410+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.415690+0000) 2019-10-13T11:04:42.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:42.410+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.415967+0000) 2019-10-13T11:04:42.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:42.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:42.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:42.419+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:42.423442+0000) 2019-10-13T11:04:44.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:44.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:44.109087+0000) 2019-10-13T11:04:44.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:44.394+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:44.395185+0000) 2019-10-13T11:04:46.768 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:04:46.768 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:46.772 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:04:46.772 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:04:47.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:47.418+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.422440+0000) 2019-10-13T11:04:47.424 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:47.418+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.423234+0000) 2019-10-13T11:04:47.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:47.418+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.423584+0000) 2019-10-13T11:04:47.425 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:47.418+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.424296+0000) 2019-10-13T11:04:47.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:47.418+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.424061+0000) 2019-10-13T11:04:47.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:47.423+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.424598+0000) 2019-10-13T11:04:47.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:47.418+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.425397+0000) 2019-10-13T11:04:47.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:47.422+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:47.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:47.427+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.431878+0000) 2019-10-13T11:04:47.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:47.762+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:47.767494+0000) 2019-10-13T11:04:48.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:48.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:48.537834+0000) 2019-10-13T11:04:51.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:51.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.243588+0000) 2019-10-13T11:04:51.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:51.258+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.265673+0000) 2019-10-13T11:04:51.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:51.571+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.573287+0000) 2019-10-13T11:04:51.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:51.787+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:51.788965+0000) 2019-10-13T11:04:52.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:52.426+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.431124+0000) 2019-10-13T11:04:52.436 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:52.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.431457+0000) 2019-10-13T11:04:52.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:52.426+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.432313+0000) 2019-10-13T11:04:52.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:52.426+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.433005+0000) 2019-10-13T11:04:52.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:52.430+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.433787+0000) 2019-10-13T11:04:52.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:52.430+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.434069+0000) 2019-10-13T11:04:52.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:52.430+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.434392+0000) 2019-10-13T11:04:52.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:52.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130723 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:52.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:52.439+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:52.440706+0000) 2019-10-13T11:04:54.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:54.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:54.109283+0000) 2019-10-13T11:04:54.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:54.394+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:54.395400+0000) 2019-10-13T11:04:57.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:57.434+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.441292+0000) 2019-10-13T11:04:57.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:04:57.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.441301+0000) 2019-10-13T11:04:57.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:04:57.438+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.442001+0000) 2019-10-13T11:04:57.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:04:57.438+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.442294+0000) 2019-10-13T11:04:57.444 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:04:57.438+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.442804+0000) 2019-10-13T11:04:57.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:57.438+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.444032+0000) 2019-10-13T11:04:57.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:04:57.438+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.444698+0000) 2019-10-13T11:04:57.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:04:57.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:04:57.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:04:57.443+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.447091+0000) 2019-10-13T11:04:57.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:04:57.762+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:57.767672+0000) 2019-10-13T11:04:58.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:04:58.534+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:04:58.538034+0000) 2019-10-13T11:05:01.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:01.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.243791+0000) 2019-10-13T11:05:01.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:01.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.265900+0000) 2019-10-13T11:05:01.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:01.575+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.579843+0000) 2019-10-13T11:05:01.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:01.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:01.789156+0000) 2019-10-13T11:05:02.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:02.446+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.451357+0000) 2019-10-13T11:05:02.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:02.446+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.451941+0000) 2019-10-13T11:05:02.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:02.446+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.452325+0000) 2019-10-13T11:05:02.454 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:02.446+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.452646+0000) 2019-10-13T11:05:02.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:02.446+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.452994+0000) 2019-10-13T11:05:02.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:02.446+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.453370+0000) 2019-10-13T11:05:02.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:02.450+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.453979+0000) 2019-10-13T11:05:02.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:02.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:02.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:02.454+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:02.459833+0000) 2019-10-13T11:05:04.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:04.102+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:04.109480+0000) 2019-10-13T11:05:04.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:04.394+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:04.395608+0000) 2019-10-13T11:05:07.461 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:07.454+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.459696+0000) 2019-10-13T11:05:07.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:07.457+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.461895+0000) 2019-10-13T11:05:07.464 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:07.457+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.463039+0000) 2019-10-13T11:05:07.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:07.457+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.462726+0000) 2019-10-13T11:05:07.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:07.457+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.464387+0000) 2019-10-13T11:05:07.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:07.457+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.464613+0000) 2019-10-13T11:05:07.466 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:07.457+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.465679+0000) 2019-10-13T11:05:07.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:07.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:07.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:07.466+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.471610+0000) 2019-10-13T11:05:07.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:07.761+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:07.767843+0000) 2019-10-13T11:05:08.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:08.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:08.538230+0000) 2019-10-13T11:05:11.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:11.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.244033+0000) 2019-10-13T11:05:11.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:11.261+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.266087+0000) 2019-10-13T11:05:11.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:11.582+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.586363+0000) 2019-10-13T11:05:11.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:11.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:11.789322+0000) 2019-10-13T11:05:12.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:12.465+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.471498+0000) 2019-10-13T11:05:12.473 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:12.465+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.472111+0000) 2019-10-13T11:05:12.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:12.469+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.473876+0000) 2019-10-13T11:05:12.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:12.469+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.474725+0000) 2019-10-13T11:05:12.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:12.470+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.474617+0000) 2019-10-13T11:05:12.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:12.469+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.475501+0000) 2019-10-13T11:05:12.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:12.469+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.477349+0000) 2019-10-13T11:05:12.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:12.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:12.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:12.478+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:12.482147+0000) 2019-10-13T11:05:14.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:14.105+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:14.109702+0000) 2019-10-13T11:05:14.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:14.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:14.395844+0000) 2019-10-13T11:05:16.902 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:05:16.903 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:16.907 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:05:16.907 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:17.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:17.473+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.480589+0000) 2019-10-13T11:05:17.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:17.473+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.481026+0000) 2019-10-13T11:05:17.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:17.473+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.481361+0000) 2019-10-13T11:05:17.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:17.477+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.482897+0000) 2019-10-13T11:05:17.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:17.478+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.483071+0000) 2019-10-13T11:05:17.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:17.486+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.488378+0000) 2019-10-13T11:05:17.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:17.481+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.486948+0000) 2019-10-13T11:05:17.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:17.481+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.487146+0000) 2019-10-13T11:05:17.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:17.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:17.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:17.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:17.768022+0000) 2019-10-13T11:05:18.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:18.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:18.538441+0000) 2019-10-13T11:05:21.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:21.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.244240+0000) 2019-10-13T11:05:21.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:21.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.266237+0000) 2019-10-13T11:05:21.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:21.594+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.597997+0000) 2019-10-13T11:05:21.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:21.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:21.789483+0000) 2019-10-13T11:05:22.495 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:22.489+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.491070+0000) 2019-10-13T11:05:22.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:22.489+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.491868+0000) 2019-10-13T11:05:22.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:22.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.492434+0000) 2019-10-13T11:05:22.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:22.489+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.492836+0000) 2019-10-13T11:05:22.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:22.489+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.493577+0000) 2019-10-13T11:05:22.496 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:22.490+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.495478+0000) 2019-10-13T11:05:22.499 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:22.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:22.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:22.497+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.499299+0000) 2019-10-13T11:05:22.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:22.498+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:22.503720+0000) 2019-10-13T11:05:24.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:24.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:24.109898+0000) 2019-10-13T11:05:24.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:24.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:24.396048+0000) 2019-10-13T11:05:27.503 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:27.501+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.502176+0000) 2019-10-13T11:05:27.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:27.501+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.502553+0000) 2019-10-13T11:05:27.504 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:27.501+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.503501+0000) 2019-10-13T11:05:27.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:27.498+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.503420+0000) 2019-10-13T11:05:27.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:27.501+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.504306+0000) 2019-10-13T11:05:27.505 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:27.501+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.504835+0000) 2019-10-13T11:05:27.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:27.501+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.505423+0000) 2019-10-13T11:05:27.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:27.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:27.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:27.506+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.510184+0000) 2019-10-13T11:05:27.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:27.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:27.768202+0000) 2019-10-13T11:05:28.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:28.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:28.538640+0000) 2019-10-13T11:05:31.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:31.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.244465+0000) 2019-10-13T11:05:31.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:31.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.266432+0000) 2019-10-13T11:05:31.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:31.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.604533+0000) 2019-10-13T11:05:31.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:31.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:31.789710+0000) 2019-10-13T11:05:32.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:32.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.512147+0000) 2019-10-13T11:05:32.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:32.509+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.512422+0000) 2019-10-13T11:05:32.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:32.509+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.513206+0000) 2019-10-13T11:05:32.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:32.509+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.512828+0000) 2019-10-13T11:05:32.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:32.510+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.514242+0000) 2019-10-13T11:05:32.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:32.513+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.514905+0000) 2019-10-13T11:05:32.516 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:32.513+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.515740+0000) 2019-10-13T11:05:32.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:32.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:32.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:32.518+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:32.521097+0000) 2019-10-13T11:05:34.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:34.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:34.110121+0000) 2019-10-13T11:05:34.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:34.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:34.396245+0000) 2019-10-13T11:05:37.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:37.517+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.521374+0000) 2019-10-13T11:05:37.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:37.521+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.521972+0000) 2019-10-13T11:05:37.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:37.521+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.523670+0000) 2019-10-13T11:05:37.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:37.521+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.525558+0000) 2019-10-13T11:05:37.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:37.525+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.526066+0000) 2019-10-13T11:05:37.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:37.525+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.527432+0000) 2019-10-13T11:05:37.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:37.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:37.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:37.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.527752+0000) 2019-10-13T11:05:37.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:37.530+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.533922+0000) 2019-10-13T11:05:37.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:37.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:37.768387+0000) 2019-10-13T11:05:38.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:38.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:38.538845+0000) 2019-10-13T11:05:41.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:41.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.244704+0000) 2019-10-13T11:05:41.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:41.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.266633+0000) 2019-10-13T11:05:41.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:41.610+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.610717+0000) 2019-10-13T11:05:41.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:41.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:41.789913+0000) 2019-10-13T11:05:42.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:42.529+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.530889+0000) 2019-10-13T11:05:42.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:42.530+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.533552+0000) 2019-10-13T11:05:42.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:42.533+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.534338+0000) 2019-10-13T11:05:42.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:42.533+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.534373+0000) 2019-10-13T11:05:42.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:42.533+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.534859+0000) 2019-10-13T11:05:42.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:42.533+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.535347+0000) 2019-10-13T11:05:42.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:42.533+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.535922+0000) 2019-10-13T11:05:42.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:42.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:42.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:42.538+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:42.542317+0000) 2019-10-13T11:05:44.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:44.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:44.110346+0000) 2019-10-13T11:05:44.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:44.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:44.396451+0000) 2019-10-13T11:05:47.070 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:05:47.071 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:47.074 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:05:47.075 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:05:47.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:47.537+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.540360+0000) 2019-10-13T11:05:47.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:47.537+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.541251+0000) 2019-10-13T11:05:47.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:47.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.541602+0000) 2019-10-13T11:05:47.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:47.541+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.542344+0000) 2019-10-13T11:05:47.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:47.541+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.542979+0000) 2019-10-13T11:05:47.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:47.541+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.542930+0000) 2019-10-13T11:05:47.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:47.538+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.543387+0000) 2019-10-13T11:05:47.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:47.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:47.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:47.546+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.550489+0000) 2019-10-13T11:05:47.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:47.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:47.768565+0000) 2019-10-13T11:05:48.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:48.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:48.539015+0000) 2019-10-13T11:05:51.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:51.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.244903+0000) 2019-10-13T11:05:51.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:51.265+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.266832+0000) 2019-10-13T11:05:51.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:51.614+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.617406+0000) 2019-10-13T11:05:51.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:51.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:51.790101+0000) 2019-10-13T11:05:52.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:52.549+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.550484+0000) 2019-10-13T11:05:52.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:52.549+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.550879+0000) 2019-10-13T11:05:52.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:52.549+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.551397+0000) 2019-10-13T11:05:52.554 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:52.549+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.553139+0000) 2019-10-13T11:05:52.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:52.553+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.555738+0000) 2019-10-13T11:05:52.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:52.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:52.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:52.557+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.558208+0000) 2019-10-13T11:05:52.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:52.554+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.558234+0000) 2019-10-13T11:05:52.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:52.554+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:52.558911+0000) 2019-10-13T11:05:54.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:54.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:54.110577+0000) 2019-10-13T11:05:54.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:54.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:54.396682+0000) 2019-10-13T11:05:57.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:57.561+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.562638+0000) 2019-10-13T11:05:57.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:57.561+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.562981+0000) 2019-10-13T11:05:57.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:05:57.561+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.563350+0000) 2019-10-13T11:05:57.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:05:57.558+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.564037+0000) 2019-10-13T11:05:57.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:05:57.561+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.564347+0000) 2019-10-13T11:05:57.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:05:57.561+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.564913+0000) 2019-10-13T11:05:57.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:05:57.561+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.565015+0000) 2019-10-13T11:05:57.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:05:57.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:05:57.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:05:57.566+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.569168+0000) 2019-10-13T11:05:57.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:05:57.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:57.768742+0000) 2019-10-13T11:05:58.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:05:58.537+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:05:58.539191+0000) 2019-10-13T11:06:01.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:01.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.245107+0000) 2019-10-13T11:06:01.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:01.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.267038+0000) 2019-10-13T11:06:01.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:01.622+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.624066+0000) 2019-10-13T11:06:01.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:01.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:01.790298+0000) 2019-10-13T11:06:02.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:02.568+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.571392+0000) 2019-10-13T11:06:02.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:02.568+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.571756+0000) 2019-10-13T11:06:02.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:02.570+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.572861+0000) 2019-10-13T11:06:02.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:02.568+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.573336+0000) 2019-10-13T11:06:02.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:02.572+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.573735+0000) 2019-10-13T11:06:02.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:02.572+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.574080+0000) 2019-10-13T11:06:02.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:02.572+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.574885+0000) 2019-10-13T11:06:02.578 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:02.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:02.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:02.578+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:02.582040+0000) 2019-10-13T11:06:04.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:04.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:04.110791+0000) 2019-10-13T11:06:04.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:04.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:04.396883+0000) 2019-10-13T11:06:07.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:07.576+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.580739+0000) 2019-10-13T11:06:07.582 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:07.576+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.581105+0000) 2019-10-13T11:06:07.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:07.580+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.581846+0000) 2019-10-13T11:06:07.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:07.580+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.582234+0000) 2019-10-13T11:06:07.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:07.578+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.582837+0000) 2019-10-13T11:06:07.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:07.580+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.583474+0000) 2019-10-13T11:06:07.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:07.580+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.584543+0000) 2019-10-13T11:06:07.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:07.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131187 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:07.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:07.586+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.590409+0000) 2019-10-13T11:06:07.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:07.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:07.768924+0000) 2019-10-13T11:06:08.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:08.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:08.539432+0000) 2019-10-13T11:06:11.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:11.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.245301+0000) 2019-10-13T11:06:11.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:11.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.267240+0000) 2019-10-13T11:06:11.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:11.625+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.630612+0000) 2019-10-13T11:06:11.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:11.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:11.790521+0000) 2019-10-13T11:06:12.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:12.588+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.590788+0000) 2019-10-13T11:06:12.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:12.588+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.592769+0000) 2019-10-13T11:06:12.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:12.588+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.593495+0000) 2019-10-13T11:06:12.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:12.589+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.593675+0000) 2019-10-13T11:06:12.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:12.592+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.596900+0000) 2019-10-13T11:06:12.598 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:12.592+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.597537+0000) 2019-10-13T11:06:12.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:12.596+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.598001+0000) 2019-10-13T11:06:12.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:12.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:12.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:12.597+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:12.602446+0000) 2019-10-13T11:06:14.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:14.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:14.111014+0000) 2019-10-13T11:06:14.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:14.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:14.397086+0000) 2019-10-13T11:06:17.364 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:06:17.364 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:17.367 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:06:17.368 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:17.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:17.600+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.601919+0000) 2019-10-13T11:06:17.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:17.600+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.603351+0000) 2019-10-13T11:06:17.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:17.600+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.604118+0000) 2019-10-13T11:06:17.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:17.601+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.604351+0000) 2019-10-13T11:06:17.606 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:17.600+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.604693+0000) 2019-10-13T11:06:17.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:17.600+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.604997+0000) 2019-10-13T11:06:17.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:17.600+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.605397+0000) 2019-10-13T11:06:17.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:17.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:17.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:17.605+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.611993+0000) 2019-10-13T11:06:17.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:17.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:17.769136+0000) 2019-10-13T11:06:18.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:18.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:18.539610+0000) 2019-10-13T11:06:21.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:21.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.245519+0000) 2019-10-13T11:06:21.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:21.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.267473+0000) 2019-10-13T11:06:21.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:21.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.637096+0000) 2019-10-13T11:06:21.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:21.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:21.790678+0000) 2019-10-13T11:06:22.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:22.608+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.611916+0000) 2019-10-13T11:06:22.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:22.608+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.612402+0000) 2019-10-13T11:06:22.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:22.608+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.612830+0000) 2019-10-13T11:06:22.614 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:22.608+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.613445+0000) 2019-10-13T11:06:22.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:22.612+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.613771+0000) 2019-10-13T11:06:22.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:22.609+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.614006+0000) 2019-10-13T11:06:22.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:22.612+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.615060+0000) 2019-10-13T11:06:22.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:22.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:22.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:22.617+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:22.620954+0000) 2019-10-13T11:06:24.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:24.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:24.111216+0000) 2019-10-13T11:06:24.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:24.396+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:24.397284+0000) 2019-10-13T11:06:27.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:27.620+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.622543+0000) 2019-10-13T11:06:27.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:27.620+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.623335+0000) 2019-10-13T11:06:27.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:27.620+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.623572+0000) 2019-10-13T11:06:27.625 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:27.620+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.623893+0000) 2019-10-13T11:06:27.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:27.620+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.624423+0000) 2019-10-13T11:06:27.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:27.624+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.626506+0000) 2019-10-13T11:06:27.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:27.621+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.626554+0000) 2019-10-13T11:06:27.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:27.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:27.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:27.629+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.632861+0000) 2019-10-13T11:06:27.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:27.764+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:27.769317+0000) 2019-10-13T11:06:28.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:28.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:28.539791+0000) 2019-10-13T11:06:31.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:31.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.245721+0000) 2019-10-13T11:06:31.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:31.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.267687+0000) 2019-10-13T11:06:31.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:31.645+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.648308+0000) 2019-10-13T11:06:31.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:31.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:31.790869+0000) 2019-10-13T11:06:32.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:32.628+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.630680+0000) 2019-10-13T11:06:32.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:32.628+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.631173+0000) 2019-10-13T11:06:32.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:32.628+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.631318+0000) 2019-10-13T11:06:32.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:32.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.631980+0000) 2019-10-13T11:06:32.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:32.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.632470+0000) 2019-10-13T11:06:32.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:32.628+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.632740+0000) 2019-10-13T11:06:32.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:32.629+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.635114+0000) 2019-10-13T11:06:32.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:32.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:32.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:32.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:32.640484+0000) 2019-10-13T11:06:34.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:34.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:34.111375+0000) 2019-10-13T11:06:34.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:34.392+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:34.397497+0000) 2019-10-13T11:06:37.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:37.636+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.641237+0000) 2019-10-13T11:06:37.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:37.640+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.642114+0000) 2019-10-13T11:06:37.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:37.640+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.642534+0000) 2019-10-13T11:06:37.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:37.637+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.644045+0000) 2019-10-13T11:06:37.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:37.640+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.644481+0000) 2019-10-13T11:06:37.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:37.644+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.646464+0000) 2019-10-13T11:06:37.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:37.641+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.647690+0000) 2019-10-13T11:06:37.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:37.644+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.648068+0000) 2019-10-13T11:06:37.649 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:37.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:37.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:37.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:37.769496+0000) 2019-10-13T11:06:38.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:38.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:38.539963+0000) 2019-10-13T11:06:41.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:41.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:41.245992+0000) 2019-10-13T11:06:41.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:41.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:41.267873+0000) 2019-10-13T11:06:41.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:41.653+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:41.654744+0000) 2019-10-13T11:06:41.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:41.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:41.791098+0000) 2019-10-13T11:06:42.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:42.648+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.651053+0000) 2019-10-13T11:06:42.652 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:42.648+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.651480+0000) 2019-10-13T11:06:42.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:42.645+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.652014+0000) 2019-10-13T11:06:42.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:42.648+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.652857+0000) 2019-10-13T11:06:42.654 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:42.648+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.653086+0000) 2019-10-13T11:06:42.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:42.652+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.654219+0000) 2019-10-13T11:06:42.655 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:42.652+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.654734+0000) 2019-10-13T11:06:42.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:42.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:42.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:42.653+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:42.659645+0000) 2019-10-13T11:06:44.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:44.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:44.111574+0000) 2019-10-13T11:06:44.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:44.396+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:44.397733+0000) 2019-10-13T11:06:47.407 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:06:47.407 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:47.411 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:06:47.411 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:06:47.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:47.660+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.663222+0000) 2019-10-13T11:06:47.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:47.660+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.663628+0000) 2019-10-13T11:06:47.665 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:47.660+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.663876+0000) 2019-10-13T11:06:47.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:47.664+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.666327+0000) 2019-10-13T11:06:47.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:47.661+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.666859+0000) 2019-10-13T11:06:47.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:47.664+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.667389+0000) 2019-10-13T11:06:47.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:47.664+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.668321+0000) 2019-10-13T11:06:47.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:47.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:47.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:47.669+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.672726+0000) 2019-10-13T11:06:47.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:47.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:47.769674+0000) 2019-10-13T11:06:48.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:48.536+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:48.540155+0000) 2019-10-13T11:06:51.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:51.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:51.246223+0000) 2019-10-13T11:06:51.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:51.264+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:51.268130+0000) 2019-10-13T11:06:51.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:51.657+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:51.661357+0000) 2019-10-13T11:06:51.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:51.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:51.791293+0000) 2019-10-13T11:06:52.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:52.668+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.671355+0000) 2019-10-13T11:06:52.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:52.668+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.672596+0000) 2019-10-13T11:06:52.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:52.668+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.672892+0000) 2019-10-13T11:06:52.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:52.669+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.673521+0000) 2019-10-13T11:06:52.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:52.672+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.675734+0000) 2019-10-13T11:06:52.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:52.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:52.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:52.676+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.677982+0000) 2019-10-13T11:06:52.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:52.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.678544+0000) 2019-10-13T11:06:52.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:52.681+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:52.684355+0000) 2019-10-13T11:06:54.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:54.108+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:54.111777+0000) 2019-10-13T11:06:54.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:54.396+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:54.398004+0000) 2019-10-13T11:06:57.682 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:06:57.675+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.681726+0000) 2019-10-13T11:06:57.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:57.679+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.683000+0000) 2019-10-13T11:06:57.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:06:57.679+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.683322+0000) 2019-10-13T11:06:57.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:06:57.679+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.684133+0000) 2019-10-13T11:06:57.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:06:57.683+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.686663+0000) 2019-10-13T11:06:57.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:06:57.681+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.686663+0000) 2019-10-13T11:06:57.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:57.683+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.687577+0000) 2019-10-13T11:06:57.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:06:57.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131497 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:06:57.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:06:57.689+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.693396+0000) 2019-10-13T11:06:57.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:06:57.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:57.769891+0000) 2019-10-13T11:06:58.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:06:58.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:06:58.540329+0000) 2019-10-13T11:07:01.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:01.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:01.246417+0000) 2019-10-13T11:07:01.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:01.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:01.268322+0000) 2019-10-13T11:07:01.668 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:01.661+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:01.667122+0000) 2019-10-13T11:07:01.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:01.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:01.791480+0000) 2019-10-13T11:07:02.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:02.687+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.691943+0000) 2019-10-13T11:07:02.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:02.687+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.692349+0000) 2019-10-13T11:07:02.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:02.687+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.692691+0000) 2019-10-13T11:07:02.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:02.687+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.693121+0000) 2019-10-13T11:07:02.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:02.691+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.693929+0000) 2019-10-13T11:07:02.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:02.689+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.694113+0000) 2019-10-13T11:07:02.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:02.691+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.694922+0000) 2019-10-13T11:07:02.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:02.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:02.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:02.697+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:02.702246+0000) 2019-10-13T11:07:04.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:04.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:04.111976+0000) 2019-10-13T11:07:04.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:04.395+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:04.398239+0000) 2019-10-13T11:07:07.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:07.695+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.700770+0000) 2019-10-13T11:07:07.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:07.699+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.701797+0000) 2019-10-13T11:07:07.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:07.699+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.702184+0000) 2019-10-13T11:07:07.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:07.699+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.702734+0000) 2019-10-13T11:07:07.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:07.699+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.703054+0000) 2019-10-13T11:07:07.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:07.697+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.703616+0000) 2019-10-13T11:07:07.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:07.699+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.704326+0000) 2019-10-13T11:07:07.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:07.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131558 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:07.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:07.705+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.709552+0000) 2019-10-13T11:07:07.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:07.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:07.770060+0000) 2019-10-13T11:07:08.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:08.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:08.540502+0000) 2019-10-13T11:07:11.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:11.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:11.246670+0000) 2019-10-13T11:07:11.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:11.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:11.268543+0000) 2019-10-13T11:07:11.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:11.673+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:11.673851+0000) 2019-10-13T11:07:11.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:11.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:11.791662+0000) 2019-10-13T11:07:12.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:12.705+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.710277+0000) 2019-10-13T11:07:12.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:12.707+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.711107+0000) 2019-10-13T11:07:12.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:12.707+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.712070+0000) 2019-10-13T11:07:12.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:12.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.712554+0000) 2019-10-13T11:07:12.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:12.707+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.713014+0000) 2019-10-13T11:07:12.714 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:12.707+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.713441+0000) 2019-10-13T11:07:12.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:12.711+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.714359+0000) 2019-10-13T11:07:12.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:12.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131592 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:12.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:12.713+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:12.717891+0000) 2019-10-13T11:07:14.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:14.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:14.112180+0000) 2019-10-13T11:07:14.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:14.395+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:14.398438+0000) 2019-10-13T11:07:17.480 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:07:17.480 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:17.484 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:07:17.484 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:17.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:17.715+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.720843+0000) 2019-10-13T11:07:17.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:17.719+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.721960+0000) 2019-10-13T11:07:17.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:17.719+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.722111+0000) 2019-10-13T11:07:17.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:17.719+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.723697+0000) 2019-10-13T11:07:17.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:17.719+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.724299+0000) 2019-10-13T11:07:17.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:17.719+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.723937+0000) 2019-10-13T11:07:17.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:17.720+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.724467+0000) 2019-10-13T11:07:17.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:17.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131621 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:17.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:17.724+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.730612+0000) 2019-10-13T11:07:17.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:17.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:17.770226+0000) 2019-10-13T11:07:18.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:18.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:18.540694+0000) 2019-10-13T11:07:21.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:21.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:21.246901+0000) 2019-10-13T11:07:21.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:21.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:21.268775+0000) 2019-10-13T11:07:21.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:21.676+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:21.680467+0000) 2019-10-13T11:07:21.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:21.788+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:21.791860+0000) 2019-10-13T11:07:22.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:22.727+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.729895+0000) 2019-10-13T11:07:22.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:22.727+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.730087+0000) 2019-10-13T11:07:22.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:22.727+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.730584+0000) 2019-10-13T11:07:22.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:22.727+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.731699+0000) 2019-10-13T11:07:22.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:22.727+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.732126+0000) 2019-10-13T11:07:22.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:22.731+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.734375+0000) 2019-10-13T11:07:22.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:22.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:22.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:22.732+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.736615+0000) 2019-10-13T11:07:22.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:22.732+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:22.739845+0000) 2019-10-13T11:07:24.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:24.107+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:24.112455+0000) 2019-10-13T11:07:24.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:24.395+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:24.398640+0000) 2019-10-13T11:07:27.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:27.735+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.739661+0000) 2019-10-13T11:07:27.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:27.735+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.739981+0000) 2019-10-13T11:07:27.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:27.735+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.741294+0000) 2019-10-13T11:07:27.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:27.739+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.741820+0000) 2019-10-13T11:07:27.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:27.736+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.741446+0000) 2019-10-13T11:07:27.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:27.739+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.742377+0000) 2019-10-13T11:07:27.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:27.739+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.744153+0000) 2019-10-13T11:07:27.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:27.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:27.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:27.744+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.751246+0000) 2019-10-13T11:07:27.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:27.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:27.770403+0000) 2019-10-13T11:07:28.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:28.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:28.540869+0000) 2019-10-13T11:07:31.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:31.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:31.247160+0000) 2019-10-13T11:07:31.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:31.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:31.269003+0000) 2019-10-13T11:07:31.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:31.684+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:31.686950+0000) 2019-10-13T11:07:31.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:31.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:31.792048+0000) 2019-10-13T11:07:32.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:32.747+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.750512+0000) 2019-10-13T11:07:32.752 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:32.747+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.750269+0000) 2019-10-13T11:07:32.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:32.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.750765+0000) 2019-10-13T11:07:32.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:32.747+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.751234+0000) 2019-10-13T11:07:32.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:32.747+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.752402+0000) 2019-10-13T11:07:32.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:32.747+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.753274+0000) 2019-10-13T11:07:32.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:32.751+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.753792+0000) 2019-10-13T11:07:32.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:32.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:32.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:32.752+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:32.758324+0000) 2019-10-13T11:07:34.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:34.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:34.112655+0000) 2019-10-13T11:07:34.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:34.395+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:34.398855+0000) 2019-10-13T11:07:37.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:37.755+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.759574+0000) 2019-10-13T11:07:37.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:37.755+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.759943+0000) 2019-10-13T11:07:37.761 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:37.755+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.760287+0000) 2019-10-13T11:07:37.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:37.755+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.760537+0000) 2019-10-13T11:07:37.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:37.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.760693+0000) 2019-10-13T11:07:37.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:37.759+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.762198+0000) 2019-10-13T11:07:37.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:37.759+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.763404+0000) 2019-10-13T11:07:37.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:37.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:37.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:37.764+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.769116+0000) 2019-10-13T11:07:37.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:37.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:37.770515+0000) 2019-10-13T11:07:38.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:38.535+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:38.541040+0000) 2019-10-13T11:07:41.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:41.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:41.247392+0000) 2019-10-13T11:07:41.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:41.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:41.269202+0000) 2019-10-13T11:07:41.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:41.692+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:41.693631+0000) 2019-10-13T11:07:41.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:41.788+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:41.792284+0000) 2019-10-13T11:07:42.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:42.767+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.770619+0000) 2019-10-13T11:07:42.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:42.767+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.770932+0000) 2019-10-13T11:07:42.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:42.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.771105+0000) 2019-10-13T11:07:42.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:42.767+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.772284+0000) 2019-10-13T11:07:42.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:42.767+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.772819+0000) 2019-10-13T11:07:42.774 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:42.767+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.773448+0000) 2019-10-13T11:07:42.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:42.771+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.773905+0000) 2019-10-13T11:07:42.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:42.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:42.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:42.772+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:42.778071+0000) 2019-10-13T11:07:44.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:44.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:44.112871+0000) 2019-10-13T11:07:44.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:44.395+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:44.399057+0000) 2019-10-13T11:07:47.580 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:07:47.580 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:47.583 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:07:47.584 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:07:47.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:47.767+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.770699+0000) 2019-10-13T11:07:47.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:47.775+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.780553+0000) 2019-10-13T11:07:47.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:47.775+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.781406+0000) 2019-10-13T11:07:47.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:47.779+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.782051+0000) 2019-10-13T11:07:47.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:47.779+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.783692+0000) 2019-10-13T11:07:47.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:47.783+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.785970+0000) 2019-10-13T11:07:47.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:47.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:47.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:47.783+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.788271+0000) 2019-10-13T11:07:47.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:47.784+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.788829+0000) 2019-10-13T11:07:47.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:47.788+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:47.793544+0000) 2019-10-13T11:07:48.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:48.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:48.541215+0000) 2019-10-13T11:07:51.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:51.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:51.247588+0000) 2019-10-13T11:07:51.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:51.263+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:51.269424+0000) 2019-10-13T11:07:51.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:51.696+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:51.700150+0000) 2019-10-13T11:07:51.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:51.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:51.792496+0000) 2019-10-13T11:07:52.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:52.787+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.790566+0000) 2019-10-13T11:07:52.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:52.787+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.791258+0000) 2019-10-13T11:07:52.792 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:52.787+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.791728+0000) 2019-10-13T11:07:52.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:52.787+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.792371+0000) 2019-10-13T11:07:52.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:52.787+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.792474+0000) 2019-10-13T11:07:52.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:52.787+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.793222+0000) 2019-10-13T11:07:52.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:52.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.793272+0000) 2019-10-13T11:07:52.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:52.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:52.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:52.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:52.800131+0000) 2019-10-13T11:07:54.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:54.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:54.113101+0000) 2019-10-13T11:07:54.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:54.398+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:54.399263+0000) 2019-10-13T11:07:57.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:57.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.770957+0000) 2019-10-13T11:07:57.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:07:57.794+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.799902+0000) 2019-10-13T11:07:57.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:07:57.794+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.800708+0000) 2019-10-13T11:07:57.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:57.794+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.801168+0000) 2019-10-13T11:07:57.802 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:07:57.798+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.801736+0000) 2019-10-13T11:07:57.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:07:57.798+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.803461+0000) 2019-10-13T11:07:57.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:07:57.802+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.805974+0000) 2019-10-13T11:07:57.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:07:57.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131867 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:07:57.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:07:57.800+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.807982+0000) 2019-10-13T11:07:57.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:07:57.804+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:57.810022+0000) 2019-10-13T11:07:58.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:07:58.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:07:58.541384+0000) 2019-10-13T11:08:01.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:01.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:01.247820+0000) 2019-10-13T11:08:01.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:01.262+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:01.269612+0000) 2019-10-13T11:08:01.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:01.708+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:01.711265+0000) 2019-10-13T11:08:01.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:01.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:01.792643+0000) 2019-10-13T11:08:02.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:02.806+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.811679+0000) 2019-10-13T11:08:02.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:02.806+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.812229+0000) 2019-10-13T11:08:02.813 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:02.806+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.812508+0000) 2019-10-13T11:08:02.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:02.806+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.813303+0000) 2019-10-13T11:08:02.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:02.808+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.813635+0000) 2019-10-13T11:08:02.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:02.810+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.814193+0000) 2019-10-13T11:08:02.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:02.810+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.814838+0000) 2019-10-13T11:08:02.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:02.812+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:02.817458+0000) 2019-10-13T11:08:02.818 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:02.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131899 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:04.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:04.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:04.113297+0000) 2019-10-13T11:08:04.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:04.398+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:04.399503+0000) 2019-10-13T11:08:07.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:07.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.771154+0000) 2019-10-13T11:08:07.823 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:07.818+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.821954+0000) 2019-10-13T11:08:07.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:07.818+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.822459+0000) 2019-10-13T11:08:07.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:07.818+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.822908+0000) 2019-10-13T11:08:07.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:07.816+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.822995+0000) 2019-10-13T11:08:07.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:07.818+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.824209+0000) 2019-10-13T11:08:07.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:07.818+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.824652+0000) 2019-10-13T11:08:07.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:07.818+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.825002+0000) 2019-10-13T11:08:07.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:07.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:07.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:07.824+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:07.829675+0000) 2019-10-13T11:08:08.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:08.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:08.541565+0000) 2019-10-13T11:08:11.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:11.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:11.248037+0000) 2019-10-13T11:08:11.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:11.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:11.269844+0000) 2019-10-13T11:08:11.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:11.716+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:11.718512+0000) 2019-10-13T11:08:11.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:11.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:11.792820+0000) 2019-10-13T11:08:12.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:12.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.831275+0000) 2019-10-13T11:08:12.833 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:12.826+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.832687+0000) 2019-10-13T11:08:12.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:12.826+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.833087+0000) 2019-10-13T11:08:12.834 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:12.826+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.833558+0000) 2019-10-13T11:08:12.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:12.830+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.834270+0000) 2019-10-13T11:08:12.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:12.830+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.834601+0000) 2019-10-13T11:08:12.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:12.830+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.834994+0000) 2019-10-13T11:08:12.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:12.834+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:12.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:12.840+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:12.841050+0000) 2019-10-13T11:08:14.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:14.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:14.113497+0000) 2019-10-13T11:08:14.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:14.394+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:14.399707+0000) 2019-10-13T11:08:17.651 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:08:17.651 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:17.655 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:08:17.655 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:17.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:17.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.771354+0000) 2019-10-13T11:08:17.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:17.838+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.842529+0000) 2019-10-13T11:08:17.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:17.838+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.842336+0000) 2019-10-13T11:08:17.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:17.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.843215+0000) 2019-10-13T11:08:17.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:17.838+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.843200+0000) 2019-10-13T11:08:17.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:17.838+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.843851+0000) 2019-10-13T11:08:17.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:17.838+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.844158+0000) 2019-10-13T11:08:17.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:17.838+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.845045+0000) 2019-10-13T11:08:17.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:17.842+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:17.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:17.848+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:17.849971+0000) 2019-10-13T11:08:18.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:18.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:18.541774+0000) 2019-10-13T11:08:21.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:21.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:21.248269+0000) 2019-10-13T11:08:21.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:21.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:21.270072+0000) 2019-10-13T11:08:21.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:21.723+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:21.725634+0000) 2019-10-13T11:08:21.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:21.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:21.793010+0000) 2019-10-13T11:08:22.853 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:22.846+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.852377+0000) 2019-10-13T11:08:22.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:22.846+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.852993+0000) 2019-10-13T11:08:22.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:22.850+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.854358+0000) 2019-10-13T11:08:22.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:22.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.854373+0000) 2019-10-13T11:08:22.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:22.851+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.854789+0000) 2019-10-13T11:08:22.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:22.850+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.855365+0000) 2019-10-13T11:08:22.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:22.850+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.855518+0000) 2019-10-13T11:08:22.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:22.854+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:22.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:22.859+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:22.860551+0000) 2019-10-13T11:08:24.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:24.106+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:24.113706+0000) 2019-10-13T11:08:24.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:24.398+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:24.399929+0000) 2019-10-13T11:08:27.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:27.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.771543+0000) 2019-10-13T11:08:27.864 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:27.858+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.862823+0000) 2019-10-13T11:08:27.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:27.858+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.862919+0000) 2019-10-13T11:08:27.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:27.858+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.863158+0000) 2019-10-13T11:08:27.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:27.859+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.863184+0000) 2019-10-13T11:08:27.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:27.858+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.864126+0000) 2019-10-13T11:08:27.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:27.858+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.865063+0000) 2019-10-13T11:08:27.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:27.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.865777+0000) 2019-10-13T11:08:27.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:27.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:27.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:27.867+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:27.872031+0000) 2019-10-13T11:08:28.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:28.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:28.541952+0000) 2019-10-13T11:08:31.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:31.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:31.248533+0000) 2019-10-13T11:08:31.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:31.266+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:31.270262+0000) 2019-10-13T11:08:31.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:31.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:31.732698+0000) 2019-10-13T11:08:31.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:31.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:31.793151+0000) 2019-10-13T11:08:32.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:32.866+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.871693+0000) 2019-10-13T11:08:32.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:32.866+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.872279+0000) 2019-10-13T11:08:32.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:32.871+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.873201+0000) 2019-10-13T11:08:32.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:32.866+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.873610+0000) 2019-10-13T11:08:32.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:32.866+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.873700+0000) 2019-10-13T11:08:32.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:32.870+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.874281+0000) 2019-10-13T11:08:32.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:32.870+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.874089+0000) 2019-10-13T11:08:32.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:32.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:32.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:32.875+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:32.879289+0000) 2019-10-13T11:08:34.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:34.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.113919+0000) 2019-10-13T11:08:34.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:34.398+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:34.400124+0000) 2019-10-13T11:08:37.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:37.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.771745+0000) 2019-10-13T11:08:37.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:37.879+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.881114+0000) 2019-10-13T11:08:37.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:37.878+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.881967+0000) 2019-10-13T11:08:37.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:37.878+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.882301+0000) 2019-10-13T11:08:37.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:37.878+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.882562+0000) 2019-10-13T11:08:37.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:37.878+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.883064+0000) 2019-10-13T11:08:37.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:37.878+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.883416+0000) 2019-10-13T11:08:37.886 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:37.878+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.885267+0000) 2019-10-13T11:08:37.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:37.882+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:37.889 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:37.887+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:37.888519+0000) 2019-10-13T11:08:38.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:38.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:38.542153+0000) 2019-10-13T11:08:41.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:41.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:41.248768+0000) 2019-10-13T11:08:41.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:41.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:41.270503+0000) 2019-10-13T11:08:41.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:41.743+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:41.744277+0000) 2019-10-13T11:08:41.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:41.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:41.793302+0000) 2019-10-13T11:08:42.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:42.886+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.891408+0000) 2019-10-13T11:08:42.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:42.886+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.892354+0000) 2019-10-13T11:08:42.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:42.891+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.892898+0000) 2019-10-13T11:08:42.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:42.886+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.893449+0000) 2019-10-13T11:08:42.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:42.890+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.893761+0000) 2019-10-13T11:08:42.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:42.890+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.894540+0000) 2019-10-13T11:08:42.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:42.890+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.894808+0000) 2019-10-13T11:08:42.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:42.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:42.897318+0000) 2019-10-13T11:08:42.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:42.894+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:44.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:44.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.114129+0000) 2019-10-13T11:08:44.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:44.398+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:44.400344+0000) 2019-10-13T11:08:47.765 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:08:47.765 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:47.769 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:08:47.769 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:08:47.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:47.766+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.771906+0000) 2019-10-13T11:08:47.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:47.894+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.901607+0000) 2019-10-13T11:08:47.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:47.898+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.902097+0000) 2019-10-13T11:08:47.903 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:47.898+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.902529+0000) 2019-10-13T11:08:47.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:47.898+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.903164+0000) 2019-10-13T11:08:47.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:47.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.903529+0000) 2019-10-13T11:08:47.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:47.903+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.904279+0000) 2019-10-13T11:08:47.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:47.898+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.904479+0000) 2019-10-13T11:08:47.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:47.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:47.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:47.907+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:47.909532+0000) 2019-10-13T11:08:48.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:48.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:48.542344+0000) 2019-10-13T11:08:51.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:51.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:51.248995+0000) 2019-10-13T11:08:51.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:51.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:51.270740+0000) 2019-10-13T11:08:51.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:51.747+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:51.750641+0000) 2019-10-13T11:08:51.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:51.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:51.793429+0000) 2019-10-13T11:08:52.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:52.905+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.912503+0000) 2019-10-13T11:08:52.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:52.905+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.912854+0000) 2019-10-13T11:08:52.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:52.905+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.912779+0000) 2019-10-13T11:08:52.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:52.905+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.913221+0000) 2019-10-13T11:08:52.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:52.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.913662+0000) 2019-10-13T11:08:52.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:52.909+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.914385+0000) 2019-10-13T11:08:52.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:52.909+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.915205+0000) 2019-10-13T11:08:52.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:52.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:52.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:52.915+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:52.919758+0000) 2019-10-13T11:08:54.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:54.109+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.114340+0000) 2019-10-13T11:08:54.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:54.393+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:54.400540+0000) 2019-10-13T11:08:57.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:57.765+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.772076+0000) 2019-10-13T11:08:57.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:08:57.917+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.923826+0000) 2019-10-13T11:08:57.925 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:08:57.917+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.924143+0000) 2019-10-13T11:08:57.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:08:57.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.924314+0000) 2019-10-13T11:08:57.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:08:57.917+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.924861+0000) 2019-10-13T11:08:57.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:08:57.921+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.926805+0000) 2019-10-13T11:08:57.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:08:57.921+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.927488+0000) 2019-10-13T11:08:57.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:08:57.921+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:08:57.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:57.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.928952+0000) 2019-10-13T11:08:57.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:08:57.931+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:57.933756+0000) 2019-10-13T11:08:58.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:08:58.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:08:58.542551+0000) 2019-10-13T11:09:01.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:01.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:01.249223+0000) 2019-10-13T11:09:01.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:01.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:01.270951+0000) 2019-10-13T11:09:01.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:01.759+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:01.762057+0000) 2019-10-13T11:09:01.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:01.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:01.793540+0000) 2019-10-13T11:09:02.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:02.925+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.931349+0000) 2019-10-13T11:09:02.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:02.925+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.932307+0000) 2019-10-13T11:09:02.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:02.925+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.932364+0000) 2019-10-13T11:09:02.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:02.925+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.932862+0000) 2019-10-13T11:09:02.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:02.929+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.933920+0000) 2019-10-13T11:09:02.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:02.931+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.933994+0000) 2019-10-13T11:09:02.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:02.929+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.935277+0000) 2019-10-13T11:09:02.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:02.935+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:02.936104+0000) 2019-10-13T11:09:02.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:02.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:04.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:04.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.114583+0000) 2019-10-13T11:09:04.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:04.397+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:04.400774+0000) 2019-10-13T11:09:07.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:07.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.772272+0000) 2019-10-13T11:09:07.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:07.937+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.941148+0000) 2019-10-13T11:09:07.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:07.937+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.941526+0000) 2019-10-13T11:09:07.943 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:07.941+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.941860+0000) 2019-10-13T11:09:07.944 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:07.941+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.942585+0000) 2019-10-13T11:09:07.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:07.939+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.942705+0000) 2019-10-13T11:09:07.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:07.941+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.943966+0000) 2019-10-13T11:09:07.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:07.941+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.944272+0000) 2019-10-13T11:09:07.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:07.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:07.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:07.947+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:07.950999+0000) 2019-10-13T11:09:08.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:08.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:08.542729+0000) 2019-10-13T11:09:11.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:11.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:11.249457+0000) 2019-10-13T11:09:11.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:11.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:11.271145+0000) 2019-10-13T11:09:11.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:11.771+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:11.773096+0000) 2019-10-13T11:09:11.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:11.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:11.793641+0000) 2019-10-13T11:09:12.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:12.947+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.949884+0000) 2019-10-13T11:09:12.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:12.949+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.950831+0000) 2019-10-13T11:09:12.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:12.949+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.952662+0000) 2019-10-13T11:09:12.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:12.953+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.953731+0000) 2019-10-13T11:09:12.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:12.949+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.953625+0000) 2019-10-13T11:09:12.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:12.953+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.955880+0000) 2019-10-13T11:09:12.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:12.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:12.958 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:12.957+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.957875+0000) 2019-10-13T11:09:12.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:12.959+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:12.960452+0000) 2019-10-13T11:09:14.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:14.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.114752+0000) 2019-10-13T11:09:14.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:14.397+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:14.400994+0000) 2019-10-13T11:09:17.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:17.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.772469+0000) 2019-10-13T11:09:17.879 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:09:17.880 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:17.883 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:09:17.883 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:17.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:17.957+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.961086+0000) 2019-10-13T11:09:17.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:17.961+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.964442+0000) 2019-10-13T11:09:17.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:17.961+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.965115+0000) 2019-10-13T11:09:17.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:17.965+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.966017+0000) 2019-10-13T11:09:17.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:17.963+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.965837+0000) 2019-10-13T11:09:17.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:17.965+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.967041+0000) 2019-10-13T11:09:17.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:17.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:17.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:17.965+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.967786+0000) 2019-10-13T11:09:17.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:17.967+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:17.969401+0000) 2019-10-13T11:09:18.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:18.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:18.542907+0000) 2019-10-13T11:09:21.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:21.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:21.249672+0000) 2019-10-13T11:09:21.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:21.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:21.271344+0000) 2019-10-13T11:09:21.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:21.783+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:21.784615+0000) 2019-10-13T11:09:21.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:21.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:21.793798+0000) 2019-10-13T11:09:22.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:22.969+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.971825+0000) 2019-10-13T11:09:22.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:22.969+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.972368+0000) 2019-10-13T11:09:22.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:22.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.973371+0000) 2019-10-13T11:09:22.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:22.973+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.973968+0000) 2019-10-13T11:09:22.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:22.973+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.974739+0000) 2019-10-13T11:09:22.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:22.973+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.975170+0000) 2019-10-13T11:09:22.976 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:22.973+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.975682+0000) 2019-10-13T11:09:22.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:22.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:22.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:22.979+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:22.980413+0000) 2019-10-13T11:09:24.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:24.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.115002+0000) 2019-10-13T11:09:24.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:24.397+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:24.401201+0000) 2019-10-13T11:09:27.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:27.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.772696+0000) 2019-10-13T11:09:27.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:27.978+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.980725+0000) 2019-10-13T11:09:27.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:27.977+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.981659+0000) 2019-10-13T11:09:27.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:27.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.981908+0000) 2019-10-13T11:09:27.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:27.981+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.982545+0000) 2019-10-13T11:09:27.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:27.981+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.984568+0000) 2019-10-13T11:09:27.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:27.985+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.987357+0000) 2019-10-13T11:09:27.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:27.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:27.989 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:27.985+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.988290+0000) 2019-10-13T11:09:27.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:27.990+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:27.993422+0000) 2019-10-13T11:09:28.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:28.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:28.543083+0000) 2019-10-13T11:09:31.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:31.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:31.249909+0000) 2019-10-13T11:09:31.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:31.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:31.271575+0000) 2019-10-13T11:09:31.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:31.786+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:31.791084+0000) 2019-10-13T11:09:31.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:31.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:31.793964+0000) 2019-10-13T11:09:32.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:32.989+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.991655+0000) 2019-10-13T11:09:32.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:32.989+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.992360+0000) 2019-10-13T11:09:32.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:32.989+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.992539+0000) 2019-10-13T11:09:32.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:32.989+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.992991+0000) 2019-10-13T11:09:32.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:32.990+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.993599+0000) 2019-10-13T11:09:32.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:32.993+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.993778+0000) 2019-10-13T11:09:32.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:32.993+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:32.994166+0000) 2019-10-13T11:09:32.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:32.997+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:33.003 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:32.998+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:33.002684+0000) 2019-10-13T11:09:34.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:34.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.115242+0000) 2019-10-13T11:09:34.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:34.397+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:34.401387+0000) 2019-10-13T11:09:37.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:37.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:37.772850+0000) 2019-10-13T11:09:38.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:37.997+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.001081+0000) 2019-10-13T11:09:38.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:38.001+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.002564+0000) 2019-10-13T11:09:38.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:38.001+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.002814+0000) 2019-10-13T11:09:38.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:38.001+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.003298+0000) 2019-10-13T11:09:38.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:37.998+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.003759+0000) 2019-10-13T11:09:38.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:38.001+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.004379+0000) 2019-10-13T11:09:38.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:38.001+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.004652+0000) 2019-10-13T11:09:38.008 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:38.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:38.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:38.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.011904+0000) 2019-10-13T11:09:38.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:38.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:38.543256+0000) 2019-10-13T11:09:41.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:41.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:41.250126+0000) 2019-10-13T11:09:41.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:41.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:41.271774+0000) 2019-10-13T11:09:41.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:41.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:41.794185+0000) 2019-10-13T11:09:41.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:41.794+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:41.797731+0000) 2019-10-13T11:09:43.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:43.009+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.013567+0000) 2019-10-13T11:09:43.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:43.010+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.013553+0000) 2019-10-13T11:09:43.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:43.013+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.014299+0000) 2019-10-13T11:09:43.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:43.013+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.014937+0000) 2019-10-13T11:09:43.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:43.013+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.015093+0000) 2019-10-13T11:09:43.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:43.013+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.015482+0000) 2019-10-13T11:09:43.017 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:43.013+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.016071+0000) 2019-10-13T11:09:43.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:43.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:43.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:43.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:43.020590+0000) 2019-10-13T11:09:44.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:44.113+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.115453+0000) 2019-10-13T11:09:44.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:44.401+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:44.401589+0000) 2019-10-13T11:09:47.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:47.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:47.773074+0000) 2019-10-13T11:09:48.021 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:48.016+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.021005+0000) 2019-10-13T11:09:48.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:48.016+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.021526+0000) 2019-10-13T11:09:48.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:48.018+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.022388+0000) 2019-10-13T11:09:48.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:48.020+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.024210+0000) 2019-10-13T11:09:48.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:48.020+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.024455+0000) 2019-10-13T11:09:48.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:48.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.025907+0000) 2019-10-13T11:09:48.027 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:48.024+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.026189+0000) 2019-10-13T11:09:48.027 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:09:48.028 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:48.028 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:48.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:48.031 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:09:48.031 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:09:48.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:48.030+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.032888+0000) 2019-10-13T11:09:48.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:48.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:48.543452+0000) 2019-10-13T11:09:51.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:51.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:51.250366+0000) 2019-10-13T11:09:51.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:51.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:51.271973+0000) 2019-10-13T11:09:51.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:51.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:51.794372+0000) 2019-10-13T11:09:51.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:51.802+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:51.804163+0000) 2019-10-13T11:09:53.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:53.028+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.030732+0000) 2019-10-13T11:09:53.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:53.026+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.030835+0000) 2019-10-13T11:09:53.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:53.028+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.031313+0000) 2019-10-13T11:09:53.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:53.028+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.031727+0000) 2019-10-13T11:09:53.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:53.028+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.032101+0000) 2019-10-13T11:09:53.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:53.028+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.032576+0000) 2019-10-13T11:09:53.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:53.028+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.032772+0000) 2019-10-13T11:09:53.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:53.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:53.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:53.038+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:53.042409+0000) 2019-10-13T11:09:54.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:54.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.115662+0000) 2019-10-13T11:09:54.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:54.400+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:54.401809+0000) 2019-10-13T11:09:57.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:57.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:57.773269+0000) 2019-10-13T11:09:58.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:09:58.036+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.040636+0000) 2019-10-13T11:09:58.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:09:58.036+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.041295+0000) 2019-10-13T11:09:58.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:09:58.040+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.042241+0000) 2019-10-13T11:09:58.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:58.040+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.043804+0000) 2019-10-13T11:09:58.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:09:58.044+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.046087+0000) 2019-10-13T11:09:58.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:09:58.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:09:58.049 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:09:58.046+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.048260+0000) 2019-10-13T11:09:58.050 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:09:58.044+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.049321+0000) 2019-10-13T11:09:58.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:09:58.050+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.054316+0000) 2019-10-13T11:09:58.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:09:58.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:09:58.543617+0000) 2019-10-13T11:10:01.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:01.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:01.250592+0000) 2019-10-13T11:10:01.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:01.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:01.272203+0000) 2019-10-13T11:10:01.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:01.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:01.794529+0000) 2019-10-13T11:10:01.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:01.810+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:01.810748+0000) 2019-10-13T11:10:03.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:03.048+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.051148+0000) 2019-10-13T11:10:03.053 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:03.048+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.052196+0000) 2019-10-13T11:10:03.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:03.048+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.052493+0000) 2019-10-13T11:10:03.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:03.050+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.052729+0000) 2019-10-13T11:10:03.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:03.048+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.053093+0000) 2019-10-13T11:10:03.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:03.052+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.053776+0000) 2019-10-13T11:10:03.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:03.052+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.053882+0000) 2019-10-13T11:10:03.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:03.054+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:03.058025+0000) 2019-10-13T11:10:03.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:03.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:04.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:04.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.115823+0000) 2019-10-13T11:10:04.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:04.400+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:04.402031+0000) 2019-10-13T11:10:07.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:07.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:07.773469+0000) 2019-10-13T11:10:08.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:08.056+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.061328+0000) 2019-10-13T11:10:08.064 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:08.060+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.063010+0000) 2019-10-13T11:10:08.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:08.058+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.063471+0000) 2019-10-13T11:10:08.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:08.060+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.063294+0000) 2019-10-13T11:10:08.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:08.060+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.064206+0000) 2019-10-13T11:10:08.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:08.060+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.064507+0000) 2019-10-13T11:10:08.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:08.060+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.064898+0000) 2019-10-13T11:10:08.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:08.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:08.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:08.066+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.068967+0000) 2019-10-13T11:10:08.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:08.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:08.543797+0000) 2019-10-13T11:10:11.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:11.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:11.250814+0000) 2019-10-13T11:10:11.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:11.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:11.272401+0000) 2019-10-13T11:10:11.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:11.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:11.794771+0000) 2019-10-13T11:10:11.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:11.814+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:11.817261+0000) 2019-10-13T11:10:13.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:13.068+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.070611+0000) 2019-10-13T11:10:13.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:13.066+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.071938+0000) 2019-10-13T11:10:13.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:13.072+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.074253+0000) 2019-10-13T11:10:13.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:13.072+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.074438+0000) 2019-10-13T11:10:13.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:13.072+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.075903+0000) 2019-10-13T11:10:13.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:13.072+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.076187+0000) 2019-10-13T11:10:13.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:13.072+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.077047+0000) 2019-10-13T11:10:13.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:13.076+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:13.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:13.082+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:13.084768+0000) 2019-10-13T11:10:14.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:14.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.116039+0000) 2019-10-13T11:10:14.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:14.400+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:14.402275+0000) 2019-10-13T11:10:17.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:17.768+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:17.773686+0000) 2019-10-13T11:10:18.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:18.076+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.081682+0000) 2019-10-13T11:10:18.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:18.080+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.082409+0000) 2019-10-13T11:10:18.084 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:18.080+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.084117+0000) 2019-10-13T11:10:18.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:18.080+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.084579+0000) 2019-10-13T11:10:18.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:18.080+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.084825+0000) 2019-10-13T11:10:18.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:18.080+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.085408+0000) 2019-10-13T11:10:18.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:18.082+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.087508+0000) 2019-10-13T11:10:18.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:18.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:18.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:18.090+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.095855+0000) 2019-10-13T11:10:18.158 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:10:18.158 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:18.162 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:10:18.162 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:18.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:18.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:18.543968+0000) 2019-10-13T11:10:21.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:21.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:21.251009+0000) 2019-10-13T11:10:21.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:21.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:21.272595+0000) 2019-10-13T11:10:21.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:21.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:21.794981+0000) 2019-10-13T11:10:21.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:21.818+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:21.823773+0000) 2019-10-13T11:10:23.093 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:23.088+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.092494+0000) 2019-10-13T11:10:23.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:23.088+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.092923+0000) 2019-10-13T11:10:23.095 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:23.090+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.093996+0000) 2019-10-13T11:10:23.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:23.092+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.094580+0000) 2019-10-13T11:10:23.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:23.092+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.094782+0000) 2019-10-13T11:10:23.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:23.092+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.094804+0000) 2019-10-13T11:10:23.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:23.092+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.095912+0000) 2019-10-13T11:10:23.099 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:23.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:23.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:23.098+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:23.101549+0000) 2019-10-13T11:10:24.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:24.112+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.116205+0000) 2019-10-13T11:10:24.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:24.400+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:24.402504+0000) 2019-10-13T11:10:27.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:27.772+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:27.774003+0000) 2019-10-13T11:10:28.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:28.100+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.102358+0000) 2019-10-13T11:10:28.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:28.100+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.103004+0000) 2019-10-13T11:10:28.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:28.100+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.103924+0000) 2019-10-13T11:10:28.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:28.100+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.104219+0000) 2019-10-13T11:10:28.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:28.100+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.105163+0000) 2019-10-13T11:10:28.106 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:28.102+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.105285+0000) 2019-10-13T11:10:28.106 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:28.104+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.105817+0000) 2019-10-13T11:10:28.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:28.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:28.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:28.110+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.112398+0000) 2019-10-13T11:10:28.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:28.540+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:28.544155+0000) 2019-10-13T11:10:31.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:31.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:31.251212+0000) 2019-10-13T11:10:31.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:31.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:31.272790+0000) 2019-10-13T11:10:31.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:31.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:31.795210+0000) 2019-10-13T11:10:31.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:31.825+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:31.830223+0000) 2019-10-13T11:10:33.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:33.108+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.111304+0000) 2019-10-13T11:10:33.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:33.108+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.112112+0000) 2019-10-13T11:10:33.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:33.109+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.113126+0000) 2019-10-13T11:10:33.114 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:33.108+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.113611+0000) 2019-10-13T11:10:33.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:33.112+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.114178+0000) 2019-10-13T11:10:33.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:33.112+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.114463+0000) 2019-10-13T11:10:33.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:33.112+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.115179+0000) 2019-10-13T11:10:33.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:33.112+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132818 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:33.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:33.113+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:33.119457+0000) 2019-10-13T11:10:34.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:34.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.116405+0000) 2019-10-13T11:10:34.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:34.400+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:34.402741+0000) 2019-10-13T11:10:37.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:37.772+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:37.774201+0000) 2019-10-13T11:10:38.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:38.116+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.119441+0000) 2019-10-13T11:10:38.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:38.113+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.119807+0000) 2019-10-13T11:10:38.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:38.116+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.120505+0000) 2019-10-13T11:10:38.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:38.116+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.120910+0000) 2019-10-13T11:10:38.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:38.120+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.122958+0000) 2019-10-13T11:10:38.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:38.120+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.123081+0000) 2019-10-13T11:10:38.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:38.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.124052+0000) 2019-10-13T11:10:38.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:38.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132848 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:38.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:38.125+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.130218+0000) 2019-10-13T11:10:38.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:38.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:38.544358+0000) 2019-10-13T11:10:41.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:41.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:41.251416+0000) 2019-10-13T11:10:41.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:41.268+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:41.272988+0000) 2019-10-13T11:10:41.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:41.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:41.795410+0000) 2019-10-13T11:10:41.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:41.833+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:41.839688+0000) 2019-10-13T11:10:43.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:43.124+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.129349+0000) 2019-10-13T11:10:43.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:43.128+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.130173+0000) 2019-10-13T11:10:43.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:43.128+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.130544+0000) 2019-10-13T11:10:43.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:43.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.130992+0000) 2019-10-13T11:10:43.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:43.128+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.131645+0000) 2019-10-13T11:10:43.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:43.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.132149+0000) 2019-10-13T11:10:43.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:43.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.131968+0000) 2019-10-13T11:10:43.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:43.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:43.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:43.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:43.141313+0000) 2019-10-13T11:10:44.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:44.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.116601+0000) 2019-10-13T11:10:44.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:44.399+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:44.402976+0000) 2019-10-13T11:10:47.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:47.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:47.774419+0000) 2019-10-13T11:10:48.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:48.135+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.139327+0000) 2019-10-13T11:10:48.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:48.135+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.139892+0000) 2019-10-13T11:10:48.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:48.135+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.140239+0000) 2019-10-13T11:10:48.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:48.135+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.140487+0000) 2019-10-13T11:10:48.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:48.135+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.140939+0000) 2019-10-13T11:10:48.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:48.139+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.143467+0000) 2019-10-13T11:10:48.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:48.137+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.143641+0000) 2019-10-13T11:10:48.147 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:48.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:48.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:48.145+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.148874+0000) 2019-10-13T11:10:48.278 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:10:48.279 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:48.282 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:10:48.282 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:10:48.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:48.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:48.544536+0000) 2019-10-13T11:10:51.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:51.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:51.251649+0000) 2019-10-13T11:10:51.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:51.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:51.273204+0000) 2019-10-13T11:10:51.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:51.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:51.795606+0000) 2019-10-13T11:10:51.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:51.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:51.846305+0000) 2019-10-13T11:10:53.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:53.145+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.151133+0000) 2019-10-13T11:10:53.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:53.147+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.150442+0000) 2019-10-13T11:10:53.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:53.147+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.150965+0000) 2019-10-13T11:10:53.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:53.147+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.151845+0000) 2019-10-13T11:10:53.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:53.147+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.152195+0000) 2019-10-13T11:10:53.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:53.147+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.152486+0000) 2019-10-13T11:10:53.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:53.151+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.154587+0000) 2019-10-13T11:10:53.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:53.151+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:53.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:53.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:53.160012+0000) 2019-10-13T11:10:54.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:54.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.116777+0000) 2019-10-13T11:10:54.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:54.399+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:54.403179+0000) 2019-10-13T11:10:57.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:57.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:57.774646+0000) 2019-10-13T11:10:58.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:58.155+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.160948+0000) 2019-10-13T11:10:58.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:10:58.159+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.161826+0000) 2019-10-13T11:10:58.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:10:58.159+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.162573+0000) 2019-10-13T11:10:58.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:10:58.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.162748+0000) 2019-10-13T11:10:58.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:10:58.159+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.163307+0000) 2019-10-13T11:10:58.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:10:58.159+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.163932+0000) 2019-10-13T11:10:58.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:10:58.159+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.163939+0000) 2019-10-13T11:10:58.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:10:58.163+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:10:58.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:10:58.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.175428+0000) 2019-10-13T11:10:58.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:10:58.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:10:58.544710+0000) 2019-10-13T11:11:01.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:01.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:01.251849+0000) 2019-10-13T11:11:01.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:01.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:01.273433+0000) 2019-10-13T11:11:01.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:01.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:01.795801+0000) 2019-10-13T11:11:01.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:01.849+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:01.852864+0000) 2019-10-13T11:11:03.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:03.167+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.171019+0000) 2019-10-13T11:11:03.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:03.167+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.171599+0000) 2019-10-13T11:11:03.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:03.167+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.172305+0000) 2019-10-13T11:11:03.173 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:03.169+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.172219+0000) 2019-10-13T11:11:03.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:03.167+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.172752+0000) 2019-10-13T11:11:03.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:03.167+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.173654+0000) 2019-10-13T11:11:03.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:03.171+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.174462+0000) 2019-10-13T11:11:03.177 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:03.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:03.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:03.173+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:03.179156+0000) 2019-10-13T11:11:04.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:04.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.116954+0000) 2019-10-13T11:11:04.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:04.399+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:04.403426+0000) 2019-10-13T11:11:07.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:07.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:07.774870+0000) 2019-10-13T11:11:08.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:08.175+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.180112+0000) 2019-10-13T11:11:08.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:08.175+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.181035+0000) 2019-10-13T11:11:08.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:08.177+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.181599+0000) 2019-10-13T11:11:08.184 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:08.179+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.183619+0000) 2019-10-13T11:11:08.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:08.183+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.185839+0000) 2019-10-13T11:11:08.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:08.183+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.186140+0000) 2019-10-13T11:11:08.187 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:08.183+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.186381+0000) 2019-10-13T11:11:08.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:08.183+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:08.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:08.189+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.192262+0000) 2019-10-13T11:11:08.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:08.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:08.544903+0000) 2019-10-13T11:11:11.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:11.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:11.252049+0000) 2019-10-13T11:11:11.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:11.267+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:11.273619+0000) 2019-10-13T11:11:11.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:11.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:11.796025+0000) 2019-10-13T11:11:11.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:11.853+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:11.859201+0000) 2019-10-13T11:11:13.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:13.187+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.191007+0000) 2019-10-13T11:11:13.192 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:13.187+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.191410+0000) 2019-10-13T11:11:13.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:13.189+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.192797+0000) 2019-10-13T11:11:13.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:13.187+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.193541+0000) 2019-10-13T11:11:13.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:13.191+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.195236+0000) 2019-10-13T11:11:13.197 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:13.191+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.195456+0000) 2019-10-13T11:11:13.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:13.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:13.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:13.191+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.197677+0000) 2019-10-13T11:11:13.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:13.201+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:13.204171+0000) 2019-10-13T11:11:14.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:14.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.117137+0000) 2019-10-13T11:11:14.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:14.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:14.403640+0000) 2019-10-13T11:11:17.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:17.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:17.775066+0000) 2019-10-13T11:11:18.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:18.195+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.199894+0000) 2019-10-13T11:11:18.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:18.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.200507+0000) 2019-10-13T11:11:18.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:18.195+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.201045+0000) 2019-10-13T11:11:18.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:18.195+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.201311+0000) 2019-10-13T11:11:18.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:18.199+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.202391+0000) 2019-10-13T11:11:18.203 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:18.199+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.203023+0000) 2019-10-13T11:11:18.204 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:18.199+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.203642+0000) 2019-10-13T11:11:18.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:18.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:18.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:18.205+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.210886+0000) 2019-10-13T11:11:18.392 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:11:18.392 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:18.396 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:11:18.396 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:18.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:18.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:18.545087+0000) 2019-10-13T11:11:21.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:21.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:21.252272+0000) 2019-10-13T11:11:21.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:21.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:21.273764+0000) 2019-10-13T11:11:21.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:21.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:21.796262+0000) 2019-10-13T11:11:21.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:21.865+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:21.865797+0000) 2019-10-13T11:11:23.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:23.203+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.209561+0000) 2019-10-13T11:11:23.211 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:23.207+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.210428+0000) 2019-10-13T11:11:23.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:23.207+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.211512+0000) 2019-10-13T11:11:23.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:23.207+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.211554+0000) 2019-10-13T11:11:23.213 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:23.205+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.211634+0000) 2019-10-13T11:11:23.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:23.207+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.212451+0000) 2019-10-13T11:11:23.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:23.207+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.212871+0000) 2019-10-13T11:11:23.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:23.211+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:23.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:23.213+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:23.218190+0000) 2019-10-13T11:11:24.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:24.111+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.117317+0000) 2019-10-13T11:11:24.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:24.399+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:24.403847+0000) 2019-10-13T11:11:27.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:27.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:27.775219+0000) 2019-10-13T11:11:28.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:28.215+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.220191+0000) 2019-10-13T11:11:28.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:28.215+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.220898+0000) 2019-10-13T11:11:28.222 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:28.215+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.221451+0000) 2019-10-13T11:11:28.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:28.219+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.225300+0000) 2019-10-13T11:11:28.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:28.223+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.227545+0000) 2019-10-13T11:11:28.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:28.223+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:28.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:28.223+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.228269+0000) 2019-10-13T11:11:28.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:28.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.229103+0000) 2019-10-13T11:11:28.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:28.225+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.229717+0000) 2019-10-13T11:11:28.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:28.539+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:28.545264+0000) 2019-10-13T11:11:31.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:31.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:31.252465+0000) 2019-10-13T11:11:31.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:31.271+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:31.273935+0000) 2019-10-13T11:11:31.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:31.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:31.796474+0000) 2019-10-13T11:11:31.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:31.877+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:31.877293+0000) 2019-10-13T11:11:33.232 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:33.227+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.231423+0000) 2019-10-13T11:11:33.233 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:33.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.232298+0000) 2019-10-13T11:11:33.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:33.231+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.234573+0000) 2019-10-13T11:11:33.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:33.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.234568+0000) 2019-10-13T11:11:33.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:33.231+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.236089+0000) 2019-10-13T11:11:33.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:33.235+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.238300+0000) 2019-10-13T11:11:33.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:33.235+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:33.240 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:33.235+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.239265+0000) 2019-10-13T11:11:33.247 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:33.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:33.245930+0000) 2019-10-13T11:11:34.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:34.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.117500+0000) 2019-10-13T11:11:34.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:34.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:34.404079+0000) 2019-10-13T11:11:37.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:37.771+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:37.775405+0000) 2019-10-13T11:11:38.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:38.239+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.242002+0000) 2019-10-13T11:11:38.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:38.239+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.242813+0000) 2019-10-13T11:11:38.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:38.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.245028+0000) 2019-10-13T11:11:38.247 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:38.239+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.245250+0000) 2019-10-13T11:11:38.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:38.243+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.245799+0000) 2019-10-13T11:11:38.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:38.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.246253+0000) 2019-10-13T11:11:38.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:38.240+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.246781+0000) 2019-10-13T11:11:38.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:38.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:38.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:38.248+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.253099+0000) 2019-10-13T11:11:38.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:38.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:38.545456+0000) 2019-10-13T11:11:41.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:41.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:41.252678+0000) 2019-10-13T11:11:41.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:41.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:41.274146+0000) 2019-10-13T11:11:41.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:41.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:41.796639+0000) 2019-10-13T11:11:41.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:41.880+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:41.883829+0000) 2019-10-13T11:11:43.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:43.246+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.251406+0000) 2019-10-13T11:11:43.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:43.246+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.252144+0000) 2019-10-13T11:11:43.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:43.246+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.252811+0000) 2019-10-13T11:11:43.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:43.246+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.253572+0000) 2019-10-13T11:11:43.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:43.250+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.253831+0000) 2019-10-13T11:11:43.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:43.250+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.254076+0000) 2019-10-13T11:11:43.255 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:43.248+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.254583+0000) 2019-10-13T11:11:43.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:43.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:43.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:43.256+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:43.261088+0000) 2019-10-13T11:11:44.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:44.110+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.117675+0000) 2019-10-13T11:11:44.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:44.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:44.404298+0000) 2019-10-13T11:11:47.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:47.770+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:47.775610+0000) 2019-10-13T11:11:48.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:48.258+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.262145+0000) 2019-10-13T11:11:48.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:48.258+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.262526+0000) 2019-10-13T11:11:48.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:48.258+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.264166+0000) 2019-10-13T11:11:48.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:48.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.264712+0000) 2019-10-13T11:11:48.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:48.262+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.265899+0000) 2019-10-13T11:11:48.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:48.262+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.266818+0000) 2019-10-13T11:11:48.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:48.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:48.269 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:48.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.268289+0000) 2019-10-13T11:11:48.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:48.268+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.273420+0000) 2019-10-13T11:11:48.510 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:11:48.510 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:48.513 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:11:48.514 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:11:48.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:48.538+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:48.545677+0000) 2019-10-13T11:11:51.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:51.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:51.252877+0000) 2019-10-13T11:11:51.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:51.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:51.274386+0000) 2019-10-13T11:11:51.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:51.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:51.796816+0000) 2019-10-13T11:11:51.891 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:51.888+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:51.890347+0000) 2019-10-13T11:11:53.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:53.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.271654+0000) 2019-10-13T11:11:53.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:53.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.272208+0000) 2019-10-13T11:11:53.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:53.266+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.273535+0000) 2019-10-13T11:11:53.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:53.266+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.272691+0000) 2019-10-13T11:11:53.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:53.270+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.273962+0000) 2019-10-13T11:11:53.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:53.270+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.274331+0000) 2019-10-13T11:11:53.279 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:53.270+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.275076+0000) 2019-10-13T11:11:53.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:53.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133314 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:53.280 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:53.272+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:53.278879+0000) 2019-10-13T11:11:54.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:54.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.117893+0000) 2019-10-13T11:11:54.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:54.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:54.404505+0000) 2019-10-13T11:11:57.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:57.770+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:57.775815+0000) 2019-10-13T11:11:58.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:11:58.276+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.281075+0000) 2019-10-13T11:11:58.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:11:58.278+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.281824+0000) 2019-10-13T11:11:58.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:11:58.278+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.282224+0000) 2019-10-13T11:11:58.283 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:11:58.278+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.282842+0000) 2019-10-13T11:11:58.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:11:58.278+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.283178+0000) 2019-10-13T11:11:58.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:11:58.278+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.284238+0000) 2019-10-13T11:11:58.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:58.278+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.284670+0000) 2019-10-13T11:11:58.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:11:58.282+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:11:58.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:11:58.284+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.289675+0000) 2019-10-13T11:11:58.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:11:58.542+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:11:58.545873+0000) 2019-10-13T11:12:01.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:01.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:01.253079+0000) 2019-10-13T11:12:01.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:01.270+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:01.274547+0000) 2019-10-13T11:12:01.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:01.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:01.797029+0000) 2019-10-13T11:12:01.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:01.900+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:01.902030+0000) 2019-10-13T11:12:03.292 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:03.284+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.291089+0000) 2019-10-13T11:12:03.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:03.286+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.291896+0000) 2019-10-13T11:12:03.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:03.286+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.292613+0000) 2019-10-13T11:12:03.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:03.290+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.294273+0000) 2019-10-13T11:12:03.296 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:03.290+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.295831+0000) 2019-10-13T11:12:03.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:03.290+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.296033+0000) 2019-10-13T11:12:03.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:03.290+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:03.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:03.294+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.297920+0000) 2019-10-13T11:12:03.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:03.296+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:03.300614+0000) 2019-10-13T11:12:04.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:04.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.118114+0000) 2019-10-13T11:12:04.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:04.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:04.404740+0000) 2019-10-13T11:12:07.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:07.770+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:07.776012+0000) 2019-10-13T11:12:08.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:08.294+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.301162+0000) 2019-10-13T11:12:08.303 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:08.296+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.301356+0000) 2019-10-13T11:12:08.303 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:08.298+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.301865+0000) 2019-10-13T11:12:08.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:08.298+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.302189+0000) 2019-10-13T11:12:08.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:08.298+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.302671+0000) 2019-10-13T11:12:08.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:08.298+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.303752+0000) 2019-10-13T11:12:08.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:08.298+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.303636+0000) 2019-10-13T11:12:08.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:08.302+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:08.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:08.304+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.310311+0000) 2019-10-13T11:12:08.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:08.542+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:08.546048+0000) 2019-10-13T11:12:11.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:11.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:11.253279+0000) 2019-10-13T11:12:11.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:11.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:11.274779+0000) 2019-10-13T11:12:11.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:11.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:11.797202+0000) 2019-10-13T11:12:11.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:11.904+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:11.908499+0000) 2019-10-13T11:12:13.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:13.306+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.311150+0000) 2019-10-13T11:12:13.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:13.306+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.311774+0000) 2019-10-13T11:12:13.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:13.306+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.312313+0000) 2019-10-13T11:12:13.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:13.308+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.314325+0000) 2019-10-13T11:12:13.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:13.310+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.316620+0000) 2019-10-13T11:12:13.318 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:13.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:13.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:13.314+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.318304+0000) 2019-10-13T11:12:13.319 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:13.314+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.318754+0000) 2019-10-13T11:12:13.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:13.324+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:13.329338+0000) 2019-10-13T11:12:14.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:14.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.118299+0000) 2019-10-13T11:12:14.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:14.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:14.404995+0000) 2019-10-13T11:12:17.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:17.770+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:17.776210+0000) 2019-10-13T11:12:18.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:18.314+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.320754+0000) 2019-10-13T11:12:18.322 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:18.314+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.321546+0000) 2019-10-13T11:12:18.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:18.318+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.322069+0000) 2019-10-13T11:12:18.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:18.318+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.323325+0000) 2019-10-13T11:12:18.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:18.318+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.324360+0000) 2019-10-13T11:12:18.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:18.320+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.324530+0000) 2019-10-13T11:12:18.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:18.318+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.324968+0000) 2019-10-13T11:12:18.328 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:18.322+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:18.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:18.324+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.331486+0000) 2019-10-13T11:12:18.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:18.542+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:18.546225+0000) 2019-10-13T11:12:18.616 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:12:18.616 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:18.619 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:12:18.620 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:21.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:21.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:21.253498+0000) 2019-10-13T11:12:21.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:21.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:21.274976+0000) 2019-10-13T11:12:21.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:21.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:21.797383+0000) 2019-10-13T11:12:21.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:21.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:21.915011+0000) 2019-10-13T11:12:23.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:23.326+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.331202+0000) 2019-10-13T11:12:23.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:23.326+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.331982+0000) 2019-10-13T11:12:23.333 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:23.326+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.332659+0000) 2019-10-13T11:12:23.334 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:23.326+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.333393+0000) 2019-10-13T11:12:23.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:23.330+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.333964+0000) 2019-10-13T11:12:23.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:23.330+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.334031+0000) 2019-10-13T11:12:23.335 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:23.328+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.334636+0000) 2019-10-13T11:12:23.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:23.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:23.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:23.332+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:23.338003+0000) 2019-10-13T11:12:24.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:24.114+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.118508+0000) 2019-10-13T11:12:24.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:24.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:24.405198+0000) 2019-10-13T11:12:27.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:27.770+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:27.776413+0000) 2019-10-13T11:12:28.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:28.334+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.340678+0000) 2019-10-13T11:12:28.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:28.336+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.340770+0000) 2019-10-13T11:12:28.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:28.334+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.341410+0000) 2019-10-13T11:12:28.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:28.338+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.343773+0000) 2019-10-13T11:12:28.347 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:28.338+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.344011+0000) 2019-10-13T11:12:28.347 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:28.338+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.345511+0000) 2019-10-13T11:12:28.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:28.342+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.345895+0000) 2019-10-13T11:12:28.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:28.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133528 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:28.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:28.348+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.353498+0000) 2019-10-13T11:12:28.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:28.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:28.546429+0000) 2019-10-13T11:12:31.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:31.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:31.253698+0000) 2019-10-13T11:12:31.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:31.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:31.275206+0000) 2019-10-13T11:12:31.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:31.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:31.797554+0000) 2019-10-13T11:12:31.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:31.916+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:31.921587+0000) 2019-10-13T11:12:33.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:33.346+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.350954+0000) 2019-10-13T11:12:33.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:33.346+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.351670+0000) 2019-10-13T11:12:33.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:33.346+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.353543+0000) 2019-10-13T11:12:33.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:33.350+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.353844+0000) 2019-10-13T11:12:33.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:33.348+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.354186+0000) 2019-10-13T11:12:33.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:33.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.355462+0000) 2019-10-13T11:12:33.356 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:33.350+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.355992+0000) 2019-10-13T11:12:33.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:33.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:33.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:33.356+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:33.361704+0000) 2019-10-13T11:12:34.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:34.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.118692+0000) 2019-10-13T11:12:34.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:34.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:34.405415+0000) 2019-10-13T11:12:37.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:37.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:37.776613+0000) 2019-10-13T11:12:38.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:38.357+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.361509+0000) 2019-10-13T11:12:38.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:38.357+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.362083+0000) 2019-10-13T11:12:38.363 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:38.357+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.362580+0000) 2019-10-13T11:12:38.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:38.357+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.363071+0000) 2019-10-13T11:12:38.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:38.360+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.363074+0000) 2019-10-13T11:12:38.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:38.357+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.363633+0000) 2019-10-13T11:12:38.365 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:38.357+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.364767+0000) 2019-10-13T11:12:38.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:38.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:38.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:38.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.371803+0000) 2019-10-13T11:12:38.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:38.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:38.546602+0000) 2019-10-13T11:12:41.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:41.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:41.253953+0000) 2019-10-13T11:12:41.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:41.269+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:41.275404+0000) 2019-10-13T11:12:41.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:41.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:41.797726+0000) 2019-10-13T11:12:41.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:41.924+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:41.927981+0000) 2019-10-13T11:12:43.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:43.365+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.371132+0000) 2019-10-13T11:12:43.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:43.365+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.371291+0000) 2019-10-13T11:12:43.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:43.365+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.373014+0000) 2019-10-13T11:12:43.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:43.365+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.373330+0000) 2019-10-13T11:12:43.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:43.369+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.374271+0000) 2019-10-13T11:12:43.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:43.369+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.374665+0000) 2019-10-13T11:12:43.375 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:43.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.374708+0000) 2019-10-13T11:12:43.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:43.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:43.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:43.380+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:43.380103+0000) 2019-10-13T11:12:44.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:44.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.118880+0000) 2019-10-13T11:12:44.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:44.401+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:44.405653+0000) 2019-10-13T11:12:47.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:47.769+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:47.776814+0000) 2019-10-13T11:12:48.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:48.377+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.382234+0000) 2019-10-13T11:12:48.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:48.377+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.382902+0000) 2019-10-13T11:12:48.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:48.379+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.383078+0000) 2019-10-13T11:12:48.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:48.377+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.383614+0000) 2019-10-13T11:12:48.386 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:48.377+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.385416+0000) 2019-10-13T11:12:48.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:48.381+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.386118+0000) 2019-10-13T11:12:48.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:48.381+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.386748+0000) 2019-10-13T11:12:48.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:48.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:48.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:48.391+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.392572+0000) 2019-10-13T11:12:48.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:48.541+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:48.546836+0000) 2019-10-13T11:12:48.730 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:12:48.730 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:48.734 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:12:48.734 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:12:51.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:51.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:51.254153+0000) 2019-10-13T11:12:51.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:51.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:51.275557+0000) 2019-10-13T11:12:51.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:51.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:51.797924+0000) 2019-10-13T11:12:51.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:51.935+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:51.938169+0000) 2019-10-13T11:12:53.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:53.389+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.391570+0000) 2019-10-13T11:12:53.393 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:53.389+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.392415+0000) 2019-10-13T11:12:53.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:53.389+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.393148+0000) 2019-10-13T11:12:53.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:53.391+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.393312+0000) 2019-10-13T11:12:53.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:53.393+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.393950+0000) 2019-10-13T11:12:53.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:53.393+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.395017+0000) 2019-10-13T11:12:53.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:53.393+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.395517+0000) 2019-10-13T11:12:53.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:53.393+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:53.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:53.399+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:53.401448+0000) 2019-10-13T11:12:54.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:54.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.119085+0000) 2019-10-13T11:12:54.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:54.405+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:54.405832+0000) 2019-10-13T11:12:57.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:57.773+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:57.777010+0000) 2019-10-13T11:12:58.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:12:58.397+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.401059+0000) 2019-10-13T11:12:58.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:12:58.401+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.401729+0000) 2019-10-13T11:12:58.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:12:58.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.402474+0000) 2019-10-13T11:12:58.405 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:58.401+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.404728+0000) 2019-10-13T11:12:58.407 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:12:58.405+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.406678+0000) 2019-10-13T11:12:58.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:12:58.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:12:58.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:12:58.405+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.409434+0000) 2019-10-13T11:12:58.410 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:12:58.405+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.409476+0000) 2019-10-13T11:12:58.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:12:58.411+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.416044+0000) 2019-10-13T11:12:58.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:12:58.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:12:58.547059+0000) 2019-10-13T11:13:01.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:01.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:01.254351+0000) 2019-10-13T11:13:01.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:01.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:01.275756+0000) 2019-10-13T11:13:01.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:01.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:01.798076+0000) 2019-10-13T11:13:01.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:01.943+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:01.944772+0000) 2019-10-13T11:13:03.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:03.409+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.412076+0000) 2019-10-13T11:13:03.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:03.409+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.413271+0000) 2019-10-13T11:13:03.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:03.411+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.413770+0000) 2019-10-13T11:13:03.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:03.413+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.414878+0000) 2019-10-13T11:13:03.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:03.413+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.415274+0000) 2019-10-13T11:13:03.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:03.413+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.415859+0000) 2019-10-13T11:13:03.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:03.413+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.416472+0000) 2019-10-13T11:13:03.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:03.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:03.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:03.419+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:03.420413+0000) 2019-10-13T11:13:04.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:04.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.119301+0000) 2019-10-13T11:13:04.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:04.405+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:04.406059+0000) 2019-10-13T11:13:07.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:07.773+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:07.777210+0000) 2019-10-13T11:13:08.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:08.419+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.422303+0000) 2019-10-13T11:13:08.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:08.421+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.422931+0000) 2019-10-13T11:13:08.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:08.421+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.424742+0000) 2019-10-13T11:13:08.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:08.425+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.426334+0000) 2019-10-13T11:13:08.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:08.425+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.428286+0000) 2019-10-13T11:13:08.431 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:08.429+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:08.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:08.429+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.430060+0000) 2019-10-13T11:13:08.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:08.427+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.430328+0000) 2019-10-13T11:13:08.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:08.429+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.430797+0000) 2019-10-13T11:13:08.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:08.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:08.547188+0000) 2019-10-13T11:13:11.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:11.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:11.254496+0000) 2019-10-13T11:13:11.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:11.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:11.275949+0000) 2019-10-13T11:13:11.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:11.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:11.798268+0000) 2019-10-13T11:13:11.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:11.955+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:11.956260+0000) 2019-10-13T11:13:13.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:13.429+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.433319+0000) 2019-10-13T11:13:13.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:13.433+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.433985+0000) 2019-10-13T11:13:13.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:13.433+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.435041+0000) 2019-10-13T11:13:13.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:13.433+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.435894+0000) 2019-10-13T11:13:13.437 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:13.431+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.435731+0000) 2019-10-13T11:13:13.437 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:13.433+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.436219+0000) 2019-10-13T11:13:13.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:13.433+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.437333+0000) 2019-10-13T11:13:13.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:13.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:13.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:13.439+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:13.440560+0000) 2019-10-13T11:13:14.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:14.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.119519+0000) 2019-10-13T11:13:14.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:14.405+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:14.406239+0000) 2019-10-13T11:13:17.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:17.773+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:17.777466+0000) 2019-10-13T11:13:18.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:18.439+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.442884+0000) 2019-10-13T11:13:18.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:18.441+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.444035+0000) 2019-10-13T11:13:18.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:18.445+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.445725+0000) 2019-10-13T11:13:18.449 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:18.445+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.448458+0000) 2019-10-13T11:13:18.450 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:18.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:18.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:18.449+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.450236+0000) 2019-10-13T11:13:18.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:18.449+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.450539+0000) 2019-10-13T11:13:18.452 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:18.449+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.451195+0000) 2019-10-13T11:13:18.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:18.455+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.457000+0000) 2019-10-13T11:13:18.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:18.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:18.547404+0000) 2019-10-13T11:13:18.980 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:13:18.981 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:18.984 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:13:18.984 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:21.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:21.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:21.254735+0000) 2019-10-13T11:13:21.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:21.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:21.276171+0000) 2019-10-13T11:13:21.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:21.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:21.798484+0000) 2019-10-13T11:13:21.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:21.959+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:21.962972+0000) 2019-10-13T11:13:23.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:23.449+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.453460+0000) 2019-10-13T11:13:23.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:23.453+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.453997+0000) 2019-10-13T11:13:23.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:23.453+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.456660+0000) 2019-10-13T11:13:23.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:23.453+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.457093+0000) 2019-10-13T11:13:23.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:23.455+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.457021+0000) 2019-10-13T11:13:23.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:23.453+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.457446+0000) 2019-10-13T11:13:23.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:23.457+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.457989+0000) 2019-10-13T11:13:23.460 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:23.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:23.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:23.459+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:23.462034+0000) 2019-10-13T11:13:24.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:24.117+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.119708+0000) 2019-10-13T11:13:24.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:24.405+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:24.406436+0000) 2019-10-13T11:13:27.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:27.773+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:27.777679+0000) 2019-10-13T11:13:28.465 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:28.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.464392+0000) 2019-10-13T11:13:28.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:28.461+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.464744+0000) 2019-10-13T11:13:28.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:28.461+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.465190+0000) 2019-10-13T11:13:28.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:28.463+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.465814+0000) 2019-10-13T11:13:28.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:28.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.466354+0000) 2019-10-13T11:13:28.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:28.465+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.466805+0000) 2019-10-13T11:13:28.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:28.465+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.467360+0000) 2019-10-13T11:13:28.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:28.467+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.468868+0000) 2019-10-13T11:13:28.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:28.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:28.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:28.545+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:28.547560+0000) 2019-10-13T11:13:31.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:31.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:31.254938+0000) 2019-10-13T11:13:31.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:31.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:31.276384+0000) 2019-10-13T11:13:31.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:31.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:31.798702+0000) 2019-10-13T11:13:31.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:31.967+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:31.968855+0000) 2019-10-13T11:13:33.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:33.468+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.473545+0000) 2019-10-13T11:13:33.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:33.472+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.474277+0000) 2019-10-13T11:13:33.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:33.472+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.474683+0000) 2019-10-13T11:13:33.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:33.472+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.475364+0000) 2019-10-13T11:13:33.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:33.472+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.477264+0000) 2019-10-13T11:13:33.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:33.475+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.479369+0000) 2019-10-13T11:13:33.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:33.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:33.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:33.476+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.479913+0000) 2019-10-13T11:13:33.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:33.483+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:33.485049+0000) 2019-10-13T11:13:34.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:34.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.119921+0000) 2019-10-13T11:13:34.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:34.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:34.406612+0000) 2019-10-13T11:13:37.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:37.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:37.777954+0000) 2019-10-13T11:13:38.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:38.480+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.483310+0000) 2019-10-13T11:13:38.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:38.480+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.483807+0000) 2019-10-13T11:13:38.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:38.479+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.484017+0000) 2019-10-13T11:13:38.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:38.480+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.484575+0000) 2019-10-13T11:13:38.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:38.480+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.485364+0000) 2019-10-13T11:13:38.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:38.484+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.486127+0000) 2019-10-13T11:13:38.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:38.484+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.487060+0000) 2019-10-13T11:13:38.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:38.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:38.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:38.487+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.491215+0000) 2019-10-13T11:13:38.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:38.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:38.547733+0000) 2019-10-13T11:13:41.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:41.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:41.255166+0000) 2019-10-13T11:13:41.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:41.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:41.276583+0000) 2019-10-13T11:13:41.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:41.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:41.798904+0000) 2019-10-13T11:13:41.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:41.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:41.975547+0000) 2019-10-13T11:13:43.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:43.492+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.494176+0000) 2019-10-13T11:13:43.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:43.492+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.494404+0000) 2019-10-13T11:13:43.496 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:43.491+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.494872+0000) 2019-10-13T11:13:43.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:43.492+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.495806+0000) 2019-10-13T11:13:43.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:43.492+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.495992+0000) 2019-10-13T11:13:43.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:43.492+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.496422+0000) 2019-10-13T11:13:43.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:43.492+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.496744+0000) 2019-10-13T11:13:43.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:43.495+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:43.499185+0000) 2019-10-13T11:13:43.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:43.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:44.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:44.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.120109+0000) 2019-10-13T11:13:44.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:44.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:44.406818+0000) 2019-10-13T11:13:47.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:47.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:47.778189+0000) 2019-10-13T11:13:48.504 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:48.500+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.503582+0000) 2019-10-13T11:13:48.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:48.503+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.504147+0000) 2019-10-13T11:13:48.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:48.500+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.505041+0000) 2019-10-13T11:13:48.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:48.500+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.505347+0000) 2019-10-13T11:13:48.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:48.504+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.506213+0000) 2019-10-13T11:13:48.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:48.504+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.506590+0000) 2019-10-13T11:13:48.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:48.504+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.507287+0000) 2019-10-13T11:13:48.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:48.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:48.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:48.511+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.515747+0000) 2019-10-13T11:13:48.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:48.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:48.547924+0000) 2019-10-13T11:13:49.111 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:13:49.111 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:49.115 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:13:49.115 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:13:51.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:51.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:51.255365+0000) 2019-10-13T11:13:51.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:51.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:51.276798+0000) 2019-10-13T11:13:51.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:51.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:51.799108+0000) 2019-10-13T11:13:51.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:51.978+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:51.982057+0000) 2019-10-13T11:13:53.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:53.508+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.513115+0000) 2019-10-13T11:13:53.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:53.512+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.515633+0000) 2019-10-13T11:13:53.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:53.512+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.515795+0000) 2019-10-13T11:13:53.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:53.514+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.516589+0000) 2019-10-13T11:13:53.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:53.512+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.517142+0000) 2019-10-13T11:13:53.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:53.516+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.518272+0000) 2019-10-13T11:13:53.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:53.516+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.518644+0000) 2019-10-13T11:13:53.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:53.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:53.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:53.522+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:53.524741+0000) 2019-10-13T11:13:54.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:54.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.120292+0000) 2019-10-13T11:13:54.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:54.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:54.406995+0000) 2019-10-13T11:13:57.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:57.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:57.778418+0000) 2019-10-13T11:13:58.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:58.520+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.522859+0000) 2019-10-13T11:13:58.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:13:58.520+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.524787+0000) 2019-10-13T11:13:58.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:13:58.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.524934+0000) 2019-10-13T11:13:58.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:13:58.524+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.525995+0000) 2019-10-13T11:13:58.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:13:58.524+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.526475+0000) 2019-10-13T11:13:58.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:13:58.524+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.527668+0000) 2019-10-13T11:13:58.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:13:58.524+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.528139+0000) 2019-10-13T11:13:58.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:13:58.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:13:58.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:13:58.530+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.533958+0000) 2019-10-13T11:13:58.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:13:58.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:13:58.548089+0000) 2019-10-13T11:14:01.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:01.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:01.255565+0000) 2019-10-13T11:14:01.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:01.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:01.276993+0000) 2019-10-13T11:14:01.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:01.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:01.799305+0000) 2019-10-13T11:14:01.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:01.986+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:01.988724+0000) 2019-10-13T11:14:03.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:03.530+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.533315+0000) 2019-10-13T11:14:03.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:03.532+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.533891+0000) 2019-10-13T11:14:03.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:03.532+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.534310+0000) 2019-10-13T11:14:03.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:03.532+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.534834+0000) 2019-10-13T11:14:03.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:03.532+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.535226+0000) 2019-10-13T11:14:03.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:03.532+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.535687+0000) 2019-10-13T11:14:03.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:03.532+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.536088+0000) 2019-10-13T11:14:03.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:03.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:03.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:03.538+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:03.543303+0000) 2019-10-13T11:14:04.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:04.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.120473+0000) 2019-10-13T11:14:04.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:04.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:04.407131+0000) 2019-10-13T11:14:07.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:07.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:07.778616+0000) 2019-10-13T11:14:08.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:08.540+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.543357+0000) 2019-10-13T11:14:08.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:08.542+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.544549+0000) 2019-10-13T11:14:08.546 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:08.540+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.545077+0000) 2019-10-13T11:14:08.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:08.544+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.546438+0000) 2019-10-13T11:14:08.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:08.544+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.546975+0000) 2019-10-13T11:14:08.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:08.544+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.547339+0000) 2019-10-13T11:14:08.548 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:08.544+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.546475+0000) 2019-10-13T11:14:08.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:08.544+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.548203+0000) 2019-10-13T11:14:08.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:08.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:08.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:08.546+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:08.551775+0000) 2019-10-13T11:14:11.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:11.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:11.255801+0000) 2019-10-13T11:14:11.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:11.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:11.277189+0000) 2019-10-13T11:14:11.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:11.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:11.799517+0000) 2019-10-13T11:14:11.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:11.990+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:11.994740+0000) 2019-10-13T11:14:13.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:13.552+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.554049+0000) 2019-10-13T11:14:13.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:13.552+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.554483+0000) 2019-10-13T11:14:13.555 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:13.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.554775+0000) 2019-10-13T11:14:13.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:13.552+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.555208+0000) 2019-10-13T11:14:13.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:13.552+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.555329+0000) 2019-10-13T11:14:13.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:13.550+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.555825+0000) 2019-10-13T11:14:13.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:13.552+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.556399+0000) 2019-10-13T11:14:13.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:13.554+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:13.559411+0000) 2019-10-13T11:14:13.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:13.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:14.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:14.116+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.120692+0000) 2019-10-13T11:14:14.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:14.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:14.407309+0000) 2019-10-13T11:14:17.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:17.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:17.778818+0000) 2019-10-13T11:14:18.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:18.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.548424+0000) 2019-10-13T11:14:18.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:18.558+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.563481+0000) 2019-10-13T11:14:18.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:18.560+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.565694+0000) 2019-10-13T11:14:18.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:18.564+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.567217+0000) 2019-10-13T11:14:18.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:18.564+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.567489+0000) 2019-10-13T11:14:18.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:18.564+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.568815+0000) 2019-10-13T11:14:18.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:18.564+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.569197+0000) 2019-10-13T11:14:18.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:18.564+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.569617+0000) 2019-10-13T11:14:18.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:18.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:18.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:18.574+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:18.576637+0000) 2019-10-13T11:14:19.227 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:14:19.227 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:19.231 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:14:19.231 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:21.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:21.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:21.256030+0000) 2019-10-13T11:14:21.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:21.272+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:21.277402+0000) 2019-10-13T11:14:21.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:21.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:21.799714+0000) 2019-10-13T11:14:22.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:21.998+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:22.001301+0000) 2019-10-13T11:14:23.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:23.572+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.574430+0000) 2019-10-13T11:14:23.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:23.572+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.575140+0000) 2019-10-13T11:14:23.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:23.572+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.575423+0000) 2019-10-13T11:14:23.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:23.572+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.576960+0000) 2019-10-13T11:14:23.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:23.574+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.576840+0000) 2019-10-13T11:14:23.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:23.572+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.577381+0000) 2019-10-13T11:14:23.578 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:23.576+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.577735+0000) 2019-10-13T11:14:23.581 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:23.576+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:23.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:23.578+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:23.580941+0000) 2019-10-13T11:14:24.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:24.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.120909+0000) 2019-10-13T11:14:24.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:24.404+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:24.407522+0000) 2019-10-13T11:14:27.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:27.776+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:27.779013+0000) 2019-10-13T11:14:28.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:28.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.548635+0000) 2019-10-13T11:14:28.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:28.580+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.584775+0000) 2019-10-13T11:14:28.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:28.580+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.585404+0000) 2019-10-13T11:14:28.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:28.584+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.585905+0000) 2019-10-13T11:14:28.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:28.584+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.586656+0000) 2019-10-13T11:14:28.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:28.582+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.586786+0000) 2019-10-13T11:14:28.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:28.584+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.587329+0000) 2019-10-13T11:14:28.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:28.584+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.587616+0000) 2019-10-13T11:14:28.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:28.586+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:28.589601+0000) 2019-10-13T11:14:28.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:28.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:31.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:31.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:31.256259+0000) 2019-10-13T11:14:31.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:31.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:31.277623+0000) 2019-10-13T11:14:31.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:31.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:31.799912+0000) 2019-10-13T11:14:32.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:32.002+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:32.007835+0000) 2019-10-13T11:14:33.595 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:33.591+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.594726+0000) 2019-10-13T11:14:33.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:33.591+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.594827+0000) 2019-10-13T11:14:33.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:33.591+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.596213+0000) 2019-10-13T11:14:33.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:33.591+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.596620+0000) 2019-10-13T11:14:33.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:33.595+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.598279+0000) 2019-10-13T11:14:33.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:33.594+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.598466+0000) 2019-10-13T11:14:33.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:33.595+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.599234+0000) 2019-10-13T11:14:33.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:33.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:33.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:33.598+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:33.603942+0000) 2019-10-13T11:14:34.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:34.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.121086+0000) 2019-10-13T11:14:34.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:34.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:34.407705+0000) 2019-10-13T11:14:37.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:37.775+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:37.779209+0000) 2019-10-13T11:14:38.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:38.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.548829+0000) 2019-10-13T11:14:38.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:38.599+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.603667+0000) 2019-10-13T11:14:38.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:38.602+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.604584+0000) 2019-10-13T11:14:38.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:38.599+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.605225+0000) 2019-10-13T11:14:38.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:38.603+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.606360+0000) 2019-10-13T11:14:38.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:38.603+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.606544+0000) 2019-10-13T11:14:38.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:38.603+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.607039+0000) 2019-10-13T11:14:38.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:38.603+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.607299+0000) 2019-10-13T11:14:38.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:38.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:38.616 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:38.610+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:38.615453+0000) 2019-10-13T11:14:41.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:41.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:41.256488+0000) 2019-10-13T11:14:41.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:41.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:41.277834+0000) 2019-10-13T11:14:41.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:41.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:41.800139+0000) 2019-10-13T11:14:42.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:42.010+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:42.014371+0000) 2019-10-13T11:14:43.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:43.607+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.612644+0000) 2019-10-13T11:14:43.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:43.607+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.613304+0000) 2019-10-13T11:14:43.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:43.611+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.614133+0000) 2019-10-13T11:14:43.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:43.611+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.614677+0000) 2019-10-13T11:14:43.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:43.614+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.616936+0000) 2019-10-13T11:14:43.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:43.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:43.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:43.615+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.619357+0000) 2019-10-13T11:14:43.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:43.615+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.619965+0000) 2019-10-13T11:14:43.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:43.622+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:43.625806+0000) 2019-10-13T11:14:44.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:44.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.121268+0000) 2019-10-13T11:14:44.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:44.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:44.407884+0000) 2019-10-13T11:14:47.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:47.775+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:47.779405+0000) 2019-10-13T11:14:48.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:48.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.549041+0000) 2019-10-13T11:14:48.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:48.618+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.622525+0000) 2019-10-13T11:14:48.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:48.619+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.622387+0000) 2019-10-13T11:14:48.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:48.619+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.624058+0000) 2019-10-13T11:14:48.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:48.619+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.624697+0000) 2019-10-13T11:14:48.626 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:48.619+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.625371+0000) 2019-10-13T11:14:48.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:48.619+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.625407+0000) 2019-10-13T11:14:48.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:48.623+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.625805+0000) 2019-10-13T11:14:48.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:48.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:48.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:48.630+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:48.632989+0000) 2019-10-13T11:14:49.363 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:14:49.363 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:49.367 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:14:49.367 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:14:51.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:51.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:51.256692+0000) 2019-10-13T11:14:51.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:51.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:51.278036+0000) 2019-10-13T11:14:51.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:51.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:51.800333+0000) 2019-10-13T11:14:52.022 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:52.018+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:52.020967+0000) 2019-10-13T11:14:53.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:53.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.632182+0000) 2019-10-13T11:14:53.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:53.627+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.632889+0000) 2019-10-13T11:14:53.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:53.627+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.633701+0000) 2019-10-13T11:14:53.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:53.631+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.635276+0000) 2019-10-13T11:14:53.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:53.630+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.635314+0000) 2019-10-13T11:14:53.637 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:53.631+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.635980+0000) 2019-10-13T11:14:53.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:53.631+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.637646+0000) 2019-10-13T11:14:53.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:53.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:53.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:53.638+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:53.643841+0000) 2019-10-13T11:14:54.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:54.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.121473+0000) 2019-10-13T11:14:54.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:54.407+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:54.408093+0000) 2019-10-13T11:14:57.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:57.775+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:57.779604+0000) 2019-10-13T11:14:58.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:58.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.549226+0000) 2019-10-13T11:14:58.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:14:58.639+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.641985+0000) 2019-10-13T11:14:58.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:14:58.637+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.642201+0000) 2019-10-13T11:14:58.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:14:58.639+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.644782+0000) 2019-10-13T11:14:58.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:14:58.639+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.644825+0000) 2019-10-13T11:14:58.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:14:58.639+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.645310+0000) 2019-10-13T11:14:58.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:14:58.643+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.646276+0000) 2019-10-13T11:14:58.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:14:58.643+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.646656+0000) 2019-10-13T11:14:58.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:14:58.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:14:58.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:14:58.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:14:58.652192+0000) 2019-10-13T11:15:01.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:01.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:01.256889+0000) 2019-10-13T11:15:01.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:01.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:01.278283+0000) 2019-10-13T11:15:01.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:01.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:01.800536+0000) 2019-10-13T11:15:02.028 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:02.021+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:02.027507+0000) 2019-10-13T11:15:03.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:03.649+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.653657+0000) 2019-10-13T11:15:03.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:03.651+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.654213+0000) 2019-10-13T11:15:03.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:03.651+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.654816+0000) 2019-10-13T11:15:03.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:03.651+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.655342+0000) 2019-10-13T11:15:03.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:03.651+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.655779+0000) 2019-10-13T11:15:03.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:03.651+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.656180+0000) 2019-10-13T11:15:03.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:03.651+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.656843+0000) 2019-10-13T11:15:03.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:03.655+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:03.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:03.657+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:03.662197+0000) 2019-10-13T11:15:04.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:04.115+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.121650+0000) 2019-10-13T11:15:04.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:04.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:04.408227+0000) 2019-10-13T11:15:07.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:07.775+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:07.779845+0000) 2019-10-13T11:15:08.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:08.543+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.549423+0000) 2019-10-13T11:15:08.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:08.657+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.661138+0000) 2019-10-13T11:15:08.663 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:08.659+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.661943+0000) 2019-10-13T11:15:08.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:08.659+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.662711+0000) 2019-10-13T11:15:08.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:08.659+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.663457+0000) 2019-10-13T11:15:08.664 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:08.659+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.663537+0000) 2019-10-13T11:15:08.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:08.659+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.664397+0000) 2019-10-13T11:15:08.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:08.659+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.664375+0000) 2019-10-13T11:15:08.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:08.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:08.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:08.665+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:08.670026+0000) 2019-10-13T11:15:11.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:11.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:11.257088+0000) 2019-10-13T11:15:11.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:11.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:11.278500+0000) 2019-10-13T11:15:11.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:11.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:11.800730+0000) 2019-10-13T11:15:12.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:12.029+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:12.034063+0000) 2019-10-13T11:15:13.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:13.667+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.671410+0000) 2019-10-13T11:15:13.673 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:13.667+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.672085+0000) 2019-10-13T11:15:13.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:13.669+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.672381+0000) 2019-10-13T11:15:13.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:13.667+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.672843+0000) 2019-10-13T11:15:13.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:13.667+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.673279+0000) 2019-10-13T11:15:13.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:13.667+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.673659+0000) 2019-10-13T11:15:13.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:13.671+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.674038+0000) 2019-10-13T11:15:13.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:13.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:13.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:13.677+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:13.680507+0000) 2019-10-13T11:15:14.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:14.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.121847+0000) 2019-10-13T11:15:14.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:14.407+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:14.408431+0000) 2019-10-13T11:15:17.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:17.775+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:17.780054+0000) 2019-10-13T11:15:18.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:18.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.549620+0000) 2019-10-13T11:15:18.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:18.675+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.681121+0000) 2019-10-13T11:15:18.682 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:18.675+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.681709+0000) 2019-10-13T11:15:18.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:18.679+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.682203+0000) 2019-10-13T11:15:18.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:18.679+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.682553+0000) 2019-10-13T11:15:18.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:18.679+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.683926+0000) 2019-10-13T11:15:18.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:18.681+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.684591+0000) 2019-10-13T11:15:18.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:18.679+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.685473+0000) 2019-10-13T11:15:18.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:18.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:18.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:18.685+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:18.689733+0000) 2019-10-13T11:15:19.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:15:19.493 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:19.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:15:19.497 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:21.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:21.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:21.257282+0000) 2019-10-13T11:15:21.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:21.275+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:21.278761+0000) 2019-10-13T11:15:21.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:21.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:21.800927+0000) 2019-10-13T11:15:22.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:22.037+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:22.040675+0000) 2019-10-13T11:15:23.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:23.687+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.690502+0000) 2019-10-13T11:15:23.693 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:23.687+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.691225+0000) 2019-10-13T11:15:23.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:23.687+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.691748+0000) 2019-10-13T11:15:23.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:23.687+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.692170+0000) 2019-10-13T11:15:23.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:23.687+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.692431+0000) 2019-10-13T11:15:23.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:23.687+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.693474+0000) 2019-10-13T11:15:23.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:23.689+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.693609+0000) 2019-10-13T11:15:23.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:23.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:23.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:23.693+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:23.696959+0000) 2019-10-13T11:15:24.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:24.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.122054+0000) 2019-10-13T11:15:24.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:24.403+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:24.408605+0000) 2019-10-13T11:15:27.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:27.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:27.780249+0000) 2019-10-13T11:15:28.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:28.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.549806+0000) 2019-10-13T11:15:28.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:28.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.700713+0000) 2019-10-13T11:15:28.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:28.694+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.700892+0000) 2019-10-13T11:15:28.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:28.694+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.701205+0000) 2019-10-13T11:15:28.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:28.698+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.702012+0000) 2019-10-13T11:15:28.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:28.701+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.704241+0000) 2019-10-13T11:15:28.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:28.698+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.704127+0000) 2019-10-13T11:15:28.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:28.702+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.706155+0000) 2019-10-13T11:15:28.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:28.702+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:28.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:28.709+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:28.713693+0000) 2019-10-13T11:15:31.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:31.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:31.257491+0000) 2019-10-13T11:15:31.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:31.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:31.278961+0000) 2019-10-13T11:15:31.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:31.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:31.801156+0000) 2019-10-13T11:15:32.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:32.041+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:32.047282+0000) 2019-10-13T11:15:33.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:33.706+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.711809+0000) 2019-10-13T11:15:33.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:33.706+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.713539+0000) 2019-10-13T11:15:33.714 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:33.710+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.714042+0000) 2019-10-13T11:15:33.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:33.710+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.714984+0000) 2019-10-13T11:15:33.716 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:33.709+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.714860+0000) 2019-10-13T11:15:33.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:33.710+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.716682+0000) 2019-10-13T11:15:33.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:33.710+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.717103+0000) 2019-10-13T11:15:33.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:33.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:33.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:33.717+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:33.723923+0000) 2019-10-13T11:15:34.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:34.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.122230+0000) 2019-10-13T11:15:34.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:34.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:34.408812+0000) 2019-10-13T11:15:37.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:37.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:37.780469+0000) 2019-10-13T11:15:38.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:38.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.550034+0000) 2019-10-13T11:15:38.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:38.714+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.721621+0000) 2019-10-13T11:15:38.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:38.717+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.721837+0000) 2019-10-13T11:15:38.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:38.718+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.723528+0000) 2019-10-13T11:15:38.725 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:38.718+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.724241+0000) 2019-10-13T11:15:38.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:38.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.724440+0000) 2019-10-13T11:15:38.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:38.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.724713+0000) 2019-10-13T11:15:38.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:38.718+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.725182+0000) 2019-10-13T11:15:38.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:38.725+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:38.728931+0000) 2019-10-13T11:15:38.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:38.722+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:41.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:41.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:41.257721+0000) 2019-10-13T11:15:41.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:41.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:41.279157+0000) 2019-10-13T11:15:41.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:41.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:41.801358+0000) 2019-10-13T11:15:42.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:42.049+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:42.053837+0000) 2019-10-13T11:15:43.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:43.730+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.734194+0000) 2019-10-13T11:15:43.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:43.730+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.734978+0000) 2019-10-13T11:15:43.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:43.729+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.735111+0000) 2019-10-13T11:15:43.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:43.730+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.735632+0000) 2019-10-13T11:15:43.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:43.730+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.735929+0000) 2019-10-13T11:15:43.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:43.730+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.736729+0000) 2019-10-13T11:15:43.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:43.730+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.737587+0000) 2019-10-13T11:15:43.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:43.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:43.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:43.733+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:43.740024+0000) 2019-10-13T11:15:44.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:44.118+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.122410+0000) 2019-10-13T11:15:44.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:44.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:44.408992+0000) 2019-10-13T11:15:47.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:47.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:47.780664+0000) 2019-10-13T11:15:48.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:48.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.550228+0000) 2019-10-13T11:15:48.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:48.738+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.743202+0000) 2019-10-13T11:15:48.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:48.738+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.743614+0000) 2019-10-13T11:15:48.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:48.741+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.745353+0000) 2019-10-13T11:15:48.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:48.742+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.746532+0000) 2019-10-13T11:15:48.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:48.742+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.746930+0000) 2019-10-13T11:15:48.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:48.742+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.748242+0000) 2019-10-13T11:15:48.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:48.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.750626+0000) 2019-10-13T11:15:48.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:48.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:48.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:48.749+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:48.752288+0000) 2019-10-13T11:15:49.633 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:15:49.634 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:49.637 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:15:49.637 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:15:51.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:51.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:51.257980+0000) 2019-10-13T11:15:51.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:51.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:51.279354+0000) 2019-10-13T11:15:51.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:51.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:51.801569+0000) 2019-10-13T11:15:52.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:52.057+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:52.060438+0000) 2019-10-13T11:15:53.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:53.750+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.753872+0000) 2019-10-13T11:15:53.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:53.750+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.754234+0000) 2019-10-13T11:15:53.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:53.750+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.754820+0000) 2019-10-13T11:15:53.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:53.749+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.755055+0000) 2019-10-13T11:15:53.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:53.750+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.755703+0000) 2019-10-13T11:15:53.758 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:53.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.757369+0000) 2019-10-13T11:15:53.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:53.750+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.757673+0000) 2019-10-13T11:15:53.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:53.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:53.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:53.757+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:53.762639+0000) 2019-10-13T11:15:54.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:54.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.122592+0000) 2019-10-13T11:15:54.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:54.402+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:54.409188+0000) 2019-10-13T11:15:57.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:57.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:57.780895+0000) 2019-10-13T11:15:58.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:58.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.550411+0000) 2019-10-13T11:15:58.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:15:58.758+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.765481+0000) 2019-10-13T11:15:58.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:15:58.758+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.765234+0000) 2019-10-13T11:15:58.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:15:58.762+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.766217+0000) 2019-10-13T11:15:58.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:15:58.762+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.767013+0000) 2019-10-13T11:15:58.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:15:58.762+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.766959+0000) 2019-10-13T11:15:58.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:15:58.761+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.767930+0000) 2019-10-13T11:15:58.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:15:58.762+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.768494+0000) 2019-10-13T11:15:58.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:15:58.766+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:15:58.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:15:58.769+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:15:58.772530+0000) 2019-10-13T11:16:01.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:01.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:01.258185+0000) 2019-10-13T11:16:01.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:01.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:01.279510+0000) 2019-10-13T11:16:01.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:01.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:01.801788+0000) 2019-10-13T11:16:02.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:02.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:02.066942+0000) 2019-10-13T11:16:03.776 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:03.770+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.774944+0000) 2019-10-13T11:16:03.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:03.770+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.775635+0000) 2019-10-13T11:16:03.777 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:03.770+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.776112+0000) 2019-10-13T11:16:03.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:03.770+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.776432+0000) 2019-10-13T11:16:03.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:03.770+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.777500+0000) 2019-10-13T11:16:03.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:03.774+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.778049+0000) 2019-10-13T11:16:03.780 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:03.772+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.779474+0000) 2019-10-13T11:16:03.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:03.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:03.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:03.780+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:03.786129+0000) 2019-10-13T11:16:04.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:04.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.122772+0000) 2019-10-13T11:16:04.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:04.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:04.409366+0000) 2019-10-13T11:16:07.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:07.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:07.781098+0000) 2019-10-13T11:16:08.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:08.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.550620+0000) 2019-10-13T11:16:08.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:08.778+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.784779+0000) 2019-10-13T11:16:08.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:08.778+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.785419+0000) 2019-10-13T11:16:08.787 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:08.782+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.785832+0000) 2019-10-13T11:16:08.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:08.782+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.786153+0000) 2019-10-13T11:16:08.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:08.782+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.786735+0000) 2019-10-13T11:16:08.788 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:08.782+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.787343+0000) 2019-10-13T11:16:08.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:08.780+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.787955+0000) 2019-10-13T11:16:08.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:08.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:08.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:08.788+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:08.793817+0000) 2019-10-13T11:16:11.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:11.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:11.258405+0000) 2019-10-13T11:16:11.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:11.274+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:11.279707+0000) 2019-10-13T11:16:11.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:11.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:11.801964+0000) 2019-10-13T11:16:12.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:12.076+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:12.078448+0000) 2019-10-13T11:16:13.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:13.790+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.794028+0000) 2019-10-13T11:16:13.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:13.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.794474+0000) 2019-10-13T11:16:13.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:13.790+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.794841+0000) 2019-10-13T11:16:13.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:13.790+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.795997+0000) 2019-10-13T11:16:13.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:13.790+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.797073+0000) 2019-10-13T11:16:13.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:13.794+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.797720+0000) 2019-10-13T11:16:13.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:13.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.797837+0000) 2019-10-13T11:16:13.801 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:13.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:13.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:13.796+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:13.801335+0000) 2019-10-13T11:16:14.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:14.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.122954+0000) 2019-10-13T11:16:14.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:14.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:14.409545+0000) 2019-10-13T11:16:17.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:17.774+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:17.781298+0000) 2019-10-13T11:16:18.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:18.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.550818+0000) 2019-10-13T11:16:18.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:18.798+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.804556+0000) 2019-10-13T11:16:18.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:18.802+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.805776+0000) 2019-10-13T11:16:18.807 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:18.802+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.806199+0000) 2019-10-13T11:16:18.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:18.802+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.806627+0000) 2019-10-13T11:16:18.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:18.802+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.806748+0000) 2019-10-13T11:16:18.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:18.802+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.807920+0000) 2019-10-13T11:16:18.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:18.804+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.808374+0000) 2019-10-13T11:16:18.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:18.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:18.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:18.804+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:18.811750+0000) 2019-10-13T11:16:19.746 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:16:19.746 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:19.750 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:16:19.751 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:21.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:21.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:21.258612+0000) 2019-10-13T11:16:21.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:21.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:21.279911+0000) 2019-10-13T11:16:21.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:21.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:21.802166+0000) 2019-10-13T11:16:22.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:22.080+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:22.084957+0000) 2019-10-13T11:16:23.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:23.809+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.813778+0000) 2019-10-13T11:16:23.816 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:23.809+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.815671+0000) 2019-10-13T11:16:23.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:23.809+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.817309+0000) 2019-10-13T11:16:23.820 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:23.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.819639+0000) 2019-10-13T11:16:23.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:23.816+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.820885+0000) 2019-10-13T11:16:23.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:23.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:23.822 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:23.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.821842+0000) 2019-10-13T11:16:23.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:23.813+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.821545+0000) 2019-10-13T11:16:23.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:23.816+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:23.821994+0000) 2019-10-13T11:16:24.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:24.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.123163+0000) 2019-10-13T11:16:24.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:24.405+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:24.409779+0000) 2019-10-13T11:16:27.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:27.773+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:27.781476+0000) 2019-10-13T11:16:28.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:28.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.551009+0000) 2019-10-13T11:16:28.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:28.820+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.825959+0000) 2019-10-13T11:16:28.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:28.821+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.826692+0000) 2019-10-13T11:16:28.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:28.821+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.829043+0000) 2019-10-13T11:16:28.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:28.821+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.829338+0000) 2019-10-13T11:16:28.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:28.825+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.831073+0000) 2019-10-13T11:16:28.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:28.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:28.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:28.825+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.831324+0000) 2019-10-13T11:16:28.833 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:28.825+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.831911+0000) 2019-10-13T11:16:28.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:28.828+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:28.833350+0000) 2019-10-13T11:16:31.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:31.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:31.258810+0000) 2019-10-13T11:16:31.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:31.273+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:31.280138+0000) 2019-10-13T11:16:31.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:31.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:31.802378+0000) 2019-10-13T11:16:32.092 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:32.088+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:32.091558+0000) 2019-10-13T11:16:33.835 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:33.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.834711+0000) 2019-10-13T11:16:33.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:33.829+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.835206+0000) 2019-10-13T11:16:33.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:33.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.835665+0000) 2019-10-13T11:16:33.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:33.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.836341+0000) 2019-10-13T11:16:33.839 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:33.833+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.838366+0000) 2019-10-13T11:16:33.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:33.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.840388+0000) 2019-10-13T11:16:33.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:33.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:33.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:33.837+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.842124+0000) 2019-10-13T11:16:33.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:33.844+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:33.849584+0000) 2019-10-13T11:16:34.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:34.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.123366+0000) 2019-10-13T11:16:34.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:34.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:34.409957+0000) 2019-10-13T11:16:37.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:37.781+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:37.781725+0000) 2019-10-13T11:16:38.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:38.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.551194+0000) 2019-10-13T11:16:38.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:38.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.844749+0000) 2019-10-13T11:16:38.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:38.841+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.845175+0000) 2019-10-13T11:16:38.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:38.845+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.846441+0000) 2019-10-13T11:16:38.848 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:38.845+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.848086+0000) 2019-10-13T11:16:38.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:38.844+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.848142+0000) 2019-10-13T11:16:38.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:38.849+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.850298+0000) 2019-10-13T11:16:38.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:38.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:38.853 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:38.849+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.852270+0000) 2019-10-13T11:16:38.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:38.852+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:38.857899+0000) 2019-10-13T11:16:41.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:41.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:41.259009+0000) 2019-10-13T11:16:41.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:41.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:41.280346+0000) 2019-10-13T11:16:41.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:41.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:41.802573+0000) 2019-10-13T11:16:42.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:42.100+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:42.103199+0000) 2019-10-13T11:16:43.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:43.853+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.855090+0000) 2019-10-13T11:16:43.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:43.853+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.856069+0000) 2019-10-13T11:16:43.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:43.853+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.855682+0000) 2019-10-13T11:16:43.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:43.853+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.856456+0000) 2019-10-13T11:16:43.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:43.857+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.858214+0000) 2019-10-13T11:16:43.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:43.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135077 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:43.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:43.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.861393+0000) 2019-10-13T11:16:43.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:43.861+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.861956+0000) 2019-10-13T11:16:43.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:43.860+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:43.864866+0000) 2019-10-13T11:16:44.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:44.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.123572+0000) 2019-10-13T11:16:44.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:44.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:44.410140+0000) 2019-10-13T11:16:47.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:47.781+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:47.781980+0000) 2019-10-13T11:16:48.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:48.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.551420+0000) 2019-10-13T11:16:48.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:48.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.864875+0000) 2019-10-13T11:16:48.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:48.861+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.865188+0000) 2019-10-13T11:16:48.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:48.861+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.865645+0000) 2019-10-13T11:16:48.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:48.865+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.867764+0000) 2019-10-13T11:16:48.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:48.865+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.868236+0000) 2019-10-13T11:16:48.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:48.864+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.870421+0000) 2019-10-13T11:16:48.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:48.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.870613+0000) 2019-10-13T11:16:48.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:48.869+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:48.870833+0000) 2019-10-13T11:16:48.872 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:48.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:49.870 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:16:49.870 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:49.875 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:16:49.875 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:16:51.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:51.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:51.259223+0000) 2019-10-13T11:16:51.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:51.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:51.280576+0000) 2019-10-13T11:16:51.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:51.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:51.802775+0000) 2019-10-13T11:16:52.115 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:52.112+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:52.114654+0000) 2019-10-13T11:16:53.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:53.873+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.875277+0000) 2019-10-13T11:16:53.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:53.873+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.877160+0000) 2019-10-13T11:16:53.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:53.877+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.877792+0000) 2019-10-13T11:16:53.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:53.877+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.878140+0000) 2019-10-13T11:16:53.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:53.877+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.878905+0000) 2019-10-13T11:16:53.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:53.872+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.879544+0000) 2019-10-13T11:16:53.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:53.877+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.880322+0000) 2019-10-13T11:16:53.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:53.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:53.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:53.880+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:53.885577+0000) 2019-10-13T11:16:54.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:54.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:54.123775+0000) 2019-10-13T11:16:54.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:54.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:54.410362+0000) 2019-10-13T11:16:57.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:57.781+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:57.782214+0000) 2019-10-13T11:16:58.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:58.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.551614+0000) 2019-10-13T11:16:58.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:16:58.885+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.886371+0000) 2019-10-13T11:16:58.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:16:58.885+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.886952+0000) 2019-10-13T11:16:58.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:16:58.885+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.887765+0000) 2019-10-13T11:16:58.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:16:58.885+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.888297+0000) 2019-10-13T11:16:58.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:16:58.885+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.888938+0000) 2019-10-13T11:16:58.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:16:58.885+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.889255+0000) 2019-10-13T11:16:58.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:16:58.884+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.889385+0000) 2019-10-13T11:16:58.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:16:58.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135167 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:16:58.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:16:58.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:16:58.893602+0000) 2019-10-13T11:17:01.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:01.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:01.259461+0000) 2019-10-13T11:17:01.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:01.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:01.280772+0000) 2019-10-13T11:17:01.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:01.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:01.803009+0000) 2019-10-13T11:17:02.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:02.120+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:02.121189+0000) 2019-10-13T11:17:03.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:03.893+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.895773+0000) 2019-10-13T11:17:03.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:03.893+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.896573+0000) 2019-10-13T11:17:03.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:03.897+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.899780+0000) 2019-10-13T11:17:03.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:03.897+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.899989+0000) 2019-10-13T11:17:03.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:03.901+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.901771+0000) 2019-10-13T11:17:03.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:03.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:03.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:03.901+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.903873+0000) 2019-10-13T11:17:03.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:03.904+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.904232+0000) 2019-10-13T11:17:03.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:03.904+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:03.904227+0000) 2019-10-13T11:17:04.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:04.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:04.123962+0000) 2019-10-13T11:17:04.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:04.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:04.410544+0000) 2019-10-13T11:17:07.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:07.781+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:07.782429+0000) 2019-10-13T11:17:08.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:08.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.551799+0000) 2019-10-13T11:17:08.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:08.905+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.905990+0000) 2019-10-13T11:17:08.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:08.905+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.907311+0000) 2019-10-13T11:17:08.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:08.905+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.907699+0000) 2019-10-13T11:17:08.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:08.905+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.908016+0000) 2019-10-13T11:17:08.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:08.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.909113+0000) 2019-10-13T11:17:08.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:08.905+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.909296+0000) 2019-10-13T11:17:08.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:08.909+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.910186+0000) 2019-10-13T11:17:08.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:08.909+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:08.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:08.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:08.915552+0000) 2019-10-13T11:17:11.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:11.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:11.259694+0000) 2019-10-13T11:17:11.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:11.277+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:11.280964+0000) 2019-10-13T11:17:11.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:11.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:11.803204+0000) 2019-10-13T11:17:12.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:12.127+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:12.130189+0000) 2019-10-13T11:17:13.917 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:13.913+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.916591+0000) 2019-10-13T11:17:13.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:13.917+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.918605+0000) 2019-10-13T11:17:13.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:13.917+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.919014+0000) 2019-10-13T11:17:13.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:13.917+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.919678+0000) 2019-10-13T11:17:13.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:13.915+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.919777+0000) 2019-10-13T11:17:13.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:13.917+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.920234+0000) 2019-10-13T11:17:13.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:13.921+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.922155+0000) 2019-10-13T11:17:13.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:13.921+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:13.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:13.927+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:13.930978+0000) 2019-10-13T11:17:14.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:14.121+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:14.124146+0000) 2019-10-13T11:17:14.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:14.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:14.410746+0000) 2019-10-13T11:17:17.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:17.781+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:17.782641+0000) 2019-10-13T11:17:18.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:18.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.551991+0000) 2019-10-13T11:17:18.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:18.924+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.927737+0000) 2019-10-13T11:17:18.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:18.924+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.928140+0000) 2019-10-13T11:17:18.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:18.924+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.928731+0000) 2019-10-13T11:17:18.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:18.927+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.928799+0000) 2019-10-13T11:17:18.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:18.928+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.931445+0000) 2019-10-13T11:17:18.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:18.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135289 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:18.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:18.928+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.933295+0000) 2019-10-13T11:17:18.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:18.928+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.933537+0000) 2019-10-13T11:17:18.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:18.935+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:18.938199+0000) 2019-10-13T11:17:19.951 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:17:19.951 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:19.955 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:17:19.955 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:21.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:21.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:21.259908+0000) 2019-10-13T11:17:21.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:21.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:21.281180+0000) 2019-10-13T11:17:21.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:21.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:21.803406+0000) 2019-10-13T11:17:22.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:22.135+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:22.136747+0000) 2019-10-13T11:17:23.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:23.932+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.935998+0000) 2019-10-13T11:17:23.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:23.932+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.936830+0000) 2019-10-13T11:17:23.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:23.932+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.936372+0000) 2019-10-13T11:17:23.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:23.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.937551+0000) 2019-10-13T11:17:23.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:23.936+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.937902+0000) 2019-10-13T11:17:23.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:23.936+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.938556+0000) 2019-10-13T11:17:23.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:23.935+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.938815+0000) 2019-10-13T11:17:23.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:23.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:23.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:23.943+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:23.947710+0000) 2019-10-13T11:17:24.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:24.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:24.124327+0000) 2019-10-13T11:17:24.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:24.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:24.410960+0000) 2019-10-13T11:17:27.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:27.780+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:27.782854+0000) 2019-10-13T11:17:28.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:28.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.552197+0000) 2019-10-13T11:17:28.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:28.944+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.946170+0000) 2019-10-13T11:17:28.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:28.944+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.946867+0000) 2019-10-13T11:17:28.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:28.944+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.948599+0000) 2019-10-13T11:17:28.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:28.948+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.951140+0000) 2019-10-13T11:17:28.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:28.948+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.951925+0000) 2019-10-13T11:17:28.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:28.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.952584+0000) 2019-10-13T11:17:28.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:28.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:28.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:28.948+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.953248+0000) 2019-10-13T11:17:28.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:28.955+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:28.957261+0000) 2019-10-13T11:17:31.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:31.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:31.260120+0000) 2019-10-13T11:17:31.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:31.276+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:31.281377+0000) 2019-10-13T11:17:31.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:31.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:31.803626+0000) 2019-10-13T11:17:32.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:32.139+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:32.143229+0000) 2019-10-13T11:17:33.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:33.952+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.956216+0000) 2019-10-13T11:17:33.957 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:33.952+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.956851+0000) 2019-10-13T11:17:33.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:33.956+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.959644+0000) 2019-10-13T11:17:33.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:33.956+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.960818+0000) 2019-10-13T11:17:33.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:33.956+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.961450+0000) 2019-10-13T11:17:33.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:33.960+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.961783+0000) 2019-10-13T11:17:33.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:33.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:33.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:33.959+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.962413+0000) 2019-10-13T11:17:33.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:33.967+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:33.968180+0000) 2019-10-13T11:17:34.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:34.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:34.124470+0000) 2019-10-13T11:17:34.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:34.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:34.411141+0000) 2019-10-13T11:17:37.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:37.780+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:37.783083+0000) 2019-10-13T11:17:38.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:38.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.552396+0000) 2019-10-13T11:17:38.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:38.964+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.966297+0000) 2019-10-13T11:17:38.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:38.964+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.966965+0000) 2019-10-13T11:17:38.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:38.964+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.967741+0000) 2019-10-13T11:17:38.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:38.967+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.968219+0000) 2019-10-13T11:17:38.970 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:38.964+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.969256+0000) 2019-10-13T11:17:38.970 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:38.964+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.969534+0000) 2019-10-13T11:17:38.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:38.968+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.970043+0000) 2019-10-13T11:17:38.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:38.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:38.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:38.971+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:38.973228+0000) 2019-10-13T11:17:41.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:41.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:41.260334+0000) 2019-10-13T11:17:41.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:41.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:41.281574+0000) 2019-10-13T11:17:41.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:41.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:41.803834+0000) 2019-10-13T11:17:42.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:42.151+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:42.152545+0000) 2019-10-13T11:17:43.978 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:43.972+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.977028+0000) 2019-10-13T11:17:43.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:43.975+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.978490+0000) 2019-10-13T11:17:43.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:43.976+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.978990+0000) 2019-10-13T11:17:43.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:43.976+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.980015+0000) 2019-10-13T11:17:43.981 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:43.976+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.979770+0000) 2019-10-13T11:17:43.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:43.976+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.980538+0000) 2019-10-13T11:17:43.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:43.976+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.981406+0000) 2019-10-13T11:17:43.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:43.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:43.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:43.983+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:43.986887+0000) 2019-10-13T11:17:44.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:44.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:44.124672+0000) 2019-10-13T11:17:44.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:44.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:44.411320+0000) 2019-10-13T11:17:47.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:47.780+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:47.783285+0000) 2019-10-13T11:17:48.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:48.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.552609+0000) 2019-10-13T11:17:48.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:48.984+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.987000+0000) 2019-10-13T11:17:48.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:48.984+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.987744+0000) 2019-10-13T11:17:48.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:48.984+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.988277+0000) 2019-10-13T11:17:48.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:48.987+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.988260+0000) 2019-10-13T11:17:48.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:48.984+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.989238+0000) 2019-10-13T11:17:48.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:48.984+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.989612+0000) 2019-10-13T11:17:48.991 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:48.988+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.990129+0000) 2019-10-13T11:17:48.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:48.991+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:48.992942+0000) 2019-10-13T11:17:48.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:48.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135473 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:50.074 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:17:50.075 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:50.078 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:17:50.078 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:17:51.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:51.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:51.260590+0000) 2019-10-13T11:17:51.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:51.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:51.281769+0000) 2019-10-13T11:17:51.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:51.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:51.804027+0000) 2019-10-13T11:17:52.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:52.155+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:52.158558+0000) 2019-10-13T11:17:53.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:53.996+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.997861+0000) 2019-10-13T11:17:53.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:53.996+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.998674+0000) 2019-10-13T11:17:53.999 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:53.996+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:53.999197+0000) 2019-10-13T11:17:54.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:53.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.000826+0000) 2019-10-13T11:17:54.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:53.996+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.001230+0000) 2019-10-13T11:17:54.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:54.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.004946+0000) 2019-10-13T11:17:54.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:54.000+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.005271+0000) 2019-10-13T11:17:54.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:54.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:54.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:54.007+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.009589+0000) 2019-10-13T11:17:54.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:54.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.124866+0000) 2019-10-13T11:17:54.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:54.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:54.411532+0000) 2019-10-13T11:17:57.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:57.780+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:57.783497+0000) 2019-10-13T11:17:58.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:58.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:58.552833+0000) 2019-10-13T11:17:59.009 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:17:59.004+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.008680+0000) 2019-10-13T11:17:59.011 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:17:59.008+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.010961+0000) 2019-10-13T11:17:59.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:17:59.008+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.011297+0000) 2019-10-13T11:17:59.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:17:59.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.011556+0000) 2019-10-13T11:17:59.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:17:59.008+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.012388+0000) 2019-10-13T11:17:59.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:17:59.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.012972+0000) 2019-10-13T11:17:59.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:17:59.008+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.013354+0000) 2019-10-13T11:17:59.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:17:59.012+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:17:59.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:17:59.015+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:17:59.016964+0000) 2019-10-13T11:18:01.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:01.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:01.260788+0000) 2019-10-13T11:18:01.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:01.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:01.281986+0000) 2019-10-13T11:18:01.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:01.803+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:01.804257+0000) 2019-10-13T11:18:02.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:02.163+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:02.164652+0000) 2019-10-13T11:18:04.021 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:04.016+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.019239+0000) 2019-10-13T11:18:04.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:04.016+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.020062+0000) 2019-10-13T11:18:04.022 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:04.016+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.020313+0000) 2019-10-13T11:18:04.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:04.016+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.020791+0000) 2019-10-13T11:18:04.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:04.019+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.021408+0000) 2019-10-13T11:18:04.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:04.016+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.021263+0000) 2019-10-13T11:18:04.023 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:04.020+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.022844+0000) 2019-10-13T11:18:04.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:04.024+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:04.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:04.027+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.028258+0000) 2019-10-13T11:18:04.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:04.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.125042+0000) 2019-10-13T11:18:04.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:04.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:04.411667+0000) 2019-10-13T11:18:07.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:07.780+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:07.783697+0000) 2019-10-13T11:18:08.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:08.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:08.553066+0000) 2019-10-13T11:18:09.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:09.024+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.029386+0000) 2019-10-13T11:18:09.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:09.028+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.029859+0000) 2019-10-13T11:18:09.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:09.028+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.030883+0000) 2019-10-13T11:18:09.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:09.028+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.031019+0000) 2019-10-13T11:18:09.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:09.028+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.031857+0000) 2019-10-13T11:18:09.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:09.031+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.032196+0000) 2019-10-13T11:18:09.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:09.028+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.033104+0000) 2019-10-13T11:18:09.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:09.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:09.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:09.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:09.038291+0000) 2019-10-13T11:18:11.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:11.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:11.260988+0000) 2019-10-13T11:18:11.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:11.280+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:11.282186+0000) 2019-10-13T11:18:11.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:11.803+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:11.804490+0000) 2019-10-13T11:18:12.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:12.167+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:12.171134+0000) 2019-10-13T11:18:14.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:14.036+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.040106+0000) 2019-10-13T11:18:14.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:14.038+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.040588+0000) 2019-10-13T11:18:14.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:14.036+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.041255+0000) 2019-10-13T11:18:14.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:14.040+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.042016+0000) 2019-10-13T11:18:14.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:14.040+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.042276+0000) 2019-10-13T11:18:14.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:14.040+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.042745+0000) 2019-10-13T11:18:14.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:14.040+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.043017+0000) 2019-10-13T11:18:14.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:14.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:14.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:14.050+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.053337+0000) 2019-10-13T11:18:14.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:14.120+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.125227+0000) 2019-10-13T11:18:14.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:14.408+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:14.411866+0000) 2019-10-13T11:18:17.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:17.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:17.783915+0000) 2019-10-13T11:18:18.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:18.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:18.553287+0000) 2019-10-13T11:18:19.050 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:19.043+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.049706+0000) 2019-10-13T11:18:19.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:19.047+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.051375+0000) 2019-10-13T11:18:19.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:19.047+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.051059+0000) 2019-10-13T11:18:19.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:19.047+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.051752+0000) 2019-10-13T11:18:19.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:19.050+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.052440+0000) 2019-10-13T11:18:19.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:19.047+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.052870+0000) 2019-10-13T11:18:19.054 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:19.047+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.053598+0000) 2019-10-13T11:18:19.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:19.051+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:19.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:19.058+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:19.062569+0000) 2019-10-13T11:18:20.196 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:18:20.313 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:20.318 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:18:20.318 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:21.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:21.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:21.261192+0000) 2019-10-13T11:18:21.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:21.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:21.282380+0000) 2019-10-13T11:18:21.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:21.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:21.804689+0000) 2019-10-13T11:18:22.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:22.174+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:22.177781+0000) 2019-10-13T11:18:24.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:24.055+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.059599+0000) 2019-10-13T11:18:24.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:24.055+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.060276+0000) 2019-10-13T11:18:24.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:24.055+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.060647+0000) 2019-10-13T11:18:24.062 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:24.058+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.061135+0000) 2019-10-13T11:18:24.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:24.055+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.060951+0000) 2019-10-13T11:18:24.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:24.059+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.062198+0000) 2019-10-13T11:18:24.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:24.059+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.062653+0000) 2019-10-13T11:18:24.066 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:24.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:24.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:24.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.068866+0000) 2019-10-13T11:18:24.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:24.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.125441+0000) 2019-10-13T11:18:24.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:24.407+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:24.412068+0000) 2019-10-13T11:18:27.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:27.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:27.784145+0000) 2019-10-13T11:18:28.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:28.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:28.553491+0000) 2019-10-13T11:18:29.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:29.063+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.069447+0000) 2019-10-13T11:18:29.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:29.067+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.070149+0000) 2019-10-13T11:18:29.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:29.067+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.070385+0000) 2019-10-13T11:18:29.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:29.067+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.070908+0000) 2019-10-13T11:18:29.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:29.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.071476+0000) 2019-10-13T11:18:29.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:29.066+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.071551+0000) 2019-10-13T11:18:29.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:29.067+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.072114+0000) 2019-10-13T11:18:29.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:29.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:29.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:29.074+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:29.078495+0000) 2019-10-13T11:18:31.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:31.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:31.261398+0000) 2019-10-13T11:18:31.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:31.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:31.282577+0000) 2019-10-13T11:18:31.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:31.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:31.804886+0000) 2019-10-13T11:18:32.185 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:32.182+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:32.184318+0000) 2019-10-13T11:18:34.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:34.075+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.080689+0000) 2019-10-13T11:18:34.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:34.075+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.081556+0000) 2019-10-13T11:18:34.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:34.079+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.081829+0000) 2019-10-13T11:18:34.083 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:34.078+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.082070+0000) 2019-10-13T11:18:34.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:34.079+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.084262+0000) 2019-10-13T11:18:34.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:34.083+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.086161+0000) 2019-10-13T11:18:34.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:34.083+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:34.088 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:34.083+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.088133+0000) 2019-10-13T11:18:34.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:34.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.089740+0000) 2019-10-13T11:18:34.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:34.119+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.125621+0000) 2019-10-13T11:18:34.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:34.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:34.412242+0000) 2019-10-13T11:18:37.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:37.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:37.784370+0000) 2019-10-13T11:18:38.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:38.547+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:38.553676+0000) 2019-10-13T11:18:39.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:39.087+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.091321+0000) 2019-10-13T11:18:39.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:39.087+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.092399+0000) 2019-10-13T11:18:39.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:39.087+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.092987+0000) 2019-10-13T11:18:39.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:39.087+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.093399+0000) 2019-10-13T11:18:39.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:39.091+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.093901+0000) 2019-10-13T11:18:39.095 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:39.090+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.094120+0000) 2019-10-13T11:18:39.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:39.091+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.094848+0000) 2019-10-13T11:18:39.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:39.095+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:39.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:39.094+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:39.099554+0000) 2019-10-13T11:18:41.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:41.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:41.261596+0000) 2019-10-13T11:18:41.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:41.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:41.282807+0000) 2019-10-13T11:18:41.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:41.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:41.805083+0000) 2019-10-13T11:18:42.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:42.186+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:42.190873+0000) 2019-10-13T11:18:44.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:44.095+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.101025+0000) 2019-10-13T11:18:44.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:44.099+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.104383+0000) 2019-10-13T11:18:44.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:44.099+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.104800+0000) 2019-10-13T11:18:44.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:44.099+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.105206+0000) 2019-10-13T11:18:44.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:44.102+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.105804+0000) 2019-10-13T11:18:44.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:44.103+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.106487+0000) 2019-10-13T11:18:44.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:44.103+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.106996+0000) 2019-10-13T11:18:44.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:44.106+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.108213+0000) 2019-10-13T11:18:44.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:44.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:44.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:44.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.125755+0000) 2019-10-13T11:18:44.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:44.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:44.412443+0000) 2019-10-13T11:18:47.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:47.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:47.784568+0000) 2019-10-13T11:18:48.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:48.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:48.553935+0000) 2019-10-13T11:18:49.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:49.106+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.111389+0000) 2019-10-13T11:18:49.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:49.107+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.113199+0000) 2019-10-13T11:18:49.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:49.111+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.114098+0000) 2019-10-13T11:18:49.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:49.111+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.114645+0000) 2019-10-13T11:18:49.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:49.111+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.116375+0000) 2019-10-13T11:18:49.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:49.115+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.118349+0000) 2019-10-13T11:18:49.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:49.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:49.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:49.115+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.118948+0000) 2019-10-13T11:18:49.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:49.114+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:49.119351+0000) 2019-10-13T11:18:50.387 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:18:50.387 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:50.391 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:18:50.391 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:18:51.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:51.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:51.261825+0000) 2019-10-13T11:18:51.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:51.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:51.283002+0000) 2019-10-13T11:18:51.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:51.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:51.805283+0000) 2019-10-13T11:18:52.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:52.194+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:52.197363+0000) 2019-10-13T11:18:54.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:54.118+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.121253+0000) 2019-10-13T11:18:54.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:54.119+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.122364+0000) 2019-10-13T11:18:54.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:54.119+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.123074+0000) 2019-10-13T11:18:54.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:54.119+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.124935+0000) 2019-10-13T11:18:54.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:54.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.125875+0000) 2019-10-13T11:18:54.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:54.119+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.124565+0000) 2019-10-13T11:18:54.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:54.119+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.125285+0000) 2019-10-13T11:18:54.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:54.123+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.125875+0000) 2019-10-13T11:18:54.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:54.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:18:54.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:54.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.129358+0000) 2019-10-13T11:18:54.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:54.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:54.412635+0000) 2019-10-13T11:18:57.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:57.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:57.784767+0000) 2019-10-13T11:18:58.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:58.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:58.554138+0000) 2019-10-13T11:18:59.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:18:59.131+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.134447+0000) 2019-10-13T11:18:59.136 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:18:59.131+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.134473+0000) 2019-10-13T11:18:59.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:18:59.131+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.135272+0000) 2019-10-13T11:18:59.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:18:59.131+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.136025+0000) 2019-10-13T11:18:59.137 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:18:59.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.136359+0000) 2019-10-13T11:18:59.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:18:59.131+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.136862+0000) 2019-10-13T11:18:59.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:18:59.135+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.138106+0000) 2019-10-13T11:18:59.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:18:59.134+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:18:59.138320+0000) 2019-10-13T11:18:59.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:18:59.135+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:01.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:01.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:01.261989+0000) 2019-10-13T11:19:01.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:01.279+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:01.283195+0000) 2019-10-13T11:19:01.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:01.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:01.805485+0000) 2019-10-13T11:19:02.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:02.198+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:02.203854+0000) 2019-10-13T11:19:04.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:04.123+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.126057+0000) 2019-10-13T11:19:04.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:04.135+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.141650+0000) 2019-10-13T11:19:04.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:04.138+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.141897+0000) 2019-10-13T11:19:04.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:04.139+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.142654+0000) 2019-10-13T11:19:04.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:04.139+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.142833+0000) 2019-10-13T11:19:04.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:04.139+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.143689+0000) 2019-10-13T11:19:04.145 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:04.139+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.144318+0000) 2019-10-13T11:19:04.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:04.139+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.145094+0000) 2019-10-13T11:19:04.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:04.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:04.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:04.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.151892+0000) 2019-10-13T11:19:04.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:04.407+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:04.412849+0000) 2019-10-13T11:19:07.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:07.779+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:07.784988+0000) 2019-10-13T11:19:08.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:08.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:08.554328+0000) 2019-10-13T11:19:09.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:09.147+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.151740+0000) 2019-10-13T11:19:09.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:09.151+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.153935+0000) 2019-10-13T11:19:09.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:09.151+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.155970+0000) 2019-10-13T11:19:09.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:09.151+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.156354+0000) 2019-10-13T11:19:09.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:09.151+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.157718+0000) 2019-10-13T11:19:09.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:09.155+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.158392+0000) 2019-10-13T11:19:09.160 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:09.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.159042+0000) 2019-10-13T11:19:09.160 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:09.155+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:09.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:09.162+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:09.165290+0000) 2019-10-13T11:19:11.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:11.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:11.262188+0000) 2019-10-13T11:19:11.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:11.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:11.283419+0000) 2019-10-13T11:19:11.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:11.802+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:11.805696+0000) 2019-10-13T11:19:12.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:12.210+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:12.210593+0000) 2019-10-13T11:19:14.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:14.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.126236+0000) 2019-10-13T11:19:14.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:14.158+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.162622+0000) 2019-10-13T11:19:14.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:14.158+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.163369+0000) 2019-10-13T11:19:14.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:14.158+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.164894+0000) 2019-10-13T11:19:14.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:14.158+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.165415+0000) 2019-10-13T11:19:14.168 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:14.162+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.167098+0000) 2019-10-13T11:19:14.168 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:14.162+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.167685+0000) 2019-10-13T11:19:14.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:14.162+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.168866+0000) 2019-10-13T11:19:14.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:14.162+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:14.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:14.170+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.174081+0000) 2019-10-13T11:19:14.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:14.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:14.413028+0000) 2019-10-13T11:19:17.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:17.778+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:17.785170+0000) 2019-10-13T11:19:18.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:18.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:18.554536+0000) 2019-10-13T11:19:19.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:19.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.172813+0000) 2019-10-13T11:19:19.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:19.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.172643+0000) 2019-10-13T11:19:19.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:19.166+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.173619+0000) 2019-10-13T11:19:19.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:19.170+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.174017+0000) 2019-10-13T11:19:19.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:19.170+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.174569+0000) 2019-10-13T11:19:19.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:19.170+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.174924+0000) 2019-10-13T11:19:19.176 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:19.170+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.175393+0000) 2019-10-13T11:19:19.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:19.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:19.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:19.178+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:19.183708+0000) 2019-10-13T11:19:20.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:19:20.494 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:20.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:19:20.498 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:21.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:21.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:21.262405+0000) 2019-10-13T11:19:21.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:21.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:21.283631+0000) 2019-10-13T11:19:21.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:21.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:21.805896+0000) 2019-10-13T11:19:22.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:22.213+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:22.216654+0000) 2019-10-13T11:19:24.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:24.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.126440+0000) 2019-10-13T11:19:24.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:24.178+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.183709+0000) 2019-10-13T11:19:24.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:24.178+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.184084+0000) 2019-10-13T11:19:24.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:24.182+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.186786+0000) 2019-10-13T11:19:24.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:24.182+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.187568+0000) 2019-10-13T11:19:24.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:24.181+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.187613+0000) 2019-10-13T11:19:24.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:24.182+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.188148+0000) 2019-10-13T11:19:24.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:24.182+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.188559+0000) 2019-10-13T11:19:24.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:24.186+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:24.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:24.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.192178+0000) 2019-10-13T11:19:24.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:24.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:24.413212+0000) 2019-10-13T11:19:27.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:27.778+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:27.785377+0000) 2019-10-13T11:19:28.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:28.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:28.554760+0000) 2019-10-13T11:19:29.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:29.186+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.193104+0000) 2019-10-13T11:19:29.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:29.189+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.194389+0000) 2019-10-13T11:19:29.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:29.190+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.195533+0000) 2019-10-13T11:19:29.198 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:29.190+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.197302+0000) 2019-10-13T11:19:29.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:29.194+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.199479+0000) 2019-10-13T11:19:29.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:29.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:29.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:29.198+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.201808+0000) 2019-10-13T11:19:29.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:29.198+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.202287+0000) 2019-10-13T11:19:29.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:29.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:29.204075+0000) 2019-10-13T11:19:31.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:31.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:31.262626+0000) 2019-10-13T11:19:31.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:31.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:31.283837+0000) 2019-10-13T11:19:31.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:31.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:31.806104+0000) 2019-10-13T11:19:32.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:32.221+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:32.223279+0000) 2019-10-13T11:19:34.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:34.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.126640+0000) 2019-10-13T11:19:34.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:34.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.203623+0000) 2019-10-13T11:19:34.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:34.198+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.204248+0000) 2019-10-13T11:19:34.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:34.198+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.205671+0000) 2019-10-13T11:19:34.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:34.202+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.206073+0000) 2019-10-13T11:19:34.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:34.202+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.207329+0000) 2019-10-13T11:19:34.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:34.202+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.209177+0000) 2019-10-13T11:19:34.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:34.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:34.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:34.205+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.210820+0000) 2019-10-13T11:19:34.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:34.206+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.211394+0000) 2019-10-13T11:19:34.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:34.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:34.413445+0000) 2019-10-13T11:19:37.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:37.778+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:37.785573+0000) 2019-10-13T11:19:38.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:38.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:38.554945+0000) 2019-10-13T11:19:39.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:39.210+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.215353+0000) 2019-10-13T11:19:39.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:39.210+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.215164+0000) 2019-10-13T11:19:39.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:39.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.215932+0000) 2019-10-13T11:19:39.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:39.210+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.216603+0000) 2019-10-13T11:19:39.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:39.210+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.216719+0000) 2019-10-13T11:19:39.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:39.210+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.217201+0000) 2019-10-13T11:19:39.218 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:39.214+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.217864+0000) 2019-10-13T11:19:39.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:39.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:39.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:39.217+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:39.220496+0000) 2019-10-13T11:19:41.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:41.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:41.262821+0000) 2019-10-13T11:19:41.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:41.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:41.283991+0000) 2019-10-13T11:19:41.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:41.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:41.806309+0000) 2019-10-13T11:19:42.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:42.229+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:42.234576+0000) 2019-10-13T11:19:44.127 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:44.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.126845+0000) 2019-10-13T11:19:44.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:44.218+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.224472+0000) 2019-10-13T11:19:44.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:44.221+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.224814+0000) 2019-10-13T11:19:44.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:44.222+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.227408+0000) 2019-10-13T11:19:44.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:44.222+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.228932+0000) 2019-10-13T11:19:44.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:44.222+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.229074+0000) 2019-10-13T11:19:44.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:44.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:44.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:44.226+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.231052+0000) 2019-10-13T11:19:44.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:44.226+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.231414+0000) 2019-10-13T11:19:44.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:44.229+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.233181+0000) 2019-10-13T11:19:44.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:44.406+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:44.413652+0000) 2019-10-13T11:19:47.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:47.782+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:47.785786+0000) 2019-10-13T11:19:48.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:48.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:48.555171+0000) 2019-10-13T11:19:49.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:49.230+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.236469+0000) 2019-10-13T11:19:49.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:49.230+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.236785+0000) 2019-10-13T11:19:49.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:49.230+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.237272+0000) 2019-10-13T11:19:49.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:49.234+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.238176+0000) 2019-10-13T11:19:49.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:49.234+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.238132+0000) 2019-10-13T11:19:49.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:49.233+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.238378+0000) 2019-10-13T11:19:49.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:49.234+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.239104+0000) 2019-10-13T11:19:49.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:49.234+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136204 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:49.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:49.237+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:49.240906+0000) 2019-10-13T11:19:50.626 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:19:50.626 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:50.630 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:19:50.630 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:19:51.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:51.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:51.263019+0000) 2019-10-13T11:19:51.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:51.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:51.284190+0000) 2019-10-13T11:19:51.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:51.805+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:51.806508+0000) 2019-10-13T11:19:52.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:52.237+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:52.241042+0000) 2019-10-13T11:19:54.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:54.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.127042+0000) 2019-10-13T11:19:54.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:54.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.243148+0000) 2019-10-13T11:19:54.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:54.238+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.244450+0000) 2019-10-13T11:19:54.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:54.238+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.245351+0000) 2019-10-13T11:19:54.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:54.242+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.247391+0000) 2019-10-13T11:19:54.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:54.242+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.247481+0000) 2019-10-13T11:19:54.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:54.246+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.250635+0000) 2019-10-13T11:19:54.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:54.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:54.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:54.246+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.251732+0000) 2019-10-13T11:19:54.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:54.249+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.254443+0000) 2019-10-13T11:19:54.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:54.410+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:54.413838+0000) 2019-10-13T11:19:57.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:57.782+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:57.785984+0000) 2019-10-13T11:19:58.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:58.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:58.555406+0000) 2019-10-13T11:19:59.256 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:19:59.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.254821+0000) 2019-10-13T11:19:59.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:19:59.250+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.255277+0000) 2019-10-13T11:19:59.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:19:59.250+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.255796+0000) 2019-10-13T11:19:59.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:19:59.254+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.257903+0000) 2019-10-13T11:19:59.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:19:59.254+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.259484+0000) 2019-10-13T11:19:59.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:19:59.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:19:59.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:19:59.254+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.260987+0000) 2019-10-13T11:19:59.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:19:59.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.261782+0000) 2019-10-13T11:19:59.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:19:59.261+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:19:59.265790+0000) 2019-10-13T11:20:01.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:01.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:01.263255+0000) 2019-10-13T11:20:01.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:01.278+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:01.284392+0000) 2019-10-13T11:20:01.807 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:01.805+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:01.806710+0000) 2019-10-13T11:20:02.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:02.241+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:02.247626+0000) 2019-10-13T11:20:04.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:04.122+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.127289+0000) 2019-10-13T11:20:04.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:04.257+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.264016+0000) 2019-10-13T11:20:04.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:04.258+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.264794+0000) 2019-10-13T11:20:04.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:04.262+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.266259+0000) 2019-10-13T11:20:04.267 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:04.262+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.266091+0000) 2019-10-13T11:20:04.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:04.262+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.266801+0000) 2019-10-13T11:20:04.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:04.262+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.267223+0000) 2019-10-13T11:20:04.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:04.262+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.267704+0000) 2019-10-13T11:20:04.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:04.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:04.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:04.265+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.271142+0000) 2019-10-13T11:20:04.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:04.410+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:04.414023+0000) 2019-10-13T11:20:07.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:07.782+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:07.786216+0000) 2019-10-13T11:20:08.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:08.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:08.555602+0000) 2019-10-13T11:20:09.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:09.269+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.275012+0000) 2019-10-13T11:20:09.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:09.269+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.275895+0000) 2019-10-13T11:20:09.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:09.273+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.278103+0000) 2019-10-13T11:20:09.279 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:09.273+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.278570+0000) 2019-10-13T11:20:09.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:09.273+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.279357+0000) 2019-10-13T11:20:09.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:09.273+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.280779+0000) 2019-10-13T11:20:09.281 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:09.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:09.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:09.273+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.281580+0000) 2019-10-13T11:20:09.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:09.277+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:09.283122+0000) 2019-10-13T11:20:11.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:11.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:11.263463+0000) 2019-10-13T11:20:11.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:11.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:11.284588+0000) 2019-10-13T11:20:11.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:11.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:11.806910+0000) 2019-10-13T11:20:12.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:12.253+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:12.254103+0000) 2019-10-13T11:20:14.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:14.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.127487+0000) 2019-10-13T11:20:14.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:14.277+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.283956+0000) 2019-10-13T11:20:14.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:14.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.284196+0000) 2019-10-13T11:20:14.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:14.281+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.284436+0000) 2019-10-13T11:20:14.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:14.277+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.285035+0000) 2019-10-13T11:20:14.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:14.277+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.285312+0000) 2019-10-13T11:20:14.287 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:14.281+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.286192+0000) 2019-10-13T11:20:14.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:14.281+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.287755+0000) 2019-10-13T11:20:14.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:14.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:14.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:14.289+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.293525+0000) 2019-10-13T11:20:14.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:14.409+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:14.414245+0000) 2019-10-13T11:20:17.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:17.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:17.786409+0000) 2019-10-13T11:20:18.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:18.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:18.555794+0000) 2019-10-13T11:20:19.295 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:19.289+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.294512+0000) 2019-10-13T11:20:19.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:19.289+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.294864+0000) 2019-10-13T11:20:19.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:19.289+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.295472+0000) 2019-10-13T11:20:19.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:19.289+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.297258+0000) 2019-10-13T11:20:19.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:19.293+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.298691+0000) 2019-10-13T11:20:19.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:19.293+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.299643+0000) 2019-10-13T11:20:19.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:19.293+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.300170+0000) 2019-10-13T11:20:19.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:19.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:19.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:19.297+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:19.303020+0000) 2019-10-13T11:20:20.748 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:20:20.749 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:20.753 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:20:20.753 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:21.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:21.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:21.263683+0000) 2019-10-13T11:20:21.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:21.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:21.284786+0000) 2019-10-13T11:20:21.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:21.805+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:21.807123+0000) 2019-10-13T11:20:22.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:22.261+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:22.264940+0000) 2019-10-13T11:20:24.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:24.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.127706+0000) 2019-10-13T11:20:24.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:24.301+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.303798+0000) 2019-10-13T11:20:24.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:24.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.304563+0000) 2019-10-13T11:20:24.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:24.301+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.305039+0000) 2019-10-13T11:20:24.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:24.301+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.305450+0000) 2019-10-13T11:20:24.306 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:24.301+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.305637+0000) 2019-10-13T11:20:24.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:24.305+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.306144+0000) 2019-10-13T11:20:24.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:24.305+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.306374+0000) 2019-10-13T11:20:24.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:24.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:24.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:24.309+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.313515+0000) 2019-10-13T11:20:24.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:24.413+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:24.414439+0000) 2019-10-13T11:20:27.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:27.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:27.786627+0000) 2019-10-13T11:20:28.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:28.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:28.556015+0000) 2019-10-13T11:20:29.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:29.313+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.314597+0000) 2019-10-13T11:20:29.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:29.308+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.314593+0000) 2019-10-13T11:20:29.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:29.313+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.315955+0000) 2019-10-13T11:20:29.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:29.313+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.316500+0000) 2019-10-13T11:20:29.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:29.313+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.316880+0000) 2019-10-13T11:20:29.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:29.313+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.317343+0000) 2019-10-13T11:20:29.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:29.313+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.317693+0000) 2019-10-13T11:20:29.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:29.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:29.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:29.320+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:29.324646+0000) 2019-10-13T11:20:31.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:31.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:31.263887+0000) 2019-10-13T11:20:31.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:31.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:31.284989+0000) 2019-10-13T11:20:31.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:31.800+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:31.807316+0000) 2019-10-13T11:20:32.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:32.268+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:32.271371+0000) 2019-10-13T11:20:34.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:34.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.127936+0000) 2019-10-13T11:20:34.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:34.321+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.324913+0000) 2019-10-13T11:20:34.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:34.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.325352+0000) 2019-10-13T11:20:34.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:34.321+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.325698+0000) 2019-10-13T11:20:34.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:34.325+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.326439+0000) 2019-10-13T11:20:34.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:34.320+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.327732+0000) 2019-10-13T11:20:34.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:34.325+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.328210+0000) 2019-10-13T11:20:34.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:34.325+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.328491+0000) 2019-10-13T11:20:34.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:34.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136480 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:34.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:34.328+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.334099+0000) 2019-10-13T11:20:34.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:34.413+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:34.414637+0000) 2019-10-13T11:20:37.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:37.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:37.786855+0000) 2019-10-13T11:20:38.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:38.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:38.556210+0000) 2019-10-13T11:20:39.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:39.333+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.334385+0000) 2019-10-13T11:20:39.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:39.333+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.334863+0000) 2019-10-13T11:20:39.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:39.333+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.336885+0000) 2019-10-13T11:20:39.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:39.333+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.336087+0000) 2019-10-13T11:20:39.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:39.333+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.337110+0000) 2019-10-13T11:20:39.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:39.333+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.337564+0000) 2019-10-13T11:20:39.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:39.332+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.337770+0000) 2019-10-13T11:20:39.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:39.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:39.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:39.340+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:39.344471+0000) 2019-10-13T11:20:41.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:41.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:41.264090+0000) 2019-10-13T11:20:41.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:41.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:41.285217+0000) 2019-10-13T11:20:41.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:41.804+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:41.807507+0000) 2019-10-13T11:20:42.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:42.276+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:42.277870+0000) 2019-10-13T11:20:44.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:44.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.128166+0000) 2019-10-13T11:20:44.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:44.341+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.345068+0000) 2019-10-13T11:20:44.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:44.340+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.345080+0000) 2019-10-13T11:20:44.347 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:44.341+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.345608+0000) 2019-10-13T11:20:44.348 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:44.345+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.347566+0000) 2019-10-13T11:20:44.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:44.345+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.347862+0000) 2019-10-13T11:20:44.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:44.345+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.348103+0000) 2019-10-13T11:20:44.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:44.345+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.345879+0000) 2019-10-13T11:20:44.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:44.344+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.350675+0000) 2019-10-13T11:20:44.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:44.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:44.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:44.413+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:44.414817+0000) 2019-10-13T11:20:47.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:47.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:47.787057+0000) 2019-10-13T11:20:48.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:48.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:48.556402+0000) 2019-10-13T11:20:49.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:49.353+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.355349+0000) 2019-10-13T11:20:49.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:49.353+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.356031+0000) 2019-10-13T11:20:49.359 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:49.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.356350+0000) 2019-10-13T11:20:49.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:49.352+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.357973+0000) 2019-10-13T11:20:49.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:49.353+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.357257+0000) 2019-10-13T11:20:49.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:49.353+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.357376+0000) 2019-10-13T11:20:49.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:49.357+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.358646+0000) 2019-10-13T11:20:49.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:49.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:49.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:49.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:49.364582+0000) 2019-10-13T11:20:51.078 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:20:51.079 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:51.083 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:20:51.083 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:20:51.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:51.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:51.264292+0000) 2019-10-13T11:20:51.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:51.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:51.285410+0000) 2019-10-13T11:20:51.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:51.804+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:51.807703+0000) 2019-10-13T11:20:52.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:52.280+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:52.284431+0000) 2019-10-13T11:20:54.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:54.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.128381+0000) 2019-10-13T11:20:54.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:54.360+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.365108+0000) 2019-10-13T11:20:54.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:54.361+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.365634+0000) 2019-10-13T11:20:54.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:54.365+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.366258+0000) 2019-10-13T11:20:54.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:54.365+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.366625+0000) 2019-10-13T11:20:54.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:54.365+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.367538+0000) 2019-10-13T11:20:54.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:54.365+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.367913+0000) 2019-10-13T11:20:54.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:54.365+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.368327+0000) 2019-10-13T11:20:54.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:54.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:54.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:54.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.372717+0000) 2019-10-13T11:20:54.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:54.413+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:54.414974+0000) 2019-10-13T11:20:57.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:57.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:57.787251+0000) 2019-10-13T11:20:58.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:58.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:58.556604+0000) 2019-10-13T11:20:59.379 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:20:59.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.376234+0000) 2019-10-13T11:20:59.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:20:59.373+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.375043+0000) 2019-10-13T11:20:59.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:20:59.373+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.376174+0000) 2019-10-13T11:20:59.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:20:59.373+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.376682+0000) 2019-10-13T11:20:59.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:20:59.373+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.377512+0000) 2019-10-13T11:20:59.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:20:59.373+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.377672+0000) 2019-10-13T11:20:59.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:20:59.377+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.377842+0000) 2019-10-13T11:20:59.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:20:59.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:20:59.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:20:59.380+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:20:59.384601+0000) 2019-10-13T11:21:01.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:01.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:01.264538+0000) 2019-10-13T11:21:01.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:01.281+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:01.285641+0000) 2019-10-13T11:21:01.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:01.804+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:01.807907+0000) 2019-10-13T11:21:02.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:02.284+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:02.291071+0000) 2019-10-13T11:21:04.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:04.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.128612+0000) 2019-10-13T11:21:04.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:04.381+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.385471+0000) 2019-10-13T11:21:04.387 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:04.385+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.385920+0000) 2019-10-13T11:21:04.388 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:04.380+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.387021+0000) 2019-10-13T11:21:04.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:04.385+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.387545+0000) 2019-10-13T11:21:04.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:04.385+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.389714+0000) 2019-10-13T11:21:04.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:04.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:04.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:04.389+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.391675+0000) 2019-10-13T11:21:04.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:04.389+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.392249+0000) 2019-10-13T11:21:04.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:04.392+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.397575+0000) 2019-10-13T11:21:04.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:04.413+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:04.415117+0000) 2019-10-13T11:21:07.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:07.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:07.787450+0000) 2019-10-13T11:21:08.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:08.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:08.556795+0000) 2019-10-13T11:21:09.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:09.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.395261+0000) 2019-10-13T11:21:09.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:09.392+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.396049+0000) 2019-10-13T11:21:09.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:09.392+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.396929+0000) 2019-10-13T11:21:09.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:09.392+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.397430+0000) 2019-10-13T11:21:09.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:09.392+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.397653+0000) 2019-10-13T11:21:09.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:09.396+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.398228+0000) 2019-10-13T11:21:09.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:09.396+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.398652+0000) 2019-10-13T11:21:09.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:09.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:09.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:09.400+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:09.406428+0000) 2019-10-13T11:21:11.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:11.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:11.264745+0000) 2019-10-13T11:21:11.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:11.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:11.285902+0000) 2019-10-13T11:21:11.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:11.804+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:11.808138+0000) 2019-10-13T11:21:12.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:12.296+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:12.297627+0000) 2019-10-13T11:21:14.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:14.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.128807+0000) 2019-10-13T11:21:14.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:14.400+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.405277+0000) 2019-10-13T11:21:14.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:14.400+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.405694+0000) 2019-10-13T11:21:14.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:14.404+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.406791+0000) 2019-10-13T11:21:14.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:14.404+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.407661+0000) 2019-10-13T11:21:14.408 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:14.400+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.407994+0000) 2019-10-13T11:21:14.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:14.404+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.408319+0000) 2019-10-13T11:21:14.409 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:14.404+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.408789+0000) 2019-10-13T11:21:14.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:14.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:14.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:14.408+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.413521+0000) 2019-10-13T11:21:14.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:14.412+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:14.415252+0000) 2019-10-13T11:21:17.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:17.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:17.787650+0000) 2019-10-13T11:21:18.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:18.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:18.556985+0000) 2019-10-13T11:21:19.415 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:19.408+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.414246+0000) 2019-10-13T11:21:19.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:19.412+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.415931+0000) 2019-10-13T11:21:19.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:19.412+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.416547+0000) 2019-10-13T11:21:19.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:19.412+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.416945+0000) 2019-10-13T11:21:19.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:19.412+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.417145+0000) 2019-10-13T11:21:19.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:19.412+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.417476+0000) 2019-10-13T11:21:19.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:19.416+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.417831+0000) 2019-10-13T11:21:19.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:19.416+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:19.421514+0000) 2019-10-13T11:21:19.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:19.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:21.150 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:21:21.150 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:21.154 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:21:21.154 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:21.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:21.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:21.264951+0000) 2019-10-13T11:21:21.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:21.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:21.286125+0000) 2019-10-13T11:21:21.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:21.808+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:21.808337+0000) 2019-10-13T11:21:22.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:22.300+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:22.304152+0000) 2019-10-13T11:21:24.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:24.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.129003+0000) 2019-10-13T11:21:24.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:24.412+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.415970+0000) 2019-10-13T11:21:24.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:24.420+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.425234+0000) 2019-10-13T11:21:24.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:24.424+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.426085+0000) 2019-10-13T11:21:24.427 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:24.424+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.426556+0000) 2019-10-13T11:21:24.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:24.428+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.431484+0000) 2019-10-13T11:21:24.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:24.424+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.431726+0000) 2019-10-13T11:21:24.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:24.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:24.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:24.428+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.433076+0000) 2019-10-13T11:21:24.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:24.428+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.433368+0000) 2019-10-13T11:21:24.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:24.428+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:24.433632+0000) 2019-10-13T11:21:27.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:27.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:27.787876+0000) 2019-10-13T11:21:28.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:28.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:28.557176+0000) 2019-10-13T11:21:29.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:29.432+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.436539+0000) 2019-10-13T11:21:29.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:29.432+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.437116+0000) 2019-10-13T11:21:29.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:29.436+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.439202+0000) 2019-10-13T11:21:29.440 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:29.436+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.439118+0000) 2019-10-13T11:21:29.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:29.436+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.441364+0000) 2019-10-13T11:21:29.442 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:29.440+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.441044+0000) 2019-10-13T11:21:29.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:29.440+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.442225+0000) 2019-10-13T11:21:29.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:29.440+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:29.442588+0000) 2019-10-13T11:21:29.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:29.440+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:31.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:31.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:31.265159+0000) 2019-10-13T11:21:31.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:31.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:31.286323+0000) 2019-10-13T11:21:31.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:31.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:31.808556+0000) 2019-10-13T11:21:32.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:32.307+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:32.310635+0000) 2019-10-13T11:21:34.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:34.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.129200+0000) 2019-10-13T11:21:34.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:34.412+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.416174+0000) 2019-10-13T11:21:34.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:34.444+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.445979+0000) 2019-10-13T11:21:34.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:34.444+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.446322+0000) 2019-10-13T11:21:34.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:34.444+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.446614+0000) 2019-10-13T11:21:34.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:34.444+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.447261+0000) 2019-10-13T11:21:34.449 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:34.443+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.447880+0000) 2019-10-13T11:21:34.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:34.444+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.448646+0000) 2019-10-13T11:21:34.450 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:34.444+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.449099+0000) 2019-10-13T11:21:34.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:34.452+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:34.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:34.451+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:34.454482+0000) 2019-10-13T11:21:37.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:37.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:37.788076+0000) 2019-10-13T11:21:38.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:38.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:38.557389+0000) 2019-10-13T11:21:39.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:39.452+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.456781+0000) 2019-10-13T11:21:39.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:39.452+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.457565+0000) 2019-10-13T11:21:39.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:39.456+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.458235+0000) 2019-10-13T11:21:39.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:39.452+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.457246+0000) 2019-10-13T11:21:39.460 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:39.455+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.458328+0000) 2019-10-13T11:21:39.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:39.456+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.459005+0000) 2019-10-13T11:21:39.461 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:39.456+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.460445+0000) 2019-10-13T11:21:39.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:39.463+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:39.466141+0000) 2019-10-13T11:21:39.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:39.460+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:41.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:41.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:41.265402+0000) 2019-10-13T11:21:41.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:41.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:41.286539+0000) 2019-10-13T11:21:41.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:41.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:41.808757+0000) 2019-10-13T11:21:42.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:42.315+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:42.317245+0000) 2019-10-13T11:21:44.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:44.128+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.129409+0000) 2019-10-13T11:21:44.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:44.412+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.416386+0000) 2019-10-13T11:21:44.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:44.464+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.467486+0000) 2019-10-13T11:21:44.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:44.464+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.467802+0000) 2019-10-13T11:21:44.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:44.464+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.468906+0000) 2019-10-13T11:21:44.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:44.464+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.468950+0000) 2019-10-13T11:21:44.471 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:44.468+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.470233+0000) 2019-10-13T11:21:44.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:44.467+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.470333+0000) 2019-10-13T11:21:44.471 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:44.468+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.471230+0000) 2019-10-13T11:21:44.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:44.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:44.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:44.475+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:44.477107+0000) 2019-10-13T11:21:47.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:47.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:47.788274+0000) 2019-10-13T11:21:48.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:48.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:48.557580+0000) 2019-10-13T11:21:49.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:49.476+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.478164+0000) 2019-10-13T11:21:49.480 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:49.476+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.478560+0000) 2019-10-13T11:21:49.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:49.476+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.479095+0000) 2019-10-13T11:21:49.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:49.476+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.479851+0000) 2019-10-13T11:21:49.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:49.476+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.480153+0000) 2019-10-13T11:21:49.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:49.476+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.480679+0000) 2019-10-13T11:21:49.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:49.479+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.480936+0000) 2019-10-13T11:21:49.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:49.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:49.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:49.483+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:49.487532+0000) 2019-10-13T11:21:51.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:51.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:51.265611+0000) 2019-10-13T11:21:51.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:51.284+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:51.286706+0000) 2019-10-13T11:21:51.423 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:21:51.423 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:51.427 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:21:51.427 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:21:51.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:51.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:51.808966+0000) 2019-10-13T11:21:52.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:52.323+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:52.323867+0000) 2019-10-13T11:21:54.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:54.124+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.129619+0000) 2019-10-13T11:21:54.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:54.412+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.416573+0000) 2019-10-13T11:21:54.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:54.484+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.487886+0000) 2019-10-13T11:21:54.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:54.484+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.488331+0000) 2019-10-13T11:21:54.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:54.484+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.488794+0000) 2019-10-13T11:21:54.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:54.488+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.489730+0000) 2019-10-13T11:21:54.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:54.487+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.490172+0000) 2019-10-13T11:21:54.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:54.488+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.489992+0000) 2019-10-13T11:21:54.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:54.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:54.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:54.492+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.494903+0000) 2019-10-13T11:21:54.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:54.499+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:54.500136+0000) 2019-10-13T11:21:57.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:57.784+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:57.788472+0000) 2019-10-13T11:21:58.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:58.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:58.557800+0000) 2019-10-13T11:21:59.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:21:59.496+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.498394+0000) 2019-10-13T11:21:59.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:21:59.496+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.497991+0000) 2019-10-13T11:21:59.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:21:59.496+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.498776+0000) 2019-10-13T11:21:59.502 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:21:59.496+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.501397+0000) 2019-10-13T11:21:59.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:21:59.496+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.501599+0000) 2019-10-13T11:21:59.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:21:59.499+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.501902+0000) 2019-10-13T11:21:59.505 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:21:59.500+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.504280+0000) 2019-10-13T11:21:59.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:21:59.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:21:59.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:21:59.507+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:21:59.509250+0000) 2019-10-13T11:22:01.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:01.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:01.265809+0000) 2019-10-13T11:22:01.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:01.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:01.286908+0000) 2019-10-13T11:22:01.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:01.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:01.809179+0000) 2019-10-13T11:22:02.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:02.327+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:02.330367+0000) 2019-10-13T11:22:04.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:04.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.129825+0000) 2019-10-13T11:22:04.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:04.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.416770+0000) 2019-10-13T11:22:04.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:04.507+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.508430+0000) 2019-10-13T11:22:04.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:04.503+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.509061+0000) 2019-10-13T11:22:04.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:04.507+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.509774+0000) 2019-10-13T11:22:04.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:04.507+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.510348+0000) 2019-10-13T11:22:04.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:04.507+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.511141+0000) 2019-10-13T11:22:04.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:04.507+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.511459+0000) 2019-10-13T11:22:04.512 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:04.507+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.512181+0000) 2019-10-13T11:22:04.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:04.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:04.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:04.515+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:04.518316+0000) 2019-10-13T11:22:07.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:07.783+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:07.788688+0000) 2019-10-13T11:22:08.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:08.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:08.558037+0000) 2019-10-13T11:22:09.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:09.519+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.520281+0000) 2019-10-13T11:22:09.526 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:09.515+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.518446+0000) 2019-10-13T11:22:09.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:09.515+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.520178+0000) 2019-10-13T11:22:09.527 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:09.515+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.520865+0000) 2019-10-13T11:22:09.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:09.519+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.521810+0000) 2019-10-13T11:22:09.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:09.519+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.522417+0000) 2019-10-13T11:22:09.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:09.519+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.522438+0000) 2019-10-13T11:22:09.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:09.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:09.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:09.527+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:09.529018+0000) 2019-10-13T11:22:11.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:11.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:11.266013+0000) 2019-10-13T11:22:11.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:11.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:11.287113+0000) 2019-10-13T11:22:11.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:11.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:11.809376+0000) 2019-10-13T11:22:12.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:12.335+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:12.336817+0000) 2019-10-13T11:22:14.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:14.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.130063+0000) 2019-10-13T11:22:14.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:14.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.416996+0000) 2019-10-13T11:22:14.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:14.523+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.529709+0000) 2019-10-13T11:22:14.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:14.527+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.530493+0000) 2019-10-13T11:22:14.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:14.527+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.530657+0000) 2019-10-13T11:22:14.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:14.527+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.532328+0000) 2019-10-13T11:22:14.535 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:14.531+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.534463+0000) 2019-10-13T11:22:14.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:14.531+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.534706+0000) 2019-10-13T11:22:14.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:14.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:14.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:14.531+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.536868+0000) 2019-10-13T11:22:14.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:14.535+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:14.539008+0000) 2019-10-13T11:22:17.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:17.787+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:17.788923+0000) 2019-10-13T11:22:18.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:18.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:18.558185+0000) 2019-10-13T11:22:19.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:19.535+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.539746+0000) 2019-10-13T11:22:19.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:19.535+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.540556+0000) 2019-10-13T11:22:19.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:19.539+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.542392+0000) 2019-10-13T11:22:19.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:19.539+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.542762+0000) 2019-10-13T11:22:19.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:19.539+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.544414+0000) 2019-10-13T11:22:19.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:19.539+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.544730+0000) 2019-10-13T11:22:19.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:19.543+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.546048+0000) 2019-10-13T11:22:19.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:19.543+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:19.546400+0000) 2019-10-13T11:22:19.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:19.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:21.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:21.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:21.266258+0000) 2019-10-13T11:22:21.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:21.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:21.287342+0000) 2019-10-13T11:22:21.608 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:22:21.609 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:21.612 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:22:21.612 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:21.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:21.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:21.809590+0000) 2019-10-13T11:22:22.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:22.343+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:22.343383+0000) 2019-10-13T11:22:24.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:24.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.130297+0000) 2019-10-13T11:22:24.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:24.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.417226+0000) 2019-10-13T11:22:24.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:24.547+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.550071+0000) 2019-10-13T11:22:24.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:24.547+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.551504+0000) 2019-10-13T11:22:24.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:24.547+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.552220+0000) 2019-10-13T11:22:24.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:24.547+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.552380+0000) 2019-10-13T11:22:24.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:24.551+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.553023+0000) 2019-10-13T11:22:24.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:24.547+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.553614+0000) 2019-10-13T11:22:24.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:24.551+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.553974+0000) 2019-10-13T11:22:24.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:24.555+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137158 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:24.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:24.555+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:24.559337+0000) 2019-10-13T11:22:27.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:27.783+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:27.789122+0000) 2019-10-13T11:22:28.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:28.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:28.558381+0000) 2019-10-13T11:22:29.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:29.559+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.562632+0000) 2019-10-13T11:22:29.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:29.559+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.563385+0000) 2019-10-13T11:22:29.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:29.559+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.563956+0000) 2019-10-13T11:22:29.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:29.559+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.564502+0000) 2019-10-13T11:22:29.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:29.559+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.565603+0000) 2019-10-13T11:22:29.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:29.563+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.566820+0000) 2019-10-13T11:22:29.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:29.563+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.567348+0000) 2019-10-13T11:22:29.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:29.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:29.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:29.567+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:29.570154+0000) 2019-10-13T11:22:31.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:31.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:31.266486+0000) 2019-10-13T11:22:31.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:31.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:31.287568+0000) 2019-10-13T11:22:31.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:31.807+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:31.809814+0000) 2019-10-13T11:22:32.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:32.347+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:32.349979+0000) 2019-10-13T11:22:34.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:34.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.130502+0000) 2019-10-13T11:22:34.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:34.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.417428+0000) 2019-10-13T11:22:34.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:34.567+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.572565+0000) 2019-10-13T11:22:34.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:34.571+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.574024+0000) 2019-10-13T11:22:34.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:34.571+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.574401+0000) 2019-10-13T11:22:34.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:34.575+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.576418+0000) 2019-10-13T11:22:34.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:34.575+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.579007+0000) 2019-10-13T11:22:34.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:34.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:34.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:34.575+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.580389+0000) 2019-10-13T11:22:34.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:34.575+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.580583+0000) 2019-10-13T11:22:34.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:34.579+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:34.582101+0000) 2019-10-13T11:22:37.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:37.783+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:37.789367+0000) 2019-10-13T11:22:38.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:38.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:38.558573+0000) 2019-10-13T11:22:39.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:39.578+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.583216+0000) 2019-10-13T11:22:39.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:39.579+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.584100+0000) 2019-10-13T11:22:39.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:39.579+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.584799+0000) 2019-10-13T11:22:39.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:39.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.586341+0000) 2019-10-13T11:22:39.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:39.583+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.586628+0000) 2019-10-13T11:22:39.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:39.583+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.587407+0000) 2019-10-13T11:22:39.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:39.583+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.587613+0000) 2019-10-13T11:22:39.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:39.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:39.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:39.590+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:39.594399+0000) 2019-10-13T11:22:41.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:41.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:41.266716+0000) 2019-10-13T11:22:41.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:41.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:41.287792+0000) 2019-10-13T11:22:41.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:41.806+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:41.810027+0000) 2019-10-13T11:22:42.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:42.354+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:42.356398+0000) 2019-10-13T11:22:44.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:44.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.130719+0000) 2019-10-13T11:22:44.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:44.411+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.417620+0000) 2019-10-13T11:22:44.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:44.587+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.592720+0000) 2019-10-13T11:22:44.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:44.587+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.593132+0000) 2019-10-13T11:22:44.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:44.591+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.593983+0000) 2019-10-13T11:22:44.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:44.591+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.594409+0000) 2019-10-13T11:22:44.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:44.591+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.595887+0000) 2019-10-13T11:22:44.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:44.594+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.596344+0000) 2019-10-13T11:22:44.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:44.591+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.596717+0000) 2019-10-13T11:22:44.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:44.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:44.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:44.598+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:44.602366+0000) 2019-10-13T11:22:47.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:47.783+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:47.789583+0000) 2019-10-13T11:22:48.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:48.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:48.558762+0000) 2019-10-13T11:22:49.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:49.598+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.602980+0000) 2019-10-13T11:22:49.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:49.599+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.605016+0000) 2019-10-13T11:22:49.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:49.599+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.605208+0000) 2019-10-13T11:22:49.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:49.603+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.606793+0000) 2019-10-13T11:22:49.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:49.603+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.607749+0000) 2019-10-13T11:22:49.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:49.603+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.608539+0000) 2019-10-13T11:22:49.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:49.603+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.608570+0000) 2019-10-13T11:22:49.611 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:49.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:49.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:49.610+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:49.612782+0000) 2019-10-13T11:22:51.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:51.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:51.266914+0000) 2019-10-13T11:22:51.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:51.283+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:51.287985+0000) 2019-10-13T11:22:51.726 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:22:51.726 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:51.730 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:22:51.730 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:22:51.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:51.806+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:51.810321+0000) 2019-10-13T11:22:52.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:52.362+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:52.363019+0000) 2019-10-13T11:22:54.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:54.127+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.130872+0000) 2019-10-13T11:22:54.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:54.415+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.417798+0000) 2019-10-13T11:22:54.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:54.607+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.612959+0000) 2019-10-13T11:22:54.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:54.607+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.613325+0000) 2019-10-13T11:22:54.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:54.610+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.614578+0000) 2019-10-13T11:22:54.616 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:54.611+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.615842+0000) 2019-10-13T11:22:54.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:54.611+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.616395+0000) 2019-10-13T11:22:54.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:54.611+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.616531+0000) 2019-10-13T11:22:54.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:54.611+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.617025+0000) 2019-10-13T11:22:54.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:54.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:54.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:54.618+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:54.623909+0000) 2019-10-13T11:22:57.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:57.786+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:57.789789+0000) 2019-10-13T11:22:58.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:58.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:58.558990+0000) 2019-10-13T11:22:59.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:22:59.618+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.625318+0000) 2019-10-13T11:22:59.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:22:59.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.625969+0000) 2019-10-13T11:22:59.627 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:22:59.622+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.626300+0000) 2019-10-13T11:22:59.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:22:59.622+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.626551+0000) 2019-10-13T11:22:59.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:22:59.622+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.627206+0000) 2019-10-13T11:22:59.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:22:59.622+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.627607+0000) 2019-10-13T11:22:59.629 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:22:59.622+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.628107+0000) 2019-10-13T11:22:59.632 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:22:59.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:22:59.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:22:59.630+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:22:59.632807+0000) 2019-10-13T11:23:01.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:01.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:01.267149+0000) 2019-10-13T11:23:01.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:01.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:01.288211+0000) 2019-10-13T11:23:01.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:01.806+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:01.810523+0000) 2019-10-13T11:23:02.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:02.366+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:02.369764+0000) 2019-10-13T11:23:04.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:04.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.131069+0000) 2019-10-13T11:23:04.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:04.414+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.417953+0000) 2019-10-13T11:23:04.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:04.630+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.635753+0000) 2019-10-13T11:23:04.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:04.630+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.636765+0000) 2019-10-13T11:23:04.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:04.634+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.638251+0000) 2019-10-13T11:23:04.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:04.634+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.638690+0000) 2019-10-13T11:23:04.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:04.634+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.640350+0000) 2019-10-13T11:23:04.641 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:04.634+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.640950+0000) 2019-10-13T11:23:04.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:04.634+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.641437+0000) 2019-10-13T11:23:04.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:04.634+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:04.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:04.642+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:04.646884+0000) 2019-10-13T11:23:07.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:07.786+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:07.789997+0000) 2019-10-13T11:23:08.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:08.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:08.559183+0000) 2019-10-13T11:23:09.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:09.638+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.645023+0000) 2019-10-13T11:23:09.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:09.642+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.646068+0000) 2019-10-13T11:23:09.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:09.642+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.646382+0000) 2019-10-13T11:23:09.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:09.642+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.646566+0000) 2019-10-13T11:23:09.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:09.642+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.647410+0000) 2019-10-13T11:23:09.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:09.642+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.649149+0000) 2019-10-13T11:23:09.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:09.642+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.649405+0000) 2019-10-13T11:23:09.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:09.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:09.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:09.650+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:09.654993+0000) 2019-10-13T11:23:11.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:11.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:11.267347+0000) 2019-10-13T11:23:11.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:11.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:11.288449+0000) 2019-10-13T11:23:11.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:11.806+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:11.810719+0000) 2019-10-13T11:23:12.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:12.374+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:12.376275+0000) 2019-10-13T11:23:14.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:14.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.131265+0000) 2019-10-13T11:23:14.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:14.414+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.418140+0000) 2019-10-13T11:23:14.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:14.650+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.655536+0000) 2019-10-13T11:23:14.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:14.650+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.656012+0000) 2019-10-13T11:23:14.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:14.650+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.656652+0000) 2019-10-13T11:23:14.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:14.654+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.658956+0000) 2019-10-13T11:23:14.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:14.654+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.659686+0000) 2019-10-13T11:23:14.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:14.654+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.660583+0000) 2019-10-13T11:23:14.662 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:14.654+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.661629+0000) 2019-10-13T11:23:14.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:14.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:14.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:14.662+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:14.665880+0000) 2019-10-13T11:23:17.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:17.786+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:17.790196+0000) 2019-10-13T11:23:18.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:18.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:18.559414+0000) 2019-10-13T11:23:19.666 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:19.658+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.664980+0000) 2019-10-13T11:23:19.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:19.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.666043+0000) 2019-10-13T11:23:19.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:19.662+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.666271+0000) 2019-10-13T11:23:19.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:19.662+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.666910+0000) 2019-10-13T11:23:19.668 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:19.662+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.667417+0000) 2019-10-13T11:23:19.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:19.662+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.667753+0000) 2019-10-13T11:23:19.669 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:19.662+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.668308+0000) 2019-10-13T11:23:19.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:19.666+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:19.671026+0000) 2019-10-13T11:23:19.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:19.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:21.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:21.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:21.267546+0000) 2019-10-13T11:23:21.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:21.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:21.288669+0000) 2019-10-13T11:23:21.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:21.810+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:21.810915+0000) 2019-10-13T11:23:21.847 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:23:21.847 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:21.851 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:23:21.851 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:22.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:22.382+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:22.382857+0000) 2019-10-13T11:23:24.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:24.126+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.131468+0000) 2019-10-13T11:23:24.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:24.414+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.418341+0000) 2019-10-13T11:23:24.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:24.670+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.674210+0000) 2019-10-13T11:23:24.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:24.670+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.675861+0000) 2019-10-13T11:23:24.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:24.670+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.677092+0000) 2019-10-13T11:23:24.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:24.674+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.678039+0000) 2019-10-13T11:23:24.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:24.674+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.678559+0000) 2019-10-13T11:23:24.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:24.674+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.679294+0000) 2019-10-13T11:23:24.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:24.674+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.680111+0000) 2019-10-13T11:23:24.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:24.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:24.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:24.682+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:24.684899+0000) 2019-10-13T11:23:27.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:27.786+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:27.790388+0000) 2019-10-13T11:23:28.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:28.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:28.559611+0000) 2019-10-13T11:23:29.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:29.678+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.685200+0000) 2019-10-13T11:23:29.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:29.678+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.685596+0000) 2019-10-13T11:23:29.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:29.682+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.688018+0000) 2019-10-13T11:23:29.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:29.682+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.688445+0000) 2019-10-13T11:23:29.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:29.686+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.688624+0000) 2019-10-13T11:23:29.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:29.682+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.689414+0000) 2019-10-13T11:23:29.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:29.686+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.690331+0000) 2019-10-13T11:23:29.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:29.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:29.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:29.690+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:29.694061+0000) 2019-10-13T11:23:31.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:31.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:31.267741+0000) 2019-10-13T11:23:31.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:31.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:31.288870+0000) 2019-10-13T11:23:31.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:31.806+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:31.811110+0000) 2019-10-13T11:23:32.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:32.386+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:32.389394+0000) 2019-10-13T11:23:34.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:34.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.131671+0000) 2019-10-13T11:23:34.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:34.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.418539+0000) 2019-10-13T11:23:34.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:34.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.695646+0000) 2019-10-13T11:23:34.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:34.690+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.696846+0000) 2019-10-13T11:23:34.700 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:34.694+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.699520+0000) 2019-10-13T11:23:34.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:34.694+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.699790+0000) 2019-10-13T11:23:34.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:34.694+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.700354+0000) 2019-10-13T11:23:34.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:34.694+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.700823+0000) 2019-10-13T11:23:34.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:34.694+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.701713+0000) 2019-10-13T11:23:34.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:34.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:34.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:34.702+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:34.706056+0000) 2019-10-13T11:23:37.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:37.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:37.790586+0000) 2019-10-13T11:23:38.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:38.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:38.559828+0000) 2019-10-13T11:23:39.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:39.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.706415+0000) 2019-10-13T11:23:39.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:39.702+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.707524+0000) 2019-10-13T11:23:39.708 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:39.702+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.707758+0000) 2019-10-13T11:23:39.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:39.702+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.708158+0000) 2019-10-13T11:23:39.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:39.702+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.708537+0000) 2019-10-13T11:23:39.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:39.706+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.709904+0000) 2019-10-13T11:23:39.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:39.706+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.709852+0000) 2019-10-13T11:23:39.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:39.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:39.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:39.714+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:39.716757+0000) 2019-10-13T11:23:41.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:41.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:41.267959+0000) 2019-10-13T11:23:41.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:41.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:41.289098+0000) 2019-10-13T11:23:41.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:41.810+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:41.811338+0000) 2019-10-13T11:23:42.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:42.394+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:42.396118+0000) 2019-10-13T11:23:44.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:44.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.131872+0000) 2019-10-13T11:23:44.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:44.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.418768+0000) 2019-10-13T11:23:44.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:44.710+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.716634+0000) 2019-10-13T11:23:44.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:44.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.718711+0000) 2019-10-13T11:23:44.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:44.714+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.719349+0000) 2019-10-13T11:23:44.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:44.714+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.719711+0000) 2019-10-13T11:23:44.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:44.714+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.720625+0000) 2019-10-13T11:23:44.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:44.717+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.721475+0000) 2019-10-13T11:23:44.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:44.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.721880+0000) 2019-10-13T11:23:44.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:44.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:44.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:44.725+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:44.728556+0000) 2019-10-13T11:23:47.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:47.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:47.790796+0000) 2019-10-13T11:23:48.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:48.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:48.560048+0000) 2019-10-13T11:23:49.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:49.722+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.726320+0000) 2019-10-13T11:23:49.727 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:49.722+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.726545+0000) 2019-10-13T11:23:49.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:49.722+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.729421+0000) 2019-10-13T11:23:49.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:49.726+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.731100+0000) 2019-10-13T11:23:49.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:49.726+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.731582+0000) 2019-10-13T11:23:49.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:49.726+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.731905+0000) 2019-10-13T11:23:49.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:49.729+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.732354+0000) 2019-10-13T11:23:49.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:49.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:49.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:49.733+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:49.736869+0000) 2019-10-13T11:23:51.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:51.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:51.268171+0000) 2019-10-13T11:23:51.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:51.282+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:51.289310+0000) 2019-10-13T11:23:51.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:51.805+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:51.811535+0000) 2019-10-13T11:23:51.961 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:23:51.961 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:51.965 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:23:51.966 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:23:52.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:52.397+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:52.402622+0000) 2019-10-13T11:23:54.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:54.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.132082+0000) 2019-10-13T11:23:54.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:54.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.418998+0000) 2019-10-13T11:23:54.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:54.729+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.736788+0000) 2019-10-13T11:23:54.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:54.733+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.739324+0000) 2019-10-13T11:23:54.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:54.733+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.739813+0000) 2019-10-13T11:23:54.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:54.733+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.740426+0000) 2019-10-13T11:23:54.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:54.733+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.740676+0000) 2019-10-13T11:23:54.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:54.733+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.740789+0000) 2019-10-13T11:23:54.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:54.737+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.742042+0000) 2019-10-13T11:23:54.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:54.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:54.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:54.741+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:54.744703+0000) 2019-10-13T11:23:57.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:57.785+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:57.790969+0000) 2019-10-13T11:23:58.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:58.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:58.560240+0000) 2019-10-13T11:23:59.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:23:59.741+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.746651+0000) 2019-10-13T11:23:59.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:23:59.741+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.747286+0000) 2019-10-13T11:23:59.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:23:59.741+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.749367+0000) 2019-10-13T11:23:59.750 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:23:59.745+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.749804+0000) 2019-10-13T11:23:59.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:23:59.745+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.749936+0000) 2019-10-13T11:23:59.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:23:59.745+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.751388+0000) 2019-10-13T11:23:59.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:23:59.745+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.751856+0000) 2019-10-13T11:23:59.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:23:59.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137748 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:23:59.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:23:59.753+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:23:59.756925+0000) 2019-10-13T11:24:01.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:01.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:01.268397+0000) 2019-10-13T11:24:01.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:01.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:01.289506+0000) 2019-10-13T11:24:01.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:01.809+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:01.811738+0000) 2019-10-13T11:24:02.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:02.405+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:02.409208+0000) 2019-10-13T11:24:04.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:04.125+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.132290+0000) 2019-10-13T11:24:04.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:04.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.419196+0000) 2019-10-13T11:24:04.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:04.749+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.756500+0000) 2019-10-13T11:24:04.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:04.749+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.756465+0000) 2019-10-13T11:24:04.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:04.749+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.757196+0000) 2019-10-13T11:24:04.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:04.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.757485+0000) 2019-10-13T11:24:04.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:04.753+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.758977+0000) 2019-10-13T11:24:04.760 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:04.753+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.758786+0000) 2019-10-13T11:24:04.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:04.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.759137+0000) 2019-10-13T11:24:04.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:04.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:04.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:04.757+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:04.764056+0000) 2019-10-13T11:24:07.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:07.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:07.791183+0000) 2019-10-13T11:24:08.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:08.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:08.560503+0000) 2019-10-13T11:24:09.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:09.761+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.765658+0000) 2019-10-13T11:24:09.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:09.765+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.766245+0000) 2019-10-13T11:24:09.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:09.765+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.766753+0000) 2019-10-13T11:24:09.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:09.765+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.767848+0000) 2019-10-13T11:24:09.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:09.765+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.768570+0000) 2019-10-13T11:24:09.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:09.765+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.769111+0000) 2019-10-13T11:24:09.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:09.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.769611+0000) 2019-10-13T11:24:09.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:09.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:09.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:09.769+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:09.775102+0000) 2019-10-13T11:24:11.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:11.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:11.268631+0000) 2019-10-13T11:24:11.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:11.285+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:11.289706+0000) 2019-10-13T11:24:11.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:11.809+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:11.811933+0000) 2019-10-13T11:24:12.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:12.409+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:12.415734+0000) 2019-10-13T11:24:14.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:14.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.132489+0000) 2019-10-13T11:24:14.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:14.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.419425+0000) 2019-10-13T11:24:14.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:14.773+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.776627+0000) 2019-10-13T11:24:14.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:14.777+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.778347+0000) 2019-10-13T11:24:14.779 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:14.777+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.778869+0000) 2019-10-13T11:24:14.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:14.777+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.780452+0000) 2019-10-13T11:24:14.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:14.777+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.780622+0000) 2019-10-13T11:24:14.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:14.777+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.783537+0000) 2019-10-13T11:24:14.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:14.781+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.784273+0000) 2019-10-13T11:24:14.785 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:14.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:14.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:14.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:14.785194+0000) 2019-10-13T11:24:17.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:17.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:17.791409+0000) 2019-10-13T11:24:18.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:18.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:18.560730+0000) 2019-10-13T11:24:19.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:19.785+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.788010+0000) 2019-10-13T11:24:19.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:19.785+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.788359+0000) 2019-10-13T11:24:19.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:19.785+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.789039+0000) 2019-10-13T11:24:19.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:19.789+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.790008+0000) 2019-10-13T11:24:19.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:19.789+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.790369+0000) 2019-10-13T11:24:19.792 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:19.789+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.790902+0000) 2019-10-13T11:24:19.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:19.789+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.791323+0000) 2019-10-13T11:24:19.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:19.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:19.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:19.793+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:19.796236+0000) 2019-10-13T11:24:21.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:21.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:21.268789+0000) 2019-10-13T11:24:21.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:21.289+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:21.289930+0000) 2019-10-13T11:24:21.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:21.809+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:21.812127+0000) 2019-10-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:24:22.016 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:24:22.025 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:22.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:22.421+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:22.426745+0000) 2019-10-13T11:24:24.133 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:24.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.132697+0000) 2019-10-13T11:24:24.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:24.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.419639+0000) 2019-10-13T11:24:24.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:24.793+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.797692+0000) 2019-10-13T11:24:24.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:24.797+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.798117+0000) 2019-10-13T11:24:24.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:24.797+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.798705+0000) 2019-10-13T11:24:24.800 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:24.797+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.799033+0000) 2019-10-13T11:24:24.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:24.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.799279+0000) 2019-10-13T11:24:24.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:24.797+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.800502+0000) 2019-10-13T11:24:24.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:24.801+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.804037+0000) 2019-10-13T11:24:24.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:24.801+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:24.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:24.805+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:24.808311+0000) 2019-10-13T11:24:27.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:27.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:27.791604+0000) 2019-10-13T11:24:28.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:28.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:28.560938+0000) 2019-10-13T11:24:29.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:29.805+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.808254+0000) 2019-10-13T11:24:29.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:29.805+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.808837+0000) 2019-10-13T11:24:29.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:29.809+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.810879+0000) 2019-10-13T11:24:29.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:29.805+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.811042+0000) 2019-10-13T11:24:29.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:29.809+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.811522+0000) 2019-10-13T11:24:29.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:29.809+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.812541+0000) 2019-10-13T11:24:29.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:29.809+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.813243+0000) 2019-10-13T11:24:29.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:29.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:29.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:29.813+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:29.819390+0000) 2019-10-13T11:24:31.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:31.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:31.268990+0000) 2019-10-13T11:24:31.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:31.289+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:31.290130+0000) 2019-10-13T11:24:31.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:31.809+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:31.812331+0000) 2019-10-13T11:24:32.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:32.429+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:32.433358+0000) 2019-10-13T11:24:34.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:34.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.132893+0000) 2019-10-13T11:24:34.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:34.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.419865+0000) 2019-10-13T11:24:34.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:34.817+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.818599+0000) 2019-10-13T11:24:34.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:34.817+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.818917+0000) 2019-10-13T11:24:34.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:34.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.820434+0000) 2019-10-13T11:24:34.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:34.817+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.820861+0000) 2019-10-13T11:24:34.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:34.817+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.821356+0000) 2019-10-13T11:24:34.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:34.817+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.823110+0000) 2019-10-13T11:24:34.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:34.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.823887+0000) 2019-10-13T11:24:34.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:34.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:34.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:34.825+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:34.832119+0000) 2019-10-13T11:24:37.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:37.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:37.791806+0000) 2019-10-13T11:24:38.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:38.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:38.561134+0000) 2019-10-13T11:24:39.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:39.825+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.828316+0000) 2019-10-13T11:24:39.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:39.825+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.828913+0000) 2019-10-13T11:24:39.830 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:39.825+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.829595+0000) 2019-10-13T11:24:39.832 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:39.829+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.831245+0000) 2019-10-13T11:24:39.835 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:39.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:39.836 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:39.833+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.835321+0000) 2019-10-13T11:24:39.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:39.833+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.835549+0000) 2019-10-13T11:24:39.837 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:39.833+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.836242+0000) 2019-10-13T11:24:39.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:39.833+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:39.839491+0000) 2019-10-13T11:24:41.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:41.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:41.269222+0000) 2019-10-13T11:24:41.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:41.289+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:41.290388+0000) 2019-10-13T11:24:41.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:41.809+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:41.812530+0000) 2019-10-13T11:24:42.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:42.437+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:42.442416+0000) 2019-10-13T11:24:44.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:44.129+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.133097+0000) 2019-10-13T11:24:44.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:44.417+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.420073+0000) 2019-10-13T11:24:44.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:44.837+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.837810+0000) 2019-10-13T11:24:44.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:44.837+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.838909+0000) 2019-10-13T11:24:44.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:44.837+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.839743+0000) 2019-10-13T11:24:44.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:44.833+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.839836+0000) 2019-10-13T11:24:44.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:44.837+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.840413+0000) 2019-10-13T11:24:44.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:44.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.842091+0000) 2019-10-13T11:24:44.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:44.841+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.843999+0000) 2019-10-13T11:24:44.846 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:44.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:44.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:44.841+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:44.846983+0000) 2019-10-13T11:24:47.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:47.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:47.792005+0000) 2019-10-13T11:24:48.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:48.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:48.561359+0000) 2019-10-13T11:24:49.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:49.845+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.848627+0000) 2019-10-13T11:24:49.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:49.849+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.851397+0000) 2019-10-13T11:24:49.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:49.849+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.851564+0000) 2019-10-13T11:24:49.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:49.849+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.851922+0000) 2019-10-13T11:24:49.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:49.849+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.852816+0000) 2019-10-13T11:24:49.854 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:49.849+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.853607+0000) 2019-10-13T11:24:49.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:49.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.854105+0000) 2019-10-13T11:24:49.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:49.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:49.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:49.852+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:49.859424+0000) 2019-10-13T11:24:51.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:51.264+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:51.269432+0000) 2019-10-13T11:24:51.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:51.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:51.290586+0000) 2019-10-13T11:24:51.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:51.808+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:51.812760+0000) 2019-10-13T11:24:52.144 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:24:52.145 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:52.148 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:24:52.148 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:24:52.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:52.444+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:52.448923+0000) 2019-10-13T11:24:54.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:54.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.133310+0000) 2019-10-13T11:24:54.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:54.416+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.420270+0000) 2019-10-13T11:24:54.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:54.856+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.858887+0000) 2019-10-13T11:24:54.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:54.856+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.859849+0000) 2019-10-13T11:24:54.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:54.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.860085+0000) 2019-10-13T11:24:54.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:54.860+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.861888+0000) 2019-10-13T11:24:54.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:54.860+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.864640+0000) 2019-10-13T11:24:54.866 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:54.860+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.865642+0000) 2019-10-13T11:24:54.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:54.864+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.866555+0000) 2019-10-13T11:24:54.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:54.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:54.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:54.864+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:54.867715+0000) 2019-10-13T11:24:57.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:57.788+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:57.792206+0000) 2019-10-13T11:24:58.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:58.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:58.561553+0000) 2019-10-13T11:24:59.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:24:59.868+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.870626+0000) 2019-10-13T11:24:59.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:24:59.868+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.871699+0000) 2019-10-13T11:24:59.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:24:59.868+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.872469+0000) 2019-10-13T11:24:59.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:24:59.868+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.872187+0000) 2019-10-13T11:24:59.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:24:59.868+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.873269+0000) 2019-10-13T11:24:59.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:24:59.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.872981+0000) 2019-10-13T11:24:59.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:24:59.872+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.875165+0000) 2019-10-13T11:24:59.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:24:59.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:24:59.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:24:59.872+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:24:59.879388+0000) 2019-10-13T11:25:01.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:01.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:01.269631+0000) 2019-10-13T11:25:01.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:01.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:01.290817+0000) 2019-10-13T11:25:01.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:01.808+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:01.812974+0000) 2019-10-13T11:25:02.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:02.448+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:02.455653+0000) 2019-10-13T11:25:04.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:04.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.133567+0000) 2019-10-13T11:25:04.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:04.416+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.420501+0000) 2019-10-13T11:25:04.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:04.876+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.880645+0000) 2019-10-13T11:25:04.882 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:04.876+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.881040+0000) 2019-10-13T11:25:04.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:04.876+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.882315+0000) 2019-10-13T11:25:04.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:04.880+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.882757+0000) 2019-10-13T11:25:04.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:04.880+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.883246+0000) 2019-10-13T11:25:04.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:04.880+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.883871+0000) 2019-10-13T11:25:04.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:04.880+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.884303+0000) 2019-10-13T11:25:04.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:04.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:04.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:04.880+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:04.887534+0000) 2019-10-13T11:25:07.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:07.788+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:07.792418+0000) 2019-10-13T11:25:08.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:08.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:08.561780+0000) 2019-10-13T11:25:09.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:09.888+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.891260+0000) 2019-10-13T11:25:09.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:09.888+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.891551+0000) 2019-10-13T11:25:09.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:09.888+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.892079+0000) 2019-10-13T11:25:09.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:09.888+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.893475+0000) 2019-10-13T11:25:09.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:09.892+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.894420+0000) 2019-10-13T11:25:09.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:09.892+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.895794+0000) 2019-10-13T11:25:09.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:09.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:09.899 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:09.896+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.898718+0000) 2019-10-13T11:25:09.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:09.900+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:09.904632+0000) 2019-10-13T11:25:11.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:11.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.269871+0000) 2019-10-13T11:25:11.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:11.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.291050+0000) 2019-10-13T11:25:11.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:11.808+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:11.813187+0000) 2019-10-13T11:25:12.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:12.460+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:12.467131+0000) 2019-10-13T11:25:14.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:14.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.133788+0000) 2019-10-13T11:25:14.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:14.420+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.420703+0000) 2019-10-13T11:25:14.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:14.896+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.901124+0000) 2019-10-13T11:25:14.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:14.896+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.901472+0000) 2019-10-13T11:25:14.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:14.900+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.902562+0000) 2019-10-13T11:25:14.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:14.900+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.904473+0000) 2019-10-13T11:25:14.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:14.900+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.904981+0000) 2019-10-13T11:25:14.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:14.904+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.906034+0000) 2019-10-13T11:25:14.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:14.900+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.906387+0000) 2019-10-13T11:25:14.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:14.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:14.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:14.904+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:14.910519+0000) 2019-10-13T11:25:17.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:17.788+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:17.792609+0000) 2019-10-13T11:25:18.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:18.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:18.561994+0000) 2019-10-13T11:25:19.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:19.908+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.911699+0000) 2019-10-13T11:25:19.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:19.908+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.912243+0000) 2019-10-13T11:25:19.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:19.908+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.912343+0000) 2019-10-13T11:25:19.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:19.908+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.912662+0000) 2019-10-13T11:25:19.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:19.908+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.913050+0000) 2019-10-13T11:25:19.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:19.908+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.913296+0000) 2019-10-13T11:25:19.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:19.912+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.914351+0000) 2019-10-13T11:25:19.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:19.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:19.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:19.916+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:19.920311+0000) 2019-10-13T11:25:21.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:21.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.270062+0000) 2019-10-13T11:25:21.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:21.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.291266+0000) 2019-10-13T11:25:21.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:21.812+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:21.813381+0000) 2019-10-13T11:25:22.274 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:25:22.274 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:22.278 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:25:22.278 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:22.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:22.468+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:22.473655+0000) 2019-10-13T11:25:24.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:24.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.134026+0000) 2019-10-13T11:25:24.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:24.416+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.420933+0000) 2019-10-13T11:25:24.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:24.916+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.921717+0000) 2019-10-13T11:25:24.923 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:24.920+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.922347+0000) 2019-10-13T11:25:24.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:24.920+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.924666+0000) 2019-10-13T11:25:24.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:24.920+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.925230+0000) 2019-10-13T11:25:24.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:24.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.925512+0000) 2019-10-13T11:25:24.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:24.924+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.926193+0000) 2019-10-13T11:25:24.927 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:24.924+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.927002+0000) 2019-10-13T11:25:24.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:24.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:24.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:24.928+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:24.932763+0000) 2019-10-13T11:25:27.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:27.788+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:27.792850+0000) 2019-10-13T11:25:28.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:28.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:28.562190+0000) 2019-10-13T11:25:29.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:29.932+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.933802+0000) 2019-10-13T11:25:29.935 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:29.932+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.933967+0000) 2019-10-13T11:25:29.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:29.928+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.934721+0000) 2019-10-13T11:25:29.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:29.932+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.934461+0000) 2019-10-13T11:25:29.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:29.932+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.936316+0000) 2019-10-13T11:25:29.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:29.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.936746+0000) 2019-10-13T11:25:29.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:29.936+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.938238+0000) 2019-10-13T11:25:29.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:29.932+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:29.939232+0000) 2019-10-13T11:25:29.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:29.936+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:31.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:31.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.270271+0000) 2019-10-13T11:25:31.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:31.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.291477+0000) 2019-10-13T11:25:31.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:31.812+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:31.813575+0000) 2019-10-13T11:25:32.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:32.476+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:32.480153+0000) 2019-10-13T11:25:34.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:34.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.134264+0000) 2019-10-13T11:25:34.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:34.420+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.421133+0000) 2019-10-13T11:25:34.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:34.940+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.943100+0000) 2019-10-13T11:25:34.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:34.940+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.944240+0000) 2019-10-13T11:25:34.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:34.940+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.944642+0000) 2019-10-13T11:25:34.946 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:34.940+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.945135+0000) 2019-10-13T11:25:34.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:34.940+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.945208+0000) 2019-10-13T11:25:34.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:34.940+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.945654+0000) 2019-10-13T11:25:34.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:34.940+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.945384+0000) 2019-10-13T11:25:34.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:34.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:34.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:34.948+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:34.952771+0000) 2019-10-13T11:25:37.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:37.792+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:37.793073+0000) 2019-10-13T11:25:38.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:38.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:38.562434+0000) 2019-10-13T11:25:39.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:39.948+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.953348+0000) 2019-10-13T11:25:39.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:39.952+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.955539+0000) 2019-10-13T11:25:39.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:39.952+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.956083+0000) 2019-10-13T11:25:39.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:39.948+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.956005+0000) 2019-10-13T11:25:39.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:39.952+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.956656+0000) 2019-10-13T11:25:39.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:39.952+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.957245+0000) 2019-10-13T11:25:39.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:39.956+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.959485+0000) 2019-10-13T11:25:39.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:39.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:39.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:39.960+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:39.965444+0000) 2019-10-13T11:25:41.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:41.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.270505+0000) 2019-10-13T11:25:41.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:41.288+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.291678+0000) 2019-10-13T11:25:41.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:41.812+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:41.813791+0000) 2019-10-13T11:25:42.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:42.480+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:42.486712+0000) 2019-10-13T11:25:44.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:44.132+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.134504+0000) 2019-10-13T11:25:44.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:44.416+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.421332+0000) 2019-10-13T11:25:44.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:44.960+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.964518+0000) 2019-10-13T11:25:44.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:44.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.964955+0000) 2019-10-13T11:25:44.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:44.960+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.965322+0000) 2019-10-13T11:25:44.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:44.960+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.965721+0000) 2019-10-13T11:25:44.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:44.964+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.966381+0000) 2019-10-13T11:25:44.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:44.964+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.967365+0000) 2019-10-13T11:25:44.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:44.964+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.967948+0000) 2019-10-13T11:25:44.971 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:44.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:44.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:44.968+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:44.974211+0000) 2019-10-13T11:25:47.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:47.791+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:47.793309+0000) 2019-10-13T11:25:48.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:48.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:48.562654+0000) 2019-10-13T11:25:49.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:49.971+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.974133+0000) 2019-10-13T11:25:49.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:49.972+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.974654+0000) 2019-10-13T11:25:49.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:49.971+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.975278+0000) 2019-10-13T11:25:49.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:49.971+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.976631+0000) 2019-10-13T11:25:49.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:49.971+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.976937+0000) 2019-10-13T11:25:49.978 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:49.971+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.977609+0000) 2019-10-13T11:25:49.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:49.975+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.978517+0000) 2019-10-13T11:25:49.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:49.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:49.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:49.984+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:49.984504+0000) 2019-10-13T11:25:51.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:51.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.270728+0000) 2019-10-13T11:25:51.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:51.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.291906+0000) 2019-10-13T11:25:51.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:51.812+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:51.814003+0000) 2019-10-13T11:25:52.379 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:25:52.379 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:52.383 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:25:52.383 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:25:52.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:52.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:52.492579+0000) 2019-10-13T11:25:54.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:54.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.134737+0000) 2019-10-13T11:25:54.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:54.419+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.421558+0000) 2019-10-13T11:25:54.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:54.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.984243+0000) 2019-10-13T11:25:54.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:54.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.984994+0000) 2019-10-13T11:25:54.986 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:54.979+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.984709+0000) 2019-10-13T11:25:54.986 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:54.979+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.985567+0000) 2019-10-13T11:25:54.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:54.983+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.986636+0000) 2019-10-13T11:25:54.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:54.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.987093+0000) 2019-10-13T11:25:54.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:54.984+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.987189+0000) 2019-10-13T11:25:54.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:54.992+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:54.992532+0000) 2019-10-13T11:25:54.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:54.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:25:57.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:57.787+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:57.793507+0000) 2019-10-13T11:25:58.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:58.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:58.562840+0000) 2019-10-13T11:25:59.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:25:59.991+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.994677+0000) 2019-10-13T11:25:59.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:25:59.991+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.995724+0000) 2019-10-13T11:25:59.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:25:59.995+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.996479+0000) 2019-10-13T11:25:59.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:25:59.991+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.997300+0000) 2019-10-13T11:25:59.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:25:59.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.997529+0000) 2019-10-13T11:25:59.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:25:59.995+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.998175+0000) 2019-10-13T11:25:59.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:25:59.995+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:25:59.998485+0000) 2019-10-13T11:26:00.000 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:25:59.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:00.003 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:25:59.999+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:00.002690+0000) 2019-10-13T11:26:01.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:01.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.270892+0000) 2019-10-13T11:26:01.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:01.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.292106+0000) 2019-10-13T11:26:01.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:01.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:01.814201+0000) 2019-10-13T11:26:02.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:02.499+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:02.500182+0000) 2019-10-13T11:26:04.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:04.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.134952+0000) 2019-10-13T11:26:04.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:04.419+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:04.421791+0000) 2019-10-13T11:26:05.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:04.999+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.003718+0000) 2019-10-13T11:26:05.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:05.003+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.005991+0000) 2019-10-13T11:26:05.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:05.003+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.008361+0000) 2019-10-13T11:26:05.009 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:05.003+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.008402+0000) 2019-10-13T11:26:05.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:05.007+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.009899+0000) 2019-10-13T11:26:05.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:05.007+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.009844+0000) 2019-10-13T11:26:05.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:05.007+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.010629+0000) 2019-10-13T11:26:05.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:05.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:05.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:05.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:05.010628+0000) 2019-10-13T11:26:07.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:07.787+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:07.793701+0000) 2019-10-13T11:26:08.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:08.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:08.563066+0000) 2019-10-13T11:26:10.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:10.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.014179+0000) 2019-10-13T11:26:10.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:10.011+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.015185+0000) 2019-10-13T11:26:10.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:10.011+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.015621+0000) 2019-10-13T11:26:10.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:10.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.016211+0000) 2019-10-13T11:26:10.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:10.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.016176+0000) 2019-10-13T11:26:10.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:10.011+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.016683+0000) 2019-10-13T11:26:10.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:10.011+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.017197+0000) 2019-10-13T11:26:10.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:10.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:10.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:10.019+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:10.022688+0000) 2019-10-13T11:26:11.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:11.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.271090+0000) 2019-10-13T11:26:11.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:11.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.292345+0000) 2019-10-13T11:26:11.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:11.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:11.814425+0000) 2019-10-13T11:26:12.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:12.503+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:12.506834+0000) 2019-10-13T11:26:14.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:14.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.135186+0000) 2019-10-13T11:26:14.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:14.419+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:14.422013+0000) 2019-10-13T11:26:15.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:15.019+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.023185+0000) 2019-10-13T11:26:15.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:15.019+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.023534+0000) 2019-10-13T11:26:15.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:15.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.024553+0000) 2019-10-13T11:26:15.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:15.019+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.025240+0000) 2019-10-13T11:26:15.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:15.023+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.025771+0000) 2019-10-13T11:26:15.027 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:15.023+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.026459+0000) 2019-10-13T11:26:15.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:15.023+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.026672+0000) 2019-10-13T11:26:15.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:15.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:15.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:15.031+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:15.032559+0000) 2019-10-13T11:26:17.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:17.791+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:17.793964+0000) 2019-10-13T11:26:18.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:18.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:18.563295+0000) 2019-10-13T11:26:20.035 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:20.031+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.034445+0000) 2019-10-13T11:26:20.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:20.031+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.034885+0000) 2019-10-13T11:26:20.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:20.031+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.034878+0000) 2019-10-13T11:26:20.037 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:20.031+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.035673+0000) 2019-10-13T11:26:20.037 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:20.031+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.036013+0000) 2019-10-13T11:26:20.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:20.031+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.036533+0000) 2019-10-13T11:26:20.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:20.031+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.037074+0000) 2019-10-13T11:26:20.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:20.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:20.040009+0000) 2019-10-13T11:26:20.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:20.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:21.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:21.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.271285+0000) 2019-10-13T11:26:21.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:21.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.292576+0000) 2019-10-13T11:26:21.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:21.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:21.814632+0000) 2019-10-13T11:26:22.497 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:26:22.497 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:22.501 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:26:22.501 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:22.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:22.511+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:22.513227+0000) 2019-10-13T11:26:24.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:24.131+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.135386+0000) 2019-10-13T11:26:24.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:24.419+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:24.422224+0000) 2019-10-13T11:26:25.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:25.047+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.048102+0000) 2019-10-13T11:26:25.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:25.047+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.050816+0000) 2019-10-13T11:26:25.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:25.043+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.046297+0000) 2019-10-13T11:26:25.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:25.043+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.046604+0000) 2019-10-13T11:26:25.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:25.043+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.046934+0000) 2019-10-13T11:26:25.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:25.043+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.047552+0000) 2019-10-13T11:26:25.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:25.043+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.047884+0000) 2019-10-13T11:26:25.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:25.043+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:25.048853+0000) 2019-10-13T11:26:25.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:25.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:27.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:27.791+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:27.794218+0000) 2019-10-13T11:26:28.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:28.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:28.563483+0000) 2019-10-13T11:26:30.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:30.051+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.053801+0000) 2019-10-13T11:26:30.059 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:30.051+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.056351+0000) 2019-10-13T11:26:30.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:30.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.056747+0000) 2019-10-13T11:26:30.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:30.051+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.057052+0000) 2019-10-13T11:26:30.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:30.055+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.058342+0000) 2019-10-13T11:26:30.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:30.055+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.060969+0000) 2019-10-13T11:26:30.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:30.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:30.063 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:30.059+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.062298+0000) 2019-10-13T11:26:30.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:30.067+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:30.068749+0000) 2019-10-13T11:26:31.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:31.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.271500+0000) 2019-10-13T11:26:31.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:31.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.292761+0000) 2019-10-13T11:26:31.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:31.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:31.814830+0000) 2019-10-13T11:26:32.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:32.519+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:32.520576+0000) 2019-10-13T11:26:34.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:34.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.135626+0000) 2019-10-13T11:26:34.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:34.419+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:34.422465+0000) 2019-10-13T11:26:35.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:35.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.065601+0000) 2019-10-13T11:26:35.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:35.063+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.066678+0000) 2019-10-13T11:26:35.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:35.063+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.069081+0000) 2019-10-13T11:26:35.071 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:35.067+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.071093+0000) 2019-10-13T11:26:35.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:35.067+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.071375+0000) 2019-10-13T11:26:35.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:35.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:35.074 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:35.067+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.073379+0000) 2019-10-13T11:26:35.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:35.071+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.073784+0000) 2019-10-13T11:26:35.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:35.075+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:35.076700+0000) 2019-10-13T11:26:37.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:37.791+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:37.794440+0000) 2019-10-13T11:26:38.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:38.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:38.563678+0000) 2019-10-13T11:26:40.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:40.071+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.076482+0000) 2019-10-13T11:26:40.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:40.075+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.078327+0000) 2019-10-13T11:26:40.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:40.075+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.079077+0000) 2019-10-13T11:26:40.080 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:40.075+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.078744+0000) 2019-10-13T11:26:40.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:40.075+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.079877+0000) 2019-10-13T11:26:40.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:40.075+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.080148+0000) 2019-10-13T11:26:40.081 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:40.075+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.081138+0000) 2019-10-13T11:26:40.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:40.087+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:40.088407+0000) 2019-10-13T11:26:40.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:40.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:41.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:41.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.271707+0000) 2019-10-13T11:26:41.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:41.287+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.292966+0000) 2019-10-13T11:26:41.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:41.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:41.815023+0000) 2019-10-13T11:26:42.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:42.523+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:42.527212+0000) 2019-10-13T11:26:44.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:44.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.135822+0000) 2019-10-13T11:26:44.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:44.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:44.422665+0000) 2019-10-13T11:26:45.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:45.082+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.087293+0000) 2019-10-13T11:26:45.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:45.086+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.090219+0000) 2019-10-13T11:26:45.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:45.086+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.090412+0000) 2019-10-13T11:26:45.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:45.086+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.091800+0000) 2019-10-13T11:26:45.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:45.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138790 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:45.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:45.091+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.093507+0000) 2019-10-13T11:26:45.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:45.090+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.094486+0000) 2019-10-13T11:26:45.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:45.090+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.094218+0000) 2019-10-13T11:26:45.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:45.095+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:45.097325+0000) 2019-10-13T11:26:47.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:47.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:47.794651+0000) 2019-10-13T11:26:48.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:48.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:48.563913+0000) 2019-10-13T11:26:50.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:50.090+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.096692+0000) 2019-10-13T11:26:50.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:50.095+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.097270+0000) 2019-10-13T11:26:50.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:50.090+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.097239+0000) 2019-10-13T11:26:50.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:50.094+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.097782+0000) 2019-10-13T11:26:50.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:50.094+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.097838+0000) 2019-10-13T11:26:50.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:50.094+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.099329+0000) 2019-10-13T11:26:50.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:50.094+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.099856+0000) 2019-10-13T11:26:50.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:50.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138819 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:50.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:50.103+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:50.105229+0000) 2019-10-13T11:26:51.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:51.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.271942+0000) 2019-10-13T11:26:51.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:51.286+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.293159+0000) 2019-10-13T11:26:51.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:51.811+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:51.815219+0000) 2019-10-13T11:26:52.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:52.531+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:52.533816+0000) 2019-10-13T11:26:52.630 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:26:52.631 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:52.634 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:26:52.634 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:26:54.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:54.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.136056+0000) 2019-10-13T11:26:54.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:54.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:54.422896+0000) 2019-10-13T11:26:55.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:55.098+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.105052+0000) 2019-10-13T11:26:55.106 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:26:55.103+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.106033+0000) 2019-10-13T11:26:55.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:26:55.102+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.106957+0000) 2019-10-13T11:26:55.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:26:55.102+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.107304+0000) 2019-10-13T11:26:55.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:26:55.102+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.107827+0000) 2019-10-13T11:26:55.109 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:55.102+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.107978+0000) 2019-10-13T11:26:55.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:26:55.102+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.108930+0000) 2019-10-13T11:26:55.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:26:55.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:26:55.115 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:26:55.111+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:55.114471+0000) 2019-10-13T11:26:57.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:26:57.794+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:57.794846+0000) 2019-10-13T11:26:58.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:26:58.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:26:58.564147+0000) 2019-10-13T11:27:00.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:00.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.116110+0000) 2019-10-13T11:27:00.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:00.110+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.116581+0000) 2019-10-13T11:27:00.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:00.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.117294+0000) 2019-10-13T11:27:00.119 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:00.115+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.118086+0000) 2019-10-13T11:27:00.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:00.114+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.118759+0000) 2019-10-13T11:27:00.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:00.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.119165+0000) 2019-10-13T11:27:00.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:00.114+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.119308+0000) 2019-10-13T11:27:00.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:00.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138881 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:00.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:00.119+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:00.124000+0000) 2019-10-13T11:27:01.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:01.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.272173+0000) 2019-10-13T11:27:01.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:01.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.293390+0000) 2019-10-13T11:27:01.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:01.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:01.815444+0000) 2019-10-13T11:27:02.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:02.538+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:02.540356+0000) 2019-10-13T11:27:04.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:04.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.136257+0000) 2019-10-13T11:27:04.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:04.422+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:04.423097+0000) 2019-10-13T11:27:05.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:05.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.126461+0000) 2019-10-13T11:27:05.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:05.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.127471+0000) 2019-10-13T11:27:05.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:05.122+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.128098+0000) 2019-10-13T11:27:05.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:05.126+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.130426+0000) 2019-10-13T11:27:05.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:05.126+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.132215+0000) 2019-10-13T11:27:05.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:05.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:05.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:05.130+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.134120+0000) 2019-10-13T11:27:05.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:05.130+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.134356+0000) 2019-10-13T11:27:05.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:05.130+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:05.134502+0000) 2019-10-13T11:27:07.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:07.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:07.795029+0000) 2019-10-13T11:27:08.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:08.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:08.564337+0000) 2019-10-13T11:27:10.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:10.134+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.138144+0000) 2019-10-13T11:27:10.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:10.134+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.138680+0000) 2019-10-13T11:27:10.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:10.134+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.139417+0000) 2019-10-13T11:27:10.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:10.134+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.140093+0000) 2019-10-13T11:27:10.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:10.134+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.140519+0000) 2019-10-13T11:27:10.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:10.138+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.140976+0000) 2019-10-13T11:27:10.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:10.138+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.142300+0000) 2019-10-13T11:27:10.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:10.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:10.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:10.142+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:10.147872+0000) 2019-10-13T11:27:11.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:11.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.272401+0000) 2019-10-13T11:27:11.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:11.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.293586+0000) 2019-10-13T11:27:11.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:11.810+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:11.815631+0000) 2019-10-13T11:27:12.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:12.542+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:12.546854+0000) 2019-10-13T11:27:14.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:14.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.136453+0000) 2019-10-13T11:27:14.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:14.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:14.423288+0000) 2019-10-13T11:27:15.148 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:15.142+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.147821+0000) 2019-10-13T11:27:15.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:15.142+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.148654+0000) 2019-10-13T11:27:15.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:15.146+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.150463+0000) 2019-10-13T11:27:15.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:15.146+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.150934+0000) 2019-10-13T11:27:15.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:15.146+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.151166+0000) 2019-10-13T11:27:15.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:15.146+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.153066+0000) 2019-10-13T11:27:15.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:15.146+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.153096+0000) 2019-10-13T11:27:15.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:15.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138978 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:15.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:15.154+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:15.159156+0000) 2019-10-13T11:27:17.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:17.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:17.795259+0000) 2019-10-13T11:27:18.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:18.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:18.564492+0000) 2019-10-13T11:27:20.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:20.154+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.157907+0000) 2019-10-13T11:27:20.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:20.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.158504+0000) 2019-10-13T11:27:20.160 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:20.154+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.159277+0000) 2019-10-13T11:27:20.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:20.154+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.159582+0000) 2019-10-13T11:27:20.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:20.154+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.160765+0000) 2019-10-13T11:27:20.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:20.154+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.161226+0000) 2019-10-13T11:27:20.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:20.154+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.161404+0000) 2019-10-13T11:27:20.166 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:20.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139007 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:20.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:20.162+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:20.167525+0000) 2019-10-13T11:27:21.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:21.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.272646+0000) 2019-10-13T11:27:21.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:21.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.293790+0000) 2019-10-13T11:27:21.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:21.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:21.815828+0000) 2019-10-13T11:27:22.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:22.550+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:22.552805+0000) 2019-10-13T11:27:22.890 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:27:22.890 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:22.894 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:27:22.894 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:24.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:24.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.136666+0000) 2019-10-13T11:27:24.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:24.422+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:24.423502+0000) 2019-10-13T11:27:25.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:25.162+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.168263+0000) 2019-10-13T11:27:25.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:25.162+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.169056+0000) 2019-10-13T11:27:25.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:25.166+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.169324+0000) 2019-10-13T11:27:25.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:25.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.171587+0000) 2019-10-13T11:27:25.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:25.166+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.172920+0000) 2019-10-13T11:27:25.174 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:25.166+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.173508+0000) 2019-10-13T11:27:25.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:25.170+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.174177+0000) 2019-10-13T11:27:25.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:25.170+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:25.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:25.174+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:25.178657+0000) 2019-10-13T11:27:27.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:27.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:27.795494+0000) 2019-10-13T11:27:28.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:28.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:28.564681+0000) 2019-10-13T11:27:30.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:30.174+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.179706+0000) 2019-10-13T11:27:30.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:30.174+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.180411+0000) 2019-10-13T11:27:30.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:30.174+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.181030+0000) 2019-10-13T11:27:30.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:30.174+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.181600+0000) 2019-10-13T11:27:30.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:30.178+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.182234+0000) 2019-10-13T11:27:30.183 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:30.178+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.182546+0000) 2019-10-13T11:27:30.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:30.178+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.182938+0000) 2019-10-13T11:27:30.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:30.182+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:30.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:30.186+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:30.188951+0000) 2019-10-13T11:27:31.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:31.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.272900+0000) 2019-10-13T11:27:31.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:31.290+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.293983+0000) 2019-10-13T11:27:31.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:31.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:31.816045+0000) 2019-10-13T11:27:32.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:32.558+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:32.559444+0000) 2019-10-13T11:27:34.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:34.130+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.136915+0000) 2019-10-13T11:27:34.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:34.418+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:34.423739+0000) 2019-10-13T11:27:35.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:35.186+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.190577+0000) 2019-10-13T11:27:35.192 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:35.186+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.190743+0000) 2019-10-13T11:27:35.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:35.186+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.192207+0000) 2019-10-13T11:27:35.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:35.186+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.192390+0000) 2019-10-13T11:27:35.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:35.186+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.192696+0000) 2019-10-13T11:27:35.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:35.186+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.193112+0000) 2019-10-13T11:27:35.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:35.190+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.194735+0000) 2019-10-13T11:27:35.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:35.190+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:35.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:35.194+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:35.197854+0000) 2019-10-13T11:27:37.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:37.790+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:37.795693+0000) 2019-10-13T11:27:38.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:38.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:38.564875+0000) 2019-10-13T11:27:40.200 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:40.193+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.199614+0000) 2019-10-13T11:27:40.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:40.193+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.200258+0000) 2019-10-13T11:27:40.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:40.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.200809+0000) 2019-10-13T11:27:40.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:40.197+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.202045+0000) 2019-10-13T11:27:40.205 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:40.197+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.204523+0000) 2019-10-13T11:27:40.207 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:40.202+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.206264+0000) 2019-10-13T11:27:40.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:40.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:40.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:40.201+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.207731+0000) 2019-10-13T11:27:40.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:40.214+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:40.219091+0000) 2019-10-13T11:27:41.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:41.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.273158+0000) 2019-10-13T11:27:41.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:41.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.294182+0000) 2019-10-13T11:27:41.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:41.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:41.816280+0000) 2019-10-13T11:27:42.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:42.562+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:42.565237+0000) 2019-10-13T11:27:44.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:44.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.137114+0000) 2019-10-13T11:27:44.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:44.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:44.423969+0000) 2019-10-13T11:27:45.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:45.205+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.210978+0000) 2019-10-13T11:27:45.212 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:45.205+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.211426+0000) 2019-10-13T11:27:45.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:45.205+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.212168+0000) 2019-10-13T11:27:45.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:45.205+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.212496+0000) 2019-10-13T11:27:45.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:45.205+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.212809+0000) 2019-10-13T11:27:45.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:45.205+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.213279+0000) 2019-10-13T11:27:45.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:45.210+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.214056+0000) 2019-10-13T11:27:45.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:45.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:45.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:45.218+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:45.220525+0000) 2019-10-13T11:27:47.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:47.789+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:47.795919+0000) 2019-10-13T11:27:48.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:48.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:48.565052+0000) 2019-10-13T11:27:50.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:50.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.220726+0000) 2019-10-13T11:27:50.222 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:50.213+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.221012+0000) 2019-10-13T11:27:50.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:50.217+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.222873+0000) 2019-10-13T11:27:50.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:50.217+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.223163+0000) 2019-10-13T11:27:50.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:50.218+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.223599+0000) 2019-10-13T11:27:50.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:50.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:50.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:50.221+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.227398+0000) 2019-10-13T11:27:50.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:50.221+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.228299+0000) 2019-10-13T11:27:50.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:50.230+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:50.235064+0000) 2019-10-13T11:27:51.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:51.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.273383+0000) 2019-10-13T11:27:51.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:51.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.294415+0000) 2019-10-13T11:27:51.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:51.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:51.816469+0000) 2019-10-13T11:27:52.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:52.566+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:52.571620+0000) 2019-10-13T11:27:53.012 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:27:53.013 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:53.016 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:27:53.017 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:27:54.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:54.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.137332+0000) 2019-10-13T11:27:54.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:54.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:54.424184+0000) 2019-10-13T11:27:55.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:27:55.226+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.230084+0000) 2019-10-13T11:27:55.231 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:55.229+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.230937+0000) 2019-10-13T11:27:55.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:27:55.229+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.231188+0000) 2019-10-13T11:27:55.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:27:55.229+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.231551+0000) 2019-10-13T11:27:55.233 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:27:55.229+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.232022+0000) 2019-10-13T11:27:55.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:55.233+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.235555+0000) 2019-10-13T11:27:55.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:27:55.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:27:55.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:27:55.233+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.237356+0000) 2019-10-13T11:27:55.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:27:55.234+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:55.240061+0000) 2019-10-13T11:27:57.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:27:57.793+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:57.796123+0000) 2019-10-13T11:27:58.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:27:58.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:27:58.565244+0000) 2019-10-13T11:28:00.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:00.237+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.240096+0000) 2019-10-13T11:28:00.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:00.237+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.241077+0000) 2019-10-13T11:28:00.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:00.238+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.241207+0000) 2019-10-13T11:28:00.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:00.241+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.243165+0000) 2019-10-13T11:28:00.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:00.241+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.245293+0000) 2019-10-13T11:28:00.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:00.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.246023+0000) 2019-10-13T11:28:00.247 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:00.245+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.246738+0000) 2019-10-13T11:28:00.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:00.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:00.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:00.246+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:00.249684+0000) 2019-10-13T11:28:01.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:01.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.273581+0000) 2019-10-13T11:28:01.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:01.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.294625+0000) 2019-10-13T11:28:01.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:01.814+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:01.816661+0000) 2019-10-13T11:28:02.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:02.578+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:02.578276+0000) 2019-10-13T11:28:04.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:04.133+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.137534+0000) 2019-10-13T11:28:04.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:04.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:04.424390+0000) 2019-10-13T11:28:05.252 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:05.249+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.251207+0000) 2019-10-13T11:28:05.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:05.249+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.251543+0000) 2019-10-13T11:28:05.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:05.249+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.252039+0000) 2019-10-13T11:28:05.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:05.253+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.253826+0000) 2019-10-13T11:28:05.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:05.253+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.254239+0000) 2019-10-13T11:28:05.255 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:05.250+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.254725+0000) 2019-10-13T11:28:05.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:05.253+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.255213+0000) 2019-10-13T11:28:05.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:05.253+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:05.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:05.254+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:05.257751+0000) 2019-10-13T11:28:07.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:07.793+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:07.796381+0000) 2019-10-13T11:28:08.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:08.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:08.565460+0000) 2019-10-13T11:28:10.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:10.257+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.260498+0000) 2019-10-13T11:28:10.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:10.257+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.261456+0000) 2019-10-13T11:28:10.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:10.261+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.261778+0000) 2019-10-13T11:28:10.263 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:10.257+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.262797+0000) 2019-10-13T11:28:10.264 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:10.261+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.263538+0000) 2019-10-13T11:28:10.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:10.261+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.263447+0000) 2019-10-13T11:28:10.265 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:10.261+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.264713+0000) 2019-10-13T11:28:10.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:10.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:10.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:10.265+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:10.270231+0000) 2019-10-13T11:28:11.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:11.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.273811+0000) 2019-10-13T11:28:11.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:11.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.294820+0000) 2019-10-13T11:28:11.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:11.813+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:11.816873+0000) 2019-10-13T11:28:12.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:12.581+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:12.584852+0000) 2019-10-13T11:28:14.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:14.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.137737+0000) 2019-10-13T11:28:14.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:14.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:14.424583+0000) 2019-10-13T11:28:15.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:15.269+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.270413+0000) 2019-10-13T11:28:15.272 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:15.269+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.271562+0000) 2019-10-13T11:28:15.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:15.269+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.271344+0000) 2019-10-13T11:28:15.272 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:15.269+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.272091+0000) 2019-10-13T11:28:15.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:15.269+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.272334+0000) 2019-10-13T11:28:15.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:15.269+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.273284+0000) 2019-10-13T11:28:15.274 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:15.269+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.273670+0000) 2019-10-13T11:28:15.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:15.277+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:15.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:15.273+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:15.278014+0000) 2019-10-13T11:28:17.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:17.793+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:17.796599+0000) 2019-10-13T11:28:18.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:18.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:18.565654+0000) 2019-10-13T11:28:20.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:20.277+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.280725+0000) 2019-10-13T11:28:20.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:20.277+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.281371+0000) 2019-10-13T11:28:20.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:20.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.281697+0000) 2019-10-13T11:28:20.283 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:20.277+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.282958+0000) 2019-10-13T11:28:20.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:20.281+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.283397+0000) 2019-10-13T11:28:20.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:20.281+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.284155+0000) 2019-10-13T11:28:20.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:20.281+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.284966+0000) 2019-10-13T11:28:20.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:20.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:20.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:20.289+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:20.292671+0000) 2019-10-13T11:28:21.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:21.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.273976+0000) 2019-10-13T11:28:21.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:21.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.295019+0000) 2019-10-13T11:28:21.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:21.813+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:21.817065+0000) 2019-10-13T11:28:22.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:22.589+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:22.591397+0000) 2019-10-13T11:28:23.150 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:28:23.150 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:23.154 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:28:23.154 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:24.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:24.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.138033+0000) 2019-10-13T11:28:24.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:24.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:24.424817+0000) 2019-10-13T11:28:25.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:25.289+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.290427+0000) 2019-10-13T11:28:25.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:25.289+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.292682+0000) 2019-10-13T11:28:25.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:25.289+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.293266+0000) 2019-10-13T11:28:25.295 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:25.293+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.295147+0000) 2019-10-13T11:28:25.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:25.293+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.295737+0000) 2019-10-13T11:28:25.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:25.293+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.296223+0000) 2019-10-13T11:28:25.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:25.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139420 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:25.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:25.297+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.298132+0000) 2019-10-13T11:28:25.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:25.297+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:25.303631+0000) 2019-10-13T11:28:27.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:27.793+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:27.796837+0000) 2019-10-13T11:28:28.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:28.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:28.565869+0000) 2019-10-13T11:28:30.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:30.297+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.300927+0000) 2019-10-13T11:28:30.303 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:30.297+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.303044+0000) 2019-10-13T11:28:30.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:30.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.303670+0000) 2019-10-13T11:28:30.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:30.301+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.303813+0000) 2019-10-13T11:28:30.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:30.301+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.304684+0000) 2019-10-13T11:28:30.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:30.301+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.304971+0000) 2019-10-13T11:28:30.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:30.305+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.306369+0000) 2019-10-13T11:28:30.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:30.305+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:30.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:30.309+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:30.314873+0000) 2019-10-13T11:28:31.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:31.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.274200+0000) 2019-10-13T11:28:31.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:31.293+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.295235+0000) 2019-10-13T11:28:31.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:31.813+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:31.817261+0000) 2019-10-13T11:28:32.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:32.593+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:32.598077+0000) 2019-10-13T11:28:34.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:34.137+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.138246+0000) 2019-10-13T11:28:34.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:34.421+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:34.425069+0000) 2019-10-13T11:28:35.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:35.309+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.313204+0000) 2019-10-13T11:28:35.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:35.313+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.313828+0000) 2019-10-13T11:28:35.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:35.313+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.314233+0000) 2019-10-13T11:28:35.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:35.309+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.314605+0000) 2019-10-13T11:28:35.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:35.313+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.315166+0000) 2019-10-13T11:28:35.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:35.313+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.315561+0000) 2019-10-13T11:28:35.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:35.313+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.315930+0000) 2019-10-13T11:28:35.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:35.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:35.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:35.317+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:35.320920+0000) 2019-10-13T11:28:37.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:37.792+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:37.797054+0000) 2019-10-13T11:28:38.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:38.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:38.566089+0000) 2019-10-13T11:28:40.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:40.320+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.322452+0000) 2019-10-13T11:28:40.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:40.320+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.322916+0000) 2019-10-13T11:28:40.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:40.317+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.323071+0000) 2019-10-13T11:28:40.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:40.320+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.325649+0000) 2019-10-13T11:28:40.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:40.324+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.326412+0000) 2019-10-13T11:28:40.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:40.324+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.327142+0000) 2019-10-13T11:28:40.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:40.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.327768+0000) 2019-10-13T11:28:40.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:40.324+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:40.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:40.325+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:40.331715+0000) 2019-10-13T11:28:41.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:41.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.274404+0000) 2019-10-13T11:28:41.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:41.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.295404+0000) 2019-10-13T11:28:41.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:41.813+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:41.817490+0000) 2019-10-13T11:28:42.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:42.601+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:42.604749+0000) 2019-10-13T11:28:44.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:44.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.138447+0000) 2019-10-13T11:28:44.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:44.420+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:44.425267+0000) 2019-10-13T11:28:45.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:45.328+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.332341+0000) 2019-10-13T11:28:45.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:45.328+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.333277+0000) 2019-10-13T11:28:45.334 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:45.332+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.333923+0000) 2019-10-13T11:28:45.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:45.332+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.334206+0000) 2019-10-13T11:28:45.335 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:45.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.334672+0000) 2019-10-13T11:28:45.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:45.332+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.336275+0000) 2019-10-13T11:28:45.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:45.332+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.336443+0000) 2019-10-13T11:28:45.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:45.333+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:45.339529+0000) 2019-10-13T11:28:45.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:45.336+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139546 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:47.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:47.796+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:47.797259+0000) 2019-10-13T11:28:48.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:48.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:48.566318+0000) 2019-10-13T11:28:50.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:50.340+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.343527+0000) 2019-10-13T11:28:50.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:50.340+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.343752+0000) 2019-10-13T11:28:50.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:50.340+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.344959+0000) 2019-10-13T11:28:50.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:50.340+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.345364+0000) 2019-10-13T11:28:50.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:50.344+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.346042+0000) 2019-10-13T11:28:50.347 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:50.344+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.346231+0000) 2019-10-13T11:28:50.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:50.341+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.346669+0000) 2019-10-13T11:28:50.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:50.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:50.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:50.349+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:50.353137+0000) 2019-10-13T11:28:51.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:51.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.274597+0000) 2019-10-13T11:28:51.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:51.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.295578+0000) 2019-10-13T11:28:51.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:51.813+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:51.817697+0000) 2019-10-13T11:28:52.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:52.613+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:52.616073+0000) 2019-10-13T11:28:53.268 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:28:53.268 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:53.272 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:28:53.273 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:28:54.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:54.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.138655+0000) 2019-10-13T11:28:54.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:54.420+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:54.425465+0000) 2019-10-13T11:28:55.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:28:55.349+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.353199+0000) 2019-10-13T11:28:55.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:55.352+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.354158+0000) 2019-10-13T11:28:55.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:28:55.356+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.357964+0000) 2019-10-13T11:28:55.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:28:55.356+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.359936+0000) 2019-10-13T11:28:55.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:28:55.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:28:55.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:28:55.356+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.360773+0000) 2019-10-13T11:28:55.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:28:55.357+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.361212+0000) 2019-10-13T11:28:55.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:55.356+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.360994+0000) 2019-10-13T11:28:55.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:28:55.356+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:55.360901+0000) 2019-10-13T11:28:57.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:28:57.792+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:57.797432+0000) 2019-10-13T11:28:58.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:28:58.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:28:58.566511+0000) 2019-10-13T11:29:00.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:00.360+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.363953+0000) 2019-10-13T11:29:00.365 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:00.360+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.364487+0000) 2019-10-13T11:29:00.365 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:00.360+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.364635+0000) 2019-10-13T11:29:00.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:00.361+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.364927+0000) 2019-10-13T11:29:00.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:00.360+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.365535+0000) 2019-10-13T11:29:00.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:00.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.366091+0000) 2019-10-13T11:29:00.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:00.364+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.366850+0000) 2019-10-13T11:29:00.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:00.365+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:00.369683+0000) 2019-10-13T11:29:00.371 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:00.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:01.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:01.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.274794+0000) 2019-10-13T11:29:01.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:01.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.295758+0000) 2019-10-13T11:29:01.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:01.817+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:01.817898+0000) 2019-10-13T11:29:02.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:02.621+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:02.622618+0000) 2019-10-13T11:29:04.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:04.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.138903+0000) 2019-10-13T11:29:04.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:04.420+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:04.425659+0000) 2019-10-13T11:29:05.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:05.372+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.374353+0000) 2019-10-13T11:29:05.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:05.369+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.375685+0000) 2019-10-13T11:29:05.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:05.372+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.376495+0000) 2019-10-13T11:29:05.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:05.372+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.376395+0000) 2019-10-13T11:29:05.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:05.372+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.376735+0000) 2019-10-13T11:29:05.378 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:05.372+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.377302+0000) 2019-10-13T11:29:05.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:05.372+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.377628+0000) 2019-10-13T11:29:05.381 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:05.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:05.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:05.377+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:05.381656+0000) 2019-10-13T11:29:07.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:07.792+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:07.797689+0000) 2019-10-13T11:29:08.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:08.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:08.566714+0000) 2019-10-13T11:29:10.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:10.380+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.383967+0000) 2019-10-13T11:29:10.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:10.380+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.385723+0000) 2019-10-13T11:29:10.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:10.384+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.386039+0000) 2019-10-13T11:29:10.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:10.384+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.388280+0000) 2019-10-13T11:29:10.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:10.385+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.388295+0000) 2019-10-13T11:29:10.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:10.384+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.388868+0000) 2019-10-13T11:29:10.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:10.384+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.389326+0000) 2019-10-13T11:29:10.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:10.388+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139699 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:10.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:10.385+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:10.391341+0000) 2019-10-13T11:29:11.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:11.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.274975+0000) 2019-10-13T11:29:11.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:11.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.295942+0000) 2019-10-13T11:29:11.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:11.817+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:11.818128+0000) 2019-10-13T11:29:12.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:12.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:12.634148+0000) 2019-10-13T11:29:14.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:14.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.139111+0000) 2019-10-13T11:29:14.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:14.424+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:14.425857+0000) 2019-10-13T11:29:15.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:15.392+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.395735+0000) 2019-10-13T11:29:15.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:15.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.396377+0000) 2019-10-13T11:29:15.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:15.392+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.396506+0000) 2019-10-13T11:29:15.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:15.392+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.397137+0000) 2019-10-13T11:29:15.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:15.392+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.397383+0000) 2019-10-13T11:29:15.399 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:15.396+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.398912+0000) 2019-10-13T11:29:15.400 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:15.392+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.398940+0000) 2019-10-13T11:29:15.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:15.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:15.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:15.400+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:15.404417+0000) 2019-10-13T11:29:17.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:17.796+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:17.797915+0000) 2019-10-13T11:29:18.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:18.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:18.566927+0000) 2019-10-13T11:29:20.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:20.400+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.405225+0000) 2019-10-13T11:29:20.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:20.404+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.405909+0000) 2019-10-13T11:29:20.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:20.404+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.407375+0000) 2019-10-13T11:29:20.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:20.404+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.407841+0000) 2019-10-13T11:29:20.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:20.404+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.408283+0000) 2019-10-13T11:29:20.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:20.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139761 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:20.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:20.408+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.412344+0000) 2019-10-13T11:29:20.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:20.408+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.413058+0000) 2019-10-13T11:29:20.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:20.412+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:20.417464+0000) 2019-10-13T11:29:21.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:21.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.275143+0000) 2019-10-13T11:29:21.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:21.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.296107+0000) 2019-10-13T11:29:21.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:21.812+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:21.818328+0000) 2019-10-13T11:29:22.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:22.636+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:22.640840+0000) 2019-10-13T11:29:23.370 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:29:23.371 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:23.374 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:29:23.374 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:24.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:24.136+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.139317+0000) 2019-10-13T11:29:24.426 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:24.424+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:24.426053+0000) 2019-10-13T11:29:25.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:25.412+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.415152+0000) 2019-10-13T11:29:25.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:25.412+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.416051+0000) 2019-10-13T11:29:25.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:25.412+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.415850+0000) 2019-10-13T11:29:25.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:25.412+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.416316+0000) 2019-10-13T11:29:25.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:25.412+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.416865+0000) 2019-10-13T11:29:25.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:25.416+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.418442+0000) 2019-10-13T11:29:25.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:25.416+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.419001+0000) 2019-10-13T11:29:25.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:25.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:25.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:25.420+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:25.426589+0000) 2019-10-13T11:29:27.799 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:27.796+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:27.798083+0000) 2019-10-13T11:29:28.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:28.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:28.567116+0000) 2019-10-13T11:29:30.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:30.424+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.426767+0000) 2019-10-13T11:29:30.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:30.424+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.427222+0000) 2019-10-13T11:29:30.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:30.424+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.429069+0000) 2019-10-13T11:29:30.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:30.424+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.429304+0000) 2019-10-13T11:29:30.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:30.428+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.430754+0000) 2019-10-13T11:29:30.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:30.428+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.432215+0000) 2019-10-13T11:29:30.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:30.428+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:30.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:30.432+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.434231+0000) 2019-10-13T11:29:30.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:30.432+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:30.438640+0000) 2019-10-13T11:29:31.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:31.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.275297+0000) 2019-10-13T11:29:31.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:31.292+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.296312+0000) 2019-10-13T11:29:31.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:31.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:31.818573+0000) 2019-10-13T11:29:32.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:32.640+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:32.647402+0000) 2019-10-13T11:29:34.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:34.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.139525+0000) 2019-10-13T11:29:34.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:34.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:34.426273+0000) 2019-10-13T11:29:35.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:35.431+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.436979+0000) 2019-10-13T11:29:35.438 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:35.431+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.437691+0000) 2019-10-13T11:29:35.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:35.435+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.438317+0000) 2019-10-13T11:29:35.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:35.432+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.438745+0000) 2019-10-13T11:29:35.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:35.435+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.439296+0000) 2019-10-13T11:29:35.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:35.435+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.439945+0000) 2019-10-13T11:29:35.441 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:35.435+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.440381+0000) 2019-10-13T11:29:35.445 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:35.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:35.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:35.440+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:35.446893+0000) 2019-10-13T11:29:37.799 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:37.795+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:37.798274+0000) 2019-10-13T11:29:38.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:38.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:38.567303+0000) 2019-10-13T11:29:40.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:40.443+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.448371+0000) 2019-10-13T11:29:40.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:40.443+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.448643+0000) 2019-10-13T11:29:40.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:40.447+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.449825+0000) 2019-10-13T11:29:40.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:40.447+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.450237+0000) 2019-10-13T11:29:40.451 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:40.447+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.450001+0000) 2019-10-13T11:29:40.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:40.447+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.450548+0000) 2019-10-13T11:29:40.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:40.444+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.451089+0000) 2019-10-13T11:29:40.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:40.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139887 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:40.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:40.452+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:40.457010+0000) 2019-10-13T11:29:41.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:41.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.275467+0000) 2019-10-13T11:29:41.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:41.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.296485+0000) 2019-10-13T11:29:41.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:41.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:41.818782+0000) 2019-10-13T11:29:42.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:42.652+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:42.653462+0000) 2019-10-13T11:29:44.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:44.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.139752+0000) 2019-10-13T11:29:44.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:44.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:44.426462+0000) 2019-10-13T11:29:45.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:45.455+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.459050+0000) 2019-10-13T11:29:45.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:45.455+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.459742+0000) 2019-10-13T11:29:45.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:45.455+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.459987+0000) 2019-10-13T11:29:45.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:45.455+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.460454+0000) 2019-10-13T11:29:45.463 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:45.456+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.462389+0000) 2019-10-13T11:29:45.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:45.459+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.462880+0000) 2019-10-13T11:29:45.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:45.460+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.464486+0000) 2019-10-13T11:29:45.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:45.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:45.466 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:45.459+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:45.465063+0000) 2019-10-13T11:29:47.799 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:47.795+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:47.798493+0000) 2019-10-13T11:29:48.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:48.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:48.567496+0000) 2019-10-13T11:29:50.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:50.463+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.468609+0000) 2019-10-13T11:29:50.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:50.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.468310+0000) 2019-10-13T11:29:50.470 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:50.464+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.469449+0000) 2019-10-13T11:29:50.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:50.463+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.469415+0000) 2019-10-13T11:29:50.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:50.467+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.472569+0000) 2019-10-13T11:29:50.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:50.467+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.472915+0000) 2019-10-13T11:29:50.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:50.471+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.473815+0000) 2019-10-13T11:29:50.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:50.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:50.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:50.472+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:50.477252+0000) 2019-10-13T11:29:51.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:51.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.275635+0000) 2019-10-13T11:29:51.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:51.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.296631+0000) 2019-10-13T11:29:51.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:51.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:51.818972+0000) 2019-10-13T11:29:52.661 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:52.656+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:52.660145+0000) 2019-10-13T11:29:53.456 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:29:53.456 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:53.460 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:29:53.460 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:29:54.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:54.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.139985+0000) 2019-10-13T11:29:54.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:54.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:54.426703+0000) 2019-10-13T11:29:55.480 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:29:55.475+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.479590+0000) 2019-10-13T11:29:55.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:29:55.472+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.479720+0000) 2019-10-13T11:29:55.482 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:55.475+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.480235+0000) 2019-10-13T11:29:55.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:29:55.475+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.480744+0000) 2019-10-13T11:29:55.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:29:55.475+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.481217+0000) 2019-10-13T11:29:55.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:29:55.479+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.482005+0000) 2019-10-13T11:29:55.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:55.479+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.482629+0000) 2019-10-13T11:29:55.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:29:55.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:29:55.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:29:55.480+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:55.486577+0000) 2019-10-13T11:29:57.799 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:29:57.795+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:57.798682+0000) 2019-10-13T11:29:58.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:29:58.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:29:58.567692+0000) 2019-10-13T11:30:00.488 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:00.480+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.487841+0000) 2019-10-13T11:30:00.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:00.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.488323+0000) 2019-10-13T11:30:00.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:00.483+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.488787+0000) 2019-10-13T11:30:00.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:00.483+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.489079+0000) 2019-10-13T11:30:00.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:00.487+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.490157+0000) 2019-10-13T11:30:00.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:00.487+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.490530+0000) 2019-10-13T11:30:00.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:00.487+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.491588+0000) 2019-10-13T11:30:00.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:00.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:00.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:00.492+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:00.497616+0000) 2019-10-13T11:30:01.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:01.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.275786+0000) 2019-10-13T11:30:01.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:01.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.296852+0000) 2019-10-13T11:30:01.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:01.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:01.819161+0000) 2019-10-13T11:30:02.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:02.664+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:02.666596+0000) 2019-10-13T11:30:04.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:04.135+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.140193+0000) 2019-10-13T11:30:04.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:04.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:04.426925+0000) 2019-10-13T11:30:05.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:05.491+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.497569+0000) 2019-10-13T11:30:05.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:05.495+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.498713+0000) 2019-10-13T11:30:05.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:05.492+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.498900+0000) 2019-10-13T11:30:05.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:05.495+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.499562+0000) 2019-10-13T11:30:05.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:05.495+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.499981+0000) 2019-10-13T11:30:05.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:05.495+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.500752+0000) 2019-10-13T11:30:05.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:05.495+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.500565+0000) 2019-10-13T11:30:05.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:05.499+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:05.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:05.500+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:05.507334+0000) 2019-10-13T11:30:07.799 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:07.795+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:07.798908+0000) 2019-10-13T11:30:08.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:08.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:08.567889+0000) 2019-10-13T11:30:10.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:10.503+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.508485+0000) 2019-10-13T11:30:10.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:10.507+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.510318+0000) 2019-10-13T11:30:10.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:10.504+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.510918+0000) 2019-10-13T11:30:10.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:10.507+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.511487+0000) 2019-10-13T11:30:10.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:10.507+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.513125+0000) 2019-10-13T11:30:10.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:10.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140077 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:10.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:10.511+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.515241+0000) 2019-10-13T11:30:10.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:10.511+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.515493+0000) 2019-10-13T11:30:10.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:10.512+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:10.517222+0000) 2019-10-13T11:30:11.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:11.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.275946+0000) 2019-10-13T11:30:11.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:11.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.297030+0000) 2019-10-13T11:30:11.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:11.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:11.819384+0000) 2019-10-13T11:30:12.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:12.672+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:12.673273+0000) 2019-10-13T11:30:14.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:14.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.140395+0000) 2019-10-13T11:30:14.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:14.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:14.427130+0000) 2019-10-13T11:30:15.518 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:15.516+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.517550+0000) 2019-10-13T11:30:15.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:15.515+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.518067+0000) 2019-10-13T11:30:15.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:15.515+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.518523+0000) 2019-10-13T11:30:15.520 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:15.515+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.519221+0000) 2019-10-13T11:30:15.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:15.515+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.519835+0000) 2019-10-13T11:30:15.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:15.515+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.520277+0000) 2019-10-13T11:30:15.521 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:15.515+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.520696+0000) 2019-10-13T11:30:15.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:15.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140111 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:15.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:15.528+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:15.528127+0000) 2019-10-13T11:30:17.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:17.795+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:17.799114+0000) 2019-10-13T11:30:18.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:18.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:18.568083+0000) 2019-10-13T11:30:20.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:20.523+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.528684+0000) 2019-10-13T11:30:20.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:20.527+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.529967+0000) 2019-10-13T11:30:20.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:20.527+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.530258+0000) 2019-10-13T11:30:20.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:20.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.531456+0000) 2019-10-13T11:30:20.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:20.531+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.532149+0000) 2019-10-13T11:30:20.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:20.527+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.532613+0000) 2019-10-13T11:30:20.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:20.531+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.534964+0000) 2019-10-13T11:30:20.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:20.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:20.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:20.539+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:20.540584+0000) 2019-10-13T11:30:21.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:21.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.276097+0000) 2019-10-13T11:30:21.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:21.291+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.297189+0000) 2019-10-13T11:30:21.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:21.815+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:21.819581+0000) 2019-10-13T11:30:22.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:22.675+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:22.679855+0000) 2019-10-13T11:30:23.672 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:30:23.672 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:23.676 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:30:23.676 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:24.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:24.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.140638+0000) 2019-10-13T11:30:24.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:24.423+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:24.427357+0000) 2019-10-13T11:30:25.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:25.535+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.539867+0000) 2019-10-13T11:30:25.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:25.535+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.540266+0000) 2019-10-13T11:30:25.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:25.539+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.541013+0000) 2019-10-13T11:30:25.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:25.535+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.541342+0000) 2019-10-13T11:30:25.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:25.535+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.541647+0000) 2019-10-13T11:30:25.543 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:25.539+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.542749+0000) 2019-10-13T11:30:25.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:25.539+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.543368+0000) 2019-10-13T11:30:25.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:25.543+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:25.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:25.543+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:25.547222+0000) 2019-10-13T11:30:27.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:27.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:27.799308+0000) 2019-10-13T11:30:28.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:28.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:28.568310+0000) 2019-10-13T11:30:30.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:30.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.549584+0000) 2019-10-13T11:30:30.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:30.546+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.550270+0000) 2019-10-13T11:30:30.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:30.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.551911+0000) 2019-10-13T11:30:30.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:30.551+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.552070+0000) 2019-10-13T11:30:30.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:30.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.552602+0000) 2019-10-13T11:30:30.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:30.550+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.554780+0000) 2019-10-13T11:30:30.557 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:30.550+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.557043+0000) 2019-10-13T11:30:30.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:30.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:30.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:30.559+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:30.563545+0000) 2019-10-13T11:30:31.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:31.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.276253+0000) 2019-10-13T11:30:31.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:31.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.297320+0000) 2019-10-13T11:30:31.820 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:31.815+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:31.819792+0000) 2019-10-13T11:30:32.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:32.683+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:32.686577+0000) 2019-10-13T11:30:34.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:34.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.140851+0000) 2019-10-13T11:30:34.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:34.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:34.427565+0000) 2019-10-13T11:30:35.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:35.558+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.561993+0000) 2019-10-13T11:30:35.563 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:35.558+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.562894+0000) 2019-10-13T11:30:35.565 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:35.558+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.564430+0000) 2019-10-13T11:30:35.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:35.558+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.565661+0000) 2019-10-13T11:30:35.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:35.563+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.565919+0000) 2019-10-13T11:30:35.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:35.562+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.566725+0000) 2019-10-13T11:30:35.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:35.562+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.568048+0000) 2019-10-13T11:30:35.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:35.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:35.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:35.567+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:35.571224+0000) 2019-10-13T11:30:37.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:37.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:37.799462+0000) 2019-10-13T11:30:38.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:38.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:38.568520+0000) 2019-10-13T11:30:40.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:40.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.571582+0000) 2019-10-13T11:30:40.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:40.571+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.572218+0000) 2019-10-13T11:30:40.573 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:40.566+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.572864+0000) 2019-10-13T11:30:40.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:40.570+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.574251+0000) 2019-10-13T11:30:40.576 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:40.570+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.575375+0000) 2019-10-13T11:30:40.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:40.570+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.576356+0000) 2019-10-13T11:30:40.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:40.570+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.576083+0000) 2019-10-13T11:30:40.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:40.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:40.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:40.579+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:40.583424+0000) 2019-10-13T11:30:41.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:41.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.276387+0000) 2019-10-13T11:30:41.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:41.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.297470+0000) 2019-10-13T11:30:41.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:41.819+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:41.820012+0000) 2019-10-13T11:30:42.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:42.691+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:42.693152+0000) 2019-10-13T11:30:44.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:44.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.141058+0000) 2019-10-13T11:30:44.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:44.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:44.427784+0000) 2019-10-13T11:30:45.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:45.578+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.581412+0000) 2019-10-13T11:30:45.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:45.578+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.582399+0000) 2019-10-13T11:30:45.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:45.578+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.582919+0000) 2019-10-13T11:30:45.584 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:45.578+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.583581+0000) 2019-10-13T11:30:45.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:45.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.584069+0000) 2019-10-13T11:30:45.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:45.578+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.584662+0000) 2019-10-13T11:30:45.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:45.578+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.584730+0000) 2019-10-13T11:30:45.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:45.586+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:45.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:45.587+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:45.591247+0000) 2019-10-13T11:30:47.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:47.794+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:47.799615+0000) 2019-10-13T11:30:48.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:48.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:48.568717+0000) 2019-10-13T11:30:50.594 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:50.586+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.593432+0000) 2019-10-13T11:30:50.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:50.590+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.596007+0000) 2019-10-13T11:30:50.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:50.590+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.595735+0000) 2019-10-13T11:30:50.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:50.590+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.596840+0000) 2019-10-13T11:30:50.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:50.590+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.597108+0000) 2019-10-13T11:30:50.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:50.595+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.597155+0000) 2019-10-13T11:30:50.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:50.594+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.598443+0000) 2019-10-13T11:30:50.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:50.594+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140324 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:50.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:50.603+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:50.604617+0000) 2019-10-13T11:30:51.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:51.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.276561+0000) 2019-10-13T11:30:51.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:51.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.297656+0000) 2019-10-13T11:30:51.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:51.819+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:51.820206+0000) 2019-10-13T11:30:52.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:52.695+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:52.699770+0000) 2019-10-13T11:30:53.786 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:30:53.787 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:53.790 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:30:53.791 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:30:54.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:54.134+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.141295+0000) 2019-10-13T11:30:54.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:54.422+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:54.427976+0000) 2019-10-13T11:30:55.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:30:55.603+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.605504+0000) 2019-10-13T11:30:55.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:30:55.602+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.605973+0000) 2019-10-13T11:30:55.607 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:30:55.602+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.606564+0000) 2019-10-13T11:30:55.609 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:30:55.602+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.608108+0000) 2019-10-13T11:30:55.609 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:30:55.602+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.608310+0000) 2019-10-13T11:30:55.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:55.602+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.609144+0000) 2019-10-13T11:30:55.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:55.606+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.609849+0000) 2019-10-13T11:30:55.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:30:55.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:30:55.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:30:55.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:55.617847+0000) 2019-10-13T11:30:57.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:30:57.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:57.799816+0000) 2019-10-13T11:30:58.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:30:58.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:30:58.568910+0000) 2019-10-13T11:31:00.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:00.610+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.615733+0000) 2019-10-13T11:31:00.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:00.614+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.617797+0000) 2019-10-13T11:31:00.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:00.614+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.618144+0000) 2019-10-13T11:31:00.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:00.615+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.620012+0000) 2019-10-13T11:31:00.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:00.614+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.621076+0000) 2019-10-13T11:31:00.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:00.614+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.621505+0000) 2019-10-13T11:31:00.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:00.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:00.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:00.618+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.622830+0000) 2019-10-13T11:31:00.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:00.627+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:00.629348+0000) 2019-10-13T11:31:01.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:01.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.276720+0000) 2019-10-13T11:31:01.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:01.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.297792+0000) 2019-10-13T11:31:01.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:01.819+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:01.820410+0000) 2019-10-13T11:31:02.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:02.703+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:02.706500+0000) 2019-10-13T11:31:04.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:04.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.141508+0000) 2019-10-13T11:31:04.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:04.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:04.428174+0000) 2019-10-13T11:31:05.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:05.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.625730+0000) 2019-10-13T11:31:05.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:05.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.626224+0000) 2019-10-13T11:31:05.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:05.622+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.627172+0000) 2019-10-13T11:31:05.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:05.622+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.627608+0000) 2019-10-13T11:31:05.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:05.622+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.627830+0000) 2019-10-13T11:31:05.629 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:05.622+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.628641+0000) 2019-10-13T11:31:05.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:05.626+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.631580+0000) 2019-10-13T11:31:05.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:05.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:05.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:05.631+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:05.634383+0000) 2019-10-13T11:31:07.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:07.794+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:07.800032+0000) 2019-10-13T11:31:08.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:08.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:08.569118+0000) 2019-10-13T11:31:10.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:10.630+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.635355+0000) 2019-10-13T11:31:10.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:10.630+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.635825+0000) 2019-10-13T11:31:10.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:10.630+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.636456+0000) 2019-10-13T11:31:10.637 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:10.630+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.637297+0000) 2019-10-13T11:31:10.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:10.634+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.638138+0000) 2019-10-13T11:31:10.639 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:10.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.638866+0000) 2019-10-13T11:31:10.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:10.634+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.639097+0000) 2019-10-13T11:31:10.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:10.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:10.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:10.643+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:10.644656+0000) 2019-10-13T11:31:11.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:11.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.276855+0000) 2019-10-13T11:31:11.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:11.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.297917+0000) 2019-10-13T11:31:11.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:11.819+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:11.820603+0000) 2019-10-13T11:31:12.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:12.711+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:12.713117+0000) 2019-10-13T11:31:14.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:14.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.141772+0000) 2019-10-13T11:31:14.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:14.422+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:14.428401+0000) 2019-10-13T11:31:15.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:15.642+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.645060+0000) 2019-10-13T11:31:15.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:15.642+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.646797+0000) 2019-10-13T11:31:15.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:15.642+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.647102+0000) 2019-10-13T11:31:15.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:15.642+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.648017+0000) 2019-10-13T11:31:15.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:15.643+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.647987+0000) 2019-10-13T11:31:15.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:15.642+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.648643+0000) 2019-10-13T11:31:15.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:15.642+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.648881+0000) 2019-10-13T11:31:15.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:15.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:15.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:15.651+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:15.652330+0000) 2019-10-13T11:31:17.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:17.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:17.800262+0000) 2019-10-13T11:31:18.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:18.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:18.569312+0000) 2019-10-13T11:31:20.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:20.650+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.655119+0000) 2019-10-13T11:31:20.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:20.651+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.655555+0000) 2019-10-13T11:31:20.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:20.650+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.656474+0000) 2019-10-13T11:31:20.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:20.650+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.656651+0000) 2019-10-13T11:31:20.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:20.650+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.657214+0000) 2019-10-13T11:31:20.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:20.650+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.657609+0000) 2019-10-13T11:31:20.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:20.654+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.658743+0000) 2019-10-13T11:31:20.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:20.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:20.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:20.663+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:20.664661+0000) 2019-10-13T11:31:21.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:21.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.277052+0000) 2019-10-13T11:31:21.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:21.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.298069+0000) 2019-10-13T11:31:21.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:21.819+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:21.820792+0000) 2019-10-13T11:31:22.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:22.715+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:22.719626+0000) 2019-10-13T11:31:24.102 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:31:24.102 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:24.106 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:31:24.106 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:24.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:24.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.142004+0000) 2019-10-13T11:31:24.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:24.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:24.428630+0000) 2019-10-13T11:31:25.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:25.662+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.665368+0000) 2019-10-13T11:31:25.666 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:25.662+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.665679+0000) 2019-10-13T11:31:25.667 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:25.662+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.666193+0000) 2019-10-13T11:31:25.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:25.662+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.666756+0000) 2019-10-13T11:31:25.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:25.662+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.668439+0000) 2019-10-13T11:31:25.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:25.666+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.668720+0000) 2019-10-13T11:31:25.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:25.662+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.669540+0000) 2019-10-13T11:31:25.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:25.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:25.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:25.670+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:25.673754+0000) 2019-10-13T11:31:27.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:27.793+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:27.800446+0000) 2019-10-13T11:31:28.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:28.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:28.569513+0000) 2019-10-13T11:31:30.676 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:30.669+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.675606+0000) 2019-10-13T11:31:30.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:30.669+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.676103+0000) 2019-10-13T11:31:30.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:30.673+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.678199+0000) 2019-10-13T11:31:30.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:30.673+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.678595+0000) 2019-10-13T11:31:30.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:30.674+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.679573+0000) 2019-10-13T11:31:30.681 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:30.673+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.680262+0000) 2019-10-13T11:31:30.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:30.673+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.680862+0000) 2019-10-13T11:31:30.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:30.677+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:30.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:30.682+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:30.685102+0000) 2019-10-13T11:31:31.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:31.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.277233+0000) 2019-10-13T11:31:31.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:31.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.298248+0000) 2019-10-13T11:31:31.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:31.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:31.820991+0000) 2019-10-13T11:31:32.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:32.722+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:32.726209+0000) 2019-10-13T11:31:34.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:34.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.142215+0000) 2019-10-13T11:31:34.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:34.425+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:34.428858+0000) 2019-10-13T11:31:35.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:35.681+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.685824+0000) 2019-10-13T11:31:35.687 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:35.681+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.686145+0000) 2019-10-13T11:31:35.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:35.681+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.686843+0000) 2019-10-13T11:31:35.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:35.681+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.687254+0000) 2019-10-13T11:31:35.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:35.681+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.687931+0000) 2019-10-13T11:31:35.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:35.682+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.688049+0000) 2019-10-13T11:31:35.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:35.681+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.689222+0000) 2019-10-13T11:31:35.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:35.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:35.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:35.690+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:35.694811+0000) 2019-10-13T11:31:37.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:37.797+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:37.800677+0000) 2019-10-13T11:31:38.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:38.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:38.569706+0000) 2019-10-13T11:31:40.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:40.693+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.694942+0000) 2019-10-13T11:31:40.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:40.693+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.695969+0000) 2019-10-13T11:31:40.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:40.693+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.696422+0000) 2019-10-13T11:31:40.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:40.694+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.696860+0000) 2019-10-13T11:31:40.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:40.693+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.697560+0000) 2019-10-13T11:31:40.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:40.697+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.697967+0000) 2019-10-13T11:31:40.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:40.697+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.698682+0000) 2019-10-13T11:31:40.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:40.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:40.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:40.698+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:40.703385+0000) 2019-10-13T11:31:41.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:41.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.277411+0000) 2019-10-13T11:31:41.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:41.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.298479+0000) 2019-10-13T11:31:41.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:41.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:41.821219+0000) 2019-10-13T11:31:42.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:42.730+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:42.732599+0000) 2019-10-13T11:31:44.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:44.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.142428+0000) 2019-10-13T11:31:44.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:44.425+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:44.429084+0000) 2019-10-13T11:31:45.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:45.701+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.705098+0000) 2019-10-13T11:31:45.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:45.701+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.705670+0000) 2019-10-13T11:31:45.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:45.705+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.706668+0000) 2019-10-13T11:31:45.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:45.705+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.707045+0000) 2019-10-13T11:31:45.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:45.705+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.707788+0000) 2019-10-13T11:31:45.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:45.705+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.708252+0000) 2019-10-13T11:31:45.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:45.706+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.708562+0000) 2019-10-13T11:31:45.713 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:45.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:45.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:45.710+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:45.714364+0000) 2019-10-13T11:31:47.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:47.797+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:47.800876+0000) 2019-10-13T11:31:48.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:48.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:48.569927+0000) 2019-10-13T11:31:50.716 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:50.713+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.715719+0000) 2019-10-13T11:31:50.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:50.713+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.717719+0000) 2019-10-13T11:31:50.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:50.713+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.717196+0000) 2019-10-13T11:31:50.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:50.717+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.718837+0000) 2019-10-13T11:31:50.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:50.717+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.719059+0000) 2019-10-13T11:31:50.720 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:50.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.719355+0000) 2019-10-13T11:31:50.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:50.717+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.720752+0000) 2019-10-13T11:31:50.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:50.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:50.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:50.722+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:50.725278+0000) 2019-10-13T11:31:51.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:51.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.277611+0000) 2019-10-13T11:31:51.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:51.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.298654+0000) 2019-10-13T11:31:51.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:51.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:51.821420+0000) 2019-10-13T11:31:52.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:52.738+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:52.739164+0000) 2019-10-13T11:31:54.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:54.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.142579+0000) 2019-10-13T11:31:54.206 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:31:54.206 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:54.210 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:31:54.210 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:31:54.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:54.425+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:54.429319+0000) 2019-10-13T11:31:55.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:31:55.725+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.727061+0000) 2019-10-13T11:31:55.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:55.725+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.727500+0000) 2019-10-13T11:31:55.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:31:55.726+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.728316+0000) 2019-10-13T11:31:55.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:55.725+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.728090+0000) 2019-10-13T11:31:55.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:31:55.725+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.729159+0000) 2019-10-13T11:31:55.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:31:55.725+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.729442+0000) 2019-10-13T11:31:55.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:31:55.729+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.729994+0000) 2019-10-13T11:31:55.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:31:55.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:31:55.736 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:31:55.730+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:55.735646+0000) 2019-10-13T11:31:57.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:31:57.797+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:57.801090+0000) 2019-10-13T11:31:58.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:31:58.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:31:58.570118+0000) 2019-10-13T11:32:00.739 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:00.737+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.737904+0000) 2019-10-13T11:32:00.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:00.737+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.739130+0000) 2019-10-13T11:32:00.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:00.737+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.739109+0000) 2019-10-13T11:32:00.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:00.734+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.739920+0000) 2019-10-13T11:32:00.741 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:00.737+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.740235+0000) 2019-10-13T11:32:00.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:00.737+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.740752+0000) 2019-10-13T11:32:00.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:00.737+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.740985+0000) 2019-10-13T11:32:00.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:00.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:00.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:00.742+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:00.746051+0000) 2019-10-13T11:32:01.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:01.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.277787+0000) 2019-10-13T11:32:01.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:01.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.298851+0000) 2019-10-13T11:32:01.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:01.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:01.821572+0000) 2019-10-13T11:32:02.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:02.742+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:02.745706+0000) 2019-10-13T11:32:04.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:04.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.142792+0000) 2019-10-13T11:32:04.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:04.425+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:04.429535+0000) 2019-10-13T11:32:05.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:05.745+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.748443+0000) 2019-10-13T11:32:05.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:05.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.748545+0000) 2019-10-13T11:32:05.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:05.745+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.749561+0000) 2019-10-13T11:32:05.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:05.749+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.749820+0000) 2019-10-13T11:32:05.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:05.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.750341+0000) 2019-10-13T11:32:05.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:05.749+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.750670+0000) 2019-10-13T11:32:05.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:05.749+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.751698+0000) 2019-10-13T11:32:05.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:05.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:05.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:05.754+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:05.757110+0000) 2019-10-13T11:32:07.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:07.797+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:07.801291+0000) 2019-10-13T11:32:08.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:08.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:08.570327+0000) 2019-10-13T11:32:10.759 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:10.757+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.758874+0000) 2019-10-13T11:32:10.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:10.758+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.760529+0000) 2019-10-13T11:32:10.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:10.757+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.761384+0000) 2019-10-13T11:32:10.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:10.757+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.761632+0000) 2019-10-13T11:32:10.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:10.761+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.762621+0000) 2019-10-13T11:32:10.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:10.761+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.763747+0000) 2019-10-13T11:32:10.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:10.761+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.764593+0000) 2019-10-13T11:32:10.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:10.761+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:10.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:10.762+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:10.766531+0000) 2019-10-13T11:32:11.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:11.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.277923+0000) 2019-10-13T11:32:11.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:11.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.299061+0000) 2019-10-13T11:32:11.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:11.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:11.821779+0000) 2019-10-13T11:32:12.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:12.750+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:12.752243+0000) 2019-10-13T11:32:14.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:14.141+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.142993+0000) 2019-10-13T11:32:14.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:14.429+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:14.429732+0000) 2019-10-13T11:32:15.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:15.769+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.770425+0000) 2019-10-13T11:32:15.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:15.769+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.770763+0000) 2019-10-13T11:32:15.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:15.770+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.772404+0000) 2019-10-13T11:32:15.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:15.769+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.772756+0000) 2019-10-13T11:32:15.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:15.773+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.775067+0000) 2019-10-13T11:32:15.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:15.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.775737+0000) 2019-10-13T11:32:15.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:15.773+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.776145+0000) 2019-10-13T11:32:15.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:15.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:15.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:15.778+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:15.781096+0000) 2019-10-13T11:32:17.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:17.801+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:17.801496+0000) 2019-10-13T11:32:18.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:18.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:18.570561+0000) 2019-10-13T11:32:20.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:20.777+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.779758+0000) 2019-10-13T11:32:20.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:20.777+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.780248+0000) 2019-10-13T11:32:20.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:20.777+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.781101+0000) 2019-10-13T11:32:20.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:20.777+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.781556+0000) 2019-10-13T11:32:20.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:20.781+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.782190+0000) 2019-10-13T11:32:20.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:20.778+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.782720+0000) 2019-10-13T11:32:20.785 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:20.781+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.784492+0000) 2019-10-13T11:32:20.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:20.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:20.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:20.786+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:20.790554+0000) 2019-10-13T11:32:21.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:21.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.278140+0000) 2019-10-13T11:32:21.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:21.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.299237+0000) 2019-10-13T11:32:21.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:21.818+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:21.821973+0000) 2019-10-13T11:32:22.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:22.758+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:22.758747+0000) 2019-10-13T11:32:24.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:24.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.143201+0000) 2019-10-13T11:32:24.344 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:32:24.345 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:24.348 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:32:24.348 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:24.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:24.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:24.429940+0000) 2019-10-13T11:32:25.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:25.784+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.789401+0000) 2019-10-13T11:32:25.790 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:25.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.789797+0000) 2019-10-13T11:32:25.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:25.788+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.790471+0000) 2019-10-13T11:32:25.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:25.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.790578+0000) 2019-10-13T11:32:25.792 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:25.788+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.791234+0000) 2019-10-13T11:32:25.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:25.788+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.792082+0000) 2019-10-13T11:32:25.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:25.788+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.792995+0000) 2019-10-13T11:32:25.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:25.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:25.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:25.794+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:25.798518+0000) 2019-10-13T11:32:27.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:27.800+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:27.801696+0000) 2019-10-13T11:32:28.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:28.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:28.570801+0000) 2019-10-13T11:32:30.801 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:30.796+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.800618+0000) 2019-10-13T11:32:30.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:30.800+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.802774+0000) 2019-10-13T11:32:30.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:30.798+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.803209+0000) 2019-10-13T11:32:30.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:30.800+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.804875+0000) 2019-10-13T11:32:30.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:30.800+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.805562+0000) 2019-10-13T11:32:30.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:30.804+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.806673+0000) 2019-10-13T11:32:30.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:30.804+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.807270+0000) 2019-10-13T11:32:30.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:30.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:30.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:30.806+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:30.811458+0000) 2019-10-13T11:32:31.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:31.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.278318+0000) 2019-10-13T11:32:31.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:31.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.299432+0000) 2019-10-13T11:32:31.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:31.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:31.822196+0000) 2019-10-13T11:32:32.766 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:32.761+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:32.765189+0000) 2019-10-13T11:32:34.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:34.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.143421+0000) 2019-10-13T11:32:34.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:34.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:34.430160+0000) 2019-10-13T11:32:35.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:35.808+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.811052+0000) 2019-10-13T11:32:35.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:35.808+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.811989+0000) 2019-10-13T11:32:35.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:35.808+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.812284+0000) 2019-10-13T11:32:35.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:35.808+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.812630+0000) 2019-10-13T11:32:35.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:35.809+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.812924+0000) 2019-10-13T11:32:35.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:35.808+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.813651+0000) 2019-10-13T11:32:35.815 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:35.812+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.815106+0000) 2019-10-13T11:32:35.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:35.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:35.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:35.813+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:35.819539+0000) 2019-10-13T11:32:37.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:37.800+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:37.801912+0000) 2019-10-13T11:32:38.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:38.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:38.570956+0000) 2019-10-13T11:32:40.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:40.820+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.822626+0000) 2019-10-13T11:32:40.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:40.820+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.823338+0000) 2019-10-13T11:32:40.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:40.820+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.823668+0000) 2019-10-13T11:32:40.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:40.817+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.822940+0000) 2019-10-13T11:32:40.826 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:40.820+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.824076+0000) 2019-10-13T11:32:40.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:40.820+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.824299+0000) 2019-10-13T11:32:40.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:40.820+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.824943+0000) 2019-10-13T11:32:40.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:40.824+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:40.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:40.825+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:40.831169+0000) 2019-10-13T11:32:41.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:41.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.278497+0000) 2019-10-13T11:32:41.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:41.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.299628+0000) 2019-10-13T11:32:41.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:41.817+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:41.822394+0000) 2019-10-13T11:32:42.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:42.769+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:42.771890+0000) 2019-10-13T11:32:44.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:44.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.143646+0000) 2019-10-13T11:32:44.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:44.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:44.430364+0000) 2019-10-13T11:32:45.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:45.832+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.833816+0000) 2019-10-13T11:32:45.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:45.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.835552+0000) 2019-10-13T11:32:45.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:45.832+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.836073+0000) 2019-10-13T11:32:45.837 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:45.832+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.836548+0000) 2019-10-13T11:32:45.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:45.832+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.836965+0000) 2019-10-13T11:32:45.838 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:45.832+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.837195+0000) 2019-10-13T11:32:45.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:45.832+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.837569+0000) 2019-10-13T11:32:45.840 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:45.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:45.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:45.841+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:45.844598+0000) 2019-10-13T11:32:47.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:47.800+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:47.802111+0000) 2019-10-13T11:32:48.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:48.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:48.571150+0000) 2019-10-13T11:32:50.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:50.840+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.843144+0000) 2019-10-13T11:32:50.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:50.837+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.843231+0000) 2019-10-13T11:32:50.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:50.840+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.843712+0000) 2019-10-13T11:32:50.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:50.840+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.844874+0000) 2019-10-13T11:32:50.846 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:50.840+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.845624+0000) 2019-10-13T11:32:50.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:50.844+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.846479+0000) 2019-10-13T11:32:50.847 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:50.844+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.846712+0000) 2019-10-13T11:32:50.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:50.844+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:50.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:50.849+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:50.852443+0000) 2019-10-13T11:32:51.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:51.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.278687+0000) 2019-10-13T11:32:51.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:51.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.299804+0000) 2019-10-13T11:32:51.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:51.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:51.822632+0000) 2019-10-13T11:32:52.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:52.773+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:52.778440+0000) 2019-10-13T11:32:54.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:54.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.143845+0000) 2019-10-13T11:32:54.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:54.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:54.430564+0000) 2019-10-13T11:32:54.450 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:32:54.450 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:54.453 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:32:54.453 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:32:55.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:55.848+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.852718+0000) 2019-10-13T11:32:55.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:55.848+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.853549+0000) 2019-10-13T11:32:55.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:32:55.852+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.855309+0000) 2019-10-13T11:32:55.858 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:32:55.852+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.857370+0000) 2019-10-13T11:32:55.858 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:32:55.856+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.858273+0000) 2019-10-13T11:32:55.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:32:55.853+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.858453+0000) 2019-10-13T11:32:55.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:32:55.856+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.858813+0000) 2019-10-13T11:32:55.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:32:55.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:32:55.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:32:55.857+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:55.863918+0000) 2019-10-13T11:32:57.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:32:57.800+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:57.802329+0000) 2019-10-13T11:32:58.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:32:58.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:32:58.571356+0000) 2019-10-13T11:33:00.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:00.860+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.863527+0000) 2019-10-13T11:33:00.866 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:00.860+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.865631+0000) 2019-10-13T11:33:00.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:00.864+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.867421+0000) 2019-10-13T11:33:00.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:00.864+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.868378+0000) 2019-10-13T11:33:00.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:00.864+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.868567+0000) 2019-10-13T11:33:00.869 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:00.864+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.869043+0000) 2019-10-13T11:33:00.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:00.865+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.869388+0000) 2019-10-13T11:33:00.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:00.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141118 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:00.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:00.869+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:00.875595+0000) 2019-10-13T11:33:01.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:01.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.278857+0000) 2019-10-13T11:33:01.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:01.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.299976+0000) 2019-10-13T11:33:01.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:01.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:01.822830+0000) 2019-10-13T11:33:02.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:02.781+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:02.785107+0000) 2019-10-13T11:33:04.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:04.140+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.144045+0000) 2019-10-13T11:33:04.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:04.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:04.430811+0000) 2019-10-13T11:33:05.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:05.872+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.874796+0000) 2019-10-13T11:33:05.876 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:05.872+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.875448+0000) 2019-10-13T11:33:05.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:05.869+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.875841+0000) 2019-10-13T11:33:05.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:05.876+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.880634+0000) 2019-10-13T11:33:05.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:05.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:05.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:05.880+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.881818+0000) 2019-10-13T11:33:05.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:05.880+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.881823+0000) 2019-10-13T11:33:05.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:05.880+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.882086+0000) 2019-10-13T11:33:05.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:05.877+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:05.882385+0000) 2019-10-13T11:33:07.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:07.800+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:07.802525+0000) 2019-10-13T11:33:08.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:08.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:08.571549+0000) 2019-10-13T11:33:10.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:10.880+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.885236+0000) 2019-10-13T11:33:10.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:10.880+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.885532+0000) 2019-10-13T11:33:10.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:10.881+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.885814+0000) 2019-10-13T11:33:10.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:10.884+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.886396+0000) 2019-10-13T11:33:10.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:10.884+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.886631+0000) 2019-10-13T11:33:10.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:10.884+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.887597+0000) 2019-10-13T11:33:10.889 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:10.884+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.888325+0000) 2019-10-13T11:33:10.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:10.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:10.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:10.889+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:10.895572+0000) 2019-10-13T11:33:11.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:11.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.279030+0000) 2019-10-13T11:33:11.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:11.296+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.300192+0000) 2019-10-13T11:33:11.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:11.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:11.822983+0000) 2019-10-13T11:33:12.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:12.789+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:12.791070+0000) 2019-10-13T11:33:14.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:14.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:14.144250+0000) 2019-10-13T11:33:14.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:14.428+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:14.431033+0000) 2019-10-13T11:33:15.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:15.892+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.897489+0000) 2019-10-13T11:33:15.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:15.896+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.897784+0000) 2019-10-13T11:33:15.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:15.896+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.898184+0000) 2019-10-13T11:33:15.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:15.893+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.898929+0000) 2019-10-13T11:33:15.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:15.896+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.900022+0000) 2019-10-13T11:33:15.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:15.896+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.900279+0000) 2019-10-13T11:33:15.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:15.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.900519+0000) 2019-10-13T11:33:15.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:15.897+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:15.902194+0000) 2019-10-13T11:33:15.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:15.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:17.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:17.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:17.802751+0000) 2019-10-13T11:33:18.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:18.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:18.571740+0000) 2019-10-13T11:33:20.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:20.903+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.906457+0000) 2019-10-13T11:33:20.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:20.903+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.908729+0000) 2019-10-13T11:33:20.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:20.903+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.908521+0000) 2019-10-13T11:33:20.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:20.903+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.909608+0000) 2019-10-13T11:33:20.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:20.905+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.909738+0000) 2019-10-13T11:33:20.911 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:20.907+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.909998+0000) 2019-10-13T11:33:20.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:20.907+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.910909+0000) 2019-10-13T11:33:20.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:20.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:20.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:20.909+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:20.913356+0000) 2019-10-13T11:33:21.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:21.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:21.279241+0000) 2019-10-13T11:33:21.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:21.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:21.300436+0000) 2019-10-13T11:33:21.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:21.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:21.823154+0000) 2019-10-13T11:33:22.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:22.793+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:22.797564+0000) 2019-10-13T11:33:24.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:24.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:24.144450+0000) 2019-10-13T11:33:24.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:24.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:24.431249+0000) 2019-10-13T11:33:24.587 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:33:24.587 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:24.590 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:33:24.591 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:25.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:25.911+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.916395+0000) 2019-10-13T11:33:25.918 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:25.915+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.918007+0000) 2019-10-13T11:33:25.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:25.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:25.923 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:25.919+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.922636+0000) 2019-10-13T11:33:25.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:25.919+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.923087+0000) 2019-10-13T11:33:25.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:25.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.924214+0000) 2019-10-13T11:33:25.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:25.921+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.924514+0000) 2019-10-13T11:33:25.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:25.919+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.924512+0000) 2019-10-13T11:33:25.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:25.925+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:25.928791+0000) 2019-10-13T11:33:27.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:27.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:27.802953+0000) 2019-10-13T11:33:28.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:28.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:28.571938+0000) 2019-10-13T11:33:30.927 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:30.923+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.926590+0000) 2019-10-13T11:33:30.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:30.921+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.926844+0000) 2019-10-13T11:33:30.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:30.923+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.927615+0000) 2019-10-13T11:33:30.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:30.923+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.927705+0000) 2019-10-13T11:33:30.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:30.923+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.927926+0000) 2019-10-13T11:33:30.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:30.923+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.928438+0000) 2019-10-13T11:33:30.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:30.923+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.928776+0000) 2019-10-13T11:33:30.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:30.927+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:30.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:30.933+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:30.936614+0000) 2019-10-13T11:33:31.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:31.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:31.279419+0000) 2019-10-13T11:33:31.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:31.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:31.300630+0000) 2019-10-13T11:33:31.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:31.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:31.823344+0000) 2019-10-13T11:33:32.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:32.801+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:32.804150+0000) 2019-10-13T11:33:34.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:34.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:34.144684+0000) 2019-10-13T11:33:34.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:34.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:34.431480+0000) 2019-10-13T11:33:35.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:35.931+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.935570+0000) 2019-10-13T11:33:35.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:35.929+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.935924+0000) 2019-10-13T11:33:35.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:35.931+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.937557+0000) 2019-10-13T11:33:35.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:35.935+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.937729+0000) 2019-10-13T11:33:35.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:35.935+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.938456+0000) 2019-10-13T11:33:35.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:35.935+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.938959+0000) 2019-10-13T11:33:35.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:35.935+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.939270+0000) 2019-10-13T11:33:35.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:35.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:35.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:35.941+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:35.944265+0000) 2019-10-13T11:33:37.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:37.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:37.803149+0000) 2019-10-13T11:33:38.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:38.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:38.572131+0000) 2019-10-13T11:33:40.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:40.943+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.947462+0000) 2019-10-13T11:33:40.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:40.940+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.947745+0000) 2019-10-13T11:33:40.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:40.943+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.948336+0000) 2019-10-13T11:33:40.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:40.943+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.948630+0000) 2019-10-13T11:33:40.950 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:40.943+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.949357+0000) 2019-10-13T11:33:40.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:40.947+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.950313+0000) 2019-10-13T11:33:40.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:40.947+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.950867+0000) 2019-10-13T11:33:40.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:40.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:40.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:40.952+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:40.956540+0000) 2019-10-13T11:33:41.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:41.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:41.279596+0000) 2019-10-13T11:33:41.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:41.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:41.300810+0000) 2019-10-13T11:33:41.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:41.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:41.823522+0000) 2019-10-13T11:33:42.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:42.808+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:42.810635+0000) 2019-10-13T11:33:44.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:44.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:44.144887+0000) 2019-10-13T11:33:44.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:44.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:44.431700+0000) 2019-10-13T11:33:45.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:45.955+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.958318+0000) 2019-10-13T11:33:45.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:45.952+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.958615+0000) 2019-10-13T11:33:45.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:45.955+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.959042+0000) 2019-10-13T11:33:45.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:45.955+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.959550+0000) 2019-10-13T11:33:45.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:45.955+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.961518+0000) 2019-10-13T11:33:45.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:45.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:45.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:45.959+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.963660+0000) 2019-10-13T11:33:45.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:45.959+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.964082+0000) 2019-10-13T11:33:45.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:45.960+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:45.965667+0000) 2019-10-13T11:33:47.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:47.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:47.803343+0000) 2019-10-13T11:33:48.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:48.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:48.572317+0000) 2019-10-13T11:33:50.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:50.963+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.967624+0000) 2019-10-13T11:33:50.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:50.963+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.967838+0000) 2019-10-13T11:33:50.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:50.963+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.968361+0000) 2019-10-13T11:33:50.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:50.963+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.968662+0000) 2019-10-13T11:33:50.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:50.963+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.969293+0000) 2019-10-13T11:33:50.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:50.964+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.970118+0000) 2019-10-13T11:33:50.971 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:50.967+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.970604+0000) 2019-10-13T11:33:50.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:50.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141421 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:50.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:50.968+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:50.975005+0000) 2019-10-13T11:33:51.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:51.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:51.279771+0000) 2019-10-13T11:33:51.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:51.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:51.301019+0000) 2019-10-13T11:33:51.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:51.816+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:51.823701+0000) 2019-10-13T11:33:52.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:52.812+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:52.817016+0000) 2019-10-13T11:33:54.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:54.139+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:54.145080+0000) 2019-10-13T11:33:54.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:54.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:54.431902+0000) 2019-10-13T11:33:54.711 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:33:54.711 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:54.715 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:33:54.715 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:33:55.978 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:55.971+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.977208+0000) 2019-10-13T11:33:55.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:33:55.975+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.977875+0000) 2019-10-13T11:33:55.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:33:55.975+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.978478+0000) 2019-10-13T11:33:55.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:33:55.975+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.979168+0000) 2019-10-13T11:33:55.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:55.975+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.979563+0000) 2019-10-13T11:33:55.981 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:33:55.975+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.980474+0000) 2019-10-13T11:33:55.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:33:55.976+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.981333+0000) 2019-10-13T11:33:55.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:33:55.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:33:55.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:33:55.980+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:55.987137+0000) 2019-10-13T11:33:57.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:33:57.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:57.803559+0000) 2019-10-13T11:33:58.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:33:58.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:33:58.572509+0000) 2019-10-13T11:34:00.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:00.983+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.987261+0000) 2019-10-13T11:34:00.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:00.983+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.988544+0000) 2019-10-13T11:34:00.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:00.983+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.989294+0000) 2019-10-13T11:34:00.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:00.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.989524+0000) 2019-10-13T11:34:00.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:00.987+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.991024+0000) 2019-10-13T11:34:00.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:00.984+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.991836+0000) 2019-10-13T11:34:00.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:00.987+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.992974+0000) 2019-10-13T11:34:00.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:00.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:00.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:00.988+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:00.995116+0000) 2019-10-13T11:34:01.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:01.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:01.279981+0000) 2019-10-13T11:34:01.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:01.295+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:01.301195+0000) 2019-10-13T11:34:01.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:01.820+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:01.823879+0000) 2019-10-13T11:34:02.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:02.816+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:02.823372+0000) 2019-10-13T11:34:04.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:04.143+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:04.145277+0000) 2019-10-13T11:34:04.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:04.427+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:04.432088+0000) 2019-10-13T11:34:05.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:05.995+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.998029+0000) 2019-10-13T11:34:06.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:05.995+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:05.999289+0000) 2019-10-13T11:34:06.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:05.996+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.001868+0000) 2019-10-13T11:34:06.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:05.999+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.003994+0000) 2019-10-13T11:34:06.005 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:05.999+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141514 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:06.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:05.999+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.004978+0000) 2019-10-13T11:34:06.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:06.000+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.004998+0000) 2019-10-13T11:34:06.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:05.999+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.005401+0000) 2019-10-13T11:34:06.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:05.999+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:06.005679+0000) 2019-10-13T11:34:07.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:07.799+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:07.803753+0000) 2019-10-13T11:34:08.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:08.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:08.572717+0000) 2019-10-13T11:34:11.008 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:11.003+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.007566+0000) 2019-10-13T11:34:11.009 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:11.003+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.008203+0000) 2019-10-13T11:34:11.009 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:11.003+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.009058+0000) 2019-10-13T11:34:11.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:11.007+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.009692+0000) 2019-10-13T11:34:11.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:11.007+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.009934+0000) 2019-10-13T11:34:11.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:11.007+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.011001+0000) 2019-10-13T11:34:11.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:11.004+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.010617+0000) 2019-10-13T11:34:11.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:11.008+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.014778+0000) 2019-10-13T11:34:11.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:11.011+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:11.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:11.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.280195+0000) 2019-10-13T11:34:11.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:11.299+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.301381+0000) 2019-10-13T11:34:11.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:11.820+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:11.824056+0000) 2019-10-13T11:34:12.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:12.828+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:12.829956+0000) 2019-10-13T11:34:14.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:14.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:14.145471+0000) 2019-10-13T11:34:14.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:14.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:14.432323+0000) 2019-10-13T11:34:16.021 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:16.014+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.020197+0000) 2019-10-13T11:34:16.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:16.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.020588+0000) 2019-10-13T11:34:16.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:16.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.021567+0000) 2019-10-13T11:34:16.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:16.016+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.023623+0000) 2019-10-13T11:34:16.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:16.018+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.023976+0000) 2019-10-13T11:34:16.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:16.018+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.024308+0000) 2019-10-13T11:34:16.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:16.018+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.024685+0000) 2019-10-13T11:34:16.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:16.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:16.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:16.024+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:16.029173+0000) 2019-10-13T11:34:17.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:17.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:17.803985+0000) 2019-10-13T11:34:18.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:18.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:18.572916+0000) 2019-10-13T11:34:21.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:21.022+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.028456+0000) 2019-10-13T11:34:21.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:21.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.030508+0000) 2019-10-13T11:34:21.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:21.026+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.030928+0000) 2019-10-13T11:34:21.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:21.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.031721+0000) 2019-10-13T11:34:21.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:21.024+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.031985+0000) 2019-10-13T11:34:21.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:21.026+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.032413+0000) 2019-10-13T11:34:21.034 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:21.026+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.033412+0000) 2019-10-13T11:34:21.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:21.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:21.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:21.032+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.037426+0000) 2019-10-13T11:34:21.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:21.278+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.280374+0000) 2019-10-13T11:34:21.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:21.294+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.301587+0000) 2019-10-13T11:34:21.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:21.820+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:21.824256+0000) 2019-10-13T11:34:22.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:22.832+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:22.836365+0000) 2019-10-13T11:34:24.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:24.138+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:24.145684+0000) 2019-10-13T11:34:24.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:24.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:24.432522+0000) 2019-10-13T11:34:24.824 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:34:24.824 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:24.828 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:34:24.828 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:26.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:26.034+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.039160+0000) 2019-10-13T11:34:26.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:26.034+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.039912+0000) 2019-10-13T11:34:26.042 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:26.034+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.041449+0000) 2019-10-13T11:34:26.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:26.034+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.041690+0000) 2019-10-13T11:34:26.044 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:26.036+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.043450+0000) 2019-10-13T11:34:26.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:26.038+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.045242+0000) 2019-10-13T11:34:26.046 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:26.042+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.045820+0000) 2019-10-13T11:34:26.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:26.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141636 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:26.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:26.048+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:26.054972+0000) 2019-10-13T11:34:27.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:27.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:27.804188+0000) 2019-10-13T11:34:28.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:28.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:28.573099+0000) 2019-10-13T11:34:31.052 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:31.046+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.051137+0000) 2019-10-13T11:34:31.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:31.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.051540+0000) 2019-10-13T11:34:31.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:31.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.052129+0000) 2019-10-13T11:34:31.057 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:31.048+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.054891+0000) 2019-10-13T11:34:31.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:31.046+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.053052+0000) 2019-10-13T11:34:31.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:31.046+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.053678+0000) 2019-10-13T11:34:31.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:31.050+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.055716+0000) 2019-10-13T11:34:31.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:31.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:31.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:31.056+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.060677+0000) 2019-10-13T11:34:31.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:31.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.280553+0000) 2019-10-13T11:34:31.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:31.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.301789+0000) 2019-10-13T11:34:31.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:31.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:31.824427+0000) 2019-10-13T11:34:32.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:32.840+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:32.842874+0000) 2019-10-13T11:34:34.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:34.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:34.145950+0000) 2019-10-13T11:34:34.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:34.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:34.432714+0000) 2019-10-13T11:34:36.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:36.056+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.060860+0000) 2019-10-13T11:34:36.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:36.058+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.062508+0000) 2019-10-13T11:34:36.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:36.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.063163+0000) 2019-10-13T11:34:36.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:36.058+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.063756+0000) 2019-10-13T11:34:36.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:36.058+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.064063+0000) 2019-10-13T11:34:36.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:36.058+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.065152+0000) 2019-10-13T11:34:36.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:36.062+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.065815+0000) 2019-10-13T11:34:36.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:36.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:36.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:36.064+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:36.071953+0000) 2019-10-13T11:34:37.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:37.798+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:37.804368+0000) 2019-10-13T11:34:38.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:38.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:38.573250+0000) 2019-10-13T11:34:41.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:41.066+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.072194+0000) 2019-10-13T11:34:41.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:41.066+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.072284+0000) 2019-10-13T11:34:41.074 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:41.066+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.072976+0000) 2019-10-13T11:34:41.074 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:41.066+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.073206+0000) 2019-10-13T11:34:41.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:41.072+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.074860+0000) 2019-10-13T11:34:41.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:41.070+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.077401+0000) 2019-10-13T11:34:41.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:41.074+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.078102+0000) 2019-10-13T11:34:41.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:41.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:41.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:41.080+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.082357+0000) 2019-10-13T11:34:41.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:41.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.280728+0000) 2019-10-13T11:34:41.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:41.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.301963+0000) 2019-10-13T11:34:41.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:41.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:41.824592+0000) 2019-10-13T11:34:42.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:42.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:42.849357+0000) 2019-10-13T11:34:44.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:44.142+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:44.146151+0000) 2019-10-13T11:34:44.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:44.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:44.432913+0000) 2019-10-13T11:34:46.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:46.074+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.080832+0000) 2019-10-13T11:34:46.084 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:46.078+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.083577+0000) 2019-10-13T11:34:46.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:46.079+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.083738+0000) 2019-10-13T11:34:46.085 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:46.078+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.084169+0000) 2019-10-13T11:34:46.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:46.078+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.085057+0000) 2019-10-13T11:34:46.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:46.078+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.085314+0000) 2019-10-13T11:34:46.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:46.082+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.086222+0000) 2019-10-13T11:34:46.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:46.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141757 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:46.092 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:46.087+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:46.091908+0000) 2019-10-13T11:34:47.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:47.802+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:47.804561+0000) 2019-10-13T11:34:48.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:48.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:48.573488+0000) 2019-10-13T11:34:51.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:51.086+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.092401+0000) 2019-10-13T11:34:51.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:51.091+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.092796+0000) 2019-10-13T11:34:51.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:51.090+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.095208+0000) 2019-10-13T11:34:51.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:51.090+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.095715+0000) 2019-10-13T11:34:51.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:51.090+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.096212+0000) 2019-10-13T11:34:51.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:51.094+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.097881+0000) 2019-10-13T11:34:51.099 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:51.094+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.098190+0000) 2019-10-13T11:34:51.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:51.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141786 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:51.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:51.103+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.106204+0000) 2019-10-13T11:34:51.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:51.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.280909+0000) 2019-10-13T11:34:51.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:51.298+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.302149+0000) 2019-10-13T11:34:51.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:51.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:51.824761+0000) 2019-10-13T11:34:52.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:52.851+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:52.855671+0000) 2019-10-13T11:34:54.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:54.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:54.146381+0000) 2019-10-13T11:34:54.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:54.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:54.433171+0000) 2019-10-13T11:34:54.973 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:34:54.973 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:54.977 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:34:54.977 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:34:56.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:34:56.099+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.103199+0000) 2019-10-13T11:34:56.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:34:56.098+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.103610+0000) 2019-10-13T11:34:56.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:56.098+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.103897+0000) 2019-10-13T11:34:56.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:56.098+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.104530+0000) 2019-10-13T11:34:56.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:34:56.098+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.104759+0000) 2019-10-13T11:34:56.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:34:56.102+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.105800+0000) 2019-10-13T11:34:56.106 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:34:56.098+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.105188+0000) 2019-10-13T11:34:56.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:34:56.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:34:56.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:34:56.111+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:56.113298+0000) 2019-10-13T11:34:57.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:34:57.802+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:57.804799+0000) 2019-10-13T11:34:58.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:34:58.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:34:58.573681+0000) 2019-10-13T11:35:01.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:01.110+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.114477+0000) 2019-10-13T11:35:01.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:01.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.115408+0000) 2019-10-13T11:35:01.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:01.110+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.115752+0000) 2019-10-13T11:35:01.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:01.110+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.116865+0000) 2019-10-13T11:35:01.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:01.110+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.117334+0000) 2019-10-13T11:35:01.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:01.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.117435+0000) 2019-10-13T11:35:01.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:01.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.118049+0000) 2019-10-13T11:35:01.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:01.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:01.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:01.119+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.121862+0000) 2019-10-13T11:35:01.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:01.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.281110+0000) 2019-10-13T11:35:01.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:01.302+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.302371+0000) 2019-10-13T11:35:01.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:01.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:01.824966+0000) 2019-10-13T11:35:02.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:02.863+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:02.867167+0000) 2019-10-13T11:35:04.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:04.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:04.146608+0000) 2019-10-13T11:35:04.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:04.426+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:04.433431+0000) 2019-10-13T11:35:06.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:06.118+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.124508+0000) 2019-10-13T11:35:06.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:06.123+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.124481+0000) 2019-10-13T11:35:06.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:06.118+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.125124+0000) 2019-10-13T11:35:06.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:06.122+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.126397+0000) 2019-10-13T11:35:06.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:06.122+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.126668+0000) 2019-10-13T11:35:06.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:06.122+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.126968+0000) 2019-10-13T11:35:06.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:06.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.129376+0000) 2019-10-13T11:35:06.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:06.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:06.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:06.135+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:06.138075+0000) 2019-10-13T11:35:07.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:07.802+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:07.805000+0000) 2019-10-13T11:35:08.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:08.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:08.573956+0000) 2019-10-13T11:35:11.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:11.129+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.135047+0000) 2019-10-13T11:35:11.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:11.129+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.136144+0000) 2019-10-13T11:35:11.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:11.129+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.136408+0000) 2019-10-13T11:35:11.138 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:11.135+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.136465+0000) 2019-10-13T11:35:11.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:11.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.136975+0000) 2019-10-13T11:35:11.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:11.129+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.137359+0000) 2019-10-13T11:35:11.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:11.133+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.139021+0000) 2019-10-13T11:35:11.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:11.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:11.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:11.143+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.146143+0000) 2019-10-13T11:35:11.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:11.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.281311+0000) 2019-10-13T11:35:11.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:11.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.302545+0000) 2019-10-13T11:35:11.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:11.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:11.825138+0000) 2019-10-13T11:35:12.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:12.875+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:12.878599+0000) 2019-10-13T11:35:14.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:14.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:14.146811+0000) 2019-10-13T11:35:14.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:14.425+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:14.433634+0000) 2019-10-13T11:35:16.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:16.141+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.146279+0000) 2019-10-13T11:35:16.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:16.141+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.146713+0000) 2019-10-13T11:35:16.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:16.141+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.148491+0000) 2019-10-13T11:35:16.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:16.141+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.149387+0000) 2019-10-13T11:35:16.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:16.145+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.150685+0000) 2019-10-13T11:35:16.152 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:16.147+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.151516+0000) 2019-10-13T11:35:16.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:16.145+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.152603+0000) 2019-10-13T11:35:16.153 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:16.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:16.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:16.155+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:16.160021+0000) 2019-10-13T11:35:17.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:17.801+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:17.805197+0000) 2019-10-13T11:35:18.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:18.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:18.574155+0000) 2019-10-13T11:35:21.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:21.149+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.156653+0000) 2019-10-13T11:35:21.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:21.149+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.157192+0000) 2019-10-13T11:35:21.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:21.153+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.157767+0000) 2019-10-13T11:35:21.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:21.153+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.159569+0000) 2019-10-13T11:35:21.160 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:21.155+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.159657+0000) 2019-10-13T11:35:21.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:21.153+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.161450+0000) 2019-10-13T11:35:21.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:21.157+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.163006+0000) 2019-10-13T11:35:21.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:21.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:21.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:21.159+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.163226+0000) 2019-10-13T11:35:21.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:21.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.281523+0000) 2019-10-13T11:35:21.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:21.297+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.302698+0000) 2019-10-13T11:35:21.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:21.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:21.825343+0000) 2019-10-13T11:35:22.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:22.883+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:22.884844+0000) 2019-10-13T11:35:24.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:24.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:24.147031+0000) 2019-10-13T11:35:24.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:24.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:24.433856+0000) 2019-10-13T11:35:25.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:35:25.105 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:25.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:35:25.109 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:26.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:26.165+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.166693+0000) 2019-10-13T11:35:26.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:26.165+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.167193+0000) 2019-10-13T11:35:26.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:26.165+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.167831+0000) 2019-10-13T11:35:26.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:26.165+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.167975+0000) 2019-10-13T11:35:26.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:26.167+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.168665+0000) 2019-10-13T11:35:26.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:26.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.169033+0000) 2019-10-13T11:35:26.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:26.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.169652+0000) 2019-10-13T11:35:26.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:26.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:26.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:26.175+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:26.178519+0000) 2019-10-13T11:35:27.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:27.801+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:27.805401+0000) 2019-10-13T11:35:28.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:28.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:28.574348+0000) 2019-10-13T11:35:31.179 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:31.175+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.178096+0000) 2019-10-13T11:35:31.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:31.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.179311+0000) 2019-10-13T11:35:31.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:31.177+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.180175+0000) 2019-10-13T11:35:31.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:31.177+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.180374+0000) 2019-10-13T11:35:31.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:31.177+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.180737+0000) 2019-10-13T11:35:31.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:31.177+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.181182+0000) 2019-10-13T11:35:31.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:31.181+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.182022+0000) 2019-10-13T11:35:31.184 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:31.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142026 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:31.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:31.183+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.186414+0000) 2019-10-13T11:35:31.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:31.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.281700+0000) 2019-10-13T11:35:31.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:31.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.302885+0000) 2019-10-13T11:35:31.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:31.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:31.825568+0000) 2019-10-13T11:35:32.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:32.887+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:32.891234+0000) 2019-10-13T11:35:34.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:34.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:34.147263+0000) 2019-10-13T11:35:34.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:34.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:34.434092+0000) 2019-10-13T11:35:36.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:36.183+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.187201+0000) 2019-10-13T11:35:36.189 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:36.185+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.187574+0000) 2019-10-13T11:35:36.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:36.185+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.188379+0000) 2019-10-13T11:35:36.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:36.189+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.190261+0000) 2019-10-13T11:35:36.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:36.189+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.190581+0000) 2019-10-13T11:35:36.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:36.189+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.191255+0000) 2019-10-13T11:35:36.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:36.193+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.193820+0000) 2019-10-13T11:35:36.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:36.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142055 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:36.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:36.199+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:36.201121+0000) 2019-10-13T11:35:37.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:37.801+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:37.805608+0000) 2019-10-13T11:35:38.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:38.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:38.574557+0000) 2019-10-13T11:35:41.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:41.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.197287+0000) 2019-10-13T11:35:41.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:41.197+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.198350+0000) 2019-10-13T11:35:41.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:41.197+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.198776+0000) 2019-10-13T11:35:41.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:41.197+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.199366+0000) 2019-10-13T11:35:41.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:41.197+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.200019+0000) 2019-10-13T11:35:41.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:41.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.200817+0000) 2019-10-13T11:35:41.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:41.197+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.201059+0000) 2019-10-13T11:35:41.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:41.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142087 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:41.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:41.203+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.207466+0000) 2019-10-13T11:35:41.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:41.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.281911+0000) 2019-10-13T11:35:41.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:41.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.303058+0000) 2019-10-13T11:35:41.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:41.823+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:41.825762+0000) 2019-10-13T11:35:42.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:42.895+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:42.897714+0000) 2019-10-13T11:35:44.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:44.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:44.147487+0000) 2019-10-13T11:35:44.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:44.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:44.434295+0000) 2019-10-13T11:35:46.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:46.205+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.208761+0000) 2019-10-13T11:35:46.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:46.205+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.208900+0000) 2019-10-13T11:35:46.210 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:46.205+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.209285+0000) 2019-10-13T11:35:46.211 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:46.205+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.209591+0000) 2019-10-13T11:35:46.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:46.209+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.211488+0000) 2019-10-13T11:35:46.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:46.209+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.211669+0000) 2019-10-13T11:35:46.213 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:46.211+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.212208+0000) 2019-10-13T11:35:46.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:46.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:46.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:46.219+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:46.223189+0000) 2019-10-13T11:35:47.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:47.805+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:47.805812+0000) 2019-10-13T11:35:48.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:48.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:48.574750+0000) 2019-10-13T11:35:51.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:51.217+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.218544+0000) 2019-10-13T11:35:51.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:51.217+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.220380+0000) 2019-10-13T11:35:51.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:51.217+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.221455+0000) 2019-10-13T11:35:51.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:51.221+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.222326+0000) 2019-10-13T11:35:51.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:51.221+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.223562+0000) 2019-10-13T11:35:51.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:51.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:51.227 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:51.222+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.226623+0000) 2019-10-13T11:35:51.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:51.225+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.227357+0000) 2019-10-13T11:35:51.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:51.222+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.227487+0000) 2019-10-13T11:35:51.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:51.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.282117+0000) 2019-10-13T11:35:51.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:51.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.303238+0000) 2019-10-13T11:35:51.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:51.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:51.825944+0000) 2019-10-13T11:35:52.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:52.906+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:52.908927+0000) 2019-10-13T11:35:54.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:54.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:54.147725+0000) 2019-10-13T11:35:54.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:54.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:54.434521+0000) 2019-10-13T11:35:55.237 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:35:55.237 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:55.241 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:35:55.241 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:35:56.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:35:56.229+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.230993+0000) 2019-10-13T11:35:56.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:35:56.229+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.231750+0000) 2019-10-13T11:35:56.234 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:56.229+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.231431+0000) 2019-10-13T11:35:56.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:35:56.229+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.232071+0000) 2019-10-13T11:35:56.235 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:35:56.230+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.232686+0000) 2019-10-13T11:35:56.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:35:56.229+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.233646+0000) 2019-10-13T11:35:56.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:56.233+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.234318+0000) 2019-10-13T11:35:56.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:35:56.234+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:56.236488+0000) 2019-10-13T11:35:56.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:35:56.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:35:57.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:35:57.805+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:57.806021+0000) 2019-10-13T11:35:58.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:35:58.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:35:58.574944+0000) 2019-10-13T11:36:01.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:01.237+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.240157+0000) 2019-10-13T11:36:01.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:01.238+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.241472+0000) 2019-10-13T11:36:01.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:01.237+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.241598+0000) 2019-10-13T11:36:01.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:01.241+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.242027+0000) 2019-10-13T11:36:01.243 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:01.241+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.242298+0000) 2019-10-13T11:36:01.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:01.241+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.242800+0000) 2019-10-13T11:36:01.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:01.241+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.243309+0000) 2019-10-13T11:36:01.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:01.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:01.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:01.246+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.249246+0000) 2019-10-13T11:36:01.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:01.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.282277+0000) 2019-10-13T11:36:01.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:01.301+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.303424+0000) 2019-10-13T11:36:01.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:01.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:01.826117+0000) 2019-10-13T11:36:02.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:02.914+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:02.915028+0000) 2019-10-13T11:36:04.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:04.145+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:04.147920+0000) 2019-10-13T11:36:04.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:04.433+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:04.434713+0000) 2019-10-13T11:36:06.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:06.248+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.250249+0000) 2019-10-13T11:36:06.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:06.248+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.250812+0000) 2019-10-13T11:36:06.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:06.248+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.251454+0000) 2019-10-13T11:36:06.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:06.248+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.251920+0000) 2019-10-13T11:36:06.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:06.248+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.252685+0000) 2019-10-13T11:36:06.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:06.250+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.252848+0000) 2019-10-13T11:36:06.254 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:06.248+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.253317+0000) 2019-10-13T11:36:06.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:06.252+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:06.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:06.258+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:06.261242+0000) 2019-10-13T11:36:07.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:07.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:07.806222+0000) 2019-10-13T11:36:08.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:08.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:08.575137+0000) 2019-10-13T11:36:11.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:11.256+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.260767+0000) 2019-10-13T11:36:11.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:11.256+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.261021+0000) 2019-10-13T11:36:11.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:11.260+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.261820+0000) 2019-10-13T11:36:11.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:11.260+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.262193+0000) 2019-10-13T11:36:11.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:11.260+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.263203+0000) 2019-10-13T11:36:11.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:11.258+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.263121+0000) 2019-10-13T11:36:11.266 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:11.260+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.265327+0000) 2019-10-13T11:36:11.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:11.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:11.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:11.270+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.272185+0000) 2019-10-13T11:36:11.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:11.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.282436+0000) 2019-10-13T11:36:11.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:11.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.303601+0000) 2019-10-13T11:36:11.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:11.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:11.826282+0000) 2019-10-13T11:36:12.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:12.922+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:12.925871+0000) 2019-10-13T11:36:14.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:14.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:14.148088+0000) 2019-10-13T11:36:14.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:14.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:14.434928+0000) 2019-10-13T11:36:16.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:16.268+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.271326+0000) 2019-10-13T11:36:16.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:16.270+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.272077+0000) 2019-10-13T11:36:16.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:16.268+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.271719+0000) 2019-10-13T11:36:16.275 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:16.272+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.274585+0000) 2019-10-13T11:36:16.277 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:16.272+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.276790+0000) 2019-10-13T11:36:16.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:16.276+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.278425+0000) 2019-10-13T11:36:16.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:16.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:16.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:16.276+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.279369+0000) 2019-10-13T11:36:16.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:16.282+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:16.285551+0000) 2019-10-13T11:36:17.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:17.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:17.806453+0000) 2019-10-13T11:36:18.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:18.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:18.575332+0000) 2019-10-13T11:36:21.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:21.280+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.283195+0000) 2019-10-13T11:36:21.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:21.280+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.283644+0000) 2019-10-13T11:36:21.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:21.280+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.283992+0000) 2019-10-13T11:36:21.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:21.280+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.284814+0000) 2019-10-13T11:36:21.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:21.282+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.287894+0000) 2019-10-13T11:36:21.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:21.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:21.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:21.288+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.290575+0000) 2019-10-13T11:36:21.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:21.288+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.290809+0000) 2019-10-13T11:36:21.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:21.290+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.292635+0000) 2019-10-13T11:36:21.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:21.300+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.303795+0000) 2019-10-13T11:36:21.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:21.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:21.826472+0000) 2019-10-13T11:36:22.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:22.934+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:22.937008+0000) 2019-10-13T11:36:24.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:24.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:24.148296+0000) 2019-10-13T11:36:24.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:24.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:24.435137+0000) 2019-10-13T11:36:25.452 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:36:25.452 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:25.456 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:36:25.456 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:26.296 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:26.292+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.295187+0000) 2019-10-13T11:36:26.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:26.292+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.295499+0000) 2019-10-13T11:36:26.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:26.292+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.296750+0000) 2019-10-13T11:36:26.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:26.294+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.297191+0000) 2019-10-13T11:36:26.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:26.292+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.297541+0000) 2019-10-13T11:36:26.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:26.296+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.298361+0000) 2019-10-13T11:36:26.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:26.296+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.299071+0000) 2019-10-13T11:36:26.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:26.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:26.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:26.302+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:26.304640+0000) 2019-10-13T11:36:27.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:27.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:27.806653+0000) 2019-10-13T11:36:28.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:28.028+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.033082+0000) 2019-10-13T11:36:28.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:28.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:28.575541+0000) 2019-10-13T11:36:31.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:31.300+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.303747+0000) 2019-10-13T11:36:31.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:31.300+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.304424+0000) 2019-10-13T11:36:31.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:31.300+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.305047+0000) 2019-10-13T11:36:31.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:31.300+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.305286+0000) 2019-10-13T11:36:31.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:31.304+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.306005+0000) 2019-10-13T11:36:31.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:31.304+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.306866+0000) 2019-10-13T11:36:31.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:31.302+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.305570+0000) 2019-10-13T11:36:31.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:31.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:31.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:31.310+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.313046+0000) 2019-10-13T11:36:31.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:31.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:31.826659+0000) 2019-10-13T11:36:32.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:32.942+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:32.943299+0000) 2019-10-13T11:36:34.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:34.144+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:34.148521+0000) 2019-10-13T11:36:34.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:34.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:34.435372+0000) 2019-10-13T11:36:36.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:36.310+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.314035+0000) 2019-10-13T11:36:36.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:36.312+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.314865+0000) 2019-10-13T11:36:36.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:36.312+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.317025+0000) 2019-10-13T11:36:36.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:36.312+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.317277+0000) 2019-10-13T11:36:36.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:36.316+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.319540+0000) 2019-10-13T11:36:36.322 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:36.318+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.320943+0000) 2019-10-13T11:36:36.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:36.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:36.322 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:36.320+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.321766+0000) 2019-10-13T11:36:36.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:36.316+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:36.321670+0000) 2019-10-13T11:36:37.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:37.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:37.806851+0000) 2019-10-13T11:36:38.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:38.028+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.033290+0000) 2019-10-13T11:36:38.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:38.052+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.054513+0000) 2019-10-13T11:36:38.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:38.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:38.575730+0000) 2019-10-13T11:36:41.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:41.320+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.324947+0000) 2019-10-13T11:36:41.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:41.320+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.325497+0000) 2019-10-13T11:36:41.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:41.320+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.325591+0000) 2019-10-13T11:36:41.327 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:41.322+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.326558+0000) 2019-10-13T11:36:41.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:41.324+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.326729+0000) 2019-10-13T11:36:41.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:41.324+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.327067+0000) 2019-10-13T11:36:41.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:41.324+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.327537+0000) 2019-10-13T11:36:41.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:41.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:41.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:41.326+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.330293+0000) 2019-10-13T11:36:41.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:41.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:41.826825+0000) 2019-10-13T11:36:42.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:42.946+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:42.949823+0000) 2019-10-13T11:36:44.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:44.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:44.148738+0000) 2019-10-13T11:36:44.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:44.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:44.435572+0000) 2019-10-13T11:36:46.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:46.330+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.334963+0000) 2019-10-13T11:36:46.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:46.332+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.335716+0000) 2019-10-13T11:36:46.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:46.332+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.336018+0000) 2019-10-13T11:36:46.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:46.332+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.336522+0000) 2019-10-13T11:36:46.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:46.336+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.339067+0000) 2019-10-13T11:36:46.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:46.336+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.341102+0000) 2019-10-13T11:36:46.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:46.336+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:46.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:46.338+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.341302+0000) 2019-10-13T11:36:46.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:46.340+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:46.341962+0000) 2019-10-13T11:36:47.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:47.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:47.807009+0000) 2019-10-13T11:36:48.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:48.032+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.033495+0000) 2019-10-13T11:36:48.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:48.052+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.054704+0000) 2019-10-13T11:36:48.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:48.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:48.575925+0000) 2019-10-13T11:36:51.346 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:51.340+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.345361+0000) 2019-10-13T11:36:51.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:51.344+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.346098+0000) 2019-10-13T11:36:51.347 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:51.344+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.346505+0000) 2019-10-13T11:36:51.349 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:51.342+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.346898+0000) 2019-10-13T11:36:51.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:51.344+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.347268+0000) 2019-10-13T11:36:51.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:51.344+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.347567+0000) 2019-10-13T11:36:51.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:51.344+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.348442+0000) 2019-10-13T11:36:51.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:51.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:51.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:51.354+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.356862+0000) 2019-10-13T11:36:51.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:51.822+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:51.826991+0000) 2019-10-13T11:36:52.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:52.953+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:52.955632+0000) 2019-10-13T11:36:54.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:54.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:54.148939+0000) 2019-10-13T11:36:54.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:54.432+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:54.435790+0000) 2019-10-13T11:36:55.582 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:36:55.582 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:55.586 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:36:55.586 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:36:56.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:36:56.352+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.355219+0000) 2019-10-13T11:36:56.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:36:56.349+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.355469+0000) 2019-10-13T11:36:56.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:56.352+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.356224+0000) 2019-10-13T11:36:56.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:56.352+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.356701+0000) 2019-10-13T11:36:56.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:56.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.357882+0000) 2019-10-13T11:36:56.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:56.352+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.357623+0000) 2019-10-13T11:36:56.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:36:56.356+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.358277+0000) 2019-10-13T11:36:56.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:36:56.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:36:56.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:36:56.357+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:56.362197+0000) 2019-10-13T11:36:57.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:36:57.804+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:57.807218+0000) 2019-10-13T11:36:58.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:36:58.032+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.033694+0000) 2019-10-13T11:36:58.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:36:58.052+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.054913+0000) 2019-10-13T11:36:58.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:36:58.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:36:58.576113+0000) 2019-10-13T11:37:01.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:01.359+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.365034+0000) 2019-10-13T11:37:01.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:01.363+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.366071+0000) 2019-10-13T11:37:01.367 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:01.363+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.366835+0000) 2019-10-13T11:37:01.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:01.361+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.367083+0000) 2019-10-13T11:37:01.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:01.363+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.367418+0000) 2019-10-13T11:37:01.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:01.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.367676+0000) 2019-10-13T11:37:01.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:01.363+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.368024+0000) 2019-10-13T11:37:01.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:01.367+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:01.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:01.369+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.373607+0000) 2019-10-13T11:37:01.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:01.825+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:01.827167+0000) 2019-10-13T11:37:02.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:02.961+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:02.962071+0000) 2019-10-13T11:37:04.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:04.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:04.149138+0000) 2019-10-13T11:37:04.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:04.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:04.435954+0000) 2019-10-13T11:37:06.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:06.371+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.376760+0000) 2019-10-13T11:37:06.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:06.371+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.377476+0000) 2019-10-13T11:37:06.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:06.375+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.379820+0000) 2019-10-13T11:37:06.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:06.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.382496+0000) 2019-10-13T11:37:06.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:06.379+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:06.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:06.377+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.383478+0000) 2019-10-13T11:37:06.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:06.379+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.384468+0000) 2019-10-13T11:37:06.386 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:06.379+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.385218+0000) 2019-10-13T11:37:06.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:06.389+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:06.392436+0000) 2019-10-13T11:37:07.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:07.803+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:07.807420+0000) 2019-10-13T11:37:08.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:08.031+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.033933+0000) 2019-10-13T11:37:08.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:08.051+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.055093+0000) 2019-10-13T11:37:08.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:08.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:08.576317+0000) 2019-10-13T11:37:11.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:11.383+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.386386+0000) 2019-10-13T11:37:11.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:11.383+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.387962+0000) 2019-10-13T11:37:11.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:11.383+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.388080+0000) 2019-10-13T11:37:11.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:11.383+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.388544+0000) 2019-10-13T11:37:11.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:11.383+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.388977+0000) 2019-10-13T11:37:11.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:11.383+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.389331+0000) 2019-10-13T11:37:11.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:11.385+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.389809+0000) 2019-10-13T11:37:11.394 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:11.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:11.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:11.393+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.398338+0000) 2019-10-13T11:37:11.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:11.821+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:11.827332+0000) 2019-10-13T11:37:12.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:12.965+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:12.968589+0000) 2019-10-13T11:37:14.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:14.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:14.149335+0000) 2019-10-13T11:37:14.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:14.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:14.436176+0000) 2019-10-13T11:37:16.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:16.391+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.397113+0000) 2019-10-13T11:37:16.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:16.391+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.397319+0000) 2019-10-13T11:37:16.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:16.395+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.397771+0000) 2019-10-13T11:37:16.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:16.395+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.399754+0000) 2019-10-13T11:37:16.401 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:16.397+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.400137+0000) 2019-10-13T11:37:16.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:16.395+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.400985+0000) 2019-10-13T11:37:16.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:16.395+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.401258+0000) 2019-10-13T11:37:16.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:16.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:16.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:16.401+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:16.407909+0000) 2019-10-13T11:37:17.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:17.803+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:17.807644+0000) 2019-10-13T11:37:18.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:18.031+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.034132+0000) 2019-10-13T11:37:18.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:18.051+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.055291+0000) 2019-10-13T11:37:18.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:18.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:18.576473+0000) 2019-10-13T11:37:21.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:21.403+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.408156+0000) 2019-10-13T11:37:21.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:21.403+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.408481+0000) 2019-10-13T11:37:21.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:21.403+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.409004+0000) 2019-10-13T11:37:21.410 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:21.403+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.409306+0000) 2019-10-13T11:37:21.410 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:21.405+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.409449+0000) 2019-10-13T11:37:21.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:21.407+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.410977+0000) 2019-10-13T11:37:21.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:21.407+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.411321+0000) 2019-10-13T11:37:21.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:21.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:21.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:21.413+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.417034+0000) 2019-10-13T11:37:21.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:21.825+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:21.827544+0000) 2019-10-13T11:37:22.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:22.973+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:22.975109+0000) 2019-10-13T11:37:24.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:24.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:24.149539+0000) 2019-10-13T11:37:24.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:24.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:24.436407+0000) 2019-10-13T11:37:25.940 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:37:25.940 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:25.944 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:37:25.945 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:26.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:26.415+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.419097+0000) 2019-10-13T11:37:26.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:26.415+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.419882+0000) 2019-10-13T11:37:26.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:26.415+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.421314+0000) 2019-10-13T11:37:26.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:26.417+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.421690+0000) 2019-10-13T11:37:26.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:26.419+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.422639+0000) 2019-10-13T11:37:26.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:26.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.422949+0000) 2019-10-13T11:37:26.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:26.419+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.423280+0000) 2019-10-13T11:37:26.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:26.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:26.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:26.421+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:26.426796+0000) 2019-10-13T11:37:27.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:27.803+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:27.807803+0000) 2019-10-13T11:37:28.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:28.031+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.034332+0000) 2019-10-13T11:37:28.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:28.051+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.055487+0000) 2019-10-13T11:37:28.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:28.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:28.576705+0000) 2019-10-13T11:37:31.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:31.427+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.430934+0000) 2019-10-13T11:37:31.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:31.425+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.431203+0000) 2019-10-13T11:37:31.432 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:31.427+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.431818+0000) 2019-10-13T11:37:31.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:31.427+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.432220+0000) 2019-10-13T11:37:31.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:31.431+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.434199+0000) 2019-10-13T11:37:31.435 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:31.431+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.435018+0000) 2019-10-13T11:37:31.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:31.431+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.435481+0000) 2019-10-13T11:37:31.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:31.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:31.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:31.437+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.440905+0000) 2019-10-13T11:37:31.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:31.825+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:31.827756+0000) 2019-10-13T11:37:32.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:32.985+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:32.986645+0000) 2019-10-13T11:37:34.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:34.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:34.149766+0000) 2019-10-13T11:37:34.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:34.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:34.436634+0000) 2019-10-13T11:37:36.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:36.435+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.439213+0000) 2019-10-13T11:37:36.441 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:36.437+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.440444+0000) 2019-10-13T11:37:36.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:36.435+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.441119+0000) 2019-10-13T11:37:36.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:36.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.440768+0000) 2019-10-13T11:37:36.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:36.435+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.440970+0000) 2019-10-13T11:37:36.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:36.439+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.441916+0000) 2019-10-13T11:37:36.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:36.439+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.442821+0000) 2019-10-13T11:37:36.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:36.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:36.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:36.445+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:36.451309+0000) 2019-10-13T11:37:37.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:37.803+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:37.808023+0000) 2019-10-13T11:37:38.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:38.031+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.034533+0000) 2019-10-13T11:37:38.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:38.055+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.055719+0000) 2019-10-13T11:37:38.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:38.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:38.576916+0000) 2019-10-13T11:37:41.451 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:41.447+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.450399+0000) 2019-10-13T11:37:41.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:41.447+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.452264+0000) 2019-10-13T11:37:41.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:41.449+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.452402+0000) 2019-10-13T11:37:41.453 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:41.447+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.453230+0000) 2019-10-13T11:37:41.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:41.451+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.454372+0000) 2019-10-13T11:37:41.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:41.451+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.455561+0000) 2019-10-13T11:37:41.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:41.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:41.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:41.455+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.457808+0000) 2019-10-13T11:37:41.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:41.461+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.464439+0000) 2019-10-13T11:37:41.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:41.825+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:41.827950+0000) 2019-10-13T11:37:42.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:42.993+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:42.993288+0000) 2019-10-13T11:37:44.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:44.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:44.149971+0000) 2019-10-13T11:37:44.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:44.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:44.436834+0000) 2019-10-13T11:37:46.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:46.457+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.462955+0000) 2019-10-13T11:37:46.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:46.455+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.460974+0000) 2019-10-13T11:37:46.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:46.455+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.461551+0000) 2019-10-13T11:37:46.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:46.459+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.462532+0000) 2019-10-13T11:37:46.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:46.459+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.462412+0000) 2019-10-13T11:37:46.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:46.459+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.463576+0000) 2019-10-13T11:37:46.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:46.459+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.463813+0000) 2019-10-13T11:37:46.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:46.465+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:46.469781+0000) 2019-10-13T11:37:46.473 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:46.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:47.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:47.803+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:47.808220+0000) 2019-10-13T11:37:48.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:48.031+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.034731+0000) 2019-10-13T11:37:48.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:48.051+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.055950+0000) 2019-10-13T11:37:48.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:48.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:48.577060+0000) 2019-10-13T11:37:51.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:51.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.471328+0000) 2019-10-13T11:37:51.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:51.467+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.471985+0000) 2019-10-13T11:37:51.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:51.467+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.472626+0000) 2019-10-13T11:37:51.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:51.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.473953+0000) 2019-10-13T11:37:51.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:51.471+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.476403+0000) 2019-10-13T11:37:51.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:51.475+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.478443+0000) 2019-10-13T11:37:51.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:51.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:51.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:51.473+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.479757+0000) 2019-10-13T11:37:51.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:51.477+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.481546+0000) 2019-10-13T11:37:51.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:51.825+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:51.828123+0000) 2019-10-13T11:37:53.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:52.997+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:52.999361+0000) 2019-10-13T11:37:54.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:54.147+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:54.150186+0000) 2019-10-13T11:37:54.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:54.431+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:54.437069+0000) 2019-10-13T11:37:56.062 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:37:56.062 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:56.066 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:37:56.066 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:37:56.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:56.479+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.484201+0000) 2019-10-13T11:37:56.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:37:56.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.486243+0000) 2019-10-13T11:37:56.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:37:56.483+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.486875+0000) 2019-10-13T11:37:56.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:56.483+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.486628+0000) 2019-10-13T11:37:56.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:56.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.487881+0000) 2019-10-13T11:37:56.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:56.483+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.488706+0000) 2019-10-13T11:37:56.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:37:56.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:37:56.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:37:56.485+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.489144+0000) 2019-10-13T11:37:56.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:37:56.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:56.494426+0000) 2019-10-13T11:37:57.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:37:57.806+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:57.808451+0000) 2019-10-13T11:37:58.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:37:58.030+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.034942+0000) 2019-10-13T11:37:58.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:37:58.054+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.056178+0000) 2019-10-13T11:37:58.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:37:58.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:37:58.577253+0000) 2019-10-13T11:38:01.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:01.486+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.493307+0000) 2019-10-13T11:38:01.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:01.490+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.494160+0000) 2019-10-13T11:38:01.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:01.490+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.494877+0000) 2019-10-13T11:38:01.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:01.490+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.495517+0000) 2019-10-13T11:38:01.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:01.490+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.496096+0000) 2019-10-13T11:38:01.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:01.490+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.496382+0000) 2019-10-13T11:38:01.497 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:01.492+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.496910+0000) 2019-10-13T11:38:01.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:01.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:01.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:01.496+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.503428+0000) 2019-10-13T11:38:01.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:01.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:01.828302+0000) 2019-10-13T11:38:03.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:03.000+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:03.006022+0000) 2019-10-13T11:38:04.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:04.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:04.150443+0000) 2019-10-13T11:38:04.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:04.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:04.437272+0000) 2019-10-13T11:38:06.512 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:06.498+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.503315+0000) 2019-10-13T11:38:06.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:06.498+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.504164+0000) 2019-10-13T11:38:06.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:06.498+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.504035+0000) 2019-10-13T11:38:06.514 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:06.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.504935+0000) 2019-10-13T11:38:06.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:06.500+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.505440+0000) 2019-10-13T11:38:06.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:06.502+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.505791+0000) 2019-10-13T11:38:06.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:06.502+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.506562+0000) 2019-10-13T11:38:06.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:06.508+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:06.512429+0000) 2019-10-13T11:38:06.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:06.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:07.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:07.802+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:07.808654+0000) 2019-10-13T11:38:08.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:08.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.035145+0000) 2019-10-13T11:38:08.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:08.050+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.056380+0000) 2019-10-13T11:38:08.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:08.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:08.577475+0000) 2019-10-13T11:38:11.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:11.510+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.514245+0000) 2019-10-13T11:38:11.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:11.510+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.516311+0000) 2019-10-13T11:38:11.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:11.510+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.517705+0000) 2019-10-13T11:38:11.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:11.514+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.518064+0000) 2019-10-13T11:38:11.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:11.512+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.518390+0000) 2019-10-13T11:38:11.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:11.514+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.518909+0000) 2019-10-13T11:38:11.520 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:11.514+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.519311+0000) 2019-10-13T11:38:11.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:11.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:11.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:11.516+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.523350+0000) 2019-10-13T11:38:11.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:11.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:11.828475+0000) 2019-10-13T11:38:13.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:13.008+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:13.012504+0000) 2019-10-13T11:38:14.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:14.146+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:14.150674+0000) 2019-10-13T11:38:14.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:14.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:14.437468+0000) 2019-10-13T11:38:16.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:16.518+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.524946+0000) 2019-10-13T11:38:16.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:16.520+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.525218+0000) 2019-10-13T11:38:16.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:16.522+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.526029+0000) 2019-10-13T11:38:16.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:16.522+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.526754+0000) 2019-10-13T11:38:16.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:16.522+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.527792+0000) 2019-10-13T11:38:16.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:16.522+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.527608+0000) 2019-10-13T11:38:16.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:16.522+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.528430+0000) 2019-10-13T11:38:16.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:16.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:16.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:16.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:16.535725+0000) 2019-10-13T11:38:17.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:17.802+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:17.808880+0000) 2019-10-13T11:38:18.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:18.030+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.035340+0000) 2019-10-13T11:38:18.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:18.050+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.056579+0000) 2019-10-13T11:38:18.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:18.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:18.577697+0000) 2019-10-13T11:38:21.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:21.528+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.535631+0000) 2019-10-13T11:38:21.537 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:21.530+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.535998+0000) 2019-10-13T11:38:21.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:21.530+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.537226+0000) 2019-10-13T11:38:21.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:21.530+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.537709+0000) 2019-10-13T11:38:21.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:21.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.538071+0000) 2019-10-13T11:38:21.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:21.534+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.538762+0000) 2019-10-13T11:38:21.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:21.534+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.539205+0000) 2019-10-13T11:38:21.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:21.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:21.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:21.536+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.542982+0000) 2019-10-13T11:38:21.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:21.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:21.828649+0000) 2019-10-13T11:38:23.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:23.016+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:23.019179+0000) 2019-10-13T11:38:24.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:24.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:24.150865+0000) 2019-10-13T11:38:24.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:24.430+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:24.437674+0000) 2019-10-13T11:38:26.184 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:38:26.184 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:26.187 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:38:26.187 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:26.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:26.542+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.546174+0000) 2019-10-13T11:38:26.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:26.542+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.546634+0000) 2019-10-13T11:38:26.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:26.542+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.547025+0000) 2019-10-13T11:38:26.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:26.540+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.547255+0000) 2019-10-13T11:38:26.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:26.542+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.547979+0000) 2019-10-13T11:38:26.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:26.542+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.548284+0000) 2019-10-13T11:38:26.550 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:26.542+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.549411+0000) 2019-10-13T11:38:26.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:26.546+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:26.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:26.552+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:26.556254+0000) 2019-10-13T11:38:27.810 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:27.806+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:27.809095+0000) 2019-10-13T11:38:28.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:28.030+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.035543+0000) 2019-10-13T11:38:28.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:28.054+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.056774+0000) 2019-10-13T11:38:28.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:28.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:28.577911+0000) 2019-10-13T11:38:31.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:31.550+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.556312+0000) 2019-10-13T11:38:31.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:31.554+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.557983+0000) 2019-10-13T11:38:31.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:31.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.558213+0000) 2019-10-13T11:38:31.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:31.554+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.558817+0000) 2019-10-13T11:38:31.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:31.554+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.559002+0000) 2019-10-13T11:38:31.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:31.556+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.560145+0000) 2019-10-13T11:38:31.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:31.554+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.560515+0000) 2019-10-13T11:38:31.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:31.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:31.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:31.560+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.565218+0000) 2019-10-13T11:38:31.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:31.828+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:31.828851+0000) 2019-10-13T11:38:33.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:33.024+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:33.025803+0000) 2019-10-13T11:38:34.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:34.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:34.151085+0000) 2019-10-13T11:38:34.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:34.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:34.437909+0000) 2019-10-13T11:38:36.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:36.560+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.566810+0000) 2019-10-13T11:38:36.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:36.562+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.567551+0000) 2019-10-13T11:38:36.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:36.562+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.568146+0000) 2019-10-13T11:38:36.569 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:36.562+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.568423+0000) 2019-10-13T11:38:36.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:36.562+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.569194+0000) 2019-10-13T11:38:36.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:36.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.569907+0000) 2019-10-13T11:38:36.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:36.566+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.570562+0000) 2019-10-13T11:38:36.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:36.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:36.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:36.572+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:36.576342+0000) 2019-10-13T11:38:37.810 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:37.806+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:37.809296+0000) 2019-10-13T11:38:38.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:38.030+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.035739+0000) 2019-10-13T11:38:38.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:38.050+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.056972+0000) 2019-10-13T11:38:38.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:38.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:38.578097+0000) 2019-10-13T11:38:41.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:41.574+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.579019+0000) 2019-10-13T11:38:41.581 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:41.574+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.579932+0000) 2019-10-13T11:38:41.581 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:41.574+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.581023+0000) 2019-10-13T11:38:41.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:41.576+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.582910+0000) 2019-10-13T11:38:41.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:41.578+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.584704+0000) 2019-10-13T11:38:41.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:41.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143168 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:41.586 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:41.578+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.585654+0000) 2019-10-13T11:38:41.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:41.580+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.587412+0000) 2019-10-13T11:38:41.588 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:41.582+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.586834+0000) 2019-10-13T11:38:41.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:41.828+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:41.829024+0000) 2019-10-13T11:38:43.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:43.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:43.036829+0000) 2019-10-13T11:38:44.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:44.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:44.151283+0000) 2019-10-13T11:38:44.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:44.434+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:44.438141+0000) 2019-10-13T11:38:46.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:46.582+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.587977+0000) 2019-10-13T11:38:46.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:46.586+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.590118+0000) 2019-10-13T11:38:46.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:46.586+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.590582+0000) 2019-10-13T11:38:46.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:46.586+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.590843+0000) 2019-10-13T11:38:46.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:46.584+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.591113+0000) 2019-10-13T11:38:46.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:46.586+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.591495+0000) 2019-10-13T11:38:46.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:46.586+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.591813+0000) 2019-10-13T11:38:46.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:46.588+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:46.592163+0000) 2019-10-13T11:38:46.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:46.590+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:47.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:47.806+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:47.809526+0000) 2019-10-13T11:38:48.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:48.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.035939+0000) 2019-10-13T11:38:48.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:48.054+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.057171+0000) 2019-10-13T11:38:48.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:48.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:48.578294+0000) 2019-10-13T11:38:51.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:51.596+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.599134+0000) 2019-10-13T11:38:51.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:51.594+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.599488+0000) 2019-10-13T11:38:51.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:51.594+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.600308+0000) 2019-10-13T11:38:51.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:51.598+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.602495+0000) 2019-10-13T11:38:51.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:51.598+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.604367+0000) 2019-10-13T11:38:51.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:51.602+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:51.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:51.602+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.607637+0000) 2019-10-13T11:38:51.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:51.602+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.608144+0000) 2019-10-13T11:38:51.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:51.604+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.609362+0000) 2019-10-13T11:38:51.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:51.824+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:51.829229+0000) 2019-10-13T11:38:53.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:53.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:53.043308+0000) 2019-10-13T11:38:54.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:54.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:54.151492+0000) 2019-10-13T11:38:54.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:54.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:54.438340+0000) 2019-10-13T11:38:56.268 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:38:56.268 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:56.272 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:38:56.272 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:38:56.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:56.605+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.610429+0000) 2019-10-13T11:38:56.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:56.605+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.611272+0000) 2019-10-13T11:38:56.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:38:56.605+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.611014+0000) 2019-10-13T11:38:56.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:56.605+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.611794+0000) 2019-10-13T11:38:56.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:38:56.612+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.612645+0000) 2019-10-13T11:38:56.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:56.605+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.613584+0000) 2019-10-13T11:38:56.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:38:56.609+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.614036+0000) 2019-10-13T11:38:56.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:38:56.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143259 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:38:56.620 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:38:56.616+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:56.619728+0000) 2019-10-13T11:38:57.810 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:38:57.805+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:57.809730+0000) 2019-10-13T11:38:58.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:38:58.029+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.036134+0000) 2019-10-13T11:38:58.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:38:58.053+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.057371+0000) 2019-10-13T11:38:58.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:38:58.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:38:58.578490+0000) 2019-10-13T11:39:01.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:01.613+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.620727+0000) 2019-10-13T11:39:01.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:01.613+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.621370+0000) 2019-10-13T11:39:01.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:01.617+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.621924+0000) 2019-10-13T11:39:01.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:01.617+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.622362+0000) 2019-10-13T11:39:01.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:01.620+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.622670+0000) 2019-10-13T11:39:01.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:01.617+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.623275+0000) 2019-10-13T11:39:01.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:01.617+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.623682+0000) 2019-10-13T11:39:01.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:01.628+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.628121+0000) 2019-10-13T11:39:01.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:01.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:01.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:01.828+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:01.829402+0000) 2019-10-13T11:39:03.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:03.047+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:03.049849+0000) 2019-10-13T11:39:04.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:04.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:04.151765+0000) 2019-10-13T11:39:04.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:04.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:04.438537+0000) 2019-10-13T11:39:06.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:06.629+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.634099+0000) 2019-10-13T11:39:06.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:06.629+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.634773+0000) 2019-10-13T11:39:06.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:06.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.635183+0000) 2019-10-13T11:39:06.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:06.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.636149+0000) 2019-10-13T11:39:06.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:06.633+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.639646+0000) 2019-10-13T11:39:06.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:06.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:06.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:06.633+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.640095+0000) 2019-10-13T11:39:06.641 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:06.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.640467+0000) 2019-10-13T11:39:06.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:06.639+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:06.644024+0000) 2019-10-13T11:39:07.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:07.809+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:07.810004+0000) 2019-10-13T11:39:08.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:08.033+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.036329+0000) 2019-10-13T11:39:08.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:08.053+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.057584+0000) 2019-10-13T11:39:08.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:08.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:08.578727+0000) 2019-10-13T11:39:11.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:11.641+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.642992+0000) 2019-10-13T11:39:11.644 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:11.641+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.643556+0000) 2019-10-13T11:39:11.646 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:11.645+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.645882+0000) 2019-10-13T11:39:11.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:11.641+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.645354+0000) 2019-10-13T11:39:11.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:11.645+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.646376+0000) 2019-10-13T11:39:11.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:11.645+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.647867+0000) 2019-10-13T11:39:11.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:11.647+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.649402+0000) 2019-10-13T11:39:11.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:11.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:11.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:11.655+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.656284+0000) 2019-10-13T11:39:11.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:11.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:11.829575+0000) 2019-10-13T11:39:13.057 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:13.055+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:13.056404+0000) 2019-10-13T11:39:14.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:14.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:14.152003+0000) 2019-10-13T11:39:14.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:14.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:14.438734+0000) 2019-10-13T11:39:16.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:16.653+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.654480+0000) 2019-10-13T11:39:16.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:16.653+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.654878+0000) 2019-10-13T11:39:16.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:16.653+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.655292+0000) 2019-10-13T11:39:16.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:16.653+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.655928+0000) 2019-10-13T11:39:16.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:16.653+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.656919+0000) 2019-10-13T11:39:16.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:16.653+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.657185+0000) 2019-10-13T11:39:16.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:16.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.658360+0000) 2019-10-13T11:39:16.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:16.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:16.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:16.659+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:16.661781+0000) 2019-10-13T11:39:17.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:17.809+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:17.810230+0000) 2019-10-13T11:39:18.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:18.033+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.036530+0000) 2019-10-13T11:39:18.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:18.057+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.057779+0000) 2019-10-13T11:39:18.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:18.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:18.578917+0000) 2019-10-13T11:39:21.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:21.665+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.666274+0000) 2019-10-13T11:39:21.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:21.665+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.667138+0000) 2019-10-13T11:39:21.669 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:21.665+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.668520+0000) 2019-10-13T11:39:21.670 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:21.665+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.669202+0000) 2019-10-13T11:39:21.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:21.665+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.669565+0000) 2019-10-13T11:39:21.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:21.667+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.671148+0000) 2019-10-13T11:39:21.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:21.669+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.672303+0000) 2019-10-13T11:39:21.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:21.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:21.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:21.671+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.675427+0000) 2019-10-13T11:39:21.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:21.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:21.829745+0000) 2019-10-13T11:39:23.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:23.059+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:23.062938+0000) 2019-10-13T11:39:24.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:24.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:24.152236+0000) 2019-10-13T11:39:24.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:24.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:24.438966+0000) 2019-10-13T11:39:26.408 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:39:26.408 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:26.414 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:39:26.414 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:26.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:26.673+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.676047+0000) 2019-10-13T11:39:26.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:26.675+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.676738+0000) 2019-10-13T11:39:26.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:26.673+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.677433+0000) 2019-10-13T11:39:26.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:26.677+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.677831+0000) 2019-10-13T11:39:26.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:26.677+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.678195+0000) 2019-10-13T11:39:26.680 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:26.677+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.678706+0000) 2019-10-13T11:39:26.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:26.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.679715+0000) 2019-10-13T11:39:26.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:26.681+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:26.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:26.683+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:26.685504+0000) 2019-10-13T11:39:27.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:27.809+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:27.810462+0000) 2019-10-13T11:39:28.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:28.033+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.036745+0000) 2019-10-13T11:39:28.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:28.057+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.057974+0000) 2019-10-13T11:39:28.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:28.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:28.579126+0000) 2019-10-13T11:39:31.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:31.683+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.686641+0000) 2019-10-13T11:39:31.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:31.685+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.689628+0000) 2019-10-13T11:39:31.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:31.689+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.690177+0000) 2019-10-13T11:39:31.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:31.689+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.690491+0000) 2019-10-13T11:39:31.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:31.689+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.691010+0000) 2019-10-13T11:39:31.692 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:31.689+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.691530+0000) 2019-10-13T11:39:31.692 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:31.689+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.691945+0000) 2019-10-13T11:39:31.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:31.691+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.693580+0000) 2019-10-13T11:39:31.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:31.693+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:31.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:31.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:31.829949+0000) 2019-10-13T11:39:33.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:33.071+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:33.074061+0000) 2019-10-13T11:39:34.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:34.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:34.152435+0000) 2019-10-13T11:39:34.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:34.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:34.439196+0000) 2019-10-13T11:39:36.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:36.693+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.697155+0000) 2019-10-13T11:39:36.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:36.697+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.697977+0000) 2019-10-13T11:39:36.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:36.697+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.698281+0000) 2019-10-13T11:39:36.699 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:36.697+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.698582+0000) 2019-10-13T11:39:36.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:36.699+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.703597+0000) 2019-10-13T11:39:36.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:36.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143508 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:36.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:36.701+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.705465+0000) 2019-10-13T11:39:36.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:36.705+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.706026+0000) 2019-10-13T11:39:36.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:36.707+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:36.711961+0000) 2019-10-13T11:39:37.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:37.809+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:37.810662+0000) 2019-10-13T11:39:38.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:38.033+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.036943+0000) 2019-10-13T11:39:38.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:38.057+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.058203+0000) 2019-10-13T11:39:38.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:38.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:38.579351+0000) 2019-10-13T11:39:41.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:41.707+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.709361+0000) 2019-10-13T11:39:41.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:41.709+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.710720+0000) 2019-10-13T11:39:41.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:41.709+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.711541+0000) 2019-10-13T11:39:41.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:41.709+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.712172+0000) 2019-10-13T11:39:41.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:41.709+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.712354+0000) 2019-10-13T11:39:41.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:41.709+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.712677+0000) 2019-10-13T11:39:41.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:41.713+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.713842+0000) 2019-10-13T11:39:41.716 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:41.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:41.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:41.715+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.717344+0000) 2019-10-13T11:39:41.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:41.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:41.830155+0000) 2019-10-13T11:39:43.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:43.079+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:43.080686+0000) 2019-10-13T11:39:44.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:44.149+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:44.152646+0000) 2019-10-13T11:39:44.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:44.437+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:44.439387+0000) 2019-10-13T11:39:46.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:46.717+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.720353+0000) 2019-10-13T11:39:46.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:46.717+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.720966+0000) 2019-10-13T11:39:46.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:46.721+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.722524+0000) 2019-10-13T11:39:46.724 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:46.721+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.723213+0000) 2019-10-13T11:39:46.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:46.719+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.723465+0000) 2019-10-13T11:39:46.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:46.721+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.723000+0000) 2019-10-13T11:39:46.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:46.721+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.723899+0000) 2019-10-13T11:39:46.726 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:46.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:46.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:46.723+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:46.726005+0000) 2019-10-13T11:39:47.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:47.809+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:47.810863+0000) 2019-10-13T11:39:48.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:48.033+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.037105+0000) 2019-10-13T11:39:48.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:48.057+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.058402+0000) 2019-10-13T11:39:48.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:48.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:48.579603+0000) 2019-10-13T11:39:51.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:51.724+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.729465+0000) 2019-10-13T11:39:51.731 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:51.728+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.730133+0000) 2019-10-13T11:39:51.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:51.728+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.730716+0000) 2019-10-13T11:39:51.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:51.728+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.731405+0000) 2019-10-13T11:39:51.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:51.728+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.731630+0000) 2019-10-13T11:39:51.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:51.727+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.732053+0000) 2019-10-13T11:39:51.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:51.728+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.732482+0000) 2019-10-13T11:39:51.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:51.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:51.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:51.735+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.738004+0000) 2019-10-13T11:39:51.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:51.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:51.830333+0000) 2019-10-13T11:39:53.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:53.083+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:53.087302+0000) 2019-10-13T11:39:54.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:54.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:54.152868+0000) 2019-10-13T11:39:54.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:54.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:54.439590+0000) 2019-10-13T11:39:56.476 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:39:56.476 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:56.480 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:39:56.480 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:39:56.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:56.736+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.739441+0000) 2019-10-13T11:39:56.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:39:56.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.740003+0000) 2019-10-13T11:39:56.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:56.736+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.740880+0000) 2019-10-13T11:39:56.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:39:56.740+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.743292+0000) 2019-10-13T11:39:56.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:56.740+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.743520+0000) 2019-10-13T11:39:56.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:56.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.744878+0000) 2019-10-13T11:39:56.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:39:56.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:39:56.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:39:56.743+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.746633+0000) 2019-10-13T11:39:56.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:39:56.751+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:56.753874+0000) 2019-10-13T11:39:57.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:39:57.808+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:57.811060+0000) 2019-10-13T11:39:58.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:39:58.032+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.037322+0000) 2019-10-13T11:39:58.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:39:58.056+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.058623+0000) 2019-10-13T11:39:58.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:39:58.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:39:58.579788+0000) 2019-10-13T11:40:01.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:01.748+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.749841+0000) 2019-10-13T11:40:01.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:01.748+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.750058+0000) 2019-10-13T11:40:01.751 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:01.748+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.750419+0000) 2019-10-13T11:40:01.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:01.748+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.750744+0000) 2019-10-13T11:40:01.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:01.748+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.751550+0000) 2019-10-13T11:40:01.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:01.751+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.753203+0000) 2019-10-13T11:40:01.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:01.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.753546+0000) 2019-10-13T11:40:01.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:01.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143664 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:01.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:01.755+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.759326+0000) 2019-10-13T11:40:01.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:01.827+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:01.830490+0000) 2019-10-13T11:40:03.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:03.091+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:03.093655+0000) 2019-10-13T11:40:04.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:04.148+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:04.153103+0000) 2019-10-13T11:40:04.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:04.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:04.439785+0000) 2019-10-13T11:40:06.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:06.756+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.759597+0000) 2019-10-13T11:40:06.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:06.756+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.761226+0000) 2019-10-13T11:40:06.763 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:06.760+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.762325+0000) 2019-10-13T11:40:06.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:06.760+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.762541+0000) 2019-10-13T11:40:06.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:06.760+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.762886+0000) 2019-10-13T11:40:06.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:06.759+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.763395+0000) 2019-10-13T11:40:06.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:06.760+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.763708+0000) 2019-10-13T11:40:06.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:06.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:06.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:06.763+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:06.767428+0000) 2019-10-13T11:40:07.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:07.808+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:07.811260+0000) 2019-10-13T11:40:08.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:08.036+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.037540+0000) 2019-10-13T11:40:08.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:08.056+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.058861+0000) 2019-10-13T11:40:08.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:08.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:08.580002+0000) 2019-10-13T11:40:11.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:11.768+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.770111+0000) 2019-10-13T11:40:11.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:11.768+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.770528+0000) 2019-10-13T11:40:11.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:11.768+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.771663+0000) 2019-10-13T11:40:11.774 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:11.768+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.773694+0000) 2019-10-13T11:40:11.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:11.772+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.775449+0000) 2019-10-13T11:40:11.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:11.774+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.777166+0000) 2019-10-13T11:40:11.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:11.772+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:11.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:11.772+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.777395+0000) 2019-10-13T11:40:11.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:11.774+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.777902+0000) 2019-10-13T11:40:11.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:11.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:11.830643+0000) 2019-10-13T11:40:13.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:13.098+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:13.100136+0000) 2019-10-13T11:40:14.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:14.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:14.153319+0000) 2019-10-13T11:40:14.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:14.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:14.440017+0000) 2019-10-13T11:40:16.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:16.776+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.780099+0000) 2019-10-13T11:40:16.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:16.776+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.781703+0000) 2019-10-13T11:40:16.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:16.778+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.782011+0000) 2019-10-13T11:40:16.783 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:16.780+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.782443+0000) 2019-10-13T11:40:16.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:16.780+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.783210+0000) 2019-10-13T11:40:16.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:16.780+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.783462+0000) 2019-10-13T11:40:16.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:16.780+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.783693+0000) 2019-10-13T11:40:16.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:16.782+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:16.787788+0000) 2019-10-13T11:40:16.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:16.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:17.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:17.808+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:17.811468+0000) 2019-10-13T11:40:18.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:18.036+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.037735+0000) 2019-10-13T11:40:18.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:18.056+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.059090+0000) 2019-10-13T11:40:18.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:18.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:18.580237+0000) 2019-10-13T11:40:21.792 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:21.788+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.791145+0000) 2019-10-13T11:40:21.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:21.788+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.791449+0000) 2019-10-13T11:40:21.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:21.788+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.791854+0000) 2019-10-13T11:40:21.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:21.788+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.792167+0000) 2019-10-13T11:40:21.793 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:21.788+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.792915+0000) 2019-10-13T11:40:21.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:21.790+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.793261+0000) 2019-10-13T11:40:21.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:21.792+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.794428+0000) 2019-10-13T11:40:21.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:21.794+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.799871+0000) 2019-10-13T11:40:21.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:21.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143787 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:21.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:21.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:21.830812+0000) 2019-10-13T11:40:23.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:23.102+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:23.106657+0000) 2019-10-13T11:40:24.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:24.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:24.153542+0000) 2019-10-13T11:40:24.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:24.436+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:24.440207+0000) 2019-10-13T11:40:26.589 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:40:26.589 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:26.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:40:26.593 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:26.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:26.796+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.801387+0000) 2019-10-13T11:40:26.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:26.800+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.801826+0000) 2019-10-13T11:40:26.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:26.800+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.802608+0000) 2019-10-13T11:40:26.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:26.800+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.803418+0000) 2019-10-13T11:40:26.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:26.800+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.803806+0000) 2019-10-13T11:40:26.805 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:26.800+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.804387+0000) 2019-10-13T11:40:26.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:26.802+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.805151+0000) 2019-10-13T11:40:26.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:26.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:26.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:26.806+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:26.809472+0000) 2019-10-13T11:40:27.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:27.808+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:27.811628+0000) 2019-10-13T11:40:28.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:28.036+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.037917+0000) 2019-10-13T11:40:28.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:28.056+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.059292+0000) 2019-10-13T11:40:28.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:28.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:28.580460+0000) 2019-10-13T11:40:31.814 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:31.808+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.813366+0000) 2019-10-13T11:40:31.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:31.808+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.813612+0000) 2019-10-13T11:40:31.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:31.812+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.814160+0000) 2019-10-13T11:40:31.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:31.812+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.815116+0000) 2019-10-13T11:40:31.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:31.812+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.815554+0000) 2019-10-13T11:40:31.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:31.812+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.816193+0000) 2019-10-13T11:40:31.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:31.814+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.817408+0000) 2019-10-13T11:40:31.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:31.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143848 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:31.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:31.818+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.820965+0000) 2019-10-13T11:40:31.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:31.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:31.830977+0000) 2019-10-13T11:40:33.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:33.114+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:33.115456+0000) 2019-10-13T11:40:34.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:34.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:34.153799+0000) 2019-10-13T11:40:34.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:34.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:34.440449+0000) 2019-10-13T11:40:36.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:36.820+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.822282+0000) 2019-10-13T11:40:36.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:36.820+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.823064+0000) 2019-10-13T11:40:36.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:36.820+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.822788+0000) 2019-10-13T11:40:36.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:36.822+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.824352+0000) 2019-10-13T11:40:36.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:36.820+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.824751+0000) 2019-10-13T11:40:36.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:36.820+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.825533+0000) 2019-10-13T11:40:36.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:36.824+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.825859+0000) 2019-10-13T11:40:36.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:36.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:36.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:36.826+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:36.831320+0000) 2019-10-13T11:40:37.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:37.808+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:37.811828+0000) 2019-10-13T11:40:38.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:38.036+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.038148+0000) 2019-10-13T11:40:38.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:38.056+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.059508+0000) 2019-10-13T11:40:38.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:38.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:38.580675+0000) 2019-10-13T11:40:41.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:41.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.831182+0000) 2019-10-13T11:40:41.835 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:41.832+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.834387+0000) 2019-10-13T11:40:41.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:41.832+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.834925+0000) 2019-10-13T11:40:41.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:41.830+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.835745+0000) 2019-10-13T11:40:41.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:41.832+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.835999+0000) 2019-10-13T11:40:41.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:41.832+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.836235+0000) 2019-10-13T11:40:41.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:41.832+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.837273+0000) 2019-10-13T11:40:41.838 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:41.832+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.837414+0000) 2019-10-13T11:40:41.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:41.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143911 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:41.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:41.838+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:41.842096+0000) 2019-10-13T11:40:43.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:43.118+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:43.121969+0000) 2019-10-13T11:40:44.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:44.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:44.154004+0000) 2019-10-13T11:40:44.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:44.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:44.440650+0000) 2019-10-13T11:40:46.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:46.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.843667+0000) 2019-10-13T11:40:46.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:46.839+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.844325+0000) 2019-10-13T11:40:46.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:46.839+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.845163+0000) 2019-10-13T11:40:46.846 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:46.839+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.845546+0000) 2019-10-13T11:40:46.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:46.842+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.846138+0000) 2019-10-13T11:40:46.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:46.843+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.846891+0000) 2019-10-13T11:40:46.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:46.843+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.847600+0000) 2019-10-13T11:40:46.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:46.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:46.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:46.850+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:46.853887+0000) 2019-10-13T11:40:47.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:47.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:47.812000+0000) 2019-10-13T11:40:48.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:48.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.038366+0000) 2019-10-13T11:40:48.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:48.055+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.059710+0000) 2019-10-13T11:40:48.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:48.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:48.580895+0000) 2019-10-13T11:40:51.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:51.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.831402+0000) 2019-10-13T11:40:51.858 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:51.851+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.856702+0000) 2019-10-13T11:40:51.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:51.851+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.857560+0000) 2019-10-13T11:40:51.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:51.855+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.859434+0000) 2019-10-13T11:40:51.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:51.855+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.859960+0000) 2019-10-13T11:40:51.861 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:51.855+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.859887+0000) 2019-10-13T11:40:51.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:51.858+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.860599+0000) 2019-10-13T11:40:51.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:51.859+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.862725+0000) 2019-10-13T11:40:51.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:51.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:51.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:51.862+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:51.864262+0000) 2019-10-13T11:40:53.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:53.126+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:53.128600+0000) 2019-10-13T11:40:54.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:54.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:54.154209+0000) 2019-10-13T11:40:54.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:54.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:54.440868+0000) 2019-10-13T11:40:56.669 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:40:56.669 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:56.673 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:40:56.673 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:40:56.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:40:56.863+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.866742+0000) 2019-10-13T11:40:56.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:40:56.862+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.867416+0000) 2019-10-13T11:40:56.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:56.863+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.868148+0000) 2019-10-13T11:40:56.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:40:56.863+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.868636+0000) 2019-10-13T11:40:56.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:56.863+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.869187+0000) 2019-10-13T11:40:56.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:56.863+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.869698+0000) 2019-10-13T11:40:56.871 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:56.867+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.870192+0000) 2019-10-13T11:40:56.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:40:56.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:40:56.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:40:56.870+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:56.874961+0000) 2019-10-13T11:40:57.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:40:57.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:57.812180+0000) 2019-10-13T11:40:58.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:40:58.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.038633+0000) 2019-10-13T11:40:58.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:40:58.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.059939+0000) 2019-10-13T11:40:58.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:40:58.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:40:58.581046+0000) 2019-10-13T11:41:01.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:01.826+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.831630+0000) 2019-10-13T11:41:01.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:01.871+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.877677+0000) 2019-10-13T11:41:01.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:01.875+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.878254+0000) 2019-10-13T11:41:01.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:01.875+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.879296+0000) 2019-10-13T11:41:01.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:01.875+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.879821+0000) 2019-10-13T11:41:01.881 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:01.875+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.880440+0000) 2019-10-13T11:41:01.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:01.875+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.880683+0000) 2019-10-13T11:41:01.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:01.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.881482+0000) 2019-10-13T11:41:01.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:01.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:01.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:01.882+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:01.887400+0000) 2019-10-13T11:41:03.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:03.134+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:03.134997+0000) 2019-10-13T11:41:04.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:04.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:04.154451+0000) 2019-10-13T11:41:04.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:04.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:04.441093+0000) 2019-10-13T11:41:06.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:06.883+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.887826+0000) 2019-10-13T11:41:06.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:06.883+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.888748+0000) 2019-10-13T11:41:06.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:06.886+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.888799+0000) 2019-10-13T11:41:06.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:06.887+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.889492+0000) 2019-10-13T11:41:06.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:06.887+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.890376+0000) 2019-10-13T11:41:06.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:06.887+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.890498+0000) 2019-10-13T11:41:06.892 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:06.887+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.891311+0000) 2019-10-13T11:41:06.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:06.890+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:06.894542+0000) 2019-10-13T11:41:06.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:06.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:07.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:07.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:07.812399+0000) 2019-10-13T11:41:08.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:08.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.038837+0000) 2019-10-13T11:41:08.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:08.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.060138+0000) 2019-10-13T11:41:08.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:08.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:08.581236+0000) 2019-10-13T11:41:11.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:11.830+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.831829+0000) 2019-10-13T11:41:11.899 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:11.895+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.898157+0000) 2019-10-13T11:41:11.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:11.895+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.898501+0000) 2019-10-13T11:41:11.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:11.895+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.899032+0000) 2019-10-13T11:41:11.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:11.895+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.899363+0000) 2019-10-13T11:41:11.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:11.894+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.899754+0000) 2019-10-13T11:41:11.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:11.895+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.900597+0000) 2019-10-13T11:41:11.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:11.899+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.902005+0000) 2019-10-13T11:41:11.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:11.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:11.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:11.906+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:11.908597+0000) 2019-10-13T11:41:13.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:13.137+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:13.141563+0000) 2019-10-13T11:41:14.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:14.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:14.154655+0000) 2019-10-13T11:41:14.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:14.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:14.441329+0000) 2019-10-13T11:41:16.910 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:16.903+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.909505+0000) 2019-10-13T11:41:16.911 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:16.907+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.910077+0000) 2019-10-13T11:41:16.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:16.905+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.911712+0000) 2019-10-13T11:41:16.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:16.911+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.914297+0000) 2019-10-13T11:41:16.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:16.911+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.915055+0000) 2019-10-13T11:41:16.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:16.911+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.915872+0000) 2019-10-13T11:41:16.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:16.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:16.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:16.911+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.917082+0000) 2019-10-13T11:41:16.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:16.913+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:16.918205+0000) 2019-10-13T11:41:17.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:17.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:17.812571+0000) 2019-10-13T11:41:18.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:18.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.039037+0000) 2019-10-13T11:41:18.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:18.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.060379+0000) 2019-10-13T11:41:18.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:18.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:18.581425+0000) 2019-10-13T11:41:21.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:21.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.831997+0000) 2019-10-13T11:41:21.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:21.915+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.919895+0000) 2019-10-13T11:41:21.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:21.915+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.920949+0000) 2019-10-13T11:41:21.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:21.915+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.921104+0000) 2019-10-13T11:41:21.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:21.917+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.921903+0000) 2019-10-13T11:41:21.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:21.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.922428+0000) 2019-10-13T11:41:21.923 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:21.919+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.922946+0000) 2019-10-13T11:41:21.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:21.919+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.923375+0000) 2019-10-13T11:41:21.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:21.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:21.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:21.925+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:21.928490+0000) 2019-10-13T11:41:23.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:23.145+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:23.147966+0000) 2019-10-13T11:41:24.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:24.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:24.154872+0000) 2019-10-13T11:41:24.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:24.435+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:24.441553+0000) 2019-10-13T11:41:26.766 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:41:26.766 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:26.770 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:41:26.770 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:26.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:26.927+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.929957+0000) 2019-10-13T11:41:26.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:26.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.931144+0000) 2019-10-13T11:41:26.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:26.927+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.932954+0000) 2019-10-13T11:41:26.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:26.931+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.935345+0000) 2019-10-13T11:41:26.937 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:26.931+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.937010+0000) 2019-10-13T11:41:26.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:26.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:26.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:26.933+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.938052+0000) 2019-10-13T11:41:26.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:26.933+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.938576+0000) 2019-10-13T11:41:26.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:26.935+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:26.938711+0000) 2019-10-13T11:41:27.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:27.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:27.812763+0000) 2019-10-13T11:41:28.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:28.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.039261+0000) 2019-10-13T11:41:28.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:28.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.060593+0000) 2019-10-13T11:41:28.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:28.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:28.581566+0000) 2019-10-13T11:41:31.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:31.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.832202+0000) 2019-10-13T11:41:31.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:31.935+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.940970+0000) 2019-10-13T11:41:31.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:31.935+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.941692+0000) 2019-10-13T11:41:31.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:31.939+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.942372+0000) 2019-10-13T11:41:31.944 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:31.939+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.943310+0000) 2019-10-13T11:41:31.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:31.939+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.943904+0000) 2019-10-13T11:41:31.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:31.941+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.944221+0000) 2019-10-13T11:41:31.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:31.939+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.944448+0000) 2019-10-13T11:41:31.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:31.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:31.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:31.945+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:31.949063+0000) 2019-10-13T11:41:33.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:33.153+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:33.154238+0000) 2019-10-13T11:41:34.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:34.151+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:34.155069+0000) 2019-10-13T11:41:34.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:34.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:34.441728+0000) 2019-10-13T11:41:36.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:36.947+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.951531+0000) 2019-10-13T11:41:36.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:36.947+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.951843+0000) 2019-10-13T11:41:36.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:36.947+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.952139+0000) 2019-10-13T11:41:36.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:36.947+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.952557+0000) 2019-10-13T11:41:36.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:36.947+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.952990+0000) 2019-10-13T11:41:36.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:36.947+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.953250+0000) 2019-10-13T11:41:36.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:36.949+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.954393+0000) 2019-10-13T11:41:36.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:36.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:36.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:36.957+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:36.961333+0000) 2019-10-13T11:41:37.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:37.807+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:37.812941+0000) 2019-10-13T11:41:38.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:38.035+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.039469+0000) 2019-10-13T11:41:38.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:38.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.060808+0000) 2019-10-13T11:41:38.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:38.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:38.581781+0000) 2019-10-13T11:41:41.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:41.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.832397+0000) 2019-10-13T11:41:41.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:41.958+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.961976+0000) 2019-10-13T11:41:41.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:41.954+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.961565+0000) 2019-10-13T11:41:41.963 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:41.958+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.962432+0000) 2019-10-13T11:41:41.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:41.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.962719+0000) 2019-10-13T11:41:41.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:41.958+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.963582+0000) 2019-10-13T11:41:41.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:41.958+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.963971+0000) 2019-10-13T11:41:41.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:41.958+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.965444+0000) 2019-10-13T11:41:41.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:41.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:41.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:41.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:41.970451+0000) 2019-10-13T11:41:43.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:43.157+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:43.160884+0000) 2019-10-13T11:41:44.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:44.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:44.155272+0000) 2019-10-13T11:41:44.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:44.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:44.441961+0000) 2019-10-13T11:41:46.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:46.966+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.973403+0000) 2019-10-13T11:41:46.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:46.966+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.973548+0000) 2019-10-13T11:41:46.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:46.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.974224+0000) 2019-10-13T11:41:46.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:46.970+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.974333+0000) 2019-10-13T11:41:46.977 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:46.970+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.976515+0000) 2019-10-13T11:41:46.980 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:46.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:46.981 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:46.974+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.980004+0000) 2019-10-13T11:41:46.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:46.977+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.980236+0000) 2019-10-13T11:41:46.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:46.977+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:46.980954+0000) 2019-10-13T11:41:47.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:47.806+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:47.813110+0000) 2019-10-13T11:41:48.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:48.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.039667+0000) 2019-10-13T11:41:48.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:48.058+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.061075+0000) 2019-10-13T11:41:48.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:48.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:48.581973+0000) 2019-10-13T11:41:51.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:51.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.832593+0000) 2019-10-13T11:41:51.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:51.974+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.981408+0000) 2019-10-13T11:41:51.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:51.978+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.983675+0000) 2019-10-13T11:41:51.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:51.978+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.984016+0000) 2019-10-13T11:41:51.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:51.978+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.984487+0000) 2019-10-13T11:41:51.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:51.981+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.984930+0000) 2019-10-13T11:41:51.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:51.978+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.985290+0000) 2019-10-13T11:41:51.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:51.982+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.986201+0000) 2019-10-13T11:41:51.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:51.985+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:51.989324+0000) 2019-10-13T11:41:51.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:51.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:53.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:53.165+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:53.167339+0000) 2019-10-13T11:41:54.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:54.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:54.155499+0000) 2019-10-13T11:41:54.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:54.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:54.442183+0000) 2019-10-13T11:41:56.863 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:41:56.863 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:56.866 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:41:56.867 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:41:56.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:56.986+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.993513+0000) 2019-10-13T11:41:56.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:56.990+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.995893+0000) 2019-10-13T11:41:56.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:41:56.994+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.998214+0000) 2019-10-13T11:41:57.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:41:56.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.999321+0000) 2019-10-13T11:41:57.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:56.994+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.999473+0000) 2019-10-13T11:41:57.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:56.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:56.999756+0000) 2019-10-13T11:41:57.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:41:56.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:57.000156+0000) 2019-10-13T11:41:57.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:41:56.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:41:57.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:41:57.001+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:57.006379+0000) 2019-10-13T11:41:57.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:41:57.810+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:57.813325+0000) 2019-10-13T11:41:58.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:41:58.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.039863+0000) 2019-10-13T11:41:58.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:41:58.058+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.061269+0000) 2019-10-13T11:41:58.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:41:58.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:41:58.582185+0000) 2019-10-13T11:42:01.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:01.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:01.832809+0000) 2019-10-13T11:42:02.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:01.998+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.004142+0000) 2019-10-13T11:42:02.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:02.002+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.006281+0000) 2019-10-13T11:42:02.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:02.002+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.006991+0000) 2019-10-13T11:42:02.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:02.002+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.007743+0000) 2019-10-13T11:42:02.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:02.001+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.007981+0000) 2019-10-13T11:42:02.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:02.002+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.008270+0000) 2019-10-13T11:42:02.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:02.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.009131+0000) 2019-10-13T11:42:02.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:02.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:02.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:02.013+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:02.016115+0000) 2019-10-13T11:42:03.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:03.169+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:03.173772+0000) 2019-10-13T11:42:04.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:04.150+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:04.155690+0000) 2019-10-13T11:42:04.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:04.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:04.442402+0000) 2019-10-13T11:42:07.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:07.010+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.015539+0000) 2019-10-13T11:42:07.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:07.010+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.016831+0000) 2019-10-13T11:42:07.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:07.010+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.017126+0000) 2019-10-13T11:42:07.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:07.010+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.017360+0000) 2019-10-13T11:42:07.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:07.014+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.017807+0000) 2019-10-13T11:42:07.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:07.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.018654+0000) 2019-10-13T11:42:07.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:07.013+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.018973+0000) 2019-10-13T11:42:07.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:07.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:07.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:07.021+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.025847+0000) 2019-10-13T11:42:07.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:07.810+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:07.813501+0000) 2019-10-13T11:42:08.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:08.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.040092+0000) 2019-10-13T11:42:08.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:08.058+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.061466+0000) 2019-10-13T11:42:08.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:08.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:08.582340+0000) 2019-10-13T11:42:11.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:11.829+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:11.833023+0000) 2019-10-13T11:42:12.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:12.022+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.026748+0000) 2019-10-13T11:42:12.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:12.022+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.029297+0000) 2019-10-13T11:42:12.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:12.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.029934+0000) 2019-10-13T11:42:12.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:12.026+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.030027+0000) 2019-10-13T11:42:12.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:12.026+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.031045+0000) 2019-10-13T11:42:12.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:12.026+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.031480+0000) 2019-10-13T11:42:12.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:12.029+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.033377+0000) 2019-10-13T11:42:12.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:12.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:12.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:12.033+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:12.039650+0000) 2019-10-13T11:42:13.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:13.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:13.180012+0000) 2019-10-13T11:42:14.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:14.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:14.155893+0000) 2019-10-13T11:42:14.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:14.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:14.442604+0000) 2019-10-13T11:42:17.038 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:17.030+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.037332+0000) 2019-10-13T11:42:17.039 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:17.034+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.038004+0000) 2019-10-13T11:42:17.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:17.034+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.039905+0000) 2019-10-13T11:42:17.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:17.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.040150+0000) 2019-10-13T11:42:17.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:17.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.040481+0000) 2019-10-13T11:42:17.043 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:17.037+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.042408+0000) 2019-10-13T11:42:17.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:17.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.044358+0000) 2019-10-13T11:42:17.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:17.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:17.046 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:17.041+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.045802+0000) 2019-10-13T11:42:17.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:17.810+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:17.813672+0000) 2019-10-13T11:42:18.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:18.038+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.040297+0000) 2019-10-13T11:42:18.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:18.058+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.061683+0000) 2019-10-13T11:42:18.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:18.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:18.582534+0000) 2019-10-13T11:42:21.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:21.828+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:21.833235+0000) 2019-10-13T11:42:22.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:22.042+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.048611+0000) 2019-10-13T11:42:22.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:22.042+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.049395+0000) 2019-10-13T11:42:22.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:22.046+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.051820+0000) 2019-10-13T11:42:22.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:22.046+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.052322+0000) 2019-10-13T11:42:22.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:22.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.052930+0000) 2019-10-13T11:42:22.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:22.046+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.053478+0000) 2019-10-13T11:42:22.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:22.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:22.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:22.048+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.052389+0000) 2019-10-13T11:42:22.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:22.052+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:22.059342+0000) 2019-10-13T11:42:23.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:23.180+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:23.186423+0000) 2019-10-13T11:42:24.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:24.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:24.156112+0000) 2019-10-13T11:42:24.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:24.438+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:24.442753+0000) 2019-10-13T11:42:27.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:27.054+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.059079+0000) 2019-10-13T11:42:27.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:27.054+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.059338+0000) 2019-10-13T11:42:27.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:27.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.060039+0000) 2019-10-13T11:42:27.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:27.054+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.061042+0000) 2019-10-13T11:42:27.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:27.058+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.061803+0000) 2019-10-13T11:42:27.063 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:27.056+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.063051+0000) 2019-10-13T11:42:27.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:27.058+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.063768+0000) 2019-10-13T11:42:27.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:27.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:27.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:27.064+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.070193+0000) 2019-10-13T11:42:27.148 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:42:27.148 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:27.152 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:42:27.152 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:27.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:27.810+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:27.813854+0000) 2019-10-13T11:42:28.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:28.034+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.040500+0000) 2019-10-13T11:42:28.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:28.058+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.061947+0000) 2019-10-13T11:42:28.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:28.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:28.582730+0000) 2019-10-13T11:42:31.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:31.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:31.833432+0000) 2019-10-13T11:42:32.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:32.064+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.071137+0000) 2019-10-13T11:42:32.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:32.066+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.071642+0000) 2019-10-13T11:42:32.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:32.066+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.071894+0000) 2019-10-13T11:42:32.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:32.066+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.072193+0000) 2019-10-13T11:42:32.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:32.066+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.072488+0000) 2019-10-13T11:42:32.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:32.066+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.073328+0000) 2019-10-13T11:42:32.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:32.070+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.074708+0000) 2019-10-13T11:42:32.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:32.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:32.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:32.076+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:32.081609+0000) 2019-10-13T11:42:33.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:33.188+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:33.193006+0000) 2019-10-13T11:42:34.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:34.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:34.156314+0000) 2019-10-13T11:42:34.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:34.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:34.442953+0000) 2019-10-13T11:42:37.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:37.077+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.082299+0000) 2019-10-13T11:42:37.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:37.077+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.082701+0000) 2019-10-13T11:42:37.084 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:37.077+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.083194+0000) 2019-10-13T11:42:37.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:37.077+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.083663+0000) 2019-10-13T11:42:37.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:37.077+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.084247+0000) 2019-10-13T11:42:37.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:37.080+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.084500+0000) 2019-10-13T11:42:37.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:37.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.084999+0000) 2019-10-13T11:42:37.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:37.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:37.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:37.084+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.090657+0000) 2019-10-13T11:42:37.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:37.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:37.814048+0000) 2019-10-13T11:42:38.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:38.037+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.040707+0000) 2019-10-13T11:42:38.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:38.057+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.062140+0000) 2019-10-13T11:42:38.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:38.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:38.582925+0000) 2019-10-13T11:42:41.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:41.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:41.833625+0000) 2019-10-13T11:42:42.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:42.089+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.093398+0000) 2019-10-13T11:42:42.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:42.089+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.093797+0000) 2019-10-13T11:42:42.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:42.089+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.094240+0000) 2019-10-13T11:42:42.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:42.089+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.094683+0000) 2019-10-13T11:42:42.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:42.089+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.095416+0000) 2019-10-13T11:42:42.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:42.089+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.096379+0000) 2019-10-13T11:42:42.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:42.092+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.096882+0000) 2019-10-13T11:42:42.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:42.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:42.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:42.096+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:42.101456+0000) 2019-10-13T11:42:43.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:43.196+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:43.202632+0000) 2019-10-13T11:42:44.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:44.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:44.156531+0000) 2019-10-13T11:42:44.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:44.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:44.443152+0000) 2019-10-13T11:42:47.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:47.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.103601+0000) 2019-10-13T11:42:47.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:47.097+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.105557+0000) 2019-10-13T11:42:47.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:47.100+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.106502+0000) 2019-10-13T11:42:47.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:47.101+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.106628+0000) 2019-10-13T11:42:47.109 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:47.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.108623+0000) 2019-10-13T11:42:47.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:47.105+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.111140+0000) 2019-10-13T11:42:47.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:47.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:47.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:47.105+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.112143+0000) 2019-10-13T11:42:47.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:47.108+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.113118+0000) 2019-10-13T11:42:47.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:47.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:47.814281+0000) 2019-10-13T11:42:48.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:48.037+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.040913+0000) 2019-10-13T11:42:48.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:48.061+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.062345+0000) 2019-10-13T11:42:48.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:48.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:48.583149+0000) 2019-10-13T11:42:51.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:51.828+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:51.833815+0000) 2019-10-13T11:42:52.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:52.113+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.115282+0000) 2019-10-13T11:42:52.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:52.113+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.117170+0000) 2019-10-13T11:42:52.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:52.112+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.117633+0000) 2019-10-13T11:42:52.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:52.117+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.118229+0000) 2019-10-13T11:42:52.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:52.117+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.118900+0000) 2019-10-13T11:42:52.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:52.117+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.119808+0000) 2019-10-13T11:42:52.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:52.117+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.120155+0000) 2019-10-13T11:42:52.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:52.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:52.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:52.120+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:52.125643+0000) 2019-10-13T11:42:53.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:53.212+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:53.213374+0000) 2019-10-13T11:42:54.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:54.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:54.156735+0000) 2019-10-13T11:42:54.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:54.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:54.443364+0000) 2019-10-13T11:42:57.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:57.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.124771+0000) 2019-10-13T11:42:57.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:42:57.120+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.126213+0000) 2019-10-13T11:42:57.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:57.125+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.127915+0000) 2019-10-13T11:42:57.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:57.129+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.130896+0000) 2019-10-13T11:42:57.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:42:57.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:42:57.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:57.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.132782+0000) 2019-10-13T11:42:57.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:42:57.129+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.133199+0000) 2019-10-13T11:42:57.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:42:57.129+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.133563+0000) 2019-10-13T11:42:57.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:42:57.128+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.135222+0000) 2019-10-13T11:42:57.205 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:42:57.205 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:57.209 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:42:57.209 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:42:57.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:42:57.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:57.814461+0000) 2019-10-13T11:42:58.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:42:58.037+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.041106+0000) 2019-10-13T11:42:58.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:42:58.061+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.062563+0000) 2019-10-13T11:42:58.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:42:58.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:42:58.583335+0000) 2019-10-13T11:43:01.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:01.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:01.834021+0000) 2019-10-13T11:43:02.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:02.133+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.136914+0000) 2019-10-13T11:43:02.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:02.137+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.137781+0000) 2019-10-13T11:43:02.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:02.137+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.138467+0000) 2019-10-13T11:43:02.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:02.132+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.138611+0000) 2019-10-13T11:43:02.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:02.137+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.139073+0000) 2019-10-13T11:43:02.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:02.137+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.140264+0000) 2019-10-13T11:43:02.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:02.137+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.140811+0000) 2019-10-13T11:43:02.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:02.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144777 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:02.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:02.144+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:02.148987+0000) 2019-10-13T11:43:03.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:03.212+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:03.219753+0000) 2019-10-13T11:43:04.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:04.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:04.156974+0000) 2019-10-13T11:43:04.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:04.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:04.443562+0000) 2019-10-13T11:43:07.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:07.145+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.146723+0000) 2019-10-13T11:43:07.149 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:07.144+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.148869+0000) 2019-10-13T11:43:07.150 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:07.149+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.149732+0000) 2019-10-13T11:43:07.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:07.149+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.150099+0000) 2019-10-13T11:43:07.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:07.149+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.151159+0000) 2019-10-13T11:43:07.152 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:07.149+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.151398+0000) 2019-10-13T11:43:07.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:07.149+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.153098+0000) 2019-10-13T11:43:07.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:07.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:07.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:07.148+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.154707+0000) 2019-10-13T11:43:07.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:07.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:07.814684+0000) 2019-10-13T11:43:08.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:08.037+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.041331+0000) 2019-10-13T11:43:08.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:08.061+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.062771+0000) 2019-10-13T11:43:08.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:08.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:08.583529+0000) 2019-10-13T11:43:11.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:11.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:11.834251+0000) 2019-10-13T11:43:12.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:12.153+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.157228+0000) 2019-10-13T11:43:12.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:12.153+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.157635+0000) 2019-10-13T11:43:12.159 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:12.157+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.158357+0000) 2019-10-13T11:43:12.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:12.152+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.158645+0000) 2019-10-13T11:43:12.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:12.157+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.159189+0000) 2019-10-13T11:43:12.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:12.157+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.160004+0000) 2019-10-13T11:43:12.161 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:12.157+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.160527+0000) 2019-10-13T11:43:12.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:12.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:12.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:12.160+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:12.165701+0000) 2019-10-13T11:43:13.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:13.224+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:13.226230+0000) 2019-10-13T11:43:14.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:14.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:14.157178+0000) 2019-10-13T11:43:14.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:14.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:14.443762+0000) 2019-10-13T11:43:17.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:17.165+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.168586+0000) 2019-10-13T11:43:17.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:17.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.168782+0000) 2019-10-13T11:43:17.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:17.165+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.169263+0000) 2019-10-13T11:43:17.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:17.168+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.169611+0000) 2019-10-13T11:43:17.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:17.169+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.170242+0000) 2019-10-13T11:43:17.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:17.169+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.171073+0000) 2019-10-13T11:43:17.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:17.169+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.171766+0000) 2019-10-13T11:43:17.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:17.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:17.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:17.176+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.176975+0000) 2019-10-13T11:43:17.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:17.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:17.814860+0000) 2019-10-13T11:43:18.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:18.037+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.041502+0000) 2019-10-13T11:43:18.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:18.061+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.062948+0000) 2019-10-13T11:43:18.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:18.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:18.583747+0000) 2019-10-13T11:43:21.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:21.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:21.834444+0000) 2019-10-13T11:43:22.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:22.177+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.179098+0000) 2019-10-13T11:43:22.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:22.176+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.179341+0000) 2019-10-13T11:43:22.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:22.177+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.179755+0000) 2019-10-13T11:43:22.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:22.177+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.180795+0000) 2019-10-13T11:43:22.182 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:22.177+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.181032+0000) 2019-10-13T11:43:22.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:22.181+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.181885+0000) 2019-10-13T11:43:22.183 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:22.181+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.182688+0000) 2019-10-13T11:43:22.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:22.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:22.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:22.184+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:22.186945+0000) 2019-10-13T11:43:23.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:23.231+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:23.232830+0000) 2019-10-13T11:43:24.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:24.153+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:24.157374+0000) 2019-10-13T11:43:24.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:24.441+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:24.443920+0000) 2019-10-13T11:43:27.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:27.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.189090+0000) 2019-10-13T11:43:27.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:27.189+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.190063+0000) 2019-10-13T11:43:27.192 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:27.189+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.191230+0000) 2019-10-13T11:43:27.192 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:27.189+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.191503+0000) 2019-10-13T11:43:27.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:27.191+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.192382+0000) 2019-10-13T11:43:27.193 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:27.189+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.192682+0000) 2019-10-13T11:43:27.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:27.189+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.193075+0000) 2019-10-13T11:43:27.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:27.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:27.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:27.195+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.197557+0000) 2019-10-13T11:43:27.262 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:43:27.262 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:27.266 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:43:27.266 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:27.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:27.813+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:27.815027+0000) 2019-10-13T11:43:28.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:28.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.041717+0000) 2019-10-13T11:43:28.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:28.061+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.063129+0000) 2019-10-13T11:43:28.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:28.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:28.583933+0000) 2019-10-13T11:43:31.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:31.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:31.834646+0000) 2019-10-13T11:43:32.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:32.196+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.199347+0000) 2019-10-13T11:43:32.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:32.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.201216+0000) 2019-10-13T11:43:32.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:32.196+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.201637+0000) 2019-10-13T11:43:32.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:32.200+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.201973+0000) 2019-10-13T11:43:32.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:32.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.202604+0000) 2019-10-13T11:43:32.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:32.200+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.202840+0000) 2019-10-13T11:43:32.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:32.200+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.203256+0000) 2019-10-13T11:43:32.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:32.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:32.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:32.203+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:32.207488+0000) 2019-10-13T11:43:33.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:33.235+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:33.239326+0000) 2019-10-13T11:43:34.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:34.152+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:34.157605+0000) 2019-10-13T11:43:34.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:34.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:34.444128+0000) 2019-10-13T11:43:37.211 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:37.207+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.210547+0000) 2019-10-13T11:43:37.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:37.208+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.211762+0000) 2019-10-13T11:43:37.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:37.208+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.211815+0000) 2019-10-13T11:43:37.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:37.208+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.212283+0000) 2019-10-13T11:43:37.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:37.208+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.212760+0000) 2019-10-13T11:43:37.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:37.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.213039+0000) 2019-10-13T11:43:37.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:37.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.213495+0000) 2019-10-13T11:43:37.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:37.215+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.217240+0000) 2019-10-13T11:43:37.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:37.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:37.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:37.812+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:37.815225+0000) 2019-10-13T11:43:38.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:38.040+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.041923+0000) 2019-10-13T11:43:38.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:38.060+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.063308+0000) 2019-10-13T11:43:38.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:38.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:38.584135+0000) 2019-10-13T11:43:41.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:41.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:41.834846+0000) 2019-10-13T11:43:42.222 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:42.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.221609+0000) 2019-10-13T11:43:42.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:42.220+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.222257+0000) 2019-10-13T11:43:42.224 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:42.220+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.223443+0000) 2019-10-13T11:43:42.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:42.220+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.223806+0000) 2019-10-13T11:43:42.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:42.219+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.223971+0000) 2019-10-13T11:43:42.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:42.224+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.226018+0000) 2019-10-13T11:43:42.227 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:42.224+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.226798+0000) 2019-10-13T11:43:42.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:42.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:42.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:42.231+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:42.233521+0000) 2019-10-13T11:43:43.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:43.247+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:43.250566+0000) 2019-10-13T11:43:44.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:44.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:44.157869+0000) 2019-10-13T11:43:44.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:44.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:44.444360+0000) 2019-10-13T11:43:47.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:47.228+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.232133+0000) 2019-10-13T11:43:47.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:47.228+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.232780+0000) 2019-10-13T11:43:47.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:47.228+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.233277+0000) 2019-10-13T11:43:47.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:47.232+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.234032+0000) 2019-10-13T11:43:47.235 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:47.231+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.234916+0000) 2019-10-13T11:43:47.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:47.232+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.235212+0000) 2019-10-13T11:43:47.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:47.232+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.235872+0000) 2019-10-13T11:43:47.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:47.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:47.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:47.239+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.240522+0000) 2019-10-13T11:43:47.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:47.812+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:47.815401+0000) 2019-10-13T11:43:48.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:48.040+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.042134+0000) 2019-10-13T11:43:48.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:48.060+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.063483+0000) 2019-10-13T11:43:48.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:48.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:48.584386+0000) 2019-10-13T11:43:51.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:51.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:51.835045+0000) 2019-10-13T11:43:52.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:52.240+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.242443+0000) 2019-10-13T11:43:52.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:52.240+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.242721+0000) 2019-10-13T11:43:52.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:52.240+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.243617+0000) 2019-10-13T11:43:52.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:52.240+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.245418+0000) 2019-10-13T11:43:52.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:52.243+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.247419+0000) 2019-10-13T11:43:52.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:52.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145086 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:52.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:52.247+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.249482+0000) 2019-10-13T11:43:52.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:52.244+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.249585+0000) 2019-10-13T11:43:52.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:52.248+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:52.250309+0000) 2019-10-13T11:43:53.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:53.255+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:53.256937+0000) 2019-10-13T11:43:54.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:54.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:54.158086+0000) 2019-10-13T11:43:54.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:54.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:54.444561+0000) 2019-10-13T11:43:57.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:57.248+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.252664+0000) 2019-10-13T11:43:57.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:43:57.248+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.253435+0000) 2019-10-13T11:43:57.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:57.248+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.253554+0000) 2019-10-13T11:43:57.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:43:57.251+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.254084+0000) 2019-10-13T11:43:57.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:57.252+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.255197+0000) 2019-10-13T11:43:57.258 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:57.252+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.255660+0000) 2019-10-13T11:43:57.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:43:57.252+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.255840+0000) 2019-10-13T11:43:57.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:43:57.259+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.260408+0000) 2019-10-13T11:43:57.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:43:57.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145118 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:43:57.343 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:43:57.343 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:57.347 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:43:57.347 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:43:57.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:43:57.812+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:57.815574+0000) 2019-10-13T11:43:58.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:43:58.040+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.042308+0000) 2019-10-13T11:43:58.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:43:58.060+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.063665+0000) 2019-10-13T11:43:58.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:43:58.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:43:58.584603+0000) 2019-10-13T11:44:01.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:01.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:01.835288+0000) 2019-10-13T11:44:02.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:02.260+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.262014+0000) 2019-10-13T11:44:02.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:02.260+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.262954+0000) 2019-10-13T11:44:02.264 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:02.260+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.263580+0000) 2019-10-13T11:44:02.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:02.260+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.264213+0000) 2019-10-13T11:44:02.265 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:02.263+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.264623+0000) 2019-10-13T11:44:02.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:02.260+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.265436+0000) 2019-10-13T11:44:02.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:02.264+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.265832+0000) 2019-10-13T11:44:02.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:02.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:02.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:02.271+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:02.274025+0000) 2019-10-13T11:44:03.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:03.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:03.263448+0000) 2019-10-13T11:44:04.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:04.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:04.158292+0000) 2019-10-13T11:44:04.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:04.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:04.444756+0000) 2019-10-13T11:44:07.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:07.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.273025+0000) 2019-10-13T11:44:07.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:07.272+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.274976+0000) 2019-10-13T11:44:07.276 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:07.272+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.275447+0000) 2019-10-13T11:44:07.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:07.272+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.275913+0000) 2019-10-13T11:44:07.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:07.276+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.277854+0000) 2019-10-13T11:44:07.279 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:07.276+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.278521+0000) 2019-10-13T11:44:07.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:07.275+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.279433+0000) 2019-10-13T11:44:07.281 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:07.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:07.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:07.283+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.285026+0000) 2019-10-13T11:44:07.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:07.812+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:07.815750+0000) 2019-10-13T11:44:08.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:08.040+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.042484+0000) 2019-10-13T11:44:08.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:08.060+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.063860+0000) 2019-10-13T11:44:08.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:08.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:08.584803+0000) 2019-10-13T11:44:11.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:11.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:11.835544+0000) 2019-10-13T11:44:12.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:12.280+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.283941+0000) 2019-10-13T11:44:12.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:12.280+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.284284+0000) 2019-10-13T11:44:12.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:12.284+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.285916+0000) 2019-10-13T11:44:12.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:12.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.286692+0000) 2019-10-13T11:44:12.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:12.283+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.287237+0000) 2019-10-13T11:44:12.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:12.284+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.287578+0000) 2019-10-13T11:44:12.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:12.284+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.287891+0000) 2019-10-13T11:44:12.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:12.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145210 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:12.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:12.291+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:12.295594+0000) 2019-10-13T11:44:13.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:13.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:13.269604+0000) 2019-10-13T11:44:14.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:14.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:14.158528+0000) 2019-10-13T11:44:14.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:14.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:14.444951+0000) 2019-10-13T11:44:17.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:17.292+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.295449+0000) 2019-10-13T11:44:17.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:17.292+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.295675+0000) 2019-10-13T11:44:17.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:17.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.296688+0000) 2019-10-13T11:44:17.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:17.292+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.296667+0000) 2019-10-13T11:44:17.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:17.296+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.297821+0000) 2019-10-13T11:44:17.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:17.295+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.298152+0000) 2019-10-13T11:44:17.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:17.296+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.298650+0000) 2019-10-13T11:44:17.301 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:17.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:17.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:17.299+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.303392+0000) 2019-10-13T11:44:17.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:17.812+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:17.815956+0000) 2019-10-13T11:44:18.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:18.040+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.042663+0000) 2019-10-13T11:44:18.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:18.060+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.064038+0000) 2019-10-13T11:44:18.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:18.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:18.584996+0000) 2019-10-13T11:44:21.836 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:21.831+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:21.835770+0000) 2019-10-13T11:44:22.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:22.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.304013+0000) 2019-10-13T11:44:22.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:22.300+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.304503+0000) 2019-10-13T11:44:22.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:22.300+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.305047+0000) 2019-10-13T11:44:22.306 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:22.300+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.305417+0000) 2019-10-13T11:44:22.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:22.304+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.305936+0000) 2019-10-13T11:44:22.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:22.303+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.306967+0000) 2019-10-13T11:44:22.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:22.304+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.307374+0000) 2019-10-13T11:44:22.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:22.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:22.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:22.311+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:22.314544+0000) 2019-10-13T11:44:23.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:23.275+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:23.276204+0000) 2019-10-13T11:44:24.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:24.156+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:24.158750+0000) 2019-10-13T11:44:24.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:24.440+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:24.445150+0000) 2019-10-13T11:44:27.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:27.312+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.314888+0000) 2019-10-13T11:44:27.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:27.312+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.315229+0000) 2019-10-13T11:44:27.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:27.311+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.315615+0000) 2019-10-13T11:44:27.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:27.312+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.316042+0000) 2019-10-13T11:44:27.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:27.316+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.319387+0000) 2019-10-13T11:44:27.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:27.316+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.321063+0000) 2019-10-13T11:44:27.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:27.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:27.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:27.320+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.322754+0000) 2019-10-13T11:44:27.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:27.327+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.329977+0000) 2019-10-13T11:44:27.484 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:44:27.484 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:27.488 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:44:27.488 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:27.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:27.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:27.816195+0000) 2019-10-13T11:44:28.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:28.039+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.042833+0000) 2019-10-13T11:44:28.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:28.063+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.064247+0000) 2019-10-13T11:44:28.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:28.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:28.585219+0000) 2019-10-13T11:44:31.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:31.830+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:31.835959+0000) 2019-10-13T11:44:32.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:32.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.325516+0000) 2019-10-13T11:44:32.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:32.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.326686+0000) 2019-10-13T11:44:32.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:32.323+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.327483+0000) 2019-10-13T11:44:32.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:32.322+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.327664+0000) 2019-10-13T11:44:32.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:32.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.328069+0000) 2019-10-13T11:44:32.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:32.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.328663+0000) 2019-10-13T11:44:32.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:32.323+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.329198+0000) 2019-10-13T11:44:32.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:32.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:32.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:32.330+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:32.332620+0000) 2019-10-13T11:44:33.288 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:33.282+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:33.287621+0000) 2019-10-13T11:44:34.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:34.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:34.158981+0000) 2019-10-13T11:44:34.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:34.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:34.445351+0000) 2019-10-13T11:44:37.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:37.331+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.337032+0000) 2019-10-13T11:44:37.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:37.331+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.337608+0000) 2019-10-13T11:44:37.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:37.335+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.338017+0000) 2019-10-13T11:44:37.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:37.334+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.336705+0000) 2019-10-13T11:44:37.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:37.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.338808+0000) 2019-10-13T11:44:37.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:37.335+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.339480+0000) 2019-10-13T11:44:37.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:37.335+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.340274+0000) 2019-10-13T11:44:37.344 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:37.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145364 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:37.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:37.342+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.345489+0000) 2019-10-13T11:44:37.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:37.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:37.816379+0000) 2019-10-13T11:44:38.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:38.039+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.043014+0000) 2019-10-13T11:44:38.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:38.063+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.064428+0000) 2019-10-13T11:44:38.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:38.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:38.585391+0000) 2019-10-13T11:44:41.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:41.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:41.836160+0000) 2019-10-13T11:44:42.348 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:42.342+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.347185+0000) 2019-10-13T11:44:42.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:42.343+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.347628+0000) 2019-10-13T11:44:42.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:42.343+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.348146+0000) 2019-10-13T11:44:42.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:42.343+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.348477+0000) 2019-10-13T11:44:42.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:42.343+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.349460+0000) 2019-10-13T11:44:42.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:42.347+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.349811+0000) 2019-10-13T11:44:42.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:42.347+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.350185+0000) 2019-10-13T11:44:42.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:42.351+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:42.356 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:42.350+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:42.355455+0000) 2019-10-13T11:44:43.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:43.290+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:43.294176+0000) 2019-10-13T11:44:44.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:44.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:44.159176+0000) 2019-10-13T11:44:44.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:44.439+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:44.445581+0000) 2019-10-13T11:44:47.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:47.351+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.357366+0000) 2019-10-13T11:44:47.367 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:47.351+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.357438+0000) 2019-10-13T11:44:47.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:47.355+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.360038+0000) 2019-10-13T11:44:47.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:47.355+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.360311+0000) 2019-10-13T11:44:47.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:47.355+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.360749+0000) 2019-10-13T11:44:47.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:47.359+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.362689+0000) 2019-10-13T11:44:47.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:47.359+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:47.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:47.354+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.358036+0000) 2019-10-13T11:44:47.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:47.370+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.372390+0000) 2019-10-13T11:44:47.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:47.811+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:47.816563+0000) 2019-10-13T11:44:48.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:48.039+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.043209+0000) 2019-10-13T11:44:48.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:48.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.064605+0000) 2019-10-13T11:44:48.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:48.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:48.585595+0000) 2019-10-13T11:44:51.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:51.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:51.836369+0000) 2019-10-13T11:44:52.369 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:52.363+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.368793+0000) 2019-10-13T11:44:52.370 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:52.363+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.369185+0000) 2019-10-13T11:44:52.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:52.363+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.369502+0000) 2019-10-13T11:44:52.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:52.367+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.370134+0000) 2019-10-13T11:44:52.371 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:52.367+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.370485+0000) 2019-10-13T11:44:52.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:52.367+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.371841+0000) 2019-10-13T11:44:52.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:52.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:52.377 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:52.366+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.371480+0000) 2019-10-13T11:44:52.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:52.374+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:52.377573+0000) 2019-10-13T11:44:53.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:53.298+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:53.300641+0000) 2019-10-13T11:44:54.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:54.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:54.159411+0000) 2019-10-13T11:44:54.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:54.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:54.445794+0000) 2019-10-13T11:44:57.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:57.375+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.380455+0000) 2019-10-13T11:44:57.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:44:57.375+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.381068+0000) 2019-10-13T11:44:57.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:44:57.378+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.382524+0000) 2019-10-13T11:44:57.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:57.379+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.382912+0000) 2019-10-13T11:44:57.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:57.379+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.383277+0000) 2019-10-13T11:44:57.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:57.379+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.384025+0000) 2019-10-13T11:44:57.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:44:57.379+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.384276+0000) 2019-10-13T11:44:57.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:44:57.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145484 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:44:57.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:44:57.386+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.390283+0000) 2019-10-13T11:44:57.591 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:44:57.591 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:57.595 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:44:57.595 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:44:57.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:44:57.811+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:57.816739+0000) 2019-10-13T11:44:58.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:44:58.039+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.043408+0000) 2019-10-13T11:44:58.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:44:58.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.064779+0000) 2019-10-13T11:44:58.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:44:58.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:44:58.585791+0000) 2019-10-13T11:45:01.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:01.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:01.836596+0000) 2019-10-13T11:45:02.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:02.387+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.389985+0000) 2019-10-13T11:45:02.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:02.387+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.390298+0000) 2019-10-13T11:45:02.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:02.387+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.390898+0000) 2019-10-13T11:45:02.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:02.387+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.391586+0000) 2019-10-13T11:45:02.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:02.387+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.392223+0000) 2019-10-13T11:45:02.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:02.387+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.392298+0000) 2019-10-13T11:45:02.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:02.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.393016+0000) 2019-10-13T11:45:02.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:02.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:02.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:02.398+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:02.401970+0000) 2019-10-13T11:45:03.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:03.302+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:03.307047+0000) 2019-10-13T11:45:04.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:04.155+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:04.159644+0000) 2019-10-13T11:45:04.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:04.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:04.445995+0000) 2019-10-13T11:45:07.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:07.395+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.401467+0000) 2019-10-13T11:45:07.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:07.395+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.401263+0000) 2019-10-13T11:45:07.403 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:07.399+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.402134+0000) 2019-10-13T11:45:07.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:07.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.403178+0000) 2019-10-13T11:45:07.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:07.399+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.404899+0000) 2019-10-13T11:45:07.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:07.399+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.405488+0000) 2019-10-13T11:45:07.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:07.403+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.406235+0000) 2019-10-13T11:45:07.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:07.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:07.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:07.406+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.409185+0000) 2019-10-13T11:45:07.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:07.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:07.816916+0000) 2019-10-13T11:45:08.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:08.039+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.043538+0000) 2019-10-13T11:45:08.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:08.059+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.064962+0000) 2019-10-13T11:45:08.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:08.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:08.585978+0000) 2019-10-13T11:45:11.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:11.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:11.836792+0000) 2019-10-13T11:45:12.413 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:12.407+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.411771+0000) 2019-10-13T11:45:12.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:12.407+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.412315+0000) 2019-10-13T11:45:12.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:12.410+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.413268+0000) 2019-10-13T11:45:12.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:12.407+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.413031+0000) 2019-10-13T11:45:12.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:12.407+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.413685+0000) 2019-10-13T11:45:12.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:12.411+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.415352+0000) 2019-10-13T11:45:12.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:12.411+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.414706+0000) 2019-10-13T11:45:12.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:12.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:12.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:12.418+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:12.420636+0000) 2019-10-13T11:45:13.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:13.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:13.313549+0000) 2019-10-13T11:45:14.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:14.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:14.159842+0000) 2019-10-13T11:45:14.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:14.443+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:14.446152+0000) 2019-10-13T11:45:17.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:17.419+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.423110+0000) 2019-10-13T11:45:17.425 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:17.419+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.423434+0000) 2019-10-13T11:45:17.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:17.422+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.424461+0000) 2019-10-13T11:45:17.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:17.419+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.423967+0000) 2019-10-13T11:45:17.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:17.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.424882+0000) 2019-10-13T11:45:17.426 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:17.423+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.426015+0000) 2019-10-13T11:45:17.427 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:17.423+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.426286+0000) 2019-10-13T11:45:17.431 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:17.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145604 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:17.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:17.426+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.431000+0000) 2019-10-13T11:45:17.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:17.811+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:17.817115+0000) 2019-10-13T11:45:18.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:18.043+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.043713+0000) 2019-10-13T11:45:18.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:18.063+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.065144+0000) 2019-10-13T11:45:18.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:18.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:18.586184+0000) 2019-10-13T11:45:21.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:21.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:21.836988+0000) 2019-10-13T11:45:22.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:22.431+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.433842+0000) 2019-10-13T11:45:22.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:22.431+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.434331+0000) 2019-10-13T11:45:22.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:22.431+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.435265+0000) 2019-10-13T11:45:22.436 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:22.430+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.435553+0000) 2019-10-13T11:45:22.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:22.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.435971+0000) 2019-10-13T11:45:22.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:22.431+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.436473+0000) 2019-10-13T11:45:22.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:22.431+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.437251+0000) 2019-10-13T11:45:22.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:22.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:22.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:22.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:22.442849+0000) 2019-10-13T11:45:23.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:23.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:23.324773+0000) 2019-10-13T11:45:24.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:24.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:24.160064+0000) 2019-10-13T11:45:24.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:24.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:24.446361+0000) 2019-10-13T11:45:27.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:27.442+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.444149+0000) 2019-10-13T11:45:27.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:27.438+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.444951+0000) 2019-10-13T11:45:27.446 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:27.438+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.445420+0000) 2019-10-13T11:45:27.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:27.442+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.446900+0000) 2019-10-13T11:45:27.448 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:27.442+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.447333+0000) 2019-10-13T11:45:27.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:27.442+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.447970+0000) 2019-10-13T11:45:27.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:27.442+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.449172+0000) 2019-10-13T11:45:27.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:27.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:27.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:27.458+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.460186+0000) 2019-10-13T11:45:27.701 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:45:27.702 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:27.706 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:45:27.706 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:27.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:27.810+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:27.817304+0000) 2019-10-13T11:45:28.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:28.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.043936+0000) 2019-10-13T11:45:28.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:28.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.065325+0000) 2019-10-13T11:45:28.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:28.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:28.586335+0000) 2019-10-13T11:45:31.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:31.834+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:31.837201+0000) 2019-10-13T11:45:32.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:32.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.455480+0000) 2019-10-13T11:45:32.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:32.450+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.455863+0000) 2019-10-13T11:45:32.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:32.450+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.457680+0000) 2019-10-13T11:45:32.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:32.454+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.458294+0000) 2019-10-13T11:45:32.460 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:32.454+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.459216+0000) 2019-10-13T11:45:32.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:32.454+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.459902+0000) 2019-10-13T11:45:32.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:32.454+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.460480+0000) 2019-10-13T11:45:32.464 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:32.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145691 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:32.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:32.462+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:32.464305+0000) 2019-10-13T11:45:33.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:33.325+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:33.331355+0000) 2019-10-13T11:45:34.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:34.154+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:34.160300+0000) 2019-10-13T11:45:34.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:34.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:34.446603+0000) 2019-10-13T11:45:37.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:37.461+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.466482+0000) 2019-10-13T11:45:37.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:37.462+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.466990+0000) 2019-10-13T11:45:37.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:37.462+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.468662+0000) 2019-10-13T11:45:37.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:37.462+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.468612+0000) 2019-10-13T11:45:37.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:37.462+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.469363+0000) 2019-10-13T11:45:37.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:37.466+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.469825+0000) 2019-10-13T11:45:37.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:37.466+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.470536+0000) 2019-10-13T11:45:37.474 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:37.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:37.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:37.473+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.479368+0000) 2019-10-13T11:45:37.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:37.814+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:37.817409+0000) 2019-10-13T11:45:38.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:38.038+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.044173+0000) 2019-10-13T11:45:38.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:38.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.065498+0000) 2019-10-13T11:45:38.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:38.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:38.586541+0000) 2019-10-13T11:45:41.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:41.833+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:41.837446+0000) 2019-10-13T11:45:42.479 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:42.474+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.478009+0000) 2019-10-13T11:45:42.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:42.474+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.478387+0000) 2019-10-13T11:45:42.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:42.474+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.479220+0000) 2019-10-13T11:45:42.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:42.474+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.481660+0000) 2019-10-13T11:45:42.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:42.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:42.486 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:42.481+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.485306+0000) 2019-10-13T11:45:42.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:42.482+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.485755+0000) 2019-10-13T11:45:42.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:42.482+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.485730+0000) 2019-10-13T11:45:42.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:42.485+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:42.492058+0000) 2019-10-13T11:45:43.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:43.333+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:43.337841+0000) 2019-10-13T11:45:44.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:44.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:44.160510+0000) 2019-10-13T11:45:44.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:44.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:44.446804+0000) 2019-10-13T11:45:47.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:47.485+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.489254+0000) 2019-10-13T11:45:47.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:47.486+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.489895+0000) 2019-10-13T11:45:47.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:47.486+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.491675+0000) 2019-10-13T11:45:47.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:47.486+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.492010+0000) 2019-10-13T11:45:47.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:47.490+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.493776+0000) 2019-10-13T11:45:47.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:47.490+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.494270+0000) 2019-10-13T11:45:47.495 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:47.490+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.494639+0000) 2019-10-13T11:45:47.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:47.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:47.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:47.497+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.501448+0000) 2019-10-13T11:45:47.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:47.814+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:47.817590+0000) 2019-10-13T11:45:48.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:48.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.044384+0000) 2019-10-13T11:45:48.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:48.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.065676+0000) 2019-10-13T11:45:48.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:48.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:48.586712+0000) 2019-10-13T11:45:51.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:51.833+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:51.837646+0000) 2019-10-13T11:45:52.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:52.494+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.499324+0000) 2019-10-13T11:45:52.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:52.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.502017+0000) 2019-10-13T11:45:52.503 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:52.498+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.502254+0000) 2019-10-13T11:45:52.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:52.501+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.505979+0000) 2019-10-13T11:45:52.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:52.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145811 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:52.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:52.501+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.506891+0000) 2019-10-13T11:45:52.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:52.502+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.506967+0000) 2019-10-13T11:45:52.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:52.502+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.507243+0000) 2019-10-13T11:45:52.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:52.502+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:52.507422+0000) 2019-10-13T11:45:53.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:53.341+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:53.344254+0000) 2019-10-13T11:45:54.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:54.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:54.160720+0000) 2019-10-13T11:45:54.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:54.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:54.447020+0000) 2019-10-13T11:45:57.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:57.506+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.511503+0000) 2019-10-13T11:45:57.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:45:57.506+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.512438+0000) 2019-10-13T11:45:57.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:57.506+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.513345+0000) 2019-10-13T11:45:57.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:57.506+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.513614+0000) 2019-10-13T11:45:57.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:45:57.506+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.513405+0000) 2019-10-13T11:45:57.516 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:57.510+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.513893+0000) 2019-10-13T11:45:57.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:45:57.509+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.514301+0000) 2019-10-13T11:45:57.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:45:57.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:45:57.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:45:57.517+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.520889+0000) 2019-10-13T11:45:57.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:45:57.814+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:57.817829+0000) 2019-10-13T11:45:57.823 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:45:57.823 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:57.827 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:45:57.827 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:45:58.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:45:58.038+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.044560+0000) 2019-10-13T11:45:58.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:45:58.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.065872+0000) 2019-10-13T11:45:58.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:45:58.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:45:58.586958+0000) 2019-10-13T11:46:01.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:01.837+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:01.837840+0000) 2019-10-13T11:46:02.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:02.514+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.520208+0000) 2019-10-13T11:46:02.521 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:02.514+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.520463+0000) 2019-10-13T11:46:02.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:02.517+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.520923+0000) 2019-10-13T11:46:02.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:02.518+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.522311+0000) 2019-10-13T11:46:02.523 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:02.518+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.522671+0000) 2019-10-13T11:46:02.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:02.518+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.523108+0000) 2019-10-13T11:46:02.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:02.518+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.523681+0000) 2019-10-13T11:46:02.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:02.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:02.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:02.525+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:02.530912+0000) 2019-10-13T11:46:03.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:03.349+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:03.350680+0000) 2019-10-13T11:46:04.161 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:04.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:04.160927+0000) 2019-10-13T11:46:04.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:04.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:04.447225+0000) 2019-10-13T11:46:07.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:07.525+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.530095+0000) 2019-10-13T11:46:07.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:07.526+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.532035+0000) 2019-10-13T11:46:07.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:07.526+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.531488+0000) 2019-10-13T11:46:07.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:07.526+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.533168+0000) 2019-10-13T11:46:07.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:07.526+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.533003+0000) 2019-10-13T11:46:07.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:07.526+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.532569+0000) 2019-10-13T11:46:07.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:07.530+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.534032+0000) 2019-10-13T11:46:07.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:07.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:07.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:07.537+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.541022+0000) 2019-10-13T11:46:07.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:07.814+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:07.818007+0000) 2019-10-13T11:46:08.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:08.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.044751+0000) 2019-10-13T11:46:08.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:08.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.066079+0000) 2019-10-13T11:46:08.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:08.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:08.587135+0000) 2019-10-13T11:46:11.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:11.833+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:11.838066+0000) 2019-10-13T11:46:12.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:12.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.541140+0000) 2019-10-13T11:46:12.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:12.538+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.543216+0000) 2019-10-13T11:46:12.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:12.538+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.543622+0000) 2019-10-13T11:46:12.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:12.541+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.545679+0000) 2019-10-13T11:46:12.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:12.538+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.545350+0000) 2019-10-13T11:46:12.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:12.542+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.546029+0000) 2019-10-13T11:46:12.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:12.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.546306+0000) 2019-10-13T11:46:12.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:12.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:12.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:12.549+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:12.553160+0000) 2019-10-13T11:46:13.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:13.353+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:13.357263+0000) 2019-10-13T11:46:14.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:14.158+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:14.161148+0000) 2019-10-13T11:46:14.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:14.442+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:14.447449+0000) 2019-10-13T11:46:17.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:17.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.552157+0000) 2019-10-13T11:46:17.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:17.546+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.552779+0000) 2019-10-13T11:46:17.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:17.546+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.553308+0000) 2019-10-13T11:46:17.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:17.549+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.553586+0000) 2019-10-13T11:46:17.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:17.550+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.554279+0000) 2019-10-13T11:46:17.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:17.550+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.554768+0000) 2019-10-13T11:46:17.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:17.550+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.555614+0000) 2019-10-13T11:46:17.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:17.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:17.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:17.557+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.562028+0000) 2019-10-13T11:46:17.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:17.814+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:17.818200+0000) 2019-10-13T11:46:18.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:18.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.044923+0000) 2019-10-13T11:46:18.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:18.066+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.066261+0000) 2019-10-13T11:46:18.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:18.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:18.587309+0000) 2019-10-13T11:46:21.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:21.837+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:21.838265+0000) 2019-10-13T11:46:22.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:22.557+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.562417+0000) 2019-10-13T11:46:22.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:22.557+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.562933+0000) 2019-10-13T11:46:22.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:22.557+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.563706+0000) 2019-10-13T11:46:22.565 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:22.557+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.564081+0000) 2019-10-13T11:46:22.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:22.561+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.566576+0000) 2019-10-13T11:46:22.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:22.565+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.568665+0000) 2019-10-13T11:46:22.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:22.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:22.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:22.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.570475+0000) 2019-10-13T11:46:22.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:22.565+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:22.570937+0000) 2019-10-13T11:46:23.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:23.361+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:23.363827+0000) 2019-10-13T11:46:24.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:24.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:24.161352+0000) 2019-10-13T11:46:24.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:24.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:24.447644+0000) 2019-10-13T11:46:27.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:27.565+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.573702+0000) 2019-10-13T11:46:27.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:27.569+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.574070+0000) 2019-10-13T11:46:27.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:27.569+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.575636+0000) 2019-10-13T11:46:27.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:27.569+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.575958+0000) 2019-10-13T11:46:27.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:27.569+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.576247+0000) 2019-10-13T11:46:27.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:27.573+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.578505+0000) 2019-10-13T11:46:27.580 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:27.573+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.579197+0000) 2019-10-13T11:46:27.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:27.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:27.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:27.581+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.584744+0000) 2019-10-13T11:46:27.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:27.817+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:27.818434+0000) 2019-10-13T11:46:27.945 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:46:27.946 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:27.949 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:46:27.950 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:28.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:28.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.045099+0000) 2019-10-13T11:46:28.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:28.065+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.066436+0000) 2019-10-13T11:46:28.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:28.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:28.587531+0000) 2019-10-13T11:46:31.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:31.833+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:31.838455+0000) 2019-10-13T11:46:32.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:32.577+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.584190+0000) 2019-10-13T11:46:32.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:32.577+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.584659+0000) 2019-10-13T11:46:32.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:32.577+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.585267+0000) 2019-10-13T11:46:32.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:32.577+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.585624+0000) 2019-10-13T11:46:32.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:32.581+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.586501+0000) 2019-10-13T11:46:32.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:32.581+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.588719+0000) 2019-10-13T11:46:32.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:32.589+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.590225+0000) 2019-10-13T11:46:32.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:32.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:32.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:32.593+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:32.599683+0000) 2019-10-13T11:46:33.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:33.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:33.370468+0000) 2019-10-13T11:46:34.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:34.157+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:34.161554+0000) 2019-10-13T11:46:34.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:34.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:34.447851+0000) 2019-10-13T11:46:37.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:37.593+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.595874+0000) 2019-10-13T11:46:37.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:37.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.596380+0000) 2019-10-13T11:46:37.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:37.593+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.597094+0000) 2019-10-13T11:46:37.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:37.597+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.599475+0000) 2019-10-13T11:46:37.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:37.601+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.601928+0000) 2019-10-13T11:46:37.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:37.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:37.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:37.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.602824+0000) 2019-10-13T11:46:37.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:37.601+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.603219+0000) 2019-10-13T11:46:37.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:37.605+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.608878+0000) 2019-10-13T11:46:37.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:37.817+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:37.818610+0000) 2019-10-13T11:46:38.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:38.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.045277+0000) 2019-10-13T11:46:38.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:38.065+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.066642+0000) 2019-10-13T11:46:38.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:38.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:38.587732+0000) 2019-10-13T11:46:41.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:41.836+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:41.838682+0000) 2019-10-13T11:46:42.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:42.605+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.605967+0000) 2019-10-13T11:46:42.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:42.605+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.606399+0000) 2019-10-13T11:46:42.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:42.605+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.607469+0000) 2019-10-13T11:46:42.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:42.604+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.608806+0000) 2019-10-13T11:46:42.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:42.605+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.609217+0000) 2019-10-13T11:46:42.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:42.605+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.609399+0000) 2019-10-13T11:46:42.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:42.609+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.609878+0000) 2019-10-13T11:46:42.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:42.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:42.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:42.612+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:42.616784+0000) 2019-10-13T11:46:43.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:43.372+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:43.376934+0000) 2019-10-13T11:46:44.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:44.161+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:44.161786+0000) 2019-10-13T11:46:44.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:44.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:44.448049+0000) 2019-10-13T11:46:47.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:47.613+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.617632+0000) 2019-10-13T11:46:47.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:47.617+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.617983+0000) 2019-10-13T11:46:47.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:47.617+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.618670+0000) 2019-10-13T11:46:47.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:47.617+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.619003+0000) 2019-10-13T11:46:47.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:47.617+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.619569+0000) 2019-10-13T11:46:47.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:47.617+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.619914+0000) 2019-10-13T11:46:47.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:47.616+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.620377+0000) 2019-10-13T11:46:47.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:47.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:47.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:47.624+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.629010+0000) 2019-10-13T11:46:47.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:47.817+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:47.818816+0000) 2019-10-13T11:46:48.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:48.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.045469+0000) 2019-10-13T11:46:48.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:48.065+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.066854+0000) 2019-10-13T11:46:48.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:48.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:48.587952+0000) 2019-10-13T11:46:51.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:51.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:51.838907+0000) 2019-10-13T11:46:52.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:52.621+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.625636+0000) 2019-10-13T11:46:52.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:52.625+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.626199+0000) 2019-10-13T11:46:52.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:52.625+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.626528+0000) 2019-10-13T11:46:52.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:52.625+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.627011+0000) 2019-10-13T11:46:52.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:52.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.629444+0000) 2019-10-13T11:46:52.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:52.625+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.628894+0000) 2019-10-13T11:46:52.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:52.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.630236+0000) 2019-10-13T11:46:52.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:52.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146164 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:52.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:52.632+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:52.638982+0000) 2019-10-13T11:46:53.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:53.380+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:53.383371+0000) 2019-10-13T11:46:54.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:54.161+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:54.161987+0000) 2019-10-13T11:46:54.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:54.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:54.448205+0000) 2019-10-13T11:46:57.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:46:57.628+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.635766+0000) 2019-10-13T11:46:57.637 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:57.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.636683+0000) 2019-10-13T11:46:57.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:46:57.633+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.637676+0000) 2019-10-13T11:46:57.638 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:57.637+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.637963+0000) 2019-10-13T11:46:57.639 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:57.637+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.638638+0000) 2019-10-13T11:46:57.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:57.637+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.638989+0000) 2019-10-13T11:46:57.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:46:57.637+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.639383+0000) 2019-10-13T11:46:57.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:46:57.636+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.642515+0000) 2019-10-13T11:46:57.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:46:57.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146196 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:46:57.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:46:57.817+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:57.818998+0000) 2019-10-13T11:46:58.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:46:58.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.045644+0000) 2019-10-13T11:46:58.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:46:58.065+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.067047+0000) 2019-10-13T11:46:58.085 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:46:58.085 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:58.089 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:46:58.089 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:46:58.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:46:58.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:46:58.588139+0000) 2019-10-13T11:47:01.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:01.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:01.839099+0000) 2019-10-13T11:47:02.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:02.645+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.646033+0000) 2019-10-13T11:47:02.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:02.645+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.647781+0000) 2019-10-13T11:47:02.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:02.644+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.648389+0000) 2019-10-13T11:47:02.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:02.645+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.648916+0000) 2019-10-13T11:47:02.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:02.645+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.649326+0000) 2019-10-13T11:47:02.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:02.645+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.649669+0000) 2019-10-13T11:47:02.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:02.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.649218+0000) 2019-10-13T11:47:02.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:02.648+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:02.653869+0000) 2019-10-13T11:47:02.655 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:02.653+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146224 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:03.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:03.388+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:03.390014+0000) 2019-10-13T11:47:04.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:04.161+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:04.162205+0000) 2019-10-13T11:47:04.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:04.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:04.448422+0000) 2019-10-13T11:47:07.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:07.652+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.657013+0000) 2019-10-13T11:47:07.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:07.653+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.657509+0000) 2019-10-13T11:47:07.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:07.657+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.659867+0000) 2019-10-13T11:47:07.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:07.657+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.660892+0000) 2019-10-13T11:47:07.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:07.661+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.662449+0000) 2019-10-13T11:47:07.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:07.661+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.662632+0000) 2019-10-13T11:47:07.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:07.661+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.664131+0000) 2019-10-13T11:47:07.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:07.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:07.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:07.660+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.665575+0000) 2019-10-13T11:47:07.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:07.817+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:07.819197+0000) 2019-10-13T11:47:08.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:08.045+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.045853+0000) 2019-10-13T11:47:08.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:08.065+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.067240+0000) 2019-10-13T11:47:08.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:08.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:08.588323+0000) 2019-10-13T11:47:11.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:11.832+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:11.839292+0000) 2019-10-13T11:47:12.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:12.665+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.668845+0000) 2019-10-13T11:47:12.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:12.665+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.669494+0000) 2019-10-13T11:47:12.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:12.669+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.669922+0000) 2019-10-13T11:47:12.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:12.669+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.673367+0000) 2019-10-13T11:47:12.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:12.669+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.673069+0000) 2019-10-13T11:47:12.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:12.668+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.673764+0000) 2019-10-13T11:47:12.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:12.673+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.673988+0000) 2019-10-13T11:47:12.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:12.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:12.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:12.672+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:12.678611+0000) 2019-10-13T11:47:13.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:13.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:13.398994+0000) 2019-10-13T11:47:14.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:14.161+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:14.162411+0000) 2019-10-13T11:47:14.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:14.445+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:14.448627+0000) 2019-10-13T11:47:17.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:17.676+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.679591+0000) 2019-10-13T11:47:17.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:17.676+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.679721+0000) 2019-10-13T11:47:17.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:17.676+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.680429+0000) 2019-10-13T11:47:17.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:17.676+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.681677+0000) 2019-10-13T11:47:17.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:17.680+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.682017+0000) 2019-10-13T11:47:17.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:17.680+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.682674+0000) 2019-10-13T11:47:17.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:17.680+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.683295+0000) 2019-10-13T11:47:17.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:17.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146314 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:17.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:17.680+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.687700+0000) 2019-10-13T11:47:17.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:17.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:17.819418+0000) 2019-10-13T11:47:18.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:18.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.046027+0000) 2019-10-13T11:47:18.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:18.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.067412+0000) 2019-10-13T11:47:18.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:18.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:18.588505+0000) 2019-10-13T11:47:21.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:21.836+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:21.839478+0000) 2019-10-13T11:47:22.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:22.684+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.689325+0000) 2019-10-13T11:47:22.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:22.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.690113+0000) 2019-10-13T11:47:22.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:22.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.690739+0000) 2019-10-13T11:47:22.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:22.688+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.692806+0000) 2019-10-13T11:47:22.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:22.692+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.694854+0000) 2019-10-13T11:47:22.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:22.692+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.694372+0000) 2019-10-13T11:47:22.696 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:22.692+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.695106+0000) 2019-10-13T11:47:22.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:22.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:22.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:22.692+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:22.697533+0000) 2019-10-13T11:47:23.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:23.404+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:23.405472+0000) 2019-10-13T11:47:24.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:24.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:24.162613+0000) 2019-10-13T11:47:24.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:24.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:24.448878+0000) 2019-10-13T11:47:27.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:27.696+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.700355+0000) 2019-10-13T11:47:27.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:27.696+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.700870+0000) 2019-10-13T11:47:27.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:27.700+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.701822+0000) 2019-10-13T11:47:27.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:27.700+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.702341+0000) 2019-10-13T11:47:27.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:27.696+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.702680+0000) 2019-10-13T11:47:27.704 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:27.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.702913+0000) 2019-10-13T11:47:27.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:27.700+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.703538+0000) 2019-10-13T11:47:27.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:27.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:27.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:27.704+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.708954+0000) 2019-10-13T11:47:27.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:27.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:27.819605+0000) 2019-10-13T11:47:28.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:28.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.046206+0000) 2019-10-13T11:47:28.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:28.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.067592+0000) 2019-10-13T11:47:28.308 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:47:28.308 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:28.312 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:47:28.312 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:28.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:28.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:28.588707+0000) 2019-10-13T11:47:31.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:31.836+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:31.839656+0000) 2019-10-13T11:47:32.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:32.708+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.709800+0000) 2019-10-13T11:47:32.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:32.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.712269+0000) 2019-10-13T11:47:32.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:32.708+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.712823+0000) 2019-10-13T11:47:32.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:32.708+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.713211+0000) 2019-10-13T11:47:32.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:32.712+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.715711+0000) 2019-10-13T11:47:32.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:32.712+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.717371+0000) 2019-10-13T11:47:32.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:32.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:32.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:32.716+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.718591+0000) 2019-10-13T11:47:32.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:32.712+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:32.719481+0000) 2019-10-13T11:47:33.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:33.408+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:33.412056+0000) 2019-10-13T11:47:34.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:34.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:34.162815+0000) 2019-10-13T11:47:34.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:34.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:34.449077+0000) 2019-10-13T11:47:37.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:37.716+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.721079+0000) 2019-10-13T11:47:37.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:37.720+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.722241+0000) 2019-10-13T11:47:37.723 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:37.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.722324+0000) 2019-10-13T11:47:37.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:37.720+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.723123+0000) 2019-10-13T11:47:37.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:37.720+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.723548+0000) 2019-10-13T11:47:37.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:37.724+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.724498+0000) 2019-10-13T11:47:37.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:37.720+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.724785+0000) 2019-10-13T11:47:37.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:37.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:37.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:37.728+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.730173+0000) 2019-10-13T11:47:37.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:37.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:37.819791+0000) 2019-10-13T11:47:38.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:38.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.046374+0000) 2019-10-13T11:47:38.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:38.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.067764+0000) 2019-10-13T11:47:38.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:38.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:38.588928+0000) 2019-10-13T11:47:41.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:41.836+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:41.839847+0000) 2019-10-13T11:47:42.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:42.728+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.732822+0000) 2019-10-13T11:47:42.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:42.728+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.733110+0000) 2019-10-13T11:47:42.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:42.728+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.732751+0000) 2019-10-13T11:47:42.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:42.732+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.734316+0000) 2019-10-13T11:47:42.735 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:42.732+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.734819+0000) 2019-10-13T11:47:42.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:42.732+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.735223+0000) 2019-10-13T11:47:42.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:42.732+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.735611+0000) 2019-10-13T11:47:42.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:42.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:42.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:42.744+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:42.744475+0000) 2019-10-13T11:47:43.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:43.415+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:43.418681+0000) 2019-10-13T11:47:44.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:44.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:44.163029+0000) 2019-10-13T11:47:44.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:44.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:44.449276+0000) 2019-10-13T11:47:47.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:47.740+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.743356+0000) 2019-10-13T11:47:47.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:47.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.744591+0000) 2019-10-13T11:47:47.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:47.740+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.745393+0000) 2019-10-13T11:47:47.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:47.744+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.747340+0000) 2019-10-13T11:47:47.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:47.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146490 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:47.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:47.747+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.749240+0000) 2019-10-13T11:47:47.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:47.744+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.749492+0000) 2019-10-13T11:47:47.751 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:47.744+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.749631+0000) 2019-10-13T11:47:47.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:47.747+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.749807+0000) 2019-10-13T11:47:47.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:47.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:47.820028+0000) 2019-10-13T11:47:48.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:48.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.046549+0000) 2019-10-13T11:47:48.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:48.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.067965+0000) 2019-10-13T11:47:48.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:48.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:48.589108+0000) 2019-10-13T11:47:51.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:51.835+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:51.840040+0000) 2019-10-13T11:47:52.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:52.751+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.754842+0000) 2019-10-13T11:47:52.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:52.752+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.756523+0000) 2019-10-13T11:47:52.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:52.752+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.756977+0000) 2019-10-13T11:47:52.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:52.752+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.757442+0000) 2019-10-13T11:47:52.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:52.756+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.757797+0000) 2019-10-13T11:47:52.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:52.756+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.758381+0000) 2019-10-13T11:47:52.759 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:52.756+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.758774+0000) 2019-10-13T11:47:52.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:52.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:52.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:52.759+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:52.760885+0000) 2019-10-13T11:47:53.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:53.423+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:53.425231+0000) 2019-10-13T11:47:54.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:54.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:54.163246+0000) 2019-10-13T11:47:54.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:54.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:54.449481+0000) 2019-10-13T11:47:57.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:47:57.759+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.763690+0000) 2019-10-13T11:47:57.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:47:57.760+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.765402+0000) 2019-10-13T11:47:57.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:57.764+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.765784+0000) 2019-10-13T11:47:57.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:57.764+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.767206+0000) 2019-10-13T11:47:57.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:57.764+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.768091+0000) 2019-10-13T11:47:57.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:47:57.764+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.768486+0000) 2019-10-13T11:47:57.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:57.764+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.769295+0000) 2019-10-13T11:47:57.771 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:47:57.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:47:57.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:47:57.771+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.772533+0000) 2019-10-13T11:47:57.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:47:57.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:57.820247+0000) 2019-10-13T11:47:58.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:47:58.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.046677+0000) 2019-10-13T11:47:58.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:47:58.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.068097+0000) 2019-10-13T11:47:58.415 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:47:58.415 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:58.419 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:47:58.419 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:47:58.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:47:58.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:47:58.589292+0000) 2019-10-13T11:48:01.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:01.839+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:01.840275+0000) 2019-10-13T11:48:02.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:02.771+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.773376+0000) 2019-10-13T11:48:02.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:02.772+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.774034+0000) 2019-10-13T11:48:02.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:02.772+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.774511+0000) 2019-10-13T11:48:02.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:02.772+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.775203+0000) 2019-10-13T11:48:02.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:02.772+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.775173+0000) 2019-10-13T11:48:02.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:02.772+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.775659+0000) 2019-10-13T11:48:02.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:02.772+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.776755+0000) 2019-10-13T11:48:02.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:02.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:02.782 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:02.779+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:02.782184+0000) 2019-10-13T11:48:03.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:03.427+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:03.431733+0000) 2019-10-13T11:48:04.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:04.160+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:04.163428+0000) 2019-10-13T11:48:04.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:04.444+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:04.449695+0000) 2019-10-13T11:48:07.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:07.780+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.783264+0000) 2019-10-13T11:48:07.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:07.780+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.783950+0000) 2019-10-13T11:48:07.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:07.783+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.784647+0000) 2019-10-13T11:48:07.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:07.780+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.785097+0000) 2019-10-13T11:48:07.786 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:07.780+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.785576+0000) 2019-10-13T11:48:07.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:07.784+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.785950+0000) 2019-10-13T11:48:07.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:07.784+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.786460+0000) 2019-10-13T11:48:07.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:07.788+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:07.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:07.791+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.792773+0000) 2019-10-13T11:48:07.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:07.816+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:07.820454+0000) 2019-10-13T11:48:08.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:08.044+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.046868+0000) 2019-10-13T11:48:08.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:08.064+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.068296+0000) 2019-10-13T11:48:08.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:08.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:08.589482+0000) 2019-10-13T11:48:11.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:11.839+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:11.840492+0000) 2019-10-13T11:48:12.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:12.791+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.794352+0000) 2019-10-13T11:48:12.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:12.791+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.794757+0000) 2019-10-13T11:48:12.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:12.791+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.795084+0000) 2019-10-13T11:48:12.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:12.791+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.795415+0000) 2019-10-13T11:48:12.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:12.791+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.795924+0000) 2019-10-13T11:48:12.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:12.791+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.797135+0000) 2019-10-13T11:48:12.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:12.795+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.799892+0000) 2019-10-13T11:48:12.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:12.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146638 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:12.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:12.799+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:12.802317+0000) 2019-10-13T11:48:13.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:13.435+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:13.438489+0000) 2019-10-13T11:48:14.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:14.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:14.163628+0000) 2019-10-13T11:48:14.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:14.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:14.449928+0000) 2019-10-13T11:48:17.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:17.799+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.805033+0000) 2019-10-13T11:48:17.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:17.803+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.805938+0000) 2019-10-13T11:48:17.807 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:17.803+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.806338+0000) 2019-10-13T11:48:17.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:17.807+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.810228+0000) 2019-10-13T11:48:17.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:17.807+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.811660+0000) 2019-10-13T11:48:17.813 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:17.807+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.812047+0000) 2019-10-13T11:48:17.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:17.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:17.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:17.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.814086+0000) 2019-10-13T11:48:17.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:17.811+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.814757+0000) 2019-10-13T11:48:17.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:17.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:17.820578+0000) 2019-10-13T11:48:18.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:18.043+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.047057+0000) 2019-10-13T11:48:18.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:18.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.068509+0000) 2019-10-13T11:48:18.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:18.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:18.589670+0000) 2019-10-13T11:48:21.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:21.839+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:21.840709+0000) 2019-10-13T11:48:22.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:22.815+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.816511+0000) 2019-10-13T11:48:22.818 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:22.811+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.817410+0000) 2019-10-13T11:48:22.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:22.815+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.818295+0000) 2019-10-13T11:48:22.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:22.815+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.819782+0000) 2019-10-13T11:48:22.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:22.815+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.821664+0000) 2019-10-13T11:48:22.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:22.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:22.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:22.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.825571+0000) 2019-10-13T11:48:22.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:22.819+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.825438+0000) 2019-10-13T11:48:22.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:22.827+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:22.830417+0000) 2019-10-13T11:48:23.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:23.443+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:23.445049+0000) 2019-10-13T11:48:24.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:24.163+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:24.163870+0000) 2019-10-13T11:48:24.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:24.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:24.450128+0000) 2019-10-13T11:48:27.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:27.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.820735+0000) 2019-10-13T11:48:27.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:27.827+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.830027+0000) 2019-10-13T11:48:27.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:27.827+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.830368+0000) 2019-10-13T11:48:27.832 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:27.827+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.831379+0000) 2019-10-13T11:48:27.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:27.827+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.831806+0000) 2019-10-13T11:48:27.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:27.827+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.832662+0000) 2019-10-13T11:48:27.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:27.827+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.833321+0000) 2019-10-13T11:48:27.835 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:27.831+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.834925+0000) 2019-10-13T11:48:27.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:27.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:27.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:27.835+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:27.838517+0000) 2019-10-13T11:48:28.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:28.043+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.047239+0000) 2019-10-13T11:48:28.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:28.063+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.068683+0000) 2019-10-13T11:48:28.527 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:48:28.527 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:28.531 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:48:28.531 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:28.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:28.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:28.589905+0000) 2019-10-13T11:48:31.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:31.839+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:31.840923+0000) 2019-10-13T11:48:32.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:32.835+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.838381+0000) 2019-10-13T11:48:32.840 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:32.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.839274+0000) 2019-10-13T11:48:32.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:32.839+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.840792+0000) 2019-10-13T11:48:32.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:32.835+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.841054+0000) 2019-10-13T11:48:32.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:32.835+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.841361+0000) 2019-10-13T11:48:32.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:32.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.841959+0000) 2019-10-13T11:48:32.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:32.839+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.842305+0000) 2019-10-13T11:48:32.846 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:32.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:32.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:32.843+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:32.847088+0000) 2019-10-13T11:48:33.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:33.451+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:33.451541+0000) 2019-10-13T11:48:34.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:34.163+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:34.164066+0000) 2019-10-13T11:48:34.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:34.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:34.450327+0000) 2019-10-13T11:48:37.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:37.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.820970+0000) 2019-10-13T11:48:37.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:37.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.849012+0000) 2019-10-13T11:48:37.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:37.843+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.849645+0000) 2019-10-13T11:48:37.852 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:37.847+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.851457+0000) 2019-10-13T11:48:37.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:37.851+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.854724+0000) 2019-10-13T11:48:37.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:37.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:37.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:37.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.856784+0000) 2019-10-13T11:48:37.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:37.851+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.857168+0000) 2019-10-13T11:48:37.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:37.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.857190+0000) 2019-10-13T11:48:37.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:37.859+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:37.863981+0000) 2019-10-13T11:48:38.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:38.043+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.047423+0000) 2019-10-13T11:48:38.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:38.063+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.068852+0000) 2019-10-13T11:48:38.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:38.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:38.590103+0000) 2019-10-13T11:48:41.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:41.839+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:41.841119+0000) 2019-10-13T11:48:42.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:42.855+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.859285+0000) 2019-10-13T11:48:42.861 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:42.855+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.861096+0000) 2019-10-13T11:48:42.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:42.859+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.863604+0000) 2019-10-13T11:48:42.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:42.859+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.864059+0000) 2019-10-13T11:48:42.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:42.859+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.864511+0000) 2019-10-13T11:48:42.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:42.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.864964+0000) 2019-10-13T11:48:42.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:42.859+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.865593+0000) 2019-10-13T11:48:42.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:42.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:42.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:42.867+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:42.870331+0000) 2019-10-13T11:48:43.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:43.455+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:43.458053+0000) 2019-10-13T11:48:44.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:44.163+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:44.164263+0000) 2019-10-13T11:48:44.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:44.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:44.450532+0000) 2019-10-13T11:48:47.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:47.815+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.821197+0000) 2019-10-13T11:48:47.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:47.867+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.871790+0000) 2019-10-13T11:48:47.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:47.867+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.872025+0000) 2019-10-13T11:48:47.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:47.867+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.872439+0000) 2019-10-13T11:48:47.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:47.867+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.873003+0000) 2019-10-13T11:48:47.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:47.867+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.873686+0000) 2019-10-13T11:48:47.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:47.871+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.874314+0000) 2019-10-13T11:48:47.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:47.871+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.875200+0000) 2019-10-13T11:48:47.878 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:47.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146842 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:47.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:47.875+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:47.877945+0000) 2019-10-13T11:48:48.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:48.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.047619+0000) 2019-10-13T11:48:48.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:48.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.069022+0000) 2019-10-13T11:48:48.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:48.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:48.590299+0000) 2019-10-13T11:48:51.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:51.838+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:51.841330+0000) 2019-10-13T11:48:52.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:52.875+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.880183+0000) 2019-10-13T11:48:52.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:52.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.882323+0000) 2019-10-13T11:48:52.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:52.879+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.882799+0000) 2019-10-13T11:48:52.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:52.879+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.883376+0000) 2019-10-13T11:48:52.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:52.879+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.884342+0000) 2019-10-13T11:48:52.886 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:52.879+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.885385+0000) 2019-10-13T11:48:52.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:52.883+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.887579+0000) 2019-10-13T11:48:52.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:52.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:52.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:52.890+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:52.893077+0000) 2019-10-13T11:48:53.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:53.462+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:53.464590+0000) 2019-10-13T11:48:54.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:54.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:54.164487+0000) 2019-10-13T11:48:54.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:54.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:54.450719+0000) 2019-10-13T11:48:57.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:57.819+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.821406+0000) 2019-10-13T11:48:57.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:48:57.887+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.892065+0000) 2019-10-13T11:48:57.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:57.887+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.893425+0000) 2019-10-13T11:48:57.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:57.891+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.893922+0000) 2019-10-13T11:48:57.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:48:57.894+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.896505+0000) 2019-10-13T11:48:57.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:48:57.894+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.897742+0000) 2019-10-13T11:48:57.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:48:57.895+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.898422+0000) 2019-10-13T11:48:57.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:48:57.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:48:57.900 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:57.895+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.899224+0000) 2019-10-13T11:48:57.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:48:57.895+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:57.899388+0000) 2019-10-13T11:48:58.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:48:58.043+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.047795+0000) 2019-10-13T11:48:58.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:48:58.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.069195+0000) 2019-10-13T11:48:58.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:48:58.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:48:58.590445+0000) 2019-10-13T11:48:58.648 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:48:58.648 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:48:58.652 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:48:58.652 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:01.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:01.838+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:01.841524+0000) 2019-10-13T11:49:02.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:02.898+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.902858+0000) 2019-10-13T11:49:02.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:02.899+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.903482+0000) 2019-10-13T11:49:02.904 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:02.899+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.903898+0000) 2019-10-13T11:49:02.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:02.899+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.904631+0000) 2019-10-13T11:49:02.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:02.903+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.905734+0000) 2019-10-13T11:49:02.906 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:02.899+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.905357+0000) 2019-10-13T11:49:02.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:02.903+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.907958+0000) 2019-10-13T11:49:02.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:02.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:02.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:02.910+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:02.912483+0000) 2019-10-13T11:49:03.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:03.470+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:03.471218+0000) 2019-10-13T11:49:04.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:04.159+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:04.164691+0000) 2019-10-13T11:49:04.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:04.447+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:04.450951+0000) 2019-10-13T11:49:07.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:07.818+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.821613+0000) 2019-10-13T11:49:07.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:07.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.912773+0000) 2019-10-13T11:49:07.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:07.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.913406+0000) 2019-10-13T11:49:07.914 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:07.906+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.913646+0000) 2019-10-13T11:49:07.916 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:07.910+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.915649+0000) 2019-10-13T11:49:07.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:07.910+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.915853+0000) 2019-10-13T11:49:07.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:07.914+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.918886+0000) 2019-10-13T11:49:07.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:07.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.919542+0000) 2019-10-13T11:49:07.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:07.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146958 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:07.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:07.922+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:07.926442+0000) 2019-10-13T11:49:08.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:08.046+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.047976+0000) 2019-10-13T11:49:08.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:08.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.069374+0000) 2019-10-13T11:49:08.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:08.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:08.590643+0000) 2019-10-13T11:49:11.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:11.838+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:11.841712+0000) 2019-10-13T11:49:12.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:12.922+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.924246+0000) 2019-10-13T11:49:12.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:12.918+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.925006+0000) 2019-10-13T11:49:12.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:12.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.925165+0000) 2019-10-13T11:49:12.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:12.918+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.925501+0000) 2019-10-13T11:49:12.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:12.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.927444+0000) 2019-10-13T11:49:12.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:12.922+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.927910+0000) 2019-10-13T11:49:12.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:12.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.926673+0000) 2019-10-13T11:49:12.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:12.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:12.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:12.930+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:12.932852+0000) 2019-10-13T11:49:13.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:13.474+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:13.477642+0000) 2019-10-13T11:49:14.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:14.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.164923+0000) 2019-10-13T11:49:14.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:14.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:14.451199+0000) 2019-10-13T11:49:17.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:17.818+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.821815+0000) 2019-10-13T11:49:17.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:17.930+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.935604+0000) 2019-10-13T11:49:17.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:17.930+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.936377+0000) 2019-10-13T11:49:17.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:17.930+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.936756+0000) 2019-10-13T11:49:17.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:17.934+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.937039+0000) 2019-10-13T11:49:17.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:17.930+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.937439+0000) 2019-10-13T11:49:17.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:17.934+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.938249+0000) 2019-10-13T11:49:17.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:17.934+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.938589+0000) 2019-10-13T11:49:17.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:17.938+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:17.940964+0000) 2019-10-13T11:49:17.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:17.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:18.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:18.046+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.048163+0000) 2019-10-13T11:49:18.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:18.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.069550+0000) 2019-10-13T11:49:18.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:18.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:18.590819+0000) 2019-10-13T11:49:21.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:21.838+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:21.841909+0000) 2019-10-13T11:49:22.947 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:22.938+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.945418+0000) 2019-10-13T11:49:22.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:22.942+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.946527+0000) 2019-10-13T11:49:22.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:22.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.947030+0000) 2019-10-13T11:49:22.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:22.942+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.947244+0000) 2019-10-13T11:49:22.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:22.942+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.947709+0000) 2019-10-13T11:49:22.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:22.946+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.948584+0000) 2019-10-13T11:49:22.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:22.942+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.948958+0000) 2019-10-13T11:49:22.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:22.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:22.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:22.950+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:22.954645+0000) 2019-10-13T11:49:23.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:23.482+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:23.484359+0000) 2019-10-13T11:49:24.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:24.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.165170+0000) 2019-10-13T11:49:24.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:24.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:24.451402+0000) 2019-10-13T11:49:27.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:27.818+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.822055+0000) 2019-10-13T11:49:27.957 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:27.950+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.955934+0000) 2019-10-13T11:49:27.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:27.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.957219+0000) 2019-10-13T11:49:27.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:27.954+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.957649+0000) 2019-10-13T11:49:27.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:27.954+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.958407+0000) 2019-10-13T11:49:27.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:27.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.958613+0000) 2019-10-13T11:49:27.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:27.954+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.958855+0000) 2019-10-13T11:49:27.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:27.954+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.959338+0000) 2019-10-13T11:49:27.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:27.958+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:27.963116+0000) 2019-10-13T11:49:27.964 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:27.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:28.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:28.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.048371+0000) 2019-10-13T11:49:28.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:28.062+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.069700+0000) 2019-10-13T11:49:28.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:28.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:28.590999+0000) 2019-10-13T11:49:28.686 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:49:28.686 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:28.689 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:49:28.690 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:31.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:31.838+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:31.842128+0000) 2019-10-13T11:49:32.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:32.962+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.965671+0000) 2019-10-13T11:49:32.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:32.962+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.966603+0000) 2019-10-13T11:49:32.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:32.962+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.967552+0000) 2019-10-13T11:49:32.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:32.962+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.968224+0000) 2019-10-13T11:49:32.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:32.962+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.968266+0000) 2019-10-13T11:49:32.970 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:32.962+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.968009+0000) 2019-10-13T11:49:32.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:32.962+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.968843+0000) 2019-10-13T11:49:32.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:32.970+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147103 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:32.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:32.970+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:32.975756+0000) 2019-10-13T11:49:33.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:33.490+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:33.490879+0000) 2019-10-13T11:49:34.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:34.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.165387+0000) 2019-10-13T11:49:34.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:34.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:34.451606+0000) 2019-10-13T11:49:37.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:37.818+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.822253+0000) 2019-10-13T11:49:37.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:37.970+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.977474+0000) 2019-10-13T11:49:37.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:37.974+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.978244+0000) 2019-10-13T11:49:37.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:37.974+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.978763+0000) 2019-10-13T11:49:37.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:37.974+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.979405+0000) 2019-10-13T11:49:37.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:37.974+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.979611+0000) 2019-10-13T11:49:37.981 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:37.974+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.980390+0000) 2019-10-13T11:49:37.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:37.974+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.980807+0000) 2019-10-13T11:49:37.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:37.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:37.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:37.982+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:37.985442+0000) 2019-10-13T11:49:38.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:38.046+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.048558+0000) 2019-10-13T11:49:38.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:38.066+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.069907+0000) 2019-10-13T11:49:38.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:38.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:38.591162+0000) 2019-10-13T11:49:41.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:41.842+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:41.842357+0000) 2019-10-13T11:49:42.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:42.982+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.987429+0000) 2019-10-13T11:49:42.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:42.982+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.988119+0000) 2019-10-13T11:49:42.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:42.982+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.989250+0000) 2019-10-13T11:49:42.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:42.982+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.989657+0000) 2019-10-13T11:49:42.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:42.986+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.990230+0000) 2019-10-13T11:49:42.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:42.986+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.990540+0000) 2019-10-13T11:49:42.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:42.986+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.991034+0000) 2019-10-13T11:49:42.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:42.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:42.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:42.994+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:42.998651+0000) 2019-10-13T11:49:43.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:43.494+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:43.497298+0000) 2019-10-13T11:49:44.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:44.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.165621+0000) 2019-10-13T11:49:44.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:44.446+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:44.451833+0000) 2019-10-13T11:49:47.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:47.822+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:47.822451+0000) 2019-10-13T11:49:48.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:47.994+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:47.999219+0000) 2019-10-13T11:49:48.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:47.994+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:47.999470+0000) 2019-10-13T11:49:48.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:47.994+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:47.999943+0000) 2019-10-13T11:49:48.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:47.994+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.000603+0000) 2019-10-13T11:49:48.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:47.994+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.001539+0000) 2019-10-13T11:49:48.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:47.998+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.001299+0000) 2019-10-13T11:49:48.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:47.998+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.002420+0000) 2019-10-13T11:49:48.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:48.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:48.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:48.002+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.006734+0000) 2019-10-13T11:49:48.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:48.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.048771+0000) 2019-10-13T11:49:48.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:48.066+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.070052+0000) 2019-10-13T11:49:48.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:48.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:48.591314+0000) 2019-10-13T11:49:51.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:51.842+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:51.842550+0000) 2019-10-13T11:49:53.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:53.006+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.008753+0000) 2019-10-13T11:49:53.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:53.006+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.010567+0000) 2019-10-13T11:49:53.011 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:53.006+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.010942+0000) 2019-10-13T11:49:53.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:53.006+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.012321+0000) 2019-10-13T11:49:53.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:53.006+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.012541+0000) 2019-10-13T11:49:53.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:53.010+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.012912+0000) 2019-10-13T11:49:53.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:53.006+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.013217+0000) 2019-10-13T11:49:53.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:53.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:53.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:53.014+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.019984+0000) 2019-10-13T11:49:53.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:53.501+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:53.503584+0000) 2019-10-13T11:49:54.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:54.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.165861+0000) 2019-10-13T11:49:54.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:54.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:54.452057+0000) 2019-10-13T11:49:57.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:57.822+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:57.822664+0000) 2019-10-13T11:49:58.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:58.014+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.019864+0000) 2019-10-13T11:49:58.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:49:58.017+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.021667+0000) 2019-10-13T11:49:58.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:58.018+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.022130+0000) 2019-10-13T11:49:58.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:49:58.018+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.022561+0000) 2019-10-13T11:49:58.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:58.018+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.022877+0000) 2019-10-13T11:49:58.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:49:58.018+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.023168+0000) 2019-10-13T11:49:58.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:49:58.018+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.023825+0000) 2019-10-13T11:49:58.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:49:58.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147249 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:49:58.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:49:58.029+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.032135+0000) 2019-10-13T11:49:58.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:49:58.042+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.048935+0000) 2019-10-13T11:49:58.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:49:58.066+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.070242+0000) 2019-10-13T11:49:58.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:49:58.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:49:58.591510+0000) 2019-10-13T11:49:58.800 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:49:58.800 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:49:58.804 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:49:58.804 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:01.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:01.841+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:01.842746+0000) 2019-10-13T11:50:03.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:03.025+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.029807+0000) 2019-10-13T11:50:03.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:03.025+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.031631+0000) 2019-10-13T11:50:03.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:03.025+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.032166+0000) 2019-10-13T11:50:03.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:03.025+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.032525+0000) 2019-10-13T11:50:03.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:03.025+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.032858+0000) 2019-10-13T11:50:03.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:03.025+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.033480+0000) 2019-10-13T11:50:03.037 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:03.029+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.036295+0000) 2019-10-13T11:50:03.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:03.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:03.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:03.033+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.039450+0000) 2019-10-13T11:50:03.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:03.505+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:03.510084+0000) 2019-10-13T11:50:04.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:04.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.166074+0000) 2019-10-13T11:50:04.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:04.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:04.452260+0000) 2019-10-13T11:50:07.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:07.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:07.822897+0000) 2019-10-13T11:50:08.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:08.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.042362+0000) 2019-10-13T11:50:08.044 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:08.037+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.043970+0000) 2019-10-13T11:50:08.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:08.037+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.044478+0000) 2019-10-13T11:50:08.047 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:08.041+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.046305+0000) 2019-10-13T11:50:08.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:08.041+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.046577+0000) 2019-10-13T11:50:08.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:08.041+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.048305+0000) 2019-10-13T11:50:08.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:08.041+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.048473+0000) 2019-10-13T11:50:08.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:08.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147306 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:08.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:08.041+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.049087+0000) 2019-10-13T11:50:08.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:08.049+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.052622+0000) 2019-10-13T11:50:08.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:08.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.070448+0000) 2019-10-13T11:50:08.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:08.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:08.591684+0000) 2019-10-13T11:50:11.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:11.841+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:11.842939+0000) 2019-10-13T11:50:13.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:13.045+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.052280+0000) 2019-10-13T11:50:13.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:13.045+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.052822+0000) 2019-10-13T11:50:13.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:13.049+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.054456+0000) 2019-10-13T11:50:13.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:13.049+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.054695+0000) 2019-10-13T11:50:13.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:13.049+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.056249+0000) 2019-10-13T11:50:13.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:13.053+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.058068+0000) 2019-10-13T11:50:13.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:13.053+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.058369+0000) 2019-10-13T11:50:13.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:13.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:13.065 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:13.061+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.064239+0000) 2019-10-13T11:50:13.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:13.513+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:13.516578+0000) 2019-10-13T11:50:14.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:14.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.166290+0000) 2019-10-13T11:50:14.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:14.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:14.452492+0000) 2019-10-13T11:50:17.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:17.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:17.823119+0000) 2019-10-13T11:50:18.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:18.045+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.049294+0000) 2019-10-13T11:50:18.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:18.061+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.062872+0000) 2019-10-13T11:50:18.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:18.061+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.064279+0000) 2019-10-13T11:50:18.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:18.061+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.066004+0000) 2019-10-13T11:50:18.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:18.065+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.066109+0000) 2019-10-13T11:50:18.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:18.065+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.066415+0000) 2019-10-13T11:50:18.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:18.065+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.066825+0000) 2019-10-13T11:50:18.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:18.065+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.068835+0000) 2019-10-13T11:50:18.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:18.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:18.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:18.065+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.070628+0000) 2019-10-13T11:50:18.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:18.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.070600+0000) 2019-10-13T11:50:18.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:18.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:18.591868+0000) 2019-10-13T11:50:21.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:21.841+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:21.843134+0000) 2019-10-13T11:50:23.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:23.069+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.073057+0000) 2019-10-13T11:50:23.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:23.069+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.073328+0000) 2019-10-13T11:50:23.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:23.073+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.073723+0000) 2019-10-13T11:50:23.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:23.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.074146+0000) 2019-10-13T11:50:23.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:23.073+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.074549+0000) 2019-10-13T11:50:23.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:23.073+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.075262+0000) 2019-10-13T11:50:23.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:23.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.075023+0000) 2019-10-13T11:50:23.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:23.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:23.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:23.077+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.082671+0000) 2019-10-13T11:50:23.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:23.517+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:23.522945+0000) 2019-10-13T11:50:24.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:24.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.166510+0000) 2019-10-13T11:50:24.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:24.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:24.452724+0000) 2019-10-13T11:50:27.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:27.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:27.823326+0000) 2019-10-13T11:50:28.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:28.045+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.049484+0000) 2019-10-13T11:50:28.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:28.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.070788+0000) 2019-10-13T11:50:28.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:28.081+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.083879+0000) 2019-10-13T11:50:28.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:28.081+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.084677+0000) 2019-10-13T11:50:28.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:28.081+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.085291+0000) 2019-10-13T11:50:28.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:28.081+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.085881+0000) 2019-10-13T11:50:28.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:28.085+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.086347+0000) 2019-10-13T11:50:28.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:28.085+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.086731+0000) 2019-10-13T11:50:28.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:28.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.087144+0000) 2019-10-13T11:50:28.092 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:28.085+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.091615+0000) 2019-10-13T11:50:28.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:28.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:28.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:28.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:28.592043+0000) 2019-10-13T11:50:28.922 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:50:28.922 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:28.925 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:50:28.926 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:31.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:31.841+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:31.843331+0000) 2019-10-13T11:50:33.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:33.093+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.095219+0000) 2019-10-13T11:50:33.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:33.093+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.096497+0000) 2019-10-13T11:50:33.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:33.093+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.097191+0000) 2019-10-13T11:50:33.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:33.093+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.097571+0000) 2019-10-13T11:50:33.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:33.093+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.098509+0000) 2019-10-13T11:50:33.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:33.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.098564+0000) 2019-10-13T11:50:33.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:33.097+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.098808+0000) 2019-10-13T11:50:33.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:33.097+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.102007+0000) 2019-10-13T11:50:33.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:33.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:33.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:33.529+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:33.529453+0000) 2019-10-13T11:50:34.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:34.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.166675+0000) 2019-10-13T11:50:34.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:34.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:34.452923+0000) 2019-10-13T11:50:37.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:37.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:37.823525+0000) 2019-10-13T11:50:38.050 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:38.045+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.049695+0000) 2019-10-13T11:50:38.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:38.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.070985+0000) 2019-10-13T11:50:38.106 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:38.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.105003+0000) 2019-10-13T11:50:38.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:38.105+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.106063+0000) 2019-10-13T11:50:38.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:38.105+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.106880+0000) 2019-10-13T11:50:38.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:38.105+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.108883+0000) 2019-10-13T11:50:38.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:38.109+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.110838+0000) 2019-10-13T11:50:38.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:38.105+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.111071+0000) 2019-10-13T11:50:38.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:38.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.111793+0000) 2019-10-13T11:50:38.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:38.109+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.113694+0000) 2019-10-13T11:50:38.115 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:38.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:38.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:38.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:38.592231+0000) 2019-10-13T11:50:41.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:41.841+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:41.843562+0000) 2019-10-13T11:50:43.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:43.113+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.117655+0000) 2019-10-13T11:50:43.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:43.117+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.120100+0000) 2019-10-13T11:50:43.121 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:43.117+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.120662+0000) 2019-10-13T11:50:43.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:43.117+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.121477+0000) 2019-10-13T11:50:43.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:43.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.122131+0000) 2019-10-13T11:50:43.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:43.121+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.121887+0000) 2019-10-13T11:50:43.123 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:43.117+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.122403+0000) 2019-10-13T11:50:43.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:43.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:43.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:43.125+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.128857+0000) 2019-10-13T11:50:43.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:43.533+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:43.536046+0000) 2019-10-13T11:50:44.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:44.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.166911+0000) 2019-10-13T11:50:44.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:44.449+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:44.453081+0000) 2019-10-13T11:50:47.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:47.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:47.823735+0000) 2019-10-13T11:50:48.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:48.049+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.049897+0000) 2019-10-13T11:50:48.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:48.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.071214+0000) 2019-10-13T11:50:48.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:48.125+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.128566+0000) 2019-10-13T11:50:48.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:48.125+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.128995+0000) 2019-10-13T11:50:48.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:48.125+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.131053+0000) 2019-10-13T11:50:48.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:48.129+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.131406+0000) 2019-10-13T11:50:48.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:48.129+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.133361+0000) 2019-10-13T11:50:48.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:48.133+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.134544+0000) 2019-10-13T11:50:48.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:48.133+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.134847+0000) 2019-10-13T11:50:48.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:48.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147541 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:48.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:48.133+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.139969+0000) 2019-10-13T11:50:48.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:48.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:48.592412+0000) 2019-10-13T11:50:51.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:51.837+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:51.843756+0000) 2019-10-13T11:50:53.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:53.137+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.139832+0000) 2019-10-13T11:50:53.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:53.137+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.140384+0000) 2019-10-13T11:50:53.142 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:53.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.140830+0000) 2019-10-13T11:50:53.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:53.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.141930+0000) 2019-10-13T11:50:53.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:53.141+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.142173+0000) 2019-10-13T11:50:53.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:53.137+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.143891+0000) 2019-10-13T11:50:53.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:53.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:53.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:53.141+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.147919+0000) 2019-10-13T11:50:53.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:53.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.148296+0000) 2019-10-13T11:50:53.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:53.537+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:53.542654+0000) 2019-10-13T11:50:54.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:54.165+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.167120+0000) 2019-10-13T11:50:54.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:54.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:54.453318+0000) 2019-10-13T11:50:57.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:57.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:57.823959+0000) 2019-10-13T11:50:58.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:58.049+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.050084+0000) 2019-10-13T11:50:58.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:58.069+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.071409+0000) 2019-10-13T11:50:58.151 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:50:58.149+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.150414+0000) 2019-10-13T11:50:58.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:58.149+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.151299+0000) 2019-10-13T11:50:58.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:50:58.149+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.152255+0000) 2019-10-13T11:50:58.155 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:50:58.153+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.154293+0000) 2019-10-13T11:50:58.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:50:58.149+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.154204+0000) 2019-10-13T11:50:58.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:50:58.153+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.155816+0000) 2019-10-13T11:50:58.158 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:50:58.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:50:58.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:50:58.157+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.158078+0000) 2019-10-13T11:50:58.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:50:58.153+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.158151+0000) 2019-10-13T11:50:58.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:50:58.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:50:58.592588+0000) 2019-10-13T11:50:59.071 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:50:59.071 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:50:59.075 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:50:59.075 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:01.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:01.840+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:01.843952+0000) 2019-10-13T11:51:03.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:03.156+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.161073+0000) 2019-10-13T11:51:03.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:03.156+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.161588+0000) 2019-10-13T11:51:03.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:03.160+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.163366+0000) 2019-10-13T11:51:03.165 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:03.160+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.163679+0000) 2019-10-13T11:51:03.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:03.160+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.164264+0000) 2019-10-13T11:51:03.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:03.160+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.165132+0000) 2019-10-13T11:51:03.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:03.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.166152+0000) 2019-10-13T11:51:03.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:03.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:03.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:03.164+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.171801+0000) 2019-10-13T11:51:03.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:03.552+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:03.554239+0000) 2019-10-13T11:51:04.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:04.164+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.167324+0000) 2019-10-13T11:51:04.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:04.448+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:04.453520+0000) 2019-10-13T11:51:07.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:07.820+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:07.824161+0000) 2019-10-13T11:51:08.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:08.048+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.050301+0000) 2019-10-13T11:51:08.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:08.068+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.071606+0000) 2019-10-13T11:51:08.173 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:08.168+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.172185+0000) 2019-10-13T11:51:08.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:08.168+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.172972+0000) 2019-10-13T11:51:08.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:08.168+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.173392+0000) 2019-10-13T11:51:08.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:08.172+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.173956+0000) 2019-10-13T11:51:08.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:08.172+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.174807+0000) 2019-10-13T11:51:08.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:08.172+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.175032+0000) 2019-10-13T11:51:08.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:08.172+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.175417+0000) 2019-10-13T11:51:08.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:08.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:08.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:08.176+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.181135+0000) 2019-10-13T11:51:08.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:08.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:08.592770+0000) 2019-10-13T11:51:11.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:11.840+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:11.844153+0000) 2019-10-13T11:51:13.183 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:13.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.182289+0000) 2019-10-13T11:51:13.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:13.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.183700+0000) 2019-10-13T11:51:13.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:13.180+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.184501+0000) 2019-10-13T11:51:13.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:13.180+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.185048+0000) 2019-10-13T11:51:13.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:13.180+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.185585+0000) 2019-10-13T11:51:13.187 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:13.184+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.186423+0000) 2019-10-13T11:51:13.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:13.184+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.186674+0000) 2019-10-13T11:51:13.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:13.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:13.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:13.184+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.190493+0000) 2019-10-13T11:51:13.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:13.556+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:13.560915+0000) 2019-10-13T11:51:14.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:14.164+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.167555+0000) 2019-10-13T11:51:14.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:14.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:14.453722+0000) 2019-10-13T11:51:17.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:17.820+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:17.824386+0000) 2019-10-13T11:51:18.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:18.048+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.050498+0000) 2019-10-13T11:51:18.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:18.068+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.071810+0000) 2019-10-13T11:51:18.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:18.192+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.194382+0000) 2019-10-13T11:51:18.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:18.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.194695+0000) 2019-10-13T11:51:18.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:18.192+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.195076+0000) 2019-10-13T11:51:18.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:18.192+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.195445+0000) 2019-10-13T11:51:18.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:18.188+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.194266+0000) 2019-10-13T11:51:18.198 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:18.192+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.196679+0000) 2019-10-13T11:51:18.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:18.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.196528+0000) 2019-10-13T11:51:18.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:18.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:18.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:18.196+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.203330+0000) 2019-10-13T11:51:18.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:18.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:18.592978+0000) 2019-10-13T11:51:21.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:21.840+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:21.844383+0000) 2019-10-13T11:51:23.206 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:23.200+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.205339+0000) 2019-10-13T11:51:23.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:23.204+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.207160+0000) 2019-10-13T11:51:23.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:23.204+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.207719+0000) 2019-10-13T11:51:23.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:23.204+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.208014+0000) 2019-10-13T11:51:23.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:23.204+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.208457+0000) 2019-10-13T11:51:23.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:23.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.209085+0000) 2019-10-13T11:51:23.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:23.208+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.209813+0000) 2019-10-13T11:51:23.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:23.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:23.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:23.212+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.216169+0000) 2019-10-13T11:51:23.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:23.564+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:23.567330+0000) 2019-10-13T11:51:24.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:24.164+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.167691+0000) 2019-10-13T11:51:24.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:24.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:24.453952+0000) 2019-10-13T11:51:27.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:27.820+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:27.824603+0000) 2019-10-13T11:51:28.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:28.048+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.050696+0000) 2019-10-13T11:51:28.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:28.068+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.072009+0000) 2019-10-13T11:51:28.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:28.212+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.216776+0000) 2019-10-13T11:51:28.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:28.212+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.217124+0000) 2019-10-13T11:51:28.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:28.212+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.217456+0000) 2019-10-13T11:51:28.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:28.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.217772+0000) 2019-10-13T11:51:28.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:28.216+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.218336+0000) 2019-10-13T11:51:28.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:28.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.218948+0000) 2019-10-13T11:51:28.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:28.216+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.219666+0000) 2019-10-13T11:51:28.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:28.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:28.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:28.220+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.225822+0000) 2019-10-13T11:51:28.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:28.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:28.593159+0000) 2019-10-13T11:51:29.178 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:51:29.178 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:29.182 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:51:29.182 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:31.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:31.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:31.844612+0000) 2019-10-13T11:51:33.227 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:33.224+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.226288+0000) 2019-10-13T11:51:33.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:33.224+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.228560+0000) 2019-10-13T11:51:33.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:33.224+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.229207+0000) 2019-10-13T11:51:33.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:33.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:33.236 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:33.228+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.234236+0000) 2019-10-13T11:51:33.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:33.232+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.234596+0000) 2019-10-13T11:51:33.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:33.232+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.235095+0000) 2019-10-13T11:51:33.237 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:33.232+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.234865+0000) 2019-10-13T11:51:33.238 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:33.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.235876+0000) 2019-10-13T11:51:33.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:33.568+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:33.573944+0000) 2019-10-13T11:51:34.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:34.164+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.167869+0000) 2019-10-13T11:51:34.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:34.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:34.454158+0000) 2019-10-13T11:51:37.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:37.820+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:37.824828+0000) 2019-10-13T11:51:38.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:38.048+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.050911+0000) 2019-10-13T11:51:38.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:38.068+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.072211+0000) 2019-10-13T11:51:38.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:38.236+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.238498+0000) 2019-10-13T11:51:38.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:38.236+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.240302+0000) 2019-10-13T11:51:38.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:38.236+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.240702+0000) 2019-10-13T11:51:38.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:38.236+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.241054+0000) 2019-10-13T11:51:38.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:38.240+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.241858+0000) 2019-10-13T11:51:38.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:38.240+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.242627+0000) 2019-10-13T11:51:38.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:38.240+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.243523+0000) 2019-10-13T11:51:38.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:38.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:38.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:38.244+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.248522+0000) 2019-10-13T11:51:38.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:38.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:38.593350+0000) 2019-10-13T11:51:41.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:41.840+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:41.844837+0000) 2019-10-13T11:51:43.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:43.244+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.249288+0000) 2019-10-13T11:51:43.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:43.244+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.249624+0000) 2019-10-13T11:51:43.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:43.248+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.250225+0000) 2019-10-13T11:51:43.251 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:43.248+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.250542+0000) 2019-10-13T11:51:43.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:43.244+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.251242+0000) 2019-10-13T11:51:43.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:43.248+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.251580+0000) 2019-10-13T11:51:43.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:43.248+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.252005+0000) 2019-10-13T11:51:43.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:43.252+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:43.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:43.252+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.256620+0000) 2019-10-13T11:51:43.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:43.580+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:43.580565+0000) 2019-10-13T11:51:44.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:44.164+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.168046+0000) 2019-10-13T11:51:44.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:44.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:44.454309+0000) 2019-10-13T11:51:47.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:47.824+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:47.825026+0000) 2019-10-13T11:51:48.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:48.048+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.051119+0000) 2019-10-13T11:51:48.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:48.068+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.072427+0000) 2019-10-13T11:51:48.261 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:48.256+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.259933+0000) 2019-10-13T11:51:48.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:48.260+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.262540+0000) 2019-10-13T11:51:48.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:48.260+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.262645+0000) 2019-10-13T11:51:48.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:48.260+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.263066+0000) 2019-10-13T11:51:48.264 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:48.256+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.263809+0000) 2019-10-13T11:51:48.265 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:48.260+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.264316+0000) 2019-10-13T11:51:48.265 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:48.260+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.265211+0000) 2019-10-13T11:51:48.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:48.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:48.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:48.264+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.268827+0000) 2019-10-13T11:51:48.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:48.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:48.593527+0000) 2019-10-13T11:51:51.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:51.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:51.845060+0000) 2019-10-13T11:51:53.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:53.268+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.270016+0000) 2019-10-13T11:51:53.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:53.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.270353+0000) 2019-10-13T11:51:53.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:53.268+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.270654+0000) 2019-10-13T11:51:53.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:53.268+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.271126+0000) 2019-10-13T11:51:53.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:53.268+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.272523+0000) 2019-10-13T11:51:53.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:53.268+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.273178+0000) 2019-10-13T11:51:53.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:53.272+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.273981+0000) 2019-10-13T11:51:53.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:53.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:53.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:53.276+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.280519+0000) 2019-10-13T11:51:53.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:53.584+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:53.586948+0000) 2019-10-13T11:51:54.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:54.168+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.168242+0000) 2019-10-13T11:51:54.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:54.452+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:54.454500+0000) 2019-10-13T11:51:57.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:57.819+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:57.825255+0000) 2019-10-13T11:51:58.052 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:58.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.051314+0000) 2019-10-13T11:51:58.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:58.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.072619+0000) 2019-10-13T11:51:58.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:51:58.275+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.281165+0000) 2019-10-13T11:51:58.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:51:58.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.281592+0000) 2019-10-13T11:51:58.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:58.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.282630+0000) 2019-10-13T11:51:58.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:51:58.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.283214+0000) 2019-10-13T11:51:58.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:51:58.280+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.283791+0000) 2019-10-13T11:51:58.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:51:58.279+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.284176+0000) 2019-10-13T11:51:58.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:51:58.279+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.284855+0000) 2019-10-13T11:51:58.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:51:58.288+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.288245+0000) 2019-10-13T11:51:58.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:51:58.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:51:58.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:51:58.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:51:58.593709+0000) 2019-10-13T11:51:59.304 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:51:59.305 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:51:59.308 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:51:59.309 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:01.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:01.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:01.845254+0000) 2019-10-13T11:52:03.292 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:03.287+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.291880+0000) 2019-10-13T11:52:03.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:03.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.294166+0000) 2019-10-13T11:52:03.295 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:03.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.294588+0000) 2019-10-13T11:52:03.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:03.291+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.294931+0000) 2019-10-13T11:52:03.296 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:03.291+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.295403+0000) 2019-10-13T11:52:03.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:03.291+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.297712+0000) 2019-10-13T11:52:03.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:03.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:03.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:03.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.300310+0000) 2019-10-13T11:52:03.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:03.304+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.306308+0000) 2019-10-13T11:52:03.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:03.592+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:03.593296+0000) 2019-10-13T11:52:04.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:04.167+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.168419+0000) 2019-10-13T11:52:04.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:04.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:04.454734+0000) 2019-10-13T11:52:07.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:07.819+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:07.825463+0000) 2019-10-13T11:52:08.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:08.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.051547+0000) 2019-10-13T11:52:08.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:08.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.072835+0000) 2019-10-13T11:52:08.303 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:08.299+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.302435+0000) 2019-10-13T11:52:08.304 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:08.299+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.303414+0000) 2019-10-13T11:52:08.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:08.299+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.303950+0000) 2019-10-13T11:52:08.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:08.299+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.304162+0000) 2019-10-13T11:52:08.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:08.299+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.304698+0000) 2019-10-13T11:52:08.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:08.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.305470+0000) 2019-10-13T11:52:08.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:08.299+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.305703+0000) 2019-10-13T11:52:08.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:08.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:08.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:08.311+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.312452+0000) 2019-10-13T11:52:08.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:08.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:08.593984+0000) 2019-10-13T11:52:11.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:11.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:11.845456+0000) 2019-10-13T11:52:13.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:13.311+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.314657+0000) 2019-10-13T11:52:13.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:13.311+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.315422+0000) 2019-10-13T11:52:13.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:13.311+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.315632+0000) 2019-10-13T11:52:13.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:13.311+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.315954+0000) 2019-10-13T11:52:13.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:13.311+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.316378+0000) 2019-10-13T11:52:13.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:13.311+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.316676+0000) 2019-10-13T11:52:13.318 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:13.315+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.318212+0000) 2019-10-13T11:52:13.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:13.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:13.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:13.319+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.323632+0000) 2019-10-13T11:52:13.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:13.595+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:13.599787+0000) 2019-10-13T11:52:14.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:14.167+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.168598+0000) 2019-10-13T11:52:14.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:14.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:14.454933+0000) 2019-10-13T11:52:17.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:17.823+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:17.825662+0000) 2019-10-13T11:52:18.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:18.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.051746+0000) 2019-10-13T11:52:18.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:18.071+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.073048+0000) 2019-10-13T11:52:18.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:18.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.325004+0000) 2019-10-13T11:52:18.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:18.319+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.325220+0000) 2019-10-13T11:52:18.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:18.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.326105+0000) 2019-10-13T11:52:18.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:18.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.326583+0000) 2019-10-13T11:52:18.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:18.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.327020+0000) 2019-10-13T11:52:18.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:18.323+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.327620+0000) 2019-10-13T11:52:18.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:18.323+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.328731+0000) 2019-10-13T11:52:18.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:18.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:18.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:18.335+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.336574+0000) 2019-10-13T11:52:18.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:18.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:18.594117+0000) 2019-10-13T11:52:21.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:21.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:21.845651+0000) 2019-10-13T11:52:23.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:23.331+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.335734+0000) 2019-10-13T11:52:23.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:23.331+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.337107+0000) 2019-10-13T11:52:23.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:23.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.337874+0000) 2019-10-13T11:52:23.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:23.335+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.338092+0000) 2019-10-13T11:52:23.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:23.335+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.338702+0000) 2019-10-13T11:52:23.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:23.335+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.339138+0000) 2019-10-13T11:52:23.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:23.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.339484+0000) 2019-10-13T11:52:23.344 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:23.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:23.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:23.343+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.346237+0000) 2019-10-13T11:52:23.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:23.603+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:23.606369+0000) 2019-10-13T11:52:24.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:24.167+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.168795+0000) 2019-10-13T11:52:24.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:24.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:24.455132+0000) 2019-10-13T11:52:27.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:27.823+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:27.825867+0000) 2019-10-13T11:52:28.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:28.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.051965+0000) 2019-10-13T11:52:28.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:28.067+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.073258+0000) 2019-10-13T11:52:28.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:28.343+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.346399+0000) 2019-10-13T11:52:28.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:28.343+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.347244+0000) 2019-10-13T11:52:28.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:28.343+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.347421+0000) 2019-10-13T11:52:28.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:28.343+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.347777+0000) 2019-10-13T11:52:28.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:28.343+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.348600+0000) 2019-10-13T11:52:28.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:28.347+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.349504+0000) 2019-10-13T11:52:28.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:28.347+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.350209+0000) 2019-10-13T11:52:28.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:28.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:28.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:28.355+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.356275+0000) 2019-10-13T11:52:28.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:28.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:28.594296+0000) 2019-10-13T11:52:29.633 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:52:29.633 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:29.647 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:52:29.647 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:31.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:31.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:31.845848+0000) 2019-10-13T11:52:33.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:33.351+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.356658+0000) 2019-10-13T11:52:33.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:33.351+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.357005+0000) 2019-10-13T11:52:33.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:33.355+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.357796+0000) 2019-10-13T11:52:33.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:33.355+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.358478+0000) 2019-10-13T11:52:33.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:33.355+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.358860+0000) 2019-10-13T11:52:33.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:33.355+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.359703+0000) 2019-10-13T11:52:33.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:33.355+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.360828+0000) 2019-10-13T11:52:33.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:33.359+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:33.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:33.363+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.366219+0000) 2019-10-13T11:52:33.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:33.611+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:33.612999+0000) 2019-10-13T11:52:34.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:34.163+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.168984+0000) 2019-10-13T11:52:34.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:34.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:34.455334+0000) 2019-10-13T11:52:37.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:37.823+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:37.826061+0000) 2019-10-13T11:52:38.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:38.047+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.052174+0000) 2019-10-13T11:52:38.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:38.071+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.073453+0000) 2019-10-13T11:52:38.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:38.363+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.367883+0000) 2019-10-13T11:52:38.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:38.363+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.368729+0000) 2019-10-13T11:52:38.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:38.367+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.370067+0000) 2019-10-13T11:52:38.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:38.367+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.370665+0000) 2019-10-13T11:52:38.371 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:38.367+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.370723+0000) 2019-10-13T11:52:38.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:38.367+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.371192+0000) 2019-10-13T11:52:38.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:38.367+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.371390+0000) 2019-10-13T11:52:38.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:38.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:38.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:38.371+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.375871+0000) 2019-10-13T11:52:38.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:38.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:38.594498+0000) 2019-10-13T11:52:41.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:41.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:41.846046+0000) 2019-10-13T11:52:43.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:43.375+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.377830+0000) 2019-10-13T11:52:43.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:43.375+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.379170+0000) 2019-10-13T11:52:43.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:43.379+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.380176+0000) 2019-10-13T11:52:43.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:43.375+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.380447+0000) 2019-10-13T11:52:43.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:43.375+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.380838+0000) 2019-10-13T11:52:43.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:43.375+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.381173+0000) 2019-10-13T11:52:43.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:43.375+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.381683+0000) 2019-10-13T11:52:43.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:43.379+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:43.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:43.383+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.385761+0000) 2019-10-13T11:52:43.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:43.615+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:43.618578+0000) 2019-10-13T11:52:44.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:44.163+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.169176+0000) 2019-10-13T11:52:44.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:44.451+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:44.455553+0000) 2019-10-13T11:52:47.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:47.823+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:47.826286+0000) 2019-10-13T11:52:48.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:48.051+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.052391+0000) 2019-10-13T11:52:48.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:48.071+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.073651+0000) 2019-10-13T11:52:48.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:48.387+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.388134+0000) 2019-10-13T11:52:48.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:48.383+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.388875+0000) 2019-10-13T11:52:48.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:48.383+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.389531+0000) 2019-10-13T11:52:48.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:48.387+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.390781+0000) 2019-10-13T11:52:48.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:48.387+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.391294+0000) 2019-10-13T11:52:48.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:48.387+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.391784+0000) 2019-10-13T11:52:48.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:48.387+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.392369+0000) 2019-10-13T11:52:48.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:48.391+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:48.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:48.395+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.396563+0000) 2019-10-13T11:52:48.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:48.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:48.594687+0000) 2019-10-13T11:52:51.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:51.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:51.846251+0000) 2019-10-13T11:52:53.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:53.394+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.399224+0000) 2019-10-13T11:52:53.401 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:53.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.400198+0000) 2019-10-13T11:52:53.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:53.394+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.400129+0000) 2019-10-13T11:52:53.401 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:53.394+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.400597+0000) 2019-10-13T11:52:53.402 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:53.394+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.401308+0000) 2019-10-13T11:52:53.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:53.398+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.401960+0000) 2019-10-13T11:52:53.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:53.398+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.402041+0000) 2019-10-13T11:52:53.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:53.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:53.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:53.407+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.411312+0000) 2019-10-13T11:52:53.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:53.623+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:53.624879+0000) 2019-10-13T11:52:54.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:54.162+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.169394+0000) 2019-10-13T11:52:54.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:54.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:54.455752+0000) 2019-10-13T11:52:57.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:57.822+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:57.826436+0000) 2019-10-13T11:52:58.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:58.050+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.052619+0000) 2019-10-13T11:52:58.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:58.070+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.073909+0000) 2019-10-13T11:52:58.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:52:58.406+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.410091+0000) 2019-10-13T11:52:58.412 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:52:58.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.410452+0000) 2019-10-13T11:52:58.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:52:58.406+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.410788+0000) 2019-10-13T11:52:58.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:58.406+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.411240+0000) 2019-10-13T11:52:58.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:52:58.406+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.411877+0000) 2019-10-13T11:52:58.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:52:58.406+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.412192+0000) 2019-10-13T11:52:58.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:52:58.406+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.413309+0000) 2019-10-13T11:52:58.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:52:58.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:52:58.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:52:58.415+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.418691+0000) 2019-10-13T11:52:58.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:52:58.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:52:58.594861+0000) 2019-10-13T11:52:59.756 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:52:59.756 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:52:59.760 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:52:59.760 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:01.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:01.843+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:01.846452+0000) 2019-10-13T11:53:03.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:03.414+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.421082+0000) 2019-10-13T11:53:03.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:03.414+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.421450+0000) 2019-10-13T11:53:03.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:03.418+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.422581+0000) 2019-10-13T11:53:03.423 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:03.418+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.422602+0000) 2019-10-13T11:53:03.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:03.418+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.423116+0000) 2019-10-13T11:53:03.424 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:03.418+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.423558+0000) 2019-10-13T11:53:03.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:03.419+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.423829+0000) 2019-10-13T11:53:03.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:03.422+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148343 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:03.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:03.431+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.433864+0000) 2019-10-13T11:53:03.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:03.631+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:03.631529+0000) 2019-10-13T11:53:04.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:04.166+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.169585+0000) 2019-10-13T11:53:04.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:04.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:04.455949+0000) 2019-10-13T11:53:07.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:07.822+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:07.826626+0000) 2019-10-13T11:53:08.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:08.046+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.052816+0000) 2019-10-13T11:53:08.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:08.070+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.074134+0000) 2019-10-13T11:53:08.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:08.431+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.432279+0000) 2019-10-13T11:53:08.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:08.426+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.432887+0000) 2019-10-13T11:53:08.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:08.426+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.433434+0000) 2019-10-13T11:53:08.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:08.430+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.434147+0000) 2019-10-13T11:53:08.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:08.430+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.435217+0000) 2019-10-13T11:53:08.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:08.430+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.435893+0000) 2019-10-13T11:53:08.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:08.430+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.436403+0000) 2019-10-13T11:53:08.440 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:08.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:08.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:08.435+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.439885+0000) 2019-10-13T11:53:08.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:08.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:08.595062+0000) 2019-10-13T11:53:11.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:11.842+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:11.846666+0000) 2019-10-13T11:53:13.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:13.438+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.442652+0000) 2019-10-13T11:53:13.444 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:13.438+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.443180+0000) 2019-10-13T11:53:13.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:13.438+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.443708+0000) 2019-10-13T11:53:13.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:13.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.444046+0000) 2019-10-13T11:53:13.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:13.438+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.444416+0000) 2019-10-13T11:53:13.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:13.438+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.444806+0000) 2019-10-13T11:53:13.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:13.438+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.445509+0000) 2019-10-13T11:53:13.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:13.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:13.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:13.446+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.451403+0000) 2019-10-13T11:53:13.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:13.634+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:13.637897+0000) 2019-10-13T11:53:14.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:14.166+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.169799+0000) 2019-10-13T11:53:14.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:14.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:14.456154+0000) 2019-10-13T11:53:17.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:17.826+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:17.826826+0000) 2019-10-13T11:53:18.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:18.050+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.053096+0000) 2019-10-13T11:53:18.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:18.070+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.074363+0000) 2019-10-13T11:53:18.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:18.450+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.454134+0000) 2019-10-13T11:53:18.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:18.450+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.454750+0000) 2019-10-13T11:53:18.458 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:18.450+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.457496+0000) 2019-10-13T11:53:18.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:18.454+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.457916+0000) 2019-10-13T11:53:18.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:18.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.457848+0000) 2019-10-13T11:53:18.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:18.454+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.458296+0000) 2019-10-13T11:53:18.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:18.454+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.458567+0000) 2019-10-13T11:53:18.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:18.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:18.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:18.466+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.468135+0000) 2019-10-13T11:53:18.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:18.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:18.595240+0000) 2019-10-13T11:53:21.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:21.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:21.846877+0000) 2019-10-13T11:53:23.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:23.458+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.465295+0000) 2019-10-13T11:53:23.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:23.458+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.465639+0000) 2019-10-13T11:53:23.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:23.462+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.466691+0000) 2019-10-13T11:53:23.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:23.462+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.467083+0000) 2019-10-13T11:53:23.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:23.462+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.467599+0000) 2019-10-13T11:53:23.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:23.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.467984+0000) 2019-10-13T11:53:23.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:23.462+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.468514+0000) 2019-10-13T11:53:23.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:23.466+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:23.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:23.470+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.474823+0000) 2019-10-13T11:53:23.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:23.642+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:23.644472+0000) 2019-10-13T11:53:24.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:24.166+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.170023+0000) 2019-10-13T11:53:24.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:24.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:24.456357+0000) 2019-10-13T11:53:27.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:27.826+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:27.827031+0000) 2019-10-13T11:53:28.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:28.046+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.053329+0000) 2019-10-13T11:53:28.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:28.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.074568+0000) 2019-10-13T11:53:28.479 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:28.474+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.478898+0000) 2019-10-13T11:53:28.480 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:28.474+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.479209+0000) 2019-10-13T11:53:28.480 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:28.474+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.479737+0000) 2019-10-13T11:53:28.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:28.474+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.481307+0000) 2019-10-13T11:53:28.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:28.474+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.481474+0000) 2019-10-13T11:53:28.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:28.478+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.482022+0000) 2019-10-13T11:53:28.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:28.478+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.483295+0000) 2019-10-13T11:53:28.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:28.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:28.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:28.486+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.490878+0000) 2019-10-13T11:53:28.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:28.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:28.595433+0000) 2019-10-13T11:53:29.812 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:53:29.812 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:29.816 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:53:29.816 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:31.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:31.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:31.847073+0000) 2019-10-13T11:53:33.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:33.482+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.488378+0000) 2019-10-13T11:53:33.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:33.482+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.489034+0000) 2019-10-13T11:53:33.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:33.486+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.489316+0000) 2019-10-13T11:53:33.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:33.486+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.490133+0000) 2019-10-13T11:53:33.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:33.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.491186+0000) 2019-10-13T11:53:33.492 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:33.486+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.491480+0000) 2019-10-13T11:53:33.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:33.486+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.491881+0000) 2019-10-13T11:53:33.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:33.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:33.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:33.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.497075+0000) 2019-10-13T11:53:33.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:33.650+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:33.650944+0000) 2019-10-13T11:53:34.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:34.166+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.170206+0000) 2019-10-13T11:53:34.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:34.450+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:34.456550+0000) 2019-10-13T11:53:37.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:37.822+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:37.827227+0000) 2019-10-13T11:53:38.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:38.050+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.053553+0000) 2019-10-13T11:53:38.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:38.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.074764+0000) 2019-10-13T11:53:38.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:38.494+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.498714+0000) 2019-10-13T11:53:38.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:38.494+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.499551+0000) 2019-10-13T11:53:38.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:38.494+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.499274+0000) 2019-10-13T11:53:38.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:38.494+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.499821+0000) 2019-10-13T11:53:38.501 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:38.494+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.500395+0000) 2019-10-13T11:53:38.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:38.494+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.500632+0000) 2019-10-13T11:53:38.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:38.494+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.501184+0000) 2019-10-13T11:53:38.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:38.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:38.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:38.502+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.505361+0000) 2019-10-13T11:53:38.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:38.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:38.595612+0000) 2019-10-13T11:53:41.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:41.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:41.847269+0000) 2019-10-13T11:53:43.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:43.502+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.508226+0000) 2019-10-13T11:53:43.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:43.506+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.510199+0000) 2019-10-13T11:53:43.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:43.506+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.511087+0000) 2019-10-13T11:53:43.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:43.506+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.513711+0000) 2019-10-13T11:53:43.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:43.510+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.513819+0000) 2019-10-13T11:53:43.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:43.510+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.514274+0000) 2019-10-13T11:53:43.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:43.510+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.514645+0000) 2019-10-13T11:53:43.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:43.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:43.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:43.514+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.516853+0000) 2019-10-13T11:53:43.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:43.654+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:43.657325+0000) 2019-10-13T11:53:44.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:44.170+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.170399+0000) 2019-10-13T11:53:44.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:44.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:44.456730+0000) 2019-10-13T11:53:47.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:47.821+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:47.827424+0000) 2019-10-13T11:53:48.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:48.049+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.053790+0000) 2019-10-13T11:53:48.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:48.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.074966+0000) 2019-10-13T11:53:48.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:48.513+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.519141+0000) 2019-10-13T11:53:48.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:48.513+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.519971+0000) 2019-10-13T11:53:48.521 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:48.513+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.521032+0000) 2019-10-13T11:53:48.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:48.517+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.522497+0000) 2019-10-13T11:53:48.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:48.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:48.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:48.521+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.525936+0000) 2019-10-13T11:53:48.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:48.521+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.526422+0000) 2019-10-13T11:53:48.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:48.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.524908+0000) 2019-10-13T11:53:48.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:48.522+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.526697+0000) 2019-10-13T11:53:48.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:48.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:48.595789+0000) 2019-10-13T11:53:51.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:51.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:51.847467+0000) 2019-10-13T11:53:53.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:53.521+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.528357+0000) 2019-10-13T11:53:53.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:53.521+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.529430+0000) 2019-10-13T11:53:53.530 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:53.525+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.529912+0000) 2019-10-13T11:53:53.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:53.525+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.530358+0000) 2019-10-13T11:53:53.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:53.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.531294+0000) 2019-10-13T11:53:53.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:53.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.531650+0000) 2019-10-13T11:53:53.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:53.526+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.531969+0000) 2019-10-13T11:53:53.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:53.530+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.535398+0000) 2019-10-13T11:53:53.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:53.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148645 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:53.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:53.662+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:53.663867+0000) 2019-10-13T11:53:54.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:54.169+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.170585+0000) 2019-10-13T11:53:54.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:54.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:54.456910+0000) 2019-10-13T11:53:57.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:57.825+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:57.827614+0000) 2019-10-13T11:53:58.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:58.053+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.054009+0000) 2019-10-13T11:53:58.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:58.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.075166+0000) 2019-10-13T11:53:58.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:53:58.533+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.539464+0000) 2019-10-13T11:53:58.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:53:58.533+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.541376+0000) 2019-10-13T11:53:58.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:53:58.537+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.541781+0000) 2019-10-13T11:53:58.545 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:53:58.537+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.544081+0000) 2019-10-13T11:53:58.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:53:58.542+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.544213+0000) 2019-10-13T11:53:58.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:53:58.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.544984+0000) 2019-10-13T11:53:58.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:58.541+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.546399+0000) 2019-10-13T11:53:58.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:53:58.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:53:58.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:53:58.550+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.554503+0000) 2019-10-13T11:53:58.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:53:58.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:53:58.596008+0000) 2019-10-13T11:53:59.909 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:53:59.910 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:53:59.913 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:53:59.913 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:01.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:01.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:01.847667+0000) 2019-10-13T11:54:03.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:03.549+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.550348+0000) 2019-10-13T11:54:03.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:03.546+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.550834+0000) 2019-10-13T11:54:03.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:03.549+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.551503+0000) 2019-10-13T11:54:03.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:03.549+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.551939+0000) 2019-10-13T11:54:03.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:03.549+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.552237+0000) 2019-10-13T11:54:03.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:03.549+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.552602+0000) 2019-10-13T11:54:03.555 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:03.553+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.554715+0000) 2019-10-13T11:54:03.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:03.554+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.557463+0000) 2019-10-13T11:54:03.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:03.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:03.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:03.666+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:03.670378+0000) 2019-10-13T11:54:04.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:04.169+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.170775+0000) 2019-10-13T11:54:04.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:04.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:04.457123+0000) 2019-10-13T11:54:07.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:07.825+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:07.827817+0000) 2019-10-13T11:54:08.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:08.053+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.054253+0000) 2019-10-13T11:54:08.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:08.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.075370+0000) 2019-10-13T11:54:08.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:08.557+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.561663+0000) 2019-10-13T11:54:08.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:08.561+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.562068+0000) 2019-10-13T11:54:08.563 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:08.561+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.562829+0000) 2019-10-13T11:54:08.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:08.558+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.563328+0000) 2019-10-13T11:54:08.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:08.561+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.564892+0000) 2019-10-13T11:54:08.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:08.561+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.565198+0000) 2019-10-13T11:54:08.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:08.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.568958+0000) 2019-10-13T11:54:08.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:08.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:08.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:08.570+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.572761+0000) 2019-10-13T11:54:08.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:08.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:08.596193+0000) 2019-10-13T11:54:11.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:11.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:11.847861+0000) 2019-10-13T11:54:13.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:13.569+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.572423+0000) 2019-10-13T11:54:13.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:13.569+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.572902+0000) 2019-10-13T11:54:13.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:13.569+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.573121+0000) 2019-10-13T11:54:13.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:13.570+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.573505+0000) 2019-10-13T11:54:13.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:13.573+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.574145+0000) 2019-10-13T11:54:13.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:13.573+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.575262+0000) 2019-10-13T11:54:13.576 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:13.573+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.575733+0000) 2019-10-13T11:54:13.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:13.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:13.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:13.578+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.581016+0000) 2019-10-13T11:54:13.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:13.674+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:13.676920+0000) 2019-10-13T11:54:14.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:14.169+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.170974+0000) 2019-10-13T11:54:14.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:14.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:14.457336+0000) 2019-10-13T11:54:17.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:17.825+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:17.828047+0000) 2019-10-13T11:54:18.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:18.053+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.054501+0000) 2019-10-13T11:54:18.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:18.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.075567+0000) 2019-10-13T11:54:18.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:18.581+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.582496+0000) 2019-10-13T11:54:18.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:18.581+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.582921+0000) 2019-10-13T11:54:18.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:18.581+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.583163+0000) 2019-10-13T11:54:18.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:18.581+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.583508+0000) 2019-10-13T11:54:18.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:18.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.584578+0000) 2019-10-13T11:54:18.586 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:18.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.585843+0000) 2019-10-13T11:54:18.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:18.581+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.586244+0000) 2019-10-13T11:54:18.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:18.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:18.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:18.585+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.590488+0000) 2019-10-13T11:54:18.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:18.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:18.596336+0000) 2019-10-13T11:54:21.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:21.845+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:21.848085+0000) 2019-10-13T11:54:23.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:23.589+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.592621+0000) 2019-10-13T11:54:23.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:23.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.594341+0000) 2019-10-13T11:54:23.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:23.593+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.594950+0000) 2019-10-13T11:54:23.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:23.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.598167+0000) 2019-10-13T11:54:23.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:23.597+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.599485+0000) 2019-10-13T11:54:23.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:23.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:23.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:23.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.601313+0000) 2019-10-13T11:54:23.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:23.597+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.601712+0000) 2019-10-13T11:54:23.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:23.605+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.610898+0000) 2019-10-13T11:54:23.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:23.681+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:23.687475+0000) 2019-10-13T11:54:24.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:24.169+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.171174+0000) 2019-10-13T11:54:24.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:24.453+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:24.457509+0000) 2019-10-13T11:54:27.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:27.825+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:27.828247+0000) 2019-10-13T11:54:28.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:28.053+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.054701+0000) 2019-10-13T11:54:28.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:28.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.075764+0000) 2019-10-13T11:54:28.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:28.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.596524+0000) 2019-10-13T11:54:28.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:28.601+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.603666+0000) 2019-10-13T11:54:28.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:28.601+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.604219+0000) 2019-10-13T11:54:28.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:28.605+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.605821+0000) 2019-10-13T11:54:28.607 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:28.605+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.606314+0000) 2019-10-13T11:54:28.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:28.605+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.606749+0000) 2019-10-13T11:54:28.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:28.601+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.607040+0000) 2019-10-13T11:54:28.608 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:28.605+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.607452+0000) 2019-10-13T11:54:28.611 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:28.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:28.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:28.605+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:28.611450+0000) 2019-10-13T11:54:30.033 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:54:30.033 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:30.037 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:54:30.037 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:54:31.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:31.845+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:31.848373+0000) 2019-10-13T11:54:33.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:33.613+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.614425+0000) 2019-10-13T11:54:33.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:33.613+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.615374+0000) 2019-10-13T11:54:33.616 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:33.613+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.615872+0000) 2019-10-13T11:54:33.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:33.613+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.616139+0000) 2019-10-13T11:54:33.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:33.613+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.616380+0000) 2019-10-13T11:54:33.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:33.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.616979+0000) 2019-10-13T11:54:33.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:33.613+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.617407+0000) 2019-10-13T11:54:33.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:33.617+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.620349+0000) 2019-10-13T11:54:33.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:33.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:33.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:33.689+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:33.694458+0000) 2019-10-13T11:54:34.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:34.169+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.171357+0000) 2019-10-13T11:54:34.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:34.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:34.457692+0000) 2019-10-13T11:54:37.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:37.825+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:37.828439+0000) 2019-10-13T11:54:38.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:38.053+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.054928+0000) 2019-10-13T11:54:38.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:38.073+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.075999+0000) 2019-10-13T11:54:38.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:38.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.596724+0000) 2019-10-13T11:54:38.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:38.621+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.624819+0000) 2019-10-13T11:54:38.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:38.621+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.625150+0000) 2019-10-13T11:54:38.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:38.625+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.627131+0000) 2019-10-13T11:54:38.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:38.625+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.627729+0000) 2019-10-13T11:54:38.629 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:38.625+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.628209+0000) 2019-10-13T11:54:38.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:38.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.628750+0000) 2019-10-13T11:54:38.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:38.625+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.629185+0000) 2019-10-13T11:54:38.633 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:38.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:38.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:38.629+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:38.635143+0000) 2019-10-13T11:54:41.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:41.845+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:41.848569+0000) 2019-10-13T11:54:43.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:43.629+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.634598+0000) 2019-10-13T11:54:43.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:43.632+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.635783+0000) 2019-10-13T11:54:43.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:43.632+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.636108+0000) 2019-10-13T11:54:43.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:43.632+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.636756+0000) 2019-10-13T11:54:43.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:43.632+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.637289+0000) 2019-10-13T11:54:43.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:43.636+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.637937+0000) 2019-10-13T11:54:43.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:43.636+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.639086+0000) 2019-10-13T11:54:43.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:43.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:43.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:43.641+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.646496+0000) 2019-10-13T11:54:43.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:43.701+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:43.704764+0000) 2019-10-13T11:54:44.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:44.168+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.171548+0000) 2019-10-13T11:54:44.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:44.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:44.457899+0000) 2019-10-13T11:54:47.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:47.824+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:47.828651+0000) 2019-10-13T11:54:48.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:48.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.055136+0000) 2019-10-13T11:54:48.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:48.072+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.076231+0000) 2019-10-13T11:54:48.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:48.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.596921+0000) 2019-10-13T11:54:48.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:48.640+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.644575+0000) 2019-10-13T11:54:48.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:48.644+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.645952+0000) 2019-10-13T11:54:48.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:48.644+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.645947+0000) 2019-10-13T11:54:48.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:48.644+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.648255+0000) 2019-10-13T11:54:48.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:48.644+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.648891+0000) 2019-10-13T11:54:48.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:48.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.651089+0000) 2019-10-13T11:54:48.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:48.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.652983+0000) 2019-10-13T11:54:48.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:48.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:48.655 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:48.652+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:48.654285+0000) 2019-10-13T11:54:51.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:51.845+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:51.848761+0000) 2019-10-13T11:54:53.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:53.653+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.657116+0000) 2019-10-13T11:54:53.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:53.652+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.657505+0000) 2019-10-13T11:54:53.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:53.656+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.658804+0000) 2019-10-13T11:54:53.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:53.656+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.659239+0000) 2019-10-13T11:54:53.661 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:53.656+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.659559+0000) 2019-10-13T11:54:53.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:53.656+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.660235+0000) 2019-10-13T11:54:53.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:53.656+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.657905+0000) 2019-10-13T11:54:53.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:53.657+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.662390+0000) 2019-10-13T11:54:53.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:53.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:53.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:53.709+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:53.714582+0000) 2019-10-13T11:54:54.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:54.168+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.171771+0000) 2019-10-13T11:54:54.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:54.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:54.458077+0000) 2019-10-13T11:54:57.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:57.824+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:57.828880+0000) 2019-10-13T11:54:58.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:58.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.055366+0000) 2019-10-13T11:54:58.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:58.072+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.076412+0000) 2019-10-13T11:54:58.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:58.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.597129+0000) 2019-10-13T11:54:58.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:54:58.668+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.669820+0000) 2019-10-13T11:54:58.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:54:58.668+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.670293+0000) 2019-10-13T11:54:58.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:54:58.665+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.670925+0000) 2019-10-13T11:54:58.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:54:58.668+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.671971+0000) 2019-10-13T11:54:58.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:54:58.668+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.672185+0000) 2019-10-13T11:54:58.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:54:58.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.673946+0000) 2019-10-13T11:54:58.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:54:58.672+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.674253+0000) 2019-10-13T11:54:58.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:54:58.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:54:58.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:54:58.677+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:54:58.681363+0000) 2019-10-13T11:55:00.157 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:55:00.157 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:00.160 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:55:00.161 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:01.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:01.845+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:01.848955+0000) 2019-10-13T11:55:03.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:03.677+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.680430+0000) 2019-10-13T11:55:03.681 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:03.676+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.681123+0000) 2019-10-13T11:55:03.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:03.676+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.681648+0000) 2019-10-13T11:55:03.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:03.680+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.681777+0000) 2019-10-13T11:55:03.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:03.680+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.682562+0000) 2019-10-13T11:55:03.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:03.680+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.683674+0000) 2019-10-13T11:55:03.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:03.680+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.683993+0000) 2019-10-13T11:55:03.688 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:03.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:03.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:03.685+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.689148+0000) 2019-10-13T11:55:03.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:03.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:03.721174+0000) 2019-10-13T11:55:04.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:04.168+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.171952+0000) 2019-10-13T11:55:04.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:04.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:04.458252+0000) 2019-10-13T11:55:07.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:07.824+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:07.829074+0000) 2019-10-13T11:55:08.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:08.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.055567+0000) 2019-10-13T11:55:08.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:08.072+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.076608+0000) 2019-10-13T11:55:08.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:08.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.597335+0000) 2019-10-13T11:55:08.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:08.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.690830+0000) 2019-10-13T11:55:08.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:08.685+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.691266+0000) 2019-10-13T11:55:08.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:08.688+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.692463+0000) 2019-10-13T11:55:08.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:08.688+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.692727+0000) 2019-10-13T11:55:08.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:08.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.693502+0000) 2019-10-13T11:55:08.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:08.692+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.693997+0000) 2019-10-13T11:55:08.695 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:08.692+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.694390+0000) 2019-10-13T11:55:08.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:08.693+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:08.698096+0000) 2019-10-13T11:55:08.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:08.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:11.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:11.849+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:11.849155+0000) 2019-10-13T11:55:13.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:13.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.701503+0000) 2019-10-13T11:55:13.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:13.700+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.702828+0000) 2019-10-13T11:55:13.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:13.700+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.703629+0000) 2019-10-13T11:55:13.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:13.697+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.703956+0000) 2019-10-13T11:55:13.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:13.700+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.704104+0000) 2019-10-13T11:55:13.705 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:13.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.704420+0000) 2019-10-13T11:55:13.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:13.700+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.704906+0000) 2019-10-13T11:55:13.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:13.705+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.708446+0000) 2019-10-13T11:55:13.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:13.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:13.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:13.721+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:13.725807+0000) 2019-10-13T11:55:14.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:14.168+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.172134+0000) 2019-10-13T11:55:14.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:14.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:14.458430+0000) 2019-10-13T11:55:17.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:17.828+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:17.829276+0000) 2019-10-13T11:55:18.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:18.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.055739+0000) 2019-10-13T11:55:18.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:18.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.076823+0000) 2019-10-13T11:55:18.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:18.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.597543+0000) 2019-10-13T11:55:18.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:18.708+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.713227+0000) 2019-10-13T11:55:18.714 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:18.708+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.713556+0000) 2019-10-13T11:55:18.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:18.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.714126+0000) 2019-10-13T11:55:18.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:18.713+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.716415+0000) 2019-10-13T11:55:18.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:18.712+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.716579+0000) 2019-10-13T11:55:18.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:18.712+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.717063+0000) 2019-10-13T11:55:18.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:18.712+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.717653+0000) 2019-10-13T11:55:18.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:18.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:18.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:18.717+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:18.722176+0000) 2019-10-13T11:55:21.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:21.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:21.849390+0000) 2019-10-13T11:55:23.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:23.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.725318+0000) 2019-10-13T11:55:23.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:23.724+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.727469+0000) 2019-10-13T11:55:23.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:23.720+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.727912+0000) 2019-10-13T11:55:23.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:23.724+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.728205+0000) 2019-10-13T11:55:23.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:23.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.728924+0000) 2019-10-13T11:55:23.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:23.724+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.729560+0000) 2019-10-13T11:55:23.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:23.728+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.729880+0000) 2019-10-13T11:55:23.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:23.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:23.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:23.732+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.736194+0000) 2019-10-13T11:55:23.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:23.732+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:23.736252+0000) 2019-10-13T11:55:24.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:24.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.172313+0000) 2019-10-13T11:55:24.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:24.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:24.458645+0000) 2019-10-13T11:55:27.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:27.828+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:27.829476+0000) 2019-10-13T11:55:28.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:28.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.055945+0000) 2019-10-13T11:55:28.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:28.072+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.077022+0000) 2019-10-13T11:55:28.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:28.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.597776+0000) 2019-10-13T11:55:28.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:28.732+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.736238+0000) 2019-10-13T11:55:28.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:28.732+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.736709+0000) 2019-10-13T11:55:28.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:28.732+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.738608+0000) 2019-10-13T11:55:28.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:28.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.738957+0000) 2019-10-13T11:55:28.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:28.736+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.739266+0000) 2019-10-13T11:55:28.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:28.736+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.740881+0000) 2019-10-13T11:55:28.742 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:28.736+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.741251+0000) 2019-10-13T11:55:28.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:28.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:28.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:28.744+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:28.751817+0000) 2019-10-13T11:55:30.283 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:55:30.283 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:30.287 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:55:30.287 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:55:31.850 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:31.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:31.849584+0000) 2019-10-13T11:55:33.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:33.732+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.736474+0000) 2019-10-13T11:55:33.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:33.748+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.750065+0000) 2019-10-13T11:55:33.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:33.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.750471+0000) 2019-10-13T11:55:33.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:33.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.751485+0000) 2019-10-13T11:55:33.761 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:33.748+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.751043+0000) 2019-10-13T11:55:33.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:33.748+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.751920+0000) 2019-10-13T11:55:33.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:33.748+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.752697+0000) 2019-10-13T11:55:33.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:33.748+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.752803+0000) 2019-10-13T11:55:33.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:33.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:33.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:33.756+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:33.761933+0000) 2019-10-13T11:55:34.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:34.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.172509+0000) 2019-10-13T11:55:34.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:34.456+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:34.458828+0000) 2019-10-13T11:55:37.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:37.824+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:37.829668+0000) 2019-10-13T11:55:38.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:38.052+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.056149+0000) 2019-10-13T11:55:38.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:38.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.077219+0000) 2019-10-13T11:55:38.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:38.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.597931+0000) 2019-10-13T11:55:38.761 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:38.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.760141+0000) 2019-10-13T11:55:38.761 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:38.756+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.760490+0000) 2019-10-13T11:55:38.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:38.755+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.761210+0000) 2019-10-13T11:55:38.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:38.759+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.761961+0000) 2019-10-13T11:55:38.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:38.759+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.762247+0000) 2019-10-13T11:55:38.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:38.759+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.764192+0000) 2019-10-13T11:55:38.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:38.759+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.764427+0000) 2019-10-13T11:55:38.768 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:38.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:38.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:38.764+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:38.770955+0000) 2019-10-13T11:55:41.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:41.848+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:41.849784+0000) 2019-10-13T11:55:43.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:43.740+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.742945+0000) 2019-10-13T11:55:43.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:43.767+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.771117+0000) 2019-10-13T11:55:43.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:43.767+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.771891+0000) 2019-10-13T11:55:43.773 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:43.767+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.772676+0000) 2019-10-13T11:55:43.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:43.768+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.773145+0000) 2019-10-13T11:55:43.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:43.767+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.773699+0000) 2019-10-13T11:55:43.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:43.771+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.774304+0000) 2019-10-13T11:55:43.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:43.771+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.774881+0000) 2019-10-13T11:55:43.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:43.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:43.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:43.776+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:43.780140+0000) 2019-10-13T11:55:44.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:44.167+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.172756+0000) 2019-10-13T11:55:44.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:44.455+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:44.459003+0000) 2019-10-13T11:55:47.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:47.827+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:47.829901+0000) 2019-10-13T11:55:48.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:48.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.056356+0000) 2019-10-13T11:55:48.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:48.075+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.077414+0000) 2019-10-13T11:55:48.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:48.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.598124+0000) 2019-10-13T11:55:48.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:48.775+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.781701+0000) 2019-10-13T11:55:48.783 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:48.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.782131+0000) 2019-10-13T11:55:48.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:48.779+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.782644+0000) 2019-10-13T11:55:48.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:48.779+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.783053+0000) 2019-10-13T11:55:48.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:48.776+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.783720+0000) 2019-10-13T11:55:48.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:48.779+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.783994+0000) 2019-10-13T11:55:48.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:48.779+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.784759+0000) 2019-10-13T11:55:48.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:48.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:48.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:48.784+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:48.790855+0000) 2019-10-13T11:55:51.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:51.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:51.849971+0000) 2019-10-13T11:55:53.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:53.748+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.749533+0000) 2019-10-13T11:55:53.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:53.787+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.792673+0000) 2019-10-13T11:55:53.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:53.787+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.793190+0000) 2019-10-13T11:55:53.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:53.791+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.793951+0000) 2019-10-13T11:55:53.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:53.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.794386+0000) 2019-10-13T11:55:53.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:53.791+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.794615+0000) 2019-10-13T11:55:53.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:53.791+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.795330+0000) 2019-10-13T11:55:53.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:53.791+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.794902+0000) 2019-10-13T11:55:53.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:53.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:53.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:53.792+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:53.799761+0000) 2019-10-13T11:55:54.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:54.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.172974+0000) 2019-10-13T11:55:54.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:54.455+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:54.459188+0000) 2019-10-13T11:55:57.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:57.827+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:57.830105+0000) 2019-10-13T11:55:58.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:58.051+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.056586+0000) 2019-10-13T11:55:58.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:58.075+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.077628+0000) 2019-10-13T11:55:58.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:58.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.598373+0000) 2019-10-13T11:55:58.803 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:55:58.799+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.802319+0000) 2019-10-13T11:55:58.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:55:58.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.803127+0000) 2019-10-13T11:55:58.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:55:58.799+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.802632+0000) 2019-10-13T11:55:58.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:55:58.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.804122+0000) 2019-10-13T11:55:58.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:55:58.800+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.805988+0000) 2019-10-13T11:55:58.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:55:58.803+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.806814+0000) 2019-10-13T11:55:58.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:55:58.803+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.807116+0000) 2019-10-13T11:55:58.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:55:58.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:55:58.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:55:58.808+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:55:58.812717+0000) 2019-10-13T11:56:00.597 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:56:00.598 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:00.602 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:56:00.602 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:01.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:01.844+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:01.850170+0000) 2019-10-13T11:56:03.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:03.752+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.756209+0000) 2019-10-13T11:56:03.813 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:03.807+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.813054+0000) 2019-10-13T11:56:03.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:03.807+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.813467+0000) 2019-10-13T11:56:03.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:03.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.814445+0000) 2019-10-13T11:56:03.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:03.811+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.814478+0000) 2019-10-13T11:56:03.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:03.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.815664+0000) 2019-10-13T11:56:03.816 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:03.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.815684+0000) 2019-10-13T11:56:03.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:03.811+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.816419+0000) 2019-10-13T11:56:03.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:03.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149421 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:03.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:03.816+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:03.821670+0000) 2019-10-13T11:56:04.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:04.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.173158+0000) 2019-10-13T11:56:04.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:04.455+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:04.459384+0000) 2019-10-13T11:56:07.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:07.827+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:07.830336+0000) 2019-10-13T11:56:08.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:08.051+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.056781+0000) 2019-10-13T11:56:08.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:08.075+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.077844+0000) 2019-10-13T11:56:08.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:08.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.598568+0000) 2019-10-13T11:56:08.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:08.819+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.825200+0000) 2019-10-13T11:56:08.827 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:08.820+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.825722+0000) 2019-10-13T11:56:08.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:08.823+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.826391+0000) 2019-10-13T11:56:08.828 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:08.823+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.826545+0000) 2019-10-13T11:56:08.828 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:08.823+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.827995+0000) 2019-10-13T11:56:08.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:08.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.828338+0000) 2019-10-13T11:56:08.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:08.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.828772+0000) 2019-10-13T11:56:08.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:08.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:08.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:08.828+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:08.835835+0000) 2019-10-13T11:56:11.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:11.848+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:11.850364+0000) 2019-10-13T11:56:13.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:13.760+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.762695+0000) 2019-10-13T11:56:13.836 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:13.831+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.834524+0000) 2019-10-13T11:56:13.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:13.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.835218+0000) 2019-10-13T11:56:13.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:13.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.835436+0000) 2019-10-13T11:56:13.837 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:13.831+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.835761+0000) 2019-10-13T11:56:13.837 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:13.832+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.836438+0000) 2019-10-13T11:56:13.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:13.835+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.841533+0000) 2019-10-13T11:56:13.843 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:13.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.842591+0000) 2019-10-13T11:56:13.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:13.836+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:13.842930+0000) 2019-10-13T11:56:13.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:13.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:14.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:14.167+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.173364+0000) 2019-10-13T11:56:14.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:14.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:14.459566+0000) 2019-10-13T11:56:17.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:17.827+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:17.830566+0000) 2019-10-13T11:56:18.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:18.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.056976+0000) 2019-10-13T11:56:18.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:18.075+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.078041+0000) 2019-10-13T11:56:18.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:18.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.598786+0000) 2019-10-13T11:56:18.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:18.843+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.847201+0000) 2019-10-13T11:56:18.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:18.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.848125+0000) 2019-10-13T11:56:18.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:18.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.848074+0000) 2019-10-13T11:56:18.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:18.843+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.848479+0000) 2019-10-13T11:56:18.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:18.843+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.849037+0000) 2019-10-13T11:56:18.850 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:18.843+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.849261+0000) 2019-10-13T11:56:18.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:18.847+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.849903+0000) 2019-10-13T11:56:18.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:18.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:18.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:18.852+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:18.855692+0000) 2019-10-13T11:56:21.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:21.848+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:21.850516+0000) 2019-10-13T11:56:23.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:23.768+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.769243+0000) 2019-10-13T11:56:23.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:23.851+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.857714+0000) 2019-10-13T11:56:23.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:23.855+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.858166+0000) 2019-10-13T11:56:23.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:23.855+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.859025+0000) 2019-10-13T11:56:23.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:23.855+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.859725+0000) 2019-10-13T11:56:23.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:23.860+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.860276+0000) 2019-10-13T11:56:23.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:23.855+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.860510+0000) 2019-10-13T11:56:23.861 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:23.855+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.861079+0000) 2019-10-13T11:56:23.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:23.864+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:23.864845+0000) 2019-10-13T11:56:23.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:23.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149539 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:24.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:24.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.173561+0000) 2019-10-13T11:56:24.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:24.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:24.459760+0000) 2019-10-13T11:56:27.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:27.827+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:27.830778+0000) 2019-10-13T11:56:28.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:28.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.057178+0000) 2019-10-13T11:56:28.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:28.075+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.078264+0000) 2019-10-13T11:56:28.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:28.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.598979+0000) 2019-10-13T11:56:28.869 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:28.863+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.868592+0000) 2019-10-13T11:56:28.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:28.863+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.868802+0000) 2019-10-13T11:56:28.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:28.863+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.869251+0000) 2019-10-13T11:56:28.871 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:28.867+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.870939+0000) 2019-10-13T11:56:28.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:28.867+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.871518+0000) 2019-10-13T11:56:28.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:28.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.873621+0000) 2019-10-13T11:56:28.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:28.871+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.874684+0000) 2019-10-13T11:56:28.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:28.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:28.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:28.879+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:28.881427+0000) 2019-10-13T11:56:30.644 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:56:30.644 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:30.648 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:56:30.648 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:56:31.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:31.847+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:31.850709+0000) 2019-10-13T11:56:33.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:33.771+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.775719+0000) 2019-10-13T11:56:33.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:33.875+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.880592+0000) 2019-10-13T11:56:33.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:33.875+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.880819+0000) 2019-10-13T11:56:33.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:33.879+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.883266+0000) 2019-10-13T11:56:33.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:33.879+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.883835+0000) 2019-10-13T11:56:33.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:33.883+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.885872+0000) 2019-10-13T11:56:33.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:33.883+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.886001+0000) 2019-10-13T11:56:33.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:33.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:33.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:33.887+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.888477+0000) 2019-10-13T11:56:33.897 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:33.895+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:33.896634+0000) 2019-10-13T11:56:34.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:34.170+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.173809+0000) 2019-10-13T11:56:34.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:34.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:34.459937+0000) 2019-10-13T11:56:37.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:37.826+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:37.830975+0000) 2019-10-13T11:56:38.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:38.054+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.057375+0000) 2019-10-13T11:56:38.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:38.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.078468+0000) 2019-10-13T11:56:38.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:38.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.599172+0000) 2019-10-13T11:56:38.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:38.886+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.891386+0000) 2019-10-13T11:56:38.893 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:38.886+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.892467+0000) 2019-10-13T11:56:38.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:38.886+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.893643+0000) 2019-10-13T11:56:38.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:38.890+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.893929+0000) 2019-10-13T11:56:38.895 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:38.890+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.894060+0000) 2019-10-13T11:56:38.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:38.891+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.894916+0000) 2019-10-13T11:56:38.896 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:38.890+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.895303+0000) 2019-10-13T11:56:38.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:38.894+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:38.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:38.899+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:38.900370+0000) 2019-10-13T11:56:41.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:41.847+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:41.850904+0000) 2019-10-13T11:56:43.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:43.779+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.782265+0000) 2019-10-13T11:56:43.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:43.898+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.902294+0000) 2019-10-13T11:56:43.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:43.899+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.902916+0000) 2019-10-13T11:56:43.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:43.898+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.903973+0000) 2019-10-13T11:56:43.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:43.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.903480+0000) 2019-10-13T11:56:43.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:43.898+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.904561+0000) 2019-10-13T11:56:43.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:43.898+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.905099+0000) 2019-10-13T11:56:43.906 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:43.898+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.905410+0000) 2019-10-13T11:56:43.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:43.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:43.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:43.911+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:43.912214+0000) 2019-10-13T11:56:44.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:44.170+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.174046+0000) 2019-10-13T11:56:44.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:44.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:44.460114+0000) 2019-10-13T11:56:47.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:47.830+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:47.831180+0000) 2019-10-13T11:56:48.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:48.050+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.057570+0000) 2019-10-13T11:56:48.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:48.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.078666+0000) 2019-10-13T11:56:48.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:48.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.599371+0000) 2019-10-13T11:56:48.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:48.906+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.913300+0000) 2019-10-13T11:56:48.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:48.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.913668+0000) 2019-10-13T11:56:48.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:48.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.914528+0000) 2019-10-13T11:56:48.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:48.910+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.915095+0000) 2019-10-13T11:56:48.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:48.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.915485+0000) 2019-10-13T11:56:48.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:48.910+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.915824+0000) 2019-10-13T11:56:48.917 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:48.910+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.916120+0000) 2019-10-13T11:56:48.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:48.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:48.924 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:48.919+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:48.923920+0000) 2019-10-13T11:56:51.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:51.847+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:51.851093+0000) 2019-10-13T11:56:53.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:53.787+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.788909+0000) 2019-10-13T11:56:53.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:53.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.925450+0000) 2019-10-13T11:56:53.927 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:53.918+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.925491+0000) 2019-10-13T11:56:53.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:53.918+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.925155+0000) 2019-10-13T11:56:53.927 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:53.922+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.925898+0000) 2019-10-13T11:56:53.927 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:53.922+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.926391+0000) 2019-10-13T11:56:53.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:53.922+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.927376+0000) 2019-10-13T11:56:53.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:53.922+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.928420+0000) 2019-10-13T11:56:53.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:53.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:53.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:53.931+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:53.934520+0000) 2019-10-13T11:56:54.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:54.170+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.174227+0000) 2019-10-13T11:56:54.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:54.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:54.460302+0000) 2019-10-13T11:56:57.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:57.826+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:57.831383+0000) 2019-10-13T11:56:58.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:58.054+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.057819+0000) 2019-10-13T11:56:58.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:58.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.078862+0000) 2019-10-13T11:56:58.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:58.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.599598+0000) 2019-10-13T11:56:58.936 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:56:58.930+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.935727+0000) 2019-10-13T11:56:58.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:56:58.930+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.936098+0000) 2019-10-13T11:56:58.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:56:58.930+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.936604+0000) 2019-10-13T11:56:58.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:56:58.930+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.937125+0000) 2019-10-13T11:56:58.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:56:58.935+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.937536+0000) 2019-10-13T11:56:58.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:56:58.934+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.938459+0000) 2019-10-13T11:56:58.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:56:58.934+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.938590+0000) 2019-10-13T11:56:58.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:56:58.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:56:58.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:56:58.943+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:56:58.944582+0000) 2019-10-13T11:57:00.800 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:57:00.800 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:00.804 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:57:00.804 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:01.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:01.847+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:01.851312+0000) 2019-10-13T11:57:03.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:03.791+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.795535+0000) 2019-10-13T11:57:03.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:03.942+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.946647+0000) 2019-10-13T11:57:03.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:03.942+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.947115+0000) 2019-10-13T11:57:03.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:03.942+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.947295+0000) 2019-10-13T11:57:03.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:03.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.948455+0000) 2019-10-13T11:57:03.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:03.947+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.949154+0000) 2019-10-13T11:57:03.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:03.942+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.949625+0000) 2019-10-13T11:57:03.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:03.946+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.950134+0000) 2019-10-13T11:57:03.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:03.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:03.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:03.951+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:03.955437+0000) 2019-10-13T11:57:04.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:04.170+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:04.174426+0000) 2019-10-13T11:57:04.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:04.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:04.460504+0000) 2019-10-13T11:57:07.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:07.830+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:07.831616+0000) 2019-10-13T11:57:08.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:08.054+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.058044+0000) 2019-10-13T11:57:08.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:08.078+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.079093+0000) 2019-10-13T11:57:08.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:08.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.599824+0000) 2019-10-13T11:57:08.958 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:08.950+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.957154+0000) 2019-10-13T11:57:08.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:08.950+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.957580+0000) 2019-10-13T11:57:08.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:08.954+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.957893+0000) 2019-10-13T11:57:08.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:08.955+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.958359+0000) 2019-10-13T11:57:08.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:08.954+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.958681+0000) 2019-10-13T11:57:08.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:08.954+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.959337+0000) 2019-10-13T11:57:08.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:08.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.960307+0000) 2019-10-13T11:57:08.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:08.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:08.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:08.963+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:08.966070+0000) 2019-10-13T11:57:11.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:11.847+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:11.851527+0000) 2019-10-13T11:57:13.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:13.799+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.802091+0000) 2019-10-13T11:57:13.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:13.962+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.967659+0000) 2019-10-13T11:57:13.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:13.966+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.970007+0000) 2019-10-13T11:57:13.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:13.966+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.970104+0000) 2019-10-13T11:57:13.971 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:13.966+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.970573+0000) 2019-10-13T11:57:13.971 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:13.966+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.970953+0000) 2019-10-13T11:57:13.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:13.967+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.971306+0000) 2019-10-13T11:57:13.972 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:13.966+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.972127+0000) 2019-10-13T11:57:13.976 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:13.970+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:13.978 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:13.975+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:13.977884+0000) 2019-10-13T11:57:14.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:14.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:14.174611+0000) 2019-10-13T11:57:14.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:14.454+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:14.460640+0000) 2019-10-13T11:57:17.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:17.830+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:17.831832+0000) 2019-10-13T11:57:18.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:18.054+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.058248+0000) 2019-10-13T11:57:18.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:18.074+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.079293+0000) 2019-10-13T11:57:18.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:18.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.600074+0000) 2019-10-13T11:57:18.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:18.974+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.978173+0000) 2019-10-13T11:57:18.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:18.974+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.978529+0000) 2019-10-13T11:57:18.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:18.974+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.981010+0000) 2019-10-13T11:57:18.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:18.978+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.982932+0000) 2019-10-13T11:57:18.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:18.979+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.983410+0000) 2019-10-13T11:57:18.985 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:18.978+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.984329+0000) 2019-10-13T11:57:18.985 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:18.978+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.984686+0000) 2019-10-13T11:57:18.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:18.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:18.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:18.987+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:18.989523+0000) 2019-10-13T11:57:21.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:21.851+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:21.851740+0000) 2019-10-13T11:57:23.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:23.807+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.811138+0000) 2019-10-13T11:57:23.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:23.986+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.989385+0000) 2019-10-13T11:57:23.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:23.986+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.989988+0000) 2019-10-13T11:57:23.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:23.986+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.990847+0000) 2019-10-13T11:57:23.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:23.987+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.991611+0000) 2019-10-13T11:57:23.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:23.986+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.992014+0000) 2019-10-13T11:57:23.993 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:23.986+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.992226+0000) 2019-10-13T11:57:23.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:23.986+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:23.992837+0000) 2019-10-13T11:57:23.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:23.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149897 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:24.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:23.999+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:24.000238+0000) 2019-10-13T11:57:24.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:24.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:24.174796+0000) 2019-10-13T11:57:24.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:24.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:24.460820+0000) 2019-10-13T11:57:27.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:27.830+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:27.832056+0000) 2019-10-13T11:57:28.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:28.054+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.058449+0000) 2019-10-13T11:57:28.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:28.078+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.079519+0000) 2019-10-13T11:57:28.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:28.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:28.600269+0000) 2019-10-13T11:57:29.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:28.994+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.000948+0000) 2019-10-13T11:57:29.002 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:28.994+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.001278+0000) 2019-10-13T11:57:29.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:28.994+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.001673+0000) 2019-10-13T11:57:29.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:28.998+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.002249+0000) 2019-10-13T11:57:29.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:28.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.003080+0000) 2019-10-13T11:57:29.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:28.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.003929+0000) 2019-10-13T11:57:29.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:28.998+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.004594+0000) 2019-10-13T11:57:29.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:29.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:29.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:29.007+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:29.010603+0000) 2019-10-13T11:57:30.964 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:57:30.964 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:30.968 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:57:30.968 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:57:31.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:31.846+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:31.851930+0000) 2019-10-13T11:57:33.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:33.814+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:33.817725+0000) 2019-10-13T11:57:34.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:34.005+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.012417+0000) 2019-10-13T11:57:34.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:34.005+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.013052+0000) 2019-10-13T11:57:34.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:34.005+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.013587+0000) 2019-10-13T11:57:34.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:34.009+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.014282+0000) 2019-10-13T11:57:34.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:34.009+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.014813+0000) 2019-10-13T11:57:34.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:34.010+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.015292+0000) 2019-10-13T11:57:34.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:34.009+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.015450+0000) 2019-10-13T11:57:34.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:34.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:34.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:34.018+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.020873+0000) 2019-10-13T11:57:34.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:34.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.175008+0000) 2019-10-13T11:57:34.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:34.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:34.460993+0000) 2019-10-13T11:57:37.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:37.829+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:37.832258+0000) 2019-10-13T11:57:38.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:38.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.058691+0000) 2019-10-13T11:57:38.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:38.077+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.079721+0000) 2019-10-13T11:57:38.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:38.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:38.600460+0000) 2019-10-13T11:57:39.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:39.017+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.023039+0000) 2019-10-13T11:57:39.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:39.017+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.023549+0000) 2019-10-13T11:57:39.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:39.017+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.023938+0000) 2019-10-13T11:57:39.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:39.017+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.024889+0000) 2019-10-13T11:57:39.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:39.017+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.025055+0000) 2019-10-13T11:57:39.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:39.022+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.025406+0000) 2019-10-13T11:57:39.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:39.021+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.026275+0000) 2019-10-13T11:57:39.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:39.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:39.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:39.030+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:39.032614+0000) 2019-10-13T11:57:41.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:41.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:41.852130+0000) 2019-10-13T11:57:43.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:43.822+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:43.823976+0000) 2019-10-13T11:57:44.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:44.029+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.033976+0000) 2019-10-13T11:57:44.035 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:44.029+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.034317+0000) 2019-10-13T11:57:44.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:44.029+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.035025+0000) 2019-10-13T11:57:44.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:44.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.035481+0000) 2019-10-13T11:57:44.036 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:44.029+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.035805+0000) 2019-10-13T11:57:44.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:44.034+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.036759+0000) 2019-10-13T11:57:44.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:44.033+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.038095+0000) 2019-10-13T11:57:44.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:44.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:44.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:44.038+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.042674+0000) 2019-10-13T11:57:44.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:44.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.175199+0000) 2019-10-13T11:57:44.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:44.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:44.461168+0000) 2019-10-13T11:57:47.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:47.829+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:47.832482+0000) 2019-10-13T11:57:48.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:48.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.058842+0000) 2019-10-13T11:57:48.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:48.077+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.079915+0000) 2019-10-13T11:57:48.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:48.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:48.600680+0000) 2019-10-13T11:57:49.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:49.045+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.045990+0000) 2019-10-13T11:57:49.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:49.045+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.046391+0000) 2019-10-13T11:57:49.048 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:49.042+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.047206+0000) 2019-10-13T11:57:49.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:49.045+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.047921+0000) 2019-10-13T11:57:49.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:49.045+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.048430+0000) 2019-10-13T11:57:49.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:49.045+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.049095+0000) 2019-10-13T11:57:49.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:49.049+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.050653+0000) 2019-10-13T11:57:49.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:49.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:49.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:49.050+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:49.053469+0000) 2019-10-13T11:57:51.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:51.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:51.852319+0000) 2019-10-13T11:57:53.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:53.826+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:53.830543+0000) 2019-10-13T11:57:54.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:54.053+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.056090+0000) 2019-10-13T11:57:54.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:54.053+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.056995+0000) 2019-10-13T11:57:54.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:54.057+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.057990+0000) 2019-10-13T11:57:54.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:54.057+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.058411+0000) 2019-10-13T11:57:54.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:54.054+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.058787+0000) 2019-10-13T11:57:54.060 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:54.057+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.059347+0000) 2019-10-13T11:57:54.060 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:54.057+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.060114+0000) 2019-10-13T11:57:54.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:54.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:54.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:54.062+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.065278+0000) 2019-10-13T11:57:54.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:54.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.175381+0000) 2019-10-13T11:57:54.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:54.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:54.461366+0000) 2019-10-13T11:57:57.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:57.829+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:57.832704+0000) 2019-10-13T11:57:58.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:58.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.059054+0000) 2019-10-13T11:57:58.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:58.077+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.080139+0000) 2019-10-13T11:57:58.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:58.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:58.600879+0000) 2019-10-13T11:57:59.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:57:59.062+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.067241+0000) 2019-10-13T11:57:59.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:57:59.065+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.067892+0000) 2019-10-13T11:57:59.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:57:59.065+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.068517+0000) 2019-10-13T11:57:59.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:57:59.065+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.069405+0000) 2019-10-13T11:57:59.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:57:59.069+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.070222+0000) 2019-10-13T11:57:59.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:57:59.069+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.070579+0000) 2019-10-13T11:57:59.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:57:59.069+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.070899+0000) 2019-10-13T11:57:59.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:57:59.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:57:59.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:57:59.074+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:57:59.076131+0000) 2019-10-13T11:58:01.111 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:58:01.111 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:01.115 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:58:01.115 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:01.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:01.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:01.852485+0000) 2019-10-13T11:58:03.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:03.834+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:03.836967+0000) 2019-10-13T11:58:04.078 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:04.073+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.077533+0000) 2019-10-13T11:58:04.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:04.077+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.077952+0000) 2019-10-13T11:58:04.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:04.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.078359+0000) 2019-10-13T11:58:04.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:04.074+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.079103+0000) 2019-10-13T11:58:04.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:04.077+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.079217+0000) 2019-10-13T11:58:04.080 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:04.077+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.079641+0000) 2019-10-13T11:58:04.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:04.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.080243+0000) 2019-10-13T11:58:04.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:04.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:04.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:04.082+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.086161+0000) 2019-10-13T11:58:04.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:04.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.175564+0000) 2019-10-13T11:58:04.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:04.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:04.461539+0000) 2019-10-13T11:58:07.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:07.829+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:07.832943+0000) 2019-10-13T11:58:08.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:08.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.059279+0000) 2019-10-13T11:58:08.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:08.077+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.080335+0000) 2019-10-13T11:58:08.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:08.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:08.601093+0000) 2019-10-13T11:58:09.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:09.085+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.088613+0000) 2019-10-13T11:58:09.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:09.085+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.089141+0000) 2019-10-13T11:58:09.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:09.089+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.090114+0000) 2019-10-13T11:58:09.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:09.086+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.090603+0000) 2019-10-13T11:58:09.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:09.089+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.090259+0000) 2019-10-13T11:58:09.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:09.089+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.090965+0000) 2019-10-13T11:58:09.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:09.089+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.092501+0000) 2019-10-13T11:58:09.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:09.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:09.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:09.094+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:09.097609+0000) 2019-10-13T11:58:11.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:11.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:11.852674+0000) 2019-10-13T11:58:13.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:13.838+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:13.843418+0000) 2019-10-13T11:58:14.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:14.094+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.099778+0000) 2019-10-13T11:58:14.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:14.097+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.100227+0000) 2019-10-13T11:58:14.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:14.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.101201+0000) 2019-10-13T11:58:14.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:14.097+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.101415+0000) 2019-10-13T11:58:14.102 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:14.101+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.102079+0000) 2019-10-13T11:58:14.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:14.101+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.102863+0000) 2019-10-13T11:58:14.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:14.101+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.103252+0000) 2019-10-13T11:58:14.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:14.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:14.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:14.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.113344+0000) 2019-10-13T11:58:14.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:14.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.175741+0000) 2019-10-13T11:58:14.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:14.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:14.461719+0000) 2019-10-13T11:58:17.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:17.829+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:17.833170+0000) 2019-10-13T11:58:18.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:18.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.059540+0000) 2019-10-13T11:58:18.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:18.077+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.080531+0000) 2019-10-13T11:58:18.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:18.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:18.601290+0000) 2019-10-13T11:58:19.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:19.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.110103+0000) 2019-10-13T11:58:19.112 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:19.109+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.111526+0000) 2019-10-13T11:58:19.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:19.109+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.112146+0000) 2019-10-13T11:58:19.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:19.109+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.112192+0000) 2019-10-13T11:58:19.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:19.110+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.112730+0000) 2019-10-13T11:58:19.114 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:19.109+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.113115+0000) 2019-10-13T11:58:19.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:19.113+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.113877+0000) 2019-10-13T11:58:19.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:19.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:19.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:19.114+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:19.117652+0000) 2019-10-13T11:58:21.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:21.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:21.852864+0000) 2019-10-13T11:58:23.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:23.846+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:23.849529+0000) 2019-10-13T11:58:24.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:24.121+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.122551+0000) 2019-10-13T11:58:24.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:24.121+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.123188+0000) 2019-10-13T11:58:24.124 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:24.121+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.123557+0000) 2019-10-13T11:58:24.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:24.121+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.124384+0000) 2019-10-13T11:58:24.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:24.125+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.126364+0000) 2019-10-13T11:58:24.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:24.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.127458+0000) 2019-10-13T11:58:24.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:24.125+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.128252+0000) 2019-10-13T11:58:24.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:24.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:24.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:24.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.130654+0000) 2019-10-13T11:58:24.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:24.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.175929+0000) 2019-10-13T11:58:24.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:24.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:24.461939+0000) 2019-10-13T11:58:27.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:27.828+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:27.833321+0000) 2019-10-13T11:58:28.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:28.056+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.059768+0000) 2019-10-13T11:58:28.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:28.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.080739+0000) 2019-10-13T11:58:28.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:28.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:28.601534+0000) 2019-10-13T11:58:29.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:29.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.131957+0000) 2019-10-13T11:58:29.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:29.128+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.132441+0000) 2019-10-13T11:58:29.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:29.128+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.133146+0000) 2019-10-13T11:58:29.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:29.128+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.133446+0000) 2019-10-13T11:58:29.135 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:29.130+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.134865+0000) 2019-10-13T11:58:29.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:29.132+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.136290+0000) 2019-10-13T11:58:29.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:29.132+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.136533+0000) 2019-10-13T11:58:29.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:29.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:29.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:29.134+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:29.139567+0000) 2019-10-13T11:58:31.241 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:58:31.241 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:31.245 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:58:31.245 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:58:31.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:31.850+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:31.853059+0000) 2019-10-13T11:58:33.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:33.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:33.855882+0000) 2019-10-13T11:58:34.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:34.140+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.141896+0000) 2019-10-13T11:58:34.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:34.140+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.142163+0000) 2019-10-13T11:58:34.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:34.140+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.142893+0000) 2019-10-13T11:58:34.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:34.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.143112+0000) 2019-10-13T11:58:34.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:34.140+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.143598+0000) 2019-10-13T11:58:34.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:34.140+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.144034+0000) 2019-10-13T11:58:34.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:34.142+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.144395+0000) 2019-10-13T11:58:34.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:34.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150323 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:34.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:34.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.151389+0000) 2019-10-13T11:58:34.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:34.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.176063+0000) 2019-10-13T11:58:34.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:34.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:34.462133+0000) 2019-10-13T11:58:37.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:37.828+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:37.833483+0000) 2019-10-13T11:58:38.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:38.056+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.059968+0000) 2019-10-13T11:58:38.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:38.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.080937+0000) 2019-10-13T11:58:38.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:38.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:38.601772+0000) 2019-10-13T11:58:39.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:39.148+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.152404+0000) 2019-10-13T11:58:39.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:39.148+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.153141+0000) 2019-10-13T11:58:39.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:39.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.153589+0000) 2019-10-13T11:58:39.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:39.152+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.154220+0000) 2019-10-13T11:58:39.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:39.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.154589+0000) 2019-10-13T11:58:39.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:39.152+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.156522+0000) 2019-10-13T11:58:39.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:39.152+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.156834+0000) 2019-10-13T11:58:39.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:39.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:39.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:39.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:39.162495+0000) 2019-10-13T11:58:41.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:41.849+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:41.853260+0000) 2019-10-13T11:58:43.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:43.861+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:43.862420+0000) 2019-10-13T11:58:44.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:44.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.162932+0000) 2019-10-13T11:58:44.164 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:44.160+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.163805+0000) 2019-10-13T11:58:44.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:44.160+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.164191+0000) 2019-10-13T11:58:44.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:44.160+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.164699+0000) 2019-10-13T11:58:44.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:44.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.166870+0000) 2019-10-13T11:58:44.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:44.164+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.168713+0000) 2019-10-13T11:58:44.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:44.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.168551+0000) 2019-10-13T11:58:44.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:44.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:44.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:44.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.172619+0000) 2019-10-13T11:58:44.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:44.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.176192+0000) 2019-10-13T11:58:44.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:44.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:44.462354+0000) 2019-10-13T11:58:47.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:47.828+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:47.833699+0000) 2019-10-13T11:58:48.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:48.056+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.060160+0000) 2019-10-13T11:58:48.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:48.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.081164+0000) 2019-10-13T11:58:48.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:48.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:48.601967+0000) 2019-10-13T11:58:49.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:49.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.175449+0000) 2019-10-13T11:58:49.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:49.173+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.176359+0000) 2019-10-13T11:58:49.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:49.172+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.176929+0000) 2019-10-13T11:58:49.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:49.172+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.177180+0000) 2019-10-13T11:58:49.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:49.176+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.177733+0000) 2019-10-13T11:58:49.179 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:49.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.178030+0000) 2019-10-13T11:58:49.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:49.176+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.178527+0000) 2019-10-13T11:58:49.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:49.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:49.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:49.181+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:49.186856+0000) 2019-10-13T11:58:51.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:51.849+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:51.853454+0000) 2019-10-13T11:58:53.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:53.865+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:53.868982+0000) 2019-10-13T11:58:54.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:54.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.176354+0000) 2019-10-13T11:58:54.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:54.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.186513+0000) 2019-10-13T11:58:54.187 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:54.184+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.187086+0000) 2019-10-13T11:58:54.188 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:54.184+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.187282+0000) 2019-10-13T11:58:54.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:54.184+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.189176+0000) 2019-10-13T11:58:54.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:54.184+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.189526+0000) 2019-10-13T11:58:54.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:54.188+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.191342+0000) 2019-10-13T11:58:54.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:54.189+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.192624+0000) 2019-10-13T11:58:54.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:54.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:54.200 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:54.193+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.199510+0000) 2019-10-13T11:58:54.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:54.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:54.462533+0000) 2019-10-13T11:58:57.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:57.832+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:57.833924+0000) 2019-10-13T11:58:58.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:58.056+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.060351+0000) 2019-10-13T11:58:58.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:58.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.081361+0000) 2019-10-13T11:58:58.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:58.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:58.602160+0000) 2019-10-13T11:58:59.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:58:59.192+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.197183+0000) 2019-10-13T11:58:59.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:58:59.192+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.197293+0000) 2019-10-13T11:58:59.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:58:59.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.198171+0000) 2019-10-13T11:58:59.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:58:59.196+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.198599+0000) 2019-10-13T11:58:59.199 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:58:59.193+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.198981+0000) 2019-10-13T11:58:59.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:58:59.196+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.199328+0000) 2019-10-13T11:58:59.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:58:59.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.199858+0000) 2019-10-13T11:58:59.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:58:59.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:58:59.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:58:59.205+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:58:59.208587+0000) 2019-10-13T11:59:01.369 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:59:01.369 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:01.373 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:59:01.373 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:01.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:01.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:01.853658+0000) 2019-10-13T11:59:03.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:03.873+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:03.875482+0000) 2019-10-13T11:59:04.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:04.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.176560+0000) 2019-10-13T11:59:04.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:04.201+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.207979+0000) 2019-10-13T11:59:04.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:04.204+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.209389+0000) 2019-10-13T11:59:04.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:04.208+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.210288+0000) 2019-10-13T11:59:04.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:04.208+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.210424+0000) 2019-10-13T11:59:04.211 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:04.208+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.210956+0000) 2019-10-13T11:59:04.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:04.208+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.211484+0000) 2019-10-13T11:59:04.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:04.208+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.212438+0000) 2019-10-13T11:59:04.217 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:04.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:04.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:04.213+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.217727+0000) 2019-10-13T11:59:04.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:04.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:04.462747+0000) 2019-10-13T11:59:07.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:07.832+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:07.834123+0000) 2019-10-13T11:59:08.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:08.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.060584+0000) 2019-10-13T11:59:08.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:08.076+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.081564+0000) 2019-10-13T11:59:08.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:08.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:08.602357+0000) 2019-10-13T11:59:09.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:09.217+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.220416+0000) 2019-10-13T11:59:09.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:09.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.220866+0000) 2019-10-13T11:59:09.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:09.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.221325+0000) 2019-10-13T11:59:09.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:09.220+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.221999+0000) 2019-10-13T11:59:09.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:09.220+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.222850+0000) 2019-10-13T11:59:09.224 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:09.220+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.223635+0000) 2019-10-13T11:59:09.224 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:09.220+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.224117+0000) 2019-10-13T11:59:09.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:09.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:09.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:09.225+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:09.228972+0000) 2019-10-13T11:59:11.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:11.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:11.853854+0000) 2019-10-13T11:59:13.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:13.881+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:13.881969+0000) 2019-10-13T11:59:14.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:14.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.176781+0000) 2019-10-13T11:59:14.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:14.228+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.231160+0000) 2019-10-13T11:59:14.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:14.228+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.231891+0000) 2019-10-13T11:59:14.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:14.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.232034+0000) 2019-10-13T11:59:14.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:14.228+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.232944+0000) 2019-10-13T11:59:14.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:14.229+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.233568+0000) 2019-10-13T11:59:14.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:14.228+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.233676+0000) 2019-10-13T11:59:14.235 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:14.232+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.234039+0000) 2019-10-13T11:59:14.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:14.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:14.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:14.233+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.239564+0000) 2019-10-13T11:59:14.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:14.460+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:14.462956+0000) 2019-10-13T11:59:17.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:17.832+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:17.834339+0000) 2019-10-13T11:59:18.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:18.056+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.060794+0000) 2019-10-13T11:59:18.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:18.080+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.081821+0000) 2019-10-13T11:59:18.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:18.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:18.602588+0000) 2019-10-13T11:59:19.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:19.236+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.241589+0000) 2019-10-13T11:59:19.243 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:19.240+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.242097+0000) 2019-10-13T11:59:19.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:19.240+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.242662+0000) 2019-10-13T11:59:19.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:19.240+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.243461+0000) 2019-10-13T11:59:19.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:19.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.243844+0000) 2019-10-13T11:59:19.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:19.240+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.244586+0000) 2019-10-13T11:59:19.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:19.240+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.244911+0000) 2019-10-13T11:59:19.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:19.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:19.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:19.245+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:19.250638+0000) 2019-10-13T11:59:21.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:21.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:21.854065+0000) 2019-10-13T11:59:23.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:23.885+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:23.887951+0000) 2019-10-13T11:59:24.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:24.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.177007+0000) 2019-10-13T11:59:24.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:24.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.253119+0000) 2019-10-13T11:59:24.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:24.251+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.254966+0000) 2019-10-13T11:59:24.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:24.251+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.255376+0000) 2019-10-13T11:59:24.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:24.251+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.255618+0000) 2019-10-13T11:59:24.259 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:24.255+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.258003+0000) 2019-10-13T11:59:24.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:24.253+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.259108+0000) 2019-10-13T11:59:24.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:24.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:24.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:24.255+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.260752+0000) 2019-10-13T11:59:24.263 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:24.257+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.261668+0000) 2019-10-13T11:59:24.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:24.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:24.463133+0000) 2019-10-13T11:59:27.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:27.831+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:27.834537+0000) 2019-10-13T11:59:28.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:28.059+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.061008+0000) 2019-10-13T11:59:28.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:28.079+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.082017+0000) 2019-10-13T11:59:28.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:28.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:28.602777+0000) 2019-10-13T11:59:29.264 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:29.259+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.263216+0000) 2019-10-13T11:59:29.265 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:29.259+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.264729+0000) 2019-10-13T11:59:29.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:29.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.265008+0000) 2019-10-13T11:59:29.266 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:29.259+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.265354+0000) 2019-10-13T11:59:29.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:29.263+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.265861+0000) 2019-10-13T11:59:29.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:29.263+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.266830+0000) 2019-10-13T11:59:29.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:29.261+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.267566+0000) 2019-10-13T11:59:29.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:29.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:29.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:29.269+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:29.273669+0000) 2019-10-13T11:59:31.511 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T11:59:31.511 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:31.515 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T11:59:31.515 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T11:59:31.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:31.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:31.854310+0000) 2019-10-13T11:59:33.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:33.893+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:33.894525+0000) 2019-10-13T11:59:34.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:34.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.177211+0000) 2019-10-13T11:59:34.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:34.267+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.273657+0000) 2019-10-13T11:59:34.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:34.271+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.274361+0000) 2019-10-13T11:59:34.275 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:34.271+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.275136+0000) 2019-10-13T11:59:34.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:34.271+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.275390+0000) 2019-10-13T11:59:34.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:34.271+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.275947+0000) 2019-10-13T11:59:34.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:34.271+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.276666+0000) 2019-10-13T11:59:34.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:34.273+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.277430+0000) 2019-10-13T11:59:34.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:34.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:34.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:34.281+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.285823+0000) 2019-10-13T11:59:34.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:34.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:34.463319+0000) 2019-10-13T11:59:37.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:37.831+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:37.834738+0000) 2019-10-13T11:59:38.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:38.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.061234+0000) 2019-10-13T11:59:38.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:38.079+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.082214+0000) 2019-10-13T11:59:38.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:38.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:38.602986+0000) 2019-10-13T11:59:39.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:39.279+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.284913+0000) 2019-10-13T11:59:39.286 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:39.279+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.285567+0000) 2019-10-13T11:59:39.287 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:39.283+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.286491+0000) 2019-10-13T11:59:39.289 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:39.283+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.288547+0000) 2019-10-13T11:59:39.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:39.283+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.289056+0000) 2019-10-13T11:59:39.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:39.283+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.289727+0000) 2019-10-13T11:59:39.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:39.285+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.290448+0000) 2019-10-13T11:59:39.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:39.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:39.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:39.289+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:39.295375+0000) 2019-10-13T11:59:41.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:41.848+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:41.854513+0000) 2019-10-13T11:59:43.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:43.896+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:43.900972+0000) 2019-10-13T11:59:44.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:44.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.177429+0000) 2019-10-13T11:59:44.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:44.291+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.294995+0000) 2019-10-13T11:59:44.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:44.291+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.297418+0000) 2019-10-13T11:59:44.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:44.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.297712+0000) 2019-10-13T11:59:44.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:44.295+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.298146+0000) 2019-10-13T11:59:44.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:44.295+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.299067+0000) 2019-10-13T11:59:44.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:44.295+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.299494+0000) 2019-10-13T11:59:44.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:44.296+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.300407+0000) 2019-10-13T11:59:44.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:44.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150748 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:44.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:44.300+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.305743+0000) 2019-10-13T11:59:44.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:44.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:44.463559+0000) 2019-10-13T11:59:47.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:47.831+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:47.834944+0000) 2019-10-13T11:59:48.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:48.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.061432+0000) 2019-10-13T11:59:48.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:48.079+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.082425+0000) 2019-10-13T11:59:48.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:48.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:48.603175+0000) 2019-10-13T11:59:49.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:49.303+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.305967+0000) 2019-10-13T11:59:49.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:49.300+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.306526+0000) 2019-10-13T11:59:49.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:49.303+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.306818+0000) 2019-10-13T11:59:49.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:49.303+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.308781+0000) 2019-10-13T11:59:49.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:49.307+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.310734+0000) 2019-10-13T11:59:49.312 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:49.307+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.311706+0000) 2019-10-13T11:59:49.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:49.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150780 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:49.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:49.311+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.313871+0000) 2019-10-13T11:59:49.316 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:49.308+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:49.315420+0000) 2019-10-13T11:59:51.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:51.848+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:51.854735+0000) 2019-10-13T11:59:53.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:53.900+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:53.907434+0000) 2019-10-13T11:59:54.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:54.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.177653+0000) 2019-10-13T11:59:54.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:54.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.316675+0000) 2019-10-13T11:59:54.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:54.311+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.317185+0000) 2019-10-13T11:59:54.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:54.312+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.318747+0000) 2019-10-13T11:59:54.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:54.315+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.319061+0000) 2019-10-13T11:59:54.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:54.315+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.319220+0000) 2019-10-13T11:59:54.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:54.315+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.319757+0000) 2019-10-13T11:59:54.322 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:54.315+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.320246+0000) 2019-10-13T11:59:54.324 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:54.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:54.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:54.320+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.325443+0000) 2019-10-13T11:59:54.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:54.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:54.463736+0000) 2019-10-13T11:59:57.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:57.831+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:57.835225+0000) 2019-10-13T11:59:58.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:58.055+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.061665+0000) 2019-10-13T11:59:58.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:58.079+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.082661+0000) 2019-10-13T11:59:58.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:58.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:58.603399+0000) 2019-10-13T11:59:59.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T11:59:59.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.328011+0000) 2019-10-13T11:59:59.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T11:59:59.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.328457+0000) 2019-10-13T11:59:59.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T11:59:59.323+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.328967+0000) 2019-10-13T11:59:59.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T11:59:59.323+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.329435+0000) 2019-10-13T11:59:59.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T11:59:59.327+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.330955+0000) 2019-10-13T11:59:59.332 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T11:59:59.324+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.330946+0000) 2019-10-13T11:59:59.332 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T11:59:59.327+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.331738+0000) 2019-10-13T11:59:59.335 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T11:59:59.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T11:59:59.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T11:59:59.332+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T10:59:59.337844+0000) 2019-10-13T12:00:01.632 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:00:01.632 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:01.636 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:00:01.636 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:01.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:01.852+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:01.854932+0000) 2019-10-13T12:00:03.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:03.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:03.913959+0000) 2019-10-13T12:00:04.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:04.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.177938+0000) 2019-10-13T12:00:04.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:04.335+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.338338+0000) 2019-10-13T12:00:04.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:04.335+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.338665+0000) 2019-10-13T12:00:04.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:04.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.339080+0000) 2019-10-13T12:00:04.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:04.335+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.341463+0000) 2019-10-13T12:00:04.342 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:04.335+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.339417+0000) 2019-10-13T12:00:04.345 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:04.339+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.344006+0000) 2019-10-13T12:00:04.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:04.340+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.346109+0000) 2019-10-13T12:00:04.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:04.340+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.346261+0000) 2019-10-13T12:00:04.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:04.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:04.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:04.459+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:04.463930+0000) 2019-10-13T12:00:07.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:07.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:07.835429+0000) 2019-10-13T12:00:08.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:08.059+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.061893+0000) 2019-10-13T12:00:08.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:08.079+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.082862+0000) 2019-10-13T12:00:08.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:08.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:08.603620+0000) 2019-10-13T12:00:09.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:09.343+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.349268+0000) 2019-10-13T12:00:09.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:09.347+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.350858+0000) 2019-10-13T12:00:09.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:09.347+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.352758+0000) 2019-10-13T12:00:09.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:09.347+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.353130+0000) 2019-10-13T12:00:09.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:09.347+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.353723+0000) 2019-10-13T12:00:09.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:09.351+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.355026+0000) 2019-10-13T12:00:09.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:09.348+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.356045+0000) 2019-10-13T12:00:09.357 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:09.351+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:09.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:09.352+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:09.358039+0000) 2019-10-13T12:00:11.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:11.852+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:11.855164+0000) 2019-10-13T12:00:13.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:13.916+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:13.920471+0000) 2019-10-13T12:00:14.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:14.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.178139+0000) 2019-10-13T12:00:14.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:14.355+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.359611+0000) 2019-10-13T12:00:14.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:14.355+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.360318+0000) 2019-10-13T12:00:14.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:14.355+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.360634+0000) 2019-10-13T12:00:14.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:14.355+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.361078+0000) 2019-10-13T12:00:14.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:14.359+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.361985+0000) 2019-10-13T12:00:14.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:14.359+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.362623+0000) 2019-10-13T12:00:14.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:14.356+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.362681+0000) 2019-10-13T12:00:14.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:14.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:14.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:14.364+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.369832+0000) 2019-10-13T12:00:14.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:14.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:14.464135+0000) 2019-10-13T12:00:17.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:17.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:17.835624+0000) 2019-10-13T12:00:18.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:18.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.062088+0000) 2019-10-13T12:00:18.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:18.082+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.083068+0000) 2019-10-13T12:00:18.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:18.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:18.603815+0000) 2019-10-13T12:00:19.372 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:19.366+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.371351+0000) 2019-10-13T12:00:19.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:19.366+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.371831+0000) 2019-10-13T12:00:19.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:19.366+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.372170+0000) 2019-10-13T12:00:19.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:19.368+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.373046+0000) 2019-10-13T12:00:19.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:19.366+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.373213+0000) 2019-10-13T12:00:19.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:19.370+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.374049+0000) 2019-10-13T12:00:19.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:19.370+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.374513+0000) 2019-10-13T12:00:19.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:19.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:19.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:19.372+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:19.379103+0000) 2019-10-13T12:00:21.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:21.852+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:21.855356+0000) 2019-10-13T12:00:23.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:23.924+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:23.926992+0000) 2019-10-13T12:00:24.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:24.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.178358+0000) 2019-10-13T12:00:24.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:24.374+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.381009+0000) 2019-10-13T12:00:24.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:24.378+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.381874+0000) 2019-10-13T12:00:24.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:24.378+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.382716+0000) 2019-10-13T12:00:24.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:24.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.383441+0000) 2019-10-13T12:00:24.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:24.378+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.383700+0000) 2019-10-13T12:00:24.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:24.380+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.384392+0000) 2019-10-13T12:00:24.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:24.378+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.384894+0000) 2019-10-13T12:00:24.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:24.380+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.387980+0000) 2019-10-13T12:00:24.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:24.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:24.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:24.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:24.464315+0000) 2019-10-13T12:00:27.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:27.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:27.835827+0000) 2019-10-13T12:00:28.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:28.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.062306+0000) 2019-10-13T12:00:28.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:28.082+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.083272+0000) 2019-10-13T12:00:28.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:28.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:28.603966+0000) 2019-10-13T12:00:29.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:29.386+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.391315+0000) 2019-10-13T12:00:29.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:29.386+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.392102+0000) 2019-10-13T12:00:29.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:29.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.392415+0000) 2019-10-13T12:00:29.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:29.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.393307+0000) 2019-10-13T12:00:29.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:29.386+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.393661+0000) 2019-10-13T12:00:29.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:29.390+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.394007+0000) 2019-10-13T12:00:29.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:29.388+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.395535+0000) 2019-10-13T12:00:29.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:29.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:29.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:29.396+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:29.400880+0000) 2019-10-13T12:00:31.739 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:00:31.739 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:31.743 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:00:31.743 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:00:31.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:31.852+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:31.855553+0000) 2019-10-13T12:00:33.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:33.932+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:33.933600+0000) 2019-10-13T12:00:34.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:34.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.178559+0000) 2019-10-13T12:00:34.403 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:34.398+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.402128+0000) 2019-10-13T12:00:34.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:34.396+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.403111+0000) 2019-10-13T12:00:34.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:34.398+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.403519+0000) 2019-10-13T12:00:34.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:34.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.404560+0000) 2019-10-13T12:00:34.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:34.398+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.405230+0000) 2019-10-13T12:00:34.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:34.402+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.405807+0000) 2019-10-13T12:00:34.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:34.402+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.406253+0000) 2019-10-13T12:00:34.410 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:34.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:34.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:34.404+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.410375+0000) 2019-10-13T12:00:34.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:34.462+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:34.464511+0000) 2019-10-13T12:00:37.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:37.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:37.836029+0000) 2019-10-13T12:00:38.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:38.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.062545+0000) 2019-10-13T12:00:38.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:38.078+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.083491+0000) 2019-10-13T12:00:38.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:38.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:38.604204+0000) 2019-10-13T12:00:39.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:39.406+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.412955+0000) 2019-10-13T12:00:39.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:39.412+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.413750+0000) 2019-10-13T12:00:39.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:39.410+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.414730+0000) 2019-10-13T12:00:39.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:39.410+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.415369+0000) 2019-10-13T12:00:39.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:39.410+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.415656+0000) 2019-10-13T12:00:39.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:39.410+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.415334+0000) 2019-10-13T12:00:39.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:39.410+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.416670+0000) 2019-10-13T12:00:39.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:39.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:39.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:39.420+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:39.423682+0000) 2019-10-13T12:00:41.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:41.852+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:41.855758+0000) 2019-10-13T12:00:43.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:43.936+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:43.939973+0000) 2019-10-13T12:00:44.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:44.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.178780+0000) 2019-10-13T12:00:44.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:44.418+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.423887+0000) 2019-10-13T12:00:44.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:44.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.424526+0000) 2019-10-13T12:00:44.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:44.418+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.424699+0000) 2019-10-13T12:00:44.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:44.418+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.425192+0000) 2019-10-13T12:00:44.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:44.422+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.426234+0000) 2019-10-13T12:00:44.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:44.424+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.426764+0000) 2019-10-13T12:00:44.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:44.422+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.426882+0000) 2019-10-13T12:00:44.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:44.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:44.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:44.432+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.434985+0000) 2019-10-13T12:00:44.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:44.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:44.464691+0000) 2019-10-13T12:00:47.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:47.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:47.836259+0000) 2019-10-13T12:00:48.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:48.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.062742+0000) 2019-10-13T12:00:48.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:48.082+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.083694+0000) 2019-10-13T12:00:48.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:48.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:48.604420+0000) 2019-10-13T12:00:49.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:49.430+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.434437+0000) 2019-10-13T12:00:49.442 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:49.430+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.435441+0000) 2019-10-13T12:00:49.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:49.430+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.436237+0000) 2019-10-13T12:00:49.442 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:49.430+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.436986+0000) 2019-10-13T12:00:49.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:49.434+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.438140+0000) 2019-10-13T12:00:49.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:49.434+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.438881+0000) 2019-10-13T12:00:49.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:49.435+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.437527+0000) 2019-10-13T12:00:49.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:49.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:49.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:49.443+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:49.444884+0000) 2019-10-13T12:00:51.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:51.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:51.855970+0000) 2019-10-13T12:00:53.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:53.943+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:53.946496+0000) 2019-10-13T12:00:54.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:54.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.178983+0000) 2019-10-13T12:00:54.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:54.438+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.445575+0000) 2019-10-13T12:00:54.447 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:54.442+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.446375+0000) 2019-10-13T12:00:54.448 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:54.442+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.446763+0000) 2019-10-13T12:00:54.449 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:54.442+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.447239+0000) 2019-10-13T12:00:54.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:54.442+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.447915+0000) 2019-10-13T12:00:54.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:54.442+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.448200+0000) 2019-10-13T12:00:54.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:54.447+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.448794+0000) 2019-10-13T12:00:54.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:54.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:54.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:54.455+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.456369+0000) 2019-10-13T12:00:54.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:54.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:54.464811+0000) 2019-10-13T12:00:57.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:57.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:57.836453+0000) 2019-10-13T12:00:58.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:58.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.062951+0000) 2019-10-13T12:00:58.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:58.082+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.083895+0000) 2019-10-13T12:00:58.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:58.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:58.604671+0000) 2019-10-13T12:00:59.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:00:59.450+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.455541+0000) 2019-10-13T12:00:59.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:00:59.450+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.456149+0000) 2019-10-13T12:00:59.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:00:59.454+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.458361+0000) 2019-10-13T12:00:59.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:00:59.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.459004+0000) 2019-10-13T12:00:59.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:00:59.454+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.461342+0000) 2019-10-13T12:00:59.462 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:00:59.459+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.461827+0000) 2019-10-13T12:00:59.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:00:59.458+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.462159+0000) 2019-10-13T12:00:59.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:00:59.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:00:59.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:00:59.467+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:00:59.468517+0000) 2019-10-13T12:01:01.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:01.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:01.856188+0000) 2019-10-13T12:01:01.863 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:01:01.863 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:01.867 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:01:01.867 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:03.954 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:03.951+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:03.952998+0000) 2019-10-13T12:01:04.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:04.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.179188+0000) 2019-10-13T12:01:04.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:04.458+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.464962+0000) 2019-10-13T12:01:04.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:04.462+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.467603+0000) 2019-10-13T12:01:04.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:04.462+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.469424+0000) 2019-10-13T12:01:04.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:04.466+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.469885+0000) 2019-10-13T12:01:04.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:04.466+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.470449+0000) 2019-10-13T12:01:04.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:04.466+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.472766+0000) 2019-10-13T12:01:04.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:04.466+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.473577+0000) 2019-10-13T12:01:04.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:04.471+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.474302+0000) 2019-10-13T12:01:04.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:04.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:04.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:04.479+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:04.480549+0000) 2019-10-13T12:01:07.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:07.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:07.836662+0000) 2019-10-13T12:01:08.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:08.058+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.063145+0000) 2019-10-13T12:01:08.084 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:08.078+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.084126+0000) 2019-10-13T12:01:08.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:08.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:08.604867+0000) 2019-10-13T12:01:09.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:09.474+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.478452+0000) 2019-10-13T12:01:09.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:09.474+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.479152+0000) 2019-10-13T12:01:09.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:09.474+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.478908+0000) 2019-10-13T12:01:09.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:09.474+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.479611+0000) 2019-10-13T12:01:09.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:09.479+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.480131+0000) 2019-10-13T12:01:09.481 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:09.474+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.480668+0000) 2019-10-13T12:01:09.482 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:09.474+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.480951+0000) 2019-10-13T12:01:09.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:09.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:09.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:09.487+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:09.488349+0000) 2019-10-13T12:01:11.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:11.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:11.856415+0000) 2019-10-13T12:01:13.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:13.955+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:13.959589+0000) 2019-10-13T12:01:14.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:14.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.179373+0000) 2019-10-13T12:01:14.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:14.457+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.465134+0000) 2019-10-13T12:01:14.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:14.485+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.490170+0000) 2019-10-13T12:01:14.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:14.485+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.490494+0000) 2019-10-13T12:01:14.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:14.485+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.491081+0000) 2019-10-13T12:01:14.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:14.487+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.491419+0000) 2019-10-13T12:01:14.493 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:14.485+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.491733+0000) 2019-10-13T12:01:14.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:14.485+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.492101+0000) 2019-10-13T12:01:14.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:14.485+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.492467+0000) 2019-10-13T12:01:14.497 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:14.489+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:14.500 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:14.495+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:14.499912+0000) 2019-10-13T12:01:17.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:17.833+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:17.836868+0000) 2019-10-13T12:01:18.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:18.057+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.063343+0000) 2019-10-13T12:01:18.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:18.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.084332+0000) 2019-10-13T12:01:18.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:18.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:18.605066+0000) 2019-10-13T12:01:19.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:19.493+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.499607+0000) 2019-10-13T12:01:19.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:19.497+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.502268+0000) 2019-10-13T12:01:19.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:19.497+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.502785+0000) 2019-10-13T12:01:19.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:19.497+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.503560+0000) 2019-10-13T12:01:19.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:19.503+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.504608+0000) 2019-10-13T12:01:19.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:19.497+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.503923+0000) 2019-10-13T12:01:19.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:19.497+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.504996+0000) 2019-10-13T12:01:19.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:19.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:19.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:19.507+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:19.511763+0000) 2019-10-13T12:01:21.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:21.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:21.856608+0000) 2019-10-13T12:01:23.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:23.963+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:23.966020+0000) 2019-10-13T12:01:24.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:24.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.179575+0000) 2019-10-13T12:01:24.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:24.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.465331+0000) 2019-10-13T12:01:24.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:24.505+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.510642+0000) 2019-10-13T12:01:24.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:24.507+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.511138+0000) 2019-10-13T12:01:24.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:24.505+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.511530+0000) 2019-10-13T12:01:24.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:24.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.512660+0000) 2019-10-13T12:01:24.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:24.505+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.513009+0000) 2019-10-13T12:01:24.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:24.509+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.513992+0000) 2019-10-13T12:01:24.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:24.509+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.514790+0000) 2019-10-13T12:01:24.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:24.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:24.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:24.519+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:24.522271+0000) 2019-10-13T12:01:27.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:27.833+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:27.837076+0000) 2019-10-13T12:01:28.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:28.061+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.063545+0000) 2019-10-13T12:01:28.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:28.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.084493+0000) 2019-10-13T12:01:28.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:28.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:28.605261+0000) 2019-10-13T12:01:29.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:29.521+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.523286+0000) 2019-10-13T12:01:29.526 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:29.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.525980+0000) 2019-10-13T12:01:29.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:29.525+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.526270+0000) 2019-10-13T12:01:29.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:29.523+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.526911+0000) 2019-10-13T12:01:29.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:29.525+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.527492+0000) 2019-10-13T12:01:29.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:29.525+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.527961+0000) 2019-10-13T12:01:29.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:29.525+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.528828+0000) 2019-10-13T12:01:29.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:29.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:29.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:29.531+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:29.534313+0000) 2019-10-13T12:01:31.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:31.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:31.856867+0000) 2019-10-13T12:01:32.186 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:01:32.186 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:32.190 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:01:32.190 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:01:33.973 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:33.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:33.972458+0000) 2019-10-13T12:01:34.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:34.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.179770+0000) 2019-10-13T12:01:34.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:34.461+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.465528+0000) 2019-10-13T12:01:34.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:34.529+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.533539+0000) 2019-10-13T12:01:34.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:34.533+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.534263+0000) 2019-10-13T12:01:34.535 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:34.533+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.534553+0000) 2019-10-13T12:01:34.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:34.533+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.534776+0000) 2019-10-13T12:01:34.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:34.533+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.535708+0000) 2019-10-13T12:01:34.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:34.533+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.536178+0000) 2019-10-13T12:01:34.537 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:34.535+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.536847+0000) 2019-10-13T12:01:34.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:34.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151425 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:34.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:34.539+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:34.541516+0000) 2019-10-13T12:01:37.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:37.833+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:37.837306+0000) 2019-10-13T12:01:38.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:38.061+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.063743+0000) 2019-10-13T12:01:38.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:38.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.084725+0000) 2019-10-13T12:01:38.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:38.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:38.605451+0000) 2019-10-13T12:01:39.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:39.541+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.543267+0000) 2019-10-13T12:01:39.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:39.541+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.544783+0000) 2019-10-13T12:01:39.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:39.543+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.545176+0000) 2019-10-13T12:01:39.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:39.541+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.545507+0000) 2019-10-13T12:01:39.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:39.545+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.545876+0000) 2019-10-13T12:01:39.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:39.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.546441+0000) 2019-10-13T12:01:39.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:39.545+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.546553+0000) 2019-10-13T12:01:39.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:39.547+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:39.550642+0000) 2019-10-13T12:01:39.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:39.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:41.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:41.855+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:41.857067+0000) 2019-10-13T12:01:43.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:43.975+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:43.979124+0000) 2019-10-13T12:01:44.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:44.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.179967+0000) 2019-10-13T12:01:44.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:44.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.465784+0000) 2019-10-13T12:01:44.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:44.553+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.556107+0000) 2019-10-13T12:01:44.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:44.553+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.556408+0000) 2019-10-13T12:01:44.558 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:44.553+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.556984+0000) 2019-10-13T12:01:44.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:44.553+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.557243+0000) 2019-10-13T12:01:44.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:44.557+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.557862+0000) 2019-10-13T12:01:44.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:44.555+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.558505+0000) 2019-10-13T12:01:44.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:44.557+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.559662+0000) 2019-10-13T12:01:44.563 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:44.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:44.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:44.563+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:44.564770+0000) 2019-10-13T12:01:47.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:47.833+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:47.837538+0000) 2019-10-13T12:01:48.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:48.061+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.063941+0000) 2019-10-13T12:01:48.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:48.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.084934+0000) 2019-10-13T12:01:48.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:48.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:48.605643+0000) 2019-10-13T12:01:49.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:49.565+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.566220+0000) 2019-10-13T12:01:49.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:49.563+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.566680+0000) 2019-10-13T12:01:49.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:49.565+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.567072+0000) 2019-10-13T12:01:49.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:49.565+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.567954+0000) 2019-10-13T12:01:49.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:49.565+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.568394+0000) 2019-10-13T12:01:49.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:49.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.569171+0000) 2019-10-13T12:01:49.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:49.565+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.569412+0000) 2019-10-13T12:01:49.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:49.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:49.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:49.571+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:49.575405+0000) 2019-10-13T12:01:51.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:51.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:51.857267+0000) 2019-10-13T12:01:53.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:53.982+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:53.985675+0000) 2019-10-13T12:01:54.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:54.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.180203+0000) 2019-10-13T12:01:54.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:54.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.466033+0000) 2019-10-13T12:01:54.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:54.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.577215+0000) 2019-10-13T12:01:54.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:54.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.577522+0000) 2019-10-13T12:01:54.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:54.577+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.578197+0000) 2019-10-13T12:01:54.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:54.577+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.580156+0000) 2019-10-13T12:01:54.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:54.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.581018+0000) 2019-10-13T12:01:54.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:54.581+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.583059+0000) 2019-10-13T12:01:54.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:54.581+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.583519+0000) 2019-10-13T12:01:54.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:54.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:54.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:54.586+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:54.588937+0000) 2019-10-13T12:01:57.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:57.837+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:57.837768+0000) 2019-10-13T12:01:58.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:58.061+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.064169+0000) 2019-10-13T12:01:58.085 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:58.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.085154+0000) 2019-10-13T12:01:58.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:58.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:58.605861+0000) 2019-10-13T12:01:59.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:01:59.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.587975+0000) 2019-10-13T12:01:59.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:01:59.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.588482+0000) 2019-10-13T12:01:59.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:01:59.585+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.588735+0000) 2019-10-13T12:01:59.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:01:59.589+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.591099+0000) 2019-10-13T12:01:59.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:01:59.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.591493+0000) 2019-10-13T12:01:59.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:01:59.589+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.591704+0000) 2019-10-13T12:01:59.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:01:59.589+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.592419+0000) 2019-10-13T12:01:59.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:01:59.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:01:59.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:01:59.594+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:01:59.599091+0000) 2019-10-13T12:02:01.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:01.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:01.857469+0000) 2019-10-13T12:02:02.237 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:02:02.237 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:02.241 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:02:02.241 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:03.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:03.990+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:03.992172+0000) 2019-10-13T12:02:04.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:04.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.180433+0000) 2019-10-13T12:02:04.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:04.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.466240+0000) 2019-10-13T12:02:04.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:04.597+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.599319+0000) 2019-10-13T12:02:04.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:04.594+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.599719+0000) 2019-10-13T12:02:04.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:04.597+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.600487+0000) 2019-10-13T12:02:04.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:04.597+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.600496+0000) 2019-10-13T12:02:04.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:04.597+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.600908+0000) 2019-10-13T12:02:04.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:04.601+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.602074+0000) 2019-10-13T12:02:04.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:04.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.602591+0000) 2019-10-13T12:02:04.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:04.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151609 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:04.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:04.606+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:04.609800+0000) 2019-10-13T12:02:07.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:07.837+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:07.838022+0000) 2019-10-13T12:02:08.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:08.061+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.064369+0000) 2019-10-13T12:02:08.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:08.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.085353+0000) 2019-10-13T12:02:08.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:08.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:08.606076+0000) 2019-10-13T12:02:09.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:09.609+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.609855+0000) 2019-10-13T12:02:09.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:09.609+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.610525+0000) 2019-10-13T12:02:09.611 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:09.609+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.611029+0000) 2019-10-13T12:02:09.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:09.609+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.611599+0000) 2019-10-13T12:02:09.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:09.606+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.611542+0000) 2019-10-13T12:02:09.614 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:09.609+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.613703+0000) 2019-10-13T12:02:09.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:09.613+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.616288+0000) 2019-10-13T12:02:09.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:09.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:09.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:09.618+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:09.620365+0000) 2019-10-13T12:02:11.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:11.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:11.857686+0000) 2019-10-13T12:02:13.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:13.994+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:13.998746+0000) 2019-10-13T12:02:14.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:14.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.180637+0000) 2019-10-13T12:02:14.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:14.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.466440+0000) 2019-10-13T12:02:14.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:14.616+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.621266+0000) 2019-10-13T12:02:14.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:14.618+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.621689+0000) 2019-10-13T12:02:14.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:14.620+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.622462+0000) 2019-10-13T12:02:14.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:14.620+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.624666+0000) 2019-10-13T12:02:14.626 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:14.620+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.625540+0000) 2019-10-13T12:02:14.626 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:14.620+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.625540+0000) 2019-10-13T12:02:14.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:14.624+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.625812+0000) 2019-10-13T12:02:14.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:14.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:14.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:14.626+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:14.630486+0000) 2019-10-13T12:02:17.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:17.836+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:17.838245+0000) 2019-10-13T12:02:18.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:18.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.064567+0000) 2019-10-13T12:02:18.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:18.080+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.085571+0000) 2019-10-13T12:02:18.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:18.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:18.606294+0000) 2019-10-13T12:02:19.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:19.626+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.631316+0000) 2019-10-13T12:02:19.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:19.628+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.631676+0000) 2019-10-13T12:02:19.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:19.628+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.633479+0000) 2019-10-13T12:02:19.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:19.632+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.633721+0000) 2019-10-13T12:02:19.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:19.632+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.634871+0000) 2019-10-13T12:02:19.637 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:19.632+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.636617+0000) 2019-10-13T12:02:19.638 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:19.632+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.636949+0000) 2019-10-13T12:02:19.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:19.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:19.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:19.642+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:19.644615+0000) 2019-10-13T12:02:21.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:21.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:21.857887+0000) 2019-10-13T12:02:24.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:24.002+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.005185+0000) 2019-10-13T12:02:24.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:24.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.180838+0000) 2019-10-13T12:02:24.467 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:24.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.466673+0000) 2019-10-13T12:02:24.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:24.638+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.643137+0000) 2019-10-13T12:02:24.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:24.640+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.644143+0000) 2019-10-13T12:02:24.646 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:24.640+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.645494+0000) 2019-10-13T12:02:24.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:24.644+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.645726+0000) 2019-10-13T12:02:24.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:24.644+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.646148+0000) 2019-10-13T12:02:24.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:24.644+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.646493+0000) 2019-10-13T12:02:24.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:24.644+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.647093+0000) 2019-10-13T12:02:24.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:24.646+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:24.650068+0000) 2019-10-13T12:02:24.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:24.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151733 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:27.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:27.836+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:27.838507+0000) 2019-10-13T12:02:28.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:28.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.064760+0000) 2019-10-13T12:02:28.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:28.084+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.085823+0000) 2019-10-13T12:02:28.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:28.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:28.606492+0000) 2019-10-13T12:02:29.655 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:29.652+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.654879+0000) 2019-10-13T12:02:29.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:29.652+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.656199+0000) 2019-10-13T12:02:29.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:29.654+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.657294+0000) 2019-10-13T12:02:29.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:29.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.657596+0000) 2019-10-13T12:02:29.659 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:29.656+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.658021+0000) 2019-10-13T12:02:29.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:29.656+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.658584+0000) 2019-10-13T12:02:29.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:29.656+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.658963+0000) 2019-10-13T12:02:29.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:29.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:29.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:29.662+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:29.665515+0000) 2019-10-13T12:02:31.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:31.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:31.858077+0000) 2019-10-13T12:02:32.469 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:02:32.469 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:32.473 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:02:32.473 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:02:34.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:34.014+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.016502+0000) 2019-10-13T12:02:34.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:34.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.181049+0000) 2019-10-13T12:02:34.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:34.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.466904+0000) 2019-10-13T12:02:34.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:34.664+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.666939+0000) 2019-10-13T12:02:34.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:34.664+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.667210+0000) 2019-10-13T12:02:34.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:34.664+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.667549+0000) 2019-10-13T12:02:34.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:34.666+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.669203+0000) 2019-10-13T12:02:34.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:34.664+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.668114+0000) 2019-10-13T12:02:34.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:34.664+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.668660+0000) 2019-10-13T12:02:34.672 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:34.668+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.670065+0000) 2019-10-13T12:02:34.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:34.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:34.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:34.674+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:34.676970+0000) 2019-10-13T12:02:37.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:37.836+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:37.838708+0000) 2019-10-13T12:02:38.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:38.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.064970+0000) 2019-10-13T12:02:38.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:38.084+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.086059+0000) 2019-10-13T12:02:38.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:38.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:38.606689+0000) 2019-10-13T12:02:39.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:39.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.677331+0000) 2019-10-13T12:02:39.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:39.678+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.681139+0000) 2019-10-13T12:02:39.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:39.676+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.678036+0000) 2019-10-13T12:02:39.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:39.676+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.678663+0000) 2019-10-13T12:02:39.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:39.676+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.679033+0000) 2019-10-13T12:02:39.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:39.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.679547+0000) 2019-10-13T12:02:39.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:39.676+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.680171+0000) 2019-10-13T12:02:39.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:39.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:39.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:39.686+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:39.688448+0000) 2019-10-13T12:02:41.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:41.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:41.858287+0000) 2019-10-13T12:02:44.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:44.022+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.023021+0000) 2019-10-13T12:02:44.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:44.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.181276+0000) 2019-10-13T12:02:44.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:44.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.467135+0000) 2019-10-13T12:02:44.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:44.684+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.688547+0000) 2019-10-13T12:02:44.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:44.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.690572+0000) 2019-10-13T12:02:44.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:44.688+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.691074+0000) 2019-10-13T12:02:44.692 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:44.688+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.691941+0000) 2019-10-13T12:02:44.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:44.688+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.693165+0000) 2019-10-13T12:02:44.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:44.688+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.693519+0000) 2019-10-13T12:02:44.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:44.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.694217+0000) 2019-10-13T12:02:44.695 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:44.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:44.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:44.694+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:44.698539+0000) 2019-10-13T12:02:47.839 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:47.836+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:47.838905+0000) 2019-10-13T12:02:48.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:48.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.065198+0000) 2019-10-13T12:02:48.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:48.084+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.086259+0000) 2019-10-13T12:02:48.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:48.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:48.606896+0000) 2019-10-13T12:02:49.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:49.696+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.698890+0000) 2019-10-13T12:02:49.701 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:49.696+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.701046+0000) 2019-10-13T12:02:49.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:49.696+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.701307+0000) 2019-10-13T12:02:49.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:49.700+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.702910+0000) 2019-10-13T12:02:49.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:49.700+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.703717+0000) 2019-10-13T12:02:49.705 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:49.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.705018+0000) 2019-10-13T12:02:49.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:49.702+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.705687+0000) 2019-10-13T12:02:49.706 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:49.702+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:49.706028+0000) 2019-10-13T12:02:49.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:49.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:51.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:51.854+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:51.858480+0000) 2019-10-13T12:02:54.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:54.026+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.029655+0000) 2019-10-13T12:02:54.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:54.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.181509+0000) 2019-10-13T12:02:54.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:54.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.467364+0000) 2019-10-13T12:02:54.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:54.708+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.711222+0000) 2019-10-13T12:02:54.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:54.708+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.711935+0000) 2019-10-13T12:02:54.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:54.708+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.712357+0000) 2019-10-13T12:02:54.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:54.710+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.714514+0000) 2019-10-13T12:02:54.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:54.712+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.714707+0000) 2019-10-13T12:02:54.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:54.712+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.717002+0000) 2019-10-13T12:02:54.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:54.712+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:54.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:54.712+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.717408+0000) 2019-10-13T12:02:54.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:54.718+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:54.721800+0000) 2019-10-13T12:02:57.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:57.836+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:57.839122+0000) 2019-10-13T12:02:58.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:58.060+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.065414+0000) 2019-10-13T12:02:58.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:58.084+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.086461+0000) 2019-10-13T12:02:58.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:58.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:58.607086+0000) 2019-10-13T12:02:59.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:02:59.716+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.720177+0000) 2019-10-13T12:02:59.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:02:59.716+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.721211+0000) 2019-10-13T12:02:59.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:02:59.720+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.721914+0000) 2019-10-13T12:02:59.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:02:59.720+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.722493+0000) 2019-10-13T12:02:59.723 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:02:59.717+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.722785+0000) 2019-10-13T12:02:59.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:02:59.720+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.723048+0000) 2019-10-13T12:02:59.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:02:59.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.723513+0000) 2019-10-13T12:02:59.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:02:59.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:02:59.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:02:59.729+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:02:59.733578+0000) 2019-10-13T12:03:01.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:01.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:01.858738+0000) 2019-10-13T12:03:02.573 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:03:02.573 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:02.577 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:03:02.577 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:04.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:04.033+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.036183+0000) 2019-10-13T12:03:04.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:04.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.181718+0000) 2019-10-13T12:03:04.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:04.464+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.467594+0000) 2019-10-13T12:03:04.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:04.728+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.730931+0000) 2019-10-13T12:03:04.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:04.728+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.731316+0000) 2019-10-13T12:03:04.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:04.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.732029+0000) 2019-10-13T12:03:04.733 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:04.728+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.732722+0000) 2019-10-13T12:03:04.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:04.728+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.732995+0000) 2019-10-13T12:03:04.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:04.732+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.733796+0000) 2019-10-13T12:03:04.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:04.732+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.734338+0000) 2019-10-13T12:03:04.738 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:04.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:04.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:04.733+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:04.739285+0000) 2019-10-13T12:03:07.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:07.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:07.839347+0000) 2019-10-13T12:03:08.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:08.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.065607+0000) 2019-10-13T12:03:08.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:08.083+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.086674+0000) 2019-10-13T12:03:08.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:08.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:08.607317+0000) 2019-10-13T12:03:09.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:09.735+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.741314+0000) 2019-10-13T12:03:09.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:09.737+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.741775+0000) 2019-10-13T12:03:09.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:09.739+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.742161+0000) 2019-10-13T12:03:09.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:09.739+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.744080+0000) 2019-10-13T12:03:09.745 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:09.739+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.744487+0000) 2019-10-13T12:03:09.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:09.743+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.746678+0000) 2019-10-13T12:03:09.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:09.743+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.747004+0000) 2019-10-13T12:03:09.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:09.743+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:09.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:09.749+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:09.754725+0000) 2019-10-13T12:03:11.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:11.857+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:11.858926+0000) 2019-10-13T12:03:14.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:14.041+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.042556+0000) 2019-10-13T12:03:14.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:14.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.181954+0000) 2019-10-13T12:03:14.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:14.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.467838+0000) 2019-10-13T12:03:14.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:14.747+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.753340+0000) 2019-10-13T12:03:14.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:14.747+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.753707+0000) 2019-10-13T12:03:14.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:14.751+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.754153+0000) 2019-10-13T12:03:14.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:14.749+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.754789+0000) 2019-10-13T12:03:14.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:14.751+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.755322+0000) 2019-10-13T12:03:14.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:14.751+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.755563+0000) 2019-10-13T12:03:14.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:14.751+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.756008+0000) 2019-10-13T12:03:14.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:14.755+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:14.764 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:14.757+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:14.763852+0000) 2019-10-13T12:03:17.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:17.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:17.839527+0000) 2019-10-13T12:03:18.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:18.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.065821+0000) 2019-10-13T12:03:18.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:18.083+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.086891+0000) 2019-10-13T12:03:18.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:18.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:18.607539+0000) 2019-10-13T12:03:19.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:19.759+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.763775+0000) 2019-10-13T12:03:19.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:19.759+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.764518+0000) 2019-10-13T12:03:19.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:19.759+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.765280+0000) 2019-10-13T12:03:19.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:19.759+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.765590+0000) 2019-10-13T12:03:19.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:19.761+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.766637+0000) 2019-10-13T12:03:19.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:19.763+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.767142+0000) 2019-10-13T12:03:19.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:19.763+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.767579+0000) 2019-10-13T12:03:19.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:19.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:19.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:19.769+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:19.773640+0000) 2019-10-13T12:03:21.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:21.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:21.859136+0000) 2019-10-13T12:03:24.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:24.045+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.049084+0000) 2019-10-13T12:03:24.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:24.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.182173+0000) 2019-10-13T12:03:24.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:24.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.468063+0000) 2019-10-13T12:03:24.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:24.771+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.776690+0000) 2019-10-13T12:03:24.778 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:24.771+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.777610+0000) 2019-10-13T12:03:24.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:24.775+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.778007+0000) 2019-10-13T12:03:24.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:24.773+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.778497+0000) 2019-10-13T12:03:24.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:24.775+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.778773+0000) 2019-10-13T12:03:24.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:24.775+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.779469+0000) 2019-10-13T12:03:24.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:24.775+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.779827+0000) 2019-10-13T12:03:24.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:24.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:24.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:24.781+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:24.786671+0000) 2019-10-13T12:03:27.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:27.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:27.839732+0000) 2019-10-13T12:03:28.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:28.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.066045+0000) 2019-10-13T12:03:28.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:28.083+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.087091+0000) 2019-10-13T12:03:28.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:28.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:28.607767+0000) 2019-10-13T12:03:29.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:29.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.785574+0000) 2019-10-13T12:03:29.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:29.783+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.786141+0000) 2019-10-13T12:03:29.788 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:29.783+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.787262+0000) 2019-10-13T12:03:29.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:29.783+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.787826+0000) 2019-10-13T12:03:29.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:29.783+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.787436+0000) 2019-10-13T12:03:29.789 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:29.785+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.788723+0000) 2019-10-13T12:03:29.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:29.783+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.788342+0000) 2019-10-13T12:03:29.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:29.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:29.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:29.789+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:29.793295+0000) 2019-10-13T12:03:31.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:31.857+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:31.859328+0000) 2019-10-13T12:03:32.720 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:03:32.720 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:32.724 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:03:32.724 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:03:34.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:34.049+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.055562+0000) 2019-10-13T12:03:34.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:34.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.182385+0000) 2019-10-13T12:03:34.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:34.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.468287+0000) 2019-10-13T12:03:34.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:34.791+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.796162+0000) 2019-10-13T12:03:34.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:34.793+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.799353+0000) 2019-10-13T12:03:34.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:34.795+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.799793+0000) 2019-10-13T12:03:34.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:34.795+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.800131+0000) 2019-10-13T12:03:34.801 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:34.795+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.800515+0000) 2019-10-13T12:03:34.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:34.795+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.801308+0000) 2019-10-13T12:03:34.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:34.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.801984+0000) 2019-10-13T12:03:34.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:34.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:34.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:34.801+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:34.805382+0000) 2019-10-13T12:03:37.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:37.839+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:37.839926+0000) 2019-10-13T12:03:38.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:38.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.066242+0000) 2019-10-13T12:03:38.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:38.083+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.087292+0000) 2019-10-13T12:03:38.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:38.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:38.607971+0000) 2019-10-13T12:03:39.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:39.803+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.807118+0000) 2019-10-13T12:03:39.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:39.803+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.807738+0000) 2019-10-13T12:03:39.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:39.803+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.808041+0000) 2019-10-13T12:03:39.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:39.805+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.808390+0000) 2019-10-13T12:03:39.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:39.803+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.809156+0000) 2019-10-13T12:03:39.810 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:39.807+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.809819+0000) 2019-10-13T12:03:39.811 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:39.807+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.810610+0000) 2019-10-13T12:03:39.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:39.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:39.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:39.813+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:39.816982+0000) 2019-10-13T12:03:41.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:41.857+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:41.859541+0000) 2019-10-13T12:03:44.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:44.057+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.062087+0000) 2019-10-13T12:03:44.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:44.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.182622+0000) 2019-10-13T12:03:44.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:44.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.468471+0000) 2019-10-13T12:03:44.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:44.811+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.816741+0000) 2019-10-13T12:03:44.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:44.815+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.818596+0000) 2019-10-13T12:03:44.820 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:44.815+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.818573+0000) 2019-10-13T12:03:44.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:44.815+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.819055+0000) 2019-10-13T12:03:44.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:44.815+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.819364+0000) 2019-10-13T12:03:44.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:44.813+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.819783+0000) 2019-10-13T12:03:44.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:44.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.820526+0000) 2019-10-13T12:03:44.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:44.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:44.823503+0000) 2019-10-13T12:03:44.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:44.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:47.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:47.839+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:47.840140+0000) 2019-10-13T12:03:48.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:48.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.066485+0000) 2019-10-13T12:03:48.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:48.087+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.087521+0000) 2019-10-13T12:03:48.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:48.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:48.608193+0000) 2019-10-13T12:03:49.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:49.823+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.829070+0000) 2019-10-13T12:03:49.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:49.827+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.831650+0000) 2019-10-13T12:03:49.833 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:49.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.832237+0000) 2019-10-13T12:03:49.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:49.827+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.832375+0000) 2019-10-13T12:03:49.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:49.831+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.833986+0000) 2019-10-13T12:03:49.836 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:49.831+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.835488+0000) 2019-10-13T12:03:49.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:49.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:49.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:49.833+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.838185+0000) 2019-10-13T12:03:49.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:49.837+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:49.840695+0000) 2019-10-13T12:03:51.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:51.853+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:51.859738+0000) 2019-10-13T12:03:54.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:54.065+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.068459+0000) 2019-10-13T12:03:54.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:54.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.182860+0000) 2019-10-13T12:03:54.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:54.463+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.468698+0000) 2019-10-13T12:03:54.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:54.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.841142+0000) 2019-10-13T12:03:54.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:54.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.841305+0000) 2019-10-13T12:03:54.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:54.839+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.841914+0000) 2019-10-13T12:03:54.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:54.841+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.844241+0000) 2019-10-13T12:03:54.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:54.839+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.844583+0000) 2019-10-13T12:03:54.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:54.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.845029+0000) 2019-10-13T12:03:54.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:54.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152296 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:54.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:54.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.849599+0000) 2019-10-13T12:03:54.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:54.849+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:54.853476+0000) 2019-10-13T12:03:57.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:57.835+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:57.840309+0000) 2019-10-13T12:03:58.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:58.063+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.066683+0000) 2019-10-13T12:03:58.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:58.087+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.087723+0000) 2019-10-13T12:03:58.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:58.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:58.608415+0000) 2019-10-13T12:03:59.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:03:59.847+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.852481+0000) 2019-10-13T12:03:59.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:03:59.849+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.852852+0000) 2019-10-13T12:03:59.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:03:59.851+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.854150+0000) 2019-10-13T12:03:59.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:03:59.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.853885+0000) 2019-10-13T12:03:59.855 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:03:59.851+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.854622+0000) 2019-10-13T12:03:59.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:03:59.851+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.855570+0000) 2019-10-13T12:03:59.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:03:59.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.856044+0000) 2019-10-13T12:03:59.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:03:59.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:03:59.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:03:59.857+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:03:59.861728+0000) 2019-10-13T12:04:01.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:01.856+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:01.859945+0000) 2019-10-13T12:04:02.885 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:04:02.886 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:02.889 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:04:02.890 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:04.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:04.076+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.079731+0000) 2019-10-13T12:04:04.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:04.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.183068+0000) 2019-10-13T12:04:04.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:04.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.468896+0000) 2019-10-13T12:04:04.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:04.858+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.863357+0000) 2019-10-13T12:04:04.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:04.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.863763+0000) 2019-10-13T12:04:04.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:04.858+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.864478+0000) 2019-10-13T12:04:04.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:04.858+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.864863+0000) 2019-10-13T12:04:04.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:04.862+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.865961+0000) 2019-10-13T12:04:04.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:04.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.866352+0000) 2019-10-13T12:04:04.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:04.862+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.867468+0000) 2019-10-13T12:04:04.872 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:04.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:04.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:04.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:04.871815+0000) 2019-10-13T12:04:07.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:07.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:07.840505+0000) 2019-10-13T12:04:08.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:08.062+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.066885+0000) 2019-10-13T12:04:08.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:08.086+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.087918+0000) 2019-10-13T12:04:08.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:08.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:08.608569+0000) 2019-10-13T12:04:09.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:09.870+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.874891+0000) 2019-10-13T12:04:09.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:09.868+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.875266+0000) 2019-10-13T12:04:09.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:09.870+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.875513+0000) 2019-10-13T12:04:09.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:09.870+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.875782+0000) 2019-10-13T12:04:09.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:09.870+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.876740+0000) 2019-10-13T12:04:09.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:09.870+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.876854+0000) 2019-10-13T12:04:09.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:09.874+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.878445+0000) 2019-10-13T12:04:09.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:09.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152390 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:09.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:09.880+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:09.884496+0000) 2019-10-13T12:04:11.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:11.856+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:11.860154+0000) 2019-10-13T12:04:14.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:14.084+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.086434+0000) 2019-10-13T12:04:14.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:14.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.183308+0000) 2019-10-13T12:04:14.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:14.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.469106+0000) 2019-10-13T12:04:14.886 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:14.878+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.885657+0000) 2019-10-13T12:04:14.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:14.882+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.886177+0000) 2019-10-13T12:04:14.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:14.882+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.886833+0000) 2019-10-13T12:04:14.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:14.882+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.887114+0000) 2019-10-13T12:04:14.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:14.880+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.887902+0000) 2019-10-13T12:04:14.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:14.882+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.887396+0000) 2019-10-13T12:04:14.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:14.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.889573+0000) 2019-10-13T12:04:14.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:14.884+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:14.891617+0000) 2019-10-13T12:04:14.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:14.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:17.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:17.838+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:17.840704+0000) 2019-10-13T12:04:18.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:18.066+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.067085+0000) 2019-10-13T12:04:18.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:18.086+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.088119+0000) 2019-10-13T12:04:18.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:18.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:18.608773+0000) 2019-10-13T12:04:19.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:19.890+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.895812+0000) 2019-10-13T12:04:19.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:19.890+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.896188+0000) 2019-10-13T12:04:19.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:19.894+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.898144+0000) 2019-10-13T12:04:19.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:19.894+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.898153+0000) 2019-10-13T12:04:19.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:19.892+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.898623+0000) 2019-10-13T12:04:19.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:19.894+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.899217+0000) 2019-10-13T12:04:19.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:19.894+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.899945+0000) 2019-10-13T12:04:19.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:19.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:19.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:19.900+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:19.905075+0000) 2019-10-13T12:04:21.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:21.856+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:21.860347+0000) 2019-10-13T12:04:24.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:24.092+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.093036+0000) 2019-10-13T12:04:24.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:24.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.183514+0000) 2019-10-13T12:04:24.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:24.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.469303+0000) 2019-10-13T12:04:24.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:24.900+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.907114+0000) 2019-10-13T12:04:24.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:24.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.907926+0000) 2019-10-13T12:04:24.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:24.902+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.908268+0000) 2019-10-13T12:04:24.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:24.902+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.909128+0000) 2019-10-13T12:04:24.910 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:24.902+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.909511+0000) 2019-10-13T12:04:24.910 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:24.902+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.909196+0000) 2019-10-13T12:04:24.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:24.906+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.910623+0000) 2019-10-13T12:04:24.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:24.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:24.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:24.912+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:24.917363+0000) 2019-10-13T12:04:27.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:27.834+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:27.840864+0000) 2019-10-13T12:04:28.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:28.062+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.067282+0000) 2019-10-13T12:04:28.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:28.082+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.088352+0000) 2019-10-13T12:04:28.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:28.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:28.609001+0000) 2019-10-13T12:04:29.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:29.914+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.918753+0000) 2019-10-13T12:04:29.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:29.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.919171+0000) 2019-10-13T12:04:29.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:29.914+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.919736+0000) 2019-10-13T12:04:29.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:29.914+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.920256+0000) 2019-10-13T12:04:29.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:29.916+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.920552+0000) 2019-10-13T12:04:29.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:29.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.921878+0000) 2019-10-13T12:04:29.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:29.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.922237+0000) 2019-10-13T12:04:29.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:29.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152516 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:29.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:29.920+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:29.926353+0000) 2019-10-13T12:04:31.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:31.860+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:31.860565+0000) 2019-10-13T12:04:33.005 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:04:33.006 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:33.009 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:04:33.009 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:04:34.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:34.092+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.099266+0000) 2019-10-13T12:04:34.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:34.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.183702+0000) 2019-10-13T12:04:34.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:34.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.469502+0000) 2019-10-13T12:04:34.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:34.924+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.928791+0000) 2019-10-13T12:04:34.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:34.922+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.929312+0000) 2019-10-13T12:04:34.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:34.926+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.930018+0000) 2019-10-13T12:04:34.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:34.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.929697+0000) 2019-10-13T12:04:34.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:34.926+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.930622+0000) 2019-10-13T12:04:34.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:34.926+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.931292+0000) 2019-10-13T12:04:34.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:34.926+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.931702+0000) 2019-10-13T12:04:34.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:34.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:34.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:34.932+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:34.937812+0000) 2019-10-13T12:04:37.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:37.838+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:37.841083+0000) 2019-10-13T12:04:38.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:38.062+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.067465+0000) 2019-10-13T12:04:38.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:38.086+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.088577+0000) 2019-10-13T12:04:38.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:38.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:38.609196+0000) 2019-10-13T12:04:39.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:39.934+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.940874+0000) 2019-10-13T12:04:39.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:39.934+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.941642+0000) 2019-10-13T12:04:39.943 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:39.938+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.942220+0000) 2019-10-13T12:04:39.943 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:39.938+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.942472+0000) 2019-10-13T12:04:39.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:39.938+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.942787+0000) 2019-10-13T12:04:39.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:39.936+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.943302+0000) 2019-10-13T12:04:39.944 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:39.938+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.943532+0000) 2019-10-13T12:04:39.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:39.940+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:39.946419+0000) 2019-10-13T12:04:39.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:39.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:41.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:41.860+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:41.860774+0000) 2019-10-13T12:04:44.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:44.104+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.105649+0000) 2019-10-13T12:04:44.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:44.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.183907+0000) 2019-10-13T12:04:44.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:44.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.469766+0000) 2019-10-13T12:04:44.951 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:44.946+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.950124+0000) 2019-10-13T12:04:44.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:44.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.950755+0000) 2019-10-13T12:04:44.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:44.946+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.951137+0000) 2019-10-13T12:04:44.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:44.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.951687+0000) 2019-10-13T12:04:44.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:44.944+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.952066+0000) 2019-10-13T12:04:44.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:44.946+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.952719+0000) 2019-10-13T12:04:44.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:44.946+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.953466+0000) 2019-10-13T12:04:44.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:44.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152612 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:44.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:44.952+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:44.959235+0000) 2019-10-13T12:04:47.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:47.838+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:47.841314+0000) 2019-10-13T12:04:48.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:48.062+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.067669+0000) 2019-10-13T12:04:48.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:48.086+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.088807+0000) 2019-10-13T12:04:48.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:48.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:48.609425+0000) 2019-10-13T12:04:49.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:49.954+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.961201+0000) 2019-10-13T12:04:49.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:49.956+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.963553+0000) 2019-10-13T12:04:49.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:49.958+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.963971+0000) 2019-10-13T12:04:49.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:49.958+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.964364+0000) 2019-10-13T12:04:49.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:49.958+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.964815+0000) 2019-10-13T12:04:49.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:49.958+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.965284+0000) 2019-10-13T12:04:49.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:49.962+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.966467+0000) 2019-10-13T12:04:49.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:49.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:49.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:49.964+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:49.969690+0000) 2019-10-13T12:04:51.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:51.856+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:51.860982+0000) 2019-10-13T12:04:54.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:54.108+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.112192+0000) 2019-10-13T12:04:54.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:54.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.184064+0000) 2019-10-13T12:04:54.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:54.466+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.469973+0000) 2019-10-13T12:04:54.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:54.966+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.972807+0000) 2019-10-13T12:04:54.974 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:54.966+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.973563+0000) 2019-10-13T12:04:54.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:54.968+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.974082+0000) 2019-10-13T12:04:54.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:54.970+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.974725+0000) 2019-10-13T12:04:54.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:54.970+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.975461+0000) 2019-10-13T12:04:54.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:54.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.975855+0000) 2019-10-13T12:04:54.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:54.970+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.976621+0000) 2019-10-13T12:04:54.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:54.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:54.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:54.976+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:54.983699+0000) 2019-10-13T12:04:57.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:57.837+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:57.841547+0000) 2019-10-13T12:04:58.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:58.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.067862+0000) 2019-10-13T12:04:58.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:58.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.089052+0000) 2019-10-13T12:04:58.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:58.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:58.609620+0000) 2019-10-13T12:04:59.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:04:59.977+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.984733+0000) 2019-10-13T12:04:59.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:04:59.977+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.985487+0000) 2019-10-13T12:04:59.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:04:59.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.986190+0000) 2019-10-13T12:04:59.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:04:59.980+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.986699+0000) 2019-10-13T12:04:59.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:04:59.981+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.987104+0000) 2019-10-13T12:04:59.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:04:59.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.987779+0000) 2019-10-13T12:04:59.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:04:59.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.988052+0000) 2019-10-13T12:04:59.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:04:59.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:04:59.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:04:59.996+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:04:59.996425+0000) 2019-10-13T12:05:01.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:01.860+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:01.861209+0000) 2019-10-13T12:05:03.141 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:05:03.141 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:03.145 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:05:03.145 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:04.119 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:04.116+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.118723+0000) 2019-10-13T12:05:04.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:04.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.184285+0000) 2019-10-13T12:05:04.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:04.465+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.470148+0000) 2019-10-13T12:05:04.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:04.989+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.996321+0000) 2019-10-13T12:05:05.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:04.993+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.999359+0000) 2019-10-13T12:05:05.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:04.993+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:04.999605+0000) 2019-10-13T12:05:05.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:04.993+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.000126+0000) 2019-10-13T12:05:05.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:05.000+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.001107+0000) 2019-10-13T12:05:05.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:04.993+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.001518+0000) 2019-10-13T12:05:05.002 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:04.997+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.002227+0000) 2019-10-13T12:05:05.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:04.997+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152738 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:05.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:05.004+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:05.006206+0000) 2019-10-13T12:05:07.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:07.837+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:07.841775+0000) 2019-10-13T12:05:08.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:08.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.068083+0000) 2019-10-13T12:05:08.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:08.081+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.089252+0000) 2019-10-13T12:05:08.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:08.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:08.609838+0000) 2019-10-13T12:05:10.009 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:10.001+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.007317+0000) 2019-10-13T12:05:10.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:10.005+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.010004+0000) 2019-10-13T12:05:10.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:10.005+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.010638+0000) 2019-10-13T12:05:10.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:10.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.009195+0000) 2019-10-13T12:05:10.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:10.005+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.011455+0000) 2019-10-13T12:05:10.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:10.005+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.012096+0000) 2019-10-13T12:05:10.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:10.005+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.012624+0000) 2019-10-13T12:05:10.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:10.009+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:10.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:10.019+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:10.020320+0000) 2019-10-13T12:05:11.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:11.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:11.861450+0000) 2019-10-13T12:05:14.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:14.123+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:14.125346+0000) 2019-10-13T12:05:14.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:14.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:14.184491+0000) 2019-10-13T12:05:14.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:14.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:14.470345+0000) 2019-10-13T12:05:15.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:15.013+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.017201+0000) 2019-10-13T12:05:15.020 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:15.017+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.019493+0000) 2019-10-13T12:05:15.021 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:15.017+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.021128+0000) 2019-10-13T12:05:15.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:15.021+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.022388+0000) 2019-10-13T12:05:15.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:15.019+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.022805+0000) 2019-10-13T12:05:15.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:15.021+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.022948+0000) 2019-10-13T12:05:15.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:15.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.023048+0000) 2019-10-13T12:05:15.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:15.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:15.028 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:15.023+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:15.027693+0000) 2019-10-13T12:05:17.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:17.841+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:17.841980+0000) 2019-10-13T12:05:18.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:18.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.068321+0000) 2019-10-13T12:05:18.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:18.085+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.089473+0000) 2019-10-13T12:05:18.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:18.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:18.610071+0000) 2019-10-13T12:05:20.030 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:20.025+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.029004+0000) 2019-10-13T12:05:20.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:20.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.030746+0000) 2019-10-13T12:05:20.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:20.029+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.032858+0000) 2019-10-13T12:05:20.034 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:20.029+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.033103+0000) 2019-10-13T12:05:20.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:20.029+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.033675+0000) 2019-10-13T12:05:20.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:20.031+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.034325+0000) 2019-10-13T12:05:20.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:20.033+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.034989+0000) 2019-10-13T12:05:20.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:20.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:20.042 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:20.039+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:20.041896+0000) 2019-10-13T12:05:21.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:21.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:21.861651+0000) 2019-10-13T12:05:24.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:24.131+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:24.132280+0000) 2019-10-13T12:05:24.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:24.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:24.184733+0000) 2019-10-13T12:05:24.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:24.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:24.470518+0000) 2019-10-13T12:05:25.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:25.037+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.039668+0000) 2019-10-13T12:05:25.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:25.037+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.040741+0000) 2019-10-13T12:05:25.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:25.041+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.042945+0000) 2019-10-13T12:05:25.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:25.041+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.042944+0000) 2019-10-13T12:05:25.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:25.041+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.044082+0000) 2019-10-13T12:05:25.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:25.041+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.044709+0000) 2019-10-13T12:05:25.045 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:25.043+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.045167+0000) 2019-10-13T12:05:25.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:25.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:25.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:25.047+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:25.050707+0000) 2019-10-13T12:05:27.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:27.841+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:27.842183+0000) 2019-10-13T12:05:28.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:28.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.068546+0000) 2019-10-13T12:05:28.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:28.089+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.089702+0000) 2019-10-13T12:05:28.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:28.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:28.610263+0000) 2019-10-13T12:05:30.051 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:30.049+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.050289+0000) 2019-10-13T12:05:30.051 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:30.049+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.051128+0000) 2019-10-13T12:05:30.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:30.049+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.053545+0000) 2019-10-13T12:05:30.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:30.053+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.053970+0000) 2019-10-13T12:05:30.057 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:30.055+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.056859+0000) 2019-10-13T12:05:30.059 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:30.057+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.058372+0000) 2019-10-13T12:05:30.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:30.057+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.058698+0000) 2019-10-13T12:05:30.060 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:30.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:30.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:30.059+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:30.060871+0000) 2019-10-13T12:05:31.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:31.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:31.861845+0000) 2019-10-13T12:05:33.250 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:05:33.250 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:33.253 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:05:33.254 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:05:34.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:34.139+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:34.143552+0000) 2019-10-13T12:05:34.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:34.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:34.184934+0000) 2019-10-13T12:05:34.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:34.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:34.470725+0000) 2019-10-13T12:05:35.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:35.061+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.062591+0000) 2019-10-13T12:05:35.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:35.061+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.064404+0000) 2019-10-13T12:05:35.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:35.061+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.065123+0000) 2019-10-13T12:05:35.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:35.061+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.065480+0000) 2019-10-13T12:05:35.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:35.065+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.065974+0000) 2019-10-13T12:05:35.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:35.063+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.066467+0000) 2019-10-13T12:05:35.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:35.065+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.067000+0000) 2019-10-13T12:05:35.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:35.067+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:35.068879+0000) 2019-10-13T12:05:35.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:35.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:37.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:37.841+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:37.842400+0000) 2019-10-13T12:05:38.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:38.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.068766+0000) 2019-10-13T12:05:38.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:38.089+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.089902+0000) 2019-10-13T12:05:38.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:38.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:38.610475+0000) 2019-10-13T12:05:40.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:40.073+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.073993+0000) 2019-10-13T12:05:40.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:40.073+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.074383+0000) 2019-10-13T12:05:40.076 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:40.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.074989+0000) 2019-10-13T12:05:40.076 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:40.071+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.075795+0000) 2019-10-13T12:05:40.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:40.073+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.075942+0000) 2019-10-13T12:05:40.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:40.073+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.077042+0000) 2019-10-13T12:05:40.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:40.073+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.077114+0000) 2019-10-13T12:05:40.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:40.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:40.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:40.079+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:40.082824+0000) 2019-10-13T12:05:41.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:41.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:41.862036+0000) 2019-10-13T12:05:44.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:44.147+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:44.149889+0000) 2019-10-13T12:05:44.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:44.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:44.185127+0000) 2019-10-13T12:05:44.471 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:44.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:44.470936+0000) 2019-10-13T12:05:45.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:45.081+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.084097+0000) 2019-10-13T12:05:45.085 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:45.081+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.084734+0000) 2019-10-13T12:05:45.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:45.085+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.085941+0000) 2019-10-13T12:05:45.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:45.085+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.086708+0000) 2019-10-13T12:05:45.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:45.083+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.087436+0000) 2019-10-13T12:05:45.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:45.085+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.087693+0000) 2019-10-13T12:05:45.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:45.085+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.088326+0000) 2019-10-13T12:05:45.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:45.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:45.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:45.091+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:45.093598+0000) 2019-10-13T12:05:47.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:47.841+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:47.842598+0000) 2019-10-13T12:05:48.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:48.065+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.068964+0000) 2019-10-13T12:05:48.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:48.089+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.090116+0000) 2019-10-13T12:05:48.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:48.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:48.610678+0000) 2019-10-13T12:05:50.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:50.093+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.094456+0000) 2019-10-13T12:05:50.096 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:50.093+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.094821+0000) 2019-10-13T12:05:50.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:50.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.095293+0000) 2019-10-13T12:05:50.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:50.093+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.095747+0000) 2019-10-13T12:05:50.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:50.093+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.095858+0000) 2019-10-13T12:05:50.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:50.095+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.096610+0000) 2019-10-13T12:05:50.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:50.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.098422+0000) 2019-10-13T12:05:50.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:50.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:50.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:50.099+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:50.102721+0000) 2019-10-13T12:05:51.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:51.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:51.862265+0000) 2019-10-13T12:05:54.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:54.155+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:54.156366+0000) 2019-10-13T12:05:54.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:54.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:54.185354+0000) 2019-10-13T12:05:54.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:54.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:54.471199+0000) 2019-10-13T12:05:55.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:55.104+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.106563+0000) 2019-10-13T12:05:55.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:55.104+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.106925+0000) 2019-10-13T12:05:55.108 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:05:55.104+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.107341+0000) 2019-10-13T12:05:55.109 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:05:55.107+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.108158+0000) 2019-10-13T12:05:55.110 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:55.104+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.108498+0000) 2019-10-13T12:05:55.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:05:55.104+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.108966+0000) 2019-10-13T12:05:55.110 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:55.104+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.109339+0000) 2019-10-13T12:05:55.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:05:55.108+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:05:55.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:05:55.115+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:55.116145+0000) 2019-10-13T12:05:57.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:05:57.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:57.842803+0000) 2019-10-13T12:05:58.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:05:58.064+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.069172+0000) 2019-10-13T12:05:58.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:05:58.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.090328+0000) 2019-10-13T12:05:58.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:05:58.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:05:58.610898+0000) 2019-10-13T12:06:00.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:00.112+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.116020+0000) 2019-10-13T12:06:00.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:00.112+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.116493+0000) 2019-10-13T12:06:00.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:00.112+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.116688+0000) 2019-10-13T12:06:00.120 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:00.116+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.120002+0000) 2019-10-13T12:06:00.124 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:00.119+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.123176+0000) 2019-10-13T12:06:00.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:00.120+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:00.125 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:00.119+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.123910+0000) 2019-10-13T12:06:00.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:00.120+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.124853+0000) 2019-10-13T12:06:00.126 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:00.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:00.124367+0000) 2019-10-13T12:06:01.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:01.859+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:01.862478+0000) 2019-10-13T12:06:03.412 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:06:03.412 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:03.416 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:06:03.416 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:04.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:04.159+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:04.162992+0000) 2019-10-13T12:06:04.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:04.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:04.185588+0000) 2019-10-13T12:06:04.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:04.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:04.471449+0000) 2019-10-13T12:06:05.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:05.124+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.127444+0000) 2019-10-13T12:06:05.128 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:05.124+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.127848+0000) 2019-10-13T12:06:05.131 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:05.128+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.130236+0000) 2019-10-13T12:06:05.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:05.128+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.130500+0000) 2019-10-13T12:06:05.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:05.131+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.133967+0000) 2019-10-13T12:06:05.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:05.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.134821+0000) 2019-10-13T12:06:05.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:05.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:05.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:05.132+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.135478+0000) 2019-10-13T12:06:05.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:05.135+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:05.136224+0000) 2019-10-13T12:06:07.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:07.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:07.843005+0000) 2019-10-13T12:06:08.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:08.064+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.069368+0000) 2019-10-13T12:06:08.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:08.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.090558+0000) 2019-10-13T12:06:08.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:08.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:08.611100+0000) 2019-10-13T12:06:10.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:10.136+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.138748+0000) 2019-10-13T12:06:10.140 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:10.136+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.139353+0000) 2019-10-13T12:06:10.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:10.135+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.139843+0000) 2019-10-13T12:06:10.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:10.136+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.140537+0000) 2019-10-13T12:06:10.142 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:10.136+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.141453+0000) 2019-10-13T12:06:10.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:10.140+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.142253+0000) 2019-10-13T12:06:10.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:10.140+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.142359+0000) 2019-10-13T12:06:10.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:10.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:10.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:10.143+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:10.147473+0000) 2019-10-13T12:06:11.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:11.858+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:11.862707+0000) 2019-10-13T12:06:14.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:14.166+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:14.169645+0000) 2019-10-13T12:06:14.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:14.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:14.185835+0000) 2019-10-13T12:06:14.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:14.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:14.471685+0000) 2019-10-13T12:06:15.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:15.148+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.149819+0000) 2019-10-13T12:06:15.151 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:15.148+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.150099+0000) 2019-10-13T12:06:15.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:15.148+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.150516+0000) 2019-10-13T12:06:15.152 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:15.148+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.150713+0000) 2019-10-13T12:06:15.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:15.148+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.150951+0000) 2019-10-13T12:06:15.152 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:15.146+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.151743+0000) 2019-10-13T12:06:15.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:15.148+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.151532+0000) 2019-10-13T12:06:15.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:15.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:15.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:15.154+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:15.158846+0000) 2019-10-13T12:06:17.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:17.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:17.843214+0000) 2019-10-13T12:06:18.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:18.064+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.069587+0000) 2019-10-13T12:06:18.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:18.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.090752+0000) 2019-10-13T12:06:18.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:18.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:18.611293+0000) 2019-10-13T12:06:20.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:20.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.161000+0000) 2019-10-13T12:06:20.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:20.156+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.161201+0000) 2019-10-13T12:06:20.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:20.160+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.161927+0000) 2019-10-13T12:06:20.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:20.160+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.162595+0000) 2019-10-13T12:06:20.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:20.160+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.163049+0000) 2019-10-13T12:06:20.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:20.158+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.163198+0000) 2019-10-13T12:06:20.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:20.160+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.162243+0000) 2019-10-13T12:06:20.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:20.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:20.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:20.166+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:20.170424+0000) 2019-10-13T12:06:21.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:21.858+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:21.862917+0000) 2019-10-13T12:06:24.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:24.174+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:24.176145+0000) 2019-10-13T12:06:24.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:24.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:24.186039+0000) 2019-10-13T12:06:24.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:24.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:24.471884+0000) 2019-10-13T12:06:25.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:25.168+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.172310+0000) 2019-10-13T12:06:25.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:25.168+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.172389+0000) 2019-10-13T12:06:25.174 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:25.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.172743+0000) 2019-10-13T12:06:25.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:25.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.173054+0000) 2019-10-13T12:06:25.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:25.172+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.174559+0000) 2019-10-13T12:06:25.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:25.172+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.176375+0000) 2019-10-13T12:06:25.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:25.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:25.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:25.174+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.178695+0000) 2019-10-13T12:06:25.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:25.182+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:25.185888+0000) 2019-10-13T12:06:27.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:27.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:27.843472+0000) 2019-10-13T12:06:28.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:28.068+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.069795+0000) 2019-10-13T12:06:28.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:28.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.090900+0000) 2019-10-13T12:06:28.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:28.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:28.611492+0000) 2019-10-13T12:06:30.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:30.180+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.182021+0000) 2019-10-13T12:06:30.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:30.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.182370+0000) 2019-10-13T12:06:30.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:30.180+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.182730+0000) 2019-10-13T12:06:30.185 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:30.180+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.184923+0000) 2019-10-13T12:06:30.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:30.180+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.185181+0000) 2019-10-13T12:06:30.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:30.184+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.187084+0000) 2019-10-13T12:06:30.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:30.182+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.187881+0000) 2019-10-13T12:06:30.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:30.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:30.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:30.194+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:30.196542+0000) 2019-10-13T12:06:31.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:31.858+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:31.863105+0000) 2019-10-13T12:06:33.527 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:06:33.527 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:33.530 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:06:33.531 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:06:34.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:34.178+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:34.182772+0000) 2019-10-13T12:06:34.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:34.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:34.186312+0000) 2019-10-13T12:06:34.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:34.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:34.472102+0000) 2019-10-13T12:06:35.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:35.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.192486+0000) 2019-10-13T12:06:35.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:35.188+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.193155+0000) 2019-10-13T12:06:35.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:35.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.193980+0000) 2019-10-13T12:06:35.198 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:35.192+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.197504+0000) 2019-10-13T12:06:35.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:35.196+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.197855+0000) 2019-10-13T12:06:35.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:35.196+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.198233+0000) 2019-10-13T12:06:35.199 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:35.194+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.198852+0000) 2019-10-13T12:06:35.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:35.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:35.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:35.202+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:35.204646+0000) 2019-10-13T12:06:37.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:37.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:37.843671+0000) 2019-10-13T12:06:38.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:38.068+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.070016+0000) 2019-10-13T12:06:38.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:38.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.091134+0000) 2019-10-13T12:06:38.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:38.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:38.611687+0000) 2019-10-13T12:06:40.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:40.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.203430+0000) 2019-10-13T12:06:40.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:40.202+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.204306+0000) 2019-10-13T12:06:40.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:40.200+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.203922+0000) 2019-10-13T12:06:40.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:40.200+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.204618+0000) 2019-10-13T12:06:40.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:40.200+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.205331+0000) 2019-10-13T12:06:40.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:40.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.205993+0000) 2019-10-13T12:06:40.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:40.204+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.206239+0000) 2019-10-13T12:06:40.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:40.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:40.213 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:40.210+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:40.212846+0000) 2019-10-13T12:06:41.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:41.858+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:41.863303+0000) 2019-10-13T12:06:44.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:44.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.186517+0000) 2019-10-13T12:06:44.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:44.186+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.189432+0000) 2019-10-13T12:06:44.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:44.468+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:44.472363+0000) 2019-10-13T12:06:45.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:45.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.213501+0000) 2019-10-13T12:06:45.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:45.210+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.213871+0000) 2019-10-13T12:06:45.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:45.212+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.214214+0000) 2019-10-13T12:06:45.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:45.212+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.215013+0000) 2019-10-13T12:06:45.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:45.212+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.217403+0000) 2019-10-13T12:06:45.218 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:45.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.217731+0000) 2019-10-13T12:06:45.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:45.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.218718+0000) 2019-10-13T12:06:45.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:45.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153367 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:45.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:45.218+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:45.223588+0000) 2019-10-13T12:06:47.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:47.840+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:47.843873+0000) 2019-10-13T12:06:48.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:48.068+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.070215+0000) 2019-10-13T12:06:48.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:48.088+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.091368+0000) 2019-10-13T12:06:48.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:48.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:48.611883+0000) 2019-10-13T12:06:50.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:50.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.224511+0000) 2019-10-13T12:06:50.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:50.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.224817+0000) 2019-10-13T12:06:50.227 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:50.222+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.226329+0000) 2019-10-13T12:06:50.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:50.223+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.226645+0000) 2019-10-13T12:06:50.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:50.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.226667+0000) 2019-10-13T12:06:50.228 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:50.223+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.227075+0000) 2019-10-13T12:06:50.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:50.223+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.227922+0000) 2019-10-13T12:06:50.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:50.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:50.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:50.230+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:50.235422+0000) 2019-10-13T12:06:51.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:51.858+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:51.863530+0000) 2019-10-13T12:06:54.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:54.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.186723+0000) 2019-10-13T12:06:54.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:54.194+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.195905+0000) 2019-10-13T12:06:54.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:54.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:54.472563+0000) 2019-10-13T12:06:55.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:06:55.231+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.235756+0000) 2019-10-13T12:06:55.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:06:55.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.236449+0000) 2019-10-13T12:06:55.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:55.231+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.236738+0000) 2019-10-13T12:06:55.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:55.231+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.237142+0000) 2019-10-13T12:06:55.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:55.231+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.237546+0000) 2019-10-13T12:06:55.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:55.235+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.238194+0000) 2019-10-13T12:06:55.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:06:55.234+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.238434+0000) 2019-10-13T12:06:55.244 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:06:55.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:06:55.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:06:55.242+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:55.244987+0000) 2019-10-13T12:06:57.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:06:57.839+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:57.844090+0000) 2019-10-13T12:06:58.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:06:58.067+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.070417+0000) 2019-10-13T12:06:58.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:06:58.087+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.091565+0000) 2019-10-13T12:06:58.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:06:58.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:06:58.612091+0000) 2019-10-13T12:07:00.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:00.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.246677+0000) 2019-10-13T12:07:00.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:00.243+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.247179+0000) 2019-10-13T12:07:00.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:00.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.247746+0000) 2019-10-13T12:07:00.250 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:00.243+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.249086+0000) 2019-10-13T12:07:00.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:00.243+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.249583+0000) 2019-10-13T12:07:00.250 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:00.247+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.250039+0000) 2019-10-13T12:07:00.251 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:00.246+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.250706+0000) 2019-10-13T12:07:00.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:00.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:00.260 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:00.254+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:00.259261+0000) 2019-10-13T12:07:01.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:01.862+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:01.863751+0000) 2019-10-13T12:07:03.624 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:07:03.624 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:03.628 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:07:03.628 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:04.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:04.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.186932+0000) 2019-10-13T12:07:04.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:04.198+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.202105+0000) 2019-10-13T12:07:04.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:04.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:04.472797+0000) 2019-10-13T12:07:05.259 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:05.255+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.258080+0000) 2019-10-13T12:07:05.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:05.255+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.258699+0000) 2019-10-13T12:07:05.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:05.255+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.259379+0000) 2019-10-13T12:07:05.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:05.255+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.260742+0000) 2019-10-13T12:07:05.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:05.255+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.260925+0000) 2019-10-13T12:07:05.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:05.255+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.261364+0000) 2019-10-13T12:07:05.263 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:05.258+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.261901+0000) 2019-10-13T12:07:05.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:05.262+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:05.266648+0000) 2019-10-13T12:07:05.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:05.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:07.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:07.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:07.844293+0000) 2019-10-13T12:07:08.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:08.067+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.070614+0000) 2019-10-13T12:07:08.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:08.087+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.091777+0000) 2019-10-13T12:07:08.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:08.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:08.612289+0000) 2019-10-13T12:07:10.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:10.267+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.270245+0000) 2019-10-13T12:07:10.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:10.267+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.270834+0000) 2019-10-13T12:07:10.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:10.270+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.272780+0000) 2019-10-13T12:07:10.274 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:10.267+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.273098+0000) 2019-10-13T12:07:10.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:10.271+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.274080+0000) 2019-10-13T12:07:10.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:10.271+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.273861+0000) 2019-10-13T12:07:10.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:10.271+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.274686+0000) 2019-10-13T12:07:10.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:10.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:10.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:10.274+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:10.278881+0000) 2019-10-13T12:07:11.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:11.862+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:11.863944+0000) 2019-10-13T12:07:14.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:14.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.187140+0000) 2019-10-13T12:07:14.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:14.206+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.208622+0000) 2019-10-13T12:07:14.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:14.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:14.473010+0000) 2019-10-13T12:07:15.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:15.275+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.281111+0000) 2019-10-13T12:07:15.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:15.278+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.281569+0000) 2019-10-13T12:07:15.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:15.279+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.281982+0000) 2019-10-13T12:07:15.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:15.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.283065+0000) 2019-10-13T12:07:15.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:15.279+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.283493+0000) 2019-10-13T12:07:15.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:15.279+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.284521+0000) 2019-10-13T12:07:15.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:15.279+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.284745+0000) 2019-10-13T12:07:15.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:15.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:15.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:15.286+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:15.290871+0000) 2019-10-13T12:07:17.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:17.839+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:17.844496+0000) 2019-10-13T12:07:18.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:18.067+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.070827+0000) 2019-10-13T12:07:18.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:18.091+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.091974+0000) 2019-10-13T12:07:18.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:18.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:18.612538+0000) 2019-10-13T12:07:20.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:20.287+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.292041+0000) 2019-10-13T12:07:20.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:20.287+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.292455+0000) 2019-10-13T12:07:20.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:20.287+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.292985+0000) 2019-10-13T12:07:20.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:20.289+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.294051+0000) 2019-10-13T12:07:20.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:20.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.296514+0000) 2019-10-13T12:07:20.299 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:20.295+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.298265+0000) 2019-10-13T12:07:20.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:20.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.298848+0000) 2019-10-13T12:07:20.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:20.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:20.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:20.297+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:20.303357+0000) 2019-10-13T12:07:21.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:21.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:21.864143+0000) 2019-10-13T12:07:24.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:24.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.187339+0000) 2019-10-13T12:07:24.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:24.209+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.215189+0000) 2019-10-13T12:07:24.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:24.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:24.473211+0000) 2019-10-13T12:07:25.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:25.299+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.302495+0000) 2019-10-13T12:07:25.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:25.297+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.303232+0000) 2019-10-13T12:07:25.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:25.299+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.303696+0000) 2019-10-13T12:07:25.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:25.299+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.304433+0000) 2019-10-13T12:07:25.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:25.299+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.305450+0000) 2019-10-13T12:07:25.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:25.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.305977+0000) 2019-10-13T12:07:25.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:25.303+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.306446+0000) 2019-10-13T12:07:25.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:25.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:25.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:25.309+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:25.312154+0000) 2019-10-13T12:07:27.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:27.839+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:27.844723+0000) 2019-10-13T12:07:28.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:28.067+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.071025+0000) 2019-10-13T12:07:28.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:28.087+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.092201+0000) 2019-10-13T12:07:28.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:28.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:28.612763+0000) 2019-10-13T12:07:30.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:30.311+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.314908+0000) 2019-10-13T12:07:30.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:30.311+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.315126+0000) 2019-10-13T12:07:30.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:30.311+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.317393+0000) 2019-10-13T12:07:30.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:30.315+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.317831+0000) 2019-10-13T12:07:30.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:30.315+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.318424+0000) 2019-10-13T12:07:30.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:30.315+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.320079+0000) 2019-10-13T12:07:30.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:30.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:30.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:30.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.322360+0000) 2019-10-13T12:07:30.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:30.321+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:30.325099+0000) 2019-10-13T12:07:31.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:31.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:31.864338+0000) 2019-10-13T12:07:33.689 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:07:33.689 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:33.693 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:07:33.693 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:07:34.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:34.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.187543+0000) 2019-10-13T12:07:34.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:34.221+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.221838+0000) 2019-10-13T12:07:34.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:34.467+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:34.473442+0000) 2019-10-13T12:07:35.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:35.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.326135+0000) 2019-10-13T12:07:35.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:35.323+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.326707+0000) 2019-10-13T12:07:35.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:35.323+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.327036+0000) 2019-10-13T12:07:35.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:35.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.327322+0000) 2019-10-13T12:07:35.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:35.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.327871+0000) 2019-10-13T12:07:35.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:35.325+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.328300+0000) 2019-10-13T12:07:35.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:35.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.329190+0000) 2019-10-13T12:07:35.335 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:35.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153680 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:35.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:35.333+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:35.336103+0000) 2019-10-13T12:07:37.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:37.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:37.844948+0000) 2019-10-13T12:07:38.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:38.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.071260+0000) 2019-10-13T12:07:38.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:38.091+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.092401+0000) 2019-10-13T12:07:38.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:38.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:38.612961+0000) 2019-10-13T12:07:40.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:40.331+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.337688+0000) 2019-10-13T12:07:40.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:40.335+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.338331+0000) 2019-10-13T12:07:40.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:40.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.339380+0000) 2019-10-13T12:07:40.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:40.335+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.339909+0000) 2019-10-13T12:07:40.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:40.335+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.340514+0000) 2019-10-13T12:07:40.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:40.335+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.341653+0000) 2019-10-13T12:07:40.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:40.337+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.341100+0000) 2019-10-13T12:07:40.346 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:40.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:40.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:40.345+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:40.349680+0000) 2019-10-13T12:07:41.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:41.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:41.864526+0000) 2019-10-13T12:07:44.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:44.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.187787+0000) 2019-10-13T12:07:44.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:44.225+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.228681+0000) 2019-10-13T12:07:44.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:44.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:44.473641+0000) 2019-10-13T12:07:45.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:45.346+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.349802+0000) 2019-10-13T12:07:45.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:45.346+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.349861+0000) 2019-10-13T12:07:45.351 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:45.346+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.350709+0000) 2019-10-13T12:07:45.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:45.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.351124+0000) 2019-10-13T12:07:45.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:45.346+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.351708+0000) 2019-10-13T12:07:45.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:45.346+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.352507+0000) 2019-10-13T12:07:45.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:45.349+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.353549+0000) 2019-10-13T12:07:45.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:45.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:45.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:45.353+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:45.358271+0000) 2019-10-13T12:07:47.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:47.838+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:47.845165+0000) 2019-10-13T12:07:48.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:48.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.071480+0000) 2019-10-13T12:07:48.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:48.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.092601+0000) 2019-10-13T12:07:48.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:48.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:48.613160+0000) 2019-10-13T12:07:50.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:50.354+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.360657+0000) 2019-10-13T12:07:50.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:50.358+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.361835+0000) 2019-10-13T12:07:50.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:50.357+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.362644+0000) 2019-10-13T12:07:50.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:50.358+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.362924+0000) 2019-10-13T12:07:50.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:50.358+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.363760+0000) 2019-10-13T12:07:50.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:50.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.364058+0000) 2019-10-13T12:07:50.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:50.358+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.364653+0000) 2019-10-13T12:07:50.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:50.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:50.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:50.365+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:50.371616+0000) 2019-10-13T12:07:51.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:51.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:51.864721+0000) 2019-10-13T12:07:54.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:54.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.187984+0000) 2019-10-13T12:07:54.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:54.233+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.235058+0000) 2019-10-13T12:07:54.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:54.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:54.473842+0000) 2019-10-13T12:07:55.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:55.366+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.371498+0000) 2019-10-13T12:07:55.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:55.366+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.372635+0000) 2019-10-13T12:07:55.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:07:55.366+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.373621+0000) 2019-10-13T12:07:55.375 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:07:55.369+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.374025+0000) 2019-10-13T12:07:55.375 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:55.370+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.374664+0000) 2019-10-13T12:07:55.375 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:07:55.370+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.375165+0000) 2019-10-13T12:07:55.376 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:55.370+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.375977+0000) 2019-10-13T12:07:55.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:07:55.373+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:55.379327+0000) 2019-10-13T12:07:55.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:07:55.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:07:57.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:07:57.838+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:57.845395+0000) 2019-10-13T12:07:58.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:07:58.066+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.071705+0000) 2019-10-13T12:07:58.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:07:58.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.092759+0000) 2019-10-13T12:07:58.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:07:58.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:07:58.613351+0000) 2019-10-13T12:08:00.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:00.378+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.382400+0000) 2019-10-13T12:08:00.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:00.378+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.383042+0000) 2019-10-13T12:08:00.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:00.381+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.384383+0000) 2019-10-13T12:08:00.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:00.378+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.384900+0000) 2019-10-13T12:08:00.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:00.378+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.385238+0000) 2019-10-13T12:08:00.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:00.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.385636+0000) 2019-10-13T12:08:00.387 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:00.382+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.386848+0000) 2019-10-13T12:08:00.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:00.385+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:00.389234+0000) 2019-10-13T12:08:00.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:00.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:01.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:01.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:01.864914+0000) 2019-10-13T12:08:03.785 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:08:03.786 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:03.789 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:08:03.789 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:04.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:04.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.188195+0000) 2019-10-13T12:08:04.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:04.237+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.241564+0000) 2019-10-13T12:08:04.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:04.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:04.474037+0000) 2019-10-13T12:08:05.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:05.390+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.393487+0000) 2019-10-13T12:08:05.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:05.390+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.394948+0000) 2019-10-13T12:08:05.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:05.393+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.397132+0000) 2019-10-13T12:08:05.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:05.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.397433+0000) 2019-10-13T12:08:05.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:05.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.397940+0000) 2019-10-13T12:08:05.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:05.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.398563+0000) 2019-10-13T12:08:05.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:05.394+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.398696+0000) 2019-10-13T12:08:05.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:05.397+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:05.400778+0000) 2019-10-13T12:08:05.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:05.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:07.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:07.842+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:07.845655+0000) 2019-10-13T12:08:08.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:08.066+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.071915+0000) 2019-10-13T12:08:08.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:08.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.092997+0000) 2019-10-13T12:08:08.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:08.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:08.613550+0000) 2019-10-13T12:08:10.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:10.398+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.404749+0000) 2019-10-13T12:08:10.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:10.398+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.405563+0000) 2019-10-13T12:08:10.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:10.402+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.406302+0000) 2019-10-13T12:08:10.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:10.401+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.406805+0000) 2019-10-13T12:08:10.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:10.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.407228+0000) 2019-10-13T12:08:10.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:10.402+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.408735+0000) 2019-10-13T12:08:10.410 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:10.402+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.409376+0000) 2019-10-13T12:08:10.413 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:10.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:10.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:10.413+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:10.416550+0000) 2019-10-13T12:08:11.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:11.861+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:11.865135+0000) 2019-10-13T12:08:14.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:14.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.188436+0000) 2019-10-13T12:08:14.249 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:14.245+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.248169+0000) 2019-10-13T12:08:14.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:14.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:14.474257+0000) 2019-10-13T12:08:15.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:15.410+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.416406+0000) 2019-10-13T12:08:15.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:15.410+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.416767+0000) 2019-10-13T12:08:15.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:15.414+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.418677+0000) 2019-10-13T12:08:15.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:15.414+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.419494+0000) 2019-10-13T12:08:15.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:15.414+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.419639+0000) 2019-10-13T12:08:15.422 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:15.418+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.421858+0000) 2019-10-13T12:08:15.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:15.417+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.422492+0000) 2019-10-13T12:08:15.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:15.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:15.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:15.425+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:15.430511+0000) 2019-10-13T12:08:17.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:17.842+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:17.845858+0000) 2019-10-13T12:08:18.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:18.066+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.072129+0000) 2019-10-13T12:08:18.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:18.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.093179+0000) 2019-10-13T12:08:18.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:18.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:18.613790+0000) 2019-10-13T12:08:20.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:20.422+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.426884+0000) 2019-10-13T12:08:20.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:20.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.427241+0000) 2019-10-13T12:08:20.428 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:20.422+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.427636+0000) 2019-10-13T12:08:20.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:20.422+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.429581+0000) 2019-10-13T12:08:20.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:20.425+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.431831+0000) 2019-10-13T12:08:20.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:20.426+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.433539+0000) 2019-10-13T12:08:20.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:20.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:20.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:20.430+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.435051+0000) 2019-10-13T12:08:20.442 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:20.437+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:20.441845+0000) 2019-10-13T12:08:21.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:21.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:21.865332+0000) 2019-10-13T12:08:24.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:24.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.188669+0000) 2019-10-13T12:08:24.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:24.252+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.257226+0000) 2019-10-13T12:08:24.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:24.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:24.474512+0000) 2019-10-13T12:08:25.438 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:25.430+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.437587+0000) 2019-10-13T12:08:25.440 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:25.432+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.439140+0000) 2019-10-13T12:08:25.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:25.434+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.441197+0000) 2019-10-13T12:08:25.444 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:25.438+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.443602+0000) 2019-10-13T12:08:25.446 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:25.438+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.445635+0000) 2019-10-13T12:08:25.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:25.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:25.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:25.442+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.446776+0000) 2019-10-13T12:08:25.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:25.442+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.447128+0000) 2019-10-13T12:08:25.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:25.448+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:25.452924+0000) 2019-10-13T12:08:27.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:27.842+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:27.846018+0000) 2019-10-13T12:08:28.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:28.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.072304+0000) 2019-10-13T12:08:28.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:28.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.093377+0000) 2019-10-13T12:08:28.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:28.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:28.613992+0000) 2019-10-13T12:08:30.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:30.446+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.451075+0000) 2019-10-13T12:08:30.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:30.446+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.452140+0000) 2019-10-13T12:08:30.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:30.450+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.454404+0000) 2019-10-13T12:08:30.455 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:30.448+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.454814+0000) 2019-10-13T12:08:30.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:30.450+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.455205+0000) 2019-10-13T12:08:30.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:30.450+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.456055+0000) 2019-10-13T12:08:30.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:30.450+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.456761+0000) 2019-10-13T12:08:30.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:30.452+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:30.457235+0000) 2019-10-13T12:08:30.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:30.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:31.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:31.860+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:31.865537+0000) 2019-10-13T12:08:33.927 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:08:33.927 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:33.931 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:08:33.931 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:08:34.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:34.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.188876+0000) 2019-10-13T12:08:34.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:34.260+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.263763+0000) 2019-10-13T12:08:34.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:34.470+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:34.474733+0000) 2019-10-13T12:08:35.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:35.454+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.461178+0000) 2019-10-13T12:08:35.462 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:35.454+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.461659+0000) 2019-10-13T12:08:35.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:35.458+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.462946+0000) 2019-10-13T12:08:35.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:35.458+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.463378+0000) 2019-10-13T12:08:35.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:35.458+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.463714+0000) 2019-10-13T12:08:35.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:35.460+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.464243+0000) 2019-10-13T12:08:35.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:35.458+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.465091+0000) 2019-10-13T12:08:35.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:35.464+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:35.468995+0000) 2019-10-13T12:08:35.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:35.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:37.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:37.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:37.846242+0000) 2019-10-13T12:08:38.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:38.066+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.072475+0000) 2019-10-13T12:08:38.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:38.086+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.093606+0000) 2019-10-13T12:08:38.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:38.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:38.614210+0000) 2019-10-13T12:08:40.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:40.468+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.472487+0000) 2019-10-13T12:08:40.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:40.466+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.473532+0000) 2019-10-13T12:08:40.477 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:40.470+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.475827+0000) 2019-10-13T12:08:40.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:40.470+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.475461+0000) 2019-10-13T12:08:40.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:40.470+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.477321+0000) 2019-10-13T12:08:40.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:40.474+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.478505+0000) 2019-10-13T12:08:40.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:40.474+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.479332+0000) 2019-10-13T12:08:40.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:40.474+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:40.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:40.476+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:40.483073+0000) 2019-10-13T12:08:41.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:41.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:41.865743+0000) 2019-10-13T12:08:44.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:44.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.189088+0000) 2019-10-13T12:08:44.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:44.268+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.270256+0000) 2019-10-13T12:08:44.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:44.469+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:44.474935+0000) 2019-10-13T12:08:45.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:45.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.483931+0000) 2019-10-13T12:08:45.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:45.477+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.484334+0000) 2019-10-13T12:08:45.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:45.477+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.484766+0000) 2019-10-13T12:08:45.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:45.477+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.485551+0000) 2019-10-13T12:08:45.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:45.481+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.486074+0000) 2019-10-13T12:08:45.488 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:45.484+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.488237+0000) 2019-10-13T12:08:45.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:45.485+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.489918+0000) 2019-10-13T12:08:45.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:45.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:45.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:45.492+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:45.498341+0000) 2019-10-13T12:08:47.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:47.841+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:47.846459+0000) 2019-10-13T12:08:48.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:48.069+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.072641+0000) 2019-10-13T12:08:48.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:48.089+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.093801+0000) 2019-10-13T12:08:48.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:48.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:48.614421+0000) 2019-10-13T12:08:50.498 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:50.489+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.496975+0000) 2019-10-13T12:08:50.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:50.493+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.497791+0000) 2019-10-13T12:08:50.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:50.492+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.498046+0000) 2019-10-13T12:08:50.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:50.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.498216+0000) 2019-10-13T12:08:50.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:50.493+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.498884+0000) 2019-10-13T12:08:50.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:50.493+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.500152+0000) 2019-10-13T12:08:50.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:50.493+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.500561+0000) 2019-10-13T12:08:50.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:50.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:50.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:50.500+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:50.504964+0000) 2019-10-13T12:08:51.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:51.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:51.865998+0000) 2019-10-13T12:08:54.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:54.185+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.189292+0000) 2019-10-13T12:08:54.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:54.272+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.276712+0000) 2019-10-13T12:08:54.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:54.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:54.475138+0000) 2019-10-13T12:08:55.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:08:55.505+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.506931+0000) 2019-10-13T12:08:55.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:08:55.505+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.508149+0000) 2019-10-13T12:08:55.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:55.505+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.508559+0000) 2019-10-13T12:08:55.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:55.505+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.508593+0000) 2019-10-13T12:08:55.510 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:55.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.509101+0000) 2019-10-13T12:08:55.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:08:55.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.509767+0000) 2019-10-13T12:08:55.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:55.509+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.510330+0000) 2019-10-13T12:08:55.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:08:55.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154184 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:08:55.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:08:55.512+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:55.519475+0000) 2019-10-13T12:08:57.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:08:57.845+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:57.846670+0000) 2019-10-13T12:08:58.073 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:08:58.069+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.072818+0000) 2019-10-13T12:08:58.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:08:58.093+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.094009+0000) 2019-10-13T12:08:58.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:08:58.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:08:58.614621+0000) 2019-10-13T12:09:00.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:00.513+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.517549+0000) 2019-10-13T12:09:00.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:00.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.518009+0000) 2019-10-13T12:09:00.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:00.517+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.518903+0000) 2019-10-13T12:09:00.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:00.517+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.519553+0000) 2019-10-13T12:09:00.521 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:00.517+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.520187+0000) 2019-10-13T12:09:00.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:00.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.522219+0000) 2019-10-13T12:09:00.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:00.521+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.522552+0000) 2019-10-13T12:09:00.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:00.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:00.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:00.524+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:00.529025+0000) 2019-10-13T12:09:01.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:01.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:01.866195+0000) 2019-10-13T12:09:04.042 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:09:04.043 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:04.046 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:09:04.047 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:04.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:04.185+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.189553+0000) 2019-10-13T12:09:04.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:04.280+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.283218+0000) 2019-10-13T12:09:04.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:04.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:04.475342+0000) 2019-10-13T12:09:05.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:05.525+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.528442+0000) 2019-10-13T12:09:05.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:05.525+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.528931+0000) 2019-10-13T12:09:05.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:05.525+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.529506+0000) 2019-10-13T12:09:05.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:05.529+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.530043+0000) 2019-10-13T12:09:05.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:05.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.531161+0000) 2019-10-13T12:09:05.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:05.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.531773+0000) 2019-10-13T12:09:05.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:05.529+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.533389+0000) 2019-10-13T12:09:05.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:05.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:05.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:05.532+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:05.537057+0000) 2019-10-13T12:09:07.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:07.845+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:07.846929+0000) 2019-10-13T12:09:08.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:08.069+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.073026+0000) 2019-10-13T12:09:08.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:08.093+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.094192+0000) 2019-10-13T12:09:08.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:08.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:08.614860+0000) 2019-10-13T12:09:10.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:10.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.540102+0000) 2019-10-13T12:09:10.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:10.537+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.540942+0000) 2019-10-13T12:09:10.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:10.537+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.541507+0000) 2019-10-13T12:09:10.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:10.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.542229+0000) 2019-10-13T12:09:10.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:10.541+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.542846+0000) 2019-10-13T12:09:10.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:10.541+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.543328+0000) 2019-10-13T12:09:10.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:10.541+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.543799+0000) 2019-10-13T12:09:10.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:10.540+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:10.546720+0000) 2019-10-13T12:09:10.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:10.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:11.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:11.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:11.866423+0000) 2019-10-13T12:09:14.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:14.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.189775+0000) 2019-10-13T12:09:14.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:14.288+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.289883+0000) 2019-10-13T12:09:14.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:14.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:14.475550+0000) 2019-10-13T12:09:15.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:15.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.552199+0000) 2019-10-13T12:09:15.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:15.549+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.552610+0000) 2019-10-13T12:09:15.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:15.549+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.553185+0000) 2019-10-13T12:09:15.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:15.549+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.553631+0000) 2019-10-13T12:09:15.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:15.553+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.553730+0000) 2019-10-13T12:09:15.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:15.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.554349+0000) 2019-10-13T12:09:15.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:15.553+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.554944+0000) 2019-10-13T12:09:15.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:15.552+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:15.557942+0000) 2019-10-13T12:09:15.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:15.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:17.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:17.845+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:17.847144+0000) 2019-10-13T12:09:18.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:18.069+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.073202+0000) 2019-10-13T12:09:18.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:18.093+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.094369+0000) 2019-10-13T12:09:18.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:18.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:18.615077+0000) 2019-10-13T12:09:20.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:20.561+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.562071+0000) 2019-10-13T12:09:20.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:20.561+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.562283+0000) 2019-10-13T12:09:20.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:20.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.564483+0000) 2019-10-13T12:09:20.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:20.561+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.564197+0000) 2019-10-13T12:09:20.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:20.561+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.564699+0000) 2019-10-13T12:09:20.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:20.561+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.564890+0000) 2019-10-13T12:09:20.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:20.561+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.565168+0000) 2019-10-13T12:09:20.570 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:20.564+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:20.570104+0000) 2019-10-13T12:09:20.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:20.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:21.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:21.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:21.866616+0000) 2019-10-13T12:09:24.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:24.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.189975+0000) 2019-10-13T12:09:24.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:24.296+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.296461+0000) 2019-10-13T12:09:24.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:24.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:24.475750+0000) 2019-10-13T12:09:25.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:25.569+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.573386+0000) 2019-10-13T12:09:25.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:25.573+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.575282+0000) 2019-10-13T12:09:25.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:25.573+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.575677+0000) 2019-10-13T12:09:25.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:25.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.576146+0000) 2019-10-13T12:09:25.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:25.577+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.578641+0000) 2019-10-13T12:09:25.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:25.580+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.580442+0000) 2019-10-13T12:09:25.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:25.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:25.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:25.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.581934+0000) 2019-10-13T12:09:25.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:25.584+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:25.584786+0000) 2019-10-13T12:09:27.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:27.845+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:27.847301+0000) 2019-10-13T12:09:28.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:28.069+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.073393+0000) 2019-10-13T12:09:28.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:28.093+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.094563+0000) 2019-10-13T12:09:28.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:28.613+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:28.615298+0000) 2019-10-13T12:09:30.586 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:30.581+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.585190+0000) 2019-10-13T12:09:30.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:30.585+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.587348+0000) 2019-10-13T12:09:30.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:30.585+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.587520+0000) 2019-10-13T12:09:30.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:30.589+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.589926+0000) 2019-10-13T12:09:30.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:30.589+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.590118+0000) 2019-10-13T12:09:30.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:30.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.590344+0000) 2019-10-13T12:09:30.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:30.591+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.592565+0000) 2019-10-13T12:09:30.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:30.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:30.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:30.591+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:30.594657+0000) 2019-10-13T12:09:31.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:31.863+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:31.866806+0000) 2019-10-13T12:09:34.157 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:09:34.157 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:34.162 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:09:34.163 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:09:34.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:34.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.190167+0000) 2019-10-13T12:09:34.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:34.299+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.302866+0000) 2019-10-13T12:09:34.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:34.473+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:34.475952+0000) 2019-10-13T12:09:35.598 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:35.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.597264+0000) 2019-10-13T12:09:35.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:35.597+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.598925+0000) 2019-10-13T12:09:35.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:35.597+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.599137+0000) 2019-10-13T12:09:35.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:35.597+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.601422+0000) 2019-10-13T12:09:35.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:35.599+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.601778+0000) 2019-10-13T12:09:35.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:35.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.601988+0000) 2019-10-13T12:09:35.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:35.601+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.602618+0000) 2019-10-13T12:09:35.606 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:35.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:35.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:35.607+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:35.609146+0000) 2019-10-13T12:09:37.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:37.844+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:37.847501+0000) 2019-10-13T12:09:38.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:38.068+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.073595+0000) 2019-10-13T12:09:38.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:38.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.094763+0000) 2019-10-13T12:09:38.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:38.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:38.615528+0000) 2019-10-13T12:09:40.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:40.604+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.608623+0000) 2019-10-13T12:09:40.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:40.604+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.609377+0000) 2019-10-13T12:09:40.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:40.604+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.609518+0000) 2019-10-13T12:09:40.611 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:40.608+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.610078+0000) 2019-10-13T12:09:40.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:40.608+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.610633+0000) 2019-10-13T12:09:40.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:40.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.611307+0000) 2019-10-13T12:09:40.612 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:40.608+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.611869+0000) 2019-10-13T12:09:40.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:40.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:40.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:40.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:40.617828+0000) 2019-10-13T12:09:41.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:41.863+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:41.867014+0000) 2019-10-13T12:09:44.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:44.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.190376+0000) 2019-10-13T12:09:44.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:44.307+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.309405+0000) 2019-10-13T12:09:44.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:44.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:44.476158+0000) 2019-10-13T12:09:45.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:45.616+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.621208+0000) 2019-10-13T12:09:45.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:45.620+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.623049+0000) 2019-10-13T12:09:45.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:45.620+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.623359+0000) 2019-10-13T12:09:45.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:45.620+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.623731+0000) 2019-10-13T12:09:45.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:45.620+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.624143+0000) 2019-10-13T12:09:45.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:45.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.625833+0000) 2019-10-13T12:09:45.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:45.624+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.626078+0000) 2019-10-13T12:09:45.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:45.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:45.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:45.627+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:45.629132+0000) 2019-10-13T12:09:47.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:47.844+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:47.847738+0000) 2019-10-13T12:09:48.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:48.072+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.073813+0000) 2019-10-13T12:09:48.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:48.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.094917+0000) 2019-10-13T12:09:48.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:48.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:48.615726+0000) 2019-10-13T12:09:50.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:50.628+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.632326+0000) 2019-10-13T12:09:50.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:50.628+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.632710+0000) 2019-10-13T12:09:50.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:50.628+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.633357+0000) 2019-10-13T12:09:50.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:50.631+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.633892+0000) 2019-10-13T12:09:50.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:50.632+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.634532+0000) 2019-10-13T12:09:50.635 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:50.632+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.635235+0000) 2019-10-13T12:09:50.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:50.632+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.635653+0000) 2019-10-13T12:09:50.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:50.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:50.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:50.639+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:50.643580+0000) 2019-10-13T12:09:51.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:51.863+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:51.867231+0000) 2019-10-13T12:09:54.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:54.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.190576+0000) 2019-10-13T12:09:54.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:54.319+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.320674+0000) 2019-10-13T12:09:54.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:54.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:54.476371+0000) 2019-10-13T12:09:55.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:55.640+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.642610+0000) 2019-10-13T12:09:55.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:09:55.639+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.643630+0000) 2019-10-13T12:09:55.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:09:55.640+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.643960+0000) 2019-10-13T12:09:55.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:55.640+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.644394+0000) 2019-10-13T12:09:55.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:55.640+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.645310+0000) 2019-10-13T12:09:55.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:55.644+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.646433+0000) 2019-10-13T12:09:55.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:09:55.644+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.645896+0000) 2019-10-13T12:09:55.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:09:55.647+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:55.651083+0000) 2019-10-13T12:09:55.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:09:55.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:09:57.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:09:57.844+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:57.847958+0000) 2019-10-13T12:09:58.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:09:58.072+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.074040+0000) 2019-10-13T12:09:58.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:09:58.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.095074+0000) 2019-10-13T12:09:58.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:09:58.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:09:58.615916+0000) 2019-10-13T12:10:00.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:00.652+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.654989+0000) 2019-10-13T12:10:00.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:00.652+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.656816+0000) 2019-10-13T12:10:00.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:00.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.657416+0000) 2019-10-13T12:10:00.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:00.656+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.658172+0000) 2019-10-13T12:10:00.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:00.656+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.658590+0000) 2019-10-13T12:10:00.660 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:00.656+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.659383+0000) 2019-10-13T12:10:00.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:00.656+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.659498+0000) 2019-10-13T12:10:00.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:00.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154592 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:00.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:00.659+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:00.663778+0000) 2019-10-13T12:10:01.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:01.863+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:01.867444+0000) 2019-10-13T12:10:04.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:04.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.190838+0000) 2019-10-13T12:10:04.282 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:10:04.282 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:04.286 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:10:04.286 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:04.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:04.323+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.327221+0000) 2019-10-13T12:10:04.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:04.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:04.476589+0000) 2019-10-13T12:10:05.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:05.664+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.666584+0000) 2019-10-13T12:10:05.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:05.664+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.666853+0000) 2019-10-13T12:10:05.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:05.664+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.667204+0000) 2019-10-13T12:10:05.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:05.664+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.667815+0000) 2019-10-13T12:10:05.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:05.667+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.668158+0000) 2019-10-13T12:10:05.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:05.664+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.668695+0000) 2019-10-13T12:10:05.670 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:05.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.669350+0000) 2019-10-13T12:10:05.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:05.671+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:05.672816+0000) 2019-10-13T12:10:05.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:05.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:07.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:07.844+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:07.848183+0000) 2019-10-13T12:10:08.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:08.072+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.074275+0000) 2019-10-13T12:10:08.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:08.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.095285+0000) 2019-10-13T12:10:08.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:08.612+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:08.616108+0000) 2019-10-13T12:10:10.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:10.672+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.677086+0000) 2019-10-13T12:10:10.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:10.672+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.676826+0000) 2019-10-13T12:10:10.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:10.675+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.679410+0000) 2019-10-13T12:10:10.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:10.676+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.681321+0000) 2019-10-13T12:10:10.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:10.680+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.682453+0000) 2019-10-13T12:10:10.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:10.680+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.682946+0000) 2019-10-13T12:10:10.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:10.680+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.683296+0000) 2019-10-13T12:10:10.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:10.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:10.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:10.691+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:10.693057+0000) 2019-10-13T12:10:11.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:11.867+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:11.867635+0000) 2019-10-13T12:10:14.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:14.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.191005+0000) 2019-10-13T12:10:14.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:14.335+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.338248+0000) 2019-10-13T12:10:14.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:14.472+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:14.476830+0000) 2019-10-13T12:10:15.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:15.684+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.687801+0000) 2019-10-13T12:10:15.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:15.684+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.688259+0000) 2019-10-13T12:10:15.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:15.684+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.688847+0000) 2019-10-13T12:10:15.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:15.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.690652+0000) 2019-10-13T12:10:15.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:15.688+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.690706+0000) 2019-10-13T12:10:15.692 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:15.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.691051+0000) 2019-10-13T12:10:15.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:15.687+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.691934+0000) 2019-10-13T12:10:15.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:15.695+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:15.696659+0000) 2019-10-13T12:10:15.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:15.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:17.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:17.844+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:17.848442+0000) 2019-10-13T12:10:18.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:18.072+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.074493+0000) 2019-10-13T12:10:18.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:18.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.095516+0000) 2019-10-13T12:10:18.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:18.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:18.616305+0000) 2019-10-13T12:10:20.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:20.696+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.699276+0000) 2019-10-13T12:10:20.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:20.696+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.699837+0000) 2019-10-13T12:10:20.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:20.696+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.700017+0000) 2019-10-13T12:10:20.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:20.696+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.700900+0000) 2019-10-13T12:10:20.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:20.696+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.701202+0000) 2019-10-13T12:10:20.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:20.696+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.701703+0000) 2019-10-13T12:10:20.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:20.699+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.702805+0000) 2019-10-13T12:10:20.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:20.703+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:20.706922+0000) 2019-10-13T12:10:20.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:20.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:21.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:21.863+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:21.867824+0000) 2019-10-13T12:10:24.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:24.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.191239+0000) 2019-10-13T12:10:24.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:24.343+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.344665+0000) 2019-10-13T12:10:24.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:24.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:24.477013+0000) 2019-10-13T12:10:25.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:25.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.709904+0000) 2019-10-13T12:10:25.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:25.708+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.710932+0000) 2019-10-13T12:10:25.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:25.708+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.711697+0000) 2019-10-13T12:10:25.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:25.708+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.713028+0000) 2019-10-13T12:10:25.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:25.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.713283+0000) 2019-10-13T12:10:25.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:25.712+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.713957+0000) 2019-10-13T12:10:25.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:25.711+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.715026+0000) 2019-10-13T12:10:25.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:25.715+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:25.718106+0000) 2019-10-13T12:10:25.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:25.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:27.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:27.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:27.848652+0000) 2019-10-13T12:10:28.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:28.072+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.074716+0000) 2019-10-13T12:10:28.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:28.092+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.095721+0000) 2019-10-13T12:10:28.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:28.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:28.616535+0000) 2019-10-13T12:10:30.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:30.720+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.722524+0000) 2019-10-13T12:10:30.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:30.720+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.724741+0000) 2019-10-13T12:10:30.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:30.723+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.725027+0000) 2019-10-13T12:10:30.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:30.720+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.725262+0000) 2019-10-13T12:10:30.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:30.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.725558+0000) 2019-10-13T12:10:30.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:30.724+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.726115+0000) 2019-10-13T12:10:30.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:30.724+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.726626+0000) 2019-10-13T12:10:30.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:30.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:30.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:30.731+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:30.733028+0000) 2019-10-13T12:10:31.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:31.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:31.868014+0000) 2019-10-13T12:10:34.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:34.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.191440+0000) 2019-10-13T12:10:34.352 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:34.346+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.351067+0000) 2019-10-13T12:10:34.427 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:10:34.428 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:34.432 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:10:34.432 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:10:34.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:34.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:34.477208+0000) 2019-10-13T12:10:35.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:35.727+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.733625+0000) 2019-10-13T12:10:35.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:35.731+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.735894+0000) 2019-10-13T12:10:35.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:35.731+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.736190+0000) 2019-10-13T12:10:35.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:35.731+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.737692+0000) 2019-10-13T12:10:35.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:35.734+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.737954+0000) 2019-10-13T12:10:35.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:35.735+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.738655+0000) 2019-10-13T12:10:35.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:35.735+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.739120+0000) 2019-10-13T12:10:35.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:35.735+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:35.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:35.742+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:35.744100+0000) 2019-10-13T12:10:37.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:37.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:37.848849+0000) 2019-10-13T12:10:38.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:38.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.074950+0000) 2019-10-13T12:10:38.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:38.091+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.095922+0000) 2019-10-13T12:10:38.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:38.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:38.616723+0000) 2019-10-13T12:10:40.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:40.739+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.744901+0000) 2019-10-13T12:10:40.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:40.743+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.746986+0000) 2019-10-13T12:10:40.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:40.743+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.747951+0000) 2019-10-13T12:10:40.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:40.743+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.748410+0000) 2019-10-13T12:10:40.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:40.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.749526+0000) 2019-10-13T12:10:40.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:40.747+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.750120+0000) 2019-10-13T12:10:40.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:40.747+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.750183+0000) 2019-10-13T12:10:40.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:40.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:40.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:40.750+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:40.755574+0000) 2019-10-13T12:10:41.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:41.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:41.868205+0000) 2019-10-13T12:10:44.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:44.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.191677+0000) 2019-10-13T12:10:44.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:44.354+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.357593+0000) 2019-10-13T12:10:44.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:44.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:44.477375+0000) 2019-10-13T12:10:45.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:45.751+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.757392+0000) 2019-10-13T12:10:45.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:45.755+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.758048+0000) 2019-10-13T12:10:45.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:45.755+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.758648+0000) 2019-10-13T12:10:45.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:45.754+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.759710+0000) 2019-10-13T12:10:45.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:45.755+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.760070+0000) 2019-10-13T12:10:45.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:45.755+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.760440+0000) 2019-10-13T12:10:45.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:45.755+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.760868+0000) 2019-10-13T12:10:45.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:45.759+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:45.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:45.762+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:45.767369+0000) 2019-10-13T12:10:47.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:47.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:47.849056+0000) 2019-10-13T12:10:48.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:48.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.075178+0000) 2019-10-13T12:10:48.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:48.095+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.096147+0000) 2019-10-13T12:10:48.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:48.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:48.616924+0000) 2019-10-13T12:10:50.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:50.763+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.768929+0000) 2019-10-13T12:10:50.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:50.767+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.769969+0000) 2019-10-13T12:10:50.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:50.767+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.770267+0000) 2019-10-13T12:10:50.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:50.767+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.770816+0000) 2019-10-13T12:10:50.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:50.766+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.771397+0000) 2019-10-13T12:10:50.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:50.767+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.771438+0000) 2019-10-13T12:10:50.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:50.767+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.771881+0000) 2019-10-13T12:10:50.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:50.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:50.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:50.770+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:50.775448+0000) 2019-10-13T12:10:51.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:51.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:51.868429+0000) 2019-10-13T12:10:54.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:54.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.191889+0000) 2019-10-13T12:10:54.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:54.362+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.367111+0000) 2019-10-13T12:10:54.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:54.471+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:54.477576+0000) 2019-10-13T12:10:55.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:10:55.771+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.777363+0000) 2019-10-13T12:10:55.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:55.775+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.778051+0000) 2019-10-13T12:10:55.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:55.775+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.779616+0000) 2019-10-13T12:10:55.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:10:55.778+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.780149+0000) 2019-10-13T12:10:55.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:55.775+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.780799+0000) 2019-10-13T12:10:55.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:10:55.775+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.780781+0000) 2019-10-13T12:10:55.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:55.775+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.781358+0000) 2019-10-13T12:10:55.785 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:10:55.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:10:55.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:10:55.782+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:55.786569+0000) 2019-10-13T12:10:57.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:10:57.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:57.849284+0000) 2019-10-13T12:10:58.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:10:58.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.075375+0000) 2019-10-13T12:10:58.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:10:58.091+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.096372+0000) 2019-10-13T12:10:58.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:10:58.611+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:10:58.617133+0000) 2019-10-13T12:11:00.789 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:00.783+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.788190+0000) 2019-10-13T12:11:00.790 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:00.783+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.789423+0000) 2019-10-13T12:11:00.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:00.787+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.790101+0000) 2019-10-13T12:11:00.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:00.787+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.790415+0000) 2019-10-13T12:11:00.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:00.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.791007+0000) 2019-10-13T12:11:00.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:00.787+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.793057+0000) 2019-10-13T12:11:00.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:00.791+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.794145+0000) 2019-10-13T12:11:00.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:00.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:00.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:00.794+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:00.799293+0000) 2019-10-13T12:11:01.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:01.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:01.868619+0000) 2019-10-13T12:11:04.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:04.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.192096+0000) 2019-10-13T12:11:04.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:04.370+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.373617+0000) 2019-10-13T12:11:04.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:04.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:04.477832+0000) 2019-10-13T12:11:04.542 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:11:04.542 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:04.546 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:11:04.546 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:05.800 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:05.795+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.799824+0000) 2019-10-13T12:11:05.801 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:05.795+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.800441+0000) 2019-10-13T12:11:05.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:05.798+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.801218+0000) 2019-10-13T12:11:05.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:05.795+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.801397+0000) 2019-10-13T12:11:05.802 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:05.799+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.802014+0000) 2019-10-13T12:11:05.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:05.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.802888+0000) 2019-10-13T12:11:05.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:05.799+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.803290+0000) 2019-10-13T12:11:05.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:05.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:05.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:05.806+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:05.810949+0000) 2019-10-13T12:11:07.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:07.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:07.849484+0000) 2019-10-13T12:11:08.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:08.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.075581+0000) 2019-10-13T12:11:08.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:08.095+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.096567+0000) 2019-10-13T12:11:08.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:08.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:08.617328+0000) 2019-10-13T12:11:10.812 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:10.806+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.811059+0000) 2019-10-13T12:11:10.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:10.807+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.811514+0000) 2019-10-13T12:11:10.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:10.807+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.812338+0000) 2019-10-13T12:11:10.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:10.807+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.812758+0000) 2019-10-13T12:11:10.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:10.807+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.813658+0000) 2019-10-13T12:11:10.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:10.807+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.813260+0000) 2019-10-13T12:11:10.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:10.811+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.814532+0000) 2019-10-13T12:11:10.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:10.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155034 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:10.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:10.818+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:10.820652+0000) 2019-10-13T12:11:11.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:11.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:11.868807+0000) 2019-10-13T12:11:14.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:14.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.192326+0000) 2019-10-13T12:11:14.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:14.378+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.379995+0000) 2019-10-13T12:11:14.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:14.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:14.478043+0000) 2019-10-13T12:11:15.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:15.815+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.821475+0000) 2019-10-13T12:11:15.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:15.819+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.821976+0000) 2019-10-13T12:11:15.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:15.819+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.824756+0000) 2019-10-13T12:11:15.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:15.826+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.828198+0000) 2019-10-13T12:11:15.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:15.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:15.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:15.827+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.830025+0000) 2019-10-13T12:11:15.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:15.826+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.830879+0000) 2019-10-13T12:11:15.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:15.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.830614+0000) 2019-10-13T12:11:15.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:15.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:15.830318+0000) 2019-10-13T12:11:17.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:17.843+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:17.849683+0000) 2019-10-13T12:11:18.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:18.071+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.075810+0000) 2019-10-13T12:11:18.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:18.091+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.096792+0000) 2019-10-13T12:11:18.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:18.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:18.617582+0000) 2019-10-13T12:11:20.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:20.831+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.834169+0000) 2019-10-13T12:11:20.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:20.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.834570+0000) 2019-10-13T12:11:20.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:20.831+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.833897+0000) 2019-10-13T12:11:20.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:20.830+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.835428+0000) 2019-10-13T12:11:20.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:20.831+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.835268+0000) 2019-10-13T12:11:20.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:20.831+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.836028+0000) 2019-10-13T12:11:20.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:20.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.836632+0000) 2019-10-13T12:11:20.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:20.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:20.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:20.838+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:20.842620+0000) 2019-10-13T12:11:21.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:21.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:21.869016+0000) 2019-10-13T12:11:24.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:24.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.192524+0000) 2019-10-13T12:11:24.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:24.382+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.386459+0000) 2019-10-13T12:11:24.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:24.475+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:24.478269+0000) 2019-10-13T12:11:25.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:25.839+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.844038+0000) 2019-10-13T12:11:25.845 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:25.839+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.844152+0000) 2019-10-13T12:11:25.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:25.839+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.844791+0000) 2019-10-13T12:11:25.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:25.843+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.847393+0000) 2019-10-13T12:11:25.848 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:25.842+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.847993+0000) 2019-10-13T12:11:25.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:25.843+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.848987+0000) 2019-10-13T12:11:25.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:25.843+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.849465+0000) 2019-10-13T12:11:25.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:25.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155129 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:25.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:25.850+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:25.853668+0000) 2019-10-13T12:11:27.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:27.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:27.849919+0000) 2019-10-13T12:11:28.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:28.074+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.076006+0000) 2019-10-13T12:11:28.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:28.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.096988+0000) 2019-10-13T12:11:28.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:28.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:28.617774+0000) 2019-10-13T12:11:30.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:30.850+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.854632+0000) 2019-10-13T12:11:30.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:30.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.855411+0000) 2019-10-13T12:11:30.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:30.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.856639+0000) 2019-10-13T12:11:30.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:30.850+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.856894+0000) 2019-10-13T12:11:30.858 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:30.854+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.857934+0000) 2019-10-13T12:11:30.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:30.854+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.858040+0000) 2019-10-13T12:11:30.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:30.854+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.858512+0000) 2019-10-13T12:11:30.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:30.858+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:30.862505+0000) 2019-10-13T12:11:30.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:30.858+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:31.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:31.866+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:31.869204+0000) 2019-10-13T12:11:34.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:34.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.192727+0000) 2019-10-13T12:11:34.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:34.390+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.393016+0000) 2019-10-13T12:11:34.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:34.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:34.478487+0000) 2019-10-13T12:11:34.664 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:11:34.664 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:34.667 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:11:34.667 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:11:35.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:35.862+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.865898+0000) 2019-10-13T12:11:35.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:35.862+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.866313+0000) 2019-10-13T12:11:35.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:35.862+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.866688+0000) 2019-10-13T12:11:35.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:35.862+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.867487+0000) 2019-10-13T12:11:35.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:35.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.868013+0000) 2019-10-13T12:11:35.869 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:35.865+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.868559+0000) 2019-10-13T12:11:35.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:35.862+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.869135+0000) 2019-10-13T12:11:35.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:35.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:35.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:35.869+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:35.875900+0000) 2019-10-13T12:11:37.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:37.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:37.850122+0000) 2019-10-13T12:11:38.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:38.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.076230+0000) 2019-10-13T12:11:38.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:38.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.097235+0000) 2019-10-13T12:11:38.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:38.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:38.617963+0000) 2019-10-13T12:11:40.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:40.870+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.877229+0000) 2019-10-13T12:11:40.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:40.870+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.877695+0000) 2019-10-13T12:11:40.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:40.874+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.878970+0000) 2019-10-13T12:11:40.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:40.873+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.879321+0000) 2019-10-13T12:11:40.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:40.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.879608+0000) 2019-10-13T12:11:40.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:40.874+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.880305+0000) 2019-10-13T12:11:40.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:40.874+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.880869+0000) 2019-10-13T12:11:40.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:40.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155224 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:40.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:40.881+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:40.887815+0000) 2019-10-13T12:11:41.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:41.865+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:41.869397+0000) 2019-10-13T12:11:44.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:44.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.192960+0000) 2019-10-13T12:11:44.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:44.397+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.399546+0000) 2019-10-13T12:11:44.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:44.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:44.478721+0000) 2019-10-13T12:11:45.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:45.882+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.888658+0000) 2019-10-13T12:11:45.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:45.886+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.890019+0000) 2019-10-13T12:11:45.891 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:45.886+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.890013+0000) 2019-10-13T12:11:45.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:45.886+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.890753+0000) 2019-10-13T12:11:45.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:45.886+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.892649+0000) 2019-10-13T12:11:45.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:45.886+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.892927+0000) 2019-10-13T12:11:45.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:45.889+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.893473+0000) 2019-10-13T12:11:45.897 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:45.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:45.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:45.897+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:45.900474+0000) 2019-10-13T12:11:47.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:47.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:47.850343+0000) 2019-10-13T12:11:48.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:48.074+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.076424+0000) 2019-10-13T12:11:48.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:48.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.097432+0000) 2019-10-13T12:11:48.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:48.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:48.618155+0000) 2019-10-13T12:11:50.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:50.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.899404+0000) 2019-10-13T12:11:50.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:50.894+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.900449+0000) 2019-10-13T12:11:50.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:50.894+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.900077+0000) 2019-10-13T12:11:50.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:50.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.900974+0000) 2019-10-13T12:11:50.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:50.894+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.901218+0000) 2019-10-13T12:11:50.903 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:50.898+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.902982+0000) 2019-10-13T12:11:50.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:50.901+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.904477+0000) 2019-10-13T12:11:50.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:50.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:50.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:50.905+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:50.909243+0000) 2019-10-13T12:11:51.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:51.865+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:51.869584+0000) 2019-10-13T12:11:54.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:54.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.193151+0000) 2019-10-13T12:11:54.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:54.401+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.406173+0000) 2019-10-13T12:11:54.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:54.474+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:54.478938+0000) 2019-10-13T12:11:55.912 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:11:55.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.911621+0000) 2019-10-13T12:11:55.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:11:55.909+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.912131+0000) 2019-10-13T12:11:55.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:55.906+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.912323+0000) 2019-10-13T12:11:55.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:11:55.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.912955+0000) 2019-10-13T12:11:55.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:55.906+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.913200+0000) 2019-10-13T12:11:55.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:55.910+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.913858+0000) 2019-10-13T12:11:55.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:55.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.914340+0000) 2019-10-13T12:11:55.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:11:55.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:11:55.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:11:55.917+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:55.921171+0000) 2019-10-13T12:11:57.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:11:57.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:57.850565+0000) 2019-10-13T12:11:58.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:11:58.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.076652+0000) 2019-10-13T12:11:58.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:11:58.090+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.097624+0000) 2019-10-13T12:11:58.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:11:58.614+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:11:58.618387+0000) 2019-10-13T12:12:00.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:00.918+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.923479+0000) 2019-10-13T12:12:00.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:00.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.924614+0000) 2019-10-13T12:12:00.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:00.918+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.924733+0000) 2019-10-13T12:12:00.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:00.921+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.925357+0000) 2019-10-13T12:12:00.926 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:00.922+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.925843+0000) 2019-10-13T12:12:00.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:00.922+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.926367+0000) 2019-10-13T12:12:00.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:00.922+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.927149+0000) 2019-10-13T12:12:00.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:00.925+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:00.929211+0000) 2019-10-13T12:12:00.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:00.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:01.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:01.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:01.869754+0000) 2019-10-13T12:12:04.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:04.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.193351+0000) 2019-10-13T12:12:04.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:04.409+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.412760+0000) 2019-10-13T12:12:04.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:04.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:04.479140+0000) 2019-10-13T12:12:04.761 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:12:04.761 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:04.765 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:12:04.765 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:05.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:05.929+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.933557+0000) 2019-10-13T12:12:05.935 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:05.930+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.934414+0000) 2019-10-13T12:12:05.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:05.930+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.934833+0000) 2019-10-13T12:12:05.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:05.930+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.935981+0000) 2019-10-13T12:12:05.937 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:05.930+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.936694+0000) 2019-10-13T12:12:05.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:05.930+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.937045+0000) 2019-10-13T12:12:05.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:05.930+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.937651+0000) 2019-10-13T12:12:05.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:05.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:05.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:05.937+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:05.941383+0000) 2019-10-13T12:12:07.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:07.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:07.850789+0000) 2019-10-13T12:12:08.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:08.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.076885+0000) 2019-10-13T12:12:08.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:08.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.097843+0000) 2019-10-13T12:12:08.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:08.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:08.618590+0000) 2019-10-13T12:12:10.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:10.938+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.944334+0000) 2019-10-13T12:12:10.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:10.938+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.945573+0000) 2019-10-13T12:12:10.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:10.941+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.947288+0000) 2019-10-13T12:12:10.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:10.942+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.949187+0000) 2019-10-13T12:12:10.951 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:10.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.950936+0000) 2019-10-13T12:12:10.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:10.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:10.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:10.946+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.952823+0000) 2019-10-13T12:12:10.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:10.950+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.954557+0000) 2019-10-13T12:12:10.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:10.953+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:10.959569+0000) 2019-10-13T12:12:11.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:11.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:11.869928+0000) 2019-10-13T12:12:14.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:14.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.193550+0000) 2019-10-13T12:12:14.420 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:14.413+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.419118+0000) 2019-10-13T12:12:14.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:14.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:14.479340+0000) 2019-10-13T12:12:15.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:15.953+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.956201+0000) 2019-10-13T12:12:15.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:15.950+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.956509+0000) 2019-10-13T12:12:15.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:15.950+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.957392+0000) 2019-10-13T12:12:15.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:15.954+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.958096+0000) 2019-10-13T12:12:15.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:15.954+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.958626+0000) 2019-10-13T12:12:15.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:15.954+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.958443+0000) 2019-10-13T12:12:15.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:15.954+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.959565+0000) 2019-10-13T12:12:15.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:15.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:15.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:15.957+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:15.963924+0000) 2019-10-13T12:12:17.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:17.846+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:17.850990+0000) 2019-10-13T12:12:18.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:18.070+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.077099+0000) 2019-10-13T12:12:18.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:18.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.098084+0000) 2019-10-13T12:12:18.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:18.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:18.618787+0000) 2019-10-13T12:12:20.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:20.962+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.966820+0000) 2019-10-13T12:12:20.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:20.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.966486+0000) 2019-10-13T12:12:20.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:20.962+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.967127+0000) 2019-10-13T12:12:20.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:20.965+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.968232+0000) 2019-10-13T12:12:20.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:20.962+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.967957+0000) 2019-10-13T12:12:20.969 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:20.962+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.968739+0000) 2019-10-13T12:12:20.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:20.962+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.969647+0000) 2019-10-13T12:12:20.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:20.970+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:20.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:20.969+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:20.975646+0000) 2019-10-13T12:12:21.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:21.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:21.870116+0000) 2019-10-13T12:12:24.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:24.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.193789+0000) 2019-10-13T12:12:24.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:24.421+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.425672+0000) 2019-10-13T12:12:24.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:24.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:24.479493+0000) 2019-10-13T12:12:25.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:25.973+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.978596+0000) 2019-10-13T12:12:25.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:25.973+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.978984+0000) 2019-10-13T12:12:25.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:25.973+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.979462+0000) 2019-10-13T12:12:25.981 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:25.973+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.979742+0000) 2019-10-13T12:12:25.981 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:25.973+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.980037+0000) 2019-10-13T12:12:25.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:25.977+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.981442+0000) 2019-10-13T12:12:25.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:25.973+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.981571+0000) 2019-10-13T12:12:25.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:25.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:25.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:25.985+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:25.990745+0000) 2019-10-13T12:12:27.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:27.845+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:27.851193+0000) 2019-10-13T12:12:28.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:28.073+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.077304+0000) 2019-10-13T12:12:28.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:28.097+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.098314+0000) 2019-10-13T12:12:28.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:28.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:28.618981+0000) 2019-10-13T12:12:30.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:30.985+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.989492+0000) 2019-10-13T12:12:30.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:30.985+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.990242+0000) 2019-10-13T12:12:30.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:30.985+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.990368+0000) 2019-10-13T12:12:30.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:30.985+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.990802+0000) 2019-10-13T12:12:30.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:30.985+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.991329+0000) 2019-10-13T12:12:30.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:30.985+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.991951+0000) 2019-10-13T12:12:30.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:30.985+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.992594+0000) 2019-10-13T12:12:30.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:30.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:31.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:30.993+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:30.999518+0000) 2019-10-13T12:12:31.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:31.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:31.870308+0000) 2019-10-13T12:12:34.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:34.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.194051+0000) 2019-10-13T12:12:34.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:34.429+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.432145+0000) 2019-10-13T12:12:34.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:34.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:34.479705+0000) 2019-10-13T12:12:35.122 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:12:35.122 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:35.126 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:12:35.126 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:12:36.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:35.993+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.000850+0000) 2019-10-13T12:12:36.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:35.993+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.001329+0000) 2019-10-13T12:12:36.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:35.993+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.001584+0000) 2019-10-13T12:12:36.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:35.997+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.002616+0000) 2019-10-13T12:12:36.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:35.997+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.003327+0000) 2019-10-13T12:12:36.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:35.997+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.004018+0000) 2019-10-13T12:12:36.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:35.997+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.004374+0000) 2019-10-13T12:12:36.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:36.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:36.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:36.005+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:36.011035+0000) 2019-10-13T12:12:37.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:37.849+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:37.851424+0000) 2019-10-13T12:12:38.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:38.073+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.077518+0000) 2019-10-13T12:12:38.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:38.097+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.098513+0000) 2019-10-13T12:12:38.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:38.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:38.619156+0000) 2019-10-13T12:12:41.012 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:41.009+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.010948+0000) 2019-10-13T12:12:41.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:41.009+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.013597+0000) 2019-10-13T12:12:41.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:41.013+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.014175+0000) 2019-10-13T12:12:41.015 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:41.013+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.014610+0000) 2019-10-13T12:12:41.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:41.013+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.015202+0000) 2019-10-13T12:12:41.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:41.012+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.016332+0000) 2019-10-13T12:12:41.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:41.017+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.017852+0000) 2019-10-13T12:12:41.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:41.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:41.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:41.016+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.023112+0000) 2019-10-13T12:12:41.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:41.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:41.870482+0000) 2019-10-13T12:12:44.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:44.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.194259+0000) 2019-10-13T12:12:44.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:44.436+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.438679+0000) 2019-10-13T12:12:44.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:44.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:44.479902+0000) 2019-10-13T12:12:46.025 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:46.021+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.024043+0000) 2019-10-13T12:12:46.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:46.021+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.024251+0000) 2019-10-13T12:12:46.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:46.020+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.027049+0000) 2019-10-13T12:12:46.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:46.021+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.025430+0000) 2019-10-13T12:12:46.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:46.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.025884+0000) 2019-10-13T12:12:46.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:46.025+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.027407+0000) 2019-10-13T12:12:46.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:46.025+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.027655+0000) 2019-10-13T12:12:46.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:46.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:46.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:46.028+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:46.034545+0000) 2019-10-13T12:12:47.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:47.849+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:47.851624+0000) 2019-10-13T12:12:48.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:48.077+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.077779+0000) 2019-10-13T12:12:48.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:48.097+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.098715+0000) 2019-10-13T12:12:48.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:48.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:48.619369+0000) 2019-10-13T12:12:51.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:51.029+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.033702+0000) 2019-10-13T12:12:51.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:51.033+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.034182+0000) 2019-10-13T12:12:51.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:51.032+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.036328+0000) 2019-10-13T12:12:51.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:51.033+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.037302+0000) 2019-10-13T12:12:51.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:51.037+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.037983+0000) 2019-10-13T12:12:51.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:51.037+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.038256+0000) 2019-10-13T12:12:51.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:51.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.038791+0000) 2019-10-13T12:12:51.042 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:51.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:51.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:51.036+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.043652+0000) 2019-10-13T12:12:51.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:51.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:51.870651+0000) 2019-10-13T12:12:54.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:54.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.194479+0000) 2019-10-13T12:12:54.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:54.444+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.444880+0000) 2019-10-13T12:12:54.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:54.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:54.480109+0000) 2019-10-13T12:12:56.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:56.041+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.045238+0000) 2019-10-13T12:12:56.046 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:56.045+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.046021+0000) 2019-10-13T12:12:56.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:56.045+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.046447+0000) 2019-10-13T12:12:56.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:12:56.045+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.047436+0000) 2019-10-13T12:12:56.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:56.045+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.047959+0000) 2019-10-13T12:12:56.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:12:56.045+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.048601+0000) 2019-10-13T12:12:56.049 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:12:56.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.048901+0000) 2019-10-13T12:12:56.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:12:56.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:12:56.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:12:56.048+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:56.055519+0000) 2019-10-13T12:12:57.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:12:57.849+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:57.851824+0000) 2019-10-13T12:12:58.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:12:58.077+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.078001+0000) 2019-10-13T12:12:58.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:12:58.097+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.098953+0000) 2019-10-13T12:12:58.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:12:58.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:12:58.619570+0000) 2019-10-13T12:13:01.057 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:01.053+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.056521+0000) 2019-10-13T12:13:01.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:01.053+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.056909+0000) 2019-10-13T12:13:01.058 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:01.057+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.057802+0000) 2019-10-13T12:13:01.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:01.057+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.057843+0000) 2019-10-13T12:13:01.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:01.057+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.058862+0000) 2019-10-13T12:13:01.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:01.057+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.059098+0000) 2019-10-13T12:13:01.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:01.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.059529+0000) 2019-10-13T12:13:01.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:01.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:01.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:01.060+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.065480+0000) 2019-10-13T12:13:01.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:01.868+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:01.870823+0000) 2019-10-13T12:13:04.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:04.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.194701+0000) 2019-10-13T12:13:04.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:04.448+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.451594+0000) 2019-10-13T12:13:04.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:04.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:04.480333+0000) 2019-10-13T12:13:05.254 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:13:05.254 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:05.258 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:13:05.258 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:06.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:06.065+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.067440+0000) 2019-10-13T12:13:06.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:06.064+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.068291+0000) 2019-10-13T12:13:06.069 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:06.065+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.068839+0000) 2019-10-13T12:13:06.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:06.065+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.069299+0000) 2019-10-13T12:13:06.070 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:06.069+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.070164+0000) 2019-10-13T12:13:06.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:06.069+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.070819+0000) 2019-10-13T12:13:06.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:06.069+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.071114+0000) 2019-10-13T12:13:06.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:06.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:06.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:06.072+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:06.076691+0000) 2019-10-13T12:13:07.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:07.849+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:07.852052+0000) 2019-10-13T12:13:08.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:08.077+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.078203+0000) 2019-10-13T12:13:08.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:08.097+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.099170+0000) 2019-10-13T12:13:08.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:08.617+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:08.619823+0000) 2019-10-13T12:13:11.078 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:11.073+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.077304+0000) 2019-10-13T12:13:11.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:11.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.079685+0000) 2019-10-13T12:13:11.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:11.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.079995+0000) 2019-10-13T12:13:11.081 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:11.076+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.080864+0000) 2019-10-13T12:13:11.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:11.077+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.081093+0000) 2019-10-13T12:13:11.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:11.081+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.082514+0000) 2019-10-13T12:13:11.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:11.081+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.082795+0000) 2019-10-13T12:13:11.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:11.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:11.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:11.080+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.087155+0000) 2019-10-13T12:13:11.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:11.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:11.870999+0000) 2019-10-13T12:13:14.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:14.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.194946+0000) 2019-10-13T12:13:14.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:14.456+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.458251+0000) 2019-10-13T12:13:14.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:14.477+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:14.480549+0000) 2019-10-13T12:13:16.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:16.085+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.088373+0000) 2019-10-13T12:13:16.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:16.085+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.089354+0000) 2019-10-13T12:13:16.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:16.089+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.089945+0000) 2019-10-13T12:13:16.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:16.089+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.090665+0000) 2019-10-13T12:13:16.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:16.089+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.091321+0000) 2019-10-13T12:13:16.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:16.084+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.092068+0000) 2019-10-13T12:13:16.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:16.089+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.092771+0000) 2019-10-13T12:13:16.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:16.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:16.095101+0000) 2019-10-13T12:13:16.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:16.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155820 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:17.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:17.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:17.852256+0000) 2019-10-13T12:13:18.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:18.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.078418+0000) 2019-10-13T12:13:18.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:18.096+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.099371+0000) 2019-10-13T12:13:18.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:18.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:18.620046+0000) 2019-10-13T12:13:21.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:21.096+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.100326+0000) 2019-10-13T12:13:21.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:21.096+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.100906+0000) 2019-10-13T12:13:21.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:21.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.101344+0000) 2019-10-13T12:13:21.103 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:21.100+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.102768+0000) 2019-10-13T12:13:21.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:21.100+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.103248+0000) 2019-10-13T12:13:21.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:21.100+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.104192+0000) 2019-10-13T12:13:21.105 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:21.100+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.104756+0000) 2019-10-13T12:13:21.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:21.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:21.110 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:21.104+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.109820+0000) 2019-10-13T12:13:21.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:21.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:21.871172+0000) 2019-10-13T12:13:24.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:24.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.195167+0000) 2019-10-13T12:13:24.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:24.464+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.464896+0000) 2019-10-13T12:13:24.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:24.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:24.480751+0000) 2019-10-13T12:13:26.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:26.108+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.111578+0000) 2019-10-13T12:13:26.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:26.108+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.112169+0000) 2019-10-13T12:13:26.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:26.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.112623+0000) 2019-10-13T12:13:26.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:26.112+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.114351+0000) 2019-10-13T12:13:26.115 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:26.112+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.114627+0000) 2019-10-13T12:13:26.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:26.112+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.115288+0000) 2019-10-13T12:13:26.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:26.112+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.115681+0000) 2019-10-13T12:13:26.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:26.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:26.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:26.116+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:26.122793+0000) 2019-10-13T12:13:27.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:27.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:27.852482+0000) 2019-10-13T12:13:28.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:28.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.078662+0000) 2019-10-13T12:13:28.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:28.096+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.099567+0000) 2019-10-13T12:13:28.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:28.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:28.620287+0000) 2019-10-13T12:13:31.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:31.120+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.124721+0000) 2019-10-13T12:13:31.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:31.120+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.125183+0000) 2019-10-13T12:13:31.126 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:31.120+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.125620+0000) 2019-10-13T12:13:31.126 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:31.124+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.126163+0000) 2019-10-13T12:13:31.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:31.124+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.127668+0000) 2019-10-13T12:13:31.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:31.124+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.128594+0000) 2019-10-13T12:13:31.129 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:31.124+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.128897+0000) 2019-10-13T12:13:31.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:31.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:31.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:31.128+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.135153+0000) 2019-10-13T12:13:31.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:31.864+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:31.871338+0000) 2019-10-13T12:13:34.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:34.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.195384+0000) 2019-10-13T12:13:34.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:34.468+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.471612+0000) 2019-10-13T12:13:34.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:34.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:34.480962+0000) 2019-10-13T12:13:35.385 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:13:35.385 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:35.389 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:13:35.389 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:13:36.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:36.128+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.133649+0000) 2019-10-13T12:13:36.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:36.132+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.134454+0000) 2019-10-13T12:13:36.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:36.132+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.134658+0000) 2019-10-13T12:13:36.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:36.132+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.135283+0000) 2019-10-13T12:13:36.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:36.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.135690+0000) 2019-10-13T12:13:36.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:36.132+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.136305+0000) 2019-10-13T12:13:36.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:36.136+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.138553+0000) 2019-10-13T12:13:36.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:36.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:36.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:36.136+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:36.142349+0000) 2019-10-13T12:13:37.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:37.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:37.852679+0000) 2019-10-13T12:13:38.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:38.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.078856+0000) 2019-10-13T12:13:38.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:38.096+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.099766+0000) 2019-10-13T12:13:38.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:38.616+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:38.620518+0000) 2019-10-13T12:13:41.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:41.140+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.144032+0000) 2019-10-13T12:13:41.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:41.140+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.144836+0000) 2019-10-13T12:13:41.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:41.140+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.145081+0000) 2019-10-13T12:13:41.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:41.140+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.145362+0000) 2019-10-13T12:13:41.146 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:41.144+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.146088+0000) 2019-10-13T12:13:41.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:41.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.146849+0000) 2019-10-13T12:13:41.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:41.144+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.147362+0000) 2019-10-13T12:13:41.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:41.148+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.150709+0000) 2019-10-13T12:13:41.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:41.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:41.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:41.868+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:41.871509+0000) 2019-10-13T12:13:44.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:44.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.195584+0000) 2019-10-13T12:13:44.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:44.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.478174+0000) 2019-10-13T12:13:44.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:44.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:44.481204+0000) 2019-10-13T12:13:46.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:46.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.155379+0000) 2019-10-13T12:13:46.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:46.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.155789+0000) 2019-10-13T12:13:46.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:46.152+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.156413+0000) 2019-10-13T12:13:46.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:46.156+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.158223+0000) 2019-10-13T12:13:46.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:46.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.160554+0000) 2019-10-13T12:13:46.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:46.156+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.160912+0000) 2019-10-13T12:13:46.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:46.159+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.161794+0000) 2019-10-13T12:13:46.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:46.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:46.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:46.163+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:46.165192+0000) 2019-10-13T12:13:47.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:47.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:47.852890+0000) 2019-10-13T12:13:48.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:48.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.079057+0000) 2019-10-13T12:13:48.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:48.096+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.099959+0000) 2019-10-13T12:13:48.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:48.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:48.620712+0000) 2019-10-13T12:13:51.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:51.163+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.166313+0000) 2019-10-13T12:13:51.168 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:51.164+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.166696+0000) 2019-10-13T12:13:51.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:51.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.167220+0000) 2019-10-13T12:13:51.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:51.164+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.167660+0000) 2019-10-13T12:13:51.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:51.164+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.168148+0000) 2019-10-13T12:13:51.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:51.164+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.168736+0000) 2019-10-13T12:13:51.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:51.164+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.169695+0000) 2019-10-13T12:13:51.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:51.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156040 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:51.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:51.175+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.176565+0000) 2019-10-13T12:13:51.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:51.867+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:51.871681+0000) 2019-10-13T12:13:54.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:54.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.195787+0000) 2019-10-13T12:13:54.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:54.476+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.481390+0000) 2019-10-13T12:13:54.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:54.483+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:54.484671+0000) 2019-10-13T12:13:56.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:13:56.172+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.177577+0000) 2019-10-13T12:13:56.180 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:56.176+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.179360+0000) 2019-10-13T12:13:56.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:13:56.179+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.181234+0000) 2019-10-13T12:13:56.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:13:56.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.181532+0000) 2019-10-13T12:13:56.182 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:56.180+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.182046+0000) 2019-10-13T12:13:56.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:56.180+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.183774+0000) 2019-10-13T12:13:56.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:56.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.184150+0000) 2019-10-13T12:13:56.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:13:56.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:13:56.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:13:56.187+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:56.191500+0000) 2019-10-13T12:13:57.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:13:57.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:57.853069+0000) 2019-10-13T12:13:58.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:13:58.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.079279+0000) 2019-10-13T12:13:58.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:13:58.096+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.100171+0000) 2019-10-13T12:13:58.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:13:58.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:13:58.620935+0000) 2019-10-13T12:14:01.190 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:01.184+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.188987+0000) 2019-10-13T12:14:01.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:01.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.190016+0000) 2019-10-13T12:14:01.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:01.188+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.190517+0000) 2019-10-13T12:14:01.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:01.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.191073+0000) 2019-10-13T12:14:01.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:01.188+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.192424+0000) 2019-10-13T12:14:01.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:01.188+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.192799+0000) 2019-10-13T12:14:01.194 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:01.191+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.193369+0000) 2019-10-13T12:14:01.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:01.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:01.201 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:01.199+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.200645+0000) 2019-10-13T12:14:01.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:01.867+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:01.871852+0000) 2019-10-13T12:14:04.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:04.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.195989+0000) 2019-10-13T12:14:04.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:04.480+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.481593+0000) 2019-10-13T12:14:04.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:04.487+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:04.491201+0000) 2019-10-13T12:14:05.517 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:14:05.517 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:05.521 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:14:05.521 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:06.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:06.196+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.201554+0000) 2019-10-13T12:14:06.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:06.199+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.202787+0000) 2019-10-13T12:14:06.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:06.200+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.203154+0000) 2019-10-13T12:14:06.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:06.200+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.204453+0000) 2019-10-13T12:14:06.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:06.200+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.204632+0000) 2019-10-13T12:14:06.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:06.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.206520+0000) 2019-10-13T12:14:06.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:06.204+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.207638+0000) 2019-10-13T12:14:06.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:06.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:06.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:06.211+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:06.215997+0000) 2019-10-13T12:14:07.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:07.848+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:07.853272+0000) 2019-10-13T12:14:08.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:08.076+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.079498+0000) 2019-10-13T12:14:08.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:08.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.100369+0000) 2019-10-13T12:14:08.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:08.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:08.621130+0000) 2019-10-13T12:14:11.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:11.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.212096+0000) 2019-10-13T12:14:11.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:11.208+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.212800+0000) 2019-10-13T12:14:11.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:11.211+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.213981+0000) 2019-10-13T12:14:11.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:11.212+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.213733+0000) 2019-10-13T12:14:11.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:11.212+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.214305+0000) 2019-10-13T12:14:11.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:11.212+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.215309+0000) 2019-10-13T12:14:11.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:11.212+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.215961+0000) 2019-10-13T12:14:11.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:11.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156163 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:11.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:11.219+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.220888+0000) 2019-10-13T12:14:11.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:11.867+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:11.872028+0000) 2019-10-13T12:14:14.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:14.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.196185+0000) 2019-10-13T12:14:14.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:14.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.481810+0000) 2019-10-13T12:14:14.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:14.495+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:14.497750+0000) 2019-10-13T12:14:16.223 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:16.219+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.222262+0000) 2019-10-13T12:14:16.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:16.219+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.224367+0000) 2019-10-13T12:14:16.225 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:16.219+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.224786+0000) 2019-10-13T12:14:16.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:16.223+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.226082+0000) 2019-10-13T12:14:16.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:16.223+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.227346+0000) 2019-10-13T12:14:16.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:16.223+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.228290+0000) 2019-10-13T12:14:16.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:16.223+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.229515+0000) 2019-10-13T12:14:16.232 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:16.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:16.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:16.231+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:16.233886+0000) 2019-10-13T12:14:17.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:17.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:17.853494+0000) 2019-10-13T12:14:18.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:18.075+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.079726+0000) 2019-10-13T12:14:18.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:18.095+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.100569+0000) 2019-10-13T12:14:18.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:18.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:18.621314+0000) 2019-10-13T12:14:21.235 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:21.231+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.234686+0000) 2019-10-13T12:14:21.236 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:21.231+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.235074+0000) 2019-10-13T12:14:21.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:21.235+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.236649+0000) 2019-10-13T12:14:21.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:21.235+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.238512+0000) 2019-10-13T12:14:21.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:21.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.240876+0000) 2019-10-13T12:14:21.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:21.235+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.241471+0000) 2019-10-13T12:14:21.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:21.239+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.241901+0000) 2019-10-13T12:14:21.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:21.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156226 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:21.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:21.243+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.247757+0000) 2019-10-13T12:14:21.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:21.871+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:21.872169+0000) 2019-10-13T12:14:24.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:24.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.196384+0000) 2019-10-13T12:14:24.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:24.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.482042+0000) 2019-10-13T12:14:24.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:24.503+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:24.504288+0000) 2019-10-13T12:14:26.246 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:26.239+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.245587+0000) 2019-10-13T12:14:26.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:26.243+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.246834+0000) 2019-10-13T12:14:26.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:26.243+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.248229+0000) 2019-10-13T12:14:26.250 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:26.247+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.249516+0000) 2019-10-13T12:14:26.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:26.247+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.249792+0000) 2019-10-13T12:14:26.251 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:26.247+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.250219+0000) 2019-10-13T12:14:26.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:26.247+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.250442+0000) 2019-10-13T12:14:26.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:26.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:26.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:26.255+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:26.256953+0000) 2019-10-13T12:14:27.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:27.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:27.853696+0000) 2019-10-13T12:14:28.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:28.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.079921+0000) 2019-10-13T12:14:28.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:28.095+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.100770+0000) 2019-10-13T12:14:28.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:28.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:28.621464+0000) 2019-10-13T12:14:31.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:31.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.258676+0000) 2019-10-13T12:14:31.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:31.255+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.259187+0000) 2019-10-13T12:14:31.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:31.255+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.259590+0000) 2019-10-13T12:14:31.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:31.255+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.260505+0000) 2019-10-13T12:14:31.261 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:31.259+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.260813+0000) 2019-10-13T12:14:31.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:31.255+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.261515+0000) 2019-10-13T12:14:31.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:31.255+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.261631+0000) 2019-10-13T12:14:31.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:31.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156287 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:31.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:31.263+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.266795+0000) 2019-10-13T12:14:31.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:31.871+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:31.872302+0000) 2019-10-13T12:14:34.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:34.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.196586+0000) 2019-10-13T12:14:34.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:34.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.482245+0000) 2019-10-13T12:14:34.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:34.507+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:34.510409+0000) 2019-10-13T12:14:35.606 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:14:35.607 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:35.611 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:14:35.611 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:14:36.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:36.263+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.269178+0000) 2019-10-13T12:14:36.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:36.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.269898+0000) 2019-10-13T12:14:36.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:36.267+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.270942+0000) 2019-10-13T12:14:36.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:36.267+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.271429+0000) 2019-10-13T12:14:36.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:36.267+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.271795+0000) 2019-10-13T12:14:36.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:36.267+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.272798+0000) 2019-10-13T12:14:36.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:36.267+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.273458+0000) 2019-10-13T12:14:36.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:36.275+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:36.277276+0000) 2019-10-13T12:14:36.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:36.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:37.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:37.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:37.853989+0000) 2019-10-13T12:14:38.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:38.075+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.080114+0000) 2019-10-13T12:14:38.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:38.095+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.100969+0000) 2019-10-13T12:14:38.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:38.615+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:38.621658+0000) 2019-10-13T12:14:41.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:41.279+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.281545+0000) 2019-10-13T12:14:41.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:41.279+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.282697+0000) 2019-10-13T12:14:41.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:41.279+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.283256+0000) 2019-10-13T12:14:41.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:41.279+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.283272+0000) 2019-10-13T12:14:41.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:41.279+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.282964+0000) 2019-10-13T12:14:41.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:41.279+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.281837+0000) 2019-10-13T12:14:41.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:41.279+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.284487+0000) 2019-10-13T12:14:41.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:41.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:41.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:41.287+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.290495+0000) 2019-10-13T12:14:41.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:41.871+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:41.872473+0000) 2019-10-13T12:14:44.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:44.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:44.196796+0000) 2019-10-13T12:14:44.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:44.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:44.482444+0000) 2019-10-13T12:14:44.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:44.515+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:44.516947+0000) 2019-10-13T12:14:46.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:46.287+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.292699+0000) 2019-10-13T12:14:46.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:46.287+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.293071+0000) 2019-10-13T12:14:46.295 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:46.291+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.294247+0000) 2019-10-13T12:14:46.295 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:46.287+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.293495+0000) 2019-10-13T12:14:46.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:46.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.294645+0000) 2019-10-13T12:14:46.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:46.291+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.294960+0000) 2019-10-13T12:14:46.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:46.291+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.295881+0000) 2019-10-13T12:14:46.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:46.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:46.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:46.299+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:46.303637+0000) 2019-10-13T12:14:47.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:47.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:47.854197+0000) 2019-10-13T12:14:48.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:48.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.080317+0000) 2019-10-13T12:14:48.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:48.099+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.101164+0000) 2019-10-13T12:14:48.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:48.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:48.621851+0000) 2019-10-13T12:14:51.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:51.299+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.303130+0000) 2019-10-13T12:14:51.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:51.299+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.304153+0000) 2019-10-13T12:14:51.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:51.299+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.304555+0000) 2019-10-13T12:14:51.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:51.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.305191+0000) 2019-10-13T12:14:51.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:51.299+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.305318+0000) 2019-10-13T12:14:51.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:51.303+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.306137+0000) 2019-10-13T12:14:51.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:51.302+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.307097+0000) 2019-10-13T12:14:51.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:51.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156411 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:51.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:51.310+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.315026+0000) 2019-10-13T12:14:51.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:51.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:51.872686+0000) 2019-10-13T12:14:54.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:54.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:54.197025+0000) 2019-10-13T12:14:54.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:54.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:54.482648+0000) 2019-10-13T12:14:54.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:54.522+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:54.523360+0000) 2019-10-13T12:14:56.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:56.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.315904+0000) 2019-10-13T12:14:56.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:56.311+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.316819+0000) 2019-10-13T12:14:56.318 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:14:56.314+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.317123+0000) 2019-10-13T12:14:56.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:56.315+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.318944+0000) 2019-10-13T12:14:56.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:14:56.315+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.319520+0000) 2019-10-13T12:14:56.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:14:56.315+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.319522+0000) 2019-10-13T12:14:56.321 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:56.315+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.319731+0000) 2019-10-13T12:14:56.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:14:56.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:14:56.327 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:14:56.322+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:56.326281+0000) 2019-10-13T12:14:57.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:14:57.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:57.854429+0000) 2019-10-13T12:14:58.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:14:58.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.080500+0000) 2019-10-13T12:14:58.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:14:58.099+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.101377+0000) 2019-10-13T12:14:58.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:14:58.619+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:14:58.622047+0000) 2019-10-13T12:15:01.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:01.323+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.326265+0000) 2019-10-13T12:15:01.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:01.323+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.326399+0000) 2019-10-13T12:15:01.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:01.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.327320+0000) 2019-10-13T12:15:01.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:01.323+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.327918+0000) 2019-10-13T12:15:01.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:01.326+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.328405+0000) 2019-10-13T12:15:01.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:01.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.328557+0000) 2019-10-13T12:15:01.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:01.323+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.328796+0000) 2019-10-13T12:15:01.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:01.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:01.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:01.334+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.338254+0000) 2019-10-13T12:15:01.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:01.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:01.872873+0000) 2019-10-13T12:15:04.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:04.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:04.197245+0000) 2019-10-13T12:15:04.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:04.479+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:04.482848+0000) 2019-10-13T12:15:04.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:04.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:04.529347+0000) 2019-10-13T12:15:05.718 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:15:05.718 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:05.722 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:15:05.722 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:06.339 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:06.334+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.338038+0000) 2019-10-13T12:15:06.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:06.331+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.337295+0000) 2019-10-13T12:15:06.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:06.335+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.338313+0000) 2019-10-13T12:15:06.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:06.335+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.339434+0000) 2019-10-13T12:15:06.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:06.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.339943+0000) 2019-10-13T12:15:06.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:06.335+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.340179+0000) 2019-10-13T12:15:06.341 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:06.335+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.341006+0000) 2019-10-13T12:15:06.344 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:06.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:06.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:06.346+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:06.348582+0000) 2019-10-13T12:15:07.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:07.851+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:07.854628+0000) 2019-10-13T12:15:08.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:08.078+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.080693+0000) 2019-10-13T12:15:08.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:08.094+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.101591+0000) 2019-10-13T12:15:08.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:08.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:08.622255+0000) 2019-10-13T12:15:11.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:11.342+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.347232+0000) 2019-10-13T12:15:11.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:11.342+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.347756+0000) 2019-10-13T12:15:11.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:11.342+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.348358+0000) 2019-10-13T12:15:11.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:11.342+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.348734+0000) 2019-10-13T12:15:11.351 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:11.346+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.350786+0000) 2019-10-13T12:15:11.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:11.346+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.352295+0000) 2019-10-13T12:15:11.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:11.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:11.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:11.350+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.355087+0000) 2019-10-13T12:15:11.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:11.358+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.362255+0000) 2019-10-13T12:15:11.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:11.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:11.873059+0000) 2019-10-13T12:15:14.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:14.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:14.197456+0000) 2019-10-13T12:15:14.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:14.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:14.483052+0000) 2019-10-13T12:15:14.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:14.534+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:14.535741+0000) 2019-10-13T12:15:16.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:16.350+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.357498+0000) 2019-10-13T12:15:16.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:16.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.358138+0000) 2019-10-13T12:15:16.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:16.354+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.358384+0000) 2019-10-13T12:15:16.360 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:16.354+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.359615+0000) 2019-10-13T12:15:16.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:16.358+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.360296+0000) 2019-10-13T12:15:16.361 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:16.354+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.360498+0000) 2019-10-13T12:15:16.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:16.354+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.361035+0000) 2019-10-13T12:15:16.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:16.362+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:16.365579+0000) 2019-10-13T12:15:16.366 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:16.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:17.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:17.850+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:17.854830+0000) 2019-10-13T12:15:18.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:18.074+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.080913+0000) 2019-10-13T12:15:18.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:18.098+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.101813+0000) 2019-10-13T12:15:18.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:18.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:18.622455+0000) 2019-10-13T12:15:21.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:21.362+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.369661+0000) 2019-10-13T12:15:21.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:21.366+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.370124+0000) 2019-10-13T12:15:21.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:21.366+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.370493+0000) 2019-10-13T12:15:21.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:21.366+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.371341+0000) 2019-10-13T12:15:21.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:21.366+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.372014+0000) 2019-10-13T12:15:21.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:21.366+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.371840+0000) 2019-10-13T12:15:21.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:21.366+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.372234+0000) 2019-10-13T12:15:21.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:21.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:21.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:21.374+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.378215+0000) 2019-10-13T12:15:21.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:21.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:21.873225+0000) 2019-10-13T12:15:24.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:24.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:24.197658+0000) 2019-10-13T12:15:24.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:24.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:24.483258+0000) 2019-10-13T12:15:24.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:24.538+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:24.542165+0000) 2019-10-13T12:15:26.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:26.374+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.380032+0000) 2019-10-13T12:15:26.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:26.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.381183+0000) 2019-10-13T12:15:26.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:26.374+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.381403+0000) 2019-10-13T12:15:26.382 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:26.378+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.382055+0000) 2019-10-13T12:15:26.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:26.378+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.382696+0000) 2019-10-13T12:15:26.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:26.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.382753+0000) 2019-10-13T12:15:26.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:26.378+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.384008+0000) 2019-10-13T12:15:26.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:26.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156628 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:26.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:26.386+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:26.389045+0000) 2019-10-13T12:15:27.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:27.854+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:27.855056+0000) 2019-10-13T12:15:28.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:28.074+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.081118+0000) 2019-10-13T12:15:28.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:28.098+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.102023+0000) 2019-10-13T12:15:28.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:28.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:28.622673+0000) 2019-10-13T12:15:31.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:31.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.392714+0000) 2019-10-13T12:15:31.394 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:31.390+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.393866+0000) 2019-10-13T12:15:31.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:31.390+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.394429+0000) 2019-10-13T12:15:31.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:31.390+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.395117+0000) 2019-10-13T12:15:31.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:31.394+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.397327+0000) 2019-10-13T12:15:31.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:31.390+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.397561+0000) 2019-10-13T12:15:31.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:31.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:31.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:31.394+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.400211+0000) 2019-10-13T12:15:31.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:31.398+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.401892+0000) 2019-10-13T12:15:31.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:31.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:31.873390+0000) 2019-10-13T12:15:34.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:34.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:34.197859+0000) 2019-10-13T12:15:34.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:34.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:34.483462+0000) 2019-10-13T12:15:34.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:34.546+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:34.548773+0000) 2019-10-13T12:15:35.879 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:15:35.879 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:35.882 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:15:35.883 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:15:36.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:36.398+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.402431+0000) 2019-10-13T12:15:36.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:36.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.403120+0000) 2019-10-13T12:15:36.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:36.398+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.404220+0000) 2019-10-13T12:15:36.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:36.398+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.404722+0000) 2019-10-13T12:15:36.405 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:36.402+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.405182+0000) 2019-10-13T12:15:36.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:36.398+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.405508+0000) 2019-10-13T12:15:36.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:36.402+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.406290+0000) 2019-10-13T12:15:36.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:36.406+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:36.409826+0000) 2019-10-13T12:15:36.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:36.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:37.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:37.854+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:37.855256+0000) 2019-10-13T12:15:38.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:38.078+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.081331+0000) 2019-10-13T12:15:38.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:38.098+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.102258+0000) 2019-10-13T12:15:38.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:38.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:38.622884+0000) 2019-10-13T12:15:41.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:41.410+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.414545+0000) 2019-10-13T12:15:41.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:41.410+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.415029+0000) 2019-10-13T12:15:41.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:41.410+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.416026+0000) 2019-10-13T12:15:41.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:41.410+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.415744+0000) 2019-10-13T12:15:41.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:41.410+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.416550+0000) 2019-10-13T12:15:41.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:41.414+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.417825+0000) 2019-10-13T12:15:41.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:41.418+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.422315+0000) 2019-10-13T12:15:41.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:41.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156722 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:41.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:41.422+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.425839+0000) 2019-10-13T12:15:41.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:41.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:41.873563+0000) 2019-10-13T12:15:44.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:44.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:44.198014+0000) 2019-10-13T12:15:44.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:44.478+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:44.483657+0000) 2019-10-13T12:15:44.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:44.554+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:44.555264+0000) 2019-10-13T12:15:46.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:46.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.427176+0000) 2019-10-13T12:15:46.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:46.422+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.427471+0000) 2019-10-13T12:15:46.429 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:46.422+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.428628+0000) 2019-10-13T12:15:46.429 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:46.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.429125+0000) 2019-10-13T12:15:46.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:46.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.429508+0000) 2019-10-13T12:15:46.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:46.426+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.429885+0000) 2019-10-13T12:15:46.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:46.426+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.430384+0000) 2019-10-13T12:15:46.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:46.430+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:46.433694+0000) 2019-10-13T12:15:46.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:46.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:47.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:47.850+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:47.855451+0000) 2019-10-13T12:15:48.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:48.074+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.081533+0000) 2019-10-13T12:15:48.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:48.102+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.102458+0000) 2019-10-13T12:15:48.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:48.618+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:48.623077+0000) 2019-10-13T12:15:51.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:51.434+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.437699+0000) 2019-10-13T12:15:51.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:51.434+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.438460+0000) 2019-10-13T12:15:51.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:51.434+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.439198+0000) 2019-10-13T12:15:51.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:51.434+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.439270+0000) 2019-10-13T12:15:51.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:51.434+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.438746+0000) 2019-10-13T12:15:51.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:51.434+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.439905+0000) 2019-10-13T12:15:51.441 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:51.434+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.440240+0000) 2019-10-13T12:15:51.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:51.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:51.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:51.442+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.446844+0000) 2019-10-13T12:15:51.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:51.870+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:51.873740+0000) 2019-10-13T12:15:54.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:54.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:54.198208+0000) 2019-10-13T12:15:54.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:54.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:54.483849+0000) 2019-10-13T12:15:54.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:54.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:54.561908+0000) 2019-10-13T12:15:56.450 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:56.442+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.448994+0000) 2019-10-13T12:15:56.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:15:56.446+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.450514+0000) 2019-10-13T12:15:56.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:15:56.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.451526+0000) 2019-10-13T12:15:56.453 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:56.446+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.451543+0000) 2019-10-13T12:15:56.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:56.446+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.451865+0000) 2019-10-13T12:15:56.453 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:15:56.446+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.451893+0000) 2019-10-13T12:15:56.453 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:56.446+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.452117+0000) 2019-10-13T12:15:56.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:15:56.453+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:56.457149+0000) 2019-10-13T12:15:56.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:15:56.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:15:57.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:15:57.854+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:57.855650+0000) 2019-10-13T12:15:58.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:15:58.078+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.081780+0000) 2019-10-13T12:15:58.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:15:58.102+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.102608+0000) 2019-10-13T12:15:58.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:15:58.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:15:58.623267+0000) 2019-10-13T12:16:01.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:01.454+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.460941+0000) 2019-10-13T12:16:01.462 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:01.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.461705+0000) 2019-10-13T12:16:01.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:01.458+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.462118+0000) 2019-10-13T12:16:01.463 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:01.457+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.462570+0000) 2019-10-13T12:16:01.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:01.458+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.462834+0000) 2019-10-13T12:16:01.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:01.458+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.463337+0000) 2019-10-13T12:16:01.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:01.458+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.463424+0000) 2019-10-13T12:16:01.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:01.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:01.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:01.465+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.470838+0000) 2019-10-13T12:16:01.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:01.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:01.873944+0000) 2019-10-13T12:16:04.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:04.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:04.198418+0000) 2019-10-13T12:16:04.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:04.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:04.484047+0000) 2019-10-13T12:16:04.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:04.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:04.568345+0000) 2019-10-13T12:16:06.087 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:16:06.087 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:06.091 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:16:06.091 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:06.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:06.469+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.472432+0000) 2019-10-13T12:16:06.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:06.465+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.472709+0000) 2019-10-13T12:16:06.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:06.465+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.473544+0000) 2019-10-13T12:16:06.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:06.469+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.473797+0000) 2019-10-13T12:16:06.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:06.469+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.474465+0000) 2019-10-13T12:16:06.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:06.469+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.474930+0000) 2019-10-13T12:16:06.476 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:06.469+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.475379+0000) 2019-10-13T12:16:06.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:06.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:06.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:06.477+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:06.480842+0000) 2019-10-13T12:16:07.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:07.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:07.855868+0000) 2019-10-13T12:16:08.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:08.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.081974+0000) 2019-10-13T12:16:08.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:08.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.102808+0000) 2019-10-13T12:16:08.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:08.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:08.623462+0000) 2019-10-13T12:16:11.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:11.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.483368+0000) 2019-10-13T12:16:11.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:11.477+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.484050+0000) 2019-10-13T12:16:11.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:11.477+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.484392+0000) 2019-10-13T12:16:11.486 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:11.481+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.484894+0000) 2019-10-13T12:16:11.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:11.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.485400+0000) 2019-10-13T12:16:11.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:11.481+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.485736+0000) 2019-10-13T12:16:11.487 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:11.481+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.486149+0000) 2019-10-13T12:16:11.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:11.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156907 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:11.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:11.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.495116+0000) 2019-10-13T12:16:11.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:11.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:11.874079+0000) 2019-10-13T12:16:14.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:14.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:14.198674+0000) 2019-10-13T12:16:14.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:14.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:14.484246+0000) 2019-10-13T12:16:14.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:14.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:14.574872+0000) 2019-10-13T12:16:16.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:16.489+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.495422+0000) 2019-10-13T12:16:16.497 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:16.489+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.495944+0000) 2019-10-13T12:16:16.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:16.489+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.496445+0000) 2019-10-13T12:16:16.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:16.489+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.496944+0000) 2019-10-13T12:16:16.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:16.489+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.497085+0000) 2019-10-13T12:16:16.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:16.493+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.497778+0000) 2019-10-13T12:16:16.499 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:16.493+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.497984+0000) 2019-10-13T12:16:16.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:16.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:16.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:16.501+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:16.504667+0000) 2019-10-13T12:16:17.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:17.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:17.856094+0000) 2019-10-13T12:16:18.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:18.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.082165+0000) 2019-10-13T12:16:18.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:18.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.103002+0000) 2019-10-13T12:16:18.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:18.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:18.623663+0000) 2019-10-13T12:16:21.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:21.501+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.505667+0000) 2019-10-13T12:16:21.507 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:21.505+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.506761+0000) 2019-10-13T12:16:21.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:21.501+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.507064+0000) 2019-10-13T12:16:21.508 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:21.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.507232+0000) 2019-10-13T12:16:21.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:21.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.508535+0000) 2019-10-13T12:16:21.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:21.505+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.508850+0000) 2019-10-13T12:16:21.510 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:21.505+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.509205+0000) 2019-10-13T12:16:21.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:21.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:21.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:21.513+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.516739+0000) 2019-10-13T12:16:21.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:21.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:21.874232+0000) 2019-10-13T12:16:24.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:24.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:24.198881+0000) 2019-10-13T12:16:24.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:24.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:24.484443+0000) 2019-10-13T12:16:24.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:24.577+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:24.581287+0000) 2019-10-13T12:16:26.518 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:26.513+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.517461+0000) 2019-10-13T12:16:26.520 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:26.513+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.519100+0000) 2019-10-13T12:16:26.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:26.517+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.518813+0000) 2019-10-13T12:16:26.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:26.517+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.519490+0000) 2019-10-13T12:16:26.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:26.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.519965+0000) 2019-10-13T12:16:26.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:26.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.523933+0000) 2019-10-13T12:16:26.526 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:26.521+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.525358+0000) 2019-10-13T12:16:26.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:26.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:26.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:26.525+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:26.528520+0000) 2019-10-13T12:16:27.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:27.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:27.856322+0000) 2019-10-13T12:16:28.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:28.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.082362+0000) 2019-10-13T12:16:28.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:28.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.103230+0000) 2019-10-13T12:16:28.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:28.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:28.623884+0000) 2019-10-13T12:16:31.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:31.525+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.529318+0000) 2019-10-13T12:16:31.533 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:31.529+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.532371+0000) 2019-10-13T12:16:31.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:31.529+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.532526+0000) 2019-10-13T12:16:31.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:31.529+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.532991+0000) 2019-10-13T12:16:31.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:31.533+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.535809+0000) 2019-10-13T12:16:31.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:31.533+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.538053+0000) 2019-10-13T12:16:31.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:31.537+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:31.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:31.537+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.538677+0000) 2019-10-13T12:16:31.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:31.541+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.545779+0000) 2019-10-13T12:16:31.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:31.869+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:31.874401+0000) 2019-10-13T12:16:34.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:34.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:34.199114+0000) 2019-10-13T12:16:34.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:34.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:34.484685+0000) 2019-10-13T12:16:34.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:34.585+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:34.587840+0000) 2019-10-13T12:16:36.209 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:16:36.209 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:36.213 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:16:36.213 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:16:36.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:36.537+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.541566+0000) 2019-10-13T12:16:36.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:36.537+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.543074+0000) 2019-10-13T12:16:36.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:36.541+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.543518+0000) 2019-10-13T12:16:36.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:36.541+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.543885+0000) 2019-10-13T12:16:36.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:36.541+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.544444+0000) 2019-10-13T12:16:36.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:36.541+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.544842+0000) 2019-10-13T12:16:36.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:36.541+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.545379+0000) 2019-10-13T12:16:36.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:36.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:36.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:36.545+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:36.551837+0000) 2019-10-13T12:16:37.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:37.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:37.856505+0000) 2019-10-13T12:16:38.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:38.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.082513+0000) 2019-10-13T12:16:38.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:38.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.103427+0000) 2019-10-13T12:16:38.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:38.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:38.624093+0000) 2019-10-13T12:16:41.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:41.549+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.552434+0000) 2019-10-13T12:16:41.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:41.553+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.555375+0000) 2019-10-13T12:16:41.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:41.553+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.556863+0000) 2019-10-13T12:16:41.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:41.553+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.558224+0000) 2019-10-13T12:16:41.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:41.557+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.559151+0000) 2019-10-13T12:16:41.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:41.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:41.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:41.557+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.561187+0000) 2019-10-13T12:16:41.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:41.557+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.561412+0000) 2019-10-13T12:16:41.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:41.557+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.561639+0000) 2019-10-13T12:16:41.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:41.873+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:41.874591+0000) 2019-10-13T12:16:44.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:44.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:44.199271+0000) 2019-10-13T12:16:44.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:44.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:44.484918+0000) 2019-10-13T12:16:44.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:44.593+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:44.594327+0000) 2019-10-13T12:16:46.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:46.561+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.563974+0000) 2019-10-13T12:16:46.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:46.561+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.564232+0000) 2019-10-13T12:16:46.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:46.561+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.564792+0000) 2019-10-13T12:16:46.566 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:46.561+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.564529+0000) 2019-10-13T12:16:46.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:46.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.565853+0000) 2019-10-13T12:16:46.567 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:46.565+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.566723+0000) 2019-10-13T12:16:46.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:46.565+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.567034+0000) 2019-10-13T12:16:46.572 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:46.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:46.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:46.569+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:46.572602+0000) 2019-10-13T12:16:47.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:47.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:47.856714+0000) 2019-10-13T12:16:48.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:48.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.082713+0000) 2019-10-13T12:16:48.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:48.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.103650+0000) 2019-10-13T12:16:48.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:48.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:48.624300+0000) 2019-10-13T12:16:51.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:51.573+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.575094+0000) 2019-10-13T12:16:51.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:51.569+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.575995+0000) 2019-10-13T12:16:51.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:51.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.575586+0000) 2019-10-13T12:16:51.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:51.573+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.576501+0000) 2019-10-13T12:16:51.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:51.573+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.576476+0000) 2019-10-13T12:16:51.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:51.573+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.577657+0000) 2019-10-13T12:16:51.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:51.577+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.578241+0000) 2019-10-13T12:16:51.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:51.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:51.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:51.577+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.583518+0000) 2019-10-13T12:16:51.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:51.873+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:51.874775+0000) 2019-10-13T12:16:54.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:54.197+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:54.199477+0000) 2019-10-13T12:16:54.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:54.481+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:54.485143+0000) 2019-10-13T12:16:54.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:54.597+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:54.600921+0000) 2019-10-13T12:16:56.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:16:56.585+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.586728+0000) 2019-10-13T12:16:56.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:56.585+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.586810+0000) 2019-10-13T12:16:56.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:56.585+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.587175+0000) 2019-10-13T12:16:56.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:16:56.581+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.587563+0000) 2019-10-13T12:16:56.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:16:56.585+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.588145+0000) 2019-10-13T12:16:56.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:56.585+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.588702+0000) 2019-10-13T12:16:56.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:56.585+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.588981+0000) 2019-10-13T12:16:56.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:16:56.589+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:56.592413+0000) 2019-10-13T12:16:56.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:16:56.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:16:57.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:16:57.853+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:57.856941+0000) 2019-10-13T12:16:58.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:16:58.081+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.082918+0000) 2019-10-13T12:16:58.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:16:58.101+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.103846+0000) 2019-10-13T12:16:58.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:16:58.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:16:58.624498+0000) 2019-10-13T12:17:01.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:01.592+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.596138+0000) 2019-10-13T12:17:01.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:01.592+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.597400+0000) 2019-10-13T12:17:01.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:01.596+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.599736+0000) 2019-10-13T12:17:01.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:01.596+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.600157+0000) 2019-10-13T12:17:01.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:01.596+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.600728+0000) 2019-10-13T12:17:01.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:01.596+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.600981+0000) 2019-10-13T12:17:01.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:01.600+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.601944+0000) 2019-10-13T12:17:01.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:01.596+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.603399+0000) 2019-10-13T12:17:01.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:01.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:01.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:01.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:01.874945+0000) 2019-10-13T12:17:04.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:04.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:04.199713+0000) 2019-10-13T12:17:04.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:04.480+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:04.485370+0000) 2019-10-13T12:17:04.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:04.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:04.607468+0000) 2019-10-13T12:17:06.322 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:17:06.323 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:06.326 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:17:06.326 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:06.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:06.600+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.607864+0000) 2019-10-13T12:17:06.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:06.604+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.607970+0000) 2019-10-13T12:17:06.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:06.604+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.609573+0000) 2019-10-13T12:17:06.611 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:06.608+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.610256+0000) 2019-10-13T12:17:06.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:06.608+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.610983+0000) 2019-10-13T12:17:06.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:06.608+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.611417+0000) 2019-10-13T12:17:06.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:06.612+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.615135+0000) 2019-10-13T12:17:06.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:06.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:06.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:06.612+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:06.619075+0000) 2019-10-13T12:17:07.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:07.852+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:07.857159+0000) 2019-10-13T12:17:08.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:08.080+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.083134+0000) 2019-10-13T12:17:08.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:08.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.104043+0000) 2019-10-13T12:17:08.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:08.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:08.624689+0000) 2019-10-13T12:17:11.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:11.612+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.619126+0000) 2019-10-13T12:17:11.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:11.616+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.619280+0000) 2019-10-13T12:17:11.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:11.616+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.619611+0000) 2019-10-13T12:17:11.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:11.616+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.619973+0000) 2019-10-13T12:17:11.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:11.616+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.620834+0000) 2019-10-13T12:17:11.622 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:11.616+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.621204+0000) 2019-10-13T12:17:11.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:11.616+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.621506+0000) 2019-10-13T12:17:11.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:11.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:11.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:11.624+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.628786+0000) 2019-10-13T12:17:11.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:11.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:11.875157+0000) 2019-10-13T12:17:14.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:14.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:14.199950+0000) 2019-10-13T12:17:14.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:14.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:14.485566+0000) 2019-10-13T12:17:14.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:14.612+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:14.613881+0000) 2019-10-13T12:17:16.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:16.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.630132+0000) 2019-10-13T12:17:16.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:16.628+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.631009+0000) 2019-10-13T12:17:16.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:16.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.631350+0000) 2019-10-13T12:17:16.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:16.628+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.631936+0000) 2019-10-13T12:17:16.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:16.628+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.632330+0000) 2019-10-13T12:17:16.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:16.628+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.632460+0000) 2019-10-13T12:17:16.635 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:16.632+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.634322+0000) 2019-10-13T12:17:16.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:16.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:16.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:16.632+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:16.638473+0000) 2019-10-13T12:17:17.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:17.856+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:17.857356+0000) 2019-10-13T12:17:18.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:18.080+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.083354+0000) 2019-10-13T12:17:18.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:18.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.104238+0000) 2019-10-13T12:17:18.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:18.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:18.624887+0000) 2019-10-13T12:17:21.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:21.636+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.640793+0000) 2019-10-13T12:17:21.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:21.640+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.641897+0000) 2019-10-13T12:17:21.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:21.640+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.643390+0000) 2019-10-13T12:17:21.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:21.640+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.643695+0000) 2019-10-13T12:17:21.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:21.640+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.645176+0000) 2019-10-13T12:17:21.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:21.640+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.645497+0000) 2019-10-13T12:17:21.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:21.644+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.646886+0000) 2019-10-13T12:17:21.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:21.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:21.655 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:21.648+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.654977+0000) 2019-10-13T12:17:21.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:21.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:21.875357+0000) 2019-10-13T12:17:24.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:24.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:24.200151+0000) 2019-10-13T12:17:24.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:24.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:24.485782+0000) 2019-10-13T12:17:24.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:24.624+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:24.625007+0000) 2019-10-13T12:17:26.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:26.652+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.654865+0000) 2019-10-13T12:17:26.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:26.652+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.655202+0000) 2019-10-13T12:17:26.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:26.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.655986+0000) 2019-10-13T12:17:26.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:26.652+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.656205+0000) 2019-10-13T12:17:26.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:26.652+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.657569+0000) 2019-10-13T12:17:26.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:26.656+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.658102+0000) 2019-10-13T12:17:26.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:26.656+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.659126+0000) 2019-10-13T12:17:26.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:26.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:26.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:26.656+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:26.663689+0000) 2019-10-13T12:17:27.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:27.856+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:27.857570+0000) 2019-10-13T12:17:28.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:28.080+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.083547+0000) 2019-10-13T12:17:28.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:28.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.104396+0000) 2019-10-13T12:17:28.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:28.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:28.625115+0000) 2019-10-13T12:17:31.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:31.664+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.667049+0000) 2019-10-13T12:17:31.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:31.664+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.667658+0000) 2019-10-13T12:17:31.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:31.664+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.668079+0000) 2019-10-13T12:17:31.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:31.664+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.668256+0000) 2019-10-13T12:17:31.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:31.664+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.668674+0000) 2019-10-13T12:17:31.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:31.664+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.669097+0000) 2019-10-13T12:17:31.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:31.664+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.669276+0000) 2019-10-13T12:17:31.671 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:31.668+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.669985+0000) 2019-10-13T12:17:31.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:31.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:31.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:31.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:31.875523+0000) 2019-10-13T12:17:34.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:34.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:34.200387+0000) 2019-10-13T12:17:34.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:34.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:34.485985+0000) 2019-10-13T12:17:34.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:34.628+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:34.631657+0000) 2019-10-13T12:17:36.496 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:17:36.496 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:36.500 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:17:36.500 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:17:36.678 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:36.672+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.676958+0000) 2019-10-13T12:17:36.680 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:36.676+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.679287+0000) 2019-10-13T12:17:36.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:36.676+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.681682+0000) 2019-10-13T12:17:36.685 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:36.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:36.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:36.680+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.685487+0000) 2019-10-13T12:17:36.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:36.680+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.685467+0000) 2019-10-13T12:17:36.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:36.684+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.685807+0000) 2019-10-13T12:17:36.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:36.680+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.685567+0000) 2019-10-13T12:17:36.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:36.684+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:36.690880+0000) 2019-10-13T12:17:37.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:37.856+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:37.857778+0000) 2019-10-13T12:17:38.084 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:38.080+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.083749+0000) 2019-10-13T12:17:38.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:38.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.104618+0000) 2019-10-13T12:17:38.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:38.620+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:38.625309+0000) 2019-10-13T12:17:41.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:41.684+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.688211+0000) 2019-10-13T12:17:41.689 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:41.684+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.688915+0000) 2019-10-13T12:17:41.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:41.684+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.689568+0000) 2019-10-13T12:17:41.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:41.684+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.690248+0000) 2019-10-13T12:17:41.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:41.688+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.690258+0000) 2019-10-13T12:17:41.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:41.688+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.690609+0000) 2019-10-13T12:17:41.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:41.684+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.691193+0000) 2019-10-13T12:17:41.692 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:41.688+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.691782+0000) 2019-10-13T12:17:41.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:41.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157465 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:41.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:41.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:41.875694+0000) 2019-10-13T12:17:44.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:44.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:44.200604+0000) 2019-10-13T12:17:44.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:44.484+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:44.486204+0000) 2019-10-13T12:17:44.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:44.632+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:44.638208+0000) 2019-10-13T12:17:46.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:46.696+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.700193+0000) 2019-10-13T12:17:46.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:46.696+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.700509+0000) 2019-10-13T12:17:46.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:46.700+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.702323+0000) 2019-10-13T12:17:46.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:46.696+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.702711+0000) 2019-10-13T12:17:46.704 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:46.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.703354+0000) 2019-10-13T12:17:46.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:46.700+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.704362+0000) 2019-10-13T12:17:46.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:46.700+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.705338+0000) 2019-10-13T12:17:46.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:46.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:46.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:46.708+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:46.713516+0000) 2019-10-13T12:17:47.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:47.856+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:47.858012+0000) 2019-10-13T12:17:48.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:48.080+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.084004+0000) 2019-10-13T12:17:48.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:48.100+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.104814+0000) 2019-10-13T12:17:48.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:48.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:48.625506+0000) 2019-10-13T12:17:51.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:51.708+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.710184+0000) 2019-10-13T12:17:51.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:51.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.711006+0000) 2019-10-13T12:17:51.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:51.704+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.712044+0000) 2019-10-13T12:17:51.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:51.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.711470+0000) 2019-10-13T12:17:51.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:51.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.712124+0000) 2019-10-13T12:17:51.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:51.708+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.712492+0000) 2019-10-13T12:17:51.714 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:51.708+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.713184+0000) 2019-10-13T12:17:51.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:51.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:51.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:51.716+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.722103+0000) 2019-10-13T12:17:51.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:51.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:51.875868+0000) 2019-10-13T12:17:54.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:54.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:54.200798+0000) 2019-10-13T12:17:54.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:54.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:54.486432+0000) 2019-10-13T12:17:54.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:54.644+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:54.644732+0000) 2019-10-13T12:17:56.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:56.715+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.721291+0000) 2019-10-13T12:17:56.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:56.719+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.721864+0000) 2019-10-13T12:17:56.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:56.719+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.722936+0000) 2019-10-13T12:17:56.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:56.719+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.723317+0000) 2019-10-13T12:17:56.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:17:56.719+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.723705+0000) 2019-10-13T12:17:56.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:17:56.724+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.725001+0000) 2019-10-13T12:17:56.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:17:56.723+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.726434+0000) 2019-10-13T12:17:56.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:17:56.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:17:56.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:17:56.728+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:56.733326+0000) 2019-10-13T12:17:57.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:17:57.855+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:57.858215+0000) 2019-10-13T12:17:58.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:17:58.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.084229+0000) 2019-10-13T12:17:58.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:17:58.099+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.105014+0000) 2019-10-13T12:17:58.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:17:58.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:17:58.625770+0000) 2019-10-13T12:18:01.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:01.727+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.730661+0000) 2019-10-13T12:18:01.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:01.727+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.731344+0000) 2019-10-13T12:18:01.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:01.727+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.731800+0000) 2019-10-13T12:18:01.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:01.727+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.732591+0000) 2019-10-13T12:18:01.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:01.727+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.732944+0000) 2019-10-13T12:18:01.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:01.732+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.734357+0000) 2019-10-13T12:18:01.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:01.731+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.735769+0000) 2019-10-13T12:18:01.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:01.735+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157589 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:01.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:01.740+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.741943+0000) 2019-10-13T12:18:01.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:01.872+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:01.876059+0000) 2019-10-13T12:18:04.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:04.199+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:04.201033+0000) 2019-10-13T12:18:04.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:04.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:04.486649+0000) 2019-10-13T12:18:04.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:04.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:04.650823+0000) 2019-10-13T12:18:06.542 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:18:06.542 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:06.547 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:18:06.547 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:06.744 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:06.739+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.742772+0000) 2019-10-13T12:18:06.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:06.739+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.743139+0000) 2019-10-13T12:18:06.745 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:06.739+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.743602+0000) 2019-10-13T12:18:06.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:06.739+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.743774+0000) 2019-10-13T12:18:06.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:06.739+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.744064+0000) 2019-10-13T12:18:06.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:06.743+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.744530+0000) 2019-10-13T12:18:06.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:06.739+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.745170+0000) 2019-10-13T12:18:06.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:06.747+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:06.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:06.747+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:06.752044+0000) 2019-10-13T12:18:07.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:07.855+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:07.858413+0000) 2019-10-13T12:18:08.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:08.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.084442+0000) 2019-10-13T12:18:08.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:08.103+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.105228+0000) 2019-10-13T12:18:08.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:08.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:08.625967+0000) 2019-10-13T12:18:11.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:11.747+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.753064+0000) 2019-10-13T12:18:11.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:11.747+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.753562+0000) 2019-10-13T12:18:11.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:11.751+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.754033+0000) 2019-10-13T12:18:11.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:11.751+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.754450+0000) 2019-10-13T12:18:11.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:11.751+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.755157+0000) 2019-10-13T12:18:11.756 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:11.751+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.755617+0000) 2019-10-13T12:18:11.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:11.751+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.756084+0000) 2019-10-13T12:18:11.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:11.755+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:11.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:11.759+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.763173+0000) 2019-10-13T12:18:11.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:11.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:11.876269+0000) 2019-10-13T12:18:14.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:14.199+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:14.201269+0000) 2019-10-13T12:18:14.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:14.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:14.486892+0000) 2019-10-13T12:18:14.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:14.655+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:14.656987+0000) 2019-10-13T12:18:16.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:16.763+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.764413+0000) 2019-10-13T12:18:16.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:16.759+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.764993+0000) 2019-10-13T12:18:16.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:16.759+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.765423+0000) 2019-10-13T12:18:16.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:16.763+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.766142+0000) 2019-10-13T12:18:16.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:16.763+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.766719+0000) 2019-10-13T12:18:16.769 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:16.763+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.768686+0000) 2019-10-13T12:18:16.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:16.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:16.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:16.767+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.772250+0000) 2019-10-13T12:18:16.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:16.775+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:16.777420+0000) 2019-10-13T12:18:17.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:17.855+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:17.858624+0000) 2019-10-13T12:18:18.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:18.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.084635+0000) 2019-10-13T12:18:18.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:18.099+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.105441+0000) 2019-10-13T12:18:18.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:18.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:18.626166+0000) 2019-10-13T12:18:21.777 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:21.771+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.776022+0000) 2019-10-13T12:18:21.777 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:21.771+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.776498+0000) 2019-10-13T12:18:21.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:21.771+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.776808+0000) 2019-10-13T12:18:21.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:21.771+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.777706+0000) 2019-10-13T12:18:21.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:21.775+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.777983+0000) 2019-10-13T12:18:21.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:21.775+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.778221+0000) 2019-10-13T12:18:21.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:21.775+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.778734+0000) 2019-10-13T12:18:21.783 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:21.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:21.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:21.783+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.786692+0000) 2019-10-13T12:18:21.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:21.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:21.876441+0000) 2019-10-13T12:18:24.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:24.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:24.201498+0000) 2019-10-13T12:18:24.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:24.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:24.487096+0000) 2019-10-13T12:18:24.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:24.663+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:24.663717+0000) 2019-10-13T12:18:26.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:26.783+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.785785+0000) 2019-10-13T12:18:26.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:26.783+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.786310+0000) 2019-10-13T12:18:26.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:26.783+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.786613+0000) 2019-10-13T12:18:26.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:26.783+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.787955+0000) 2019-10-13T12:18:26.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:26.783+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.788635+0000) 2019-10-13T12:18:26.791 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:26.783+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.789672+0000) 2019-10-13T12:18:26.791 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:26.787+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.789939+0000) 2019-10-13T12:18:26.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:26.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:26.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:26.791+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:26.794467+0000) 2019-10-13T12:18:27.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:27.855+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:27.858853+0000) 2019-10-13T12:18:28.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:28.083+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.084846+0000) 2019-10-13T12:18:28.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:28.103+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.105655+0000) 2019-10-13T12:18:28.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:28.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:28.626355+0000) 2019-10-13T12:18:31.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:31.791+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.796381+0000) 2019-10-13T12:18:31.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:31.791+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.797169+0000) 2019-10-13T12:18:31.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:31.791+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.797573+0000) 2019-10-13T12:18:31.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:31.795+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.798001+0000) 2019-10-13T12:18:31.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:31.795+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.798224+0000) 2019-10-13T12:18:31.800 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:31.795+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.798857+0000) 2019-10-13T12:18:31.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:31.795+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.799177+0000) 2019-10-13T12:18:31.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:31.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:31.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:31.803+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.804754+0000) 2019-10-13T12:18:31.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:31.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:31.876590+0000) 2019-10-13T12:18:34.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:34.199+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:34.201656+0000) 2019-10-13T12:18:34.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:34.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:34.487298+0000) 2019-10-13T12:18:34.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:34.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:34.670279+0000) 2019-10-13T12:18:36.664 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:18:36.664 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:36.668 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:18:36.668 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:18:36.808 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:36.803+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.807529+0000) 2019-10-13T12:18:36.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:36.803+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.808357+0000) 2019-10-13T12:18:36.811 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:36.807+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.810359+0000) 2019-10-13T12:18:36.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:36.807+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.810608+0000) 2019-10-13T12:18:36.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:36.807+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.811762+0000) 2019-10-13T12:18:36.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:36.811+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.812588+0000) 2019-10-13T12:18:36.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:36.807+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.812785+0000) 2019-10-13T12:18:36.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:36.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:36.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:36.819+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:36.820453+0000) 2019-10-13T12:18:37.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:37.855+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:37.859056+0000) 2019-10-13T12:18:38.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:38.083+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.085045+0000) 2019-10-13T12:18:38.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:38.103+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.105927+0000) 2019-10-13T12:18:38.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:38.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:38.626552+0000) 2019-10-13T12:18:41.819 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:41.815+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.818682+0000) 2019-10-13T12:18:41.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:41.815+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.819750+0000) 2019-10-13T12:18:41.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:41.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.820279+0000) 2019-10-13T12:18:41.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:41.819+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.820750+0000) 2019-10-13T12:18:41.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:41.815+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.821042+0000) 2019-10-13T12:18:41.823 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:41.815+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.821369+0000) 2019-10-13T12:18:41.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:41.819+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.822101+0000) 2019-10-13T12:18:41.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:41.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:41.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:41.827+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.829042+0000) 2019-10-13T12:18:41.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:41.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:41.876773+0000) 2019-10-13T12:18:44.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:44.199+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:44.201910+0000) 2019-10-13T12:18:44.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:44.483+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:44.487508+0000) 2019-10-13T12:18:44.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:44.675+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:44.676761+0000) 2019-10-13T12:18:46.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:46.827+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.829822+0000) 2019-10-13T12:18:46.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:46.827+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.831061+0000) 2019-10-13T12:18:46.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:46.827+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.831085+0000) 2019-10-13T12:18:46.832 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:46.827+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.831326+0000) 2019-10-13T12:18:46.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:46.827+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.831863+0000) 2019-10-13T12:18:46.834 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:46.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.833161+0000) 2019-10-13T12:18:46.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:46.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.833243+0000) 2019-10-13T12:18:46.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:46.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157868 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:46.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:46.835+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:46.839392+0000) 2019-10-13T12:18:47.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:47.859+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:47.859271+0000) 2019-10-13T12:18:48.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:48.079+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.085261+0000) 2019-10-13T12:18:48.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:48.103+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.106129+0000) 2019-10-13T12:18:48.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:48.623+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:48.626779+0000) 2019-10-13T12:18:51.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:51.838+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.841964+0000) 2019-10-13T12:18:51.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:51.838+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.843787+0000) 2019-10-13T12:18:51.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:51.838+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.844145+0000) 2019-10-13T12:18:51.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:51.842+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.846414+0000) 2019-10-13T12:18:51.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:51.847+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.848361+0000) 2019-10-13T12:18:51.851 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:51.846+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.850175+0000) 2019-10-13T12:18:51.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:51.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157897 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:51.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:51.846+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.850778+0000) 2019-10-13T12:18:51.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:51.851+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.855855+0000) 2019-10-13T12:18:51.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:51.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:51.876957+0000) 2019-10-13T12:18:54.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:54.198+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:54.202109+0000) 2019-10-13T12:18:54.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:54.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:54.487738+0000) 2019-10-13T12:18:54.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:54.683+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:54.683394+0000) 2019-10-13T12:18:56.855 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:18:56.850+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.854261+0000) 2019-10-13T12:18:56.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:18:56.851+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.854898+0000) 2019-10-13T12:18:56.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:56.850+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.855055+0000) 2019-10-13T12:18:56.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:18:56.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.855738+0000) 2019-10-13T12:18:56.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:56.850+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.856050+0000) 2019-10-13T12:18:56.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:56.850+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.856920+0000) 2019-10-13T12:18:56.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:56.854+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.858234+0000) 2019-10-13T12:18:56.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:18:56.854+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:18:56.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:18:56.863+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:56.864744+0000) 2019-10-13T12:18:57.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:18:57.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:57.859474+0000) 2019-10-13T12:18:58.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:18:58.082+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.085493+0000) 2019-10-13T12:18:58.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:18:58.102+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.106345+0000) 2019-10-13T12:18:58.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:18:58.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:18:58.626948+0000) 2019-10-13T12:19:01.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:01.858+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.864865+0000) 2019-10-13T12:19:01.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:01.863+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.865744+0000) 2019-10-13T12:19:01.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:01.858+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.865376+0000) 2019-10-13T12:19:01.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:01.862+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.866617+0000) 2019-10-13T12:19:01.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:01.862+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.866737+0000) 2019-10-13T12:19:01.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:01.862+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.867547+0000) 2019-10-13T12:19:01.869 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:01.862+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.868630+0000) 2019-10-13T12:19:01.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:01.867+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.871898+0000) 2019-10-13T12:19:01.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:01.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157960 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:01.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:01.875+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:01.877116+0000) 2019-10-13T12:19:04.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:04.198+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:04.202309+0000) 2019-10-13T12:19:04.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:04.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:04.487963+0000) 2019-10-13T12:19:04.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:04.687+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:04.689991+0000) 2019-10-13T12:19:06.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:06.870+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.876299+0000) 2019-10-13T12:19:06.877 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:06.870+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.876830+0000) 2019-10-13T12:19:06.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:06.870+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.877243+0000) 2019-10-13T12:19:06.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:06.874+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.877990+0000) 2019-10-13T12:19:06.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:06.875+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.879162+0000) 2019-10-13T12:19:06.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:06.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.879723+0000) 2019-10-13T12:19:06.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:06.874+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.880482+0000) 2019-10-13T12:19:06.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:06.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:06.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:06.879+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:06.883657+0000) 2019-10-13T12:19:06.942 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:19:06.942 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:06.946 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:19:06.946 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:07.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:07.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:07.859673+0000) 2019-10-13T12:19:08.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:08.078+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.085702+0000) 2019-10-13T12:19:08.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:08.102+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.106565+0000) 2019-10-13T12:19:08.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:08.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:08.627127+0000) 2019-10-13T12:19:11.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:11.874+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.877285+0000) 2019-10-13T12:19:11.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:11.882+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.886293+0000) 2019-10-13T12:19:11.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:11.882+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.886954+0000) 2019-10-13T12:19:11.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:11.882+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.888028+0000) 2019-10-13T12:19:11.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:11.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.888834+0000) 2019-10-13T12:19:11.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:11.882+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.888994+0000) 2019-10-13T12:19:11.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:11.882+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.889544+0000) 2019-10-13T12:19:11.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:11.886+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.889949+0000) 2019-10-13T12:19:11.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:11.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158021 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:11.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:11.894+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:11.897462+0000) 2019-10-13T12:19:14.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:14.198+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:14.202506+0000) 2019-10-13T12:19:14.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:14.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:14.488197+0000) 2019-10-13T12:19:14.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:14.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:14.701018+0000) 2019-10-13T12:19:16.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:16.894+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.898479+0000) 2019-10-13T12:19:16.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:16.894+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.900381+0000) 2019-10-13T12:19:16.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:16.894+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.900106+0000) 2019-10-13T12:19:16.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:16.894+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.901043+0000) 2019-10-13T12:19:16.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:16.894+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.901461+0000) 2019-10-13T12:19:16.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:16.898+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.902800+0000) 2019-10-13T12:19:16.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:16.898+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.903108+0000) 2019-10-13T12:19:16.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:16.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:16.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:16.906+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:16.908312+0000) 2019-10-13T12:19:17.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:17.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:17.859855+0000) 2019-10-13T12:19:18.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:18.082+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.086003+0000) 2019-10-13T12:19:18.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:18.102+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.106781+0000) 2019-10-13T12:19:18.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:18.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:18.627324+0000) 2019-10-13T12:19:21.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:21.874+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.877518+0000) 2019-10-13T12:19:21.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:21.902+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.908694+0000) 2019-10-13T12:19:21.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:21.906+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.912205+0000) 2019-10-13T12:19:21.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:21.910+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.912602+0000) 2019-10-13T12:19:21.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:21.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.912972+0000) 2019-10-13T12:19:21.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:21.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.913794+0000) 2019-10-13T12:19:21.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:21.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.913937+0000) 2019-10-13T12:19:21.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:21.910+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.914517+0000) 2019-10-13T12:19:21.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:21.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:21.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:21.918+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:21.921090+0000) 2019-10-13T12:19:24.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:24.202+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:24.202739+0000) 2019-10-13T12:19:24.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:24.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:24.488402+0000) 2019-10-13T12:19:24.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:24.702+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:24.707539+0000) 2019-10-13T12:19:26.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:26.918+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.920661+0000) 2019-10-13T12:19:26.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:26.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.920836+0000) 2019-10-13T12:19:26.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:26.914+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.921323+0000) 2019-10-13T12:19:26.922 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:26.914+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.921599+0000) 2019-10-13T12:19:26.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:26.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.922580+0000) 2019-10-13T12:19:26.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:26.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.922925+0000) 2019-10-13T12:19:26.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:26.918+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.923921+0000) 2019-10-13T12:19:26.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:26.922+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:26.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:26.926+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:26.930178+0000) 2019-10-13T12:19:27.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:27.854+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:27.860030+0000) 2019-10-13T12:19:28.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:28.082+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.086204+0000) 2019-10-13T12:19:28.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:28.106+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.107016+0000) 2019-10-13T12:19:28.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:28.622+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:28.627516+0000) 2019-10-13T12:19:31.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:31.874+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.877709+0000) 2019-10-13T12:19:31.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:31.926+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.932048+0000) 2019-10-13T12:19:31.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:31.930+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.933964+0000) 2019-10-13T12:19:31.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:31.930+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.934641+0000) 2019-10-13T12:19:31.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:31.930+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.934918+0000) 2019-10-13T12:19:31.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:31.930+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.935394+0000) 2019-10-13T12:19:31.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:31.930+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.935774+0000) 2019-10-13T12:19:31.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:31.930+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.936102+0000) 2019-10-13T12:19:31.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:31.934+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:31.938614+0000) 2019-10-13T12:19:31.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:31.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:34.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:34.202+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:34.202992+0000) 2019-10-13T12:19:34.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:34.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:34.488633+0000) 2019-10-13T12:19:34.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:34.710+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:34.714204+0000) 2019-10-13T12:19:36.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:36.938+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.942439+0000) 2019-10-13T12:19:36.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:36.938+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.943163+0000) 2019-10-13T12:19:36.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:36.938+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.943498+0000) 2019-10-13T12:19:36.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:36.938+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.943946+0000) 2019-10-13T12:19:36.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:36.938+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.944215+0000) 2019-10-13T12:19:36.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:36.938+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.944388+0000) 2019-10-13T12:19:36.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:36.938+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.944989+0000) 2019-10-13T12:19:36.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:36.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:36.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:36.950+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:36.952105+0000) 2019-10-13T12:19:36.983 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:19:36.983 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:36.987 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:19:36.987 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:19:37.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:37.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:37.860222+0000) 2019-10-13T12:19:38.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:38.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.086407+0000) 2019-10-13T12:19:38.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:38.106+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.107272+0000) 2019-10-13T12:19:38.628 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:38.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:38.627688+0000) 2019-10-13T12:19:41.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:41.874+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.877943+0000) 2019-10-13T12:19:41.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:41.946+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.952256+0000) 2019-10-13T12:19:41.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:41.946+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.953080+0000) 2019-10-13T12:19:41.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:41.946+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.953381+0000) 2019-10-13T12:19:41.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:41.950+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.954135+0000) 2019-10-13T12:19:41.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:41.950+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.955017+0000) 2019-10-13T12:19:41.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:41.950+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.955310+0000) 2019-10-13T12:19:41.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:41.950+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.957005+0000) 2019-10-13T12:19:41.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:41.958+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:41.962020+0000) 2019-10-13T12:19:41.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:41.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:44.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:44.198+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:44.203188+0000) 2019-10-13T12:19:44.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:44.482+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:44.488832+0000) 2019-10-13T12:19:44.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:44.718+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:44.720630+0000) 2019-10-13T12:19:46.965 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:46.957+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.964550+0000) 2019-10-13T12:19:46.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:46.962+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.964986+0000) 2019-10-13T12:19:46.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:46.957+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.965258+0000) 2019-10-13T12:19:46.966 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:46.957+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.965507+0000) 2019-10-13T12:19:46.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:46.961+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.967121+0000) 2019-10-13T12:19:46.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:46.961+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.967734+0000) 2019-10-13T12:19:46.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:46.961+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.969545+0000) 2019-10-13T12:19:46.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:46.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:46.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:46.970+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:46.975725+0000) 2019-10-13T12:19:47.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:47.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:47.860416+0000) 2019-10-13T12:19:48.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:48.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.086603+0000) 2019-10-13T12:19:48.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:48.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.107469+0000) 2019-10-13T12:19:48.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:48.621+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:48.627915+0000) 2019-10-13T12:19:51.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:51.874+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.878154+0000) 2019-10-13T12:19:51.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:51.969+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.975285+0000) 2019-10-13T12:19:51.977 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:51.969+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.975816+0000) 2019-10-13T12:19:51.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:51.974+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.976302+0000) 2019-10-13T12:19:51.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:51.969+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.977029+0000) 2019-10-13T12:19:51.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:51.969+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.977214+0000) 2019-10-13T12:19:51.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:51.973+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.978185+0000) 2019-10-13T12:19:51.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:51.973+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.978487+0000) 2019-10-13T12:19:51.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:51.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:51.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:51.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:51.984291+0000) 2019-10-13T12:19:54.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:54.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:54.203346+0000) 2019-10-13T12:19:54.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:54.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:54.489029+0000) 2019-10-13T12:19:54.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:54.726+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:54.727093+0000) 2019-10-13T12:19:56.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:56.981+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.987634+0000) 2019-10-13T12:19:56.989 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:19:56.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.987925+0000) 2019-10-13T12:19:56.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:56.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.988138+0000) 2019-10-13T12:19:56.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:56.981+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.988712+0000) 2019-10-13T12:19:56.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:19:56.986+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.991027+0000) 2019-10-13T12:19:56.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:19:56.985+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.991314+0000) 2019-10-13T12:19:56.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:56.985+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:56.992278+0000) 2019-10-13T12:19:56.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:19:56.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:19:57.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:19:56.998+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.000458+0000) 2019-10-13T12:19:57.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:19:57.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:57.860623+0000) 2019-10-13T12:19:58.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:19:58.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.086816+0000) 2019-10-13T12:19:58.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:19:58.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.107667+0000) 2019-10-13T12:19:58.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:19:58.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:19:58.628104+0000) 2019-10-13T12:20:01.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:01.878+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:01.878347+0000) 2019-10-13T12:20:02.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:01.997+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:01.999077+0000) 2019-10-13T12:20:02.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:01.997+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:01.999572+0000) 2019-10-13T12:20:02.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:01.997+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.000115+0000) 2019-10-13T12:20:02.002 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:01.997+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.001620+0000) 2019-10-13T12:20:02.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:01.998+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.002723+0000) 2019-10-13T12:20:02.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:02.001+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.002949+0000) 2019-10-13T12:20:02.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:02.001+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.004623+0000) 2019-10-13T12:20:02.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:02.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:02.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:02.006+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:02.008543+0000) 2019-10-13T12:20:04.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:04.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:04.203570+0000) 2019-10-13T12:20:04.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:04.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:04.489271+0000) 2019-10-13T12:20:04.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:04.730+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:04.733578+0000) 2019-10-13T12:20:07.010 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:07.009+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.009719+0000) 2019-10-13T12:20:07.011 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:07.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.010931+0000) 2019-10-13T12:20:07.012 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:07.009+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.011652+0000) 2019-10-13T12:20:07.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:07.009+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.012146+0000) 2019-10-13T12:20:07.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:07.009+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.012308+0000) 2019-10-13T12:20:07.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:07.010+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.012660+0000) 2019-10-13T12:20:07.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:07.009+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.012801+0000) 2019-10-13T12:20:07.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:07.014+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.017452+0000) 2019-10-13T12:20:07.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:07.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:07.040 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:20:07.040 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:07.043 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:20:07.044 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:07.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:07.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:07.860830+0000) 2019-10-13T12:20:08.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:08.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.087030+0000) 2019-10-13T12:20:08.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:08.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.107882+0000) 2019-10-13T12:20:08.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:08.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:08.628314+0000) 2019-10-13T12:20:11.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:11.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:11.878549+0000) 2019-10-13T12:20:12.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:12.017+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.021530+0000) 2019-10-13T12:20:12.022 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:12.017+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.021697+0000) 2019-10-13T12:20:12.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:12.021+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.022327+0000) 2019-10-13T12:20:12.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:12.021+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.024025+0000) 2019-10-13T12:20:12.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:12.025+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.025858+0000) 2019-10-13T12:20:12.028 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:12.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:12.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:12.025+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.028167+0000) 2019-10-13T12:20:12.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:12.025+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.029114+0000) 2019-10-13T12:20:12.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:12.033+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:12.036474+0000) 2019-10-13T12:20:14.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:14.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:14.203821+0000) 2019-10-13T12:20:14.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:14.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:14.489469+0000) 2019-10-13T12:20:14.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:14.733+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:14.740068+0000) 2019-10-13T12:20:17.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:17.029+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.031301+0000) 2019-10-13T12:20:17.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:17.029+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.032114+0000) 2019-10-13T12:20:17.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:17.029+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.032565+0000) 2019-10-13T12:20:17.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:17.033+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.034552+0000) 2019-10-13T12:20:17.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:17.033+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.036700+0000) 2019-10-13T12:20:17.038 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:17.033+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.037676+0000) 2019-10-13T12:20:17.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:17.033+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.038505+0000) 2019-10-13T12:20:17.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:17.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.038837+0000) 2019-10-13T12:20:17.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:17.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:17.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:17.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:17.861015+0000) 2019-10-13T12:20:18.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:18.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.087266+0000) 2019-10-13T12:20:18.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:18.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.108138+0000) 2019-10-13T12:20:18.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:18.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:18.628511+0000) 2019-10-13T12:20:21.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:21.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:21.878763+0000) 2019-10-13T12:20:22.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:22.041+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.042398+0000) 2019-10-13T12:20:22.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:22.041+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.042820+0000) 2019-10-13T12:20:22.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:22.041+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.043594+0000) 2019-10-13T12:20:22.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:22.041+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.044594+0000) 2019-10-13T12:20:22.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:22.041+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.045287+0000) 2019-10-13T12:20:22.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:22.041+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.045567+0000) 2019-10-13T12:20:22.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:22.045+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.046516+0000) 2019-10-13T12:20:22.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:22.045+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:22.049553+0000) 2019-10-13T12:20:22.051 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:22.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158456 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:24.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:24.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:24.204047+0000) 2019-10-13T12:20:24.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:24.485+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:24.489704+0000) 2019-10-13T12:20:24.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:24.745+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:24.746300+0000) 2019-10-13T12:20:27.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:27.053+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.053843+0000) 2019-10-13T12:20:27.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:27.053+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.054330+0000) 2019-10-13T12:20:27.055 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:27.053+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.054787+0000) 2019-10-13T12:20:27.057 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:27.053+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.056980+0000) 2019-10-13T12:20:27.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:27.057+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.060231+0000) 2019-10-13T12:20:27.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:27.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:27.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:27.061+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.062281+0000) 2019-10-13T12:20:27.063 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:27.057+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.062651+0000) 2019-10-13T12:20:27.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:27.069+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.072767+0000) 2019-10-13T12:20:27.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:27.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:27.861193+0000) 2019-10-13T12:20:28.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:28.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.087487+0000) 2019-10-13T12:20:28.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:28.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.108337+0000) 2019-10-13T12:20:28.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:28.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:28.628668+0000) 2019-10-13T12:20:31.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:31.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:31.878984+0000) 2019-10-13T12:20:32.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:32.061+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.065103+0000) 2019-10-13T12:20:32.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:32.061+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.065436+0000) 2019-10-13T12:20:32.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:32.065+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.066414+0000) 2019-10-13T12:20:32.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:32.065+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.066798+0000) 2019-10-13T12:20:32.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:32.065+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.068049+0000) 2019-10-13T12:20:32.069 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:32.065+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.068393+0000) 2019-10-13T12:20:32.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:32.065+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.068938+0000) 2019-10-13T12:20:32.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:32.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:32.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:32.069+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:32.073772+0000) 2019-10-13T12:20:34.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:34.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:34.204265+0000) 2019-10-13T12:20:34.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:34.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:34.489946+0000) 2019-10-13T12:20:34.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:34.749+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:34.755101+0000) 2019-10-13T12:20:37.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:37.073+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.076221+0000) 2019-10-13T12:20:37.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:37.073+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.076527+0000) 2019-10-13T12:20:37.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:37.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.077528+0000) 2019-10-13T12:20:37.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:37.073+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.077560+0000) 2019-10-13T12:20:37.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:37.073+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.077634+0000) 2019-10-13T12:20:37.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:37.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.078316+0000) 2019-10-13T12:20:37.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:37.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.079074+0000) 2019-10-13T12:20:37.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:37.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:37.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:37.081+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.085738+0000) 2019-10-13T12:20:37.170 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:20:37.170 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:37.179 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:20:37.179 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:20:37.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:37.857+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:37.861390+0000) 2019-10-13T12:20:38.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:38.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.087682+0000) 2019-10-13T12:20:38.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:38.105+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.108570+0000) 2019-10-13T12:20:38.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:38.625+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:38.628842+0000) 2019-10-13T12:20:41.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:41.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:41.879244+0000) 2019-10-13T12:20:42.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:42.084+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.087141+0000) 2019-10-13T12:20:42.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:42.085+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.088205+0000) 2019-10-13T12:20:42.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:42.084+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.089414+0000) 2019-10-13T12:20:42.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:42.088+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.090367+0000) 2019-10-13T12:20:42.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:42.088+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.090806+0000) 2019-10-13T12:20:42.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:42.088+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.091249+0000) 2019-10-13T12:20:42.092 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:42.088+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.091752+0000) 2019-10-13T12:20:42.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:42.092+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:42.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:42.093+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:42.096369+0000) 2019-10-13T12:20:44.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:44.200+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:44.204468+0000) 2019-10-13T12:20:44.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:44.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:44.490145+0000) 2019-10-13T12:20:44.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:44.757+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:44.761496+0000) 2019-10-13T12:20:47.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:47.096+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.099052+0000) 2019-10-13T12:20:47.100 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:47.096+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.099234+0000) 2019-10-13T12:20:47.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:47.096+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.099720+0000) 2019-10-13T12:20:47.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:47.096+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.100145+0000) 2019-10-13T12:20:47.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:47.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.100621+0000) 2019-10-13T12:20:47.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:47.097+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.101588+0000) 2019-10-13T12:20:47.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:47.096+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.101572+0000) 2019-10-13T12:20:47.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:47.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:47.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:47.105+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.108279+0000) 2019-10-13T12:20:47.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:47.860+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:47.861589+0000) 2019-10-13T12:20:48.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:48.084+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.087918+0000) 2019-10-13T12:20:48.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:48.104+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.108766+0000) 2019-10-13T12:20:48.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:48.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:48.629013+0000) 2019-10-13T12:20:51.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:51.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:51.879438+0000) 2019-10-13T12:20:52.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:52.105+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.109407+0000) 2019-10-13T12:20:52.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:52.104+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.109673+0000) 2019-10-13T12:20:52.111 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:52.108+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.110377+0000) 2019-10-13T12:20:52.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:52.108+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.112807+0000) 2019-10-13T12:20:52.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:52.112+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.115278+0000) 2019-10-13T12:20:52.118 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:52.112+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.117114+0000) 2019-10-13T12:20:52.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:52.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:52.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:52.116+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.118484+0000) 2019-10-13T12:20:52.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:52.113+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:52.119453+0000) 2019-10-13T12:20:54.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:54.200+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:54.204684+0000) 2019-10-13T12:20:54.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:54.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:54.490394+0000) 2019-10-13T12:20:54.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:54.765+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:54.767972+0000) 2019-10-13T12:20:57.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:20:57.116+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.121140+0000) 2019-10-13T12:20:57.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:57.120+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.122178+0000) 2019-10-13T12:20:57.140 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:57.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.122545+0000) 2019-10-13T12:20:57.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:57.120+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.124395+0000) 2019-10-13T12:20:57.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:20:57.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:20:57.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:57.124+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.129139+0000) 2019-10-13T12:20:57.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:20:57.128+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.129826+0000) 2019-10-13T12:20:57.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:20:57.121+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.127010+0000) 2019-10-13T12:20:57.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:20:57.121+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.127984+0000) 2019-10-13T12:20:57.862 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:20:57.860+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:57.861795+0000) 2019-10-13T12:20:58.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:20:58.084+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.088150+0000) 2019-10-13T12:20:58.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:20:58.104+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.108992+0000) 2019-10-13T12:20:58.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:20:58.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:20:58.629158+0000) 2019-10-13T12:21:01.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:01.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:01.879660+0000) 2019-10-13T12:21:02.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:02.128+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.132312+0000) 2019-10-13T12:21:02.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:02.128+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.133621+0000) 2019-10-13T12:21:02.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:02.132+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.133783+0000) 2019-10-13T12:21:02.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:02.132+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.133965+0000) 2019-10-13T12:21:02.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:02.132+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.134420+0000) 2019-10-13T12:21:02.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:02.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.135297+0000) 2019-10-13T12:21:02.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:02.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.135832+0000) 2019-10-13T12:21:02.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:02.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:02.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:02.137+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:02.141642+0000) 2019-10-13T12:21:04.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:04.200+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:04.204931+0000) 2019-10-13T12:21:04.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:04.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:04.490597+0000) 2019-10-13T12:21:04.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:04.769+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:04.774510+0000) 2019-10-13T12:21:07.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:07.137+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.143554+0000) 2019-10-13T12:21:07.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:07.140+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.144061+0000) 2019-10-13T12:21:07.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:07.140+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.144363+0000) 2019-10-13T12:21:07.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:07.144+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.146353+0000) 2019-10-13T12:21:07.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:07.144+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.146849+0000) 2019-10-13T12:21:07.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:07.144+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.148640+0000) 2019-10-13T12:21:07.151 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:07.148+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.150339+0000) 2019-10-13T12:21:07.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:07.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:07.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:07.153+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.156599+0000) 2019-10-13T12:21:07.290 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:21:07.290 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:07.294 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:21:07.294 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:07.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:07.860+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:07.861989+0000) 2019-10-13T12:21:08.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:08.084+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.088359+0000) 2019-10-13T12:21:08.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:08.104+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.109185+0000) 2019-10-13T12:21:08.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:08.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:08.629341+0000) 2019-10-13T12:21:11.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:11.877+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:11.879892+0000) 2019-10-13T12:21:12.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:12.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.154913+0000) 2019-10-13T12:21:12.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:12.152+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.155164+0000) 2019-10-13T12:21:12.157 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:12.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.156191+0000) 2019-10-13T12:21:12.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:12.153+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.157531+0000) 2019-10-13T12:21:12.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:12.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.157129+0000) 2019-10-13T12:21:12.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:12.156+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.157773+0000) 2019-10-13T12:21:12.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:12.156+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.158400+0000) 2019-10-13T12:21:12.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:12.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158764 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:12.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:12.161+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:12.164682+0000) 2019-10-13T12:21:14.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:14.204+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:14.205132+0000) 2019-10-13T12:21:14.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:14.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:14.490813+0000) 2019-10-13T12:21:14.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:14.781+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:14.785851+0000) 2019-10-13T12:21:17.167 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:17.164+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.166485+0000) 2019-10-13T12:21:17.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:17.164+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.168601+0000) 2019-10-13T12:21:17.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:17.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.169084+0000) 2019-10-13T12:21:17.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:17.164+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.169336+0000) 2019-10-13T12:21:17.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:17.168+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.170810+0000) 2019-10-13T12:21:17.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:17.168+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.172448+0000) 2019-10-13T12:21:17.173 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:17.168+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.172828+0000) 2019-10-13T12:21:17.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:17.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:17.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:17.172+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.179873+0000) 2019-10-13T12:21:17.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:17.860+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:17.862169+0000) 2019-10-13T12:21:18.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:18.084+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.088542+0000) 2019-10-13T12:21:18.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:18.104+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.109386+0000) 2019-10-13T12:21:18.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:18.624+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:18.629579+0000) 2019-10-13T12:21:21.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:21.876+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:21.880143+0000) 2019-10-13T12:21:22.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:22.172+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.176384+0000) 2019-10-13T12:21:22.178 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:22.172+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.178014+0000) 2019-10-13T12:21:22.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:22.176+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.179821+0000) 2019-10-13T12:21:22.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:22.176+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.180294+0000) 2019-10-13T12:21:22.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:22.176+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.180752+0000) 2019-10-13T12:21:22.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:22.176+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.180938+0000) 2019-10-13T12:21:22.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:22.176+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.181323+0000) 2019-10-13T12:21:22.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:22.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158827 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:22.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:22.180+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:22.188011+0000) 2019-10-13T12:21:24.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:24.204+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:24.205328+0000) 2019-10-13T12:21:24.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:24.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:24.491011+0000) 2019-10-13T12:21:24.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:24.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:24.792398+0000) 2019-10-13T12:21:27.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:27.188+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.190838+0000) 2019-10-13T12:21:27.192 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:27.184+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.191306+0000) 2019-10-13T12:21:27.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:27.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.191738+0000) 2019-10-13T12:21:27.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:27.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.192635+0000) 2019-10-13T12:21:27.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:27.188+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.192371+0000) 2019-10-13T12:21:27.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:27.188+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.193589+0000) 2019-10-13T12:21:27.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:27.192+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.195939+0000) 2019-10-13T12:21:27.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:27.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:27.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:27.196+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.201516+0000) 2019-10-13T12:21:27.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:27.860+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:27.862349+0000) 2019-10-13T12:21:28.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:28.084+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.088714+0000) 2019-10-13T12:21:28.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:28.108+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.109558+0000) 2019-10-13T12:21:28.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:28.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:28.629776+0000) 2019-10-13T12:21:31.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:31.876+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:31.880342+0000) 2019-10-13T12:21:32.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:32.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.201205+0000) 2019-10-13T12:21:32.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:32.200+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.201773+0000) 2019-10-13T12:21:32.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:32.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.202433+0000) 2019-10-13T12:21:32.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:32.200+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.203075+0000) 2019-10-13T12:21:32.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:32.200+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.205171+0000) 2019-10-13T12:21:32.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:32.204+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.207318+0000) 2019-10-13T12:21:32.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:32.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:32.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:32.204+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.209032+0000) 2019-10-13T12:21:32.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:32.208+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:32.213213+0000) 2019-10-13T12:21:34.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:34.200+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:34.205539+0000) 2019-10-13T12:21:34.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:34.488+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:34.491249+0000) 2019-10-13T12:21:34.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:34.792+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:34.798910+0000) 2019-10-13T12:21:37.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:37.207+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.212539+0000) 2019-10-13T12:21:37.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:37.207+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.213054+0000) 2019-10-13T12:21:37.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:37.207+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.213220+0000) 2019-10-13T12:21:37.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:37.207+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.213542+0000) 2019-10-13T12:21:37.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:37.211+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.214604+0000) 2019-10-13T12:21:37.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:37.211+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.214798+0000) 2019-10-13T12:21:37.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:37.208+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.215334+0000) 2019-10-13T12:21:37.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:37.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:37.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:37.216+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.221870+0000) 2019-10-13T12:21:37.338 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:21:37.339 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:37.342 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:21:37.342 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:21:37.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:37.859+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:37.862546+0000) 2019-10-13T12:21:38.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:38.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.088889+0000) 2019-10-13T12:21:38.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:38.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.109735+0000) 2019-10-13T12:21:38.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:38.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:38.630005+0000) 2019-10-13T12:21:41.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:41.876+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:41.880568+0000) 2019-10-13T12:21:42.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:42.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.222790+0000) 2019-10-13T12:21:42.224 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:42.219+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.223180+0000) 2019-10-13T12:21:42.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:42.219+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.223946+0000) 2019-10-13T12:21:42.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:42.219+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.224195+0000) 2019-10-13T12:21:42.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:42.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.224736+0000) 2019-10-13T12:21:42.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:42.220+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.225889+0000) 2019-10-13T12:21:42.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:42.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.226127+0000) 2019-10-13T12:21:42.232 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:42.228+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:42.232194+0000) 2019-10-13T12:21:42.233 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:42.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:44.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:44.203+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:44.205731+0000) 2019-10-13T12:21:44.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:44.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:44.491453+0000) 2019-10-13T12:21:44.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:44.804+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:44.805292+0000) 2019-10-13T12:21:47.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:47.231+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.235362+0000) 2019-10-13T12:21:47.238 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:47.231+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.237364+0000) 2019-10-13T12:21:47.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:47.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.237888+0000) 2019-10-13T12:21:47.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:47.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.238655+0000) 2019-10-13T12:21:47.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:47.235+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.240221+0000) 2019-10-13T12:21:47.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:47.235+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.241437+0000) 2019-10-13T12:21:47.243 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:47.239+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.242726+0000) 2019-10-13T12:21:47.244 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:47.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:47.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:47.240+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.245131+0000) 2019-10-13T12:21:47.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:47.859+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:47.862756+0000) 2019-10-13T12:21:48.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:48.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.089069+0000) 2019-10-13T12:21:48.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:48.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.110009+0000) 2019-10-13T12:21:48.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:48.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:48.630228+0000) 2019-10-13T12:21:51.881 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:51.880+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:51.880764+0000) 2019-10-13T12:21:52.248 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:52.243+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.246873+0000) 2019-10-13T12:21:52.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:52.243+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.247336+0000) 2019-10-13T12:21:52.249 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:52.240+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.247919+0000) 2019-10-13T12:21:52.250 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:52.243+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.249452+0000) 2019-10-13T12:21:52.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:52.247+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.249980+0000) 2019-10-13T12:21:52.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:52.247+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.251888+0000) 2019-10-13T12:21:52.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:52.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159012 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:52.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:52.251+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.254920+0000) 2019-10-13T12:21:52.260 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:52.252+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:52.259873+0000) 2019-10-13T12:21:54.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:54.203+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:54.205945+0000) 2019-10-13T12:21:54.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:54.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:54.491650+0000) 2019-10-13T12:21:54.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:54.808+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:54.811943+0000) 2019-10-13T12:21:57.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:57.255+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.258345+0000) 2019-10-13T12:21:57.260 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:57.255+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.259550+0000) 2019-10-13T12:21:57.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:21:57.256+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.261441+0000) 2019-10-13T12:21:57.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:57.255+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.261508+0000) 2019-10-13T12:21:57.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:21:57.259+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.261839+0000) 2019-10-13T12:21:57.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:21:57.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.262603+0000) 2019-10-13T12:21:57.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:57.259+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.263000+0000) 2019-10-13T12:21:57.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:21:57.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:21:57.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:21:57.260+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.267976+0000) 2019-10-13T12:21:57.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:21:57.859+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:57.862934+0000) 2019-10-13T12:21:58.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:21:58.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.089269+0000) 2019-10-13T12:21:58.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:21:58.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.110189+0000) 2019-10-13T12:21:58.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:21:58.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:21:58.630438+0000) 2019-10-13T12:22:01.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:01.880+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:01.880959+0000) 2019-10-13T12:22:02.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:02.267+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.270019+0000) 2019-10-13T12:22:02.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:02.267+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.270499+0000) 2019-10-13T12:22:02.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:02.267+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.270975+0000) 2019-10-13T12:22:02.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:02.267+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.271895+0000) 2019-10-13T12:22:02.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:02.267+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.272115+0000) 2019-10-13T12:22:02.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:02.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.272842+0000) 2019-10-13T12:22:02.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:02.267+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.272968+0000) 2019-10-13T12:22:02.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:02.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:02.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:02.272+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:02.278762+0000) 2019-10-13T12:22:04.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:04.203+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:04.206147+0000) 2019-10-13T12:22:04.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:04.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:04.491864+0000) 2019-10-13T12:22:04.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:04.820+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:04.821238+0000) 2019-10-13T12:22:07.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:07.275+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.279484+0000) 2019-10-13T12:22:07.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:07.275+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.280840+0000) 2019-10-13T12:22:07.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:07.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.281402+0000) 2019-10-13T12:22:07.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:07.275+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.281524+0000) 2019-10-13T12:22:07.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:07.279+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.282671+0000) 2019-10-13T12:22:07.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:07.276+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.283283+0000) 2019-10-13T12:22:07.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:07.279+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.283506+0000) 2019-10-13T12:22:07.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:07.280+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.286388+0000) 2019-10-13T12:22:07.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:07.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:07.415 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:22:07.415 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:07.419 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:22:07.419 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:07.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:07.859+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:07.863117+0000) 2019-10-13T12:22:08.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:08.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.089445+0000) 2019-10-13T12:22:08.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:08.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.110385+0000) 2019-10-13T12:22:08.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:08.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:08.630631+0000) 2019-10-13T12:22:11.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:11.880+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:11.881194+0000) 2019-10-13T12:22:12.291 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:12.287+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.290585+0000) 2019-10-13T12:22:12.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:12.287+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.291141+0000) 2019-10-13T12:22:12.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:12.288+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.292247+0000) 2019-10-13T12:22:12.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:12.287+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.292524+0000) 2019-10-13T12:22:12.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:12.287+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.293334+0000) 2019-10-13T12:22:12.294 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:12.287+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.293394+0000) 2019-10-13T12:22:12.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:12.287+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.293657+0000) 2019-10-13T12:22:12.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:12.292+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:12.297629+0000) 2019-10-13T12:22:12.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:12.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:14.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:14.203+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:14.206379+0000) 2019-10-13T12:22:14.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:14.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:14.492078+0000) 2019-10-13T12:22:14.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:14.824+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:14.827614+0000) 2019-10-13T12:22:17.302 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:17.300+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.301806+0000) 2019-10-13T12:22:17.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:17.299+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.302236+0000) 2019-10-13T12:22:17.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:17.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.303005+0000) 2019-10-13T12:22:17.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:17.299+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.303531+0000) 2019-10-13T12:22:17.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:17.303+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.307113+0000) 2019-10-13T12:22:17.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:17.303+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.308676+0000) 2019-10-13T12:22:17.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:17.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159169 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:17.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:17.307+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.310920+0000) 2019-10-13T12:22:17.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:17.316+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.317472+0000) 2019-10-13T12:22:17.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:17.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:17.863299+0000) 2019-10-13T12:22:18.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:18.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.089618+0000) 2019-10-13T12:22:18.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:18.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.110568+0000) 2019-10-13T12:22:18.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:18.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:18.630786+0000) 2019-10-13T12:22:21.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:21.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:21.881397+0000) 2019-10-13T12:22:22.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:22.311+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.315258+0000) 2019-10-13T12:22:22.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:22.311+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.315518+0000) 2019-10-13T12:22:22.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:22.311+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.317099+0000) 2019-10-13T12:22:22.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:22.311+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.317628+0000) 2019-10-13T12:22:22.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:22.315+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.318309+0000) 2019-10-13T12:22:22.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:22.315+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.318555+0000) 2019-10-13T12:22:22.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:22.315+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.319672+0000) 2019-10-13T12:22:22.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:22.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:22.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:22.323+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:22.325214+0000) 2019-10-13T12:22:24.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:24.203+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:24.206591+0000) 2019-10-13T12:22:24.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:24.487+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:24.492221+0000) 2019-10-13T12:22:24.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:24.831+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:24.834095+0000) 2019-10-13T12:22:27.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:27.319+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.323994+0000) 2019-10-13T12:22:27.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:27.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.324242+0000) 2019-10-13T12:22:27.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:27.319+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.325000+0000) 2019-10-13T12:22:27.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:27.323+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.327944+0000) 2019-10-13T12:22:27.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:27.323+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.329237+0000) 2019-10-13T12:22:27.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:27.327+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.331072+0000) 2019-10-13T12:22:27.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:27.331+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.332138+0000) 2019-10-13T12:22:27.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:27.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:27.333 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:27.327+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.332669+0000) 2019-10-13T12:22:27.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:27.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:27.863474+0000) 2019-10-13T12:22:28.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:28.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.089818+0000) 2019-10-13T12:22:28.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:28.107+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.110767+0000) 2019-10-13T12:22:28.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:28.627+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:28.630981+0000) 2019-10-13T12:22:31.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:31.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:31.881608+0000) 2019-10-13T12:22:32.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:32.330+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.335884+0000) 2019-10-13T12:22:32.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:32.330+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.336157+0000) 2019-10-13T12:22:32.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:32.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.337309+0000) 2019-10-13T12:22:32.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:32.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.337556+0000) 2019-10-13T12:22:32.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:32.334+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.338400+0000) 2019-10-13T12:22:32.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:32.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.339143+0000) 2019-10-13T12:22:32.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:32.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.339256+0000) 2019-10-13T12:22:32.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:32.339+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:32.342304+0000) 2019-10-13T12:22:32.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:32.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159260 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:34.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:34.202+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:34.206811+0000) 2019-10-13T12:22:34.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:34.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:34.492448+0000) 2019-10-13T12:22:34.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:34.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:34.839884+0000) 2019-10-13T12:22:37.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:37.342+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.347877+0000) 2019-10-13T12:22:37.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:37.342+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.348474+0000) 2019-10-13T12:22:37.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:37.342+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.349164+0000) 2019-10-13T12:22:37.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:37.346+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.350313+0000) 2019-10-13T12:22:37.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:37.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.354639+0000) 2019-10-13T12:22:37.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:37.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:37.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:37.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.355531+0000) 2019-10-13T12:22:37.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:37.350+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.355166+0000) 2019-10-13T12:22:37.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:37.355+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.358870+0000) 2019-10-13T12:22:37.481 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:22:37.481 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:37.485 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:22:37.485 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:22:37.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:37.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:37.863661+0000) 2019-10-13T12:22:38.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:38.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.089990+0000) 2019-10-13T12:22:38.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:38.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.110941+0000) 2019-10-13T12:22:38.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:38.626+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:38.631200+0000) 2019-10-13T12:22:41.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:41.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:41.881804+0000) 2019-10-13T12:22:42.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:42.354+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.358067+0000) 2019-10-13T12:22:42.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:42.354+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.359134+0000) 2019-10-13T12:22:42.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:42.359+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.361155+0000) 2019-10-13T12:22:42.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:42.354+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.361400+0000) 2019-10-13T12:22:42.363 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:42.358+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.362654+0000) 2019-10-13T12:22:42.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:42.358+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.363469+0000) 2019-10-13T12:22:42.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:42.358+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.364244+0000) 2019-10-13T12:22:42.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:42.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:42.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:42.367+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:42.369423+0000) 2019-10-13T12:22:44.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:44.206+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:44.207022+0000) 2019-10-13T12:22:44.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:44.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:44.492664+0000) 2019-10-13T12:22:44.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:44.843+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:44.846408+0000) 2019-10-13T12:22:47.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:47.366+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.370708+0000) 2019-10-13T12:22:47.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:47.366+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.371819+0000) 2019-10-13T12:22:47.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:47.366+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.372234+0000) 2019-10-13T12:22:47.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:47.371+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.372644+0000) 2019-10-13T12:22:47.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:47.370+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.374589+0000) 2019-10-13T12:22:47.375 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:47.370+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.374873+0000) 2019-10-13T12:22:47.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:47.370+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.376697+0000) 2019-10-13T12:22:47.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:47.379+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.381824+0000) 2019-10-13T12:22:47.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:47.374+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:47.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:47.862+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:47.863861+0000) 2019-10-13T12:22:48.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:48.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.090169+0000) 2019-10-13T12:22:48.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:48.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.111123+0000) 2019-10-13T12:22:48.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:48.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:48.631439+0000) 2019-10-13T12:22:51.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:51.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:51.882031+0000) 2019-10-13T12:22:52.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:52.378+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.382412+0000) 2019-10-13T12:22:52.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:52.379+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.383122+0000) 2019-10-13T12:22:52.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:52.378+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.383366+0000) 2019-10-13T12:22:52.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:52.378+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.384323+0000) 2019-10-13T12:22:52.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:52.378+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.385137+0000) 2019-10-13T12:22:52.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:52.378+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.385622+0000) 2019-10-13T12:22:52.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:52.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.385401+0000) 2019-10-13T12:22:52.390 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:52.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159382 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:52.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:52.387+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:52.390364+0000) 2019-10-13T12:22:54.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:54.202+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:54.207258+0000) 2019-10-13T12:22:54.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:54.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:54.492867+0000) 2019-10-13T12:22:54.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:54.855+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:54.857708+0000) 2019-10-13T12:22:57.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:22:57.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.393011+0000) 2019-10-13T12:22:57.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:57.390+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.395317+0000) 2019-10-13T12:22:57.396 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:57.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.395603+0000) 2019-10-13T12:22:57.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:57.390+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.397069+0000) 2019-10-13T12:22:57.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:57.390+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.397290+0000) 2019-10-13T12:22:57.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:22:57.395+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.397903+0000) 2019-10-13T12:22:57.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:22:57.394+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.398505+0000) 2019-10-13T12:22:57.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:22:57.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:22:57.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:22:57.403+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.404905+0000) 2019-10-13T12:22:57.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:22:57.862+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:57.864043+0000) 2019-10-13T12:22:58.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:22:58.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.090354+0000) 2019-10-13T12:22:58.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:22:58.106+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.111298+0000) 2019-10-13T12:22:58.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:22:58.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:22:58.631639+0000) 2019-10-13T12:23:01.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:01.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:01.882221+0000) 2019-10-13T12:23:02.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:02.398+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.403079+0000) 2019-10-13T12:23:02.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:02.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.405135+0000) 2019-10-13T12:23:02.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:02.402+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.407144+0000) 2019-10-13T12:23:02.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:02.402+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.408627+0000) 2019-10-13T12:23:02.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:02.406+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.409802+0000) 2019-10-13T12:23:02.411 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:02.406+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.410783+0000) 2019-10-13T12:23:02.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:02.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:02.412 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:02.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.411809+0000) 2019-10-13T12:23:02.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:02.411+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:02.413125+0000) 2019-10-13T12:23:04.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:04.202+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:04.207460+0000) 2019-10-13T12:23:04.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:04.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:04.493078+0000) 2019-10-13T12:23:04.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:04.863+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:04.864320+0000) 2019-10-13T12:23:07.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:07.410+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.414914+0000) 2019-10-13T12:23:07.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:07.410+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.415677+0000) 2019-10-13T12:23:07.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:07.410+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.416927+0000) 2019-10-13T12:23:07.418 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:07.410+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.417182+0000) 2019-10-13T12:23:07.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:07.410+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.417340+0000) 2019-10-13T12:23:07.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:07.414+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.417940+0000) 2019-10-13T12:23:07.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:07.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.418897+0000) 2019-10-13T12:23:07.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:07.419+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.421807+0000) 2019-10-13T12:23:07.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:07.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:07.560 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:23:07.561 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:07.564 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:23:07.565 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:07.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:07.858+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:07.864227+0000) 2019-10-13T12:23:08.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:08.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.090594+0000) 2019-10-13T12:23:08.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:08.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.111511+0000) 2019-10-13T12:23:08.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:08.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:08.631868+0000) 2019-10-13T12:23:11.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:11.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:11.882417+0000) 2019-10-13T12:23:12.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:12.418+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.425232+0000) 2019-10-13T12:23:12.427 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:12.418+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.425687+0000) 2019-10-13T12:23:12.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:12.422+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.426966+0000) 2019-10-13T12:23:12.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:12.422+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.427294+0000) 2019-10-13T12:23:12.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:12.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.429240+0000) 2019-10-13T12:23:12.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:12.426+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.430346+0000) 2019-10-13T12:23:12.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:12.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.431329+0000) 2019-10-13T12:23:12.434 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:12.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:12.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:12.435+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:12.436846+0000) 2019-10-13T12:23:14.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:14.206+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:14.207671+0000) 2019-10-13T12:23:14.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:14.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:14.493291+0000) 2019-10-13T12:23:14.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:14.867+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:14.870777+0000) 2019-10-13T12:23:17.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:17.430+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.436545+0000) 2019-10-13T12:23:17.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:17.434+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.438389+0000) 2019-10-13T12:23:17.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:17.435+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.439063+0000) 2019-10-13T12:23:17.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:17.438+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.442944+0000) 2019-10-13T12:23:17.444 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:17.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:17.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:17.438+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.444475+0000) 2019-10-13T12:23:17.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:17.438+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.445030+0000) 2019-10-13T12:23:17.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:17.438+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.444772+0000) 2019-10-13T12:23:17.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:17.443+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.445907+0000) 2019-10-13T12:23:17.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:17.862+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:17.864451+0000) 2019-10-13T12:23:18.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:18.086+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.090770+0000) 2019-10-13T12:23:18.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:18.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.111694+0000) 2019-10-13T12:23:18.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:18.630+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:18.632063+0000) 2019-10-13T12:23:21.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:21.879+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:21.882613+0000) 2019-10-13T12:23:22.448 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:22.442+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.447542+0000) 2019-10-13T12:23:22.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:22.442+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.448333+0000) 2019-10-13T12:23:22.449 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:22.447+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.448772+0000) 2019-10-13T12:23:22.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:22.446+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.450634+0000) 2019-10-13T12:23:22.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:22.446+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.451109+0000) 2019-10-13T12:23:22.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:22.446+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.451710+0000) 2019-10-13T12:23:22.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:22.450+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.453892+0000) 2019-10-13T12:23:22.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:22.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:22.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:22.455+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:22.457443+0000) 2019-10-13T12:23:24.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:24.206+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:24.207875+0000) 2019-10-13T12:23:24.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:24.486+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:24.493523+0000) 2019-10-13T12:23:24.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:24.875+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:24.877281+0000) 2019-10-13T12:23:27.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:27.453+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.458936+0000) 2019-10-13T12:23:27.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:27.453+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.459350+0000) 2019-10-13T12:23:27.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:27.453+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.459590+0000) 2019-10-13T12:23:27.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:27.453+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.460296+0000) 2019-10-13T12:23:27.461 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:27.453+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.460539+0000) 2019-10-13T12:23:27.462 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:27.458+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.461522+0000) 2019-10-13T12:23:27.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:27.457+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.462463+0000) 2019-10-13T12:23:27.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:27.461+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:27.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:27.466+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.468391+0000) 2019-10-13T12:23:27.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:27.861+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:27.864625+0000) 2019-10-13T12:23:28.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:28.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.090965+0000) 2019-10-13T12:23:28.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:28.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.111899+0000) 2019-10-13T12:23:28.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:28.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:28.632269+0000) 2019-10-13T12:23:31.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:31.878+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:31.882800+0000) 2019-10-13T12:23:32.471 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:32.465+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.470024+0000) 2019-10-13T12:23:32.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:32.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.471802+0000) 2019-10-13T12:23:32.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:32.465+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.472537+0000) 2019-10-13T12:23:32.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:32.465+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.473108+0000) 2019-10-13T12:23:32.474 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:32.470+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.473756+0000) 2019-10-13T12:23:32.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:32.469+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.474093+0000) 2019-10-13T12:23:32.475 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:32.469+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.474512+0000) 2019-10-13T12:23:32.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:32.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:32.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:32.478+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:32.481122+0000) 2019-10-13T12:23:34.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:34.201+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:34.208074+0000) 2019-10-13T12:23:34.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:34.489+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:34.493778+0000) 2019-10-13T12:23:34.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:34.878+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:34.883859+0000) 2019-10-13T12:23:37.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:37.473+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.480515+0000) 2019-10-13T12:23:37.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:37.477+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.482143+0000) 2019-10-13T12:23:37.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:37.477+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.483198+0000) 2019-10-13T12:23:37.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:37.477+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.482622+0000) 2019-10-13T12:23:37.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:37.481+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.485853+0000) 2019-10-13T12:23:37.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:37.482+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.486130+0000) 2019-10-13T12:23:37.487 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:37.481+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.486506+0000) 2019-10-13T12:23:37.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:37.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159655 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:37.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:37.486+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.490362+0000) 2019-10-13T12:23:37.662 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:23:37.662 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:37.666 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:23:37.666 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:23:37.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:37.861+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:37.864826+0000) 2019-10-13T12:23:38.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:38.085+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.091140+0000) 2019-10-13T12:23:38.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:38.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.112113+0000) 2019-10-13T12:23:38.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:38.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:38.632513+0000) 2019-10-13T12:23:41.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:41.878+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:41.882997+0000) 2019-10-13T12:23:42.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:42.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.491976+0000) 2019-10-13T12:23:42.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:42.489+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.493361+0000) 2019-10-13T12:23:42.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:42.489+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.493658+0000) 2019-10-13T12:23:42.510 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:42.493+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.496943+0000) 2019-10-13T12:23:42.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:42.494+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.497477+0000) 2019-10-13T12:23:42.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:42.497+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.497772+0000) 2019-10-13T12:23:42.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:42.497+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.498366+0000) 2019-10-13T12:23:42.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:42.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:42.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:42.498+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:42.503354+0000) 2019-10-13T12:23:44.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:44.205+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:44.208275+0000) 2019-10-13T12:23:44.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:44.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:44.493977+0000) 2019-10-13T12:23:44.891 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:44.886+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:44.890431+0000) 2019-10-13T12:23:47.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:47.501+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.504870+0000) 2019-10-13T12:23:47.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:47.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.505774+0000) 2019-10-13T12:23:47.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:47.505+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.506427+0000) 2019-10-13T12:23:47.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:47.505+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.507176+0000) 2019-10-13T12:23:47.509 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:47.502+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.507701+0000) 2019-10-13T12:23:47.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:47.505+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.507193+0000) 2019-10-13T12:23:47.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:47.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.508045+0000) 2019-10-13T12:23:47.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:47.509+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:47.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:47.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.512849+0000) 2019-10-13T12:23:47.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:47.861+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:47.865044+0000) 2019-10-13T12:23:48.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:48.089+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.091317+0000) 2019-10-13T12:23:48.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:48.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.112312+0000) 2019-10-13T12:23:48.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:48.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:48.632708+0000) 2019-10-13T12:23:51.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:51.878+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:51.883212+0000) 2019-10-13T12:23:52.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:52.510+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.515151+0000) 2019-10-13T12:23:52.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:52.513+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.515368+0000) 2019-10-13T12:23:52.517 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:52.513+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.516122+0000) 2019-10-13T12:23:52.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:52.513+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.516382+0000) 2019-10-13T12:23:52.518 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:52.513+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.517195+0000) 2019-10-13T12:23:52.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:52.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.518443+0000) 2019-10-13T12:23:52.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:52.517+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.519801+0000) 2019-10-13T12:23:52.524 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:52.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:52.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:52.518+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:52.524019+0000) 2019-10-13T12:23:54.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:54.205+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:54.208472+0000) 2019-10-13T12:23:54.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:54.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:54.494177+0000) 2019-10-13T12:23:54.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:54.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:54.897011+0000) 2019-10-13T12:23:57.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:57.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.526168+0000) 2019-10-13T12:23:57.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:23:57.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.526707+0000) 2019-10-13T12:23:57.528 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:57.525+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.527418+0000) 2019-10-13T12:23:57.529 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:23:57.525+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.528125+0000) 2019-10-13T12:23:57.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:57.525+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.528460+0000) 2019-10-13T12:23:57.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:57.525+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.529254+0000) 2019-10-13T12:23:57.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:23:57.529+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.530005+0000) 2019-10-13T12:23:57.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:23:57.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:23:57.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:23:57.534+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.537650+0000) 2019-10-13T12:23:57.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:23:57.861+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:57.865226+0000) 2019-10-13T12:23:58.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:23:58.089+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.091549+0000) 2019-10-13T12:23:58.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:23:58.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.112490+0000) 2019-10-13T12:23:58.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:23:58.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:23:58.632900+0000) 2019-10-13T12:24:01.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:01.878+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:01.883407+0000) 2019-10-13T12:24:02.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:02.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.537374+0000) 2019-10-13T12:24:02.539 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:02.537+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.537900+0000) 2019-10-13T12:24:02.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:02.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.538163+0000) 2019-10-13T12:24:02.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:02.537+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.538677+0000) 2019-10-13T12:24:02.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:02.537+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.539505+0000) 2019-10-13T12:24:02.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:02.537+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.540720+0000) 2019-10-13T12:24:02.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:02.537+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.541546+0000) 2019-10-13T12:24:02.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:02.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:02.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:02.542+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:02.545968+0000) 2019-10-13T12:24:04.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:04.205+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:04.208692+0000) 2019-10-13T12:24:04.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:04.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:04.494379+0000) 2019-10-13T12:24:04.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:04.898+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:04.903623+0000) 2019-10-13T12:24:07.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:07.545+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.548153+0000) 2019-10-13T12:24:07.551 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:07.549+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.550181+0000) 2019-10-13T12:24:07.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:07.546+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.551575+0000) 2019-10-13T12:24:07.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:07.549+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.551677+0000) 2019-10-13T12:24:07.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:07.549+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.552146+0000) 2019-10-13T12:24:07.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:07.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.552442+0000) 2019-10-13T12:24:07.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:07.549+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.552789+0000) 2019-10-13T12:24:07.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:07.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:07.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:07.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.557014+0000) 2019-10-13T12:24:07.775 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:24:07.775 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:07.779 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:24:07.779 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:07.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:07.861+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:07.865458+0000) 2019-10-13T12:24:08.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:08.089+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.091723+0000) 2019-10-13T12:24:08.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:08.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.112659+0000) 2019-10-13T12:24:08.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:08.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:08.633130+0000) 2019-10-13T12:24:11.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:11.882+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:11.883604+0000) 2019-10-13T12:24:12.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:12.557+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.559941+0000) 2019-10-13T12:24:12.561 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:12.557+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.560515+0000) 2019-10-13T12:24:12.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:12.561+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.562397+0000) 2019-10-13T12:24:12.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:12.561+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.562802+0000) 2019-10-13T12:24:12.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:12.561+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.563030+0000) 2019-10-13T12:24:12.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:12.562+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.564851+0000) 2019-10-13T12:24:12.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:12.561+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.565075+0000) 2019-10-13T12:24:12.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:12.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:12.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:12.570+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:12.573652+0000) 2019-10-13T12:24:14.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:14.205+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:14.208875+0000) 2019-10-13T12:24:14.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:14.493+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:14.494577+0000) 2019-10-13T12:24:14.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:14.906+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:14.910250+0000) 2019-10-13T12:24:17.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:17.569+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.571003+0000) 2019-10-13T12:24:17.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:17.569+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.571475+0000) 2019-10-13T12:24:17.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:17.569+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.572224+0000) 2019-10-13T12:24:17.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:17.570+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.572859+0000) 2019-10-13T12:24:17.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:17.569+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.573445+0000) 2019-10-13T12:24:17.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:17.573+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.574351+0000) 2019-10-13T12:24:17.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:17.573+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.574610+0000) 2019-10-13T12:24:17.580 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:17.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:17.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:17.578+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.580314+0000) 2019-10-13T12:24:17.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:17.865+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:17.865633+0000) 2019-10-13T12:24:18.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:18.089+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.091906+0000) 2019-10-13T12:24:18.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:18.109+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.112867+0000) 2019-10-13T12:24:18.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:18.629+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:18.633326+0000) 2019-10-13T12:24:21.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:21.882+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:21.883827+0000) 2019-10-13T12:24:22.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:22.578+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.582278+0000) 2019-10-13T12:24:22.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:22.580+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.583934+0000) 2019-10-13T12:24:22.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:22.580+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.584278+0000) 2019-10-13T12:24:22.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:22.584+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.586488+0000) 2019-10-13T12:24:22.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:22.584+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.587274+0000) 2019-10-13T12:24:22.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:22.584+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.588981+0000) 2019-10-13T12:24:22.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:22.588+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.590718+0000) 2019-10-13T12:24:22.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:22.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:22.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:22.590+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:22.592358+0000) 2019-10-13T12:24:24.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:24.208+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:24.209091+0000) 2019-10-13T12:24:24.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:24.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:24.494795+0000) 2019-10-13T12:24:24.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:24.918+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:24.921159+0000) 2019-10-13T12:24:27.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:27.592+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.594885+0000) 2019-10-13T12:24:27.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:27.590+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.595267+0000) 2019-10-13T12:24:27.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:27.592+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.595528+0000) 2019-10-13T12:24:27.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:27.592+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.596488+0000) 2019-10-13T12:24:27.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:27.592+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.596842+0000) 2019-10-13T12:24:27.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:27.592+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.597040+0000) 2019-10-13T12:24:27.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:27.592+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.597582+0000) 2019-10-13T12:24:27.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:27.600+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:27.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:27.602+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.604843+0000) 2019-10-13T12:24:27.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:27.864+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:27.865764+0000) 2019-10-13T12:24:28.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:28.088+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.092085+0000) 2019-10-13T12:24:28.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:28.112+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.113082+0000) 2019-10-13T12:24:28.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:28.628+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:28.633515+0000) 2019-10-13T12:24:31.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:31.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:31.884037+0000) 2019-10-13T12:24:32.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:32.600+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.605507+0000) 2019-10-13T12:24:32.607 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:32.604+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.606329+0000) 2019-10-13T12:24:32.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:32.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.607826+0000) 2019-10-13T12:24:32.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:32.604+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.608042+0000) 2019-10-13T12:24:32.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:32.604+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.608376+0000) 2019-10-13T12:24:32.610 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:32.604+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.608267+0000) 2019-10-13T12:24:32.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:32.604+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.608870+0000) 2019-10-13T12:24:32.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:32.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:32.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:32.609+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:32.614795+0000) 2019-10-13T12:24:34.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:34.208+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:34.209320+0000) 2019-10-13T12:24:34.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:34.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:34.495004+0000) 2019-10-13T12:24:34.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:34.925+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:34.927786+0000) 2019-10-13T12:24:37.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:37.613+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.616545+0000) 2019-10-13T12:24:37.619 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:37.616+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.617988+0000) 2019-10-13T12:24:37.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:37.616+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.618818+0000) 2019-10-13T12:24:37.620 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:37.616+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.618616+0000) 2019-10-13T12:24:37.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:37.616+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.620996+0000) 2019-10-13T12:24:37.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:37.620+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.621840+0000) 2019-10-13T12:24:37.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:37.620+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.622140+0000) 2019-10-13T12:24:37.625 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:37.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160015 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:37.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:37.621+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.626550+0000) 2019-10-13T12:24:37.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:37.864+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:37.865918+0000) 2019-10-13T12:24:37.914 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:24:37.915 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:37.918 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:24:37.918 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:24:38.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:38.088+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.092261+0000) 2019-10-13T12:24:38.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:38.108+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.113289+0000) 2019-10-13T12:24:38.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:38.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:38.633764+0000) 2019-10-13T12:24:41.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:41.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:41.884260+0000) 2019-10-13T12:24:42.628 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:42.621+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.627574+0000) 2019-10-13T12:24:42.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:42.624+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.629569+0000) 2019-10-13T12:24:42.631 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:42.628+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.629924+0000) 2019-10-13T12:24:42.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:42.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.630933+0000) 2019-10-13T12:24:42.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:42.628+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.630570+0000) 2019-10-13T12:24:42.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:42.628+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.631104+0000) 2019-10-13T12:24:42.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:42.628+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.633625+0000) 2019-10-13T12:24:42.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:42.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160043 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:42.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:42.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:42.641046+0000) 2019-10-13T12:24:44.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:44.204+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:44.209516+0000) 2019-10-13T12:24:44.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:44.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:44.495210+0000) 2019-10-13T12:24:44.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:44.929+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:44.934274+0000) 2019-10-13T12:24:47.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:47.636+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.639178+0000) 2019-10-13T12:24:47.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:47.636+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.639737+0000) 2019-10-13T12:24:47.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:47.636+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.640180+0000) 2019-10-13T12:24:47.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:47.636+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.641114+0000) 2019-10-13T12:24:47.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:47.637+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.641481+0000) 2019-10-13T12:24:47.644 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:47.640+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.643123+0000) 2019-10-13T12:24:47.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:47.640+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.643378+0000) 2019-10-13T12:24:47.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:47.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:47.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:47.645+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.649427+0000) 2019-10-13T12:24:47.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:47.864+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:47.866098+0000) 2019-10-13T12:24:48.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:48.088+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.092448+0000) 2019-10-13T12:24:48.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:48.108+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.113465+0000) 2019-10-13T12:24:48.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:48.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:48.634006+0000) 2019-10-13T12:24:51.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:51.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:51.884491+0000) 2019-10-13T12:24:52.651 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:52.648+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.650566+0000) 2019-10-13T12:24:52.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:52.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.651299+0000) 2019-10-13T12:24:52.653 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:52.648+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.651835+0000) 2019-10-13T12:24:52.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:52.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.651911+0000) 2019-10-13T12:24:52.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:52.648+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.653251+0000) 2019-10-13T12:24:52.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:52.652+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.654222+0000) 2019-10-13T12:24:52.656 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:52.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.655124+0000) 2019-10-13T12:24:52.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:52.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:52.661 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:52.657+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:52.660950+0000) 2019-10-13T12:24:54.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:54.204+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:54.209716+0000) 2019-10-13T12:24:54.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:54.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:54.495438+0000) 2019-10-13T12:24:54.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:54.937+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:54.940819+0000) 2019-10-13T12:24:57.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:24:57.660+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.662041+0000) 2019-10-13T12:24:57.664 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:57.660+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.662664+0000) 2019-10-13T12:24:57.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:24:57.660+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.663352+0000) 2019-10-13T12:24:57.664 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:57.660+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.663827+0000) 2019-10-13T12:24:57.665 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:24:57.661+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.664246+0000) 2019-10-13T12:24:57.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:57.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.664557+0000) 2019-10-13T12:24:57.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:57.660+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.665683+0000) 2019-10-13T12:24:57.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:24:57.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:24:57.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:24:57.669+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.672658+0000) 2019-10-13T12:24:57.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:24:57.864+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:57.866277+0000) 2019-10-13T12:24:58.093 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:24:58.088+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.092623+0000) 2019-10-13T12:24:58.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:24:58.108+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.113642+0000) 2019-10-13T12:24:58.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:24:58.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:24:58.634188+0000) 2019-10-13T12:25:01.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:01.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:01.884713+0000) 2019-10-13T12:25:02.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:02.672+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.674179+0000) 2019-10-13T12:25:02.676 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:02.672+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.674902+0000) 2019-10-13T12:25:02.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:02.672+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.675422+0000) 2019-10-13T12:25:02.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:02.672+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.676088+0000) 2019-10-13T12:25:02.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:02.673+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.677117+0000) 2019-10-13T12:25:02.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:02.672+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.677227+0000) 2019-10-13T12:25:02.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:02.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.677369+0000) 2019-10-13T12:25:02.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:02.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160163 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:02.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:02.677+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:02.683032+0000) 2019-10-13T12:25:04.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:04.208+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:04.209951+0000) 2019-10-13T12:25:04.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:04.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:04.495599+0000) 2019-10-13T12:25:04.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:04.945+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:04.947386+0000) 2019-10-13T12:25:07.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:07.684+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.685787+0000) 2019-10-13T12:25:07.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:07.684+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.687108+0000) 2019-10-13T12:25:07.688 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:07.684+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.687396+0000) 2019-10-13T12:25:07.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:07.684+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.687979+0000) 2019-10-13T12:25:07.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:07.685+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.688415+0000) 2019-10-13T12:25:07.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:07.684+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.688649+0000) 2019-10-13T12:25:07.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:07.688+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.690149+0000) 2019-10-13T12:25:07.695 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:07.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:07.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:07.689+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.695686+0000) 2019-10-13T12:25:07.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:07.864+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:07.866472+0000) 2019-10-13T12:25:08.028 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:25:08.028 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:08.032 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:25:08.032 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:08.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:08.088+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.092814+0000) 2019-10-13T12:25:08.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:08.112+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.113832+0000) 2019-10-13T12:25:08.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:08.632+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:08.634390+0000) 2019-10-13T12:25:11.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:11.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:11.884921+0000) 2019-10-13T12:25:12.699 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:12.696+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.698167+0000) 2019-10-13T12:25:12.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:12.696+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.700304+0000) 2019-10-13T12:25:12.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:12.696+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.700479+0000) 2019-10-13T12:25:12.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:12.697+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.701238+0000) 2019-10-13T12:25:12.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:12.700+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.702954+0000) 2019-10-13T12:25:12.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:12.700+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.704622+0000) 2019-10-13T12:25:12.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:12.700+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.705202+0000) 2019-10-13T12:25:12.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:12.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:12.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:12.713+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:12.716972+0000) 2019-10-13T12:25:14.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:14.208+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:14.210157+0000) 2019-10-13T12:25:14.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:14.492+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:14.495802+0000) 2019-10-13T12:25:14.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:14.949+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:14.953986+0000) 2019-10-13T12:25:17.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:17.707+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.711092+0000) 2019-10-13T12:25:17.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:17.707+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.711439+0000) 2019-10-13T12:25:17.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:17.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.711768+0000) 2019-10-13T12:25:17.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:17.707+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.713412+0000) 2019-10-13T12:25:17.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:17.709+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.714670+0000) 2019-10-13T12:25:17.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:17.711+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.715162+0000) 2019-10-13T12:25:17.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:17.711+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.715910+0000) 2019-10-13T12:25:17.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:17.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:17.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:17.717+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.721715+0000) 2019-10-13T12:25:17.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:17.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:17.866647+0000) 2019-10-13T12:25:18.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:18.091+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.093028+0000) 2019-10-13T12:25:18.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:18.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.114018+0000) 2019-10-13T12:25:18.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:18.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:18.634569+0000) 2019-10-13T12:25:21.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:21.881+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:21.885117+0000) 2019-10-13T12:25:22.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:22.715+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.721623+0000) 2019-10-13T12:25:22.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:22.719+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.722978+0000) 2019-10-13T12:25:22.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:22.719+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.723206+0000) 2019-10-13T12:25:22.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:22.719+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.723710+0000) 2019-10-13T12:25:22.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:22.719+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.724788+0000) 2019-10-13T12:25:22.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:22.719+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.725068+0000) 2019-10-13T12:25:22.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:22.721+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.725855+0000) 2019-10-13T12:25:22.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:22.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160283 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:22.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:22.729+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:22.734366+0000) 2019-10-13T12:25:24.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:24.207+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:24.210370+0000) 2019-10-13T12:25:24.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:24.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:24.496003+0000) 2019-10-13T12:25:24.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:24.957+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:24.960338+0000) 2019-10-13T12:25:27.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:27.731+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.734117+0000) 2019-10-13T12:25:27.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:27.731+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.735004+0000) 2019-10-13T12:25:27.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:27.731+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.736317+0000) 2019-10-13T12:25:27.738 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:27.731+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.736533+0000) 2019-10-13T12:25:27.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:27.731+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.737607+0000) 2019-10-13T12:25:27.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:27.733+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.738071+0000) 2019-10-13T12:25:27.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:27.735+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.738295+0000) 2019-10-13T12:25:27.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:27.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:27.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:27.741+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.745633+0000) 2019-10-13T12:25:27.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:27.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:27.866824+0000) 2019-10-13T12:25:28.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:28.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.093161+0000) 2019-10-13T12:25:28.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:28.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.114199+0000) 2019-10-13T12:25:28.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:28.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:28.634757+0000) 2019-10-13T12:25:31.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:31.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:31.885307+0000) 2019-10-13T12:25:32.747 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:32.743+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.746278+0000) 2019-10-13T12:25:32.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:32.745+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.748297+0000) 2019-10-13T12:25:32.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:32.747+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.749902+0000) 2019-10-13T12:25:32.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:32.747+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.751928+0000) 2019-10-13T12:25:32.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:32.751+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.754053+0000) 2019-10-13T12:25:32.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:32.749+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.754326+0000) 2019-10-13T12:25:32.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:32.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:32.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:32.751+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.754925+0000) 2019-10-13T12:25:32.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:32.751+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:32.755179+0000) 2019-10-13T12:25:34.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:34.207+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:34.210559+0000) 2019-10-13T12:25:34.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:34.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:34.496201+0000) 2019-10-13T12:25:34.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:34.960+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:34.966801+0000) 2019-10-13T12:25:37.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:37.751+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.757330+0000) 2019-10-13T12:25:37.759 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:37.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.758011+0000) 2019-10-13T12:25:37.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:37.755+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.761391+0000) 2019-10-13T12:25:37.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:37.759+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.763598+0000) 2019-10-13T12:25:37.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:37.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.763964+0000) 2019-10-13T12:25:37.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:37.759+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.764374+0000) 2019-10-13T12:25:37.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:37.759+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.764932+0000) 2019-10-13T12:25:37.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:37.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:37.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:37.768+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.773865+0000) 2019-10-13T12:25:37.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:37.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:37.867004+0000) 2019-10-13T12:25:38.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:38.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.093363+0000) 2019-10-13T12:25:38.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:38.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.114381+0000) 2019-10-13T12:25:38.170 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:25:38.170 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:38.174 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:25:38.174 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:25:38.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:38.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:38.634928+0000) 2019-10-13T12:25:41.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:41.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:41.885504+0000) 2019-10-13T12:25:42.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:42.767+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.769901+0000) 2019-10-13T12:25:42.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:42.764+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.770809+0000) 2019-10-13T12:25:42.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:42.767+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.772354+0000) 2019-10-13T12:25:42.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:42.771+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.774447+0000) 2019-10-13T12:25:42.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:42.771+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.774891+0000) 2019-10-13T12:25:42.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:42.771+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.776466+0000) 2019-10-13T12:25:42.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:42.771+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.776074+0000) 2019-10-13T12:25:42.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:42.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:42.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:42.776+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:42.782573+0000) 2019-10-13T12:25:44.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:44.207+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:44.210759+0000) 2019-10-13T12:25:44.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:44.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:44.496407+0000) 2019-10-13T12:25:44.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:44.968+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:44.972886+0000) 2019-10-13T12:25:47.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:47.775+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.781155+0000) 2019-10-13T12:25:47.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:47.779+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.781813+0000) 2019-10-13T12:25:47.783 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:47.779+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.782450+0000) 2019-10-13T12:25:47.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:47.779+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.784279+0000) 2019-10-13T12:25:47.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:47.780+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.787771+0000) 2019-10-13T12:25:47.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:47.783+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.785816+0000) 2019-10-13T12:25:47.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:47.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:47.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:47.783+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.788617+0000) 2019-10-13T12:25:47.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:47.788+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.794882+0000) 2019-10-13T12:25:47.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:47.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:47.867207+0000) 2019-10-13T12:25:48.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:48.087+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.093544+0000) 2019-10-13T12:25:48.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:48.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.114561+0000) 2019-10-13T12:25:48.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:48.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:48.635107+0000) 2019-10-13T12:25:51.886 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:51.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:51.885698+0000) 2019-10-13T12:25:52.792 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:52.787+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.791124+0000) 2019-10-13T12:25:52.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:52.791+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.793999+0000) 2019-10-13T12:25:52.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:52.791+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.794226+0000) 2019-10-13T12:25:52.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:52.788+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.794719+0000) 2019-10-13T12:25:52.796 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:52.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.795160+0000) 2019-10-13T12:25:52.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:52.791+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.795968+0000) 2019-10-13T12:25:52.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:52.791+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.796493+0000) 2019-10-13T12:25:52.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:52.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:52.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:52.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:52.801818+0000) 2019-10-13T12:25:54.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:54.207+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:54.210959+0000) 2019-10-13T12:25:54.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:54.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:54.496610+0000) 2019-10-13T12:25:54.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:54.972+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:54.979495+0000) 2019-10-13T12:25:57.803 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:25:57.799+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.802542+0000) 2019-10-13T12:25:57.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:25:57.796+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.803453+0000) 2019-10-13T12:25:57.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:57.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.803759+0000) 2019-10-13T12:25:57.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:57.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.804866+0000) 2019-10-13T12:25:57.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:57.799+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.804437+0000) 2019-10-13T12:25:57.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:57.799+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.805336+0000) 2019-10-13T12:25:57.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:25:57.803+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.806951+0000) 2019-10-13T12:25:57.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:25:57.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:25:57.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:25:57.804+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.811974+0000) 2019-10-13T12:25:57.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:25:57.863+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:57.867415+0000) 2019-10-13T12:25:58.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:25:58.091+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.093779+0000) 2019-10-13T12:25:58.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:25:58.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.114739+0000) 2019-10-13T12:25:58.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:25:58.635+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:25:58.635323+0000) 2019-10-13T12:26:01.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:01.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:01.885900+0000) 2019-10-13T12:26:02.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:02.811+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.814704+0000) 2019-10-13T12:26:02.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:02.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.815366+0000) 2019-10-13T12:26:02.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:02.811+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.815923+0000) 2019-10-13T12:26:02.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:02.811+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.816056+0000) 2019-10-13T12:26:02.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:02.811+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.816596+0000) 2019-10-13T12:26:02.819 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:02.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.818191+0000) 2019-10-13T12:26:02.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:02.815+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.818330+0000) 2019-10-13T12:26:02.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:02.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:02.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:02.820+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:02.825000+0000) 2019-10-13T12:26:04.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:04.207+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:04.211130+0000) 2019-10-13T12:26:04.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:04.491+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:04.496845+0000) 2019-10-13T12:26:04.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:04.984+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:04.985538+0000) 2019-10-13T12:26:07.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:07.823+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.828598+0000) 2019-10-13T12:26:07.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:07.823+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.829118+0000) 2019-10-13T12:26:07.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:07.828+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.832237+0000) 2019-10-13T12:26:07.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:07.827+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.833585+0000) 2019-10-13T12:26:07.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:07.828+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.833921+0000) 2019-10-13T12:26:07.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:07.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:07.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:07.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.834752+0000) 2019-10-13T12:26:07.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:07.831+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.835195+0000) 2019-10-13T12:26:07.843 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:07.831+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.834566+0000) 2019-10-13T12:26:07.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:07.867+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:07.867631+0000) 2019-10-13T12:26:08.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:08.091+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.093971+0000) 2019-10-13T12:26:08.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:08.111+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.114916+0000) 2019-10-13T12:26:08.222 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:26:08.223 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:08.229 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:26:08.229 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:08.636 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:08.631+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:08.635498+0000) 2019-10-13T12:26:11.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:11.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:11.886100+0000) 2019-10-13T12:26:12.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:12.831+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.837450+0000) 2019-10-13T12:26:12.839 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:12.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.837935+0000) 2019-10-13T12:26:12.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:12.835+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.841254+0000) 2019-10-13T12:26:12.842 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:12.835+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.840978+0000) 2019-10-13T12:26:12.843 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:12.839+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.842310+0000) 2019-10-13T12:26:12.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:12.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.842947+0000) 2019-10-13T12:26:12.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:12.839+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.843335+0000) 2019-10-13T12:26:12.846 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:12.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160583 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:12.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:12.844+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:12.850181+0000) 2019-10-13T12:26:14.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:14.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:14.211378+0000) 2019-10-13T12:26:14.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:14.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:14.497050+0000) 2019-10-13T12:26:14.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:14.984+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:14.991997+0000) 2019-10-13T12:26:17.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:17.842+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.849578+0000) 2019-10-13T12:26:17.851 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:17.846+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.850093+0000) 2019-10-13T12:26:17.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:17.846+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.850696+0000) 2019-10-13T12:26:17.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:17.846+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.851369+0000) 2019-10-13T12:26:17.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:17.846+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.852166+0000) 2019-10-13T12:26:17.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:17.848+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.852773+0000) 2019-10-13T12:26:17.854 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:17.846+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.853210+0000) 2019-10-13T12:26:17.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:17.848+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.855252+0000) 2019-10-13T12:26:17.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:17.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:17.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:17.862+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:17.867787+0000) 2019-10-13T12:26:18.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:18.090+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.094146+0000) 2019-10-13T12:26:18.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:18.114+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.115092+0000) 2019-10-13T12:26:18.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:18.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:18.635693+0000) 2019-10-13T12:26:21.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:21.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:21.886298+0000) 2019-10-13T12:26:22.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:22.854+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.860749+0000) 2019-10-13T12:26:22.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:22.860+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.864829+0000) 2019-10-13T12:26:22.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:22.858+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.865232+0000) 2019-10-13T12:26:22.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:22.858+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.865593+0000) 2019-10-13T12:26:22.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:22.862+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.866034+0000) 2019-10-13T12:26:22.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:22.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.866866+0000) 2019-10-13T12:26:22.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:22.862+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.867187+0000) 2019-10-13T12:26:22.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:22.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:22.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:22.864+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:22.871909+0000) 2019-10-13T12:26:24.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:24.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:24.211595+0000) 2019-10-13T12:26:24.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:24.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:24.497255+0000) 2019-10-13T12:26:24.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:24.992+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:24.997980+0000) 2019-10-13T12:26:27.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:27.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.867995+0000) 2019-10-13T12:26:27.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:27.866+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.871835+0000) 2019-10-13T12:26:27.873 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:27.866+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.872219+0000) 2019-10-13T12:26:27.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:27.866+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.872900+0000) 2019-10-13T12:26:27.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:27.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.875465+0000) 2019-10-13T12:26:27.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:27.870+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.877195+0000) 2019-10-13T12:26:27.880 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:27.874+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.879239+0000) 2019-10-13T12:26:27.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:27.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:27.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:27.872+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.879454+0000) 2019-10-13T12:26:27.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:27.874+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:27.879749+0000) 2019-10-13T12:26:28.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:28.090+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.094329+0000) 2019-10-13T12:26:28.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:28.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.115290+0000) 2019-10-13T12:26:28.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:28.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:28.635867+0000) 2019-10-13T12:26:31.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:31.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:31.886488+0000) 2019-10-13T12:26:32.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:32.878+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.882208+0000) 2019-10-13T12:26:32.884 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:32.878+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.883563+0000) 2019-10-13T12:26:32.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:32.878+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.883984+0000) 2019-10-13T12:26:32.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:32.880+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.884556+0000) 2019-10-13T12:26:32.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:32.878+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.884696+0000) 2019-10-13T12:26:32.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:32.878+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.885277+0000) 2019-10-13T12:26:32.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:32.882+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.885979+0000) 2019-10-13T12:26:32.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:32.884+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:32.889769+0000) 2019-10-13T12:26:32.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:32.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:34.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:34.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:34.211800+0000) 2019-10-13T12:26:34.498 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:34.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:34.497454+0000) 2019-10-13T12:26:35.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:35.004+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:35.004536+0000) 2019-10-13T12:26:37.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:37.862+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.868187+0000) 2019-10-13T12:26:37.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:37.892+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.893795+0000) 2019-10-13T12:26:37.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:37.890+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.896219+0000) 2019-10-13T12:26:37.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:37.890+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.896468+0000) 2019-10-13T12:26:37.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:37.890+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.896783+0000) 2019-10-13T12:26:37.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:37.890+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.897177+0000) 2019-10-13T12:26:37.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:37.890+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.897523+0000) 2019-10-13T12:26:37.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:37.894+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.898258+0000) 2019-10-13T12:26:37.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:37.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:37.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:37.904+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:37.904921+0000) 2019-10-13T12:26:38.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:38.090+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.094501+0000) 2019-10-13T12:26:38.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:38.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.115498+0000) 2019-10-13T12:26:38.350 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:26:38.350 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:38.384 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:26:38.385 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:26:38.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:38.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:38.636074+0000) 2019-10-13T12:26:41.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:41.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:41.886679+0000) 2019-10-13T12:26:42.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:42.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.905304+0000) 2019-10-13T12:26:42.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:42.898+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.905655+0000) 2019-10-13T12:26:42.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:42.902+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.906207+0000) 2019-10-13T12:26:42.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:42.903+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.906742+0000) 2019-10-13T12:26:42.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:42.902+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.908027+0000) 2019-10-13T12:26:42.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:42.902+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.907694+0000) 2019-10-13T12:26:42.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:42.902+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.908733+0000) 2019-10-13T12:26:42.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:42.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:42.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:42.911+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:42.913519+0000) 2019-10-13T12:26:44.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:44.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:44.212044+0000) 2019-10-13T12:26:44.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:44.490+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:44.497656+0000) 2019-10-13T12:26:45.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:45.007+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:45.011099+0000) 2019-10-13T12:26:47.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:47.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.868403+0000) 2019-10-13T12:26:47.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:47.914+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.917958+0000) 2019-10-13T12:26:47.919 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:47.914+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.918134+0000) 2019-10-13T12:26:47.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:47.914+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.918362+0000) 2019-10-13T12:26:47.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:47.914+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.918994+0000) 2019-10-13T12:26:47.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:47.914+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.919281+0000) 2019-10-13T12:26:47.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:47.914+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.919495+0000) 2019-10-13T12:26:47.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:47.919+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.921137+0000) 2019-10-13T12:26:47.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:47.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:47.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:47.923+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:47.925208+0000) 2019-10-13T12:26:48.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:48.094+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.094682+0000) 2019-10-13T12:26:48.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:48.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.115688+0000) 2019-10-13T12:26:48.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:48.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:48.636264+0000) 2019-10-13T12:26:51.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:51.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:51.886891+0000) 2019-10-13T12:26:52.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:52.922+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.926939+0000) 2019-10-13T12:26:52.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:52.922+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.927284+0000) 2019-10-13T12:26:52.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:52.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.927441+0000) 2019-10-13T12:26:52.929 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:52.922+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.927828+0000) 2019-10-13T12:26:52.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:52.922+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.928784+0000) 2019-10-13T12:26:52.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:52.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.929118+0000) 2019-10-13T12:26:52.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:52.927+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.930666+0000) 2019-10-13T12:26:52.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:52.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:52.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:52.935+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:52.936471+0000) 2019-10-13T12:26:54.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:54.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:54.212302+0000) 2019-10-13T12:26:54.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:54.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:54.497889+0000) 2019-10-13T12:26:55.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:55.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:55.017604+0000) 2019-10-13T12:26:57.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:57.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.868604+0000) 2019-10-13T12:26:57.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:57.934+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.938313+0000) 2019-10-13T12:26:57.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:57.934+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.938985+0000) 2019-10-13T12:26:57.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:26:57.934+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.939530+0000) 2019-10-13T12:26:57.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:26:57.939+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.940161+0000) 2019-10-13T12:26:57.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:26:57.934+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.940402+0000) 2019-10-13T12:26:57.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:26:57.934+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.940726+0000) 2019-10-13T12:26:57.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:57.934+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.941043+0000) 2019-10-13T12:26:57.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:26:57.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:26:57.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:26:57.947+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:57.948861+0000) 2019-10-13T12:26:58.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:26:58.094+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.094898+0000) 2019-10-13T12:26:58.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:26:58.114+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.115869+0000) 2019-10-13T12:26:58.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:26:58.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:26:58.636481+0000) 2019-10-13T12:27:01.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:01.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:01.887094+0000) 2019-10-13T12:27:02.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:02.946+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.949865+0000) 2019-10-13T12:27:02.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:02.946+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.951274+0000) 2019-10-13T12:27:02.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:02.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.951795+0000) 2019-10-13T12:27:02.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:02.946+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.953684+0000) 2019-10-13T12:27:02.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:02.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.955392+0000) 2019-10-13T12:27:02.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:02.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.956663+0000) 2019-10-13T12:27:02.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:02.950+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.956828+0000) 2019-10-13T12:27:02.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:02.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:02.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:02.959+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:02.960894+0000) 2019-10-13T12:27:04.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:04.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:04.212508+0000) 2019-10-13T12:27:04.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:04.494+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:04.498086+0000) 2019-10-13T12:27:05.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:05.023+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:05.024151+0000) 2019-10-13T12:27:07.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:07.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.868826+0000) 2019-10-13T12:27:07.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:07.954+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.961085+0000) 2019-10-13T12:27:07.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:07.954+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.961576+0000) 2019-10-13T12:27:07.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:07.959+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.962003+0000) 2019-10-13T12:27:07.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:07.958+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.962056+0000) 2019-10-13T12:27:07.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:07.958+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.962403+0000) 2019-10-13T12:27:07.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:07.958+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.962956+0000) 2019-10-13T12:27:07.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:07.958+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.963565+0000) 2019-10-13T12:27:07.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:07.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160911 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:07.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:07.971+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:07.973227+0000) 2019-10-13T12:27:08.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:08.094+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.095079+0000) 2019-10-13T12:27:08.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:08.110+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.116052+0000) 2019-10-13T12:27:08.462 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:27:08.462 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:08.466 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:27:08.466 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:08.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:08.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:08.636675+0000) 2019-10-13T12:27:11.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:11.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:11.887292+0000) 2019-10-13T12:27:12.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:12.967+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.971980+0000) 2019-10-13T12:27:12.973 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:12.965+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.972405+0000) 2019-10-13T12:27:12.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:12.965+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.973398+0000) 2019-10-13T12:27:12.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:12.965+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.973137+0000) 2019-10-13T12:27:12.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:12.969+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.974590+0000) 2019-10-13T12:27:12.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:12.969+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.974296+0000) 2019-10-13T12:27:12.978 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:12.969+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.975669+0000) 2019-10-13T12:27:12.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:12.975+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:12.978056+0000) 2019-10-13T12:27:12.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:12.973+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:14.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:14.209+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:14.212732+0000) 2019-10-13T12:27:14.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:14.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:14.498293+0000) 2019-10-13T12:27:15.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:15.027+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:15.030612+0000) 2019-10-13T12:27:17.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:17.865+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.869048+0000) 2019-10-13T12:27:17.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:17.977+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.984243+0000) 2019-10-13T12:27:17.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:17.977+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.984625+0000) 2019-10-13T12:27:17.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:17.977+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.985193+0000) 2019-10-13T12:27:17.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:17.983+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.985793+0000) 2019-10-13T12:27:17.987 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:17.981+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.985891+0000) 2019-10-13T12:27:17.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:17.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.986227+0000) 2019-10-13T12:27:17.988 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:17.981+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.987616+0000) 2019-10-13T12:27:17.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:17.991+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:17.993600+0000) 2019-10-13T12:27:17.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:17.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:18.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:18.093+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.095224+0000) 2019-10-13T12:27:18.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:18.113+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.116234+0000) 2019-10-13T12:27:18.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:18.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:18.636872+0000) 2019-10-13T12:27:21.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:21.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:21.887479+0000) 2019-10-13T12:27:22.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:22.989+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.997101+0000) 2019-10-13T12:27:22.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:22.995+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.998264+0000) 2019-10-13T12:27:22.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:22.993+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.998142+0000) 2019-10-13T12:27:23.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:22.993+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.998483+0000) 2019-10-13T12:27:23.000 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:22.993+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:22.999485+0000) 2019-10-13T12:27:23.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:22.993+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.000503+0000) 2019-10-13T12:27:23.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:22.993+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.001100+0000) 2019-10-13T12:27:23.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:22.997+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160998 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:23.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:23.003+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:23.007516+0000) 2019-10-13T12:27:24.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:24.209+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:24.212946+0000) 2019-10-13T12:27:24.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:24.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:24.498492+0000) 2019-10-13T12:27:25.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:25.035+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:25.037084+0000) 2019-10-13T12:27:27.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:27.865+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:27.869246+0000) 2019-10-13T12:27:28.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:28.005+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.008951+0000) 2019-10-13T12:27:28.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:28.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.009449+0000) 2019-10-13T12:27:28.011 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:28.005+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.009587+0000) 2019-10-13T12:27:28.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:28.009+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.010340+0000) 2019-10-13T12:27:28.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:28.009+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.010584+0000) 2019-10-13T12:27:28.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:28.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.012320+0000) 2019-10-13T12:27:28.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:28.009+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.012721+0000) 2019-10-13T12:27:28.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:28.015+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.016791+0000) 2019-10-13T12:27:28.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:28.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161029 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:28.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:28.093+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.095452+0000) 2019-10-13T12:27:28.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:28.113+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.116435+0000) 2019-10-13T12:27:28.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:28.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:28.637044+0000) 2019-10-13T12:27:31.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:31.883+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:31.887675+0000) 2019-10-13T12:27:33.022 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:33.017+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.020652+0000) 2019-10-13T12:27:33.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:33.017+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.021097+0000) 2019-10-13T12:27:33.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:33.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.021821+0000) 2019-10-13T12:27:33.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:33.021+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.022027+0000) 2019-10-13T12:27:33.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:33.019+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.022947+0000) 2019-10-13T12:27:33.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:33.021+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.023132+0000) 2019-10-13T12:27:33.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:33.021+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.023432+0000) 2019-10-13T12:27:33.029 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:33.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:33.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:33.027+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:33.030589+0000) 2019-10-13T12:27:34.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:34.209+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.213166+0000) 2019-10-13T12:27:34.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:34.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:34.498686+0000) 2019-10-13T12:27:35.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:35.043+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:35.043844+0000) 2019-10-13T12:27:37.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:37.865+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:37.869441+0000) 2019-10-13T12:27:38.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:38.029+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.031675+0000) 2019-10-13T12:27:38.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:38.029+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.033033+0000) 2019-10-13T12:27:38.036 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:38.033+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.034768+0000) 2019-10-13T12:27:38.036 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:38.033+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.035098+0000) 2019-10-13T12:27:38.037 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:38.033+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.036598+0000) 2019-10-13T12:27:38.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:38.035+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.037071+0000) 2019-10-13T12:27:38.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:38.033+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.037465+0000) 2019-10-13T12:27:38.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:38.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:38.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:38.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.047628+0000) 2019-10-13T12:27:38.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:38.093+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.095639+0000) 2019-10-13T12:27:38.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:38.113+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.116650+0000) 2019-10-13T12:27:38.537 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:27:38.537 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:38.541 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:27:38.541 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:27:38.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:38.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:38.637229+0000) 2019-10-13T12:27:41.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:41.887+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:41.887877+0000) 2019-10-13T12:27:43.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:43.041+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.043321+0000) 2019-10-13T12:27:43.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:43.041+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.043980+0000) 2019-10-13T12:27:43.045 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:43.043+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.044653+0000) 2019-10-13T12:27:43.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:43.041+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.045367+0000) 2019-10-13T12:27:43.047 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:43.045+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.045837+0000) 2019-10-13T12:27:43.047 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:43.041+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.045687+0000) 2019-10-13T12:27:43.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:43.045+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.046485+0000) 2019-10-13T12:27:43.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:43.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:43.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:43.051+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:43.053259+0000) 2019-10-13T12:27:44.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:44.209+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.213405+0000) 2019-10-13T12:27:44.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:44.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:44.498890+0000) 2019-10-13T12:27:45.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:45.046+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:45.050453+0000) 2019-10-13T12:27:47.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:47.869+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:47.869642+0000) 2019-10-13T12:27:48.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:48.053+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.054667+0000) 2019-10-13T12:27:48.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:48.053+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.057404+0000) 2019-10-13T12:27:48.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:48.057+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.058152+0000) 2019-10-13T12:27:48.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:48.057+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.058931+0000) 2019-10-13T12:27:48.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:48.054+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.059548+0000) 2019-10-13T12:27:48.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:48.057+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.059937+0000) 2019-10-13T12:27:48.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:48.057+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.060559+0000) 2019-10-13T12:27:48.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:48.061+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:48.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:48.058+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.063697+0000) 2019-10-13T12:27:48.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:48.093+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.095853+0000) 2019-10-13T12:27:48.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:48.113+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.116830+0000) 2019-10-13T12:27:48.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:48.633+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:48.637442+0000) 2019-10-13T12:27:51.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:51.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:51.888081+0000) 2019-10-13T12:27:53.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:53.062+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.066268+0000) 2019-10-13T12:27:53.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:53.065+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.067589+0000) 2019-10-13T12:27:53.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:53.065+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.068714+0000) 2019-10-13T12:27:53.070 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:53.065+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.068955+0000) 2019-10-13T12:27:53.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:53.069+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.069823+0000) 2019-10-13T12:27:53.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:53.065+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.069579+0000) 2019-10-13T12:27:53.072 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:53.069+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.071307+0000) 2019-10-13T12:27:53.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:53.073+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:53.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:53.070+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:53.075838+0000) 2019-10-13T12:27:54.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:54.213+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.213609+0000) 2019-10-13T12:27:54.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:54.497+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:54.499087+0000) 2019-10-13T12:27:55.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:55.054+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:55.056988+0000) 2019-10-13T12:27:57.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:57.869+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:57.869875+0000) 2019-10-13T12:27:58.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:58.077+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.079099+0000) 2019-10-13T12:27:58.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:27:58.077+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.079404+0000) 2019-10-13T12:27:58.081 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:58.077+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.080040+0000) 2019-10-13T12:27:58.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:27:58.078+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.083402+0000) 2019-10-13T12:27:58.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:27:58.081+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.085473+0000) 2019-10-13T12:27:58.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:27:58.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:27:58.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:58.085+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.086845+0000) 2019-10-13T12:27:58.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:27:58.085+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.087246+0000) 2019-10-13T12:27:58.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:27:58.090+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.093328+0000) 2019-10-13T12:27:58.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:27:58.093+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.095978+0000) 2019-10-13T12:27:58.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:27:58.113+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.117002+0000) 2019-10-13T12:27:58.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:27:58.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:27:58.637654+0000) 2019-10-13T12:28:01.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:01.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:01.888316+0000) 2019-10-13T12:28:03.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:03.089+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.090387+0000) 2019-10-13T12:28:03.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:03.089+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.090670+0000) 2019-10-13T12:28:03.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:03.089+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.091303+0000) 2019-10-13T12:28:03.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:03.089+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.091846+0000) 2019-10-13T12:28:03.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:03.090+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.092751+0000) 2019-10-13T12:28:03.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:03.089+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.093030+0000) 2019-10-13T12:28:03.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:03.089+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.093612+0000) 2019-10-13T12:28:03.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:03.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:03.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:03.098+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:03.102837+0000) 2019-10-13T12:28:04.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:04.213+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.213814+0000) 2019-10-13T12:28:04.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:04.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:04.499286+0000) 2019-10-13T12:28:05.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:05.058+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:05.063453+0000) 2019-10-13T12:28:07.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:07.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:07.870091+0000) 2019-10-13T12:28:08.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:08.092+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.096134+0000) 2019-10-13T12:28:08.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:08.096+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.100851+0000) 2019-10-13T12:28:08.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:08.096+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.101527+0000) 2019-10-13T12:28:08.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:08.100+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.102572+0000) 2019-10-13T12:28:08.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:08.100+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.103217+0000) 2019-10-13T12:28:08.106 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:08.102+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.105174+0000) 2019-10-13T12:28:08.106 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:08.100+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.105119+0000) 2019-10-13T12:28:08.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:08.104+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.106768+0000) 2019-10-13T12:28:08.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:08.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161264 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:08.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:08.106+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.110115+0000) 2019-10-13T12:28:08.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:08.112+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.117138+0000) 2019-10-13T12:28:08.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:08.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:08.637832+0000) 2019-10-13T12:28:08.652 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:28:08.653 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:08.656 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:28:08.656 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:11.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:11.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:11.888513+0000) 2019-10-13T12:28:13.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:13.108+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.112157+0000) 2019-10-13T12:28:13.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:13.112+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.113814+0000) 2019-10-13T12:28:13.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:13.114+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.116283+0000) 2019-10-13T12:28:13.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:13.112+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.115548+0000) 2019-10-13T12:28:13.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:13.112+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.116311+0000) 2019-10-13T12:28:13.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:13.112+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.116987+0000) 2019-10-13T12:28:13.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:13.112+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.117460+0000) 2019-10-13T12:28:13.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:13.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:13.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:13.118+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:13.122155+0000) 2019-10-13T12:28:14.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:14.212+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.214013+0000) 2019-10-13T12:28:14.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:14.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:14.499511+0000) 2019-10-13T12:28:15.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:15.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:15.069892+0000) 2019-10-13T12:28:17.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:17.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:17.870290+0000) 2019-10-13T12:28:18.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:18.092+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.096331+0000) 2019-10-13T12:28:18.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:18.116+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.117305+0000) 2019-10-13T12:28:18.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:18.120+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.122688+0000) 2019-10-13T12:28:18.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:18.120+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.123994+0000) 2019-10-13T12:28:18.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:18.120+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.124930+0000) 2019-10-13T12:28:18.126 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:18.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.125412+0000) 2019-10-13T12:28:18.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:18.124+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.125964+0000) 2019-10-13T12:28:18.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:18.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.126543+0000) 2019-10-13T12:28:18.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:18.124+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.126407+0000) 2019-10-13T12:28:18.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:18.126+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.130799+0000) 2019-10-13T12:28:18.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:18.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161324 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:18.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:18.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:18.638013+0000) 2019-10-13T12:28:21.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:21.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:21.888727+0000) 2019-10-13T12:28:23.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:23.132+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.134196+0000) 2019-10-13T12:28:23.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:23.132+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.135675+0000) 2019-10-13T12:28:23.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:23.132+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.136395+0000) 2019-10-13T12:28:23.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:23.132+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.136608+0000) 2019-10-13T12:28:23.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:23.132+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.136516+0000) 2019-10-13T12:28:23.138 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:23.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.137129+0000) 2019-10-13T12:28:23.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:23.132+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.137307+0000) 2019-10-13T12:28:23.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:23.138+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:23.141144+0000) 2019-10-13T12:28:23.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:23.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:24.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:24.212+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.214216+0000) 2019-10-13T12:28:24.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:24.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:24.499729+0000) 2019-10-13T12:28:25.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:25.074+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:25.076336+0000) 2019-10-13T12:28:27.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:27.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:27.870493+0000) 2019-10-13T12:28:28.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:28.092+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.096525+0000) 2019-10-13T12:28:28.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:28.116+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.117571+0000) 2019-10-13T12:28:28.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:28.144+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.146546+0000) 2019-10-13T12:28:28.148 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:28.142+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.147598+0000) 2019-10-13T12:28:28.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:28.144+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.148657+0000) 2019-10-13T12:28:28.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:28.144+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.148365+0000) 2019-10-13T12:28:28.150 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:28.144+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.149371+0000) 2019-10-13T12:28:28.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:28.148+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.151062+0000) 2019-10-13T12:28:28.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:28.146+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.152019+0000) 2019-10-13T12:28:28.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:28.148+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.153200+0000) 2019-10-13T12:28:28.155 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:28.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:28.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:28.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:28.638197+0000) 2019-10-13T12:28:31.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:31.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:31.888921+0000) 2019-10-13T12:28:33.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:33.152+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.157609+0000) 2019-10-13T12:28:33.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:33.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.158034+0000) 2019-10-13T12:28:33.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:33.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.158936+0000) 2019-10-13T12:28:33.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:33.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.159336+0000) 2019-10-13T12:28:33.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:33.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.159897+0000) 2019-10-13T12:28:33.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:33.156+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.160260+0000) 2019-10-13T12:28:33.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:33.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.160583+0000) 2019-10-13T12:28:33.175 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:33.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:33.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:33.162+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:33.167678+0000) 2019-10-13T12:28:34.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:34.212+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.214428+0000) 2019-10-13T12:28:34.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:34.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:34.499941+0000) 2019-10-13T12:28:35.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:35.078+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:35.083032+0000) 2019-10-13T12:28:37.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:37.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:37.870694+0000) 2019-10-13T12:28:38.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:38.096+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.096723+0000) 2019-10-13T12:28:38.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:38.116+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.117796+0000) 2019-10-13T12:28:38.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:38.164+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.168266+0000) 2019-10-13T12:28:38.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:38.164+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.168755+0000) 2019-10-13T12:28:38.170 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:38.164+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.169247+0000) 2019-10-13T12:28:38.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:38.164+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.169604+0000) 2019-10-13T12:28:38.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:38.168+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.170217+0000) 2019-10-13T12:28:38.173 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:38.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.172176+0000) 2019-10-13T12:28:38.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:38.172+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.174171+0000) 2019-10-13T12:28:38.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:38.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:38.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:38.178+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.181017+0000) 2019-10-13T12:28:38.639 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:38.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:38.638383+0000) 2019-10-13T12:28:38.707 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:28:38.708 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:38.712 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:28:38.712 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:28:41.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:41.886+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:41.889147+0000) 2019-10-13T12:28:43.182 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:43.176+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.180241+0000) 2019-10-13T12:28:43.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:43.176+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.180506+0000) 2019-10-13T12:28:43.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:43.180+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.182466+0000) 2019-10-13T12:28:43.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:43.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.183273+0000) 2019-10-13T12:28:43.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:43.182+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.185340+0000) 2019-10-13T12:28:43.188 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:43.184+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.187217+0000) 2019-10-13T12:28:43.188 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:43.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161472 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:43.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:43.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.188487+0000) 2019-10-13T12:28:43.195 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:43.190+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:43.194350+0000) 2019-10-13T12:28:44.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:44.212+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.214623+0000) 2019-10-13T12:28:44.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:44.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:44.500136+0000) 2019-10-13T12:28:45.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:45.086+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:45.089494+0000) 2019-10-13T12:28:47.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:47.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:47.870897+0000) 2019-10-13T12:28:48.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:48.092+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.096919+0000) 2019-10-13T12:28:48.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:48.116+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.118024+0000) 2019-10-13T12:28:48.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:48.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.190697+0000) 2019-10-13T12:28:48.192 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:48.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.191356+0000) 2019-10-13T12:28:48.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:48.188+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.192318+0000) 2019-10-13T12:28:48.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:48.188+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.193197+0000) 2019-10-13T12:28:48.194 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:48.188+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.193586+0000) 2019-10-13T12:28:48.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:48.189+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.194583+0000) 2019-10-13T12:28:48.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:48.192+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.195449+0000) 2019-10-13T12:28:48.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:48.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161503 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:48.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:48.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.201968+0000) 2019-10-13T12:28:48.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:48.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:48.638581+0000) 2019-10-13T12:28:51.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:51.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:51.889348+0000) 2019-10-13T12:28:53.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:53.200+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.205107+0000) 2019-10-13T12:28:53.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:53.200+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.205471+0000) 2019-10-13T12:28:53.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:53.204+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.207568+0000) 2019-10-13T12:28:53.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:53.204+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.207811+0000) 2019-10-13T12:28:53.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:53.204+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.209697+0000) 2019-10-13T12:28:53.211 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:53.208+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.210101+0000) 2019-10-13T12:28:53.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:53.205+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.211209+0000) 2019-10-13T12:28:53.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:53.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:53.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:53.209+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:53.213648+0000) 2019-10-13T12:28:54.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:54.212+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.214823+0000) 2019-10-13T12:28:54.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:54.496+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:54.500326+0000) 2019-10-13T12:28:55.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:55.097+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:55.100569+0000) 2019-10-13T12:28:57.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:57.868+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:57.871126+0000) 2019-10-13T12:28:58.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:58.092+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.097147+0000) 2019-10-13T12:28:58.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:58.116+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.118220+0000) 2019-10-13T12:28:58.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:28:58.212+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.215482+0000) 2019-10-13T12:28:58.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:28:58.212+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.215981+0000) 2019-10-13T12:28:58.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:58.212+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.216740+0000) 2019-10-13T12:28:58.218 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:28:58.212+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.217020+0000) 2019-10-13T12:28:58.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:28:58.212+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.217448+0000) 2019-10-13T12:28:58.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:28:58.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.218009+0000) 2019-10-13T12:28:58.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:28:58.213+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.219457+0000) 2019-10-13T12:28:58.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:28:58.217+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.221936+0000) 2019-10-13T12:28:58.223 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:28:58.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:28:58.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:28:58.636+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:28:58.638791+0000) 2019-10-13T12:29:01.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:01.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:01.889559+0000) 2019-10-13T12:29:03.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:03.223+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.225999+0000) 2019-10-13T12:29:03.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:03.223+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.227174+0000) 2019-10-13T12:29:03.228 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:03.223+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.227504+0000) 2019-10-13T12:29:03.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:03.223+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.228081+0000) 2019-10-13T12:29:03.229 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:03.223+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.228363+0000) 2019-10-13T12:29:03.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:03.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.228957+0000) 2019-10-13T12:29:03.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:03.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.229684+0000) 2019-10-13T12:29:03.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:03.229+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:03.233469+0000) 2019-10-13T12:29:03.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:03.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161592 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:04.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:04.211+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.215013+0000) 2019-10-13T12:29:04.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:04.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:04.500537+0000) 2019-10-13T12:29:05.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:05.105+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:05.107233+0000) 2019-10-13T12:29:07.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:07.867+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:07.871337+0000) 2019-10-13T12:29:08.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:08.095+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.097346+0000) 2019-10-13T12:29:08.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:08.115+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.118416+0000) 2019-10-13T12:29:08.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:08.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.238704+0000) 2019-10-13T12:29:08.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:08.237+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.240457+0000) 2019-10-13T12:29:08.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:08.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.240896+0000) 2019-10-13T12:29:08.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:08.235+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.241462+0000) 2019-10-13T12:29:08.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:08.239+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.242391+0000) 2019-10-13T12:29:08.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:08.239+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.242730+0000) 2019-10-13T12:29:08.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:08.239+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.243175+0000) 2019-10-13T12:29:08.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:08.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:08.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:08.245+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.250451+0000) 2019-10-13T12:29:08.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:08.635+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:08.638967+0000) 2019-10-13T12:29:08.837 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:29:08.838 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:08.841 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:29:08.841 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:11.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:11.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:11.889790+0000) 2019-10-13T12:29:13.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:13.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.250290+0000) 2019-10-13T12:29:13.252 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:13.247+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.249954+0000) 2019-10-13T12:29:13.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:13.247+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.250685+0000) 2019-10-13T12:29:13.253 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:13.247+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.251550+0000) 2019-10-13T12:29:13.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:13.247+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.251847+0000) 2019-10-13T12:29:13.253 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:13.249+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.252626+0000) 2019-10-13T12:29:13.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:13.251+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.254327+0000) 2019-10-13T12:29:13.259 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:13.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:13.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:13.257+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:13.260476+0000) 2019-10-13T12:29:14.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:14.211+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.215198+0000) 2019-10-13T12:29:14.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:14.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:14.500738+0000) 2019-10-13T12:29:15.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:15.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:15.113744+0000) 2019-10-13T12:29:17.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:17.867+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:17.871531+0000) 2019-10-13T12:29:18.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:18.091+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.097577+0000) 2019-10-13T12:29:18.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:18.115+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.118649+0000) 2019-10-13T12:29:18.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:18.255+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.261453+0000) 2019-10-13T12:29:18.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:18.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.263391+0000) 2019-10-13T12:29:18.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:18.259+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.263459+0000) 2019-10-13T12:29:18.265 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:18.259+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.263886+0000) 2019-10-13T12:29:18.265 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:18.259+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.264227+0000) 2019-10-13T12:29:18.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:18.259+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.264599+0000) 2019-10-13T12:29:18.266 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:18.261+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.265217+0000) 2019-10-13T12:29:18.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:18.265+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.269166+0000) 2019-10-13T12:29:18.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:18.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161683 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:18.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:18.635+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:18.639144+0000) 2019-10-13T12:29:21.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:21.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:21.890005+0000) 2019-10-13T12:29:23.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:23.271+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.274224+0000) 2019-10-13T12:29:23.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:23.271+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.274175+0000) 2019-10-13T12:29:23.276 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:23.271+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.274712+0000) 2019-10-13T12:29:23.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:23.271+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.275089+0000) 2019-10-13T12:29:23.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:23.271+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.275352+0000) 2019-10-13T12:29:23.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:23.271+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.276237+0000) 2019-10-13T12:29:23.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:23.273+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.276704+0000) 2019-10-13T12:29:23.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:23.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:23.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:23.277+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:23.282043+0000) 2019-10-13T12:29:24.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:24.211+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.215381+0000) 2019-10-13T12:29:24.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:24.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:24.500935+0000) 2019-10-13T12:29:25.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:25.117+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:25.120332+0000) 2019-10-13T12:29:27.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:27.871+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:27.871738+0000) 2019-10-13T12:29:28.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:28.095+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.097778+0000) 2019-10-13T12:29:28.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:28.115+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.118856+0000) 2019-10-13T12:29:28.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:28.279+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.284991+0000) 2019-10-13T12:29:28.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:28.283+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.286425+0000) 2019-10-13T12:29:28.288 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:28.283+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.286828+0000) 2019-10-13T12:29:28.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:28.283+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.289287+0000) 2019-10-13T12:29:28.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:28.287+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.291376+0000) 2019-10-13T12:29:28.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:28.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:28.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:28.287+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.293227+0000) 2019-10-13T12:29:28.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:28.289+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.293761+0000) 2019-10-13T12:29:28.295 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:28.289+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.294092+0000) 2019-10-13T12:29:28.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:28.635+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:28.639338+0000) 2019-10-13T12:29:31.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:31.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:31.890190+0000) 2019-10-13T12:29:33.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:33.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.296056+0000) 2019-10-13T12:29:33.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:33.291+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.296612+0000) 2019-10-13T12:29:33.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:33.293+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.297750+0000) 2019-10-13T12:29:33.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:33.295+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.299327+0000) 2019-10-13T12:29:33.303 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:33.299+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.301996+0000) 2019-10-13T12:29:33.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:33.301+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.306177+0000) 2019-10-13T12:29:33.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:33.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.304688+0000) 2019-10-13T12:29:33.312 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:33.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:33.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:33.299+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:33.305380+0000) 2019-10-13T12:29:34.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:34.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.215566+0000) 2019-10-13T12:29:34.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:34.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:34.501188+0000) 2019-10-13T12:29:35.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:35.125+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:35.126673+0000) 2019-10-13T12:29:37.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:37.867+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:37.871933+0000) 2019-10-13T12:29:38.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:38.095+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.097995+0000) 2019-10-13T12:29:38.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:38.115+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.119057+0000) 2019-10-13T12:29:38.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:38.303+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.308521+0000) 2019-10-13T12:29:38.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:38.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.309426+0000) 2019-10-13T12:29:38.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:38.303+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.309658+0000) 2019-10-13T12:29:38.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:38.307+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.310450+0000) 2019-10-13T12:29:38.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:38.305+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.311099+0000) 2019-10-13T12:29:38.312 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:38.307+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.311216+0000) 2019-10-13T12:29:38.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:38.307+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.312165+0000) 2019-10-13T12:29:38.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:38.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:38.319 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:38.313+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.317985+0000) 2019-10-13T12:29:38.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:38.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:38.639520+0000) 2019-10-13T12:29:38.914 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:29:38.914 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:38.918 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:29:38.918 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:29:41.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:41.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:41.890402+0000) 2019-10-13T12:29:43.322 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:43.315+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.321443+0000) 2019-10-13T12:29:43.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:43.319+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.323568+0000) 2019-10-13T12:29:43.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:43.321+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.324285+0000) 2019-10-13T12:29:43.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:43.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.324762+0000) 2019-10-13T12:29:43.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:43.319+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.324963+0000) 2019-10-13T12:29:43.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:43.319+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.325366+0000) 2019-10-13T12:29:43.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:43.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.326216+0000) 2019-10-13T12:29:43.328 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:43.323+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:43.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:43.329+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:43.333793+0000) 2019-10-13T12:29:44.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:44.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.215754+0000) 2019-10-13T12:29:44.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:44.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:44.501401+0000) 2019-10-13T12:29:45.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:45.129+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:45.133092+0000) 2019-10-13T12:29:47.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:47.871+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:47.872130+0000) 2019-10-13T12:29:48.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:48.095+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.098155+0000) 2019-10-13T12:29:48.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:48.115+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.119209+0000) 2019-10-13T12:29:48.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:48.327+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.330805+0000) 2019-10-13T12:29:48.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:48.327+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.331240+0000) 2019-10-13T12:29:48.334 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:48.327+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.332971+0000) 2019-10-13T12:29:48.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:48.331+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.335746+0000) 2019-10-13T12:29:48.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:48.333+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.336383+0000) 2019-10-13T12:29:48.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:48.331+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.336446+0000) 2019-10-13T12:29:48.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:48.331+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.336915+0000) 2019-10-13T12:29:48.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:48.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:48.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:48.337+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.341350+0000) 2019-10-13T12:29:48.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:48.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:48.639696+0000) 2019-10-13T12:29:51.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:51.885+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:51.890599+0000) 2019-10-13T12:29:53.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:53.339+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.342057+0000) 2019-10-13T12:29:53.344 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:53.339+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.342968+0000) 2019-10-13T12:29:53.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:53.336+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.343727+0000) 2019-10-13T12:29:53.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:53.339+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.343953+0000) 2019-10-13T12:29:53.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:53.339+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.344832+0000) 2019-10-13T12:29:53.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:53.339+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.345638+0000) 2019-10-13T12:29:53.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:53.343+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.346755+0000) 2019-10-13T12:29:53.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:53.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:53.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:53.348+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:53.352482+0000) 2019-10-13T12:29:54.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:54.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.215945+0000) 2019-10-13T12:29:54.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:54.495+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:54.501602+0000) 2019-10-13T12:29:55.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:55.132+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:55.139569+0000) 2019-10-13T12:29:57.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:57.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:57.872329+0000) 2019-10-13T12:29:58.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:58.094+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.098307+0000) 2019-10-13T12:29:58.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:58.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.119418+0000) 2019-10-13T12:29:58.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:29:58.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.353538+0000) 2019-10-13T12:29:58.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:29:58.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.353896+0000) 2019-10-13T12:29:58.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:29:58.350+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.354493+0000) 2019-10-13T12:29:58.356 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:58.350+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.354879+0000) 2019-10-13T12:29:58.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:29:58.348+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.355808+0000) 2019-10-13T12:29:58.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:29:58.350+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.356243+0000) 2019-10-13T12:29:58.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:29:58.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.358342+0000) 2019-10-13T12:29:58.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:29:58.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:29:58.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:29:58.360+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.365439+0000) 2019-10-13T12:29:58.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:29:58.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:29:58.639909+0000) 2019-10-13T12:30:01.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:01.888+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:01.890797+0000) 2019-10-13T12:30:03.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:03.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.364560+0000) 2019-10-13T12:30:03.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:03.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.364885+0000) 2019-10-13T12:30:03.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:03.362+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.366358+0000) 2019-10-13T12:30:03.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:03.362+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.366684+0000) 2019-10-13T12:30:03.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:03.362+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.367171+0000) 2019-10-13T12:30:03.369 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:03.360+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.367830+0000) 2019-10-13T12:30:03.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:03.362+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.369572+0000) 2019-10-13T12:30:03.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:03.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:03.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:03.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:03.375831+0000) 2019-10-13T12:30:04.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:04.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.216156+0000) 2019-10-13T12:30:04.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:04.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:04.501832+0000) 2019-10-13T12:30:05.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:05.140+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:05.146221+0000) 2019-10-13T12:30:07.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:07.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:07.872559+0000) 2019-10-13T12:30:08.099 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:08.094+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.098525+0000) 2019-10-13T12:30:08.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:08.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.119654+0000) 2019-10-13T12:30:08.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:08.370+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.376152+0000) 2019-10-13T12:30:08.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:08.370+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.377002+0000) 2019-10-13T12:30:08.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:08.374+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.378756+0000) 2019-10-13T12:30:08.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:08.374+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.379230+0000) 2019-10-13T12:30:08.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:08.374+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.379450+0000) 2019-10-13T12:30:08.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:08.374+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.379828+0000) 2019-10-13T12:30:08.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:08.372+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.378555+0000) 2019-10-13T12:30:08.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:08.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:08.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:08.380+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.387058+0000) 2019-10-13T12:30:08.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:08.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:08.640124+0000) 2019-10-13T12:30:09.020 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:30:09.020 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:09.024 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:30:09.024 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:11.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:11.888+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:11.890999+0000) 2019-10-13T12:30:13.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:13.382+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.386189+0000) 2019-10-13T12:30:13.388 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:13.380+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.386655+0000) 2019-10-13T12:30:13.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:13.382+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.387671+0000) 2019-10-13T12:30:13.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:13.382+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.389521+0000) 2019-10-13T12:30:13.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:13.386+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.389847+0000) 2019-10-13T12:30:13.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:13.386+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.390234+0000) 2019-10-13T12:30:13.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:13.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.390899+0000) 2019-10-13T12:30:13.394 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:13.388+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:13.393741+0000) 2019-10-13T12:30:13.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:13.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:14.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:14.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.216349+0000) 2019-10-13T12:30:14.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:14.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:14.502067+0000) 2019-10-13T12:30:15.154 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:15.152+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:15.152773+0000) 2019-10-13T12:30:17.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:17.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:17.872759+0000) 2019-10-13T12:30:18.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:18.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.098723+0000) 2019-10-13T12:30:18.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:18.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.119882+0000) 2019-10-13T12:30:18.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:18.394+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.397953+0000) 2019-10-13T12:30:18.400 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:18.394+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.399332+0000) 2019-10-13T12:30:18.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:18.394+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.400190+0000) 2019-10-13T12:30:18.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:18.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.400585+0000) 2019-10-13T12:30:18.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:18.398+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.402487+0000) 2019-10-13T12:30:18.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:18.396+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.403585+0000) 2019-10-13T12:30:18.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:18.398+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.403609+0000) 2019-10-13T12:30:18.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:18.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:18.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:18.404+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.408721+0000) 2019-10-13T12:30:18.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:18.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:18.640299+0000) 2019-10-13T12:30:21.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:21.888+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:21.891194+0000) 2019-10-13T12:30:23.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:23.402+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.409222+0000) 2019-10-13T12:30:23.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:23.406+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.410536+0000) 2019-10-13T12:30:23.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:23.406+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.411054+0000) 2019-10-13T12:30:23.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:23.406+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.411491+0000) 2019-10-13T12:30:23.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:23.406+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.411209+0000) 2019-10-13T12:30:23.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:23.406+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.412025+0000) 2019-10-13T12:30:23.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:23.408+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.412658+0000) 2019-10-13T12:30:23.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:23.412+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:23.416877+0000) 2019-10-13T12:30:23.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:23.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:24.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:24.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.216546+0000) 2019-10-13T12:30:24.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:24.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:24.502270+0000) 2019-10-13T12:30:25.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:25.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:25.159285+0000) 2019-10-13T12:30:27.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:27.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:27.872961+0000) 2019-10-13T12:30:28.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:28.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.098926+0000) 2019-10-13T12:30:28.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:28.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.120125+0000) 2019-10-13T12:30:28.422 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:28.414+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.420985+0000) 2019-10-13T12:30:28.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:28.418+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.422025+0000) 2019-10-13T12:30:28.423 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:28.416+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.422889+0000) 2019-10-13T12:30:28.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:28.418+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.424067+0000) 2019-10-13T12:30:28.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:28.418+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.424408+0000) 2019-10-13T12:30:28.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:28.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.427075+0000) 2019-10-13T12:30:28.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:28.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.427054+0000) 2019-10-13T12:30:28.431 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:28.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:28.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:28.428+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.435757+0000) 2019-10-13T12:30:28.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:28.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:28.640476+0000) 2019-10-13T12:30:31.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:31.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:31.891391+0000) 2019-10-13T12:30:33.436 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:33.430+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.434813+0000) 2019-10-13T12:30:33.437 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:33.430+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.435520+0000) 2019-10-13T12:30:33.437 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:33.430+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.435750+0000) 2019-10-13T12:30:33.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:33.430+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.436603+0000) 2019-10-13T12:30:33.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:33.430+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.437356+0000) 2019-10-13T12:30:33.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:33.432+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.437947+0000) 2019-10-13T12:30:33.439 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:33.434+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.438158+0000) 2019-10-13T12:30:33.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:33.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162137 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:33.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:33.436+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:33.443637+0000) 2019-10-13T12:30:34.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:34.210+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.216733+0000) 2019-10-13T12:30:34.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:34.498+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:34.502459+0000) 2019-10-13T12:30:35.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:35.164+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:35.165839+0000) 2019-10-13T12:30:37.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:37.866+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:37.873178+0000) 2019-10-13T12:30:38.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:38.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.099152+0000) 2019-10-13T12:30:38.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:38.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.120326+0000) 2019-10-13T12:30:38.445 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:38.438+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.444194+0000) 2019-10-13T12:30:38.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:38.440+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.444616+0000) 2019-10-13T12:30:38.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:38.442+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.446110+0000) 2019-10-13T12:30:38.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:38.442+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.446290+0000) 2019-10-13T12:30:38.448 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:38.442+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.446722+0000) 2019-10-13T12:30:38.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:38.442+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.447226+0000) 2019-10-13T12:30:38.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:38.442+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.447519+0000) 2019-10-13T12:30:38.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:38.444+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.451437+0000) 2019-10-13T12:30:38.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:38.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:38.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:38.634+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:38.640649+0000) 2019-10-13T12:30:39.118 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:30:39.118 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:39.122 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:30:39.122 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:30:41.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:41.884+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:41.891617+0000) 2019-10-13T12:30:43.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:43.452+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.455512+0000) 2019-10-13T12:30:43.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:43.450+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.456080+0000) 2019-10-13T12:30:43.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:43.450+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.456711+0000) 2019-10-13T12:30:43.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:43.450+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.456843+0000) 2019-10-13T12:30:43.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:43.450+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.457225+0000) 2019-10-13T12:30:43.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:43.454+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.457831+0000) 2019-10-13T12:30:43.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:43.454+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.459129+0000) 2019-10-13T12:30:43.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:43.456+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:43.460757+0000) 2019-10-13T12:30:43.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:43.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:44.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:44.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.216918+0000) 2019-10-13T12:30:44.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:44.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:44.502660+0000) 2019-10-13T12:30:45.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:45.172+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:45.172387+0000) 2019-10-13T12:30:47.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:47.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:47.873374+0000) 2019-10-13T12:30:48.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:48.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.099358+0000) 2019-10-13T12:30:48.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:48.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.120524+0000) 2019-10-13T12:30:48.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:48.462+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.468977+0000) 2019-10-13T12:30:48.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:48.462+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.469646+0000) 2019-10-13T12:30:48.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:48.464+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.470424+0000) 2019-10-13T12:30:48.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:48.466+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.470920+0000) 2019-10-13T12:30:48.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:48.466+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.472233+0000) 2019-10-13T12:30:48.473 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:48.466+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.472291+0000) 2019-10-13T12:30:48.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:48.466+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.472874+0000) 2019-10-13T12:30:48.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:48.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:48.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:48.472+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.477076+0000) 2019-10-13T12:30:48.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:48.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:48.640827+0000) 2019-10-13T12:30:51.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:51.888+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:51.891773+0000) 2019-10-13T12:30:53.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:53.473+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.479707+0000) 2019-10-13T12:30:53.482 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:53.480+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.480371+0000) 2019-10-13T12:30:53.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:53.473+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.480398+0000) 2019-10-13T12:30:53.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:53.477+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.482695+0000) 2019-10-13T12:30:53.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:53.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.483304+0000) 2019-10-13T12:30:53.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:53.477+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.483353+0000) 2019-10-13T12:30:53.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:53.477+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.484672+0000) 2019-10-13T12:30:53.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:53.477+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:53.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:53.488+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:53.489298+0000) 2019-10-13T12:30:54.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:54.213+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:54.217123+0000) 2019-10-13T12:30:54.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:54.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:54.502859+0000) 2019-10-13T12:30:55.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:55.176+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:55.178863+0000) 2019-10-13T12:30:57.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:57.865+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:57.873591+0000) 2019-10-13T12:30:58.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:58.097+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.099554+0000) 2019-10-13T12:30:58.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:58.117+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.120719+0000) 2019-10-13T12:30:58.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:30:58.481+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.488984+0000) 2019-10-13T12:30:58.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:30:58.487+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.489611+0000) 2019-10-13T12:30:58.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:30:58.485+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.490127+0000) 2019-10-13T12:30:58.493 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:30:58.485+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.492325+0000) 2019-10-13T12:30:58.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:58.485+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.492426+0000) 2019-10-13T12:30:58.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:30:58.489+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.494275+0000) 2019-10-13T12:30:58.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:30:58.495+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.496304+0000) 2019-10-13T12:30:58.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:30:58.489+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:30:58.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:30:58.489+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.497645+0000) 2019-10-13T12:30:58.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:30:58.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:30:58.641020+0000) 2019-10-13T12:31:01.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:01.887+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:01.891968+0000) 2019-10-13T12:31:03.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:03.493+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.501190+0000) 2019-10-13T12:31:03.503 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:03.493+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.501550+0000) 2019-10-13T12:31:03.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:03.503+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.507311+0000) 2019-10-13T12:31:03.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:03.501+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.509349+0000) 2019-10-13T12:31:03.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:03.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:03.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:03.507+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.510098+0000) 2019-10-13T12:31:03.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:03.505+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.509893+0000) 2019-10-13T12:31:03.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:03.505+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.510276+0000) 2019-10-13T12:31:03.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:03.505+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:03.510429+0000) 2019-10-13T12:31:04.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:04.213+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:04.217345+0000) 2019-10-13T12:31:04.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:04.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:04.503088+0000) 2019-10-13T12:31:05.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:05.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:05.185229+0000) 2019-10-13T12:31:07.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:07.873+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:07.873822+0000) 2019-10-13T12:31:08.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:08.097+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.099763+0000) 2019-10-13T12:31:08.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:08.117+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.120934+0000) 2019-10-13T12:31:08.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:08.509+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.512763+0000) 2019-10-13T12:31:08.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:08.509+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.513125+0000) 2019-10-13T12:31:08.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:08.513+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.516040+0000) 2019-10-13T12:31:08.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:08.515+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.516406+0000) 2019-10-13T12:31:08.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:08.513+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.516558+0000) 2019-10-13T12:31:08.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:08.519+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.520406+0000) 2019-10-13T12:31:08.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:08.517+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.520773+0000) 2019-10-13T12:31:08.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:08.521+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.521999+0000) 2019-10-13T12:31:08.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:08.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162354 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:08.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:08.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:08.641228+0000) 2019-10-13T12:31:09.234 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:31:09.234 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:09.237 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:31:09.237 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:11.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:11.891+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:11.892158+0000) 2019-10-13T12:31:13.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:13.523+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.525019+0000) 2019-10-13T12:31:13.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:13.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.525962+0000) 2019-10-13T12:31:13.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:13.525+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.525821+0000) 2019-10-13T12:31:13.528 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:13.525+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.526426+0000) 2019-10-13T12:31:13.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:13.525+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.528409+0000) 2019-10-13T12:31:13.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:13.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.528651+0000) 2019-10-13T12:31:13.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:13.525+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.529236+0000) 2019-10-13T12:31:13.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:13.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:13.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:13.535+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:13.536688+0000) 2019-10-13T12:31:14.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:14.213+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:14.217556+0000) 2019-10-13T12:31:14.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:14.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:14.503280+0000) 2019-10-13T12:31:15.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:15.187+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:15.191789+0000) 2019-10-13T12:31:17.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:17.873+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:17.874025+0000) 2019-10-13T12:31:18.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:18.097+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.099993+0000) 2019-10-13T12:31:18.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:18.117+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.121165+0000) 2019-10-13T12:31:18.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:18.533+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.537083+0000) 2019-10-13T12:31:18.539 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:18.533+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.537614+0000) 2019-10-13T12:31:18.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:18.537+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.538483+0000) 2019-10-13T12:31:18.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:18.537+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.539213+0000) 2019-10-13T12:31:18.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:18.537+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.539501+0000) 2019-10-13T12:31:18.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:18.539+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.540306+0000) 2019-10-13T12:31:18.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:18.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.540377+0000) 2019-10-13T12:31:18.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:18.543+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.544813+0000) 2019-10-13T12:31:18.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:18.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:18.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:18.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:18.641401+0000) 2019-10-13T12:31:21.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:21.891+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:21.892365+0000) 2019-10-13T12:31:23.550 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:23.545+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.549015+0000) 2019-10-13T12:31:23.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:23.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.549604+0000) 2019-10-13T12:31:23.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:23.549+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.551423+0000) 2019-10-13T12:31:23.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:23.549+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.552716+0000) 2019-10-13T12:31:23.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:23.549+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.553417+0000) 2019-10-13T12:31:23.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:23.553+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.553850+0000) 2019-10-13T12:31:23.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:23.551+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.554314+0000) 2019-10-13T12:31:23.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:23.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:23.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:23.555+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:23.558974+0000) 2019-10-13T12:31:24.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:24.217+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:24.217817+0000) 2019-10-13T12:31:24.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:24.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:24.503472+0000) 2019-10-13T12:31:25.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:25.199+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:25.203242+0000) 2019-10-13T12:31:27.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:27.873+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:27.874225+0000) 2019-10-13T12:31:28.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:28.097+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.100192+0000) 2019-10-13T12:31:28.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:28.117+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.121365+0000) 2019-10-13T12:31:28.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:28.557+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.560547+0000) 2019-10-13T12:31:28.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:28.557+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.561161+0000) 2019-10-13T12:31:28.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:28.557+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.561582+0000) 2019-10-13T12:31:28.563 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:28.561+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.562577+0000) 2019-10-13T12:31:28.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:28.563+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.564655+0000) 2019-10-13T12:31:28.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:28.561+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.564805+0000) 2019-10-13T12:31:28.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:28.565+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.566750+0000) 2019-10-13T12:31:28.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:28.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:28.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:28.567+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.570658+0000) 2019-10-13T12:31:28.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:28.637+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:28.641593+0000) 2019-10-13T12:31:31.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:31.891+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:31.892575+0000) 2019-10-13T12:31:33.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:33.569+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.572228+0000) 2019-10-13T12:31:33.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:33.569+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.573194+0000) 2019-10-13T12:31:33.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:33.573+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.573786+0000) 2019-10-13T12:31:33.577 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:33.573+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.574906+0000) 2019-10-13T12:31:33.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:33.573+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.575046+0000) 2019-10-13T12:31:33.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:33.571+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.575468+0000) 2019-10-13T12:31:33.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:33.573+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.575833+0000) 2019-10-13T12:31:33.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:33.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:33.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:33.579+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:33.583007+0000) 2019-10-13T12:31:34.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:34.217+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:34.218005+0000) 2019-10-13T12:31:34.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:34.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:34.503652+0000) 2019-10-13T12:31:35.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:35.207+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:35.209659+0000) 2019-10-13T12:31:37.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:37.873+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:37.874445+0000) 2019-10-13T12:31:38.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:38.097+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.100386+0000) 2019-10-13T12:31:38.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:38.121+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.121602+0000) 2019-10-13T12:31:38.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:38.583+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.585126+0000) 2019-10-13T12:31:38.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:38.581+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.584896+0000) 2019-10-13T12:31:38.587 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:38.581+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.585514+0000) 2019-10-13T12:31:38.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:38.585+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.586832+0000) 2019-10-13T12:31:38.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:38.585+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.586571+0000) 2019-10-13T12:31:38.589 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:38.585+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.587968+0000) 2019-10-13T12:31:38.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:38.585+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.588603+0000) 2019-10-13T12:31:38.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:38.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:38.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:38.591+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.593383+0000) 2019-10-13T12:31:38.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:38.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:38.641816+0000) 2019-10-13T12:31:39.532 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:31:39.532 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:39.536 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:31:39.536 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:31:41.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:41.891+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:41.892772+0000) 2019-10-13T12:31:43.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:43.595+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.597797+0000) 2019-10-13T12:31:43.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:43.597+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.598194+0000) 2019-10-13T12:31:43.599 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:43.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.598444+0000) 2019-10-13T12:31:43.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:43.597+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.598969+0000) 2019-10-13T12:31:43.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:43.597+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.599728+0000) 2019-10-13T12:31:43.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:43.597+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.600394+0000) 2019-10-13T12:31:43.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:43.597+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.600797+0000) 2019-10-13T12:31:43.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:43.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:43.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:43.603+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:43.605195+0000) 2019-10-13T12:31:44.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:44.217+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:44.218193+0000) 2019-10-13T12:31:44.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:44.501+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:44.503833+0000) 2019-10-13T12:31:45.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:45.215+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:45.216127+0000) 2019-10-13T12:31:47.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:47.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:47.874680+0000) 2019-10-13T12:31:48.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:48.096+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.100548+0000) 2019-10-13T12:31:48.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:48.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.121829+0000) 2019-10-13T12:31:48.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:48.604+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.608991+0000) 2019-10-13T12:31:48.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:48.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.611457+0000) 2019-10-13T12:31:48.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:48.608+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.613046+0000) 2019-10-13T12:31:48.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:48.608+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.613553+0000) 2019-10-13T12:31:48.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:48.612+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.613975+0000) 2019-10-13T12:31:48.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:48.612+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.614353+0000) 2019-10-13T12:31:48.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:48.612+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.614666+0000) 2019-10-13T12:31:48.617 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:48.612+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:48.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:48.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.617501+0000) 2019-10-13T12:31:48.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:48.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:48.641988+0000) 2019-10-13T12:31:51.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:51.891+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:51.892976+0000) 2019-10-13T12:31:53.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:53.616+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.619294+0000) 2019-10-13T12:31:53.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:53.616+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.619761+0000) 2019-10-13T12:31:53.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:53.616+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.621241+0000) 2019-10-13T12:31:53.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:53.616+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.621577+0000) 2019-10-13T12:31:53.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:53.619+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.623765+0000) 2019-10-13T12:31:53.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:53.624+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.625981+0000) 2019-10-13T12:31:53.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:53.624+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.626508+0000) 2019-10-13T12:31:53.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:53.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:53.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:53.631+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:53.632609+0000) 2019-10-13T12:31:54.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:54.216+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:54.218351+0000) 2019-10-13T12:31:54.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:54.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:54.504025+0000) 2019-10-13T12:31:55.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:55.219+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:55.222589+0000) 2019-10-13T12:31:57.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:57.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:57.874874+0000) 2019-10-13T12:31:58.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:58.096+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.100751+0000) 2019-10-13T12:31:58.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:58.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.122052+0000) 2019-10-13T12:31:58.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:31:58.628+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.630511+0000) 2019-10-13T12:31:58.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:31:58.628+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.630842+0000) 2019-10-13T12:31:58.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:58.628+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.631600+0000) 2019-10-13T12:31:58.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:31:58.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.631913+0000) 2019-10-13T12:31:58.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:31:58.628+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.632287+0000) 2019-10-13T12:31:58.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:31:58.628+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.633368+0000) 2019-10-13T12:31:58.635 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:31:58.631+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.634157+0000) 2019-10-13T12:31:58.640 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:31:58.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:31:58.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:31:58.639+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.642169+0000) 2019-10-13T12:31:58.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:31:58.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:31:58.642115+0000) 2019-10-13T12:32:01.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:01.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:01.893179+0000) 2019-10-13T12:32:03.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:03.640+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.642420+0000) 2019-10-13T12:32:03.646 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:03.640+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.644990+0000) 2019-10-13T12:32:03.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:03.640+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.645312+0000) 2019-10-13T12:32:03.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:03.646+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.649308+0000) 2019-10-13T12:32:03.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:03.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.650595+0000) 2019-10-13T12:32:03.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:03.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:03.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:03.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.651506+0000) 2019-10-13T12:32:03.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:03.648+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.651240+0000) 2019-10-13T12:32:03.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:03.648+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:03.651285+0000) 2019-10-13T12:32:04.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:04.216+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:04.218562+0000) 2019-10-13T12:32:04.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:04.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:04.504197+0000) 2019-10-13T12:32:05.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:05.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:05.229057+0000) 2019-10-13T12:32:07.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:07.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:07.875107+0000) 2019-10-13T12:32:08.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:08.100+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.100945+0000) 2019-10-13T12:32:08.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:08.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.122249+0000) 2019-10-13T12:32:08.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:08.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.642282+0000) 2019-10-13T12:32:08.654 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:08.648+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.653497+0000) 2019-10-13T12:32:08.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:08.652+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.655081+0000) 2019-10-13T12:32:08.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:08.652+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.655526+0000) 2019-10-13T12:32:08.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:08.650+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.655961+0000) 2019-10-13T12:32:08.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:08.652+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.656071+0000) 2019-10-13T12:32:08.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:08.652+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.656451+0000) 2019-10-13T12:32:08.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:08.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.656790+0000) 2019-10-13T12:32:08.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:08.658+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:08.661198+0000) 2019-10-13T12:32:08.662 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:08.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:09.671 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:32:09.671 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:09.676 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:32:09.676 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:11.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:11.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:11.893392+0000) 2019-10-13T12:32:13.666 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:13.660+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.665379+0000) 2019-10-13T12:32:13.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:13.664+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.667549+0000) 2019-10-13T12:32:13.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:13.664+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.667966+0000) 2019-10-13T12:32:13.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:13.668+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.669821+0000) 2019-10-13T12:32:13.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:13.668+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.669968+0000) 2019-10-13T12:32:13.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:13.668+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.671625+0000) 2019-10-13T12:32:13.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:13.670+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.672100+0000) 2019-10-13T12:32:13.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:13.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:13.672265+0000) 2019-10-13T12:32:13.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:13.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:14.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:14.216+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:14.218757+0000) 2019-10-13T12:32:14.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:14.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:14.504403+0000) 2019-10-13T12:32:15.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:15.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:15.235629+0000) 2019-10-13T12:32:17.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:17.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:17.875336+0000) 2019-10-13T12:32:18.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:18.096+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.101142+0000) 2019-10-13T12:32:18.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:18.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.122440+0000) 2019-10-13T12:32:18.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:18.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.642480+0000) 2019-10-13T12:32:18.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:18.676+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.678065+0000) 2019-10-13T12:32:18.680 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:18.676+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.678603+0000) 2019-10-13T12:32:18.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:18.676+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.678761+0000) 2019-10-13T12:32:18.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:18.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.679887+0000) 2019-10-13T12:32:18.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:18.676+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.680980+0000) 2019-10-13T12:32:18.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:18.678+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.681720+0000) 2019-10-13T12:32:18.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:18.680+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.682967+0000) 2019-10-13T12:32:18.686 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:18.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162781 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:18.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:18.682+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:18.687574+0000) 2019-10-13T12:32:21.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:21.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:21.893624+0000) 2019-10-13T12:32:23.690 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:23.684+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.688659+0000) 2019-10-13T12:32:23.692 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:23.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.690987+0000) 2019-10-13T12:32:23.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:23.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.693270+0000) 2019-10-13T12:32:23.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:23.688+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.693698+0000) 2019-10-13T12:32:23.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:23.692+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.694618+0000) 2019-10-13T12:32:23.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:23.692+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.695138+0000) 2019-10-13T12:32:23.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:23.690+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.696059+0000) 2019-10-13T12:32:23.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:23.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:23.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:23.698+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:23.700661+0000) 2019-10-13T12:32:24.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:24.216+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:24.218975+0000) 2019-10-13T12:32:24.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:24.500+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:24.504610+0000) 2019-10-13T12:32:25.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:25.238+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:25.242231+0000) 2019-10-13T12:32:27.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:27.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:27.875528+0000) 2019-10-13T12:32:28.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:28.096+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.101335+0000) 2019-10-13T12:32:28.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:28.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.122668+0000) 2019-10-13T12:32:28.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:28.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.642675+0000) 2019-10-13T12:32:28.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:28.696+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.700564+0000) 2019-10-13T12:32:28.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:28.696+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.701177+0000) 2019-10-13T12:32:28.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:28.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.701795+0000) 2019-10-13T12:32:28.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:28.700+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.702505+0000) 2019-10-13T12:32:28.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:28.698+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.703320+0000) 2019-10-13T12:32:28.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:28.700+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.704764+0000) 2019-10-13T12:32:28.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:28.704+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.706834+0000) 2019-10-13T12:32:28.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:28.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:28.713 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:28.710+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:28.712346+0000) 2019-10-13T12:32:31.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:31.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:31.893813+0000) 2019-10-13T12:32:33.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:33.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.713803+0000) 2019-10-13T12:32:33.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:33.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.713362+0000) 2019-10-13T12:32:33.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:33.712+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.715285+0000) 2019-10-13T12:32:33.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:33.712+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.715833+0000) 2019-10-13T12:32:33.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:33.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.716331+0000) 2019-10-13T12:32:33.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:33.712+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.716474+0000) 2019-10-13T12:32:33.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:33.712+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.717413+0000) 2019-10-13T12:32:33.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:33.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:33.723 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:33.718+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:33.722500+0000) 2019-10-13T12:32:34.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:34.216+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:34.219145+0000) 2019-10-13T12:32:34.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:34.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:34.504818+0000) 2019-10-13T12:32:35.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:35.246+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:35.248171+0000) 2019-10-13T12:32:37.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:37.872+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:37.875746+0000) 2019-10-13T12:32:38.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:38.100+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.101535+0000) 2019-10-13T12:32:38.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:38.120+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.122829+0000) 2019-10-13T12:32:38.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:38.640+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.642941+0000) 2019-10-13T12:32:38.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:38.720+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.722851+0000) 2019-10-13T12:32:38.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:38.718+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.723512+0000) 2019-10-13T12:32:38.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:38.720+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.723684+0000) 2019-10-13T12:32:38.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:38.720+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.724322+0000) 2019-10-13T12:32:38.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:38.720+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.725014+0000) 2019-10-13T12:32:38.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:38.724+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.727518+0000) 2019-10-13T12:32:38.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:38.724+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.727924+0000) 2019-10-13T12:32:38.732 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:38.726+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:38.731818+0000) 2019-10-13T12:32:38.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:38.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:39.743 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:32:39.743 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:39.746 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:32:39.747 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:32:41.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:41.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:41.894006+0000) 2019-10-13T12:32:43.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:43.731+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.735633+0000) 2019-10-13T12:32:43.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:43.731+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.736361+0000) 2019-10-13T12:32:43.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:43.731+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.737582+0000) 2019-10-13T12:32:43.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:43.735+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.737776+0000) 2019-10-13T12:32:43.739 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:43.734+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.738441+0000) 2019-10-13T12:32:43.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:43.735+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.738504+0000) 2019-10-13T12:32:43.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:43.735+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.740025+0000) 2019-10-13T12:32:43.744 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:43.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:43.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:43.738+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:43.743825+0000) 2019-10-13T12:32:44.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:44.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:44.219322+0000) 2019-10-13T12:32:44.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:44.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:44.505009+0000) 2019-10-13T12:32:45.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:45.254+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:45.254841+0000) 2019-10-13T12:32:47.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:47.871+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:47.875971+0000) 2019-10-13T12:32:48.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:48.099+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.101735+0000) 2019-10-13T12:32:48.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:48.119+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.123029+0000) 2019-10-13T12:32:48.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:48.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.643114+0000) 2019-10-13T12:32:48.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:48.742+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.746746+0000) 2019-10-13T12:32:48.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:48.743+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.746918+0000) 2019-10-13T12:32:48.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:48.743+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.748588+0000) 2019-10-13T12:32:48.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:48.743+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.749163+0000) 2019-10-13T12:32:48.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:48.743+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.749699+0000) 2019-10-13T12:32:48.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:48.747+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.750432+0000) 2019-10-13T12:32:48.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:48.747+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.752345+0000) 2019-10-13T12:32:48.756 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:48.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:48.758 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:48.754+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:48.757391+0000) 2019-10-13T12:32:51.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:51.890+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:51.894200+0000) 2019-10-13T12:32:53.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:53.751+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.757381+0000) 2019-10-13T12:32:53.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:53.754+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.758309+0000) 2019-10-13T12:32:53.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:53.755+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.757861+0000) 2019-10-13T12:32:53.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:53.755+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.761171+0000) 2019-10-13T12:32:53.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:53.755+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.761456+0000) 2019-10-13T12:32:53.763 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:53.759+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.762798+0000) 2019-10-13T12:32:53.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:53.759+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.763992+0000) 2019-10-13T12:32:53.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:53.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:53.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:53.766+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:53.769660+0000) 2019-10-13T12:32:54.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:54.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:54.219501+0000) 2019-10-13T12:32:54.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:54.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:54.505187+0000) 2019-10-13T12:32:55.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:55.258+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:55.261474+0000) 2019-10-13T12:32:57.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:57.875+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:57.876177+0000) 2019-10-13T12:32:58.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:58.099+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.101917+0000) 2019-10-13T12:32:58.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:58.119+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.123221+0000) 2019-10-13T12:32:58.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:58.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.643309+0000) 2019-10-13T12:32:58.771 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:32:58.767+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.769938+0000) 2019-10-13T12:32:58.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:32:58.767+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.770260+0000) 2019-10-13T12:32:58.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:32:58.767+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.770615+0000) 2019-10-13T12:32:58.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:32:58.770+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.774063+0000) 2019-10-13T12:32:58.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:32:58.771+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.775779+0000) 2019-10-13T12:32:58.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:32:58.774+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.777826+0000) 2019-10-13T12:32:58.779 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:32:58.775+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.778217+0000) 2019-10-13T12:32:58.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:32:58.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:32:58.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:32:58.775+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:32:58.778674+0000) 2019-10-13T12:33:01.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:01.894+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:01.894393+0000) 2019-10-13T12:33:03.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:03.779+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.781917+0000) 2019-10-13T12:33:03.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:03.779+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.782007+0000) 2019-10-13T12:33:03.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:03.779+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.782493+0000) 2019-10-13T12:33:03.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:03.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.783331+0000) 2019-10-13T12:33:03.786 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:03.782+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.784465+0000) 2019-10-13T12:33:03.786 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:03.779+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.784488+0000) 2019-10-13T12:33:03.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:03.779+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.785038+0000) 2019-10-13T12:33:03.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:03.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:03.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:03.786+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:03.790759+0000) 2019-10-13T12:33:04.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:04.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:04.219716+0000) 2019-10-13T12:33:04.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:04.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:04.505361+0000) 2019-10-13T12:33:05.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:05.266+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:05.268127+0000) 2019-10-13T12:33:07.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:07.871+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:07.876380+0000) 2019-10-13T12:33:08.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:08.099+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.102119+0000) 2019-10-13T12:33:08.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:08.119+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.123420+0000) 2019-10-13T12:33:08.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:08.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.643504+0000) 2019-10-13T12:33:08.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:08.787+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.793329+0000) 2019-10-13T12:33:08.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:08.787+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.793505+0000) 2019-10-13T12:33:08.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:08.791+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.794180+0000) 2019-10-13T12:33:08.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:08.791+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.794463+0000) 2019-10-13T12:33:08.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:08.791+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.794696+0000) 2019-10-13T12:33:08.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:08.789+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.795585+0000) 2019-10-13T12:33:08.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:08.791+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.795726+0000) 2019-10-13T12:33:08.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:08.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:08.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:08.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:08.801100+0000) 2019-10-13T12:33:09.855 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:33:09.855 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:09.859 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:33:09.859 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:11.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:11.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:11.894617+0000) 2019-10-13T12:33:13.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:13.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.804800+0000) 2019-10-13T12:33:13.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:13.799+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.805146+0000) 2019-10-13T12:33:13.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:13.799+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.805603+0000) 2019-10-13T12:33:13.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:13.803+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.806513+0000) 2019-10-13T12:33:13.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:13.801+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.807515+0000) 2019-10-13T12:33:13.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:13.803+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.808039+0000) 2019-10-13T12:33:13.810 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:13.803+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.808568+0000) 2019-10-13T12:33:13.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:13.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:13.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:13.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:13.813144+0000) 2019-10-13T12:33:14.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:14.219+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:14.219955+0000) 2019-10-13T12:33:14.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:14.499+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:14.505563+0000) 2019-10-13T12:33:15.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:15.273+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:15.274457+0000) 2019-10-13T12:33:17.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:17.875+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:17.876614+0000) 2019-10-13T12:33:18.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:18.099+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.102311+0000) 2019-10-13T12:33:18.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:18.123+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.123615+0000) 2019-10-13T12:33:18.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:18.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.643710+0000) 2019-10-13T12:33:18.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:18.811+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.817622+0000) 2019-10-13T12:33:18.820 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:18.815+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.818033+0000) 2019-10-13T12:33:18.820 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:18.815+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.818656+0000) 2019-10-13T12:33:18.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:18.815+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.819113+0000) 2019-10-13T12:33:18.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:18.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.819639+0000) 2019-10-13T12:33:18.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:18.817+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.820696+0000) 2019-10-13T12:33:18.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:18.815+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.820413+0000) 2019-10-13T12:33:18.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:18.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:18.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:18.821+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:18.824765+0000) 2019-10-13T12:33:21.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:21.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:21.894816+0000) 2019-10-13T12:33:23.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:23.823+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.827513+0000) 2019-10-13T12:33:23.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:23.823+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.828036+0000) 2019-10-13T12:33:23.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:23.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.829496+0000) 2019-10-13T12:33:23.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:23.827+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.829880+0000) 2019-10-13T12:33:23.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:23.827+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.830208+0000) 2019-10-13T12:33:23.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:23.825+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.830956+0000) 2019-10-13T12:33:23.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:23.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.830626+0000) 2019-10-13T12:33:23.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:23.829+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:23.835098+0000) 2019-10-13T12:33:23.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:23.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:24.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:24.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:24.220136+0000) 2019-10-13T12:33:24.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:24.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:24.505735+0000) 2019-10-13T12:33:25.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:25.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:25.280995+0000) 2019-10-13T12:33:27.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:27.871+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:27.876840+0000) 2019-10-13T12:33:28.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:28.099+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.102527+0000) 2019-10-13T12:33:28.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:28.123+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.123810+0000) 2019-10-13T12:33:28.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:28.639+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.643930+0000) 2019-10-13T12:33:28.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:28.835+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.838711+0000) 2019-10-13T12:33:28.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:28.835+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.839560+0000) 2019-10-13T12:33:28.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:28.835+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.840091+0000) 2019-10-13T12:33:28.841 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:28.835+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.840556+0000) 2019-10-13T12:33:28.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:28.835+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.841414+0000) 2019-10-13T12:33:28.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:28.837+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.842567+0000) 2019-10-13T12:33:28.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:28.839+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.843203+0000) 2019-10-13T12:33:28.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:28.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:28.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:28.845+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:28.849580+0000) 2019-10-13T12:33:31.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:31.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:31.895028+0000) 2019-10-13T12:33:33.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:33.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.849844+0000) 2019-10-13T12:33:33.851 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:33.847+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.850212+0000) 2019-10-13T12:33:33.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:33.847+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.851222+0000) 2019-10-13T12:33:33.853 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:33.847+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.851463+0000) 2019-10-13T12:33:33.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:33.847+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.851865+0000) 2019-10-13T12:33:33.854 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:33.847+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.852738+0000) 2019-10-13T12:33:33.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:33.849+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.854118+0000) 2019-10-13T12:33:33.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:33.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:33.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:33.853+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:33.859031+0000) 2019-10-13T12:33:34.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:34.215+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:34.220321+0000) 2019-10-13T12:33:34.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:34.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:34.505982+0000) 2019-10-13T12:33:35.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:35.285+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:35.287527+0000) 2019-10-13T12:33:37.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:37.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:37.877053+0000) 2019-10-13T12:33:38.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:38.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.102750+0000) 2019-10-13T12:33:38.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:38.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.124020+0000) 2019-10-13T12:33:38.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:38.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.644168+0000) 2019-10-13T12:33:38.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:38.858+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.865121+0000) 2019-10-13T12:33:38.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:38.861+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.865533+0000) 2019-10-13T12:33:38.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:38.862+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.866247+0000) 2019-10-13T12:33:38.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:38.862+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.866756+0000) 2019-10-13T12:33:38.868 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:38.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.867375+0000) 2019-10-13T12:33:38.869 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:38.862+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.867635+0000) 2019-10-13T12:33:38.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:38.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:38.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:38.862+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.868864+0000) 2019-10-13T12:33:38.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:38.869+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:38.872617+0000) 2019-10-13T12:33:40.133 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:33:40.133 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:40.137 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:33:40.137 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:33:41.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:41.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:41.895257+0000) 2019-10-13T12:33:43.874 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:43.866+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.872368+0000) 2019-10-13T12:33:43.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:43.866+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.873256+0000) 2019-10-13T12:33:43.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:43.869+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.873936+0000) 2019-10-13T12:33:43.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:43.870+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.874114+0000) 2019-10-13T12:33:43.875 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:43.870+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.874449+0000) 2019-10-13T12:33:43.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:43.870+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.875365+0000) 2019-10-13T12:33:43.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:43.870+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.876150+0000) 2019-10-13T12:33:43.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:43.873+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:43.878446+0000) 2019-10-13T12:33:43.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:43.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:44.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:44.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:44.220520+0000) 2019-10-13T12:33:44.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:44.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:44.506194+0000) 2019-10-13T12:33:45.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:45.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:45.294058+0000) 2019-10-13T12:33:47.878 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:47.870+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:47.877252+0000) 2019-10-13T12:33:48.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:48.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.102961+0000) 2019-10-13T12:33:48.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:48.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.124247+0000) 2019-10-13T12:33:48.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:48.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.644346+0000) 2019-10-13T12:33:48.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:48.878+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.882472+0000) 2019-10-13T12:33:48.884 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:48.878+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.882873+0000) 2019-10-13T12:33:48.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:48.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.886248+0000) 2019-10-13T12:33:48.889 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:48.882+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.887920+0000) 2019-10-13T12:33:48.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:48.882+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.889468+0000) 2019-10-13T12:33:48.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:48.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:48.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:48.885+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.891675+0000) 2019-10-13T12:33:48.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:48.886+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.891963+0000) 2019-10-13T12:33:48.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:48.893+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:48.898745+0000) 2019-10-13T12:33:51.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:51.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:51.895410+0000) 2019-10-13T12:33:53.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:53.889+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.895331+0000) 2019-10-13T12:33:53.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:53.890+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.895715+0000) 2019-10-13T12:33:53.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:53.890+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.896207+0000) 2019-10-13T12:33:53.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:53.890+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.896437+0000) 2019-10-13T12:33:53.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:53.890+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.896795+0000) 2019-10-13T12:33:53.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:53.890+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.897203+0000) 2019-10-13T12:33:53.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:53.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.897731+0000) 2019-10-13T12:33:53.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:53.898+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:33:53.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:53.901+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:53.904481+0000) 2019-10-13T12:33:54.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:54.218+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:54.220716+0000) 2019-10-13T12:33:54.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:54.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:54.506381+0000) 2019-10-13T12:33:55.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:55.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:55.300641+0000) 2019-10-13T12:33:57.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:57.874+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:57.877454+0000) 2019-10-13T12:33:58.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:58.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.103162+0000) 2019-10-13T12:33:58.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:58.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.124446+0000) 2019-10-13T12:33:58.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:58.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.644539+0000) 2019-10-13T12:33:58.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:33:58.902+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.905898+0000) 2019-10-13T12:33:58.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:33:58.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.907350+0000) 2019-10-13T12:33:58.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:33:58.902+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.908121+0000) 2019-10-13T12:33:58.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:33:58.905+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.908526+0000) 2019-10-13T12:33:58.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:33:58.902+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.908474+0000) 2019-10-13T12:33:58.910 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:33:58.902+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.908825+0000) 2019-10-13T12:33:58.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:33:58.902+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.909244+0000) 2019-10-13T12:33:58.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:33:58.909+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:33:58.913697+0000) 2019-10-13T12:33:58.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:33:58.910+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:01.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:01.889+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:01.895634+0000) 2019-10-13T12:34:03.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:03.913+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.917487+0000) 2019-10-13T12:34:03.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:03.914+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.917783+0000) 2019-10-13T12:34:03.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:03.914+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.918952+0000) 2019-10-13T12:34:03.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:03.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.919383+0000) 2019-10-13T12:34:03.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:03.914+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.919762+0000) 2019-10-13T12:34:03.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:03.914+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.920424+0000) 2019-10-13T12:34:03.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:03.914+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.920731+0000) 2019-10-13T12:34:03.926 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:03.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163414 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:03.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:03.925+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:03.928197+0000) 2019-10-13T12:34:04.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:04.218+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:04.220905+0000) 2019-10-13T12:34:04.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:04.502+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:04.506556+0000) 2019-10-13T12:34:05.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:05.305+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:05.307093+0000) 2019-10-13T12:34:07.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:07.874+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:07.877677+0000) 2019-10-13T12:34:08.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:08.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.103359+0000) 2019-10-13T12:34:08.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:08.118+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.124656+0000) 2019-10-13T12:34:08.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:08.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.644730+0000) 2019-10-13T12:34:08.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:08.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.928158+0000) 2019-10-13T12:34:08.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:08.922+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.928812+0000) 2019-10-13T12:34:08.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:08.922+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.929404+0000) 2019-10-13T12:34:08.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:08.926+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.930831+0000) 2019-10-13T12:34:08.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:08.926+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.930982+0000) 2019-10-13T12:34:08.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:08.926+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.931808+0000) 2019-10-13T12:34:08.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:08.929+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.932438+0000) 2019-10-13T12:34:08.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:08.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163447 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:08.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:08.933+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:08.939540+0000) 2019-10-13T12:34:10.183 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:34:10.183 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:10.187 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:34:10.187 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:11.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:11.892+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:11.895847+0000) 2019-10-13T12:34:13.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:13.934+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.941089+0000) 2019-10-13T12:34:13.943 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:13.934+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.941505+0000) 2019-10-13T12:34:13.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:13.938+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.942200+0000) 2019-10-13T12:34:13.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:13.938+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.942963+0000) 2019-10-13T12:34:13.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:13.938+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.943326+0000) 2019-10-13T12:34:13.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:13.940+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.945243+0000) 2019-10-13T12:34:13.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:13.938+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.945277+0000) 2019-10-13T12:34:13.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:13.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:13.954 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:13.948+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:13.953022+0000) 2019-10-13T12:34:14.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:14.214+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:14.221092+0000) 2019-10-13T12:34:14.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:14.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:14.506741+0000) 2019-10-13T12:34:15.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:15.312+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:15.313700+0000) 2019-10-13T12:34:17.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:17.874+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:17.877931+0000) 2019-10-13T12:34:18.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:18.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.103553+0000) 2019-10-13T12:34:18.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:18.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.124873+0000) 2019-10-13T12:34:18.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:18.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.644920+0000) 2019-10-13T12:34:18.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:18.944+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.951558+0000) 2019-10-13T12:34:18.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:18.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.952999+0000) 2019-10-13T12:34:18.955 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:18.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.953632+0000) 2019-10-13T12:34:18.955 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:18.950+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.954188+0000) 2019-10-13T12:34:18.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:18.950+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.954834+0000) 2019-10-13T12:34:18.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:18.950+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.955210+0000) 2019-10-13T12:34:18.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:18.950+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.955615+0000) 2019-10-13T12:34:18.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:18.952+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:18.959652+0000) 2019-10-13T12:34:18.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:18.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:21.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:21.892+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:21.896062+0000) 2019-10-13T12:34:23.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:23.960+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.964290+0000) 2019-10-13T12:34:23.966 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:23.958+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.964944+0000) 2019-10-13T12:34:23.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:23.958+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.965203+0000) 2019-10-13T12:34:23.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:23.962+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.965776+0000) 2019-10-13T12:34:23.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:23.962+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.966356+0000) 2019-10-13T12:34:23.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:23.962+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.967065+0000) 2019-10-13T12:34:23.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:23.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.967261+0000) 2019-10-13T12:34:23.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:23.964+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:23.971293+0000) 2019-10-13T12:34:23.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:23.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163535 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:24.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:24.218+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:24.221266+0000) 2019-10-13T12:34:24.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:24.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:24.506955+0000) 2019-10-13T12:34:25.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:25.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:25.320292+0000) 2019-10-13T12:34:27.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:27.878+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:27.878204+0000) 2019-10-13T12:34:28.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:28.098+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.103749+0000) 2019-10-13T12:34:28.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:28.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.125069+0000) 2019-10-13T12:34:28.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:28.638+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.645129+0000) 2019-10-13T12:34:28.977 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:28.970+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.976265+0000) 2019-10-13T12:34:28.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:28.972+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.976794+0000) 2019-10-13T12:34:28.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:28.970+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.976998+0000) 2019-10-13T12:34:28.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:28.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.977613+0000) 2019-10-13T12:34:28.979 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:28.974+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.977714+0000) 2019-10-13T12:34:28.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:28.974+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.978784+0000) 2019-10-13T12:34:28.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:28.974+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.979391+0000) 2019-10-13T12:34:28.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:28.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:28.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:28.980+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:28.987512+0000) 2019-10-13T12:34:31.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:31.892+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:31.896266+0000) 2019-10-13T12:34:33.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:33.981+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.987466+0000) 2019-10-13T12:34:34.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:33.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.987619+0000) 2019-10-13T12:34:34.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:33.984+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.988466+0000) 2019-10-13T12:34:34.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:33.988+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.996064+0000) 2019-10-13T12:34:34.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:33.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.988836+0000) 2019-10-13T12:34:34.009 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:33.985+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.989977+0000) 2019-10-13T12:34:34.009 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:33.985+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.990244+0000) 2019-10-13T12:34:34.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:33.985+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:33.990928+0000) 2019-10-13T12:34:34.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:33.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163597 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:34.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:34.217+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:34.221472+0000) 2019-10-13T12:34:34.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:34.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:34.507157+0000) 2019-10-13T12:34:35.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:35.324+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:35.326796+0000) 2019-10-13T12:34:37.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:37.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:37.878408+0000) 2019-10-13T12:34:38.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:38.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.103950+0000) 2019-10-13T12:34:38.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:38.121+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.125279+0000) 2019-10-13T12:34:38.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:38.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.645366+0000) 2019-10-13T12:34:39.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:38.993+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:38.999879+0000) 2019-10-13T12:34:39.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:38.993+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.000553+0000) 2019-10-13T12:34:39.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:38.993+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.000795+0000) 2019-10-13T12:34:39.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:38.996+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.002103+0000) 2019-10-13T12:34:39.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:38.997+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.002229+0000) 2019-10-13T12:34:39.004 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:38.997+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.002494+0000) 2019-10-13T12:34:39.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:38.997+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.003594+0000) 2019-10-13T12:34:39.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:39.000+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:39.006589+0000) 2019-10-13T12:34:39.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:39.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:40.297 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:34:40.298 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:40.302 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:34:40.302 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:34:41.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:41.892+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:41.896461+0000) 2019-10-13T12:34:44.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:44.005+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.011773+0000) 2019-10-13T12:34:44.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:44.005+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.012149+0000) 2019-10-13T12:34:44.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:44.008+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.012698+0000) 2019-10-13T12:34:44.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:44.005+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.012890+0000) 2019-10-13T12:34:44.015 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:44.005+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.013446+0000) 2019-10-13T12:34:44.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:44.005+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.013533+0000) 2019-10-13T12:34:44.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:44.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.014160+0000) 2019-10-13T12:34:44.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:44.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:44.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:44.016+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.023718+0000) 2019-10-13T12:34:44.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:44.217+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.221671+0000) 2019-10-13T12:34:44.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:44.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:44.507336+0000) 2019-10-13T12:34:45.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:45.332+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:45.333593+0000) 2019-10-13T12:34:47.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:47.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:47.878624+0000) 2019-10-13T12:34:48.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:48.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:48.104158+0000) 2019-10-13T12:34:48.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:48.121+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:48.125479+0000) 2019-10-13T12:34:48.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:48.641+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:48.645567+0000) 2019-10-13T12:34:49.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:49.021+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.023294+0000) 2019-10-13T12:34:49.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:49.021+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.024511+0000) 2019-10-13T12:34:49.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:49.021+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.025394+0000) 2019-10-13T12:34:49.027 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:49.025+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.025929+0000) 2019-10-13T12:34:49.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:49.025+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.026366+0000) 2019-10-13T12:34:49.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:49.025+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.026253+0000) 2019-10-13T12:34:49.028 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:49.020+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.027450+0000) 2019-10-13T12:34:49.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:49.020+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:49.027697+0000) 2019-10-13T12:34:49.033 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:49.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163691 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:51.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:51.896+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:51.896659+0000) 2019-10-13T12:34:54.037 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:54.033+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.035339+0000) 2019-10-13T12:34:54.037 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:54.033+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.036102+0000) 2019-10-13T12:34:54.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:54.032+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.037215+0000) 2019-10-13T12:34:54.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:54.033+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.037125+0000) 2019-10-13T12:34:54.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:54.037+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.039013+0000) 2019-10-13T12:34:54.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:54.041+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.041991+0000) 2019-10-13T12:34:54.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:54.041+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.044244+0000) 2019-10-13T12:34:54.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:54.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:54.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:54.048+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.055607+0000) 2019-10-13T12:34:54.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:54.221+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.221859+0000) 2019-10-13T12:34:54.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:54.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:54.507511+0000) 2019-10-13T12:34:55.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:55.336+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:55.339710+0000) 2019-10-13T12:34:57.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:57.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:57.878825+0000) 2019-10-13T12:34:58.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:58.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:58.104357+0000) 2019-10-13T12:34:58.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:58.121+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:58.125637+0000) 2019-10-13T12:34:58.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:58.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:58.645780+0000) 2019-10-13T12:34:59.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:34:59.045+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.048464+0000) 2019-10-13T12:34:59.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:34:59.045+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.049009+0000) 2019-10-13T12:34:59.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:34:59.044+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.052003+0000) 2019-10-13T12:34:59.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:34:59.053+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.054191+0000) 2019-10-13T12:34:59.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:34:59.048+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.055718+0000) 2019-10-13T12:34:59.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:34:59.053+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.056167+0000) 2019-10-13T12:34:59.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:34:59.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:34:59.058 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:34:59.053+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.057500+0000) 2019-10-13T12:34:59.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:34:59.053+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:34:59.057615+0000) 2019-10-13T12:35:01.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:01.896+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:01.896897+0000) 2019-10-13T12:35:04.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:04.057+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.059282+0000) 2019-10-13T12:35:04.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:04.057+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.059648+0000) 2019-10-13T12:35:04.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:04.057+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.060865+0000) 2019-10-13T12:35:04.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:04.061+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.062126+0000) 2019-10-13T12:35:04.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:04.061+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.062255+0000) 2019-10-13T12:35:04.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:04.056+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.062801+0000) 2019-10-13T12:35:04.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:04.061+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.063308+0000) 2019-10-13T12:35:04.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:04.064+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.068970+0000) 2019-10-13T12:35:04.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:04.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:04.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:04.221+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.222044+0000) 2019-10-13T12:35:04.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:04.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:04.507699+0000) 2019-10-13T12:35:05.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:05.344+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:05.347368+0000) 2019-10-13T12:35:07.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:07.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:07.879043+0000) 2019-10-13T12:35:08.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:08.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:08.104551+0000) 2019-10-13T12:35:08.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:08.125+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:08.125855+0000) 2019-10-13T12:35:08.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:08.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:08.646006+0000) 2019-10-13T12:35:09.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:09.068+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.073084+0000) 2019-10-13T12:35:09.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:09.069+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.073159+0000) 2019-10-13T12:35:09.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:09.073+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.074050+0000) 2019-10-13T12:35:09.076 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:09.073+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.075650+0000) 2019-10-13T12:35:09.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:09.073+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.076853+0000) 2019-10-13T12:35:09.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:09.077+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.078413+0000) 2019-10-13T12:35:09.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:09.077+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.079563+0000) 2019-10-13T12:35:09.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:09.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:09.093 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:09.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:09.092024+0000) 2019-10-13T12:35:10.358 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:35:10.358 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:10.362 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:35:10.362 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:11.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:11.896+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:11.897135+0000) 2019-10-13T12:35:14.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:14.081+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.083836+0000) 2019-10-13T12:35:14.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:14.081+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.084251+0000) 2019-10-13T12:35:14.086 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:14.081+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.085145+0000) 2019-10-13T12:35:14.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:14.085+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.085979+0000) 2019-10-13T12:35:14.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:14.085+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.086461+0000) 2019-10-13T12:35:14.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:14.085+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.086841+0000) 2019-10-13T12:35:14.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:14.084+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.087613+0000) 2019-10-13T12:35:14.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:14.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:14.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:14.092+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.094008+0000) 2019-10-13T12:35:14.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:14.221+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.222260+0000) 2019-10-13T12:35:14.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:14.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:14.507894+0000) 2019-10-13T12:35:15.356 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:15.351+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:15.354689+0000) 2019-10-13T12:35:17.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:17.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:17.879276+0000) 2019-10-13T12:35:18.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:18.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:18.104757+0000) 2019-10-13T12:35:18.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:18.125+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:18.126064+0000) 2019-10-13T12:35:18.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:18.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:18.646201+0000) 2019-10-13T12:35:19.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:19.091+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.095858+0000) 2019-10-13T12:35:19.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:19.093+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.096273+0000) 2019-10-13T12:35:19.098 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:19.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.097428+0000) 2019-10-13T12:35:19.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:19.097+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.097821+0000) 2019-10-13T12:35:19.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:19.097+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.098126+0000) 2019-10-13T12:35:19.099 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:19.097+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.098437+0000) 2019-10-13T12:35:19.100 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:19.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.098819+0000) 2019-10-13T12:35:19.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:19.099+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:19.102954+0000) 2019-10-13T12:35:19.105 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:19.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:21.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:21.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:21.897358+0000) 2019-10-13T12:35:24.109 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:24.103+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.107909+0000) 2019-10-13T12:35:24.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:24.105+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.108098+0000) 2019-10-13T12:35:24.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:24.105+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.108540+0000) 2019-10-13T12:35:24.110 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:24.105+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.109627+0000) 2019-10-13T12:35:24.111 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:24.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.110203+0000) 2019-10-13T12:35:24.112 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:24.109+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.110695+0000) 2019-10-13T12:35:24.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:24.109+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.110987+0000) 2019-10-13T12:35:24.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:24.113+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:24.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:24.115+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.116424+0000) 2019-10-13T12:35:24.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:24.221+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.222483+0000) 2019-10-13T12:35:24.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:24.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:24.508072+0000) 2019-10-13T12:35:25.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:25.359+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:25.361286+0000) 2019-10-13T12:35:27.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:27.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:27.879475+0000) 2019-10-13T12:35:28.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:28.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:28.104980+0000) 2019-10-13T12:35:28.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:28.125+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:28.126265+0000) 2019-10-13T12:35:28.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:28.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:28.646398+0000) 2019-10-13T12:35:29.120 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:29.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.119391+0000) 2019-10-13T12:35:29.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:29.116+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.120148+0000) 2019-10-13T12:35:29.122 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:29.116+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.120707+0000) 2019-10-13T12:35:29.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:29.116+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.121460+0000) 2019-10-13T12:35:29.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:29.116+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.120438+0000) 2019-10-13T12:35:29.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:29.120+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.122223+0000) 2019-10-13T12:35:29.124 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:29.120+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.123420+0000) 2019-10-13T12:35:29.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:29.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:29.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:29.127+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:29.128705+0000) 2019-10-13T12:35:31.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:31.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:31.897565+0000) 2019-10-13T12:35:34.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:34.128+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.131480+0000) 2019-10-13T12:35:34.133 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:34.128+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.131801+0000) 2019-10-13T12:35:34.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:34.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.132673+0000) 2019-10-13T12:35:34.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:34.131+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.133123+0000) 2019-10-13T12:35:34.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:34.128+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.133254+0000) 2019-10-13T12:35:34.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:34.132+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.133922+0000) 2019-10-13T12:35:34.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:34.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.134154+0000) 2019-10-13T12:35:34.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:34.135+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.137859+0000) 2019-10-13T12:35:34.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:34.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:34.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:34.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.222635+0000) 2019-10-13T12:35:34.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:34.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:34.508264+0000) 2019-10-13T12:35:35.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:35.363+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:35.367867+0000) 2019-10-13T12:35:37.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:37.876+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:37.879672+0000) 2019-10-13T12:35:38.106 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:38.104+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:38.105241+0000) 2019-10-13T12:35:38.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:38.124+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:38.126463+0000) 2019-10-13T12:35:38.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:38.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:38.646549+0000) 2019-10-13T12:35:39.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:39.136+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.141583+0000) 2019-10-13T12:35:39.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:39.140+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.142556+0000) 2019-10-13T12:35:39.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:39.140+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.142557+0000) 2019-10-13T12:35:39.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:39.139+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.143235+0000) 2019-10-13T12:35:39.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:39.140+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.143744+0000) 2019-10-13T12:35:39.145 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:39.140+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.144233+0000) 2019-10-13T12:35:39.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:39.140+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.145488+0000) 2019-10-13T12:35:39.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:39.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:39.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:39.147+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:39.150000+0000) 2019-10-13T12:35:40.399 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:35:40.400 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:40.408 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:35:40.409 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:35:41.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:41.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:41.897764+0000) 2019-10-13T12:35:44.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:44.148+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.152741+0000) 2019-10-13T12:35:44.154 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:44.148+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.153022+0000) 2019-10-13T12:35:44.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:44.148+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.153587+0000) 2019-10-13T12:35:44.156 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:44.151+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.154922+0000) 2019-10-13T12:35:44.156 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:44.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.155420+0000) 2019-10-13T12:35:44.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:44.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.155916+0000) 2019-10-13T12:35:44.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:44.152+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.156126+0000) 2019-10-13T12:35:44.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:44.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164023 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:44.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:44.159+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.161654+0000) 2019-10-13T12:35:44.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:44.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.222846+0000) 2019-10-13T12:35:44.509 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:44.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:44.508439+0000) 2019-10-13T12:35:45.376 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:45.371+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:45.374694+0000) 2019-10-13T12:35:47.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:47.876+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:47.879894+0000) 2019-10-13T12:35:48.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:48.100+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:48.105472+0000) 2019-10-13T12:35:48.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:48.124+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:48.126696+0000) 2019-10-13T12:35:48.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:48.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:48.646803+0000) 2019-10-13T12:35:49.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:49.160+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.163956+0000) 2019-10-13T12:35:49.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:49.160+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.164504+0000) 2019-10-13T12:35:49.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:49.163+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.165316+0000) 2019-10-13T12:35:49.167 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:49.160+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.165418+0000) 2019-10-13T12:35:49.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:49.164+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.166259+0000) 2019-10-13T12:35:49.168 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:49.164+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.166765+0000) 2019-10-13T12:35:49.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:49.164+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.167560+0000) 2019-10-13T12:35:49.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:49.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:49.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:49.171+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:49.174280+0000) 2019-10-13T12:35:51.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:51.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:51.897979+0000) 2019-10-13T12:35:54.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:54.171+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.175705+0000) 2019-10-13T12:35:54.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:54.172+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.176632+0000) 2019-10-13T12:35:54.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:54.172+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.177066+0000) 2019-10-13T12:35:54.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:54.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.177345+0000) 2019-10-13T12:35:54.179 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:54.176+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.178164+0000) 2019-10-13T12:35:54.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:54.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.178608+0000) 2019-10-13T12:35:54.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:54.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.179309+0000) 2019-10-13T12:35:54.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:54.179+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.182786+0000) 2019-10-13T12:35:54.184 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:54.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:35:54.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:54.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.223025+0000) 2019-10-13T12:35:54.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:54.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:54.508626+0000) 2019-10-13T12:35:55.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:55.379+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:55.381164+0000) 2019-10-13T12:35:57.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:57.876+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:57.880088+0000) 2019-10-13T12:35:58.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:58.104+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:58.105671+0000) 2019-10-13T12:35:58.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:58.124+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:58.126854+0000) 2019-10-13T12:35:58.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:58.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:58.646999+0000) 2019-10-13T12:35:59.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:35:59.184+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.188523+0000) 2019-10-13T12:35:59.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:35:59.184+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.188982+0000) 2019-10-13T12:35:59.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:35:59.188+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.189977+0000) 2019-10-13T12:35:59.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:35:59.188+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.190146+0000) 2019-10-13T12:35:59.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:35:59.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.190557+0000) 2019-10-13T12:35:59.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:35:59.187+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.191143+0000) 2019-10-13T12:35:59.203 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:35:59.188+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.191308+0000) 2019-10-13T12:35:59.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:35:59.191+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:35:59.194630+0000) 2019-10-13T12:35:59.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:35:59.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:01.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:01.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:01.898133+0000) 2019-10-13T12:36:04.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:04.196+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.198098+0000) 2019-10-13T12:36:04.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:04.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.199452+0000) 2019-10-13T12:36:04.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:04.196+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.199694+0000) 2019-10-13T12:36:04.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:04.196+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.200214+0000) 2019-10-13T12:36:04.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:04.196+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.201367+0000) 2019-10-13T12:36:04.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:04.196+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.200608+0000) 2019-10-13T12:36:04.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:04.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.202317+0000) 2019-10-13T12:36:04.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:04.203+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.204918+0000) 2019-10-13T12:36:04.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:04.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:04.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:04.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.223218+0000) 2019-10-13T12:36:04.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:04.504+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:04.508808+0000) 2019-10-13T12:36:05.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:05.383+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:05.387735+0000) 2019-10-13T12:36:07.881 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:07.876+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:07.880312+0000) 2019-10-13T12:36:08.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:08.104+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:08.105947+0000) 2019-10-13T12:36:08.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:08.124+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:08.127056+0000) 2019-10-13T12:36:08.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:08.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:08.647192+0000) 2019-10-13T12:36:09.210 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:09.204+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.209141+0000) 2019-10-13T12:36:09.211 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:09.208+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.210439+0000) 2019-10-13T12:36:09.213 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:09.208+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.211706+0000) 2019-10-13T12:36:09.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:09.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.211989+0000) 2019-10-13T12:36:09.213 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:09.208+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.212429+0000) 2019-10-13T12:36:09.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:09.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.212961+0000) 2019-10-13T12:36:09.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:09.211+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.213693+0000) 2019-10-13T12:36:09.218 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:09.212+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164179 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:09.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:09.219+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:09.222254+0000) 2019-10-13T12:36:10.505 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:36:10.506 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:10.509 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:36:10.509 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:11.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:11.895+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:11.898346+0000) 2019-10-13T12:36:14.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:14.216+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.220331+0000) 2019-10-13T12:36:14.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:14.216+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.221116+0000) 2019-10-13T12:36:14.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:14.216+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.221441+0000) 2019-10-13T12:36:14.223 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:14.219+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.222087+0000) 2019-10-13T12:36:14.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:14.220+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.222589+0000) 2019-10-13T12:36:14.224 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:14.220+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.222717+0000) 2019-10-13T12:36:14.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:14.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.223360+0000) 2019-10-13T12:36:14.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:14.220+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.223956+0000) 2019-10-13T12:36:14.230 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:14.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:14.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:14.227+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.229720+0000) 2019-10-13T12:36:14.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:14.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:14.509047+0000) 2019-10-13T12:36:15.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:15.391+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:15.394369+0000) 2019-10-13T12:36:17.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:17.876+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:17.880541+0000) 2019-10-13T12:36:18.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:18.104+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:18.106139+0000) 2019-10-13T12:36:18.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:18.124+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:18.127275+0000) 2019-10-13T12:36:18.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:18.644+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:18.647394+0000) 2019-10-13T12:36:19.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:19.228+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.232325+0000) 2019-10-13T12:36:19.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:19.231+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.232985+0000) 2019-10-13T12:36:19.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:19.228+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.233365+0000) 2019-10-13T12:36:19.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:19.232+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.233815+0000) 2019-10-13T12:36:19.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:19.232+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.233954+0000) 2019-10-13T12:36:19.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:19.232+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.234631+0000) 2019-10-13T12:36:19.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:19.232+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.235799+0000) 2019-10-13T12:36:19.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:19.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:19.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:19.239+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:19.240824+0000) 2019-10-13T12:36:21.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:21.894+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:21.898549+0000) 2019-10-13T12:36:24.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:24.220+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.223522+0000) 2019-10-13T12:36:24.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:24.240+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.243312+0000) 2019-10-13T12:36:24.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:24.240+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.244115+0000) 2019-10-13T12:36:24.245 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:24.242+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.244589+0000) 2019-10-13T12:36:24.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:24.240+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.244947+0000) 2019-10-13T12:36:24.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:24.240+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.245062+0000) 2019-10-13T12:36:24.247 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:24.240+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.245329+0000) 2019-10-13T12:36:24.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:24.240+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.245662+0000) 2019-10-13T12:36:24.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:24.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:24.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:24.250+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.252842+0000) 2019-10-13T12:36:24.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:24.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:24.509223+0000) 2019-10-13T12:36:25.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:25.398+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:25.400884+0000) 2019-10-13T12:36:27.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:27.875+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:27.880767+0000) 2019-10-13T12:36:28.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:28.103+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:28.106322+0000) 2019-10-13T12:36:28.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:28.123+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:28.127475+0000) 2019-10-13T12:36:28.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:28.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:28.647646+0000) 2019-10-13T12:36:29.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:29.251+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.255606+0000) 2019-10-13T12:36:29.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:29.251+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.256063+0000) 2019-10-13T12:36:29.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:29.254+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.258208+0000) 2019-10-13T12:36:29.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:29.255+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.258282+0000) 2019-10-13T12:36:29.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:29.255+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.259042+0000) 2019-10-13T12:36:29.261 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:29.255+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.259412+0000) 2019-10-13T12:36:29.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:29.255+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.259564+0000) 2019-10-13T12:36:29.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:29.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:29.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:29.262+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:29.266576+0000) 2019-10-13T12:36:31.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:31.894+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:31.898781+0000) 2019-10-13T12:36:34.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:34.219+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.223729+0000) 2019-10-13T12:36:34.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:34.263+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.266713+0000) 2019-10-13T12:36:34.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:34.263+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.267351+0000) 2019-10-13T12:36:34.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:34.263+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.267868+0000) 2019-10-13T12:36:34.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:34.263+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.268488+0000) 2019-10-13T12:36:34.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:34.263+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.269119+0000) 2019-10-13T12:36:34.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:34.266+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.269953+0000) 2019-10-13T12:36:34.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:34.267+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.270130+0000) 2019-10-13T12:36:34.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:34.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:34.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:34.270+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.275990+0000) 2019-10-13T12:36:34.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:34.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:34.509398+0000) 2019-10-13T12:36:35.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:35.402+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:35.407605+0000) 2019-10-13T12:36:37.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:37.875+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:37.880963+0000) 2019-10-13T12:36:38.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:38.103+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:38.106538+0000) 2019-10-13T12:36:38.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:38.123+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:38.127663+0000) 2019-10-13T12:36:38.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:38.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:38.647854+0000) 2019-10-13T12:36:39.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:39.275+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.278342+0000) 2019-10-13T12:36:39.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:39.275+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.279277+0000) 2019-10-13T12:36:39.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:39.275+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.280168+0000) 2019-10-13T12:36:39.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:39.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.280431+0000) 2019-10-13T12:36:39.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:39.275+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.280823+0000) 2019-10-13T12:36:39.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:39.278+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.281390+0000) 2019-10-13T12:36:39.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:39.279+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.281730+0000) 2019-10-13T12:36:39.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:39.282+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:39.285381+0000) 2019-10-13T12:36:39.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:39.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:40.611 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:36:40.611 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:40.614 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:36:40.615 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:36:41.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:41.894+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:41.898975+0000) 2019-10-13T12:36:44.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:44.219+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.223956+0000) 2019-10-13T12:36:44.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:44.283+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.289460+0000) 2019-10-13T12:36:44.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:44.286+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.289963+0000) 2019-10-13T12:36:44.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:44.287+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.290180+0000) 2019-10-13T12:36:44.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:44.287+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.291917+0000) 2019-10-13T12:36:44.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:44.287+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.291622+0000) 2019-10-13T12:36:44.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:44.287+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.292322+0000) 2019-10-13T12:36:44.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:44.287+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.290964+0000) 2019-10-13T12:36:44.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:44.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:44.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:44.294+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.298721+0000) 2019-10-13T12:36:44.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:44.503+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:44.509574+0000) 2019-10-13T12:36:45.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:45.410+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:45.414415+0000) 2019-10-13T12:36:47.882 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:47.875+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:47.881171+0000) 2019-10-13T12:36:48.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:48.103+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:48.106743+0000) 2019-10-13T12:36:48.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:48.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:48.127866+0000) 2019-10-13T12:36:48.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:48.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:48.648018+0000) 2019-10-13T12:36:49.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:49.295+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.301573+0000) 2019-10-13T12:36:49.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:49.298+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.302071+0000) 2019-10-13T12:36:49.304 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:49.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.302561+0000) 2019-10-13T12:36:49.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:49.299+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.303186+0000) 2019-10-13T12:36:49.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:49.299+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.304199+0000) 2019-10-13T12:36:49.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:49.299+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.304524+0000) 2019-10-13T12:36:49.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:49.299+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.305228+0000) 2019-10-13T12:36:49.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:49.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:49.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:49.310+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:49.314588+0000) 2019-10-13T12:36:51.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:51.898+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:51.899188+0000) 2019-10-13T12:36:54.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:54.219+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.224189+0000) 2019-10-13T12:36:54.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:54.307+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.312554+0000) 2019-10-13T12:36:54.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:54.307+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.313198+0000) 2019-10-13T12:36:54.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:54.311+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.314208+0000) 2019-10-13T12:36:54.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:54.310+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.315011+0000) 2019-10-13T12:36:54.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:54.311+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.315523+0000) 2019-10-13T12:36:54.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:54.311+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.315988+0000) 2019-10-13T12:36:54.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:54.311+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.316178+0000) 2019-10-13T12:36:54.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:54.314+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.320035+0000) 2019-10-13T12:36:54.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:54.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:54.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:54.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:54.509728+0000) 2019-10-13T12:36:55.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:55.422+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:55.425630+0000) 2019-10-13T12:36:57.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:57.879+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:57.881369+0000) 2019-10-13T12:36:58.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:58.103+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:58.106945+0000) 2019-10-13T12:36:58.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:58.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:58.128086+0000) 2019-10-13T12:36:58.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:58.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:58.648218+0000) 2019-10-13T12:36:59.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:36:59.319+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.323767+0000) 2019-10-13T12:36:59.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:36:59.319+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.324406+0000) 2019-10-13T12:36:59.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:36:59.319+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.324833+0000) 2019-10-13T12:36:59.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:36:59.323+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.325841+0000) 2019-10-13T12:36:59.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:36:59.322+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.326910+0000) 2019-10-13T12:36:59.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:36:59.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.327585+0000) 2019-10-13T12:36:59.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:36:59.323+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.328090+0000) 2019-10-13T12:36:59.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:36:59.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164482 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:36:59.336 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:36:59.330+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:36:59.335568+0000) 2019-10-13T12:37:01.900 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:01.898+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:01.899419+0000) 2019-10-13T12:37:04.225 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:04.219+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.224385+0000) 2019-10-13T12:37:04.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:04.331+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.335261+0000) 2019-10-13T12:37:04.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:04.331+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.336209+0000) 2019-10-13T12:37:04.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:04.331+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.336847+0000) 2019-10-13T12:37:04.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:04.331+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.337163+0000) 2019-10-13T12:37:04.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:04.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.337938+0000) 2019-10-13T12:37:04.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:04.335+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.338811+0000) 2019-10-13T12:37:04.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:04.334+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.339581+0000) 2019-10-13T12:37:04.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:04.338+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.343275+0000) 2019-10-13T12:37:04.346 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:04.339+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:04.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:04.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:04.509934+0000) 2019-10-13T12:37:05.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:05.430+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:05.433097+0000) 2019-10-13T12:37:07.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:07.879+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:07.881567+0000) 2019-10-13T12:37:08.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:08.103+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:08.107169+0000) 2019-10-13T12:37:08.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:08.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:08.128286+0000) 2019-10-13T12:37:08.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:08.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:08.648417+0000) 2019-10-13T12:37:09.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:09.347+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.352724+0000) 2019-10-13T12:37:09.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:09.347+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.353079+0000) 2019-10-13T12:37:09.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:09.351+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.354339+0000) 2019-10-13T12:37:09.357 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:09.351+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.355531+0000) 2019-10-13T12:37:09.357 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:09.351+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:09.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:09.351+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.355876+0000) 2019-10-13T12:37:09.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:09.351+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.356312+0000) 2019-10-13T12:37:09.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:09.354+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.357625+0000) 2019-10-13T12:37:09.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:09.358+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:09.364061+0000) 2019-10-13T12:37:10.658 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:37:10.658 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:10.662 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:37:10.662 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:11.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:11.894+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:11.899610+0000) 2019-10-13T12:37:14.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:14.223+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.224591+0000) 2019-10-13T12:37:14.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:14.355+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.358615+0000) 2019-10-13T12:37:14.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:14.354+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.359931+0000) 2019-10-13T12:37:14.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:14.355+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.360049+0000) 2019-10-13T12:37:14.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:14.355+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.360547+0000) 2019-10-13T12:37:14.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:14.355+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.360844+0000) 2019-10-13T12:37:14.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:14.355+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.361420+0000) 2019-10-13T12:37:14.363 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:14.359+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.361971+0000) 2019-10-13T12:37:14.368 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:14.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:14.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:14.366+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.368846+0000) 2019-10-13T12:37:14.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:14.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:14.510063+0000) 2019-10-13T12:37:15.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:15.438+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:15.439564+0000) 2019-10-13T12:37:17.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:17.879+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:17.881790+0000) 2019-10-13T12:37:18.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:18.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:18.107378+0000) 2019-10-13T12:37:18.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:18.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:18.128514+0000) 2019-10-13T12:37:18.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:18.643+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:18.648615+0000) 2019-10-13T12:37:19.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:19.367+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.370890+0000) 2019-10-13T12:37:19.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:19.367+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.371419+0000) 2019-10-13T12:37:19.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:19.367+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.372483+0000) 2019-10-13T12:37:19.375 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:19.370+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.374409+0000) 2019-10-13T12:37:19.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:19.371+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.375241+0000) 2019-10-13T12:37:19.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:19.371+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.376212+0000) 2019-10-13T12:37:19.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:19.375+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.378133+0000) 2019-10-13T12:37:19.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:19.375+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:19.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:19.378+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:19.383593+0000) 2019-10-13T12:37:21.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:21.898+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:21.899812+0000) 2019-10-13T12:37:24.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:24.218+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.224792+0000) 2019-10-13T12:37:24.385 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:24.378+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.383643+0000) 2019-10-13T12:37:24.386 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:24.382+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.384144+0000) 2019-10-13T12:37:24.386 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:24.378+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.384981+0000) 2019-10-13T12:37:24.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:24.378+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.385520+0000) 2019-10-13T12:37:24.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:24.382+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.386098+0000) 2019-10-13T12:37:24.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:24.382+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.386533+0000) 2019-10-13T12:37:24.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:24.382+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.386989+0000) 2019-10-13T12:37:24.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:24.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:24.394 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:24.390+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.392955+0000) 2019-10-13T12:37:24.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:24.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:24.510241+0000) 2019-10-13T12:37:25.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:25.441+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:25.445749+0000) 2019-10-13T12:37:27.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:27.878+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:27.881988+0000) 2019-10-13T12:37:28.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:28.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:28.107575+0000) 2019-10-13T12:37:28.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:28.126+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:28.128730+0000) 2019-10-13T12:37:28.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:28.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:28.648806+0000) 2019-10-13T12:37:29.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:29.390+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.394401+0000) 2019-10-13T12:37:29.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:29.390+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.396628+0000) 2019-10-13T12:37:29.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:29.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.397852+0000) 2019-10-13T12:37:29.399 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:29.394+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.398351+0000) 2019-10-13T12:37:29.400 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:29.393+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.398866+0000) 2019-10-13T12:37:29.400 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:29.394+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.398938+0000) 2019-10-13T12:37:29.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:29.394+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.399267+0000) 2019-10-13T12:37:29.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:29.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:29.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:29.397+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:29.403343+0000) 2019-10-13T12:37:31.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:31.893+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:31.900009+0000) 2019-10-13T12:37:34.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:34.222+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.224993+0000) 2019-10-13T12:37:34.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:34.398+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.404524+0000) 2019-10-13T12:37:34.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:34.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.405989+0000) 2019-10-13T12:37:34.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:34.402+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.406253+0000) 2019-10-13T12:37:34.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:34.402+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.406612+0000) 2019-10-13T12:37:34.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:34.402+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.407100+0000) 2019-10-13T12:37:34.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:34.402+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.407526+0000) 2019-10-13T12:37:34.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:34.405+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.408305+0000) 2019-10-13T12:37:34.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:34.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164694 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:34.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:34.409+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.415234+0000) 2019-10-13T12:37:34.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:34.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:34.510422+0000) 2019-10-13T12:37:35.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:35.453+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:35.456949+0000) 2019-10-13T12:37:37.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:37.878+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:37.882203+0000) 2019-10-13T12:37:38.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:38.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:38.107771+0000) 2019-10-13T12:37:38.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:38.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:38.128881+0000) 2019-10-13T12:37:38.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:38.642+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:38.648997+0000) 2019-10-13T12:37:39.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:39.410+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.416472+0000) 2019-10-13T12:37:39.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:39.410+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.417393+0000) 2019-10-13T12:37:39.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:39.413+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.418308+0000) 2019-10-13T12:37:39.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:39.414+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.418652+0000) 2019-10-13T12:37:39.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:39.414+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.420257+0000) 2019-10-13T12:37:39.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:39.414+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.420560+0000) 2019-10-13T12:37:39.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:39.414+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.421166+0000) 2019-10-13T12:37:39.426 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:39.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164727 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:39.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:39.425+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:39.428328+0000) 2019-10-13T12:37:40.731 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:37:40.732 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:40.736 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:37:40.736 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:37:41.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:41.897+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:41.900252+0000) 2019-10-13T12:37:44.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:44.222+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.225191+0000) 2019-10-13T12:37:44.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:44.425+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.428866+0000) 2019-10-13T12:37:44.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:44.426+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.430791+0000) 2019-10-13T12:37:44.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:44.426+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.432564+0000) 2019-10-13T12:37:44.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:44.426+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.433473+0000) 2019-10-13T12:37:44.435 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:44.426+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.433494+0000) 2019-10-13T12:37:44.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:44.430+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.434026+0000) 2019-10-13T12:37:44.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:44.430+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.434798+0000) 2019-10-13T12:37:44.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:44.433+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.436608+0000) 2019-10-13T12:37:44.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:44.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164754 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:44.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:44.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:44.510636+0000) 2019-10-13T12:37:45.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:45.461+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:45.463416+0000) 2019-10-13T12:37:47.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:47.878+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:47.882421+0000) 2019-10-13T12:37:48.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:48.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.107988+0000) 2019-10-13T12:37:48.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:48.126+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.129078+0000) 2019-10-13T12:37:48.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:48.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:48.649194+0000) 2019-10-13T12:37:49.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:49.434+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.440638+0000) 2019-10-13T12:37:49.443 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:49.434+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.441219+0000) 2019-10-13T12:37:49.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:49.434+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.441611+0000) 2019-10-13T12:37:49.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:49.438+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.441972+0000) 2019-10-13T12:37:49.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:49.438+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.442749+0000) 2019-10-13T12:37:49.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:49.437+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.443472+0000) 2019-10-13T12:37:49.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:49.438+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.443495+0000) 2019-10-13T12:37:49.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:49.441+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:49.446386+0000) 2019-10-13T12:37:49.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:49.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:51.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:51.897+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:51.900447+0000) 2019-10-13T12:37:54.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:54.222+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.225388+0000) 2019-10-13T12:37:54.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:54.449+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.452235+0000) 2019-10-13T12:37:54.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:54.446+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.452982+0000) 2019-10-13T12:37:54.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:54.446+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.453488+0000) 2019-10-13T12:37:54.455 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:54.450+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.454049+0000) 2019-10-13T12:37:54.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:54.450+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.454486+0000) 2019-10-13T12:37:54.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:54.450+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.455224+0000) 2019-10-13T12:37:54.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:54.446+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.453594+0000) 2019-10-13T12:37:54.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:54.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:54.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:54.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.462207+0000) 2019-10-13T12:37:54.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:54.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:54.510807+0000) 2019-10-13T12:37:55.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:55.469+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:55.470088+0000) 2019-10-13T12:37:57.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:57.878+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:57.882629+0000) 2019-10-13T12:37:58.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:58.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.108183+0000) 2019-10-13T12:37:58.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:58.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.129232+0000) 2019-10-13T12:37:58.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:58.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:58.649412+0000) 2019-10-13T12:37:59.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:37:59.458+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.462845+0000) 2019-10-13T12:37:59.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:37:59.458+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.463940+0000) 2019-10-13T12:37:59.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:37:59.458+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.464344+0000) 2019-10-13T12:37:59.466 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:37:59.458+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.464841+0000) 2019-10-13T12:37:59.466 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:37:59.461+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.465546+0000) 2019-10-13T12:37:59.467 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:37:59.458+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.465596+0000) 2019-10-13T12:37:59.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:37:59.462+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.466841+0000) 2019-10-13T12:37:59.473 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:37:59.466+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164848 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:37:59.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:37:59.469+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:37:59.472726+0000) 2019-10-13T12:38:01.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:01.897+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:01.900647+0000) 2019-10-13T12:38:04.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:04.222+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.225595+0000) 2019-10-13T12:38:04.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:04.469+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.475566+0000) 2019-10-13T12:38:04.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:04.470+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.475909+0000) 2019-10-13T12:38:04.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:04.470+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.476572+0000) 2019-10-13T12:38:04.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:04.474+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.479271+0000) 2019-10-13T12:38:04.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:04.474+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.479669+0000) 2019-10-13T12:38:04.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:04.474+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.480104+0000) 2019-10-13T12:38:04.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:04.474+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.480589+0000) 2019-10-13T12:38:04.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:04.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:04.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:04.481+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.486507+0000) 2019-10-13T12:38:04.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:04.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:04.510987+0000) 2019-10-13T12:38:05.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:05.473+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:05.476604+0000) 2019-10-13T12:38:07.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:07.882+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:07.882832+0000) 2019-10-13T12:38:08.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:08.102+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.108381+0000) 2019-10-13T12:38:08.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:08.122+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.129434+0000) 2019-10-13T12:38:08.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:08.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:08.649626+0000) 2019-10-13T12:38:09.488 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:09.482+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.486192+0000) 2019-10-13T12:38:09.488 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:09.482+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.486618+0000) 2019-10-13T12:38:09.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:09.482+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.486894+0000) 2019-10-13T12:38:09.489 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:09.481+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.487729+0000) 2019-10-13T12:38:09.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:09.482+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.488370+0000) 2019-10-13T12:38:09.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:09.482+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.488602+0000) 2019-10-13T12:38:09.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:09.482+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.489112+0000) 2019-10-13T12:38:09.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:09.489+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:09.492423+0000) 2019-10-13T12:38:09.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:09.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164910 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:10.843 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:38:10.843 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:10.847 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:38:10.847 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:11.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:11.897+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:11.900873+0000) 2019-10-13T12:38:14.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:14.222+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.225832+0000) 2019-10-13T12:38:14.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:14.493+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.498129+0000) 2019-10-13T12:38:14.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:14.494+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.498750+0000) 2019-10-13T12:38:14.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:14.494+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.500188+0000) 2019-10-13T12:38:14.502 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:14.494+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.500092+0000) 2019-10-13T12:38:14.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:14.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.501192+0000) 2019-10-13T12:38:14.503 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:14.494+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.501694+0000) 2019-10-13T12:38:14.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:14.498+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.502723+0000) 2019-10-13T12:38:14.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:14.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:14.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:14.505+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.508757+0000) 2019-10-13T12:38:14.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:14.506+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:14.511133+0000) 2019-10-13T12:38:15.484 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:15.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:15.483077+0000) 2019-10-13T12:38:17.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:17.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:17.883093+0000) 2019-10-13T12:38:18.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:18.101+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.108576+0000) 2019-10-13T12:38:18.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:18.125+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.129648+0000) 2019-10-13T12:38:18.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:18.645+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:18.649845+0000) 2019-10-13T12:38:19.511 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:19.501+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.509308+0000) 2019-10-13T12:38:19.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:19.505+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.510577+0000) 2019-10-13T12:38:19.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:19.509+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.512919+0000) 2019-10-13T12:38:19.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:19.509+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.514601+0000) 2019-10-13T12:38:19.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:19.509+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.515524+0000) 2019-10-13T12:38:19.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:19.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.515539+0000) 2019-10-13T12:38:19.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:19.509+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.516008+0000) 2019-10-13T12:38:19.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:19.513+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:19.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:19.517+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:19.523349+0000) 2019-10-13T12:38:21.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:21.897+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:21.901085+0000) 2019-10-13T12:38:24.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:24.225+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.226041+0000) 2019-10-13T12:38:24.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:24.505+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.511271+0000) 2019-10-13T12:38:24.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:24.517+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.521230+0000) 2019-10-13T12:38:24.523 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:24.517+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.521904+0000) 2019-10-13T12:38:24.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:24.517+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.522675+0000) 2019-10-13T12:38:24.524 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:24.517+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.523145+0000) 2019-10-13T12:38:24.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:24.517+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.523465+0000) 2019-10-13T12:38:24.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:24.517+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.523895+0000) 2019-10-13T12:38:24.525 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:24.517+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.524396+0000) 2019-10-13T12:38:24.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:24.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:24.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:24.525+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:24.531052+0000) 2019-10-13T12:38:25.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:25.485+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:25.489600+0000) 2019-10-13T12:38:27.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:27.877+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:27.883309+0000) 2019-10-13T12:38:28.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:28.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.108778+0000) 2019-10-13T12:38:28.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:28.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.129921+0000) 2019-10-13T12:38:28.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:28.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:28.650033+0000) 2019-10-13T12:38:29.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:29.525+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.532155+0000) 2019-10-13T12:38:29.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:29.525+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.532546+0000) 2019-10-13T12:38:29.534 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:29.528+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.533319+0000) 2019-10-13T12:38:29.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:29.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.533610+0000) 2019-10-13T12:38:29.536 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:29.529+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.534260+0000) 2019-10-13T12:38:29.536 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:29.529+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.534983+0000) 2019-10-13T12:38:29.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:29.529+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.535750+0000) 2019-10-13T12:38:29.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:29.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:29.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:29.536+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:29.541892+0000) 2019-10-13T12:38:31.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:31.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:31.901281+0000) 2019-10-13T12:38:34.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:34.225+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.226273+0000) 2019-10-13T12:38:34.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:34.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.511481+0000) 2019-10-13T12:38:34.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:34.541+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.544191+0000) 2019-10-13T12:38:34.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:34.545+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.546395+0000) 2019-10-13T12:38:34.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:34.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.546357+0000) 2019-10-13T12:38:34.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:34.540+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.547233+0000) 2019-10-13T12:38:34.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:34.545+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.547416+0000) 2019-10-13T12:38:34.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:34.545+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.548208+0000) 2019-10-13T12:38:34.549 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:34.545+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.548478+0000) 2019-10-13T12:38:34.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:34.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:34.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:34.548+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:34.554642+0000) 2019-10-13T12:38:35.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:35.488+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:35.496052+0000) 2019-10-13T12:38:37.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:37.881+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:37.883538+0000) 2019-10-13T12:38:38.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:38.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.109007+0000) 2019-10-13T12:38:38.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:38.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.130115+0000) 2019-10-13T12:38:38.651 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:38.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:38.650230+0000) 2019-10-13T12:38:39.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:39.548+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.555797+0000) 2019-10-13T12:38:39.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:39.553+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.556408+0000) 2019-10-13T12:38:39.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:39.553+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.556920+0000) 2019-10-13T12:38:39.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:39.557+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.559117+0000) 2019-10-13T12:38:39.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:39.557+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.560925+0000) 2019-10-13T12:38:39.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:39.561+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.563185+0000) 2019-10-13T12:38:39.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:39.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:39.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:39.560+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.564975+0000) 2019-10-13T12:38:39.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:39.561+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:39.564128+0000) 2019-10-13T12:38:41.073 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:38:41.073 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:41.077 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:38:41.077 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:38:41.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:41.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:41.901473+0000) 2019-10-13T12:38:44.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:44.225+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.226472+0000) 2019-10-13T12:38:44.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:44.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.511680+0000) 2019-10-13T12:38:44.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:44.565+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.567318+0000) 2019-10-13T12:38:44.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:44.565+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.568454+0000) 2019-10-13T12:38:44.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:44.565+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.568855+0000) 2019-10-13T12:38:44.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:44.565+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.568831+0000) 2019-10-13T12:38:44.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:44.565+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.569686+0000) 2019-10-13T12:38:44.572 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:44.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.570685+0000) 2019-10-13T12:38:44.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:44.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.570390+0000) 2019-10-13T12:38:44.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:44.568+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:44.575282+0000) 2019-10-13T12:38:44.576 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:44.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:45.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:45.504+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:45.506816+0000) 2019-10-13T12:38:47.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:47.881+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:47.883736+0000) 2019-10-13T12:38:48.110 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:48.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.109204+0000) 2019-10-13T12:38:48.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:48.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.130310+0000) 2019-10-13T12:38:48.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:48.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:48.650424+0000) 2019-10-13T12:38:49.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:49.577+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.579300+0000) 2019-10-13T12:38:49.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:49.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.580603+0000) 2019-10-13T12:38:49.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:49.576+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.581346+0000) 2019-10-13T12:38:49.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:49.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.583186+0000) 2019-10-13T12:38:49.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:49.581+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.585334+0000) 2019-10-13T12:38:49.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:49.585+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.586936+0000) 2019-10-13T12:38:49.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:49.580+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.587895+0000) 2019-10-13T12:38:49.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:49.585+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:49.587740+0000) 2019-10-13T12:38:49.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:49.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:51.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:51.896+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:51.901726+0000) 2019-10-13T12:38:54.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:54.225+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.226677+0000) 2019-10-13T12:38:54.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:54.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.511873+0000) 2019-10-13T12:38:54.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:54.588+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.591837+0000) 2019-10-13T12:38:54.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:54.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.592303+0000) 2019-10-13T12:38:54.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:54.589+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.593241+0000) 2019-10-13T12:38:54.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:54.589+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.593654+0000) 2019-10-13T12:38:54.595 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:54.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.594260+0000) 2019-10-13T12:38:54.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:54.593+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.594634+0000) 2019-10-13T12:38:54.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:54.593+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.595468+0000) 2019-10-13T12:38:54.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:54.592+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:54.599556+0000) 2019-10-13T12:38:54.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:54.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:55.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:55.512+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:55.513356+0000) 2019-10-13T12:38:57.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:57.881+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:57.883989+0000) 2019-10-13T12:38:58.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:58.109+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.109353+0000) 2019-10-13T12:38:58.132 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:58.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.130517+0000) 2019-10-13T12:38:58.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:58.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:58.650613+0000) 2019-10-13T12:38:59.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:38:59.600+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.603629+0000) 2019-10-13T12:38:59.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:38:59.601+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.604313+0000) 2019-10-13T12:38:59.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:38:59.601+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.604813+0000) 2019-10-13T12:38:59.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:38:59.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.605518+0000) 2019-10-13T12:38:59.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:38:59.605+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.608053+0000) 2019-10-13T12:38:59.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:38:59.604+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.611601+0000) 2019-10-13T12:38:59.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:38:59.609+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.612010+0000) 2019-10-13T12:38:59.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:38:59.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:38:59.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:38:59.609+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:38:59.613387+0000) 2019-10-13T12:39:01.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:01.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:01.901963+0000) 2019-10-13T12:39:04.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:04.225+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.226880+0000) 2019-10-13T12:39:04.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:04.509+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.512069+0000) 2019-10-13T12:39:04.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:04.613+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.616234+0000) 2019-10-13T12:39:04.618 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:04.613+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.616923+0000) 2019-10-13T12:39:04.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:04.613+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.617285+0000) 2019-10-13T12:39:04.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:04.617+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.619189+0000) 2019-10-13T12:39:04.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:04.612+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.619761+0000) 2019-10-13T12:39:04.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:04.617+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.620910+0000) 2019-10-13T12:39:04.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:04.617+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.621265+0000) 2019-10-13T12:39:04.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:04.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:04.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:04.620+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:04.625841+0000) 2019-10-13T12:39:05.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:05.516+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:05.519879+0000) 2019-10-13T12:39:07.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:07.881+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:07.884189+0000) 2019-10-13T12:39:08.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:08.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.109581+0000) 2019-10-13T12:39:08.132 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:08.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.130751+0000) 2019-10-13T12:39:08.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:08.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:08.650809+0000) 2019-10-13T12:39:09.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:09.625+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.627999+0000) 2019-10-13T12:39:09.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:09.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.628822+0000) 2019-10-13T12:39:09.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:09.625+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.628775+0000) 2019-10-13T12:39:09.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:09.624+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.629683+0000) 2019-10-13T12:39:09.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:09.625+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.629700+0000) 2019-10-13T12:39:09.632 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:09.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.630666+0000) 2019-10-13T12:39:09.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:09.629+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.630968+0000) 2019-10-13T12:39:09.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:09.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:09.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:09.632+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:09.636922+0000) 2019-10-13T12:39:11.195 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:39:11.195 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:11.199 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:39:11.199 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:11.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:11.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:11.902182+0000) 2019-10-13T12:39:14.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:14.224+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.227085+0000) 2019-10-13T12:39:14.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:14.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.512261+0000) 2019-10-13T12:39:14.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:14.636+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.638868+0000) 2019-10-13T12:39:14.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:14.636+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.640597+0000) 2019-10-13T12:39:14.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:14.636+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.641747+0000) 2019-10-13T12:39:14.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:14.640+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.642825+0000) 2019-10-13T12:39:14.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:14.640+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.643747+0000) 2019-10-13T12:39:14.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:14.644+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.646219+0000) 2019-10-13T12:39:14.648 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:14.644+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.646816+0000) 2019-10-13T12:39:14.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:14.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:14.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:14.644+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:14.649496+0000) 2019-10-13T12:39:15.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:15.524+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:15.527017+0000) 2019-10-13T12:39:17.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:17.880+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:17.884385+0000) 2019-10-13T12:39:18.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:18.108+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.109819+0000) 2019-10-13T12:39:18.132 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:18.128+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.131007+0000) 2019-10-13T12:39:18.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:18.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:18.651007+0000) 2019-10-13T12:39:19.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:19.648+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.652778+0000) 2019-10-13T12:39:19.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:19.652+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.653835+0000) 2019-10-13T12:39:19.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:19.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.654146+0000) 2019-10-13T12:39:19.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:19.648+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.654870+0000) 2019-10-13T12:39:19.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:19.652+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.654973+0000) 2019-10-13T12:39:19.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:19.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.656017+0000) 2019-10-13T12:39:19.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:19.652+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.656922+0000) 2019-10-13T12:39:19.660 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:19.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:19.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:19.656+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:19.661242+0000) 2019-10-13T12:39:21.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:21.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:21.902445+0000) 2019-10-13T12:39:24.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:24.224+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.227342+0000) 2019-10-13T12:39:24.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:24.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.512485+0000) 2019-10-13T12:39:24.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:24.656+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.662414+0000) 2019-10-13T12:39:24.664 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:24.660+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.662475+0000) 2019-10-13T12:39:24.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:24.660+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.662789+0000) 2019-10-13T12:39:24.665 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:24.660+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.663263+0000) 2019-10-13T12:39:24.665 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:24.660+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.664185+0000) 2019-10-13T12:39:24.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:24.660+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.665240+0000) 2019-10-13T12:39:24.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:24.664+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.666083+0000) 2019-10-13T12:39:24.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:24.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:24.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:24.668+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:24.672466+0000) 2019-10-13T12:39:25.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:25.532+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:25.534228+0000) 2019-10-13T12:39:27.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:27.880+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:27.884615+0000) 2019-10-13T12:39:28.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:28.108+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.110030+0000) 2019-10-13T12:39:28.132 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:28.128+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.131211+0000) 2019-10-13T12:39:28.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:28.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:28.651203+0000) 2019-10-13T12:39:29.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:29.672+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.674688+0000) 2019-10-13T12:39:29.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:29.672+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.674948+0000) 2019-10-13T12:39:29.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:29.672+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.676536+0000) 2019-10-13T12:39:29.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:29.672+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.677047+0000) 2019-10-13T12:39:29.679 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:29.672+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.677641+0000) 2019-10-13T12:39:29.679 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:29.676+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.678162+0000) 2019-10-13T12:39:29.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:29.676+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.678917+0000) 2019-10-13T12:39:29.684 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:29.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:29.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:29.684+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:29.686239+0000) 2019-10-13T12:39:31.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:31.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:31.902641+0000) 2019-10-13T12:39:34.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:34.224+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.227558+0000) 2019-10-13T12:39:34.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:34.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.512685+0000) 2019-10-13T12:39:34.689 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:34.684+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.688060+0000) 2019-10-13T12:39:34.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:34.684+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.687780+0000) 2019-10-13T12:39:34.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:34.687+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.689431+0000) 2019-10-13T12:39:34.691 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:34.684+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.688985+0000) 2019-10-13T12:39:34.691 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:34.684+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.689411+0000) 2019-10-13T12:39:34.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:34.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.691966+0000) 2019-10-13T12:39:34.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:34.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.692063+0000) 2019-10-13T12:39:34.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:34.691+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:34.692630+0000) 2019-10-13T12:39:34.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:34.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:35.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:35.539+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:35.540288+0000) 2019-10-13T12:39:37.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:37.880+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:37.884809+0000) 2019-10-13T12:39:38.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:38.108+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.110237+0000) 2019-10-13T12:39:38.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:38.128+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.131409+0000) 2019-10-13T12:39:38.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:38.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:38.651399+0000) 2019-10-13T12:39:39.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:39.696+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.698997+0000) 2019-10-13T12:39:39.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:39.696+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.699382+0000) 2019-10-13T12:39:39.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:39.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.699705+0000) 2019-10-13T12:39:39.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:39.696+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.700346+0000) 2019-10-13T12:39:39.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:39.699+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.700860+0000) 2019-10-13T12:39:39.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:39.696+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.701592+0000) 2019-10-13T12:39:39.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:39.700+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.702439+0000) 2019-10-13T12:39:39.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:39.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:39.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:39.707+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:39.708719+0000) 2019-10-13T12:39:41.291 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:39:41.291 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:41.295 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:39:41.295 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:39:41.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:41.899+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:41.902790+0000) 2019-10-13T12:39:44.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:44.224+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.227788+0000) 2019-10-13T12:39:44.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:44.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.512884+0000) 2019-10-13T12:39:44.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:44.704+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.709204+0000) 2019-10-13T12:39:44.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:44.707+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.710388+0000) 2019-10-13T12:39:44.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:44.708+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.711289+0000) 2019-10-13T12:39:44.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:44.708+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.711720+0000) 2019-10-13T12:39:44.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:44.708+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.711928+0000) 2019-10-13T12:39:44.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:44.708+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.712930+0000) 2019-10-13T12:39:44.714 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:44.708+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.713183+0000) 2019-10-13T12:39:44.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:44.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:44.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:44.719+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:44.720472+0000) 2019-10-13T12:39:45.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:45.543+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:45.546908+0000) 2019-10-13T12:39:47.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:47.880+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:47.885009+0000) 2019-10-13T12:39:48.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:48.108+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.110455+0000) 2019-10-13T12:39:48.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:48.128+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.131628+0000) 2019-10-13T12:39:48.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:48.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:48.651593+0000) 2019-10-13T12:39:49.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:49.716+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.721598+0000) 2019-10-13T12:39:49.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:49.720+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.723058+0000) 2019-10-13T12:39:49.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:49.719+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.723936+0000) 2019-10-13T12:39:49.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:49.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.724110+0000) 2019-10-13T12:39:49.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:49.720+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.724783+0000) 2019-10-13T12:39:49.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:49.724+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.726804+0000) 2019-10-13T12:39:49.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:49.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:49.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:49.728+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.731200+0000) 2019-10-13T12:39:49.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:49.739+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:49.741997+0000) 2019-10-13T12:39:51.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:51.899+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:51.903004+0000) 2019-10-13T12:39:54.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:54.224+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.228022+0000) 2019-10-13T12:39:54.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:54.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.513091+0000) 2019-10-13T12:39:54.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:54.731+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.734109+0000) 2019-10-13T12:39:54.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:54.732+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.734529+0000) 2019-10-13T12:39:54.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:54.732+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.735202+0000) 2019-10-13T12:39:54.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:54.732+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.737087+0000) 2019-10-13T12:39:54.738 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:54.732+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.737294+0000) 2019-10-13T12:39:54.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:54.736+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.739092+0000) 2019-10-13T12:39:54.741 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:54.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.739675+0000) 2019-10-13T12:39:54.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:54.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:54.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:54.751+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:54.752307+0000) 2019-10-13T12:39:55.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:55.551+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:55.553462+0000) 2019-10-13T12:39:57.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:57.884+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:57.885211+0000) 2019-10-13T12:39:58.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:58.108+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.110654+0000) 2019-10-13T12:39:58.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:58.128+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.131854+0000) 2019-10-13T12:39:58.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:58.648+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:58.651818+0000) 2019-10-13T12:39:59.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:39:59.744+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.747646+0000) 2019-10-13T12:39:59.750 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:39:59.747+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.748256+0000) 2019-10-13T12:39:59.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:39:59.744+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.748751+0000) 2019-10-13T12:39:59.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:39:59.744+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.749196+0000) 2019-10-13T12:39:59.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:39:59.744+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.748642+0000) 2019-10-13T12:39:59.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:39:59.748+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.749862+0000) 2019-10-13T12:39:59.753 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:39:59.748+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.752409+0000) 2019-10-13T12:39:59.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:39:59.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:39:59.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:39:59.755+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:39:59.758240+0000) 2019-10-13T12:40:01.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:01.899+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:01.903211+0000) 2019-10-13T12:40:04.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:04.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.228242+0000) 2019-10-13T12:40:04.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:04.508+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.513282+0000) 2019-10-13T12:40:04.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:04.756+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.758021+0000) 2019-10-13T12:40:04.760 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:04.755+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.758692+0000) 2019-10-13T12:40:04.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:04.756+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.758746+0000) 2019-10-13T12:40:04.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:04.756+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.760902+0000) 2019-10-13T12:40:04.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:04.756+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.761663+0000) 2019-10-13T12:40:04.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:04.760+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.762375+0000) 2019-10-13T12:40:04.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:04.760+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.763591+0000) 2019-10-13T12:40:04.766 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:04.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:04.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:04.763+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:04.766988+0000) 2019-10-13T12:40:05.561 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:05.559+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:05.559975+0000) 2019-10-13T12:40:07.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:07.879+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:07.885407+0000) 2019-10-13T12:40:08.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:08.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.110854+0000) 2019-10-13T12:40:08.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:08.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.132051+0000) 2019-10-13T12:40:08.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:08.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:08.652028+0000) 2019-10-13T12:40:09.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:09.763+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.769689+0000) 2019-10-13T12:40:09.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:09.763+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.769577+0000) 2019-10-13T12:40:09.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:09.767+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.770061+0000) 2019-10-13T12:40:09.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:09.767+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.770401+0000) 2019-10-13T12:40:09.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:09.767+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.771891+0000) 2019-10-13T12:40:09.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:09.767+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.772437+0000) 2019-10-13T12:40:09.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:09.767+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.773125+0000) 2019-10-13T12:40:09.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:09.775+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:09.777322+0000) 2019-10-13T12:40:09.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:09.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:11.406 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:40:11.406 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:11.409 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:40:11.410 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:11.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:11.899+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:11.903425+0000) 2019-10-13T12:40:14.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:14.223+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.228442+0000) 2019-10-13T12:40:14.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:14.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.513497+0000) 2019-10-13T12:40:14.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:14.779+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.780442+0000) 2019-10-13T12:40:14.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:14.775+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.780465+0000) 2019-10-13T12:40:14.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:14.775+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.781065+0000) 2019-10-13T12:40:14.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:14.775+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.781706+0000) 2019-10-13T12:40:14.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:14.779+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.783527+0000) 2019-10-13T12:40:14.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:14.783+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.786441+0000) 2019-10-13T12:40:14.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:14.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165679 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:14.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:14.783+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.789542+0000) 2019-10-13T12:40:14.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:14.791+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:14.795934+0000) 2019-10-13T12:40:15.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:15.563+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:15.565928+0000) 2019-10-13T12:40:17.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:17.883+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:17.885556+0000) 2019-10-13T12:40:18.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:18.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.111057+0000) 2019-10-13T12:40:18.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:18.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.132254+0000) 2019-10-13T12:40:18.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:18.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:18.652237+0000) 2019-10-13T12:40:19.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:19.787+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.791994+0000) 2019-10-13T12:40:19.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:19.787+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.792126+0000) 2019-10-13T12:40:19.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:19.787+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.792588+0000) 2019-10-13T12:40:19.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:19.791+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.795191+0000) 2019-10-13T12:40:19.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:19.791+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.795465+0000) 2019-10-13T12:40:19.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:19.791+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.796019+0000) 2019-10-13T12:40:19.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:19.795+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.796595+0000) 2019-10-13T12:40:19.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:19.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:19.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:19.803+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:19.807643+0000) 2019-10-13T12:40:21.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:21.899+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:21.903621+0000) 2019-10-13T12:40:24.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:24.227+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.228652+0000) 2019-10-13T12:40:24.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:24.507+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.513692+0000) 2019-10-13T12:40:24.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:24.799+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.804624+0000) 2019-10-13T12:40:24.806 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:24.799+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.804959+0000) 2019-10-13T12:40:24.807 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:24.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.805365+0000) 2019-10-13T12:40:24.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:24.799+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.805687+0000) 2019-10-13T12:40:24.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:24.803+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.806430+0000) 2019-10-13T12:40:24.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:24.803+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.807216+0000) 2019-10-13T12:40:24.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:24.803+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.807940+0000) 2019-10-13T12:40:24.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:24.811+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:24.813219+0000) 2019-10-13T12:40:24.814 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:24.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:25.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:25.571+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:25.572483+0000) 2019-10-13T12:40:27.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:27.883+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:27.885786+0000) 2019-10-13T12:40:28.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:28.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.111314+0000) 2019-10-13T12:40:28.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:28.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.132456+0000) 2019-10-13T12:40:28.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:28.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:28.652429+0000) 2019-10-13T12:40:29.817 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:29.811+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.816013+0000) 2019-10-13T12:40:29.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:29.811+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.817013+0000) 2019-10-13T12:40:29.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:29.811+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.817275+0000) 2019-10-13T12:40:29.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:29.815+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.818191+0000) 2019-10-13T12:40:29.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:29.815+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.818555+0000) 2019-10-13T12:40:29.820 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:29.815+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.819500+0000) 2019-10-13T12:40:29.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:29.815+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.819938+0000) 2019-10-13T12:40:29.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:29.823+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:29.824298+0000) 2019-10-13T12:40:29.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:29.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:31.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:31.903+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:31.903862+0000) 2019-10-13T12:40:34.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:34.227+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.228887+0000) 2019-10-13T12:40:34.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:34.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.513915+0000) 2019-10-13T12:40:34.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:34.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.827664+0000) 2019-10-13T12:40:34.830 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:34.823+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.828234+0000) 2019-10-13T12:40:34.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:34.823+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.828702+0000) 2019-10-13T12:40:34.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:34.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.829439+0000) 2019-10-13T12:40:34.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:34.827+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.829953+0000) 2019-10-13T12:40:34.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:34.827+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.830437+0000) 2019-10-13T12:40:34.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:34.827+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.831204+0000) 2019-10-13T12:40:34.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:34.831+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:34.835884+0000) 2019-10-13T12:40:34.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:34.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:35.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:35.575+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:35.579018+0000) 2019-10-13T12:40:37.887 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:37.883+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:37.885982+0000) 2019-10-13T12:40:38.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:38.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.111531+0000) 2019-10-13T12:40:38.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:38.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.132656+0000) 2019-10-13T12:40:38.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:38.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:38.652656+0000) 2019-10-13T12:40:39.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:39.835+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.840221+0000) 2019-10-13T12:40:39.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:39.835+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.840885+0000) 2019-10-13T12:40:39.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:39.839+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.842275+0000) 2019-10-13T12:40:39.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:39.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.842710+0000) 2019-10-13T12:40:39.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:39.839+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.843693+0000) 2019-10-13T12:40:39.846 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:39.839+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.844283+0000) 2019-10-13T12:40:39.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:39.842+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.844737+0000) 2019-10-13T12:40:39.849 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:39.842+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:39.847888+0000) 2019-10-13T12:40:39.851 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:39.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165836 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:41.511 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:40:41.511 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:41.515 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:40:41.515 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:40:41.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:41.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:41.904126+0000) 2019-10-13T12:40:44.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:44.227+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.229113+0000) 2019-10-13T12:40:44.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:44.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.514134+0000) 2019-10-13T12:40:44.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:44.851+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.855269+0000) 2019-10-13T12:40:44.857 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:44.851+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.855605+0000) 2019-10-13T12:40:44.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:44.854+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.856083+0000) 2019-10-13T12:40:44.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:44.851+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.856359+0000) 2019-10-13T12:40:44.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:44.855+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.858174+0000) 2019-10-13T12:40:44.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:44.855+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.858421+0000) 2019-10-13T12:40:44.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:44.855+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.858802+0000) 2019-10-13T12:40:44.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:44.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:44.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:44.866+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:44.869470+0000) 2019-10-13T12:40:45.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:45.582+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:45.585514+0000) 2019-10-13T12:40:47.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:47.883+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:47.886212+0000) 2019-10-13T12:40:48.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:48.107+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.111747+0000) 2019-10-13T12:40:48.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:48.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.132851+0000) 2019-10-13T12:40:48.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:48.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:48.652866+0000) 2019-10-13T12:40:49.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:49.859+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.865628+0000) 2019-10-13T12:40:49.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:49.863+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.866252+0000) 2019-10-13T12:40:49.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:49.863+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.866768+0000) 2019-10-13T12:40:49.871 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:49.863+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.869576+0000) 2019-10-13T12:40:49.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:49.866+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.871840+0000) 2019-10-13T12:40:49.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:49.870+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.874219+0000) 2019-10-13T12:40:49.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:49.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165899 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:49.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:49.871+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.874597+0000) 2019-10-13T12:40:49.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:49.871+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:49.874945+0000) 2019-10-13T12:40:51.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:51.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:51.904360+0000) 2019-10-13T12:40:54.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:54.227+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.229326+0000) 2019-10-13T12:40:54.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:54.511+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.514336+0000) 2019-10-13T12:40:54.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:54.871+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.877132+0000) 2019-10-13T12:40:54.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:54.875+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.877914+0000) 2019-10-13T12:40:54.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:54.875+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.878233+0000) 2019-10-13T12:40:54.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:54.875+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.879463+0000) 2019-10-13T12:40:54.882 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:54.875+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.879883+0000) 2019-10-13T12:40:54.882 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:54.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.880850+0000) 2019-10-13T12:40:54.883 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:54.875+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.881172+0000) 2019-10-13T12:40:54.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:54.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:54.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:54.882+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:54.886863+0000) 2019-10-13T12:40:55.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:55.590+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:55.592037+0000) 2019-10-13T12:40:57.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:57.883+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:57.886415+0000) 2019-10-13T12:40:58.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:58.111+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.111962+0000) 2019-10-13T12:40:58.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:58.127+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.132997+0000) 2019-10-13T12:40:58.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:58.647+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:58.653060+0000) 2019-10-13T12:40:59.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:40:59.883+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.889468+0000) 2019-10-13T12:40:59.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:40:59.887+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.889837+0000) 2019-10-13T12:40:59.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:40:59.887+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.890711+0000) 2019-10-13T12:40:59.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:40:59.887+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.891654+0000) 2019-10-13T12:40:59.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:40:59.890+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.892220+0000) 2019-10-13T12:40:59.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:40:59.887+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.892601+0000) 2019-10-13T12:40:59.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:40:59.887+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.892901+0000) 2019-10-13T12:40:59.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:40:59.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165959 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:40:59.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:40:59.894+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:40:59.897931+0000) 2019-10-13T12:41:01.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:01.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:01.904523+0000) 2019-10-13T12:41:04.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:04.226+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.229557+0000) 2019-10-13T12:41:04.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:04.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.514550+0000) 2019-10-13T12:41:04.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:04.894+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.900375+0000) 2019-10-13T12:41:04.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:04.894+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.900776+0000) 2019-10-13T12:41:04.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:04.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.901566+0000) 2019-10-13T12:41:04.903 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:04.898+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.902572+0000) 2019-10-13T12:41:04.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:04.898+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.902962+0000) 2019-10-13T12:41:04.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:04.898+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.903902+0000) 2019-10-13T12:41:04.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:04.902+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.905036+0000) 2019-10-13T12:41:04.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:04.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165990 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:04.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:04.906+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:04.908838+0000) 2019-10-13T12:41:05.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:05.598+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:05.598714+0000) 2019-10-13T12:41:07.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:07.882+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:07.886631+0000) 2019-10-13T12:41:08.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:08.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.112132+0000) 2019-10-13T12:41:08.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:08.126+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.133193+0000) 2019-10-13T12:41:08.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:08.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:08.653257+0000) 2019-10-13T12:41:09.913 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:09.906+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.911760+0000) 2019-10-13T12:41:09.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:09.906+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.911941+0000) 2019-10-13T12:41:09.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:09.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.912931+0000) 2019-10-13T12:41:09.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:09.906+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.913083+0000) 2019-10-13T12:41:09.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:09.906+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.913313+0000) 2019-10-13T12:41:09.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:09.914+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.916352+0000) 2019-10-13T12:41:09.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:09.910+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.916522+0000) 2019-10-13T12:41:09.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:09.918+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:09.920778+0000) 2019-10-13T12:41:09.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:09.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:11.600 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:41:11.600 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:11.604 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:41:11.604 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:11.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:11.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:11.904730+0000) 2019-10-13T12:41:14.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:14.226+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.229779+0000) 2019-10-13T12:41:14.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:14.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.514749+0000) 2019-10-13T12:41:14.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:14.918+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.923774+0000) 2019-10-13T12:41:14.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:14.918+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.924429+0000) 2019-10-13T12:41:14.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:14.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.925131+0000) 2019-10-13T12:41:14.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:14.922+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.925915+0000) 2019-10-13T12:41:14.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:14.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.926531+0000) 2019-10-13T12:41:14.928 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:14.922+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.926538+0000) 2019-10-13T12:41:14.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:14.922+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.927146+0000) 2019-10-13T12:41:14.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:14.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166051 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:14.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:14.930+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:14.932278+0000) 2019-10-13T12:41:15.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:15.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:15.605347+0000) 2019-10-13T12:41:17.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:17.882+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:17.886838+0000) 2019-10-13T12:41:18.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:18.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.112325+0000) 2019-10-13T12:41:18.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:18.130+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.133397+0000) 2019-10-13T12:41:18.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:18.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:18.653446+0000) 2019-10-13T12:41:19.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:19.930+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.934987+0000) 2019-10-13T12:41:19.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:19.930+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.935331+0000) 2019-10-13T12:41:19.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:19.930+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.936665+0000) 2019-10-13T12:41:19.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:19.934+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.937790+0000) 2019-10-13T12:41:19.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:19.930+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.937592+0000) 2019-10-13T12:41:19.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:19.934+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.937928+0000) 2019-10-13T12:41:19.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:19.934+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.939153+0000) 2019-10-13T12:41:19.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:19.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:19.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:19.942+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:19.945454+0000) 2019-10-13T12:41:21.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:21.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:21.904963+0000) 2019-10-13T12:41:24.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:24.226+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.229978+0000) 2019-10-13T12:41:24.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:24.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.514948+0000) 2019-10-13T12:41:24.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:24.942+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.947316+0000) 2019-10-13T12:41:24.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:24.942+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.949237+0000) 2019-10-13T12:41:24.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:24.942+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.949644+0000) 2019-10-13T12:41:24.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:24.946+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.950202+0000) 2019-10-13T12:41:24.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:24.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.949904+0000) 2019-10-13T12:41:24.952 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:24.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.950814+0000) 2019-10-13T12:41:24.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:24.946+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.952540+0000) 2019-10-13T12:41:24.956 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:24.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:24.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:24.954+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:24.959197+0000) 2019-10-13T12:41:25.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:25.610+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:25.611313+0000) 2019-10-13T12:41:27.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:27.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:27.887073+0000) 2019-10-13T12:41:28.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:28.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.112529+0000) 2019-10-13T12:41:28.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:28.130+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.133594+0000) 2019-10-13T12:41:28.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:28.646+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:28.653635+0000) 2019-10-13T12:41:29.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:29.954+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.959699+0000) 2019-10-13T12:41:29.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:29.954+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.960254+0000) 2019-10-13T12:41:29.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:29.958+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.961957+0000) 2019-10-13T12:41:29.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:29.958+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.962231+0000) 2019-10-13T12:41:29.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:29.958+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.962930+0000) 2019-10-13T12:41:29.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:29.958+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.964765+0000) 2019-10-13T12:41:29.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:29.958+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.965438+0000) 2019-10-13T12:41:29.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:29.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:29.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:29.966+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:29.970336+0000) 2019-10-13T12:41:31.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:31.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:31.905158+0000) 2019-10-13T12:41:34.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:34.226+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.230190+0000) 2019-10-13T12:41:34.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:34.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.515192+0000) 2019-10-13T12:41:34.971 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:34.962+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.969648+0000) 2019-10-13T12:41:34.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:34.966+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.972205+0000) 2019-10-13T12:41:34.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:34.966+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.972422+0000) 2019-10-13T12:41:34.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:34.970+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.974683+0000) 2019-10-13T12:41:34.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:34.970+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.974841+0000) 2019-10-13T12:41:34.976 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:34.970+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.975353+0000) 2019-10-13T12:41:34.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:34.970+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.976053+0000) 2019-10-13T12:41:34.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:34.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:34.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:34.978+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:34.982670+0000) 2019-10-13T12:41:35.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:35.614+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:35.618013+0000) 2019-10-13T12:41:37.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:37.882+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:37.887302+0000) 2019-10-13T12:41:38.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:38.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.112690+0000) 2019-10-13T12:41:38.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:38.130+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.133812+0000) 2019-10-13T12:41:38.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:38.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:38.653832+0000) 2019-10-13T12:41:39.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:39.978+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.981787+0000) 2019-10-13T12:41:39.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:39.978+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.982422+0000) 2019-10-13T12:41:39.984 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:39.978+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.982778+0000) 2019-10-13T12:41:39.985 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:39.978+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.983452+0000) 2019-10-13T12:41:39.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:39.982+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.984199+0000) 2019-10-13T12:41:39.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:39.978+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.984272+0000) 2019-10-13T12:41:39.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:39.978+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.985019+0000) 2019-10-13T12:41:39.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:39.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:39.987996+0000) 2019-10-13T12:41:39.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:39.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:41.696 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:41:41.696 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:41.700 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:41:41.700 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:41:41.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:41.902+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:41.905389+0000) 2019-10-13T12:41:44.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:44.230+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.230407+0000) 2019-10-13T12:41:44.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:44.510+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.515449+0000) 2019-10-13T12:41:44.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:44.986+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.993356+0000) 2019-10-13T12:41:44.995 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:44.990+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.993832+0000) 2019-10-13T12:41:44.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:44.990+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.994214+0000) 2019-10-13T12:41:44.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:44.989+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.995489+0000) 2019-10-13T12:41:44.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:44.990+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.995211+0000) 2019-10-13T12:41:44.997 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:44.990+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.994629+0000) 2019-10-13T12:41:44.997 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:44.990+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:44.995916+0000) 2019-10-13T12:41:45.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:44.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:45.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:45.001+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:45.004836+0000) 2019-10-13T12:41:45.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:45.621+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:45.624467+0000) 2019-10-13T12:41:47.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:47.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:47.887538+0000) 2019-10-13T12:41:48.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:48.106+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.112886+0000) 2019-10-13T12:41:48.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:48.130+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.134029+0000) 2019-10-13T12:41:48.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:48.650+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:48.654044+0000) 2019-10-13T12:41:50.006 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:49.998+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.004010+0000) 2019-10-13T12:41:50.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:49.998+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.004724+0000) 2019-10-13T12:41:50.007 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:49.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.005470+0000) 2019-10-13T12:41:50.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:50.002+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.005811+0000) 2019-10-13T12:41:50.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:50.002+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.006818+0000) 2019-10-13T12:41:50.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:50.001+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.007706+0000) 2019-10-13T12:41:50.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:50.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.008046+0000) 2019-10-13T12:41:50.014 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:50.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166270 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:50.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:50.009+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:50.013898+0000) 2019-10-13T12:41:51.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:51.901+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:51.905616+0000) 2019-10-13T12:41:54.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:54.230+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:54.230609+0000) 2019-10-13T12:41:54.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:54.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:54.515656+0000) 2019-10-13T12:41:55.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:41:55.010+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.016790+0000) 2019-10-13T12:41:55.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:55.010+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.017063+0000) 2019-10-13T12:41:55.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:41:55.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.017799+0000) 2019-10-13T12:41:55.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:41:55.013+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.018944+0000) 2019-10-13T12:41:55.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:55.014+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.019282+0000) 2019-10-13T12:41:55.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:55.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.019702+0000) 2019-10-13T12:41:55.021 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:55.014+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.020279+0000) 2019-10-13T12:41:55.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:41:55.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:41:55.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:55.025+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.028734+0000) 2019-10-13T12:41:55.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:41:55.629+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:55.630909+0000) 2019-10-13T12:41:57.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:41:57.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:57.887742+0000) 2019-10-13T12:41:58.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:41:58.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.113079+0000) 2019-10-13T12:41:58.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:41:58.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.134269+0000) 2019-10-13T12:41:58.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:41:58.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:41:58.654244+0000) 2019-10-13T12:42:00.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:00.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.027354+0000) 2019-10-13T12:42:00.030 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:00.021+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.027630+0000) 2019-10-13T12:42:00.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:00.025+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.029791+0000) 2019-10-13T12:42:00.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:00.025+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.030486+0000) 2019-10-13T12:42:00.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:00.025+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.030985+0000) 2019-10-13T12:42:00.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:00.025+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.031962+0000) 2019-10-13T12:42:00.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:00.025+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.032759+0000) 2019-10-13T12:42:00.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:00.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:00.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:00.033+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:00.037058+0000) 2019-10-13T12:42:01.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:01.901+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:01.905848+0000) 2019-10-13T12:42:04.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:04.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:04.230827+0000) 2019-10-13T12:42:04.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:04.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:04.515858+0000) 2019-10-13T12:42:05.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:05.033+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.039370+0000) 2019-10-13T12:42:05.042 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:05.033+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.040323+0000) 2019-10-13T12:42:05.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:05.033+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.041022+0000) 2019-10-13T12:42:05.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:05.033+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.041220+0000) 2019-10-13T12:42:05.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:05.033+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.041561+0000) 2019-10-13T12:42:05.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:05.037+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.041986+0000) 2019-10-13T12:42:05.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:05.037+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.042632+0000) 2019-10-13T12:42:05.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:05.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:05.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:05.045+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.051744+0000) 2019-10-13T12:42:05.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:05.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:05.637526+0000) 2019-10-13T12:42:07.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:07.885+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:07.887939+0000) 2019-10-13T12:42:08.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:08.105+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.113299+0000) 2019-10-13T12:42:08.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:08.129+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.134469+0000) 2019-10-13T12:42:08.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:08.649+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:08.654484+0000) 2019-10-13T12:42:10.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:10.045+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.051952+0000) 2019-10-13T12:42:10.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:10.045+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.052909+0000) 2019-10-13T12:42:10.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:10.049+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.053518+0000) 2019-10-13T12:42:10.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:10.049+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.054862+0000) 2019-10-13T12:42:10.056 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:10.049+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.055283+0000) 2019-10-13T12:42:10.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:10.053+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.058546+0000) 2019-10-13T12:42:10.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:10.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:10.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:10.053+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.060909+0000) 2019-10-13T12:42:10.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:10.061+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:10.066476+0000) 2019-10-13T12:42:11.802 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:42:11.802 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:11.806 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:42:11.806 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:11.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:11.901+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:11.906064+0000) 2019-10-13T12:42:14.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:14.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:14.231024+0000) 2019-10-13T12:42:14.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:14.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:14.516064+0000) 2019-10-13T12:42:15.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:15.061+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.064499+0000) 2019-10-13T12:42:15.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:15.061+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.065027+0000) 2019-10-13T12:42:15.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:15.061+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.065414+0000) 2019-10-13T12:42:15.067 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:15.065+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.065839+0000) 2019-10-13T12:42:15.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:15.065+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.067692+0000) 2019-10-13T12:42:15.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:15.065+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.067715+0000) 2019-10-13T12:42:15.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:15.069+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.069724+0000) 2019-10-13T12:42:15.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:15.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:15.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:15.065+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.072051+0000) 2019-10-13T12:42:15.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:15.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:15.643976+0000) 2019-10-13T12:42:17.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:17.885+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:17.888171+0000) 2019-10-13T12:42:18.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:18.109+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.113493+0000) 2019-10-13T12:42:18.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:18.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.134668+0000) 2019-10-13T12:42:18.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:18.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:18.654679+0000) 2019-10-13T12:42:20.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:20.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.076835+0000) 2019-10-13T12:42:20.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:20.073+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.077298+0000) 2019-10-13T12:42:20.079 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:20.077+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.077818+0000) 2019-10-13T12:42:20.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:20.073+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.078789+0000) 2019-10-13T12:42:20.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:20.077+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.080485+0000) 2019-10-13T12:42:20.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:20.077+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.080954+0000) 2019-10-13T12:42:20.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:20.081+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.082356+0000) 2019-10-13T12:42:20.085 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:20.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:20.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:20.081+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:20.087111+0000) 2019-10-13T12:42:21.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:21.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:21.906304+0000) 2019-10-13T12:42:24.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:24.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:24.231253+0000) 2019-10-13T12:42:24.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:24.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:24.516265+0000) 2019-10-13T12:42:25.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:25.085+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.087260+0000) 2019-10-13T12:42:25.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:25.085+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.087846+0000) 2019-10-13T12:42:25.090 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:25.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.089057+0000) 2019-10-13T12:42:25.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:25.085+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.089085+0000) 2019-10-13T12:42:25.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:25.089+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.090941+0000) 2019-10-13T12:42:25.093 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:25.089+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.091691+0000) 2019-10-13T12:42:25.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:25.093+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.093731+0000) 2019-10-13T12:42:25.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:25.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:25.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:25.093+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.098094+0000) 2019-10-13T12:42:25.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:25.649+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:25.650452+0000) 2019-10-13T12:42:27.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:27.885+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:27.888389+0000) 2019-10-13T12:42:28.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:28.109+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.113649+0000) 2019-10-13T12:42:28.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:28.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.134878+0000) 2019-10-13T12:42:28.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:28.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:28.654870+0000) 2019-10-13T12:42:30.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:30.093+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.099337+0000) 2019-10-13T12:42:30.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:30.097+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.099734+0000) 2019-10-13T12:42:30.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:30.097+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.100372+0000) 2019-10-13T12:42:30.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:30.097+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.100907+0000) 2019-10-13T12:42:30.104 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:30.101+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.102569+0000) 2019-10-13T12:42:30.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:30.101+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.102332+0000) 2019-10-13T12:42:30.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:30.105+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.105825+0000) 2019-10-13T12:42:30.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:30.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:30.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:30.113+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:30.116243+0000) 2019-10-13T12:42:31.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:31.901+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:31.906517+0000) 2019-10-13T12:42:34.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:34.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:34.231477+0000) 2019-10-13T12:42:34.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:34.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:34.516468+0000) 2019-10-13T12:42:35.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:35.105+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.111166+0000) 2019-10-13T12:42:35.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:35.109+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.111598+0000) 2019-10-13T12:42:35.114 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:35.109+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.112971+0000) 2019-10-13T12:42:35.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:35.113+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.114988+0000) 2019-10-13T12:42:35.116 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:35.113+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.115488+0000) 2019-10-13T12:42:35.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:35.113+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.117005+0000) 2019-10-13T12:42:35.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:35.113+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.116836+0000) 2019-10-13T12:42:35.121 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:35.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166547 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:35.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:35.121+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.125027+0000) 2019-10-13T12:42:35.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:35.653+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:35.657047+0000) 2019-10-13T12:42:37.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:37.885+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:37.888605+0000) 2019-10-13T12:42:38.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:38.113+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.113843+0000) 2019-10-13T12:42:38.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:38.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.135076+0000) 2019-10-13T12:42:38.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:38.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:38.655090+0000) 2019-10-13T12:42:40.125 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:40.121+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.123512+0000) 2019-10-13T12:42:40.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:40.121+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.124213+0000) 2019-10-13T12:42:40.126 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:40.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.124592+0000) 2019-10-13T12:42:40.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:40.121+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.125906+0000) 2019-10-13T12:42:40.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:40.125+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.125964+0000) 2019-10-13T12:42:40.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:40.125+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.126400+0000) 2019-10-13T12:42:40.128 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:40.125+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.126664+0000) 2019-10-13T12:42:40.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:40.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166578 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:40.134 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:40.129+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:40.133144+0000) 2019-10-13T12:42:41.871 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:42:41.871 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:41.875 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:42:41.875 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:42:41.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:41.901+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:41.906723+0000) 2019-10-13T12:42:44.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:44.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:44.231673+0000) 2019-10-13T12:42:44.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:44.513+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:44.516673+0000) 2019-10-13T12:42:45.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:45.133+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.134865+0000) 2019-10-13T12:42:45.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:45.133+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.135670+0000) 2019-10-13T12:42:45.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:45.133+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.136492+0000) 2019-10-13T12:42:45.139 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:45.137+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.138140+0000) 2019-10-13T12:42:45.140 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:45.137+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.138831+0000) 2019-10-13T12:42:45.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:45.133+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.139427+0000) 2019-10-13T12:42:45.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:45.137+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.139469+0000) 2019-10-13T12:42:45.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:45.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.141099+0000) 2019-10-13T12:42:45.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:45.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:45.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:45.657+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:45.663624+0000) 2019-10-13T12:42:47.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:47.885+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:47.888803+0000) 2019-10-13T12:42:48.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:48.113+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.114046+0000) 2019-10-13T12:42:48.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:48.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.135278+0000) 2019-10-13T12:42:48.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:48.653+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:48.655313+0000) 2019-10-13T12:42:50.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:50.145+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.147378+0000) 2019-10-13T12:42:50.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:50.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.148201+0000) 2019-10-13T12:42:50.151 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:50.144+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.148252+0000) 2019-10-13T12:42:50.151 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:50.145+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.148676+0000) 2019-10-13T12:42:50.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:50.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.149676+0000) 2019-10-13T12:42:50.153 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:50.149+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.149890+0000) 2019-10-13T12:42:50.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:50.149+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.150322+0000) 2019-10-13T12:42:50.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:50.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:50.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:50.152+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:50.158290+0000) 2019-10-13T12:42:51.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:51.900+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:51.906927+0000) 2019-10-13T12:42:54.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:54.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:54.231877+0000) 2019-10-13T12:42:54.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:54.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:54.516822+0000) 2019-10-13T12:42:55.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:55.156+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.158713+0000) 2019-10-13T12:42:55.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:55.156+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.159198+0000) 2019-10-13T12:42:55.162 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:55.156+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.160766+0000) 2019-10-13T12:42:55.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:42:55.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.161317+0000) 2019-10-13T12:42:55.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:42:55.156+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.163279+0000) 2019-10-13T12:42:55.166 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:42:55.160+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.164720+0000) 2019-10-13T12:42:55.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:55.160+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.165053+0000) 2019-10-13T12:42:55.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:42:55.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:42:55.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:55.164+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.169016+0000) 2019-10-13T12:42:55.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:42:55.668+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:55.670050+0000) 2019-10-13T12:42:57.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:42:57.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:57.889029+0000) 2019-10-13T12:42:58.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:42:58.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.114269+0000) 2019-10-13T12:42:58.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:42:58.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.135476+0000) 2019-10-13T12:42:58.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:42:58.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:42:58.655509+0000) 2019-10-13T12:43:00.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:00.168+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.170110+0000) 2019-10-13T12:43:00.173 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:00.168+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.170898+0000) 2019-10-13T12:43:00.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:00.168+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.171264+0000) 2019-10-13T12:43:00.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:00.168+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.171510+0000) 2019-10-13T12:43:00.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:00.168+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.172473+0000) 2019-10-13T12:43:00.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:00.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.171940+0000) 2019-10-13T12:43:00.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:00.168+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.173282+0000) 2019-10-13T12:43:00.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:00.172+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:00.178174+0000) 2019-10-13T12:43:00.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:00.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:01.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:01.904+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:01.907120+0000) 2019-10-13T12:43:04.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:04.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:04.232119+0000) 2019-10-13T12:43:04.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:04.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:04.517037+0000) 2019-10-13T12:43:05.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:05.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.182438+0000) 2019-10-13T12:43:05.184 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:05.180+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.182608+0000) 2019-10-13T12:43:05.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:05.180+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.183229+0000) 2019-10-13T12:43:05.185 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:05.176+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.183707+0000) 2019-10-13T12:43:05.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:05.180+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.183958+0000) 2019-10-13T12:43:05.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:05.180+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.184524+0000) 2019-10-13T12:43:05.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:05.184+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.185797+0000) 2019-10-13T12:43:05.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:05.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:05.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:05.184+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.191833+0000) 2019-10-13T12:43:05.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:05.672+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:05.678813+0000) 2019-10-13T12:43:07.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:07.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:07.889248+0000) 2019-10-13T12:43:08.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:08.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.114486+0000) 2019-10-13T12:43:08.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:08.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.135675+0000) 2019-10-13T12:43:08.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:08.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:08.655703+0000) 2019-10-13T12:43:10.195 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:10.188+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.193483+0000) 2019-10-13T12:43:10.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:10.192+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.196032+0000) 2019-10-13T12:43:10.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:10.192+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.196188+0000) 2019-10-13T12:43:10.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:10.192+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.196879+0000) 2019-10-13T12:43:10.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:10.192+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.197227+0000) 2019-10-13T12:43:10.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:10.192+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.197360+0000) 2019-10-13T12:43:10.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:10.192+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.197606+0000) 2019-10-13T12:43:10.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:10.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:10.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:10.196+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:10.202220+0000) 2019-10-13T12:43:11.908 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:11.904+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:11.907313+0000) 2019-10-13T12:43:11.921 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:43:11.921 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:11.925 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:43:11.925 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:14.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:14.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:14.232336+0000) 2019-10-13T12:43:14.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:14.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:14.517267+0000) 2019-10-13T12:43:15.206 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:15.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.204459+0000) 2019-10-13T12:43:15.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:15.204+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.206661+0000) 2019-10-13T12:43:15.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:15.204+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.208100+0000) 2019-10-13T12:43:15.211 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:15.204+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.209844+0000) 2019-10-13T12:43:15.211 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:15.208+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.209958+0000) 2019-10-13T12:43:15.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:15.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.211201+0000) 2019-10-13T12:43:15.214 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:15.208+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.212650+0000) 2019-10-13T12:43:15.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:15.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:15.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:15.212+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.216329+0000) 2019-10-13T12:43:15.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:15.684+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:15.685271+0000) 2019-10-13T12:43:17.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:17.884+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:17.889440+0000) 2019-10-13T12:43:18.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:18.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.114727+0000) 2019-10-13T12:43:18.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:18.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.135866+0000) 2019-10-13T12:43:18.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:18.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:18.655895+0000) 2019-10-13T12:43:20.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:20.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.217055+0000) 2019-10-13T12:43:20.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:20.216+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.217966+0000) 2019-10-13T12:43:20.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:20.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.219718+0000) 2019-10-13T12:43:20.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:20.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.220531+0000) 2019-10-13T12:43:20.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:20.216+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.219951+0000) 2019-10-13T12:43:20.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:20.216+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.223246+0000) 2019-10-13T12:43:20.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:20.220+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.223156+0000) 2019-10-13T12:43:20.225 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:20.220+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:20.224169+0000) 2019-10-13T12:43:20.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:20.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:21.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:21.904+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:21.907509+0000) 2019-10-13T12:43:24.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:24.232+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:24.232529+0000) 2019-10-13T12:43:24.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:24.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:24.517465+0000) 2019-10-13T12:43:25.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:25.224+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.228964+0000) 2019-10-13T12:43:25.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:25.224+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.229643+0000) 2019-10-13T12:43:25.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:25.228+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.229733+0000) 2019-10-13T12:43:25.232 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:25.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.230160+0000) 2019-10-13T12:43:25.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:25.228+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.231346+0000) 2019-10-13T12:43:25.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:25.228+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.231699+0000) 2019-10-13T12:43:25.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:25.232+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.236799+0000) 2019-10-13T12:43:25.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:25.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:25.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:25.236+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.240914+0000) 2019-10-13T12:43:25.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:25.684+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:25.691409+0000) 2019-10-13T12:43:27.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:27.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:27.889640+0000) 2019-10-13T12:43:28.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:28.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.114941+0000) 2019-10-13T12:43:28.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:28.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.136080+0000) 2019-10-13T12:43:28.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:28.652+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:28.656086+0000) 2019-10-13T12:43:30.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:30.236+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.240744+0000) 2019-10-13T12:43:30.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:30.236+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.240828+0000) 2019-10-13T12:43:30.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:30.236+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.241446+0000) 2019-10-13T12:43:30.243 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:30.240+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.242176+0000) 2019-10-13T12:43:30.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:30.240+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.242880+0000) 2019-10-13T12:43:30.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:30.240+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.243353+0000) 2019-10-13T12:43:30.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:30.240+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.244956+0000) 2019-10-13T12:43:30.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:30.244+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:30.250109+0000) 2019-10-13T12:43:30.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:30.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:31.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:31.904+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:31.907654+0000) 2019-10-13T12:43:34.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:34.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:34.232731+0000) 2019-10-13T12:43:34.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:34.512+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:34.517664+0000) 2019-10-13T12:43:35.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:35.252+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.253845+0000) 2019-10-13T12:43:35.256 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:35.252+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.254191+0000) 2019-10-13T12:43:35.256 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:35.248+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.254860+0000) 2019-10-13T12:43:35.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:35.252+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.254915+0000) 2019-10-13T12:43:35.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:35.252+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.255823+0000) 2019-10-13T12:43:35.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:35.252+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.256649+0000) 2019-10-13T12:43:35.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:35.252+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.257374+0000) 2019-10-13T12:43:35.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:35.256+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.261813+0000) 2019-10-13T12:43:35.264 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:35.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:35.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:35.700+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:35.702447+0000) 2019-10-13T12:43:37.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:37.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:37.889844+0000) 2019-10-13T12:43:38.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:38.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.115145+0000) 2019-10-13T12:43:38.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:38.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.136277+0000) 2019-10-13T12:43:38.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:38.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:38.656314+0000) 2019-10-13T12:43:40.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:40.264+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.266842+0000) 2019-10-13T12:43:40.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:40.264+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.269060+0000) 2019-10-13T12:43:40.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:40.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.269918+0000) 2019-10-13T12:43:40.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:40.268+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.271465+0000) 2019-10-13T12:43:40.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:40.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.272423+0000) 2019-10-13T12:43:40.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:40.272+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.274408+0000) 2019-10-13T12:43:40.276 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:40.272+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.274891+0000) 2019-10-13T12:43:40.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:40.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:40.277 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:40.268+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:40.275472+0000) 2019-10-13T12:43:41.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:41.904+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:41.907856+0000) 2019-10-13T12:43:42.035 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:43:42.036 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:42.039 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:43:42.039 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:43:44.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:44.228+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:44.232955+0000) 2019-10-13T12:43:44.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:44.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:44.517896+0000) 2019-10-13T12:43:45.279 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:45.276+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.277896+0000) 2019-10-13T12:43:45.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:45.272+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.278601+0000) 2019-10-13T12:43:45.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:45.276+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.279409+0000) 2019-10-13T12:43:45.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:45.276+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.280234+0000) 2019-10-13T12:43:45.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:45.276+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.280288+0000) 2019-10-13T12:43:45.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:45.276+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.280707+0000) 2019-10-13T12:43:45.283 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:45.276+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.281207+0000) 2019-10-13T12:43:45.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:45.280+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.286065+0000) 2019-10-13T12:43:45.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:45.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:45.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:45.708+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:45.708928+0000) 2019-10-13T12:43:47.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:47.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:47.890061+0000) 2019-10-13T12:43:48.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:48.112+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.115347+0000) 2019-10-13T12:43:48.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:48.132+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.136503+0000) 2019-10-13T12:43:48.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:48.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:48.656515+0000) 2019-10-13T12:43:50.291 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:50.288+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.290270+0000) 2019-10-13T12:43:50.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:50.287+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.290845+0000) 2019-10-13T12:43:50.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:50.287+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.291696+0000) 2019-10-13T12:43:50.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:50.287+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.292333+0000) 2019-10-13T12:43:50.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:50.287+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.292610+0000) 2019-10-13T12:43:50.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:50.287+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.293246+0000) 2019-10-13T12:43:50.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:50.291+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.293840+0000) 2019-10-13T12:43:50.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:50.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:50.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:50.296+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:50.299828+0000) 2019-10-13T12:43:51.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:51.908+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:51.908071+0000) 2019-10-13T12:43:54.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:54.227+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:54.233191+0000) 2019-10-13T12:43:54.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:54.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:54.518126+0000) 2019-10-13T12:43:55.303 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:55.299+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.301834+0000) 2019-10-13T12:43:55.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:43:55.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.302761+0000) 2019-10-13T12:43:55.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:55.299+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.303526+0000) 2019-10-13T12:43:55.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:55.299+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.303713+0000) 2019-10-13T12:43:55.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:43:55.303+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.304255+0000) 2019-10-13T12:43:55.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:55.299+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.304443+0000) 2019-10-13T12:43:55.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:43:55.299+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.304706+0000) 2019-10-13T12:43:55.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:43:55.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:43:55.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:55.311+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.312502+0000) 2019-10-13T12:43:55.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:43:55.711+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:55.715384+0000) 2019-10-13T12:43:57.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:43:57.887+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:57.890255+0000) 2019-10-13T12:43:58.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:43:58.111+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.115510+0000) 2019-10-13T12:43:58.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:43:58.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.136735+0000) 2019-10-13T12:43:58.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:43:58.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:43:58.656719+0000) 2019-10-13T12:44:00.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:00.311+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.315168+0000) 2019-10-13T12:44:00.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:00.311+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.315574+0000) 2019-10-13T12:44:00.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:00.311+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.316124+0000) 2019-10-13T12:44:00.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:00.311+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.317141+0000) 2019-10-13T12:44:00.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:00.315+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.317999+0000) 2019-10-13T12:44:00.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:00.315+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.318484+0000) 2019-10-13T12:44:00.320 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:00.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.319234+0000) 2019-10-13T12:44:00.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:00.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:00.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:00.319+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:00.323372+0000) 2019-10-13T12:44:01.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:01.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:01.908262+0000) 2019-10-13T12:44:04.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:04.231+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:04.233428+0000) 2019-10-13T12:44:04.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:04.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:04.518329+0000) 2019-10-13T12:44:05.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:05.323+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.326550+0000) 2019-10-13T12:44:05.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:05.323+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.327868+0000) 2019-10-13T12:44:05.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:05.323+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.328064+0000) 2019-10-13T12:44:05.330 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:05.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.328355+0000) 2019-10-13T12:44:05.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:05.327+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.329051+0000) 2019-10-13T12:44:05.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:05.323+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.329232+0000) 2019-10-13T12:44:05.331 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:05.327+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.329962+0000) 2019-10-13T12:44:05.335 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:05.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:05.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:05.335+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.336337+0000) 2019-10-13T12:44:05.723 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:05.719+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:05.721743+0000) 2019-10-13T12:44:07.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:07.887+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:07.890464+0000) 2019-10-13T12:44:08.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:08.111+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.115731+0000) 2019-10-13T12:44:08.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:08.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.136953+0000) 2019-10-13T12:44:08.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:08.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:08.656911+0000) 2019-10-13T12:44:10.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:10.331+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.337591+0000) 2019-10-13T12:44:10.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:10.335+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.338081+0000) 2019-10-13T12:44:10.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:10.335+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.338562+0000) 2019-10-13T12:44:10.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:10.335+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.339429+0000) 2019-10-13T12:44:10.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:10.335+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.340420+0000) 2019-10-13T12:44:10.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:10.335+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.340729+0000) 2019-10-13T12:44:10.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:10.335+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.341261+0000) 2019-10-13T12:44:10.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:10.343+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:10.346119+0000) 2019-10-13T12:44:10.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:10.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167137 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:11.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:11.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:11.908473+0000) 2019-10-13T12:44:12.090 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:44:12.091 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:12.095 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:44:12.095 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:14.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:14.231+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:14.233631+0000) 2019-10-13T12:44:14.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:14.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:14.518534+0000) 2019-10-13T12:44:15.353 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:15.347+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.351112+0000) 2019-10-13T12:44:15.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:15.347+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.351713+0000) 2019-10-13T12:44:15.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:15.351+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.353767+0000) 2019-10-13T12:44:15.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:15.347+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.353183+0000) 2019-10-13T12:44:15.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:15.351+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.355713+0000) 2019-10-13T12:44:15.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:15.351+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.356258+0000) 2019-10-13T12:44:15.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:15.351+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.356941+0000) 2019-10-13T12:44:15.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:15.355+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:15.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:15.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.361130+0000) 2019-10-13T12:44:15.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:15.727+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:15.728431+0000) 2019-10-13T12:44:17.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:17.887+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:17.890724+0000) 2019-10-13T12:44:18.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:18.115+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.115930+0000) 2019-10-13T12:44:18.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:18.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.137182+0000) 2019-10-13T12:44:18.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:18.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:18.657110+0000) 2019-10-13T12:44:20.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:20.359+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.362390+0000) 2019-10-13T12:44:20.367 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:20.359+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.365477+0000) 2019-10-13T12:44:20.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:20.363+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.365795+0000) 2019-10-13T12:44:20.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:20.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.366379+0000) 2019-10-13T12:44:20.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:20.363+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.367420+0000) 2019-10-13T12:44:20.369 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:20.363+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.368013+0000) 2019-10-13T12:44:20.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:20.363+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.368562+0000) 2019-10-13T12:44:20.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:20.367+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:20.372 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:20.367+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:20.371377+0000) 2019-10-13T12:44:21.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:21.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:21.908667+0000) 2019-10-13T12:44:24.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:24.231+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:24.233839+0000) 2019-10-13T12:44:24.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:24.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:24.518746+0000) 2019-10-13T12:44:25.375 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:25.371+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.374018+0000) 2019-10-13T12:44:25.378 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:25.371+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.377109+0000) 2019-10-13T12:44:25.379 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:25.375+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.377700+0000) 2019-10-13T12:44:25.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:25.375+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.381130+0000) 2019-10-13T12:44:25.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:25.379+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:25.384 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:25.379+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.382454+0000) 2019-10-13T12:44:25.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:25.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.383108+0000) 2019-10-13T12:44:25.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:25.379+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.382800+0000) 2019-10-13T12:44:25.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:25.387+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.388288+0000) 2019-10-13T12:44:25.736 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:25.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:25.735032+0000) 2019-10-13T12:44:27.892 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:27.887+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:27.890927+0000) 2019-10-13T12:44:28.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:28.111+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.116128+0000) 2019-10-13T12:44:28.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:28.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.137383+0000) 2019-10-13T12:44:28.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:28.651+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:28.657293+0000) 2019-10-13T12:44:30.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:30.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.385531+0000) 2019-10-13T12:44:30.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:30.383+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.386252+0000) 2019-10-13T12:44:30.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:30.383+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.386583+0000) 2019-10-13T12:44:30.388 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:30.383+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.387056+0000) 2019-10-13T12:44:30.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:30.383+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.387563+0000) 2019-10-13T12:44:30.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:30.383+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.388108+0000) 2019-10-13T12:44:30.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:30.383+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.389662+0000) 2019-10-13T12:44:30.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:30.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:30.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:30.391+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:30.395880+0000) 2019-10-13T12:44:31.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:31.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:31.908890+0000) 2019-10-13T12:44:34.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:34.231+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:34.234043+0000) 2019-10-13T12:44:34.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:34.515+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:34.518941+0000) 2019-10-13T12:44:35.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:35.391+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.395441+0000) 2019-10-13T12:44:35.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:35.391+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.396742+0000) 2019-10-13T12:44:35.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:35.391+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.397031+0000) 2019-10-13T12:44:35.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:35.391+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.397417+0000) 2019-10-13T12:44:35.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:35.395+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.398157+0000) 2019-10-13T12:44:35.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:35.395+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.398092+0000) 2019-10-13T12:44:35.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:35.395+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.399198+0000) 2019-10-13T12:44:35.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:35.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:35.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:35.403+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.404803+0000) 2019-10-13T12:44:35.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:35.739+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:35.741394+0000) 2019-10-13T12:44:37.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:37.887+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:37.891145+0000) 2019-10-13T12:44:38.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:38.115+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.116331+0000) 2019-10-13T12:44:38.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:38.131+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.137578+0000) 2019-10-13T12:44:38.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:38.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:38.657525+0000) 2019-10-13T12:44:40.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:40.403+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.407094+0000) 2019-10-13T12:44:40.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:40.403+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.407652+0000) 2019-10-13T12:44:40.411 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:40.403+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.409318+0000) 2019-10-13T12:44:40.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:40.403+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.409517+0000) 2019-10-13T12:44:40.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:40.403+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.409536+0000) 2019-10-13T12:44:40.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:40.407+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.410228+0000) 2019-10-13T12:44:40.412 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:40.407+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.410891+0000) 2019-10-13T12:44:40.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:40.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:40.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:40.415+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:40.418203+0000) 2019-10-13T12:44:41.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:41.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:41.909121+0000) 2019-10-13T12:44:42.196 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:44:42.196 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:42.200 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:44:42.200 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:44:44.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:44.231+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:44.234244+0000) 2019-10-13T12:44:44.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:44.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:44.519175+0000) 2019-10-13T12:44:45.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:45.414+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.419750+0000) 2019-10-13T12:44:45.423 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:45.418+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.422267+0000) 2019-10-13T12:44:45.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:45.423+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.424390+0000) 2019-10-13T12:44:45.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:45.423+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.425283+0000) 2019-10-13T12:44:45.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:45.422+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.425832+0000) 2019-10-13T12:44:45.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:45.422+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:45.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:45.422+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.428271+0000) 2019-10-13T12:44:45.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:45.422+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.428808+0000) 2019-10-13T12:44:45.431 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:45.422+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.429213+0000) 2019-10-13T12:44:45.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:45.751+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:45.752395+0000) 2019-10-13T12:44:47.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:47.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:47.891358+0000) 2019-10-13T12:44:48.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:48.110+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.116531+0000) 2019-10-13T12:44:48.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:48.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.137791+0000) 2019-10-13T12:44:48.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:48.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:48.657721+0000) 2019-10-13T12:44:50.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:50.427+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.432053+0000) 2019-10-13T12:44:50.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:50.426+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.432406+0000) 2019-10-13T12:44:50.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:50.426+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.432773+0000) 2019-10-13T12:44:50.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:50.430+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.433735+0000) 2019-10-13T12:44:50.435 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:50.430+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.434172+0000) 2019-10-13T12:44:50.436 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:50.430+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.434617+0000) 2019-10-13T12:44:50.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:50.430+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.435158+0000) 2019-10-13T12:44:50.438 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:50.435+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:50.437255+0000) 2019-10-13T12:44:50.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:50.434+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167385 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:51.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:51.907+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:51.909316+0000) 2019-10-13T12:44:54.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:54.230+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:54.234453+0000) 2019-10-13T12:44:54.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:54.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:54.519408+0000) 2019-10-13T12:44:55.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:44:55.442+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.446109+0000) 2019-10-13T12:44:55.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:44:55.442+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.447227+0000) 2019-10-13T12:44:55.450 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:55.442+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.448863+0000) 2019-10-13T12:44:55.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:55.442+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.449185+0000) 2019-10-13T12:44:55.451 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:44:55.447+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.450229+0000) 2019-10-13T12:44:55.452 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:55.442+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.449585+0000) 2019-10-13T12:44:55.452 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:55.446+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.451138+0000) 2019-10-13T12:44:55.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:55.451+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.452273+0000) 2019-10-13T12:44:55.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:44:55.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:44:55.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:44:55.759+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:55.760435+0000) 2019-10-13T12:44:57.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:44:57.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:57.891560+0000) 2019-10-13T12:44:58.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:44:58.110+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.116731+0000) 2019-10-13T12:44:58.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:44:58.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.137955+0000) 2019-10-13T12:44:58.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:44:58.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:44:58.657956+0000) 2019-10-13T12:45:00.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:00.450+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.456299+0000) 2019-10-13T12:45:00.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:00.450+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.456858+0000) 2019-10-13T12:45:00.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:00.454+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.459540+0000) 2019-10-13T12:45:00.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:00.454+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.461300+0000) 2019-10-13T12:45:00.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:00.458+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.463219+0000) 2019-10-13T12:45:00.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:00.462+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.464197+0000) 2019-10-13T12:45:00.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:00.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167446 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:00.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:00.458+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.465077+0000) 2019-10-13T12:45:00.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:00.462+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:00.465547+0000) 2019-10-13T12:45:01.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:01.906+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:01.909499+0000) 2019-10-13T12:45:04.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:04.230+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:04.234684+0000) 2019-10-13T12:45:04.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:04.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:04.519607+0000) 2019-10-13T12:45:05.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:05.462+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.468111+0000) 2019-10-13T12:45:05.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:05.462+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.469001+0000) 2019-10-13T12:45:05.472 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:05.466+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.470860+0000) 2019-10-13T12:45:05.473 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:05.466+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.472141+0000) 2019-10-13T12:45:05.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:05.466+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.472387+0000) 2019-10-13T12:45:05.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:05.466+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.473117+0000) 2019-10-13T12:45:05.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:05.470+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.475315+0000) 2019-10-13T12:45:05.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:05.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167476 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:05.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:05.474+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.480029+0000) 2019-10-13T12:45:05.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:05.762+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:05.766451+0000) 2019-10-13T12:45:07.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:07.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:07.891775+0000) 2019-10-13T12:45:08.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:08.110+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:08.116931+0000) 2019-10-13T12:45:08.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:08.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:08.138174+0000) 2019-10-13T12:45:08.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:08.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:08.658158+0000) 2019-10-13T12:45:10.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:10.474+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.479985+0000) 2019-10-13T12:45:10.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:10.478+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.481938+0000) 2019-10-13T12:45:10.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:10.478+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.482327+0000) 2019-10-13T12:45:10.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:10.478+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.482930+0000) 2019-10-13T12:45:10.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:10.478+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.483900+0000) 2019-10-13T12:45:10.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:10.478+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.484550+0000) 2019-10-13T12:45:10.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:10.478+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.484874+0000) 2019-10-13T12:45:10.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:10.482+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:10.487513+0000) 2019-10-13T12:45:10.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:10.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:11.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:11.906+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:11.909689+0000) 2019-10-13T12:45:12.243 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:45:12.243 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:12.246 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:45:12.247 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:14.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:14.234+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.234908+0000) 2019-10-13T12:45:14.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:14.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:14.519809+0000) 2019-10-13T12:45:15.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:15.486+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.491749+0000) 2019-10-13T12:45:15.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:15.486+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.492486+0000) 2019-10-13T12:45:15.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:15.490+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.493159+0000) 2019-10-13T12:45:15.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:15.490+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.494316+0000) 2019-10-13T12:45:15.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:15.490+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.494765+0000) 2019-10-13T12:45:15.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:15.490+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.495002+0000) 2019-10-13T12:45:15.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:15.490+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.495471+0000) 2019-10-13T12:45:15.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:15.498+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.500288+0000) 2019-10-13T12:45:15.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:15.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:15.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:15.774+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:15.775314+0000) 2019-10-13T12:45:17.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:17.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:17.891932+0000) 2019-10-13T12:45:18.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:18.114+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:18.117127+0000) 2019-10-13T12:45:18.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:18.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:18.138387+0000) 2019-10-13T12:45:18.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:18.654+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:18.658370+0000) 2019-10-13T12:45:20.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:20.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.504377+0000) 2019-10-13T12:45:20.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:20.498+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.504627+0000) 2019-10-13T12:45:20.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:20.502+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.505797+0000) 2019-10-13T12:45:20.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:20.502+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.506082+0000) 2019-10-13T12:45:20.508 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:20.502+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.506495+0000) 2019-10-13T12:45:20.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:20.502+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.507188+0000) 2019-10-13T12:45:20.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:20.502+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.507857+0000) 2019-10-13T12:45:20.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:20.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:20.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:20.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:20.513449+0000) 2019-10-13T12:45:21.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:21.906+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:21.909918+0000) 2019-10-13T12:45:24.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:24.234+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.235117+0000) 2019-10-13T12:45:24.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:24.514+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:24.520033+0000) 2019-10-13T12:45:25.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:25.510+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.515520+0000) 2019-10-13T12:45:25.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:25.510+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.516761+0000) 2019-10-13T12:45:25.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:25.510+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.517660+0000) 2019-10-13T12:45:25.519 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:25.514+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.518059+0000) 2019-10-13T12:45:25.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:25.514+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.518936+0000) 2019-10-13T12:45:25.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:25.514+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.519386+0000) 2019-10-13T12:45:25.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:25.514+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.519835+0000) 2019-10-13T12:45:25.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:25.518+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.524073+0000) 2019-10-13T12:45:25.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:25.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:25.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:25.778+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:25.781992+0000) 2019-10-13T12:45:27.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:27.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:27.892125+0000) 2019-10-13T12:45:28.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:28.114+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:28.117337+0000) 2019-10-13T12:45:28.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:28.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:28.138599+0000) 2019-10-13T12:45:28.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:28.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:28.658584+0000) 2019-10-13T12:45:30.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:30.522+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.527949+0000) 2019-10-13T12:45:30.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:30.526+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.530415+0000) 2019-10-13T12:45:30.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:30.526+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.529944+0000) 2019-10-13T12:45:30.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:30.526+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.530741+0000) 2019-10-13T12:45:30.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:30.526+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.531329+0000) 2019-10-13T12:45:30.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:30.526+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.531572+0000) 2019-10-13T12:45:30.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:30.526+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.533602+0000) 2019-10-13T12:45:30.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:30.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:30.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:30.534+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:30.536672+0000) 2019-10-13T12:45:31.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:31.906+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:31.910147+0000) 2019-10-13T12:45:34.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:34.234+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.235312+0000) 2019-10-13T12:45:34.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:34.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:34.520262+0000) 2019-10-13T12:45:35.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:35.534+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.539849+0000) 2019-10-13T12:45:35.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:35.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.540794+0000) 2019-10-13T12:45:35.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:35.534+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.540937+0000) 2019-10-13T12:45:35.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:35.534+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.541519+0000) 2019-10-13T12:45:35.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:35.538+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.542127+0000) 2019-10-13T12:45:35.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:35.538+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.543250+0000) 2019-10-13T12:45:35.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:35.538+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.543766+0000) 2019-10-13T12:45:35.550 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:35.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:35.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:35.546+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.549332+0000) 2019-10-13T12:45:35.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:35.786+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:35.788486+0000) 2019-10-13T12:45:37.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:37.886+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:37.892372+0000) 2019-10-13T12:45:38.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:38.114+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:38.117529+0000) 2019-10-13T12:45:38.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:38.134+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:38.138856+0000) 2019-10-13T12:45:38.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:38.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:38.658782+0000) 2019-10-13T12:45:40.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:40.546+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.551601+0000) 2019-10-13T12:45:40.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:40.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.552029+0000) 2019-10-13T12:45:40.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:40.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.552570+0000) 2019-10-13T12:45:40.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:40.550+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.554681+0000) 2019-10-13T12:45:40.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:40.550+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.555108+0000) 2019-10-13T12:45:40.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:40.550+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.555965+0000) 2019-10-13T12:45:40.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:40.550+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.557268+0000) 2019-10-13T12:45:40.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:40.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:40.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:40.558+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:40.562744+0000) 2019-10-13T12:45:41.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:41.906+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:41.910341+0000) 2019-10-13T12:45:42.490 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:45:42.491 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:42.495 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:45:42.495 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:45:44.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:44.229+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.235507+0000) 2019-10-13T12:45:44.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:44.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:44.520421+0000) 2019-10-13T12:45:45.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:45.558+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.562856+0000) 2019-10-13T12:45:45.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:45.557+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.563200+0000) 2019-10-13T12:45:45.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:45.557+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.563954+0000) 2019-10-13T12:45:45.567 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:45.561+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.566046+0000) 2019-10-13T12:45:45.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:45.561+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.566420+0000) 2019-10-13T12:45:45.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:45.561+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.567176+0000) 2019-10-13T12:45:45.569 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:45.561+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.567578+0000) 2019-10-13T12:45:45.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:45.566+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.571014+0000) 2019-10-13T12:45:45.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:45.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:45.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:45.790+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:45.794980+0000) 2019-10-13T12:45:47.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:47.889+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:47.892605+0000) 2019-10-13T12:45:48.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:48.109+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:48.117719+0000) 2019-10-13T12:45:48.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:48.133+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:48.139061+0000) 2019-10-13T12:45:48.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:48.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:48.659012+0000) 2019-10-13T12:45:50.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:50.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.575005+0000) 2019-10-13T12:45:50.577 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:50.569+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.575864+0000) 2019-10-13T12:45:50.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:50.569+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.576026+0000) 2019-10-13T12:45:50.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:50.569+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.576376+0000) 2019-10-13T12:45:50.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:50.569+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.577185+0000) 2019-10-13T12:45:50.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:50.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.577758+0000) 2019-10-13T12:45:50.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:50.573+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.578346+0000) 2019-10-13T12:45:50.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:50.578+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:50.583385+0000) 2019-10-13T12:45:50.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:50.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:51.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:51.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:51.910537+0000) 2019-10-13T12:45:54.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:54.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.235711+0000) 2019-10-13T12:45:54.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:54.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:54.520645+0000) 2019-10-13T12:45:55.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:55.585+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.587152+0000) 2019-10-13T12:45:55.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:55.585+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.587732+0000) 2019-10-13T12:45:55.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:45:55.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.587525+0000) 2019-10-13T12:45:55.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:45:55.585+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.588557+0000) 2019-10-13T12:45:55.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:55.585+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.589109+0000) 2019-10-13T12:45:55.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:45:55.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.590052+0000) 2019-10-13T12:45:55.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:55.585+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.588997+0000) 2019-10-13T12:45:55.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:45:55.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:45:55.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:55.594+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.596492+0000) 2019-10-13T12:45:55.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:45:55.798+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:55.801575+0000) 2019-10-13T12:45:57.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:45:57.889+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:57.892843+0000) 2019-10-13T12:45:58.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:45:58.117+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:58.117954+0000) 2019-10-13T12:45:58.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:45:58.137+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:58.139253+0000) 2019-10-13T12:45:58.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:45:58.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:45:58.659207+0000) 2019-10-13T12:46:00.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:00.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.597683+0000) 2019-10-13T12:46:00.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:00.597+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.598074+0000) 2019-10-13T12:46:00.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:00.597+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.599862+0000) 2019-10-13T12:46:00.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:00.601+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.602919+0000) 2019-10-13T12:46:00.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:00.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.603091+0000) 2019-10-13T12:46:00.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:00.601+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.604785+0000) 2019-10-13T12:46:00.606 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:00.602+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.605404+0000) 2019-10-13T12:46:00.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:00.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:00.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:00.610+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:00.613085+0000) 2019-10-13T12:46:01.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:01.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:01.910772+0000) 2019-10-13T12:46:04.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:04.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.235906+0000) 2019-10-13T12:46:04.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:04.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:04.520839+0000) 2019-10-13T12:46:05.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:05.605+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.609597+0000) 2019-10-13T12:46:05.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:05.609+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.610222+0000) 2019-10-13T12:46:05.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:05.605+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.611202+0000) 2019-10-13T12:46:05.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:05.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.611423+0000) 2019-10-13T12:46:05.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:05.609+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.612052+0000) 2019-10-13T12:46:05.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:05.609+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.613101+0000) 2019-10-13T12:46:05.615 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:05.613+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.613903+0000) 2019-10-13T12:46:05.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:05.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167848 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:05.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:05.613+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.620064+0000) 2019-10-13T12:46:05.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:05.805+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:05.808044+0000) 2019-10-13T12:46:07.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:07.889+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:07.893071+0000) 2019-10-13T12:46:08.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:08.117+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:08.118153+0000) 2019-10-13T12:46:08.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:08.137+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:08.139480+0000) 2019-10-13T12:46:08.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:08.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:08.659400+0000) 2019-10-13T12:46:10.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:10.617+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.621434+0000) 2019-10-13T12:46:10.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:10.621+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.621883+0000) 2019-10-13T12:46:10.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:10.621+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.622702+0000) 2019-10-13T12:46:10.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:10.621+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.622420+0000) 2019-10-13T12:46:10.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:10.621+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.623042+0000) 2019-10-13T12:46:10.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:10.621+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.624698+0000) 2019-10-13T12:46:10.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:10.621+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.625066+0000) 2019-10-13T12:46:10.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:10.625+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:10.628534+0000) 2019-10-13T12:46:10.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:10.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:11.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:11.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:11.910996+0000) 2019-10-13T12:46:12.563 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:46:12.563 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:12.566 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:46:12.567 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:14.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:14.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.236131+0000) 2019-10-13T12:46:14.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:14.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:14.521036+0000) 2019-10-13T12:46:15.636 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:15.629+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.633165+0000) 2019-10-13T12:46:15.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:15.633+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.634033+0000) 2019-10-13T12:46:15.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:15.633+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.635750+0000) 2019-10-13T12:46:15.639 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:15.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.637498+0000) 2019-10-13T12:46:15.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:15.633+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.639411+0000) 2019-10-13T12:46:15.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:15.637+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.639999+0000) 2019-10-13T12:46:15.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:15.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.641222+0000) 2019-10-13T12:46:15.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:15.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:15.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:15.641+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.647799+0000) 2019-10-13T12:46:15.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:15.813+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:15.814688+0000) 2019-10-13T12:46:17.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:17.889+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:17.893273+0000) 2019-10-13T12:46:18.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:18.117+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:18.118349+0000) 2019-10-13T12:46:18.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:18.137+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:18.139682+0000) 2019-10-13T12:46:18.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:18.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:18.659621+0000) 2019-10-13T12:46:20.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:20.641+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.644478+0000) 2019-10-13T12:46:20.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:20.641+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.645515+0000) 2019-10-13T12:46:20.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:20.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.646292+0000) 2019-10-13T12:46:20.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:20.641+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.646668+0000) 2019-10-13T12:46:20.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:20.645+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.647314+0000) 2019-10-13T12:46:20.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:20.645+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.647705+0000) 2019-10-13T12:46:20.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:20.645+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.648134+0000) 2019-10-13T12:46:20.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:20.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:20.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:20.649+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:20.653086+0000) 2019-10-13T12:46:21.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:21.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:21.911191+0000) 2019-10-13T12:46:24.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:24.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.236359+0000) 2019-10-13T12:46:24.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:24.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:24.521250+0000) 2019-10-13T12:46:25.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:25.653+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.656276+0000) 2019-10-13T12:46:25.658 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:25.653+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.656919+0000) 2019-10-13T12:46:25.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:25.653+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.657542+0000) 2019-10-13T12:46:25.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:25.657+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.659390+0000) 2019-10-13T12:46:25.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:25.657+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.661856+0000) 2019-10-13T12:46:25.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:25.661+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.663494+0000) 2019-10-13T12:46:25.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:25.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:25.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:25.665+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.665969+0000) 2019-10-13T12:46:25.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:25.669+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.672446+0000) 2019-10-13T12:46:25.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:25.817+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:25.821305+0000) 2019-10-13T12:46:27.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:27.889+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:27.893509+0000) 2019-10-13T12:46:28.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:28.117+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:28.118581+0000) 2019-10-13T12:46:28.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:28.137+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:28.139890+0000) 2019-10-13T12:46:28.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:28.657+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:28.659815+0000) 2019-10-13T12:46:30.670 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:30.665+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.668858+0000) 2019-10-13T12:46:30.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:30.665+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.669047+0000) 2019-10-13T12:46:30.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:30.665+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.669649+0000) 2019-10-13T12:46:30.672 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:30.669+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.669890+0000) 2019-10-13T12:46:30.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:30.669+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.670936+0000) 2019-10-13T12:46:30.672 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:30.669+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.670566+0000) 2019-10-13T12:46:30.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:30.669+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.671665+0000) 2019-10-13T12:46:30.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:30.677+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:30.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:30.673+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:30.678652+0000) 2019-10-13T12:46:31.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:31.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:31.911393+0000) 2019-10-13T12:46:34.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:34.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.236593+0000) 2019-10-13T12:46:34.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:34.517+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:34.521476+0000) 2019-10-13T12:46:35.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:35.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.681252+0000) 2019-10-13T12:46:35.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:35.677+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.682172+0000) 2019-10-13T12:46:35.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:35.677+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.681670+0000) 2019-10-13T12:46:35.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:35.681+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.682224+0000) 2019-10-13T12:46:35.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:35.681+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.683013+0000) 2019-10-13T12:46:35.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:35.681+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.683391+0000) 2019-10-13T12:46:35.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:35.681+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.684445+0000) 2019-10-13T12:46:35.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:35.689+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:35.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:35.685+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.690377+0000) 2019-10-13T12:46:35.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:35.825+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:35.827749+0000) 2019-10-13T12:46:37.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:37.888+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:37.893700+0000) 2019-10-13T12:46:38.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:38.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:38.118781+0000) 2019-10-13T12:46:38.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:38.136+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:38.140101+0000) 2019-10-13T12:46:38.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:38.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:38.660018+0000) 2019-10-13T12:46:40.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:40.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.693599+0000) 2019-10-13T12:46:40.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:40.689+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.694562+0000) 2019-10-13T12:46:40.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:40.692+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.694017+0000) 2019-10-13T12:46:40.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:40.692+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.694626+0000) 2019-10-13T12:46:40.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:40.692+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.696063+0000) 2019-10-13T12:46:40.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:40.692+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.696738+0000) 2019-10-13T12:46:40.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:40.692+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.697126+0000) 2019-10-13T12:46:40.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:40.697+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:40.701658+0000) 2019-10-13T12:46:40.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:40.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168065 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:41.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:41.909+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:41.911594+0000) 2019-10-13T12:46:42.695 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:46:42.695 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:42.698 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:46:42.698 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:46:44.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:44.232+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.236798+0000) 2019-10-13T12:46:44.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:44.516+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:44.521673+0000) 2019-10-13T12:46:45.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:45.700+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.704886+0000) 2019-10-13T12:46:45.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:45.701+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.706223+0000) 2019-10-13T12:46:45.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:45.704+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.707099+0000) 2019-10-13T12:46:45.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:45.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.707095+0000) 2019-10-13T12:46:45.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:45.704+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.707608+0000) 2019-10-13T12:46:45.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:45.704+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.708183+0000) 2019-10-13T12:46:45.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:45.704+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.708896+0000) 2019-10-13T12:46:45.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:45.712+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:45.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:45.709+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.715713+0000) 2019-10-13T12:46:45.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:45.833+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:45.834305+0000) 2019-10-13T12:46:47.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:47.892+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:47.893926+0000) 2019-10-13T12:46:48.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:48.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:48.118976+0000) 2019-10-13T12:46:48.141 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:48.136+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:48.140297+0000) 2019-10-13T12:46:48.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:48.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:48.660213+0000) 2019-10-13T12:46:50.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:50.712+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.717255+0000) 2019-10-13T12:46:50.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:50.712+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.717692+0000) 2019-10-13T12:46:50.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:50.716+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.720168+0000) 2019-10-13T12:46:50.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:50.716+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.720751+0000) 2019-10-13T12:46:50.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:50.717+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.721358+0000) 2019-10-13T12:46:50.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:50.716+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.721074+0000) 2019-10-13T12:46:50.723 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:50.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.721989+0000) 2019-10-13T12:46:50.726 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:50.721+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:50.725098+0000) 2019-10-13T12:46:50.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:50.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:51.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:51.905+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:51.911821+0000) 2019-10-13T12:46:54.238 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:54.236+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.237010+0000) 2019-10-13T12:46:54.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:54.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:54.521903+0000) 2019-10-13T12:46:55.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:55.724+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.729108+0000) 2019-10-13T12:46:55.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:46:55.724+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.729269+0000) 2019-10-13T12:46:55.732 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:46:55.728+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.730849+0000) 2019-10-13T12:46:55.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:55.728+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.730921+0000) 2019-10-13T12:46:55.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:55.728+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.731369+0000) 2019-10-13T12:46:55.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:46:55.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.731958+0000) 2019-10-13T12:46:55.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:55.728+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.732560+0000) 2019-10-13T12:46:55.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:46:55.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:46:55.740 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:55.733+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.738672+0000) 2019-10-13T12:46:55.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:46:55.837+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:55.840721+0000) 2019-10-13T12:46:57.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:46:57.892+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:57.894153+0000) 2019-10-13T12:46:58.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:46:58.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:58.119235+0000) 2019-10-13T12:46:58.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:46:58.136+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:58.140483+0000) 2019-10-13T12:46:58.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:46:58.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:46:58.660406+0000) 2019-10-13T12:47:00.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:00.737+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.741237+0000) 2019-10-13T12:47:00.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:00.736+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.741330+0000) 2019-10-13T12:47:00.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:00.740+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.742067+0000) 2019-10-13T12:47:00.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:00.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.741930+0000) 2019-10-13T12:47:00.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:00.740+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.742415+0000) 2019-10-13T12:47:00.745 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:00.740+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.743776+0000) 2019-10-13T12:47:00.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:00.740+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.744690+0000) 2019-10-13T12:47:00.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:00.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:00.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:00.745+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:00.749525+0000) 2019-10-13T12:47:01.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:01.909+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:01.912020+0000) 2019-10-13T12:47:04.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:04.232+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.237240+0000) 2019-10-13T12:47:04.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:04.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:04.522106+0000) 2019-10-13T12:47:05.754 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:05.748+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.752669+0000) 2019-10-13T12:47:05.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:05.748+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.752939+0000) 2019-10-13T12:47:05.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:05.748+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.753683+0000) 2019-10-13T12:47:05.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:05.752+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.757005+0000) 2019-10-13T12:47:05.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:05.756+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.759235+0000) 2019-10-13T12:47:05.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:05.757+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.761464+0000) 2019-10-13T12:47:05.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:05.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:05.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:05.760+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.762198+0000) 2019-10-13T12:47:05.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:05.757+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.763875+0000) 2019-10-13T12:47:05.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:05.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:05.847236+0000) 2019-10-13T12:47:07.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:07.892+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:07.894338+0000) 2019-10-13T12:47:08.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:08.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:08.119438+0000) 2019-10-13T12:47:08.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:08.140+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:08.140685+0000) 2019-10-13T12:47:08.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:08.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:08.660594+0000) 2019-10-13T12:47:10.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:10.764+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.767729+0000) 2019-10-13T12:47:10.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:10.768+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.769725+0000) 2019-10-13T12:47:10.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:10.768+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.770012+0000) 2019-10-13T12:47:10.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:10.768+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.770272+0000) 2019-10-13T12:47:10.772 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:10.768+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.770803+0000) 2019-10-13T12:47:10.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:10.764+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.771274+0000) 2019-10-13T12:47:10.773 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:10.768+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.771341+0000) 2019-10-13T12:47:10.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:10.768+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:10.773171+0000) 2019-10-13T12:47:10.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:10.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:11.913 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:11.908+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:11.912277+0000) 2019-10-13T12:47:12.814 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:47:12.814 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:12.818 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:47:12.818 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:14.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:14.236+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.237457+0000) 2019-10-13T12:47:14.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:14.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:14.522261+0000) 2019-10-13T12:47:15.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:15.772+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.776758+0000) 2019-10-13T12:47:15.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:15.772+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.777437+0000) 2019-10-13T12:47:15.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:15.772+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.777678+0000) 2019-10-13T12:47:15.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:15.776+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.778140+0000) 2019-10-13T12:47:15.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:15.776+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.778523+0000) 2019-10-13T12:47:15.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:15.776+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.779043+0000) 2019-10-13T12:47:15.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:15.776+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.779845+0000) 2019-10-13T12:47:15.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:15.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:15.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:15.784+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.788695+0000) 2019-10-13T12:47:15.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:15.848+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:15.853443+0000) 2019-10-13T12:47:17.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:17.892+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:17.894538+0000) 2019-10-13T12:47:18.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:18.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:18.119652+0000) 2019-10-13T12:47:18.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:18.140+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:18.140889+0000) 2019-10-13T12:47:18.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:18.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:18.660823+0000) 2019-10-13T12:47:20.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:20.788+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.791417+0000) 2019-10-13T12:47:20.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:20.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.791565+0000) 2019-10-13T12:47:20.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:20.788+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.791986+0000) 2019-10-13T12:47:20.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:20.788+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.792815+0000) 2019-10-13T12:47:20.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:20.788+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.793393+0000) 2019-10-13T12:47:20.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:20.788+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.793496+0000) 2019-10-13T12:47:20.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:20.792+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.794577+0000) 2019-10-13T12:47:20.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:20.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:20.802 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:20.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:20.800856+0000) 2019-10-13T12:47:21.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:21.908+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:21.912503+0000) 2019-10-13T12:47:24.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:24.236+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.237684+0000) 2019-10-13T12:47:24.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:24.520+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:24.522503+0000) 2019-10-13T12:47:25.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:25.796+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.801747+0000) 2019-10-13T12:47:25.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:25.800+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.802031+0000) 2019-10-13T12:47:25.805 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:25.800+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.803214+0000) 2019-10-13T12:47:25.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:25.800+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.803427+0000) 2019-10-13T12:47:25.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:25.800+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.804578+0000) 2019-10-13T12:47:25.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:25.800+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.804385+0000) 2019-10-13T12:47:25.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:25.800+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.804943+0000) 2019-10-13T12:47:25.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:25.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168346 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:25.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:25.804+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.810402+0000) 2019-10-13T12:47:25.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:25.856+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:25.860127+0000) 2019-10-13T12:47:27.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:27.892+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:27.894745+0000) 2019-10-13T12:47:28.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:28.116+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:28.119861+0000) 2019-10-13T12:47:28.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:28.136+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:28.141082+0000) 2019-10-13T12:47:28.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:28.656+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:28.661048+0000) 2019-10-13T12:47:30.815 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:30.812+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.813820+0000) 2019-10-13T12:47:30.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:30.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.814803+0000) 2019-10-13T12:47:30.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:30.812+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.814244+0000) 2019-10-13T12:47:30.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:30.812+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.814823+0000) 2019-10-13T12:47:30.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:30.812+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.815452+0000) 2019-10-13T12:47:30.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:30.812+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.815723+0000) 2019-10-13T12:47:30.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:30.812+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.816321+0000) 2019-10-13T12:47:30.819 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:30.812+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:30.817666+0000) 2019-10-13T12:47:30.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:30.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:31.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:31.912+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:31.912692+0000) 2019-10-13T12:47:34.239 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:34.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.237949+0000) 2019-10-13T12:47:34.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:34.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:34.522707+0000) 2019-10-13T12:47:35.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:35.819+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.823682+0000) 2019-10-13T12:47:35.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:35.819+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.824181+0000) 2019-10-13T12:47:35.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:35.819+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.824526+0000) 2019-10-13T12:47:35.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:35.820+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.827690+0000) 2019-10-13T12:47:35.829 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:35.823+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.827767+0000) 2019-10-13T12:47:35.830 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:35.823+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.829020+0000) 2019-10-13T12:47:35.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:35.823+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.829587+0000) 2019-10-13T12:47:35.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:35.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:35.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:35.828+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.835161+0000) 2019-10-13T12:47:35.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:35.864+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:35.867283+0000) 2019-10-13T12:47:37.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:37.891+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:37.894949+0000) 2019-10-13T12:47:38.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:38.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:38.120089+0000) 2019-10-13T12:47:38.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:38.139+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:38.141302+0000) 2019-10-13T12:47:38.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:38.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:38.661242+0000) 2019-10-13T12:47:40.838 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:40.831+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.836281+0000) 2019-10-13T12:47:40.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:40.831+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.836745+0000) 2019-10-13T12:47:40.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:40.835+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.838999+0000) 2019-10-13T12:47:40.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:40.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.840513+0000) 2019-10-13T12:47:40.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:40.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.840828+0000) 2019-10-13T12:47:40.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:40.835+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.841417+0000) 2019-10-13T12:47:40.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:40.836+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.842143+0000) 2019-10-13T12:47:40.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:40.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:40.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:40.844+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:40.850656+0000) 2019-10-13T12:47:41.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:41.912+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:41.912884+0000) 2019-10-13T12:47:42.873 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:47:42.873 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:42.877 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:47:42.877 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:47:44.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:44.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.238214+0000) 2019-10-13T12:47:44.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:44.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:44.522913+0000) 2019-10-13T12:47:45.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:45.843+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.847149+0000) 2019-10-13T12:47:45.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:45.843+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.847719+0000) 2019-10-13T12:47:45.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:45.843+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.848995+0000) 2019-10-13T12:47:45.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:45.843+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.849299+0000) 2019-10-13T12:47:45.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:45.843+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.849586+0000) 2019-10-13T12:47:45.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:45.844+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.850793+0000) 2019-10-13T12:47:45.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:45.847+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.850515+0000) 2019-10-13T12:47:45.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:45.851+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:45.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:45.852+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.856479+0000) 2019-10-13T12:47:45.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:45.868+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:45.875628+0000) 2019-10-13T12:47:47.896 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:47.891+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:47.895152+0000) 2019-10-13T12:47:48.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:48.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:48.120309+0000) 2019-10-13T12:47:48.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:48.135+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:48.141545+0000) 2019-10-13T12:47:48.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:48.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:48.661434+0000) 2019-10-13T12:47:50.861 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:50.855+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.859389+0000) 2019-10-13T12:47:50.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:50.855+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.859808+0000) 2019-10-13T12:47:50.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:50.855+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.860297+0000) 2019-10-13T12:47:50.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:50.859+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.862236+0000) 2019-10-13T12:47:50.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:50.859+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.862638+0000) 2019-10-13T12:47:50.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:50.856+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.863359+0000) 2019-10-13T12:47:50.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:50.859+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.862794+0000) 2019-10-13T12:47:50.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:50.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:50.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:50.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:50.868545+0000) 2019-10-13T12:47:51.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:51.912+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:51.913078+0000) 2019-10-13T12:47:54.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:54.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.238420+0000) 2019-10-13T12:47:54.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:54.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:54.523145+0000) 2019-10-13T12:47:55.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:55.867+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.872693+0000) 2019-10-13T12:47:55.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:55.867+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.873235+0000) 2019-10-13T12:47:55.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:47:55.871+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.874172+0000) 2019-10-13T12:47:55.876 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:47:55.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.875093+0000) 2019-10-13T12:47:55.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:47:55.871+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.876817+0000) 2019-10-13T12:47:55.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:55.871+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.876542+0000) 2019-10-13T12:47:55.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:55.871+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.877675+0000) 2019-10-13T12:47:55.881 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:47:55.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:47:55.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:55.880+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.885247+0000) 2019-10-13T12:47:55.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:47:55.880+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:55.885358+0000) 2019-10-13T12:47:57.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:47:57.891+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:57.895362+0000) 2019-10-13T12:47:58.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:47:58.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:58.120511+0000) 2019-10-13T12:47:58.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:47:58.139+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:58.141811+0000) 2019-10-13T12:47:58.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:47:58.655+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:47:58.661669+0000) 2019-10-13T12:48:00.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:00.879+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.883277+0000) 2019-10-13T12:48:00.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:00.879+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.884010+0000) 2019-10-13T12:48:00.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:00.879+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.884359+0000) 2019-10-13T12:48:00.886 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:00.879+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.884965+0000) 2019-10-13T12:48:00.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:00.879+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.885371+0000) 2019-10-13T12:48:00.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:00.880+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.886177+0000) 2019-10-13T12:48:00.888 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:00.883+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.886774+0000) 2019-10-13T12:48:00.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:00.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:00.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:00.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:00.892602+0000) 2019-10-13T12:48:01.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:01.912+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:01.913289+0000) 2019-10-13T12:48:04.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:04.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.238635+0000) 2019-10-13T12:48:04.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:04.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:04.523376+0000) 2019-10-13T12:48:05.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:05.884+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.885578+0000) 2019-10-13T12:48:05.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:05.891+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.894081+0000) 2019-10-13T12:48:05.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:05.891+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.895363+0000) 2019-10-13T12:48:05.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:05.891+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.895763+0000) 2019-10-13T12:48:05.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:05.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.896273+0000) 2019-10-13T12:48:05.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:05.891+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.896483+0000) 2019-10-13T12:48:05.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:05.891+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.897631+0000) 2019-10-13T12:48:05.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:05.895+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.898435+0000) 2019-10-13T12:48:05.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:05.900+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:05.900702+0000) 2019-10-13T12:48:05.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:05.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:07.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:07.895+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:07.895610+0000) 2019-10-13T12:48:08.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:08.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:08.120744+0000) 2019-10-13T12:48:08.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:08.139+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:08.142009+0000) 2019-10-13T12:48:08.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:08.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:08.661863+0000) 2019-10-13T12:48:10.908 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:10.903+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.906865+0000) 2019-10-13T12:48:10.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:10.903+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.907436+0000) 2019-10-13T12:48:10.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:10.907+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.908289+0000) 2019-10-13T12:48:10.910 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:10.903+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.907555+0000) 2019-10-13T12:48:10.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:10.907+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.910246+0000) 2019-10-13T12:48:10.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:10.907+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.912853+0000) 2019-10-13T12:48:10.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:10.907+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.913055+0000) 2019-10-13T12:48:10.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:10.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168623 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:10.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:10.915+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:10.917166+0000) 2019-10-13T12:48:11.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:11.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:11.913474+0000) 2019-10-13T12:48:12.992 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:48:12.992 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:12.995 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:48:12.996 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:14.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:14.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.238841+0000) 2019-10-13T12:48:14.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:14.519+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:14.523528+0000) 2019-10-13T12:48:15.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:15.891+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.892134+0000) 2019-10-13T12:48:15.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:15.915+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.918790+0000) 2019-10-13T12:48:15.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:15.915+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.919010+0000) 2019-10-13T12:48:15.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:15.915+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.919877+0000) 2019-10-13T12:48:15.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:15.915+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.920010+0000) 2019-10-13T12:48:15.922 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:15.915+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.920599+0000) 2019-10-13T12:48:15.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:15.915+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.920857+0000) 2019-10-13T12:48:15.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:15.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.921835+0000) 2019-10-13T12:48:15.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:15.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:15.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:15.927+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:15.928533+0000) 2019-10-13T12:48:17.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:17.895+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:17.895852+0000) 2019-10-13T12:48:18.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:18.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:18.120971+0000) 2019-10-13T12:48:18.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:18.139+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:18.142234+0000) 2019-10-13T12:48:18.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:18.659+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:18.662055+0000) 2019-10-13T12:48:20.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:20.927+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.930824+0000) 2019-10-13T12:48:20.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:20.927+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.931237+0000) 2019-10-13T12:48:20.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:20.927+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.931819+0000) 2019-10-13T12:48:20.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:20.927+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.932168+0000) 2019-10-13T12:48:20.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:20.927+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.932825+0000) 2019-10-13T12:48:20.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:20.927+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.933389+0000) 2019-10-13T12:48:20.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:20.931+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.934001+0000) 2019-10-13T12:48:20.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:20.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168686 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:20.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:20.939+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:20.940318+0000) 2019-10-13T12:48:21.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:21.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:21.913663+0000) 2019-10-13T12:48:24.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:24.235+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.239044+0000) 2019-10-13T12:48:24.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:24.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:24.523752+0000) 2019-10-13T12:48:25.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:25.895+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.898693+0000) 2019-10-13T12:48:25.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:25.935+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.941136+0000) 2019-10-13T12:48:25.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:25.939+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.943612+0000) 2019-10-13T12:48:25.945 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:25.939+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.943901+0000) 2019-10-13T12:48:25.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:25.943+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.944631+0000) 2019-10-13T12:48:25.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:25.939+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.945633+0000) 2019-10-13T12:48:25.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:25.943+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.945800+0000) 2019-10-13T12:48:25.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:25.943+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.946293+0000) 2019-10-13T12:48:25.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:25.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:25.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:25.947+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:25.951329+0000) 2019-10-13T12:48:27.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:27.894+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:27.896058+0000) 2019-10-13T12:48:28.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:28.118+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:28.121169+0000) 2019-10-13T12:48:28.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:28.138+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:28.142465+0000) 2019-10-13T12:48:28.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:28.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:28.662280+0000) 2019-10-13T12:48:30.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:30.951+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.953102+0000) 2019-10-13T12:48:30.955 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:30.950+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.953942+0000) 2019-10-13T12:48:30.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:30.950+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.954411+0000) 2019-10-13T12:48:30.958 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:30.950+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.956247+0000) 2019-10-13T12:48:30.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:30.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.957143+0000) 2019-10-13T12:48:30.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:30.954+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.958922+0000) 2019-10-13T12:48:30.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:30.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.959433+0000) 2019-10-13T12:48:30.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:30.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:30.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:30.959+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:30.963050+0000) 2019-10-13T12:48:31.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:31.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:31.913836+0000) 2019-10-13T12:48:34.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:34.234+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.239254+0000) 2019-10-13T12:48:34.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:34.518+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:34.523953+0000) 2019-10-13T12:48:35.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:35.903+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.905310+0000) 2019-10-13T12:48:35.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:35.958+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.965595+0000) 2019-10-13T12:48:35.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:35.962+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.965989+0000) 2019-10-13T12:48:35.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:35.962+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.966527+0000) 2019-10-13T12:48:35.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:35.962+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.966753+0000) 2019-10-13T12:48:35.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:35.962+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.967216+0000) 2019-10-13T12:48:35.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:35.967+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.968119+0000) 2019-10-13T12:48:35.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:35.962+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.969047+0000) 2019-10-13T12:48:35.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:35.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:35.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:35.971+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:35.974285+0000) 2019-10-13T12:48:37.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:37.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:37.896277+0000) 2019-10-13T12:48:38.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:38.118+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:38.121369+0000) 2019-10-13T12:48:38.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:38.138+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:38.142664+0000) 2019-10-13T12:48:38.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:38.658+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:38.662480+0000) 2019-10-13T12:48:40.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:40.970+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.976627+0000) 2019-10-13T12:48:40.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:40.970+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.977323+0000) 2019-10-13T12:48:40.979 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:40.974+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.977783+0000) 2019-10-13T12:48:40.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:40.974+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.978233+0000) 2019-10-13T12:48:40.981 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:40.974+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.978865+0000) 2019-10-13T12:48:40.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:40.979+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.980439+0000) 2019-10-13T12:48:40.982 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:40.974+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.979991+0000) 2019-10-13T12:48:40.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:40.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:40.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:40.983+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:40.987390+0000) 2019-10-13T12:48:41.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:41.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:41.913978+0000) 2019-10-13T12:48:43.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:48:43.105 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:43.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:48:43.109 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:48:44.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:44.238+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.239458+0000) 2019-10-13T12:48:44.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:44.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:44.524155+0000) 2019-10-13T12:48:45.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:45.911+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.911781+0000) 2019-10-13T12:48:45.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:45.982+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.989024+0000) 2019-10-13T12:48:45.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:45.982+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.989271+0000) 2019-10-13T12:48:45.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:45.986+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.990821+0000) 2019-10-13T12:48:45.992 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:45.986+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.991246+0000) 2019-10-13T12:48:45.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:45.986+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.992984+0000) 2019-10-13T12:48:45.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:45.991+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.994025+0000) 2019-10-13T12:48:45.997 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:45.990+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.996242+0000) 2019-10-13T12:48:45.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:45.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:46.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:45.995+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:45.999601+0000) 2019-10-13T12:48:47.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:47.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:47.896480+0000) 2019-10-13T12:48:48.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:48.118+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:48.121578+0000) 2019-10-13T12:48:48.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:48.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:48.142883+0000) 2019-10-13T12:48:48.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:48.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:48.662673+0000) 2019-10-13T12:48:51.003 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:50.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.001313+0000) 2019-10-13T12:48:51.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:50.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.003539+0000) 2019-10-13T12:48:51.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:50.998+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.003870+0000) 2019-10-13T12:48:51.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:50.998+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.004129+0000) 2019-10-13T12:48:51.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:51.003+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.004881+0000) 2019-10-13T12:48:51.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:50.998+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.005563+0000) 2019-10-13T12:48:51.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:50.998+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.005326+0000) 2019-10-13T12:48:51.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:51.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:51.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:51.011+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.012632+0000) 2019-10-13T12:48:51.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:51.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:51.914171+0000) 2019-10-13T12:48:54.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:54.238+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.239684+0000) 2019-10-13T12:48:54.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:54.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:54.524354+0000) 2019-10-13T12:48:55.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:55.915+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:55.918346+0000) 2019-10-13T12:48:56.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:48:56.006+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.012834+0000) 2019-10-13T12:48:56.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:48:56.011+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.013270+0000) 2019-10-13T12:48:56.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:56.010+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.014367+0000) 2019-10-13T12:48:56.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:56.010+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.014580+0000) 2019-10-13T12:48:56.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:56.010+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.017091+0000) 2019-10-13T12:48:56.019 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:48:56.010+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.017604+0000) 2019-10-13T12:48:56.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:56.014+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.018158+0000) 2019-10-13T12:48:56.023 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:48:56.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:48:56.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:48:56.019+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:56.023085+0000) 2019-10-13T12:48:57.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:48:57.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:57.896695+0000) 2019-10-13T12:48:58.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:48:58.118+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:58.121817+0000) 2019-10-13T12:48:58.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:48:58.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:58.143101+0000) 2019-10-13T12:48:58.664 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:48:58.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:48:58.662889+0000) 2019-10-13T12:49:01.028 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:01.022+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.025852+0000) 2019-10-13T12:49:01.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:01.022+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.026219+0000) 2019-10-13T12:49:01.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:01.023+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.027242+0000) 2019-10-13T12:49:01.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:01.022+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.026745+0000) 2019-10-13T12:49:01.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:01.022+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.027216+0000) 2019-10-13T12:49:01.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:01.022+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.027913+0000) 2019-10-13T12:49:01.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:01.022+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.028676+0000) 2019-10-13T12:49:01.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:01.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:01.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:01.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.036927+0000) 2019-10-13T12:49:01.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:01.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:01.914344+0000) 2019-10-13T12:49:04.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:04.238+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.239875+0000) 2019-10-13T12:49:04.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:04.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:04.524557+0000) 2019-10-13T12:49:05.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:05.927+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:05.928658+0000) 2019-10-13T12:49:06.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:06.030+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.037005+0000) 2019-10-13T12:49:06.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:06.030+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.037582+0000) 2019-10-13T12:49:06.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:06.035+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.038970+0000) 2019-10-13T12:49:06.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:06.034+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.039270+0000) 2019-10-13T12:49:06.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:06.034+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.039585+0000) 2019-10-13T12:49:06.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:06.034+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.040247+0000) 2019-10-13T12:49:06.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:06.034+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.040647+0000) 2019-10-13T12:49:06.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:06.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:06.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:06.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:06.047581+0000) 2019-10-13T12:49:07.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:07.894+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:07.896895+0000) 2019-10-13T12:49:08.123 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:08.118+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:08.122023+0000) 2019-10-13T12:49:08.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:08.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:08.143292+0000) 2019-10-13T12:49:08.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:08.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:08.663108+0000) 2019-10-13T12:49:11.052 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:11.046+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.050537+0000) 2019-10-13T12:49:11.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:11.046+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.051542+0000) 2019-10-13T12:49:11.053 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:11.046+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.052007+0000) 2019-10-13T12:49:11.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:11.051+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.052592+0000) 2019-10-13T12:49:11.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:11.046+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.053094+0000) 2019-10-13T12:49:11.055 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:11.046+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.052634+0000) 2019-10-13T12:49:11.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:11.050+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.054153+0000) 2019-10-13T12:49:11.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:11.059+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.062061+0000) 2019-10-13T12:49:11.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:11.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:11.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:11.911+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:11.914518+0000) 2019-10-13T12:49:13.244 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:49:13.244 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:13.247 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:49:13.248 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:14.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:14.238+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.240030+0000) 2019-10-13T12:49:14.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:14.522+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:14.524753+0000) 2019-10-13T12:49:15.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:15.930+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:15.935171+0000) 2019-10-13T12:49:16.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:16.058+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.061956+0000) 2019-10-13T12:49:16.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:16.058+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.062375+0000) 2019-10-13T12:49:16.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:16.058+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.063301+0000) 2019-10-13T12:49:16.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:16.058+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.063944+0000) 2019-10-13T12:49:16.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:16.058+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.064007+0000) 2019-10-13T12:49:16.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:16.058+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.064284+0000) 2019-10-13T12:49:16.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:16.058+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.065102+0000) 2019-10-13T12:49:16.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:16.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:16.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:16.070+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:16.074594+0000) 2019-10-13T12:49:17.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:17.890+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:17.897110+0000) 2019-10-13T12:49:18.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:18.122+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:18.122220+0000) 2019-10-13T12:49:18.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:18.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:18.143499+0000) 2019-10-13T12:49:18.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:18.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:18.663260+0000) 2019-10-13T12:49:21.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:21.070+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.073882+0000) 2019-10-13T12:49:21.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:21.070+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.076245+0000) 2019-10-13T12:49:21.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:21.070+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.076601+0000) 2019-10-13T12:49:21.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:21.070+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.077198+0000) 2019-10-13T12:49:21.080 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:21.074+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.079280+0000) 2019-10-13T12:49:21.081 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:21.078+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.080083+0000) 2019-10-13T12:49:21.082 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:21.074+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.080440+0000) 2019-10-13T12:49:21.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:21.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:21.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:21.090+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.092530+0000) 2019-10-13T12:49:21.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:21.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:21.914762+0000) 2019-10-13T12:49:24.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:24.237+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.240246+0000) 2019-10-13T12:49:24.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:24.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:24.524988+0000) 2019-10-13T12:49:25.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:25.942+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:25.946648+0000) 2019-10-13T12:49:26.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:26.081+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.085598+0000) 2019-10-13T12:49:26.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:26.082+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.086507+0000) 2019-10-13T12:49:26.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:26.081+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.086996+0000) 2019-10-13T12:49:26.089 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:26.081+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.087860+0000) 2019-10-13T12:49:26.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:26.081+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.088802+0000) 2019-10-13T12:49:26.090 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:26.081+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.089171+0000) 2019-10-13T12:49:26.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:26.085+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.090364+0000) 2019-10-13T12:49:26.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:26.090+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:26.093099+0000) 2019-10-13T12:49:26.095 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:26.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:27.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:27.893+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:27.897271+0000) 2019-10-13T12:49:28.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:28.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:28.122453+0000) 2019-10-13T12:49:28.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:28.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:28.143692+0000) 2019-10-13T12:49:28.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:28.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:28.663480+0000) 2019-10-13T12:49:31.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:31.093+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.097346+0000) 2019-10-13T12:49:31.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:31.093+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.097865+0000) 2019-10-13T12:49:31.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:31.093+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.098500+0000) 2019-10-13T12:49:31.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:31.093+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.099037+0000) 2019-10-13T12:49:31.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:31.093+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.099365+0000) 2019-10-13T12:49:31.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:31.093+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.100044+0000) 2019-10-13T12:49:31.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:31.098+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.100599+0000) 2019-10-13T12:49:31.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:31.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:31.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:31.102+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.107958+0000) 2019-10-13T12:49:31.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:31.914+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:31.914971+0000) 2019-10-13T12:49:34.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:34.233+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.240442+0000) 2019-10-13T12:49:34.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:34.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:34.525189+0000) 2019-10-13T12:49:35.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:35.950+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:35.952996+0000) 2019-10-13T12:49:36.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:36.105+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.109191+0000) 2019-10-13T12:49:36.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:36.105+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.110129+0000) 2019-10-13T12:49:36.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:36.105+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.110485+0000) 2019-10-13T12:49:36.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:36.105+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.111724+0000) 2019-10-13T12:49:36.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:36.110+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.112449+0000) 2019-10-13T12:49:36.114 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:36.105+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.112585+0000) 2019-10-13T12:49:36.114 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:36.105+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.112981+0000) 2019-10-13T12:49:36.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:36.114+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:36.116513+0000) 2019-10-13T12:49:36.120 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:36.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:37.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:37.893+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:37.897471+0000) 2019-10-13T12:49:38.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:38.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:38.122652+0000) 2019-10-13T12:49:38.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:38.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:38.143886+0000) 2019-10-13T12:49:38.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:38.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:38.663673+0000) 2019-10-13T12:49:41.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:41.121+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.121848+0000) 2019-10-13T12:49:41.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:41.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.122101+0000) 2019-10-13T12:49:41.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:41.122+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.124874+0000) 2019-10-13T12:49:41.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:41.121+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.124989+0000) 2019-10-13T12:49:41.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:41.121+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.125649+0000) 2019-10-13T12:49:41.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:41.125+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.127721+0000) 2019-10-13T12:49:41.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:41.129+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.130340+0000) 2019-10-13T12:49:41.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:41.126+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.131494+0000) 2019-10-13T12:49:41.133 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:41.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:41.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:41.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:41.915145+0000) 2019-10-13T12:49:43.291 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:49:43.291 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:43.295 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:49:43.295 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:49:44.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:44.237+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.240599+0000) 2019-10-13T12:49:44.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:44.521+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:44.525379+0000) 2019-10-13T12:49:45.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:45.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:45.959542+0000) 2019-10-13T12:49:46.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:46.133+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.134685+0000) 2019-10-13T12:49:46.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:46.133+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.134989+0000) 2019-10-13T12:49:46.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:46.133+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.135353+0000) 2019-10-13T12:49:46.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:46.133+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.136616+0000) 2019-10-13T12:49:46.138 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:46.133+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.137273+0000) 2019-10-13T12:49:46.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:46.133+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.137602+0000) 2019-10-13T12:49:46.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:46.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.138099+0000) 2019-10-13T12:49:46.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:46.138+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:46.142800+0000) 2019-10-13T12:49:46.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:46.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:47.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:47.897+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:47.897726+0000) 2019-10-13T12:49:48.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:48.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:48.122848+0000) 2019-10-13T12:49:48.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:48.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:48.144118+0000) 2019-10-13T12:49:48.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:48.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:48.663863+0000) 2019-10-13T12:49:51.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:51.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.146362+0000) 2019-10-13T12:49:51.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:51.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.146503+0000) 2019-10-13T12:49:51.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:51.145+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.149394+0000) 2019-10-13T12:49:51.152 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:51.145+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.149570+0000) 2019-10-13T12:49:51.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:51.149+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.149905+0000) 2019-10-13T12:49:51.153 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:51.146+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.151905+0000) 2019-10-13T12:49:51.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:51.149+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.152185+0000) 2019-10-13T12:49:51.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:51.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:51.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:51.154+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.158456+0000) 2019-10-13T12:49:51.916 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:51.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:51.915350+0000) 2019-10-13T12:49:54.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:54.237+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.240805+0000) 2019-10-13T12:49:54.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:54.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:54.525570+0000) 2019-10-13T12:49:55.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:55.962+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:55.966215+0000) 2019-10-13T12:49:56.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:49:56.157+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.160071+0000) 2019-10-13T12:49:56.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:56.157+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.160799+0000) 2019-10-13T12:49:56.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:49:56.157+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.161186+0000) 2019-10-13T12:49:56.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:56.157+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.161483+0000) 2019-10-13T12:49:56.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:49:56.158+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.162667+0000) 2019-10-13T12:49:56.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:56.161+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.161940+0000) 2019-10-13T12:49:56.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:56.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.162866+0000) 2019-10-13T12:49:56.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:49:56.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:49:56.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:49:56.166+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:56.169075+0000) 2019-10-13T12:49:57.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:49:57.897+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:57.897978+0000) 2019-10-13T12:49:58.124 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:49:58.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:58.123059+0000) 2019-10-13T12:49:58.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:49:58.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:58.144351+0000) 2019-10-13T12:49:58.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:49:58.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:49:58.664060+0000) 2019-10-13T12:50:01.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:01.169+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.170816+0000) 2019-10-13T12:50:01.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:01.169+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.171642+0000) 2019-10-13T12:50:01.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:01.169+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.173206+0000) 2019-10-13T12:50:01.175 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:01.169+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.173613+0000) 2019-10-13T12:50:01.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:01.173+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.174319+0000) 2019-10-13T12:50:01.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:01.173+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.174741+0000) 2019-10-13T12:50:01.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:01.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.176037+0000) 2019-10-13T12:50:01.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:01.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:01.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:01.182+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.184604+0000) 2019-10-13T12:50:01.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:01.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:01.915558+0000) 2019-10-13T12:50:04.242 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:04.237+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.241006+0000) 2019-10-13T12:50:04.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:04.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:04.525811+0000) 2019-10-13T12:50:05.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:05.970+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:05.972737+0000) 2019-10-13T12:50:06.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:06.177+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.181082+0000) 2019-10-13T12:50:06.183 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:06.178+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.182017+0000) 2019-10-13T12:50:06.184 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:06.181+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.182136+0000) 2019-10-13T12:50:06.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:06.181+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.184104+0000) 2019-10-13T12:50:06.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:06.185+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.187054+0000) 2019-10-13T12:50:06.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:06.185+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.188789+0000) 2019-10-13T12:50:06.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:06.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:06.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:06.189+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.190597+0000) 2019-10-13T12:50:06.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:06.194+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:06.197696+0000) 2019-10-13T12:50:07.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:07.897+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:07.898242+0000) 2019-10-13T12:50:08.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:08.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:08.123285+0000) 2019-10-13T12:50:08.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:08.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:08.144553+0000) 2019-10-13T12:50:08.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:08.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:08.664252+0000) 2019-10-13T12:50:11.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:11.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.194304+0000) 2019-10-13T12:50:11.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:11.193+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.194761+0000) 2019-10-13T12:50:11.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:11.193+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.195257+0000) 2019-10-13T12:50:11.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:11.193+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.196160+0000) 2019-10-13T12:50:11.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:11.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.196958+0000) 2019-10-13T12:50:11.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:11.193+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.197206+0000) 2019-10-13T12:50:11.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:11.193+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.197685+0000) 2019-10-13T12:50:11.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:11.198+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.203044+0000) 2019-10-13T12:50:11.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:11.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169364 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:11.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:11.910+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:11.915738+0000) 2019-10-13T12:50:13.422 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:50:13.422 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:13.425 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:50:13.426 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:14.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:14.237+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.241228+0000) 2019-10-13T12:50:14.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:14.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:14.525972+0000) 2019-10-13T12:50:15.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:15.974+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:15.979295+0000) 2019-10-13T12:50:16.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:16.205+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.206103+0000) 2019-10-13T12:50:16.208 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:16.205+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.206295+0000) 2019-10-13T12:50:16.208 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:16.205+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.206906+0000) 2019-10-13T12:50:16.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:16.205+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.207048+0000) 2019-10-13T12:50:16.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:16.205+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.207731+0000) 2019-10-13T12:50:16.211 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:16.205+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.209406+0000) 2019-10-13T12:50:16.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:16.210+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.212797+0000) 2019-10-13T12:50:16.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:16.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:16.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:16.218+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:16.220673+0000) 2019-10-13T12:50:17.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:17.897+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:17.898480+0000) 2019-10-13T12:50:18.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:18.121+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:18.123513+0000) 2019-10-13T12:50:18.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:18.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:18.144754+0000) 2019-10-13T12:50:18.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:18.661+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:18.664476+0000) 2019-10-13T12:50:21.220 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:21.216+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.218071+0000) 2019-10-13T12:50:21.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:21.213+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.219712+0000) 2019-10-13T12:50:21.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:21.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.219930+0000) 2019-10-13T12:50:21.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:21.216+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.220061+0000) 2019-10-13T12:50:21.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:21.216+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.220088+0000) 2019-10-13T12:50:21.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:21.216+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.220402+0000) 2019-10-13T12:50:21.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:21.216+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.221420+0000) 2019-10-13T12:50:21.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:21.224+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169428 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:21.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:21.221+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.227754+0000) 2019-10-13T12:50:21.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:21.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:21.915909+0000) 2019-10-13T12:50:24.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:24.236+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.241436+0000) 2019-10-13T12:50:24.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:24.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:24.526154+0000) 2019-10-13T12:50:25.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:25.985+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:25.986069+0000) 2019-10-13T12:50:26.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:26.228+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.230541+0000) 2019-10-13T12:50:26.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:26.228+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.230938+0000) 2019-10-13T12:50:26.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:26.228+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.231537+0000) 2019-10-13T12:50:26.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:26.228+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.232386+0000) 2019-10-13T12:50:26.234 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:26.228+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.232787+0000) 2019-10-13T12:50:26.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:26.229+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.233214+0000) 2019-10-13T12:50:26.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:26.228+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.233468+0000) 2019-10-13T12:50:26.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:26.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:26.238282+0000) 2019-10-13T12:50:26.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:26.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:27.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:27.896+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:27.898685+0000) 2019-10-13T12:50:28.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:28.120+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:28.123738+0000) 2019-10-13T12:50:28.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:28.140+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:28.144966+0000) 2019-10-13T12:50:28.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:28.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:28.664687+0000) 2019-10-13T12:50:31.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:31.237+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.243067+0000) 2019-10-13T12:50:31.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:31.240+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.244267+0000) 2019-10-13T12:50:31.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:31.240+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.244438+0000) 2019-10-13T12:50:31.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:31.240+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.245039+0000) 2019-10-13T12:50:31.247 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:31.240+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.245304+0000) 2019-10-13T12:50:31.248 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:31.244+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.246430+0000) 2019-10-13T12:50:31.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:31.244+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.247897+0000) 2019-10-13T12:50:31.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:31.245+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.248980+0000) 2019-10-13T12:50:31.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:31.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169489 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:31.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:31.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:31.916124+0000) 2019-10-13T12:50:34.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:34.236+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.241669+0000) 2019-10-13T12:50:34.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:34.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:34.526302+0000) 2019-10-13T12:50:35.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:35.989+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:35.992501+0000) 2019-10-13T12:50:36.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:36.249+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.253346+0000) 2019-10-13T12:50:36.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:36.252+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.253775+0000) 2019-10-13T12:50:36.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:36.252+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.254708+0000) 2019-10-13T12:50:36.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:36.252+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.256505+0000) 2019-10-13T12:50:36.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:36.252+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.256662+0000) 2019-10-13T12:50:36.259 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:36.252+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.257594+0000) 2019-10-13T12:50:36.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:36.256+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.258841+0000) 2019-10-13T12:50:36.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:36.257+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:36.260456+0000) 2019-10-13T12:50:36.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:36.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:37.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:37.896+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:37.898883+0000) 2019-10-13T12:50:38.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:38.120+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:38.123936+0000) 2019-10-13T12:50:38.146 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:38.144+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:38.145197+0000) 2019-10-13T12:50:38.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:38.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:38.664860+0000) 2019-10-13T12:50:41.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:41.264+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.266644+0000) 2019-10-13T12:50:41.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:41.264+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.266937+0000) 2019-10-13T12:50:41.269 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:41.264+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.267685+0000) 2019-10-13T12:50:41.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:41.265+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.268516+0000) 2019-10-13T12:50:41.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:41.264+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.268217+0000) 2019-10-13T12:50:41.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:41.264+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.269061+0000) 2019-10-13T12:50:41.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:41.264+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.269447+0000) 2019-10-13T12:50:41.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:41.269+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.273004+0000) 2019-10-13T12:50:41.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:41.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:41.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:41.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:41.916289+0000) 2019-10-13T12:50:43.544 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:50:43.544 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:43.547 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:50:43.547 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:50:44.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:44.240+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.241874+0000) 2019-10-13T12:50:44.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:44.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:44.526530+0000) 2019-10-13T12:50:46.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:45.997+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:45.998909+0000) 2019-10-13T12:50:46.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:46.272+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.276565+0000) 2019-10-13T12:50:46.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:46.272+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.276936+0000) 2019-10-13T12:50:46.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:46.276+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.278335+0000) 2019-10-13T12:50:46.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:46.276+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.279002+0000) 2019-10-13T12:50:46.282 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:46.276+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.280958+0000) 2019-10-13T12:50:46.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:46.280+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.282745+0000) 2019-10-13T12:50:46.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:46.281+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.284783+0000) 2019-10-13T12:50:46.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:46.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169583 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:46.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:46.285+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:46.289866+0000) 2019-10-13T12:50:47.900 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:47.896+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:47.899083+0000) 2019-10-13T12:50:48.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:48.120+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:48.124129+0000) 2019-10-13T12:50:48.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:48.140+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:48.145395+0000) 2019-10-13T12:50:48.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:48.660+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:48.665071+0000) 2019-10-13T12:50:51.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:51.284+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.288034+0000) 2019-10-13T12:50:51.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:51.288+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.290059+0000) 2019-10-13T12:50:51.292 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:51.288+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.290501+0000) 2019-10-13T12:50:51.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:51.288+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.291294+0000) 2019-10-13T12:50:51.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:51.288+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.291653+0000) 2019-10-13T12:50:51.294 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:51.288+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.292546+0000) 2019-10-13T12:50:51.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:51.289+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.293023+0000) 2019-10-13T12:50:51.297 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:51.293+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.296445+0000) 2019-10-13T12:50:51.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:51.292+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:51.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:51.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:51.916461+0000) 2019-10-13T12:50:54.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:54.240+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.242083+0000) 2019-10-13T12:50:54.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:54.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:54.526736+0000) 2019-10-13T12:50:56.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:56.001+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.005443+0000) 2019-10-13T12:50:56.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:56.296+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.300671+0000) 2019-10-13T12:50:56.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:56.296+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.300936+0000) 2019-10-13T12:50:56.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:50:56.296+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.301635+0000) 2019-10-13T12:50:56.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:50:56.300+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.301850+0000) 2019-10-13T12:50:56.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:50:56.297+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.303314+0000) 2019-10-13T12:50:56.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:56.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.303472+0000) 2019-10-13T12:50:56.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:56.300+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.303824+0000) 2019-10-13T12:50:56.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:50:56.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:50:56.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:50:56.305+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:56.308831+0000) 2019-10-13T12:50:57.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:50:57.896+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:57.899287+0000) 2019-10-13T12:50:58.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:50:58.120+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:58.124323+0000) 2019-10-13T12:50:58.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:50:58.144+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:58.145588+0000) 2019-10-13T12:50:58.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:50:58.664+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:50:58.665292+0000) 2019-10-13T12:51:01.313 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:01.308+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.311254+0000) 2019-10-13T12:51:01.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:01.305+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.311905+0000) 2019-10-13T12:51:01.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:01.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.312399+0000) 2019-10-13T12:51:01.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:01.308+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.312929+0000) 2019-10-13T12:51:01.315 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:01.308+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.313559+0000) 2019-10-13T12:51:01.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:01.312+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.314746+0000) 2019-10-13T12:51:01.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:01.312+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.315611+0000) 2019-10-13T12:51:01.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:01.313+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.319290+0000) 2019-10-13T12:51:01.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:01.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169676 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:01.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:01.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:01.916630+0000) 2019-10-13T12:51:04.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:04.240+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.242317+0000) 2019-10-13T12:51:04.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:04.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:04.526933+0000) 2019-10-13T12:51:06.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:06.009+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.012155+0000) 2019-10-13T12:51:06.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:06.320+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.323199+0000) 2019-10-13T12:51:06.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:06.320+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.323315+0000) 2019-10-13T12:51:06.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:06.317+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.323830+0000) 2019-10-13T12:51:06.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:06.320+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.324055+0000) 2019-10-13T12:51:06.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:06.320+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.324795+0000) 2019-10-13T12:51:06.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:06.320+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.325306+0000) 2019-10-13T12:51:06.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:06.324+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.325995+0000) 2019-10-13T12:51:06.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:06.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:06.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:06.325+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:06.331230+0000) 2019-10-13T12:51:07.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:07.896+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:07.899487+0000) 2019-10-13T12:51:08.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:08.120+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:08.124546+0000) 2019-10-13T12:51:08.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:08.144+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:08.145790+0000) 2019-10-13T12:51:08.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:08.664+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:08.665483+0000) 2019-10-13T12:51:11.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:11.328+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.333691+0000) 2019-10-13T12:51:11.336 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:11.332+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.334380+0000) 2019-10-13T12:51:11.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:11.329+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.335288+0000) 2019-10-13T12:51:11.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:11.332+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.335740+0000) 2019-10-13T12:51:11.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:11.332+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.335905+0000) 2019-10-13T12:51:11.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:11.332+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.336571+0000) 2019-10-13T12:51:11.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:11.336+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.337869+0000) 2019-10-13T12:51:11.343 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:11.340+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:11.345 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:11.337+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.343886+0000) 2019-10-13T12:51:11.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:11.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:11.916801+0000) 2019-10-13T12:51:13.598 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:51:13.598 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:13.601 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:51:13.602 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:14.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:14.240+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.242522+0000) 2019-10-13T12:51:14.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:14.524+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:14.527171+0000) 2019-10-13T12:51:16.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:16.017+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.018710+0000) 2019-10-13T12:51:16.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:16.339+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.345630+0000) 2019-10-13T12:51:16.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:16.343+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.346273+0000) 2019-10-13T12:51:16.350 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:16.343+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.348400+0000) 2019-10-13T12:51:16.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:16.345+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.349342+0000) 2019-10-13T12:51:16.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:16.343+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.349479+0000) 2019-10-13T12:51:16.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:16.347+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.350168+0000) 2019-10-13T12:51:16.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:16.347+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.351248+0000) 2019-10-13T12:51:16.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:16.349+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:16.354313+0000) 2019-10-13T12:51:16.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:16.351+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:17.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:17.895+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:17.899722+0000) 2019-10-13T12:51:18.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:18.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:18.124761+0000) 2019-10-13T12:51:18.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:18.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:18.145999+0000) 2019-10-13T12:51:18.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:18.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:18.665676+0000) 2019-10-13T12:51:21.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:21.351+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.357409+0000) 2019-10-13T12:51:21.361 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:21.355+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.359555+0000) 2019-10-13T12:51:21.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:21.357+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.360281+0000) 2019-10-13T12:51:21.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:21.355+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.360526+0000) 2019-10-13T12:51:21.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:21.355+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.360874+0000) 2019-10-13T12:51:21.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:21.355+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.361189+0000) 2019-10-13T12:51:21.363 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:21.359+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.362152+0000) 2019-10-13T12:51:21.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:21.363+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:21.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:21.365+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.369363+0000) 2019-10-13T12:51:21.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:21.913+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:21.916990+0000) 2019-10-13T12:51:24.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:24.239+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.242728+0000) 2019-10-13T12:51:24.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:24.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:24.527373+0000) 2019-10-13T12:51:26.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:26.024+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.025411+0000) 2019-10-13T12:51:26.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:26.367+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.370023+0000) 2019-10-13T12:51:26.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:26.364+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.370960+0000) 2019-10-13T12:51:26.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:26.367+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.371237+0000) 2019-10-13T12:51:26.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:26.367+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.371400+0000) 2019-10-13T12:51:26.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:26.367+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.372008+0000) 2019-10-13T12:51:26.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:26.367+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.372195+0000) 2019-10-13T12:51:26.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:26.367+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.373520+0000) 2019-10-13T12:51:26.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:26.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:26.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:26.372+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:26.379665+0000) 2019-10-13T12:51:27.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:27.895+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:27.899903+0000) 2019-10-13T12:51:28.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:28.123+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:28.124956+0000) 2019-10-13T12:51:28.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:28.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:28.146225+0000) 2019-10-13T12:51:28.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:28.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:28.665907+0000) 2019-10-13T12:51:31.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:31.375+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.381698+0000) 2019-10-13T12:51:31.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:31.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.382692+0000) 2019-10-13T12:51:31.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:31.379+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.384771+0000) 2019-10-13T12:51:31.388 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:31.383+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.386619+0000) 2019-10-13T12:51:31.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:31.384+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.388476+0000) 2019-10-13T12:51:31.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:31.387+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.390128+0000) 2019-10-13T12:51:31.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:31.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169860 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:31.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:31.387+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.390564+0000) 2019-10-13T12:51:31.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:31.392+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.397230+0000) 2019-10-13T12:51:31.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:31.916+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:31.917164+0000) 2019-10-13T12:51:34.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:34.239+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.242930+0000) 2019-10-13T12:51:34.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:34.523+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:34.527607+0000) 2019-10-13T12:51:36.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:36.028+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.031811+0000) 2019-10-13T12:51:36.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:36.387+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.393503+0000) 2019-10-13T12:51:36.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:36.391+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.393851+0000) 2019-10-13T12:51:36.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:36.388+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.394645+0000) 2019-10-13T12:51:36.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:36.391+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.395382+0000) 2019-10-13T12:51:36.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:36.391+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.395645+0000) 2019-10-13T12:51:36.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:36.391+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.396124+0000) 2019-10-13T12:51:36.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:36.391+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.396445+0000) 2019-10-13T12:51:36.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:36.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:36.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:36.400+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:36.404448+0000) 2019-10-13T12:51:37.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:37.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:37.900116+0000) 2019-10-13T12:51:38.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:38.123+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:38.125153+0000) 2019-10-13T12:51:38.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:38.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:38.146429+0000) 2019-10-13T12:51:38.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:38.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:38.666124+0000) 2019-10-13T12:51:41.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:41.399+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.404043+0000) 2019-10-13T12:51:41.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:41.399+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.405008+0000) 2019-10-13T12:51:41.407 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:41.403+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.406048+0000) 2019-10-13T12:51:41.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:41.403+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.406697+0000) 2019-10-13T12:51:41.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:41.403+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.406870+0000) 2019-10-13T12:51:41.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:41.400+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.407351+0000) 2019-10-13T12:51:41.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:41.403+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.408207+0000) 2019-10-13T12:51:41.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:41.408+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.413414+0000) 2019-10-13T12:51:41.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:41.407+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169923 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:41.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:41.916+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:41.917360+0000) 2019-10-13T12:51:43.706 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:51:43.706 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:43.710 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:51:43.710 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:51:44.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:44.239+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.243170+0000) 2019-10-13T12:51:44.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:44.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:44.527840+0000) 2019-10-13T12:51:46.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:46.040+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.042750+0000) 2019-10-13T12:51:46.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:46.411+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.416241+0000) 2019-10-13T12:51:46.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:46.411+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.416643+0000) 2019-10-13T12:51:46.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:46.412+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.418801+0000) 2019-10-13T12:51:46.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:46.415+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.418868+0000) 2019-10-13T12:51:46.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:46.415+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.419705+0000) 2019-10-13T12:51:46.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:46.415+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.420337+0000) 2019-10-13T12:51:46.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:46.415+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.421650+0000) 2019-10-13T12:51:46.428 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:46.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:46.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:46.420+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:46.427097+0000) 2019-10-13T12:51:47.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:47.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:47.900336+0000) 2019-10-13T12:51:48.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:48.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:48.125376+0000) 2019-10-13T12:51:48.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:48.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:48.146577+0000) 2019-10-13T12:51:48.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:48.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:48.666344+0000) 2019-10-13T12:51:51.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:51.423+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.429672+0000) 2019-10-13T12:51:51.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:51.424+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.430176+0000) 2019-10-13T12:51:51.432 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:51.427+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.430221+0000) 2019-10-13T12:51:51.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:51.427+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.430975+0000) 2019-10-13T12:51:51.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:51.427+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.431386+0000) 2019-10-13T12:51:51.433 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:51.427+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.432117+0000) 2019-10-13T12:51:51.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:51.427+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.432454+0000) 2019-10-13T12:51:51.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:51.428+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.435832+0000) 2019-10-13T12:51:51.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:51.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:51.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:51.912+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:51.917522+0000) 2019-10-13T12:51:54.245 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:54.243+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.243379+0000) 2019-10-13T12:51:54.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:54.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:54.528039+0000) 2019-10-13T12:51:56.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:56.044+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.049382+0000) 2019-10-13T12:51:56.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:56.439+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.443359+0000) 2019-10-13T12:51:56.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:51:56.443+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.445799+0000) 2019-10-13T12:51:56.447 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:56.443+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.445960+0000) 2019-10-13T12:51:56.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:51:56.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.446541+0000) 2019-10-13T12:51:56.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:56.443+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.446971+0000) 2019-10-13T12:51:56.449 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:51:56.440+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.447566+0000) 2019-10-13T12:51:56.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:56.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.447847+0000) 2019-10-13T12:51:56.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:51:56.447+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170015 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:51:56.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:51:56.444+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:56.451696+0000) 2019-10-13T12:51:57.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:51:57.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:57.900563+0000) 2019-10-13T12:51:58.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:51:58.119+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:58.125575+0000) 2019-10-13T12:51:58.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:51:58.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:58.146783+0000) 2019-10-13T12:51:58.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:51:58.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:51:58.666539+0000) 2019-10-13T12:52:01.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:01.451+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.454539+0000) 2019-10-13T12:52:01.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:01.451+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.456715+0000) 2019-10-13T12:52:01.459 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:01.451+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.456716+0000) 2019-10-13T12:52:01.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:01.451+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.457603+0000) 2019-10-13T12:52:01.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:01.455+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.458198+0000) 2019-10-13T12:52:01.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:01.455+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.459000+0000) 2019-10-13T12:52:01.461 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:01.452+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.459784+0000) 2019-10-13T12:52:01.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:01.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:01.468 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:01.460+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.466932+0000) 2019-10-13T12:52:01.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:01.916+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:01.917692+0000) 2019-10-13T12:52:04.245 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:04.243+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.243601+0000) 2019-10-13T12:52:04.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:04.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:04.528253+0000) 2019-10-13T12:52:06.057 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:06.052+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.056032+0000) 2019-10-13T12:52:06.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:06.459+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.465404+0000) 2019-10-13T12:52:06.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:06.463+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.466242+0000) 2019-10-13T12:52:06.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:06.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.467088+0000) 2019-10-13T12:52:06.469 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:06.463+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.467893+0000) 2019-10-13T12:52:06.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:06.463+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.467967+0000) 2019-10-13T12:52:06.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:06.463+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.468801+0000) 2019-10-13T12:52:06.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:06.464+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.469986+0000) 2019-10-13T12:52:06.475 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:06.468+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:06.473953+0000) 2019-10-13T12:52:06.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:06.471+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:07.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:07.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:07.900757+0000) 2019-10-13T12:52:08.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:08.123+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:08.125773+0000) 2019-10-13T12:52:08.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:08.143+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:08.146978+0000) 2019-10-13T12:52:08.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:08.663+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:08.666728+0000) 2019-10-13T12:52:11.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:11.470+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.477619+0000) 2019-10-13T12:52:11.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:11.470+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.477548+0000) 2019-10-13T12:52:11.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:11.472+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.479139+0000) 2019-10-13T12:52:11.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:11.474+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.479301+0000) 2019-10-13T12:52:11.481 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:11.474+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.479264+0000) 2019-10-13T12:52:11.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:11.474+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.480092+0000) 2019-10-13T12:52:11.482 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:11.474+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.480464+0000) 2019-10-13T12:52:11.487 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:11.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:11.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:11.480+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.487526+0000) 2019-10-13T12:52:11.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:11.916+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:11.917815+0000) 2019-10-13T12:52:13.835 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:52:13.835 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:13.839 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:52:13.840 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:14.245 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:14.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.243797+0000) 2019-10-13T12:52:14.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:14.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:14.528475+0000) 2019-10-13T12:52:16.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:16.060+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.062497+0000) 2019-10-13T12:52:16.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:16.482+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.488555+0000) 2019-10-13T12:52:16.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:16.482+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.489166+0000) 2019-10-13T12:52:16.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:16.486+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.491530+0000) 2019-10-13T12:52:16.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:16.484+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.492027+0000) 2019-10-13T12:52:16.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:16.490+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.493880+0000) 2019-10-13T12:52:16.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:16.490+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.493842+0000) 2019-10-13T12:52:16.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:16.490+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.496236+0000) 2019-10-13T12:52:16.499 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:16.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:16.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:16.500+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:16.504320+0000) 2019-10-13T12:52:17.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:17.898+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:17.900986+0000) 2019-10-13T12:52:18.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:18.122+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:18.125987+0000) 2019-10-13T12:52:18.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:18.146+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:18.147172+0000) 2019-10-13T12:52:18.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:18.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:18.666919+0000) 2019-10-13T12:52:21.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:21.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.501568+0000) 2019-10-13T12:52:21.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:21.498+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.502377+0000) 2019-10-13T12:52:21.504 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:21.498+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.502511+0000) 2019-10-13T12:52:21.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:21.496+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.503176+0000) 2019-10-13T12:52:21.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:21.498+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.503270+0000) 2019-10-13T12:52:21.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:21.498+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.504203+0000) 2019-10-13T12:52:21.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:21.498+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.504783+0000) 2019-10-13T12:52:21.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:21.504+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.510134+0000) 2019-10-13T12:52:21.512 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:21.506+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:21.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:21.916+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:21.918017+0000) 2019-10-13T12:52:24.245 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:24.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.244037+0000) 2019-10-13T12:52:24.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:24.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:24.528677+0000) 2019-10-13T12:52:26.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:26.068+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.069042+0000) 2019-10-13T12:52:26.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:26.510+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.514901+0000) 2019-10-13T12:52:26.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:26.510+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.515382+0000) 2019-10-13T12:52:26.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:26.516+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.518258+0000) 2019-10-13T12:52:26.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:26.514+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.520417+0000) 2019-10-13T12:52:26.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:26.514+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.520171+0000) 2019-10-13T12:52:26.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:26.514+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.521373+0000) 2019-10-13T12:52:26.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:26.518+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:26.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:26.518+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.522304+0000) 2019-10-13T12:52:26.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:26.528+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:26.528934+0000) 2019-10-13T12:52:27.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:27.898+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:27.901185+0000) 2019-10-13T12:52:28.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:28.122+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:28.126179+0000) 2019-10-13T12:52:28.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:28.146+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:28.147384+0000) 2019-10-13T12:52:28.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:28.666+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:28.667109+0000) 2019-10-13T12:52:31.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:31.518+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.525472+0000) 2019-10-13T12:52:31.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:31.523+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.526159+0000) 2019-10-13T12:52:31.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:31.522+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.528482+0000) 2019-10-13T12:52:31.530 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:31.522+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.528778+0000) 2019-10-13T12:52:31.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:31.522+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.529039+0000) 2019-10-13T12:52:31.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:31.526+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.530497+0000) 2019-10-13T12:52:31.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:31.526+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.530986+0000) 2019-10-13T12:52:31.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:31.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170232 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:31.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:31.531+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.534635+0000) 2019-10-13T12:52:31.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:31.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:31.918147+0000) 2019-10-13T12:52:34.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:34.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:34.244242+0000) 2019-10-13T12:52:34.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:34.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:34.528910+0000) 2019-10-13T12:52:36.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:36.071+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.075506+0000) 2019-10-13T12:52:36.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:36.530+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.536486+0000) 2019-10-13T12:52:36.538 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:36.535+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.537570+0000) 2019-10-13T12:52:36.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:36.534+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.537987+0000) 2019-10-13T12:52:36.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:36.534+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.538612+0000) 2019-10-13T12:52:36.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:36.534+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.539568+0000) 2019-10-13T12:52:36.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:36.534+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.539973+0000) 2019-10-13T12:52:36.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:36.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.540897+0000) 2019-10-13T12:52:36.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:36.543+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:36.544273+0000) 2019-10-13T12:52:36.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:36.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170263 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:37.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:37.898+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:37.901378+0000) 2019-10-13T12:52:38.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:38.122+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:38.126377+0000) 2019-10-13T12:52:38.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:38.146+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:38.147580+0000) 2019-10-13T12:52:38.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:38.666+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:38.667327+0000) 2019-10-13T12:52:41.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:41.542+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.548581+0000) 2019-10-13T12:52:41.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:41.542+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.549193+0000) 2019-10-13T12:52:41.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:41.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.551760+0000) 2019-10-13T12:52:41.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:41.550+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.554526+0000) 2019-10-13T12:52:41.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:41.551+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.555959+0000) 2019-10-13T12:52:41.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:41.555+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.556788+0000) 2019-10-13T12:52:41.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:41.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170295 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:41.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:41.554+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.558368+0000) 2019-10-13T12:52:41.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:41.554+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.558682+0000) 2019-10-13T12:52:41.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:41.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:41.918317+0000) 2019-10-13T12:52:43.906 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:52:43.907 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:43.911 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:52:43.911 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:52:44.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:44.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:44.244469+0000) 2019-10-13T12:52:44.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:44.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:44.529156+0000) 2019-10-13T12:52:46.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:46.079+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.082040+0000) 2019-10-13T12:52:46.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:46.554+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.560711+0000) 2019-10-13T12:52:46.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:46.558+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.562449+0000) 2019-10-13T12:52:46.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:46.558+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.562955+0000) 2019-10-13T12:52:46.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:46.558+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.565014+0000) 2019-10-13T12:52:46.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:46.562+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.568759+0000) 2019-10-13T12:52:46.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:46.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:46.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:46.567+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.570071+0000) 2019-10-13T12:52:46.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:46.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.569979+0000) 2019-10-13T12:52:46.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:46.571+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:46.574660+0000) 2019-10-13T12:52:47.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:47.894+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:47.901573+0000) 2019-10-13T12:52:48.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:48.122+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:48.126529+0000) 2019-10-13T12:52:48.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:48.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:48.147775+0000) 2019-10-13T12:52:48.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:48.662+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:48.667523+0000) 2019-10-13T12:52:51.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:51.566+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.572965+0000) 2019-10-13T12:52:51.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:51.570+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.573916+0000) 2019-10-13T12:52:51.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:51.570+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.574288+0000) 2019-10-13T12:52:51.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:51.570+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.575474+0000) 2019-10-13T12:52:51.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:51.575+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.576409+0000) 2019-10-13T12:52:51.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:51.570+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.576598+0000) 2019-10-13T12:52:51.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:51.570+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.576898+0000) 2019-10-13T12:52:51.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:51.579+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.581054+0000) 2019-10-13T12:52:51.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:51.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:51.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:51.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:51.918519+0000) 2019-10-13T12:52:54.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:54.238+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:54.244699+0000) 2019-10-13T12:52:54.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:54.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:54.529358+0000) 2019-10-13T12:52:56.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:56.087+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.088697+0000) 2019-10-13T12:52:56.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:52:56.578+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.583766+0000) 2019-10-13T12:52:56.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:52:56.578+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.584770+0000) 2019-10-13T12:52:56.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:56.578+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.585416+0000) 2019-10-13T12:52:56.587 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:56.582+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.585860+0000) 2019-10-13T12:52:56.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:52:56.583+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.587045+0000) 2019-10-13T12:52:56.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:56.582+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.587028+0000) 2019-10-13T12:52:56.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:56.582+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.587694+0000) 2019-10-13T12:52:56.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:52:56.591+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:56.593897+0000) 2019-10-13T12:52:56.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:52:56.590+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:52:57.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:52:57.898+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:57.901774+0000) 2019-10-13T12:52:58.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:52:58.126+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:58.126760+0000) 2019-10-13T12:52:58.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:52:58.142+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:58.148006+0000) 2019-10-13T12:52:58.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:52:58.666+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:52:58.667715+0000) 2019-10-13T12:53:01.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:01.590+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.597341+0000) 2019-10-13T12:53:01.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:01.595+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.597635+0000) 2019-10-13T12:53:01.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:01.594+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.599278+0000) 2019-10-13T12:53:01.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:01.594+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.600026+0000) 2019-10-13T12:53:01.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:01.594+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.600643+0000) 2019-10-13T12:53:01.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:01.598+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.602323+0000) 2019-10-13T12:53:01.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:01.598+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.603327+0000) 2019-10-13T12:53:01.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:01.598+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:01.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:01.611+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.612994+0000) 2019-10-13T12:53:01.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:01.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:01.918690+0000) 2019-10-13T12:53:04.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:04.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:04.244910+0000) 2019-10-13T12:53:04.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:04.526+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:04.529554+0000) 2019-10-13T12:53:06.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:06.095+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.099668+0000) 2019-10-13T12:53:06.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:06.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.608640+0000) 2019-10-13T12:53:06.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:06.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.609020+0000) 2019-10-13T12:53:06.612 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:06.605+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.610944+0000) 2019-10-13T12:53:06.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:06.605+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.611727+0000) 2019-10-13T12:53:06.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:06.605+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.613292+0000) 2019-10-13T12:53:06.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:06.605+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.613561+0000) 2019-10-13T12:53:06.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:06.609+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.613937+0000) 2019-10-13T12:53:06.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:06.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:06.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:06.619+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:06.620732+0000) 2019-10-13T12:53:07.903 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:07.897+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:07.901977+0000) 2019-10-13T12:53:08.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:08.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:08.126963+0000) 2019-10-13T12:53:08.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:08.141+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:08.148227+0000) 2019-10-13T12:53:08.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:08.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:08.667914+0000) 2019-10-13T12:53:11.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:11.613+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.620567+0000) 2019-10-13T12:53:11.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:11.613+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.621318+0000) 2019-10-13T12:53:11.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:11.619+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.621810+0000) 2019-10-13T12:53:11.624 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:11.617+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.621886+0000) 2019-10-13T12:53:11.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:11.617+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.622926+0000) 2019-10-13T12:53:11.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:11.617+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.623306+0000) 2019-10-13T12:53:11.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:11.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.623717+0000) 2019-10-13T12:53:11.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:11.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170480 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:11.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:11.627+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.630396+0000) 2019-10-13T12:53:11.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:11.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:11.918857+0000) 2019-10-13T12:53:14.004 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:53:14.004 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:14.008 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:53:14.008 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:14.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:14.241+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:14.245182+0000) 2019-10-13T12:53:14.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:14.525+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:14.529777+0000) 2019-10-13T12:53:16.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:16.103+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.106294+0000) 2019-10-13T12:53:16.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:16.625+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.632682+0000) 2019-10-13T12:53:16.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:16.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.633009+0000) 2019-10-13T12:53:16.635 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:16.629+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.633953+0000) 2019-10-13T12:53:16.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:16.629+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.634944+0000) 2019-10-13T12:53:16.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:16.629+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.635307+0000) 2019-10-13T12:53:16.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:16.631+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.635866+0000) 2019-10-13T12:53:16.637 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:16.629+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.636037+0000) 2019-10-13T12:53:16.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:16.639+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:16.640739+0000) 2019-10-13T12:53:16.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:16.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:17.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:17.901+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:17.902172+0000) 2019-10-13T12:53:18.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:18.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:18.127196+0000) 2019-10-13T12:53:18.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:18.145+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:18.148422+0000) 2019-10-13T12:53:18.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:18.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:18.668123+0000) 2019-10-13T12:53:21.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:21.641+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.644545+0000) 2019-10-13T12:53:21.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:21.643+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.645204+0000) 2019-10-13T12:53:21.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:21.641+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.645361+0000) 2019-10-13T12:53:21.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:21.645+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.645725+0000) 2019-10-13T12:53:21.648 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:21.645+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.646415+0000) 2019-10-13T12:53:21.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:21.645+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.646980+0000) 2019-10-13T12:53:21.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:21.645+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.647441+0000) 2019-10-13T12:53:21.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:21.653+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:21.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:21.655+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.656514+0000) 2019-10-13T12:53:21.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:21.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:21.919029+0000) 2019-10-13T12:53:24.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:24.241+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:24.245389+0000) 2019-10-13T12:53:24.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:24.529+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:24.529941+0000) 2019-10-13T12:53:26.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:26.111+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.112882+0000) 2019-10-13T12:53:26.659 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:26.655+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.657849+0000) 2019-10-13T12:53:26.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:26.657+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.658090+0000) 2019-10-13T12:53:26.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:26.657+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.658593+0000) 2019-10-13T12:53:26.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:26.657+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.659243+0000) 2019-10-13T12:53:26.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:26.657+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.661230+0000) 2019-10-13T12:53:26.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:26.657+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.661538+0000) 2019-10-13T12:53:26.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:26.661+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.662394+0000) 2019-10-13T12:53:26.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:26.663+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:26.665519+0000) 2019-10-13T12:53:26.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:26.665+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:27.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:27.901+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:27.902372+0000) 2019-10-13T12:53:28.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:28.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:28.127398+0000) 2019-10-13T12:53:28.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:28.145+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:28.148648+0000) 2019-10-13T12:53:28.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:28.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:28.668313+0000) 2019-10-13T12:53:31.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:31.665+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.669124+0000) 2019-10-13T12:53:31.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:31.667+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.670246+0000) 2019-10-13T12:53:31.672 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:31.669+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.670560+0000) 2019-10-13T12:53:31.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:31.669+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.671276+0000) 2019-10-13T12:53:31.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:31.669+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.671577+0000) 2019-10-13T12:53:31.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:31.669+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.672709+0000) 2019-10-13T12:53:31.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:31.669+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.673080+0000) 2019-10-13T12:53:31.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:31.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170604 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:31.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:31.675+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.678601+0000) 2019-10-13T12:53:31.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:31.915+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:31.919199+0000) 2019-10-13T12:53:34.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:34.245+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:34.245590+0000) 2019-10-13T12:53:34.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:34.529+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:34.530182+0000) 2019-10-13T12:53:36.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:36.122+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.124259+0000) 2019-10-13T12:53:36.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:36.678+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.681241+0000) 2019-10-13T12:53:36.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:36.677+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.681330+0000) 2019-10-13T12:53:36.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:36.681+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.682078+0000) 2019-10-13T12:53:36.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:36.681+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.682309+0000) 2019-10-13T12:53:36.685 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:36.681+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.684172+0000) 2019-10-13T12:53:36.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:36.681+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.684685+0000) 2019-10-13T12:53:36.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:36.681+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.685244+0000) 2019-10-13T12:53:36.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:36.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170634 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:36.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:36.690+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:36.694043+0000) 2019-10-13T12:53:37.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:37.901+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:37.902584+0000) 2019-10-13T12:53:38.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:38.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:38.127603+0000) 2019-10-13T12:53:38.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:38.145+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:38.148845+0000) 2019-10-13T12:53:38.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:38.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:38.668499+0000) 2019-10-13T12:53:41.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:41.690+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.692222+0000) 2019-10-13T12:53:41.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:41.689+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.692525+0000) 2019-10-13T12:53:41.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:41.689+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.693184+0000) 2019-10-13T12:53:41.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:41.689+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.693302+0000) 2019-10-13T12:53:41.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:41.693+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.694188+0000) 2019-10-13T12:53:41.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:41.693+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.694554+0000) 2019-10-13T12:53:41.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:41.693+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.695024+0000) 2019-10-13T12:53:41.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:41.698+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.700405+0000) 2019-10-13T12:53:41.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:41.697+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:41.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:41.914+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:41.919409+0000) 2019-10-13T12:53:44.125 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:53:44.125 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:44.129 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:53:44.129 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:53:44.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:44.245+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:44.245781+0000) 2019-10-13T12:53:44.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:44.529+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:44.530380+0000) 2019-10-13T12:53:46.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:46.130+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.130927+0000) 2019-10-13T12:53:46.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:46.698+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.704039+0000) 2019-10-13T12:53:46.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:46.701+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.704350+0000) 2019-10-13T12:53:46.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:46.701+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.704729+0000) 2019-10-13T12:53:46.707 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:46.701+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.705658+0000) 2019-10-13T12:53:46.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:46.705+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.705948+0000) 2019-10-13T12:53:46.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:46.705+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.708301+0000) 2019-10-13T12:53:46.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:46.705+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.708628+0000) 2019-10-13T12:53:46.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:46.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:46.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:46.714+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:46.716344+0000) 2019-10-13T12:53:47.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:47.901+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:47.902783+0000) 2019-10-13T12:53:48.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:48.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:48.127823+0000) 2019-10-13T12:53:48.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:48.145+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:48.149043+0000) 2019-10-13T12:53:48.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:48.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:48.668698+0000) 2019-10-13T12:53:51.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:51.713+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.716215+0000) 2019-10-13T12:53:51.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:51.713+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.717252+0000) 2019-10-13T12:53:51.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:51.714+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.718132+0000) 2019-10-13T12:53:51.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:51.717+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.718359+0000) 2019-10-13T12:53:51.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:51.717+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.718888+0000) 2019-10-13T12:53:51.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:51.717+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.719252+0000) 2019-10-13T12:53:51.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:51.717+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.720022+0000) 2019-10-13T12:53:51.727 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:51.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170728 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:51.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:51.722+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.726734+0000) 2019-10-13T12:53:51.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:51.914+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:51.919574+0000) 2019-10-13T12:53:54.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:54.245+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:54.245988+0000) 2019-10-13T12:53:54.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:54.529+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:54.530581+0000) 2019-10-13T12:53:56.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:56.134+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.137542+0000) 2019-10-13T12:53:56.730 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:53:56.725+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.728719+0000) 2019-10-13T12:53:56.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:56.729+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.729867+0000) 2019-10-13T12:53:56.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:56.729+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.730974+0000) 2019-10-13T12:53:56.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:53:56.726+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.731911+0000) 2019-10-13T12:53:56.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:56.729+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.731904+0000) 2019-10-13T12:53:56.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:53:56.729+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.732299+0000) 2019-10-13T12:53:56.734 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:56.729+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.732672+0000) 2019-10-13T12:53:56.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:53:56.734+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:56.737913+0000) 2019-10-13T12:53:56.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:53:56.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:53:57.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:53:57.901+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:57.902985+0000) 2019-10-13T12:53:58.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:53:58.125+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:58.128037+0000) 2019-10-13T12:53:58.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:53:58.149+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:58.149240+0000) 2019-10-13T12:53:58.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:53:58.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:53:58.668891+0000) 2019-10-13T12:54:01.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:01.736+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.741562+0000) 2019-10-13T12:54:01.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:01.740+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.742354+0000) 2019-10-13T12:54:01.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:01.740+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.744364+0000) 2019-10-13T12:54:01.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:01.740+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.744785+0000) 2019-10-13T12:54:01.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:01.744+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.746974+0000) 2019-10-13T12:54:01.749 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:01.742+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.747580+0000) 2019-10-13T12:54:01.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:01.746+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.750066+0000) 2019-10-13T12:54:01.752 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:01.748+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:01.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:01.748+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.751013+0000) 2019-10-13T12:54:01.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:01.914+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:01.919779+0000) 2019-10-13T12:54:04.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:04.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:04.246187+0000) 2019-10-13T12:54:04.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:04.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:04.530734+0000) 2019-10-13T12:54:06.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:06.142+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.144140+0000) 2019-10-13T12:54:06.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:06.752+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.754382+0000) 2019-10-13T12:54:06.756 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:06.752+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.754806+0000) 2019-10-13T12:54:06.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:06.752+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.756631+0000) 2019-10-13T12:54:06.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:06.754+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.757166+0000) 2019-10-13T12:54:06.759 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:06.752+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.757441+0000) 2019-10-13T12:54:06.759 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:06.756+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.758108+0000) 2019-10-13T12:54:06.760 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:06.756+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.759404+0000) 2019-10-13T12:54:06.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:06.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:06.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:06.762+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:06.764351+0000) 2019-10-13T12:54:07.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:07.900+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:07.903188+0000) 2019-10-13T12:54:08.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:08.124+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:08.128224+0000) 2019-10-13T12:54:08.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:08.144+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:08.149453+0000) 2019-10-13T12:54:08.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:08.664+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:08.669118+0000) 2019-10-13T12:54:11.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:11.764+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.767120+0000) 2019-10-13T12:54:11.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:11.766+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.769028+0000) 2019-10-13T12:54:11.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:11.764+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.769489+0000) 2019-10-13T12:54:11.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:11.768+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.770203+0000) 2019-10-13T12:54:11.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:11.768+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.770517+0000) 2019-10-13T12:54:11.772 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:11.768+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.771126+0000) 2019-10-13T12:54:11.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:11.768+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.771189+0000) 2019-10-13T12:54:11.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:11.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170851 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:11.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:11.774+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.777435+0000) 2019-10-13T12:54:11.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:11.918+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:11.919981+0000) 2019-10-13T12:54:14.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:14.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:14.246401+0000) 2019-10-13T12:54:14.254 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:54:14.254 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:14.258 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:54:14.258 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:14.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:14.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:14.530967+0000) 2019-10-13T12:54:16.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:16.150+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.150597+0000) 2019-10-13T12:54:16.778 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:16.772+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.776493+0000) 2019-10-13T12:54:16.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:16.772+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.777076+0000) 2019-10-13T12:54:16.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:16.772+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.777418+0000) 2019-10-13T12:54:16.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:16.776+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.777815+0000) 2019-10-13T12:54:16.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:16.776+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.778747+0000) 2019-10-13T12:54:16.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:16.778+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.780160+0000) 2019-10-13T12:54:16.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:16.776+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.780312+0000) 2019-10-13T12:54:16.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:16.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:16.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:16.782+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:16.785425+0000) 2019-10-13T12:54:17.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:17.900+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:17.903396+0000) 2019-10-13T12:54:18.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:18.124+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:18.128417+0000) 2019-10-13T12:54:18.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:18.148+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:18.149649+0000) 2019-10-13T12:54:18.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:18.668+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:18.669297+0000) 2019-10-13T12:54:21.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:21.784+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.787578+0000) 2019-10-13T12:54:21.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:21.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.788854+0000) 2019-10-13T12:54:21.792 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:21.788+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.790992+0000) 2019-10-13T12:54:21.793 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:21.788+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.792075+0000) 2019-10-13T12:54:21.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:21.788+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.792665+0000) 2019-10-13T12:54:21.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:21.788+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.793577+0000) 2019-10-13T12:54:21.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:21.792+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.794078+0000) 2019-10-13T12:54:21.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:21.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:21.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:21.794+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.797215+0000) 2019-10-13T12:54:21.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:21.918+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:21.920154+0000) 2019-10-13T12:54:24.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:24.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:24.246600+0000) 2019-10-13T12:54:24.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:24.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:24.531166+0000) 2019-10-13T12:54:26.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:26.154+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.157235+0000) 2019-10-13T12:54:26.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:26.798+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.800298+0000) 2019-10-13T12:54:26.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:26.796+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.800543+0000) 2019-10-13T12:54:26.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:26.796+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.800707+0000) 2019-10-13T12:54:26.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:26.796+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.801278+0000) 2019-10-13T12:54:26.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:26.800+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.803210+0000) 2019-10-13T12:54:26.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:26.800+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.804634+0000) 2019-10-13T12:54:26.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:26.800+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.805264+0000) 2019-10-13T12:54:26.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:26.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:26.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:26.806+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:26.809477+0000) 2019-10-13T12:54:27.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:27.900+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:27.903588+0000) 2019-10-13T12:54:28.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:28.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:28.128622+0000) 2019-10-13T12:54:28.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:28.148+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:28.149854+0000) 2019-10-13T12:54:28.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:28.668+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:28.669492+0000) 2019-10-13T12:54:31.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:31.808+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.810860+0000) 2019-10-13T12:54:31.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:31.808+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.811178+0000) 2019-10-13T12:54:31.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:31.808+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.811773+0000) 2019-10-13T12:54:31.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:31.808+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.812241+0000) 2019-10-13T12:54:31.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:31.810+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.813005+0000) 2019-10-13T12:54:31.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:31.808+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.813349+0000) 2019-10-13T12:54:31.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:31.808+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.813612+0000) 2019-10-13T12:54:31.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:31.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170975 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:31.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:31.818+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.820651+0000) 2019-10-13T12:54:31.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:31.918+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:31.920369+0000) 2019-10-13T12:54:34.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:34.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:34.246799+0000) 2019-10-13T12:54:34.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:34.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:34.531363+0000) 2019-10-13T12:54:36.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:36.162+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.163000+0000) 2019-10-13T12:54:36.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:36.820+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.823471+0000) 2019-10-13T12:54:36.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:36.818+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.823961+0000) 2019-10-13T12:54:36.826 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:36.820+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.824043+0000) 2019-10-13T12:54:36.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:36.820+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.824939+0000) 2019-10-13T12:54:36.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:36.820+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.825039+0000) 2019-10-13T12:54:36.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:36.820+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.824398+0000) 2019-10-13T12:54:36.827 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:36.824+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.826041+0000) 2019-10-13T12:54:36.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:36.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:36.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:36.830+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:36.832127+0000) 2019-10-13T12:54:37.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:37.900+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:37.903788+0000) 2019-10-13T12:54:38.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:38.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:38.128821+0000) 2019-10-13T12:54:38.151 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:38.148+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:38.150039+0000) 2019-10-13T12:54:38.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:38.668+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:38.669716+0000) 2019-10-13T12:54:41.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:41.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.834352+0000) 2019-10-13T12:54:41.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:41.832+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.834414+0000) 2019-10-13T12:54:41.836 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:41.832+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.835121+0000) 2019-10-13T12:54:41.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:41.832+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.836159+0000) 2019-10-13T12:54:41.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:41.832+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.835909+0000) 2019-10-13T12:54:41.839 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:41.836+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.837993+0000) 2019-10-13T12:54:41.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:41.836+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.838277+0000) 2019-10-13T12:54:41.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:41.837+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.843198+0000) 2019-10-13T12:54:41.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:41.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:41.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:41.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:41.920515+0000) 2019-10-13T12:54:44.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:44.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:44.247033+0000) 2019-10-13T12:54:44.319 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:54:44.320 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:44.323 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:54:44.323 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:54:44.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:44.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:44.531561+0000) 2019-10-13T12:54:46.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:46.165+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.169589+0000) 2019-10-13T12:54:46.848 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:46.844+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.846672+0000) 2019-10-13T12:54:46.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:46.844+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.847997+0000) 2019-10-13T12:54:46.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:46.844+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.848745+0000) 2019-10-13T12:54:46.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:46.844+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.848823+0000) 2019-10-13T12:54:46.851 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:46.844+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.849376+0000) 2019-10-13T12:54:46.852 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:46.848+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.850610+0000) 2019-10-13T12:54:46.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:46.845+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.851450+0000) 2019-10-13T12:54:46.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:46.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:46.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:46.853+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:46.858375+0000) 2019-10-13T12:54:47.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:47.900+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:47.903985+0000) 2019-10-13T12:54:48.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:48.124+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:48.129039+0000) 2019-10-13T12:54:48.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:48.148+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:48.150235+0000) 2019-10-13T12:54:48.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:48.668+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:48.669938+0000) 2019-10-13T12:54:51.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:51.856+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.858749+0000) 2019-10-13T12:54:51.861 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:51.856+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.859499+0000) 2019-10-13T12:54:51.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:51.856+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.859879+0000) 2019-10-13T12:54:51.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:51.857+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.860743+0000) 2019-10-13T12:54:51.862 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:51.856+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.860331+0000) 2019-10-13T12:54:51.863 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:51.856+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.861422+0000) 2019-10-13T12:54:51.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:51.860+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.862255+0000) 2019-10-13T12:54:51.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:51.861+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.865626+0000) 2019-10-13T12:54:51.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:51.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171099 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:51.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:51.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:51.920732+0000) 2019-10-13T12:54:54.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:54.244+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:54.247238+0000) 2019-10-13T12:54:54.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:54.528+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:54.531766+0000) 2019-10-13T12:54:56.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:56.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.176147+0000) 2019-10-13T12:54:56.871 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:56.864+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.869156+0000) 2019-10-13T12:54:56.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:56.868+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.870898+0000) 2019-10-13T12:54:56.873 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:54:56.868+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.871159+0000) 2019-10-13T12:54:56.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:56.868+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.871569+0000) 2019-10-13T12:54:56.874 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:56.868+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.872372+0000) 2019-10-13T12:54:56.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:54:56.869+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.872843+0000) 2019-10-13T12:54:56.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:54:56.868+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.873237+0000) 2019-10-13T12:54:56.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:54:56.872+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:54:56.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:54:56.877+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:56.881576+0000) 2019-10-13T12:54:57.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:54:57.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:57.904181+0000) 2019-10-13T12:54:58.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:54:58.123+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:58.129239+0000) 2019-10-13T12:54:58.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:54:58.147+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:58.150428+0000) 2019-10-13T12:54:58.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:54:58.667+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:54:58.670138+0000) 2019-10-13T12:55:01.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:01.875+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.879881+0000) 2019-10-13T12:55:01.882 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:01.875+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.880898+0000) 2019-10-13T12:55:01.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:01.879+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.882189+0000) 2019-10-13T12:55:01.884 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:01.879+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.882721+0000) 2019-10-13T12:55:01.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:01.879+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.883166+0000) 2019-10-13T12:55:01.885 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:01.879+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.883269+0000) 2019-10-13T12:55:01.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:01.877+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.883976+0000) 2019-10-13T12:55:01.891 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:01.885+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.889577+0000) 2019-10-13T12:55:01.891 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:01.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:01.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:01.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:01.920891+0000) 2019-10-13T12:55:04.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:04.243+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:04.247439+0000) 2019-10-13T12:55:04.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:04.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:04.531953+0000) 2019-10-13T12:55:06.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:06.181+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.182821+0000) 2019-10-13T12:55:06.895 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:06.889+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.894238+0000) 2019-10-13T12:55:06.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:06.891+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.894304+0000) 2019-10-13T12:55:06.898 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:06.891+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.894683+0000) 2019-10-13T12:55:06.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:06.891+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.894862+0000) 2019-10-13T12:55:06.898 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:06.891+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.895193+0000) 2019-10-13T12:55:06.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:06.891+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.895601+0000) 2019-10-13T12:55:06.899 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:06.891+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.897699+0000) 2019-10-13T12:55:06.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:06.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:06.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:06.901+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:06.904801+0000) 2019-10-13T12:55:07.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:07.903+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:07.904382+0000) 2019-10-13T12:55:08.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:08.127+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:08.129465+0000) 2019-10-13T12:55:08.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:08.147+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:08.150635+0000) 2019-10-13T12:55:08.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:08.667+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:08.670311+0000) 2019-10-13T12:55:11.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:11.901+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.905691+0000) 2019-10-13T12:55:11.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:11.899+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.904879+0000) 2019-10-13T12:55:11.911 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:11.899+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.905704+0000) 2019-10-13T12:55:11.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:11.903+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.906037+0000) 2019-10-13T12:55:11.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:11.903+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.907869+0000) 2019-10-13T12:55:11.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:11.907+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.909974+0000) 2019-10-13T12:55:11.914 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:11.907+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.912729+0000) 2019-10-13T12:55:11.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:11.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:11.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:11.913+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.916935+0000) 2019-10-13T12:55:11.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:11.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:11.921006+0000) 2019-10-13T12:55:14.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:14.243+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:14.247651+0000) 2019-10-13T12:55:14.469 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:55:14.469 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:14.473 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:55:14.473 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:14.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:14.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:14.532159+0000) 2019-10-13T12:55:16.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:16.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.189605+0000) 2019-10-13T12:55:16.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:16.911+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.917397+0000) 2019-10-13T12:55:16.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:16.915+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.918491+0000) 2019-10-13T12:55:16.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:16.915+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.918927+0000) 2019-10-13T12:55:16.921 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:16.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.919506+0000) 2019-10-13T12:55:16.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:16.915+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.919699+0000) 2019-10-13T12:55:16.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:16.915+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.920230+0000) 2019-10-13T12:55:16.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:16.915+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.920560+0000) 2019-10-13T12:55:16.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:16.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:16.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:16.921+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:16.926542+0000) 2019-10-13T12:55:17.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:17.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:17.904570+0000) 2019-10-13T12:55:18.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:18.127+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:18.129668+0000) 2019-10-13T12:55:18.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:18.147+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:18.150838+0000) 2019-10-13T12:55:18.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:18.667+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:18.670498+0000) 2019-10-13T12:55:21.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:21.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.921126+0000) 2019-10-13T12:55:21.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:21.923+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.929385+0000) 2019-10-13T12:55:21.932 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:21.927+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.930303+0000) 2019-10-13T12:55:21.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:21.927+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.930679+0000) 2019-10-13T12:55:21.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:21.925+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.931244+0000) 2019-10-13T12:55:21.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:21.927+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.931244+0000) 2019-10-13T12:55:21.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:21.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.932441+0000) 2019-10-13T12:55:21.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:21.927+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.932779+0000) 2019-10-13T12:55:21.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:21.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:21.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:21.933+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:21.939483+0000) 2019-10-13T12:55:24.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:24.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:24.247849+0000) 2019-10-13T12:55:24.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:24.531+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:24.532371+0000) 2019-10-13T12:55:26.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:26.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.196196+0000) 2019-10-13T12:55:26.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:26.939+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.942542+0000) 2019-10-13T12:55:26.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:26.939+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.942631+0000) 2019-10-13T12:55:26.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:26.937+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.943598+0000) 2019-10-13T12:55:26.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:26.939+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.943654+0000) 2019-10-13T12:55:26.946 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:26.939+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.943992+0000) 2019-10-13T12:55:26.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:26.939+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.945353+0000) 2019-10-13T12:55:26.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:26.943+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.945779+0000) 2019-10-13T12:55:26.952 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:26.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:26.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:26.945+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:26.950599+0000) 2019-10-13T12:55:27.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:27.899+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:27.904743+0000) 2019-10-13T12:55:28.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:28.127+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:28.129924+0000) 2019-10-13T12:55:28.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:28.147+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:28.151030+0000) 2019-10-13T12:55:28.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:28.667+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:28.670643+0000) 2019-10-13T12:55:31.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:31.917+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.921355+0000) 2019-10-13T12:55:31.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:31.947+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.953463+0000) 2019-10-13T12:55:31.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:31.951+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.955138+0000) 2019-10-13T12:55:31.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:31.951+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.955652+0000) 2019-10-13T12:55:31.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:31.951+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.956080+0000) 2019-10-13T12:55:31.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:31.951+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.956245+0000) 2019-10-13T12:55:31.958 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:31.951+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.956572+0000) 2019-10-13T12:55:31.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:31.953+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.958595+0000) 2019-10-13T12:55:31.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:31.959+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:31.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:31.961+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:31.964759+0000) 2019-10-13T12:55:34.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:34.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:34.248064+0000) 2019-10-13T12:55:34.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:34.531+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:34.532576+0000) 2019-10-13T12:55:36.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:36.201+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.202760+0000) 2019-10-13T12:55:36.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:36.963+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.966899+0000) 2019-10-13T12:55:36.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:36.963+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.967488+0000) 2019-10-13T12:55:36.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:36.965+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.968763+0000) 2019-10-13T12:55:36.971 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:36.963+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.969327+0000) 2019-10-13T12:55:36.971 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:36.963+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.969128+0000) 2019-10-13T12:55:36.971 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:36.967+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.969802+0000) 2019-10-13T12:55:36.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:36.967+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.970158+0000) 2019-10-13T12:55:36.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:36.969+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:36.974680+0000) 2019-10-13T12:55:36.977 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:36.971+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:37.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:37.903+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:37.904956+0000) 2019-10-13T12:55:38.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:38.127+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:38.130142+0000) 2019-10-13T12:55:38.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:38.147+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:38.151228+0000) 2019-10-13T12:55:38.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:38.667+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:38.670869+0000) 2019-10-13T12:55:41.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:41.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.921591+0000) 2019-10-13T12:55:41.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:41.975+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.978951+0000) 2019-10-13T12:55:41.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:41.973+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.979611+0000) 2019-10-13T12:55:41.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:41.975+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.979877+0000) 2019-10-13T12:55:41.983 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:41.979+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.981801+0000) 2019-10-13T12:55:41.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:41.979+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.981895+0000) 2019-10-13T12:55:41.984 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:41.979+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.982635+0000) 2019-10-13T12:55:41.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:41.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.983087+0000) 2019-10-13T12:55:41.989 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:41.983+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171409 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:41.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:41.985+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:41.989363+0000) 2019-10-13T12:55:44.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:44.243+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:44.248217+0000) 2019-10-13T12:55:44.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:44.527+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:44.532773+0000) 2019-10-13T12:55:44.580 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:55:44.580 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:44.584 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:55:44.584 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:55:46.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:46.208+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.209378+0000) 2019-10-13T12:55:46.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:46.984+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.991347+0000) 2019-10-13T12:55:46.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:46.991+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.994314+0000) 2019-10-13T12:55:46.996 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:46.991+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.994527+0000) 2019-10-13T12:55:46.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:46.991+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.995098+0000) 2019-10-13T12:55:46.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:46.991+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:46.996897+0000) 2019-10-13T12:55:47.001 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:46.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:47.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:46.995+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:47.000162+0000) 2019-10-13T12:55:47.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:46.995+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:47.000723+0000) 2019-10-13T12:55:47.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:47.000+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:47.006387+0000) 2019-10-13T12:55:47.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:47.903+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:47.905207+0000) 2019-10-13T12:55:48.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:48.127+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:48.130359+0000) 2019-10-13T12:55:48.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:48.151+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:48.151426+0000) 2019-10-13T12:55:48.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:48.671+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:48.671121+0000) 2019-10-13T12:55:51.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:51.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:51.921807+0000) 2019-10-13T12:55:52.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:51.999+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.003694+0000) 2019-10-13T12:55:52.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:51.999+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.004109+0000) 2019-10-13T12:55:52.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:52.000+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.004977+0000) 2019-10-13T12:55:52.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:52.003+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.005966+0000) 2019-10-13T12:55:52.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:52.003+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.006342+0000) 2019-10-13T12:55:52.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:52.003+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.006874+0000) 2019-10-13T12:55:52.008 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:52.003+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.007278+0000) 2019-10-13T12:55:52.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:52.004+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:52.011735+0000) 2019-10-13T12:55:52.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:52.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171470 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:54.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:54.246+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:54.248453+0000) 2019-10-13T12:55:54.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:54.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:54.532972+0000) 2019-10-13T12:55:56.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:56.212+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:56.215814+0000) 2019-10-13T12:55:57.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:55:57.010+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.016108+0000) 2019-10-13T12:55:57.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:55:57.010+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.016374+0000) 2019-10-13T12:55:57.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:57.010+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.016769+0000) 2019-10-13T12:55:57.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:57.010+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.017059+0000) 2019-10-13T12:55:57.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:55:57.012+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.018388+0000) 2019-10-13T12:55:57.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:57.014+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.018366+0000) 2019-10-13T12:55:57.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:57.014+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.018770+0000) 2019-10-13T12:55:57.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:55:57.020+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.024252+0000) 2019-10-13T12:55:57.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:55:57.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:55:57.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:55:57.902+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:57.905379+0000) 2019-10-13T12:55:58.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:55:58.126+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:58.130557+0000) 2019-10-13T12:55:58.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:55:58.146+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:58.151594+0000) 2019-10-13T12:55:58.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:55:58.666+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:55:58.671321+0000) 2019-10-13T12:56:01.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:01.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:01.922033+0000) 2019-10-13T12:56:02.029 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:02.022+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.027948+0000) 2019-10-13T12:56:02.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:02.022+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.028467+0000) 2019-10-13T12:56:02.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:02.026+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.031310+0000) 2019-10-13T12:56:02.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:02.026+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.033090+0000) 2019-10-13T12:56:02.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:02.028+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.035461+0000) 2019-10-13T12:56:02.038 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:02.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171533 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:02.038 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:02.030+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.036772+0000) 2019-10-13T12:56:02.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:02.030+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.037392+0000) 2019-10-13T12:56:02.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:02.040+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:02.044424+0000) 2019-10-13T12:56:04.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:04.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:04.248681+0000) 2019-10-13T12:56:04.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:04.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:04.533209+0000) 2019-10-13T12:56:06.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:06.216+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:06.222284+0000) 2019-10-13T12:56:07.041 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:07.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.040017+0000) 2019-10-13T12:56:07.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:07.034+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.040414+0000) 2019-10-13T12:56:07.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:07.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.041233+0000) 2019-10-13T12:56:07.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:07.038+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.041790+0000) 2019-10-13T12:56:07.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:07.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.044213+0000) 2019-10-13T12:56:07.046 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:07.038+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.044350+0000) 2019-10-13T12:56:07.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:07.040+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.044973+0000) 2019-10-13T12:56:07.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:07.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:07.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:07.044+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.049306+0000) 2019-10-13T12:56:07.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:07.898+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:07.905557+0000) 2019-10-13T12:56:08.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:08.126+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:08.130745+0000) 2019-10-13T12:56:08.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:08.146+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:08.151811+0000) 2019-10-13T12:56:08.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:08.670+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:08.671563+0000) 2019-10-13T12:56:11.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:11.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:11.922243+0000) 2019-10-13T12:56:12.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:12.046+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.051885+0000) 2019-10-13T12:56:12.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:12.046+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.052708+0000) 2019-10-13T12:56:12.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:12.046+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.052991+0000) 2019-10-13T12:56:12.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:12.050+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.055448+0000) 2019-10-13T12:56:12.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:12.048+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.055223+0000) 2019-10-13T12:56:12.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:12.050+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.056520+0000) 2019-10-13T12:56:12.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:12.050+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.057197+0000) 2019-10-13T12:56:12.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:12.052+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:12.059020+0000) 2019-10-13T12:56:12.061 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:12.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:14.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:14.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:14.248881+0000) 2019-10-13T12:56:14.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:14.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:14.533408+0000) 2019-10-13T12:56:14.674 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:56:14.674 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:14.678 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:56:14.678 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:16.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:16.224+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:16.228917+0000) 2019-10-13T12:56:17.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:17.056+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.062699+0000) 2019-10-13T12:56:17.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:17.058+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.063555+0000) 2019-10-13T12:56:17.065 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:17.058+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.064125+0000) 2019-10-13T12:56:17.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:17.062+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.065784+0000) 2019-10-13T12:56:17.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:17.062+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.066400+0000) 2019-10-13T12:56:17.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:17.062+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.066640+0000) 2019-10-13T12:56:17.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:17.062+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.068090+0000) 2019-10-13T12:56:17.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:17.064+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.070136+0000) 2019-10-13T12:56:17.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:17.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171628 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:17.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:17.902+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:17.905761+0000) 2019-10-13T12:56:18.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:18.126+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:18.130949+0000) 2019-10-13T12:56:18.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:18.150+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:18.152013+0000) 2019-10-13T12:56:18.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:18.670+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:18.671753+0000) 2019-10-13T12:56:21.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:21.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:21.922455+0000) 2019-10-13T12:56:22.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:22.070+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.074894+0000) 2019-10-13T12:56:22.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:22.070+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.075493+0000) 2019-10-13T12:56:22.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:22.070+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.076025+0000) 2019-10-13T12:56:22.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:22.070+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.077507+0000) 2019-10-13T12:56:22.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:22.074+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.079939+0000) 2019-10-13T12:56:22.083 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:22.078+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.081921+0000) 2019-10-13T12:56:22.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:22.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:22.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:22.076+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.083347+0000) 2019-10-13T12:56:22.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:22.076+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:22.083321+0000) 2019-10-13T12:56:24.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:24.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:24.249074+0000) 2019-10-13T12:56:24.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:24.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:24.533615+0000) 2019-10-13T12:56:26.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:26.228+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:26.235526+0000) 2019-10-13T12:56:27.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:27.082+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.085668+0000) 2019-10-13T12:56:27.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:27.082+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.087272+0000) 2019-10-13T12:56:27.090 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:27.082+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.089115+0000) 2019-10-13T12:56:27.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:27.086+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.090726+0000) 2019-10-13T12:56:27.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:27.086+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.091279+0000) 2019-10-13T12:56:27.093 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:27.086+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.091768+0000) 2019-10-13T12:56:27.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:27.088+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.092528+0000) 2019-10-13T12:56:27.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:27.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:27.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:27.088+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.095468+0000) 2019-10-13T12:56:27.907 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:27.902+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:27.905950+0000) 2019-10-13T12:56:28.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:28.130+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:28.131164+0000) 2019-10-13T12:56:28.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:28.150+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:28.152269+0000) 2019-10-13T12:56:28.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:28.670+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:28.671958+0000) 2019-10-13T12:56:31.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:31.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:31.922629+0000) 2019-10-13T12:56:32.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:32.094+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.097924+0000) 2019-10-13T12:56:32.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:32.094+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.100427+0000) 2019-10-13T12:56:32.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:32.094+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.100756+0000) 2019-10-13T12:56:32.103 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:32.094+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.101216+0000) 2019-10-13T12:56:32.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:32.096+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.103418+0000) 2019-10-13T12:56:32.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:32.098+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.103896+0000) 2019-10-13T12:56:32.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:32.098+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.105153+0000) 2019-10-13T12:56:32.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:32.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:32.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:32.104+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:32.110336+0000) 2019-10-13T12:56:34.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:34.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:34.249280+0000) 2019-10-13T12:56:34.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:34.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:34.533841+0000) 2019-10-13T12:56:36.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:36.236+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:36.242020+0000) 2019-10-13T12:56:37.111 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:37.102+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.109708+0000) 2019-10-13T12:56:37.113 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:37.106+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.112006+0000) 2019-10-13T12:56:37.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:37.106+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.112470+0000) 2019-10-13T12:56:37.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:37.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.115536+0000) 2019-10-13T12:56:37.119 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:37.112+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.117565+0000) 2019-10-13T12:56:37.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:37.114+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:37.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:37.114+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.118523+0000) 2019-10-13T12:56:37.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:37.114+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.118861+0000) 2019-10-13T12:56:37.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:37.116+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.122079+0000) 2019-10-13T12:56:37.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:37.902+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:37.906152+0000) 2019-10-13T12:56:38.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:38.130+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:38.131365+0000) 2019-10-13T12:56:38.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:38.150+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:38.152479+0000) 2019-10-13T12:56:38.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:38.666+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:38.672157+0000) 2019-10-13T12:56:41.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:41.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:41.922826+0000) 2019-10-13T12:56:42.125 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:42.118+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.123644+0000) 2019-10-13T12:56:42.126 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:42.118+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.123898+0000) 2019-10-13T12:56:42.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:42.120+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.124483+0000) 2019-10-13T12:56:42.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:42.118+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.124669+0000) 2019-10-13T12:56:42.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:42.122+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.127099+0000) 2019-10-13T12:56:42.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:42.122+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.127601+0000) 2019-10-13T12:56:42.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:42.122+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.127847+0000) 2019-10-13T12:56:42.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:42.128+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:42.128688+0000) 2019-10-13T12:56:42.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:42.126+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171781 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:44.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:44.242+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:44.249515+0000) 2019-10-13T12:56:44.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:44.530+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:44.534042+0000) 2019-10-13T12:56:44.863 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:56:44.863 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:44.867 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:56:44.867 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:56:46.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:46.248+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:46.248673+0000) 2019-10-13T12:56:47.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:47.130+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.133343+0000) 2019-10-13T12:56:47.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:47.130+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.133922+0000) 2019-10-13T12:56:47.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:47.130+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.134462+0000) 2019-10-13T12:56:47.136 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:47.130+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.134694+0000) 2019-10-13T12:56:47.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:47.130+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.136419+0000) 2019-10-13T12:56:47.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:47.136+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.138883+0000) 2019-10-13T12:56:47.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:47.138+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.142752+0000) 2019-10-13T12:56:47.144 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:47.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:47.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:47.148+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.151103+0000) 2019-10-13T12:56:47.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:47.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:47.906362+0000) 2019-10-13T12:56:48.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:48.126+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:48.131576+0000) 2019-10-13T12:56:48.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:48.150+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:48.152676+0000) 2019-10-13T12:56:48.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:48.669+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:48.672351+0000) 2019-10-13T12:56:51.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:51.919+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:51.923023+0000) 2019-10-13T12:56:52.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:52.141+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.146359+0000) 2019-10-13T12:56:52.150 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:52.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.149051+0000) 2019-10-13T12:56:52.151 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:52.147+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.149595+0000) 2019-10-13T12:56:52.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:52.145+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.150096+0000) 2019-10-13T12:56:52.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:52.145+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.151794+0000) 2019-10-13T12:56:52.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:52.145+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.152384+0000) 2019-10-13T12:56:52.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:52.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.152673+0000) 2019-10-13T12:56:52.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:52.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171842 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:52.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:52.155+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:52.159719+0000) 2019-10-13T12:56:54.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:54.241+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:54.249716+0000) 2019-10-13T12:56:54.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:54.529+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:54.534241+0000) 2019-10-13T12:56:56.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:56.251+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:56.255266+0000) 2019-10-13T12:56:57.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:57.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.158328+0000) 2019-10-13T12:56:57.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:57.153+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.158690+0000) 2019-10-13T12:56:57.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:57.153+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.161135+0000) 2019-10-13T12:56:57.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:56:57.157+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.164303+0000) 2019-10-13T12:56:57.168 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:56:57.163+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.166763+0000) 2019-10-13T12:56:57.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:56:57.161+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.166926+0000) 2019-10-13T12:56:57.169 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:56:57.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171875 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:56:57.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:57.161+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.167468+0000) 2019-10-13T12:56:57.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:56:57.167+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.168375+0000) 2019-10-13T12:56:57.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:56:57.905+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:57.906533+0000) 2019-10-13T12:56:58.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:56:58.129+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:58.131755+0000) 2019-10-13T12:56:58.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:56:58.149+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:58.152876+0000) 2019-10-13T12:56:58.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:56:58.665+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:56:58.672545+0000) 2019-10-13T12:57:01.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:01.919+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:01.923231+0000) 2019-10-13T12:57:02.173 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:02.165+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.171373+0000) 2019-10-13T12:57:02.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:02.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.172069+0000) 2019-10-13T12:57:02.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:02.171+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.172711+0000) 2019-10-13T12:57:02.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:02.165+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.172712+0000) 2019-10-13T12:57:02.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:02.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.173056+0000) 2019-10-13T12:57:02.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:02.165+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.173421+0000) 2019-10-13T12:57:02.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:02.169+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.175075+0000) 2019-10-13T12:57:02.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:02.175+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:02.178252+0000) 2019-10-13T12:57:02.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:02.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:04.251 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:04.249+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:04.249967+0000) 2019-10-13T12:57:04.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:04.533+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:04.534462+0000) 2019-10-13T12:57:06.263 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:06.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:06.261924+0000) 2019-10-13T12:57:07.184 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:07.181+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.181975+0000) 2019-10-13T12:57:07.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:07.181+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.184676+0000) 2019-10-13T12:57:07.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:07.181+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.184857+0000) 2019-10-13T12:57:07.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:07.181+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.185175+0000) 2019-10-13T12:57:07.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:07.185+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.185822+0000) 2019-10-13T12:57:07.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:07.185+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.186175+0000) 2019-10-13T12:57:07.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:07.183+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.186826+0000) 2019-10-13T12:57:07.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:07.187+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.190549+0000) 2019-10-13T12:57:07.192 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:07.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171936 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:07.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:07.905+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:07.906713+0000) 2019-10-13T12:57:08.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:08.129+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:08.131969+0000) 2019-10-13T12:57:08.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:08.149+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:08.153020+0000) 2019-10-13T12:57:08.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:08.669+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:08.672751+0000) 2019-10-13T12:57:11.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:11.919+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:11.923414+0000) 2019-10-13T12:57:12.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:12.193+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.194152+0000) 2019-10-13T12:57:12.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:12.193+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.194845+0000) 2019-10-13T12:57:12.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:12.193+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.195398+0000) 2019-10-13T12:57:12.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:12.193+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.195891+0000) 2019-10-13T12:57:12.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:12.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.196590+0000) 2019-10-13T12:57:12.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:12.197+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.197823+0000) 2019-10-13T12:57:12.200 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:12.195+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.199088+0000) 2019-10-13T12:57:12.204 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:12.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:12.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:12.203+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:12.204663+0000) 2019-10-13T12:57:14.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:14.249+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:14.250191+0000) 2019-10-13T12:57:14.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:14.533+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:14.534673+0000) 2019-10-13T12:57:14.988 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:57:14.989 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:14.992 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:57:14.992 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:16.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:16.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:16.268449+0000) 2019-10-13T12:57:17.208 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:17.205+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.206936+0000) 2019-10-13T12:57:17.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:17.205+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.207484+0000) 2019-10-13T12:57:17.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:17.207+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.208322+0000) 2019-10-13T12:57:17.212 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:17.209+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.210790+0000) 2019-10-13T12:57:17.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:17.209+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.212382+0000) 2019-10-13T12:57:17.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:17.211+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.213153+0000) 2019-10-13T12:57:17.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:17.213+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.214895+0000) 2019-10-13T12:57:17.216 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:17.213+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:17.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:17.213+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.215374+0000) 2019-10-13T12:57:17.908 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:17.905+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:17.906888+0000) 2019-10-13T12:57:18.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:18.129+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:18.132170+0000) 2019-10-13T12:57:18.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:18.149+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:18.153199+0000) 2019-10-13T12:57:18.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:18.669+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:18.672946+0000) 2019-10-13T12:57:21.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:21.919+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:21.923589+0000) 2019-10-13T12:57:22.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:22.217+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.218981+0000) 2019-10-13T12:57:22.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:22.217+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.219554+0000) 2019-10-13T12:57:22.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:22.217+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.219732+0000) 2019-10-13T12:57:22.222 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:22.217+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.220352+0000) 2019-10-13T12:57:22.222 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:22.219+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.221219+0000) 2019-10-13T12:57:22.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:22.217+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.221238+0000) 2019-10-13T12:57:22.223 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:22.217+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.221565+0000) 2019-10-13T12:57:22.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:22.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:22.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:22.227+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:22.228834+0000) 2019-10-13T12:57:24.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:24.249+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:24.250420+0000) 2019-10-13T12:57:24.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:24.533+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:24.534872+0000) 2019-10-13T12:57:26.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:26.271+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:26.275032+0000) 2019-10-13T12:57:27.231 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:27.225+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.229601+0000) 2019-10-13T12:57:27.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:27.227+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.230149+0000) 2019-10-13T12:57:27.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:27.229+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.230724+0000) 2019-10-13T12:57:27.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:27.229+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.231503+0000) 2019-10-13T12:57:27.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:27.229+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.233614+0000) 2019-10-13T12:57:27.235 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:27.233+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.234147+0000) 2019-10-13T12:57:27.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:27.233+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.234796+0000) 2019-10-13T12:57:27.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:27.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:27.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:27.239+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.240586+0000) 2019-10-13T12:57:27.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:27.905+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:27.907039+0000) 2019-10-13T12:57:28.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:28.129+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:28.132346+0000) 2019-10-13T12:57:28.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:28.153+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:28.153375+0000) 2019-10-13T12:57:28.675 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:28.669+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:28.673139+0000) 2019-10-13T12:57:31.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:31.923+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:31.923788+0000) 2019-10-13T12:57:32.246 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:32.241+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.243943+0000) 2019-10-13T12:57:32.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:32.241+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.244724+0000) 2019-10-13T12:57:32.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:32.243+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.246712+0000) 2019-10-13T12:57:32.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:32.245+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.246539+0000) 2019-10-13T12:57:32.249 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:32.245+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.248216+0000) 2019-10-13T12:57:32.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:32.247+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.248883+0000) 2019-10-13T12:57:32.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:32.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:32.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:32.249+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.250322+0000) 2019-10-13T12:57:32.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:32.249+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:32.250154+0000) 2019-10-13T12:57:34.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:34.249+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:34.250638+0000) 2019-10-13T12:57:34.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:34.533+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:34.535073+0000) 2019-10-13T12:57:36.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:36.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:36.281429+0000) 2019-10-13T12:57:37.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:37.249+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.253636+0000) 2019-10-13T12:57:37.256 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:37.253+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.254635+0000) 2019-10-13T12:57:37.256 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:37.251+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.255524+0000) 2019-10-13T12:57:37.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:37.253+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.255875+0000) 2019-10-13T12:57:37.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:37.253+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.256043+0000) 2019-10-13T12:57:37.258 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:37.253+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.256348+0000) 2019-10-13T12:57:37.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:37.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.256874+0000) 2019-10-13T12:57:37.263 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:37.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:37.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:37.263+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.265299+0000) 2019-10-13T12:57:37.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:37.905+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:37.907246+0000) 2019-10-13T12:57:38.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:38.129+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:38.132553+0000) 2019-10-13T12:57:38.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:38.149+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:38.153549+0000) 2019-10-13T12:57:38.675 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:38.669+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:38.673334+0000) 2019-10-13T12:57:41.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:41.919+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:41.924060+0000) 2019-10-13T12:57:42.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:42.261+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.264324+0000) 2019-10-13T12:57:42.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:42.261+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.264723+0000) 2019-10-13T12:57:42.267 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:42.263+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.266164+0000) 2019-10-13T12:57:42.269 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:42.265+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.267349+0000) 2019-10-13T12:57:42.269 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:42.265+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.268171+0000) 2019-10-13T12:57:42.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:42.265+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.269031+0000) 2019-10-13T12:57:42.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:42.265+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.269319+0000) 2019-10-13T12:57:42.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:42.269+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:42.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:42.275+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:42.277610+0000) 2019-10-13T12:57:44.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:44.249+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:44.250836+0000) 2019-10-13T12:57:44.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:44.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:44.535295+0000) 2019-10-13T12:57:45.061 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:57:45.061 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:45.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:57:45.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:57:46.289 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:46.283+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:46.287897+0000) 2019-10-13T12:57:47.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:47.272+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.276538+0000) 2019-10-13T12:57:47.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:47.272+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.277270+0000) 2019-10-13T12:57:47.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:47.272+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.277010+0000) 2019-10-13T12:57:47.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:47.275+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.278047+0000) 2019-10-13T12:57:47.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:47.276+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.277803+0000) 2019-10-13T12:57:47.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:47.280+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.282764+0000) 2019-10-13T12:57:47.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:47.280+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.284602+0000) 2019-10-13T12:57:47.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:47.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172184 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:47.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:47.291+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.294778+0000) 2019-10-13T12:57:47.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:47.904+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:47.907427+0000) 2019-10-13T12:57:48.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:48.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:48.132744+0000) 2019-10-13T12:57:48.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:48.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:48.153724+0000) 2019-10-13T12:57:48.675 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:48.668+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:48.673531+0000) 2019-10-13T12:57:51.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:51.923+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:51.924300+0000) 2019-10-13T12:57:52.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:52.284+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.289012+0000) 2019-10-13T12:57:52.291 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:52.284+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.289294+0000) 2019-10-13T12:57:52.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:52.288+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.290179+0000) 2019-10-13T12:57:52.293 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:52.288+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.291331+0000) 2019-10-13T12:57:52.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:52.288+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.291495+0000) 2019-10-13T12:57:52.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:52.291+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.292232+0000) 2019-10-13T12:57:52.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:52.288+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.292417+0000) 2019-10-13T12:57:52.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:52.295+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:52.296845+0000) 2019-10-13T12:57:52.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:52.292+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:54.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:54.248+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:54.251037+0000) 2019-10-13T12:57:54.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:54.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:54.535527+0000) 2019-10-13T12:57:56.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:56.290+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:56.294582+0000) 2019-10-13T12:57:57.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:57:57.296+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.300214+0000) 2019-10-13T12:57:57.302 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:57.296+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.300735+0000) 2019-10-13T12:57:57.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:57:57.300+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.302593+0000) 2019-10-13T12:57:57.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:57:57.298+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.303367+0000) 2019-10-13T12:57:57.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:57.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.304748+0000) 2019-10-13T12:57:57.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:57.304+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.305888+0000) 2019-10-13T12:57:57.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:57.304+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.306191+0000) 2019-10-13T12:57:57.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:57:57.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:57:57.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:57:57.306+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.311658+0000) 2019-10-13T12:57:57.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:57:57.904+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:57.907597+0000) 2019-10-13T12:57:58.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:57:58.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:58.132917+0000) 2019-10-13T12:57:58.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:57:58.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:58.153960+0000) 2019-10-13T12:57:58.675 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:57:58.672+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:57:58.673724+0000) 2019-10-13T12:58:01.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:01.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:01.924483+0000) 2019-10-13T12:58:02.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:02.310+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.314421+0000) 2019-10-13T12:58:02.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:02.312+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.314656+0000) 2019-10-13T12:58:02.318 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:02.312+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.316934+0000) 2019-10-13T12:58:02.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:02.312+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.317245+0000) 2019-10-13T12:58:02.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:02.312+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.317677+0000) 2019-10-13T12:58:02.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:02.314+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.318656+0000) 2019-10-13T12:58:02.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:02.316+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.319528+0000) 2019-10-13T12:58:02.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:02.316+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:02.319824+0000) 2019-10-13T12:58:02.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:02.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:04.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:04.248+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:04.251234+0000) 2019-10-13T12:58:04.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:04.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:04.535754+0000) 2019-10-13T12:58:06.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:06.298+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:06.301141+0000) 2019-10-13T12:58:07.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:07.320+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.325566+0000) 2019-10-13T12:58:07.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:07.324+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.326060+0000) 2019-10-13T12:58:07.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:07.324+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.326289+0000) 2019-10-13T12:58:07.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:07.322+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.326937+0000) 2019-10-13T12:58:07.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:07.324+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.327103+0000) 2019-10-13T12:58:07.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:07.324+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.327491+0000) 2019-10-13T12:58:07.330 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:07.324+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.327805+0000) 2019-10-13T12:58:07.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:07.326+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.331707+0000) 2019-10-13T12:58:07.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:07.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:07.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:07.904+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:07.907811+0000) 2019-10-13T12:58:08.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:08.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:08.133118+0000) 2019-10-13T12:58:08.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:08.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:08.154136+0000) 2019-10-13T12:58:08.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:08.672+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:08.673957+0000) 2019-10-13T12:58:11.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:11.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:11.924658+0000) 2019-10-13T12:58:12.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:12.332+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.337680+0000) 2019-10-13T12:58:12.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:12.332+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.337377+0000) 2019-10-13T12:58:12.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:12.334+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.338705+0000) 2019-10-13T12:58:12.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:12.336+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.339103+0000) 2019-10-13T12:58:12.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:12.336+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.340116+0000) 2019-10-13T12:58:12.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:12.336+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.340558+0000) 2019-10-13T12:58:12.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:12.336+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.340809+0000) 2019-10-13T12:58:12.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:12.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:12.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:12.350+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:12.352446+0000) 2019-10-13T12:58:14.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:14.248+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:14.251433+0000) 2019-10-13T12:58:14.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:14.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:14.535979+0000) 2019-10-13T12:58:15.120 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:58:15.120 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:15.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:58:15.125 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:16.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:16.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:16.312314+0000) 2019-10-13T12:58:17.352 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:17.348+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.350568+0000) 2019-10-13T12:58:17.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:17.348+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.351205+0000) 2019-10-13T12:58:17.353 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:17.348+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.351387+0000) 2019-10-13T12:58:17.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:17.348+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.352011+0000) 2019-10-13T12:58:17.354 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:17.348+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.352353+0000) 2019-10-13T12:58:17.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:17.348+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.353420+0000) 2019-10-13T12:58:17.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:17.350+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.354216+0000) 2019-10-13T12:58:17.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:17.354+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.359304+0000) 2019-10-13T12:58:17.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:17.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:17.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:17.904+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:17.908017+0000) 2019-10-13T12:58:18.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:18.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:18.133315+0000) 2019-10-13T12:58:18.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:18.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:18.154306+0000) 2019-10-13T12:58:18.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:18.672+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:18.674153+0000) 2019-10-13T12:58:21.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:21.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:21.924857+0000) 2019-10-13T12:58:22.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:22.360+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.362888+0000) 2019-10-13T12:58:22.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:22.360+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.364756+0000) 2019-10-13T12:58:22.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:22.360+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.365429+0000) 2019-10-13T12:58:22.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:22.364+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.369354+0000) 2019-10-13T12:58:22.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:22.366+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.371661+0000) 2019-10-13T12:58:22.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:22.368+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.371726+0000) 2019-10-13T12:58:22.374 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:22.368+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.372743+0000) 2019-10-13T12:58:22.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:22.368+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:22.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:22.370+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:22.374696+0000) 2019-10-13T12:58:24.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:24.248+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:24.251585+0000) 2019-10-13T12:58:24.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:24.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:24.536177+0000) 2019-10-13T12:58:26.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:26.318+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:26.318913+0000) 2019-10-13T12:58:27.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:27.376+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.378865+0000) 2019-10-13T12:58:27.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:27.376+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.379230+0000) 2019-10-13T12:58:27.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:27.376+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.379819+0000) 2019-10-13T12:58:27.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:27.376+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.380696+0000) 2019-10-13T12:58:27.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:27.378+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.381800+0000) 2019-10-13T12:58:27.383 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:27.380+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.382021+0000) 2019-10-13T12:58:27.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:27.380+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.382332+0000) 2019-10-13T12:58:27.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:27.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:27.389 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:27.382+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.387874+0000) 2019-10-13T12:58:27.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:27.904+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:27.908208+0000) 2019-10-13T12:58:28.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:28.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:28.133514+0000) 2019-10-13T12:58:28.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:28.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:28.154500+0000) 2019-10-13T12:58:28.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:28.672+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:28.674377+0000) 2019-10-13T12:58:31.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:31.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:31.925057+0000) 2019-10-13T12:58:32.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:32.384+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.389300+0000) 2019-10-13T12:58:32.392 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:32.388+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.389842+0000) 2019-10-13T12:58:32.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:32.388+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.390289+0000) 2019-10-13T12:58:32.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:32.388+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.390616+0000) 2019-10-13T12:58:32.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:32.386+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.391153+0000) 2019-10-13T12:58:32.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:32.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.391827+0000) 2019-10-13T12:58:32.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:32.388+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.392056+0000) 2019-10-13T12:58:32.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:32.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172463 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:32.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:32.398+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:32.400335+0000) 2019-10-13T12:58:34.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:34.248+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:34.251777+0000) 2019-10-13T12:58:34.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:34.532+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:34.536375+0000) 2019-10-13T12:58:36.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:36.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:36.325494+0000) 2019-10-13T12:58:37.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:37.396+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.400319+0000) 2019-10-13T12:58:37.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:37.396+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.400671+0000) 2019-10-13T12:58:37.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:37.398+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.401763+0000) 2019-10-13T12:58:37.404 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:37.396+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.401571+0000) 2019-10-13T12:58:37.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:37.400+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.401941+0000) 2019-10-13T12:58:37.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:37.400+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.402423+0000) 2019-10-13T12:58:37.405 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:37.400+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.403316+0000) 2019-10-13T12:58:37.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:37.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:37.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:37.406+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.410298+0000) 2019-10-13T12:58:37.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:37.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:37.908406+0000) 2019-10-13T12:58:38.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:38.128+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:38.133679+0000) 2019-10-13T12:58:38.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:38.152+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:38.154670+0000) 2019-10-13T12:58:38.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:38.672+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:38.674604+0000) 2019-10-13T12:58:41.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:41.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:41.925250+0000) 2019-10-13T12:58:42.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:42.407+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.413126+0000) 2019-10-13T12:58:42.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:42.407+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.413530+0000) 2019-10-13T12:58:42.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:42.411+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.414455+0000) 2019-10-13T12:58:42.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:42.410+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.415298+0000) 2019-10-13T12:58:42.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:42.411+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.416014+0000) 2019-10-13T12:58:42.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:42.411+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.416330+0000) 2019-10-13T12:58:42.418 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:42.411+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.415564+0000) 2019-10-13T12:58:42.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:42.418+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:42.421830+0000) 2019-10-13T12:58:42.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:42.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:44.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:44.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:44.251979+0000) 2019-10-13T12:58:44.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:44.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:44.536590+0000) 2019-10-13T12:58:45.229 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:58:45.229 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:45.233 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:58:45.233 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:58:46.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:46.330+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:46.332075+0000) 2019-10-13T12:58:47.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:47.419+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.425099+0000) 2019-10-13T12:58:47.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:47.419+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.425713+0000) 2019-10-13T12:58:47.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:47.423+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.426272+0000) 2019-10-13T12:58:47.428 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:47.423+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.427010+0000) 2019-10-13T12:58:47.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:47.423+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.427716+0000) 2019-10-13T12:58:47.429 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:47.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.428297+0000) 2019-10-13T12:58:47.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:47.423+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.428341+0000) 2019-10-13T12:58:47.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:47.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:47.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:47.434+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.436081+0000) 2019-10-13T12:58:47.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:47.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:47.908603+0000) 2019-10-13T12:58:48.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:48.131+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:48.133859+0000) 2019-10-13T12:58:48.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:48.151+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:48.154806+0000) 2019-10-13T12:58:48.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:48.671+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:48.674794+0000) 2019-10-13T12:58:51.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:51.922+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:51.925449+0000) 2019-10-13T12:58:52.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:52.431+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.437260+0000) 2019-10-13T12:58:52.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:52.435+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.438040+0000) 2019-10-13T12:58:52.440 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:52.435+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.438453+0000) 2019-10-13T12:58:52.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:52.435+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.438638+0000) 2019-10-13T12:58:52.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:52.435+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.439239+0000) 2019-10-13T12:58:52.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:52.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.439711+0000) 2019-10-13T12:58:52.441 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:52.438+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.440361+0000) 2019-10-13T12:58:52.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:52.442+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:52.444270+0000) 2019-10-13T12:58:52.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:52.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:54.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:54.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:54.252173+0000) 2019-10-13T12:58:54.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:54.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:54.536789+0000) 2019-10-13T12:58:56.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:56.338+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:56.338758+0000) 2019-10-13T12:58:57.450 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:57.443+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.448635+0000) 2019-10-13T12:58:57.451 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:58:57.447+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.450167+0000) 2019-10-13T12:58:57.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:57.447+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.450613+0000) 2019-10-13T12:58:57.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:57.447+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.451369+0000) 2019-10-13T12:58:57.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:57.447+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.451774+0000) 2019-10-13T12:58:57.454 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:58:57.447+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.451773+0000) 2019-10-13T12:58:57.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:58:57.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.452841+0000) 2019-10-13T12:58:57.459 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:58:57.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172617 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:58:57.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:58:57.454+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.459203+0000) 2019-10-13T12:58:57.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:58:57.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:57.908801+0000) 2019-10-13T12:58:58.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:58:58.131+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:58.134030+0000) 2019-10-13T12:58:58.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:58:58.151+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:58.155001+0000) 2019-10-13T12:58:58.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:58:58.671+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:58:58.675007+0000) 2019-10-13T12:59:01.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:01.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:01.925673+0000) 2019-10-13T12:59:02.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:02.455+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.460619+0000) 2019-10-13T12:59:02.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:02.455+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.461573+0000) 2019-10-13T12:59:02.464 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:02.459+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.462005+0000) 2019-10-13T12:59:02.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:02.459+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.462749+0000) 2019-10-13T12:59:02.465 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:02.459+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.463362+0000) 2019-10-13T12:59:02.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:02.459+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.463515+0000) 2019-10-13T12:59:02.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:02.461+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.464147+0000) 2019-10-13T12:59:02.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:02.465+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:02.470911+0000) 2019-10-13T12:59:02.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:02.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:04.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:04.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:04.252342+0000) 2019-10-13T12:59:04.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:04.531+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:04.536943+0000) 2019-10-13T12:59:06.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:06.341+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:06.345243+0000) 2019-10-13T12:59:07.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:07.469+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.473968+0000) 2019-10-13T12:59:07.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:07.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.474608+0000) 2019-10-13T12:59:07.478 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:07.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.476119+0000) 2019-10-13T12:59:07.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:07.471+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.476398+0000) 2019-10-13T12:59:07.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:07.471+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.476783+0000) 2019-10-13T12:59:07.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:07.471+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.477149+0000) 2019-10-13T12:59:07.479 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:07.471+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.477667+0000) 2019-10-13T12:59:07.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:07.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172676 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:07.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:07.477+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.483953+0000) 2019-10-13T12:59:07.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:07.903+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:07.908988+0000) 2019-10-13T12:59:08.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:08.131+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:08.134231+0000) 2019-10-13T12:59:08.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:08.151+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:08.155180+0000) 2019-10-13T12:59:08.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:08.671+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:08.675243+0000) 2019-10-13T12:59:11.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:11.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:11.925896+0000) 2019-10-13T12:59:12.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:12.483+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.487237+0000) 2019-10-13T12:59:12.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:12.483+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.487547+0000) 2019-10-13T12:59:12.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:12.481+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.487949+0000) 2019-10-13T12:59:12.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:12.483+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.488027+0000) 2019-10-13T12:59:12.490 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:12.483+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.488539+0000) 2019-10-13T12:59:12.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:12.483+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.488921+0000) 2019-10-13T12:59:12.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:12.483+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.489570+0000) 2019-10-13T12:59:12.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:12.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:12.493175+0000) 2019-10-13T12:59:12.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:12.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:14.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:14.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:14.252560+0000) 2019-10-13T12:59:14.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:14.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:14.537134+0000) 2019-10-13T12:59:15.351 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:59:15.351 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:15.355 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:59:15.355 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:16.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:16.349+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:16.351933+0000) 2019-10-13T12:59:17.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:17.491+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.497129+0000) 2019-10-13T12:59:17.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:17.491+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.497400+0000) 2019-10-13T12:59:17.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:17.495+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.498192+0000) 2019-10-13T12:59:17.500 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:17.493+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.498655+0000) 2019-10-13T12:59:17.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:17.495+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.499094+0000) 2019-10-13T12:59:17.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:17.495+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.499518+0000) 2019-10-13T12:59:17.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:17.495+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.500273+0000) 2019-10-13T12:59:17.508 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:17.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:17.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:17.501+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.508052+0000) 2019-10-13T12:59:17.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:17.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:17.909202+0000) 2019-10-13T12:59:18.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:18.131+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:18.134410+0000) 2019-10-13T12:59:18.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:18.155+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:18.155393+0000) 2019-10-13T12:59:18.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:18.671+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:18.675473+0000) 2019-10-13T12:59:21.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:21.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:21.926111+0000) 2019-10-13T12:59:22.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:22.507+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.509998+0000) 2019-10-13T12:59:22.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:22.507+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.510962+0000) 2019-10-13T12:59:22.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:22.507+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.511301+0000) 2019-10-13T12:59:22.513 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:22.507+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.511825+0000) 2019-10-13T12:59:22.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:22.507+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.512417+0000) 2019-10-13T12:59:22.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:22.509+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.512938+0000) 2019-10-13T12:59:22.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:22.507+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.513633+0000) 2019-10-13T12:59:22.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:22.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:22.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:22.513+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:22.519844+0000) 2019-10-13T12:59:24.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:24.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:24.252779+0000) 2019-10-13T12:59:24.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:24.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:24.537363+0000) 2019-10-13T12:59:26.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:26.357+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:26.358549+0000) 2019-10-13T12:59:27.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:27.519+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.521728+0000) 2019-10-13T12:59:27.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:27.519+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.522422+0000) 2019-10-13T12:59:27.525 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:27.517+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.523660+0000) 2019-10-13T12:59:27.526 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:27.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.524069+0000) 2019-10-13T12:59:27.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:27.519+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.524425+0000) 2019-10-13T12:59:27.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:27.519+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.524737+0000) 2019-10-13T12:59:27.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:27.519+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.524779+0000) 2019-10-13T12:59:27.532 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:27.527+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:27.533 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:27.525+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.531994+0000) 2019-10-13T12:59:27.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:27.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:27.909384+0000) 2019-10-13T12:59:28.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:28.131+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:28.134594+0000) 2019-10-13T12:59:28.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:28.151+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:28.155553+0000) 2019-10-13T12:59:28.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:28.675+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:28.675668+0000) 2019-10-13T12:59:31.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:31.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:31.926312+0000) 2019-10-13T12:59:32.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:32.531+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.533977+0000) 2019-10-13T12:59:32.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:32.529+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.534918+0000) 2019-10-13T12:59:32.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:32.531+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.534933+0000) 2019-10-13T12:59:32.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:32.531+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.535464+0000) 2019-10-13T12:59:32.538 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:32.531+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.536871+0000) 2019-10-13T12:59:32.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:32.535+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.539146+0000) 2019-10-13T12:59:32.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:32.539+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.542526+0000) 2019-10-13T12:59:32.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:32.539+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:32.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:32.537+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:32.543966+0000) 2019-10-13T12:59:34.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:34.247+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:34.252973+0000) 2019-10-13T12:59:34.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:34.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:34.537563+0000) 2019-10-13T12:59:36.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:36.361+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:36.365137+0000) 2019-10-13T12:59:37.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:37.542+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.546733+0000) 2019-10-13T12:59:37.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:37.542+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.547135+0000) 2019-10-13T12:59:37.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:37.542+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.547551+0000) 2019-10-13T12:59:37.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:37.542+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.549508+0000) 2019-10-13T12:59:37.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:37.542+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.549536+0000) 2019-10-13T12:59:37.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:37.545+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.551647+0000) 2019-10-13T12:59:37.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:37.549+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.553126+0000) 2019-10-13T12:59:37.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:37.546+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.553348+0000) 2019-10-13T12:59:37.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:37.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:37.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:37.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:37.909594+0000) 2019-10-13T12:59:38.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:38.130+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:38.134795+0000) 2019-10-13T12:59:38.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:38.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:38.155731+0000) 2019-10-13T12:59:38.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:38.670+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:38.675879+0000) 2019-10-13T12:59:41.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:41.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:41.926460+0000) 2019-10-13T12:59:42.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:42.554+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.558235+0000) 2019-10-13T12:59:42.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:42.554+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.559213+0000) 2019-10-13T12:59:42.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:42.554+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.560993+0000) 2019-10-13T12:59:42.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:42.554+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.561654+0000) 2019-10-13T12:59:42.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:42.558+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.563764+0000) 2019-10-13T12:59:42.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:42.561+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.566072+0000) 2019-10-13T12:59:42.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:42.562+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.565968+0000) 2019-10-13T12:59:42.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:42.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:42.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:42.565+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:42.570485+0000) 2019-10-13T12:59:44.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:44.246+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:44.253193+0000) 2019-10-13T12:59:44.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:44.534+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:44.537778+0000) 2019-10-13T12:59:45.465 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T12:59:45.465 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:45.492 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T12:59:45.493 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T12:59:46.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:46.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:46.371713+0000) 2019-10-13T12:59:47.571 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:47.562+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.569630+0000) 2019-10-13T12:59:47.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:47.566+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.570574+0000) 2019-10-13T12:59:47.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:47.566+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.572252+0000) 2019-10-13T12:59:47.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:47.569+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.572810+0000) 2019-10-13T12:59:47.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:47.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.572959+0000) 2019-10-13T12:59:47.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:47.566+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.573354+0000) 2019-10-13T12:59:47.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:47.570+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.574103+0000) 2019-10-13T12:59:47.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:47.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:47.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:47.577+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.580166+0000) 2019-10-13T12:59:47.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:47.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:47.909771+0000) 2019-10-13T12:59:48.136 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:48.130+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:48.134985+0000) 2019-10-13T12:59:48.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:48.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:48.155871+0000) 2019-10-13T12:59:48.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:48.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:48.676094+0000) 2019-10-13T12:59:51.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:51.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:51.926675+0000) 2019-10-13T12:59:52.582 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:52.574+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.580396+0000) 2019-10-13T12:59:52.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:52.578+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.581874+0000) 2019-10-13T12:59:52.584 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:52.578+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.582412+0000) 2019-10-13T12:59:52.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:52.578+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.582916+0000) 2019-10-13T12:59:52.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:52.578+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.583968+0000) 2019-10-13T12:59:52.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:52.581+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.584439+0000) 2019-10-13T12:59:52.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:52.578+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.584622+0000) 2019-10-13T12:59:52.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:52.582+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172945 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:52.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:52.585+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:52.591302+0000) 2019-10-13T12:59:54.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:54.250+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:54.253390+0000) 2019-10-13T12:59:54.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:54.534+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:54.537981+0000) 2019-10-13T12:59:56.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:56.373+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:56.378246+0000) 2019-10-13T12:59:57.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T12:59:57.586+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.592707+0000) 2019-10-13T12:59:57.595 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T12:59:57.586+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.593315+0000) 2019-10-13T12:59:57.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:57.590+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.593969+0000) 2019-10-13T12:59:57.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:57.590+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.594847+0000) 2019-10-13T12:59:57.597 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:57.590+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.595131+0000) 2019-10-13T12:59:57.597 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:57.590+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.595968+0000) 2019-10-13T12:59:57.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T12:59:57.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.596720+0000) 2019-10-13T12:59:57.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T12:59:57.594+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T12:59:57.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T12:59:57.597+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.602354+0000) 2019-10-13T12:59:57.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T12:59:57.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:57.909942+0000) 2019-10-13T12:59:58.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T12:59:58.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:58.135163+0000) 2019-10-13T12:59:58.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T12:59:58.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:58.156096+0000) 2019-10-13T12:59:58.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T12:59:58.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T11:59:58.676283+0000) 2019-10-13T13:00:01.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:01.921+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:01.926826+0000) 2019-10-13T13:00:02.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:02.598+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.604586+0000) 2019-10-13T13:00:02.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:02.598+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.605353+0000) 2019-10-13T13:00:02.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:02.598+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.605035+0000) 2019-10-13T13:00:02.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:02.602+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.606052+0000) 2019-10-13T13:00:02.608 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:02.602+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.606719+0000) 2019-10-13T13:00:02.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:02.601+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.607820+0000) 2019-10-13T13:00:02.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:02.602+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.607833+0000) 2019-10-13T13:00:02.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:02.609+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:02.613775+0000) 2019-10-13T13:00:02.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:02.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:04.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:04.250+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:04.253559+0000) 2019-10-13T13:00:04.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:04.534+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:04.538155+0000) 2019-10-13T13:00:06.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:06.380+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:06.384925+0000) 2019-10-13T13:00:07.619 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:07.610+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.617242+0000) 2019-10-13T13:00:07.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:07.614+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.617826+0000) 2019-10-13T13:00:07.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:07.614+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.618452+0000) 2019-10-13T13:00:07.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:07.614+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.619235+0000) 2019-10-13T13:00:07.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:07.614+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.619736+0000) 2019-10-13T13:00:07.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:07.616+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.620505+0000) 2019-10-13T13:00:07.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:07.614+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.619600+0000) 2019-10-13T13:00:07.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:07.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:07.629 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:07.620+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.627659+0000) 2019-10-13T13:00:07.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:07.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:07.910106+0000) 2019-10-13T13:00:08.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:08.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:08.135371+0000) 2019-10-13T13:00:08.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:08.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:08.156285+0000) 2019-10-13T13:00:08.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:08.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:08.676475+0000) 2019-10-13T13:00:11.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:11.920+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:11.926972+0000) 2019-10-13T13:00:12.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:12.622+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.628632+0000) 2019-10-13T13:00:12.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:12.622+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.628730+0000) 2019-10-13T13:00:12.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:12.622+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.629578+0000) 2019-10-13T13:00:12.632 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:12.626+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.630827+0000) 2019-10-13T13:00:12.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:12.626+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.631266+0000) 2019-10-13T13:00:12.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:12.626+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.631553+0000) 2019-10-13T13:00:12.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:12.628+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.632317+0000) 2019-10-13T13:00:12.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:12.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:12.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:12.632+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:12.638036+0000) 2019-10-13T13:00:14.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:14.250+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:14.253804+0000) 2019-10-13T13:00:14.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:14.538+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:14.538353+0000) 2019-10-13T13:00:15.627 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:00:15.627 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:15.630 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:00:15.630 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:16.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:16.388+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:16.391476+0000) 2019-10-13T13:00:17.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:17.638+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.642120+0000) 2019-10-13T13:00:17.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:17.638+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.642623+0000) 2019-10-13T13:00:17.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:17.638+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.643092+0000) 2019-10-13T13:00:17.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:17.640+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.645169+0000) 2019-10-13T13:00:17.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:17.638+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.645299+0000) 2019-10-13T13:00:17.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:17.638+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.645697+0000) 2019-10-13T13:00:17.648 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:17.642+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.646205+0000) 2019-10-13T13:00:17.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:17.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:17.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:17.648+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.655607+0000) 2019-10-13T13:00:17.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:17.910+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:17.910297+0000) 2019-10-13T13:00:18.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:18.130+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:18.135550+0000) 2019-10-13T13:00:18.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:18.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:18.156461+0000) 2019-10-13T13:00:18.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:18.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:18.676674+0000) 2019-10-13T13:00:21.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:21.924+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:21.927238+0000) 2019-10-13T13:00:22.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:22.646+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.652865+0000) 2019-10-13T13:00:22.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:22.648+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.653894+0000) 2019-10-13T13:00:22.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:22.650+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.655485+0000) 2019-10-13T13:00:22.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:22.650+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.655897+0000) 2019-10-13T13:00:22.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:22.650+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.656563+0000) 2019-10-13T13:00:22.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:22.650+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.656682+0000) 2019-10-13T13:00:22.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:22.650+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.657481+0000) 2019-10-13T13:00:22.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:22.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:22.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:22.660+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:22.666040+0000) 2019-10-13T13:00:24.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:24.250+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:24.254012+0000) 2019-10-13T13:00:24.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:24.538+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:24.538583+0000) 2019-10-13T13:00:26.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:26.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:26.397991+0000) 2019-10-13T13:00:27.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:27.662+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.666309+0000) 2019-10-13T13:00:27.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:27.662+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.666693+0000) 2019-10-13T13:00:27.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:27.664+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.668572+0000) 2019-10-13T13:00:27.671 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:27.666+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.670123+0000) 2019-10-13T13:00:27.672 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:27.666+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.670525+0000) 2019-10-13T13:00:27.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:27.666+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.671867+0000) 2019-10-13T13:00:27.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:27.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:27.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:27.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.674454+0000) 2019-10-13T13:00:27.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:27.680+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.685558+0000) 2019-10-13T13:00:27.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:27.906+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:27.910502+0000) 2019-10-13T13:00:28.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:28.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:28.135746+0000) 2019-10-13T13:00:28.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:28.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:28.156638+0000) 2019-10-13T13:00:28.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:28.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:28.676867+0000) 2019-10-13T13:00:31.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:31.924+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:31.927429+0000) 2019-10-13T13:00:32.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:32.672+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.678035+0000) 2019-10-13T13:00:32.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:32.673+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.681233+0000) 2019-10-13T13:00:32.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:32.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.683140+0000) 2019-10-13T13:00:32.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:32.677+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.684914+0000) 2019-10-13T13:00:32.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:32.680+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.685256+0000) 2019-10-13T13:00:32.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:32.677+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173184 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:32.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:32.681+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.686294+0000) 2019-10-13T13:00:32.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:32.681+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.686722+0000) 2019-10-13T13:00:32.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:32.681+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:32.686881+0000) 2019-10-13T13:00:34.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:34.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:34.254224+0000) 2019-10-13T13:00:34.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:34.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:34.538783+0000) 2019-10-13T13:00:36.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:36.404+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:36.404633+0000) 2019-10-13T13:00:37.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:37.681+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.688124+0000) 2019-10-13T13:00:37.690 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:37.681+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.688539+0000) 2019-10-13T13:00:37.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:37.684+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.689679+0000) 2019-10-13T13:00:37.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:37.685+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.690052+0000) 2019-10-13T13:00:37.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:37.685+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.690611+0000) 2019-10-13T13:00:37.692 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:37.685+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.690708+0000) 2019-10-13T13:00:37.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:37.685+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.691212+0000) 2019-10-13T13:00:37.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:37.692+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.696737+0000) 2019-10-13T13:00:37.699 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:37.693+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:37.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:37.909+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:37.910708+0000) 2019-10-13T13:00:38.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:38.133+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:38.135942+0000) 2019-10-13T13:00:38.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:38.153+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:38.156817+0000) 2019-10-13T13:00:38.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:38.673+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:38.677057+0000) 2019-10-13T13:00:41.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:41.924+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:41.927649+0000) 2019-10-13T13:00:42.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:42.696+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.701689+0000) 2019-10-13T13:00:42.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:42.697+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.701910+0000) 2019-10-13T13:00:42.704 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:42.697+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.702746+0000) 2019-10-13T13:00:42.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:42.697+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.703231+0000) 2019-10-13T13:00:42.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:42.697+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.704653+0000) 2019-10-13T13:00:42.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:42.701+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.706463+0000) 2019-10-13T13:00:42.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:42.701+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.708547+0000) 2019-10-13T13:00:42.712 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:42.705+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:42.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:42.712+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:42.717260+0000) 2019-10-13T13:00:44.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:44.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:44.254456+0000) 2019-10-13T13:00:44.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:44.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:44.538976+0000) 2019-10-13T13:00:45.754 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:00:45.755 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:45.758 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:00:45.759 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:00:46.412 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:46.408+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:46.411184+0000) 2019-10-13T13:00:47.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:47.713+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.714027+0000) 2019-10-13T13:00:47.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:47.713+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.714530+0000) 2019-10-13T13:00:47.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:47.708+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.715172+0000) 2019-10-13T13:00:47.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:47.713+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.715317+0000) 2019-10-13T13:00:47.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:47.713+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.715714+0000) 2019-10-13T13:00:47.718 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:47.713+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.716732+0000) 2019-10-13T13:00:47.719 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:47.713+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.717272+0000) 2019-10-13T13:00:47.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:47.716+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.722863+0000) 2019-10-13T13:00:47.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:47.721+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:47.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:47.909+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:47.910906+0000) 2019-10-13T13:00:48.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:48.133+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:48.136118+0000) 2019-10-13T13:00:48.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:48.153+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:48.156988+0000) 2019-10-13T13:00:48.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:48.673+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:48.677226+0000) 2019-10-13T13:00:51.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:51.924+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:51.927863+0000) 2019-10-13T13:00:52.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:52.725+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.726244+0000) 2019-10-13T13:00:52.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:52.725+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.726542+0000) 2019-10-13T13:00:52.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:52.725+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.726883+0000) 2019-10-13T13:00:52.729 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:52.725+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.727570+0000) 2019-10-13T13:00:52.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:52.724+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.728237+0000) 2019-10-13T13:00:52.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:52.725+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.728507+0000) 2019-10-13T13:00:52.731 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:52.725+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.729179+0000) 2019-10-13T13:00:52.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:52.728+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:52.735539+0000) 2019-10-13T13:00:52.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:52.733+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:54.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:54.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:54.254688+0000) 2019-10-13T13:00:54.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:54.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:54.539188+0000) 2019-10-13T13:00:56.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:56.416+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:56.417730+0000) 2019-10-13T13:00:57.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:00:57.737+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.738537+0000) 2019-10-13T13:00:57.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:57.737+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.739197+0000) 2019-10-13T13:00:57.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:00:57.737+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.739756+0000) 2019-10-13T13:00:57.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:57.737+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.740237+0000) 2019-10-13T13:00:57.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:00:57.736+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.740827+0000) 2019-10-13T13:00:57.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:57.737+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.741095+0000) 2019-10-13T13:00:57.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:57.741+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.744655+0000) 2019-10-13T13:00:57.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:00:57.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:00:57.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:00:57.744+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.748696+0000) 2019-10-13T13:00:57.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:00:57.909+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:57.911086+0000) 2019-10-13T13:00:58.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:00:58.133+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:58.136312+0000) 2019-10-13T13:00:58.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:00:58.153+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:58.157169+0000) 2019-10-13T13:00:58.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:00:58.673+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:00:58.677408+0000) 2019-10-13T13:01:01.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:01.924+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:01.928056+0000) 2019-10-13T13:01:02.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:02.748+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.750824+0000) 2019-10-13T13:01:02.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:02.749+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.751886+0000) 2019-10-13T13:01:02.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:02.749+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.752135+0000) 2019-10-13T13:01:02.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:02.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.752372+0000) 2019-10-13T13:01:02.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:02.749+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.753302+0000) 2019-10-13T13:01:02.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:02.753+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.754241+0000) 2019-10-13T13:01:02.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:02.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.754749+0000) 2019-10-13T13:01:02.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:02.756+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:02.758019+0000) 2019-10-13T13:01:02.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:02.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:04.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:04.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:04.254888+0000) 2019-10-13T13:01:04.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:04.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:04.539417+0000) 2019-10-13T13:01:06.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:06.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:06.424422+0000) 2019-10-13T13:01:07.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:07.761+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.763871+0000) 2019-10-13T13:01:07.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:07.761+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.763933+0000) 2019-10-13T13:01:07.767 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:07.761+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.765292+0000) 2019-10-13T13:01:07.767 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:07.765+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.766006+0000) 2019-10-13T13:01:07.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:07.765+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.767010+0000) 2019-10-13T13:01:07.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:07.765+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.767456+0000) 2019-10-13T13:01:07.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:07.767+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.768306+0000) 2019-10-13T13:01:07.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:07.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173395 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:07.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:07.771+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.774205+0000) 2019-10-13T13:01:07.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:07.909+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:07.911285+0000) 2019-10-13T13:01:08.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:08.133+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:08.136492+0000) 2019-10-13T13:01:08.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:08.157+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:08.157379+0000) 2019-10-13T13:01:08.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:08.673+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:08.677581+0000) 2019-10-13T13:01:11.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:11.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:11.928310+0000) 2019-10-13T13:01:12.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:12.773+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.774317+0000) 2019-10-13T13:01:12.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:12.773+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.776316+0000) 2019-10-13T13:01:12.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:12.775+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.777079+0000) 2019-10-13T13:01:12.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:12.777+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.778040+0000) 2019-10-13T13:01:12.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:12.777+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.778141+0000) 2019-10-13T13:01:12.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:12.777+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.779092+0000) 2019-10-13T13:01:12.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:12.777+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.779168+0000) 2019-10-13T13:01:12.785 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:12.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173424 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:12.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:12.783+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:12.786596+0000) 2019-10-13T13:01:14.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:14.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:14.255109+0000) 2019-10-13T13:01:14.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:14.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:14.539619+0000) 2019-10-13T13:01:15.872 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:01:15.872 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:15.876 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:01:15.876 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:16.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:16.427+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:16.430991+0000) 2019-10-13T13:01:17.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:17.785+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.787834+0000) 2019-10-13T13:01:17.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:17.785+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.788286+0000) 2019-10-13T13:01:17.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:17.785+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.789002+0000) 2019-10-13T13:01:17.792 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:17.789+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.790647+0000) 2019-10-13T13:01:17.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:17.791+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.793071+0000) 2019-10-13T13:01:17.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:17.793+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.793883+0000) 2019-10-13T13:01:17.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:17.791+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.794456+0000) 2019-10-13T13:01:17.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:17.793+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.794524+0000) 2019-10-13T13:01:17.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:17.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173456 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:17.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:17.909+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:17.911468+0000) 2019-10-13T13:01:18.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:18.133+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:18.136695+0000) 2019-10-13T13:01:18.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:18.157+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:18.157557+0000) 2019-10-13T13:01:18.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:18.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:18.677783+0000) 2019-10-13T13:01:21.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:21.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:21.928492+0000) 2019-10-13T13:01:22.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:22.797+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.798803+0000) 2019-10-13T13:01:22.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:22.797+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.800494+0000) 2019-10-13T13:01:22.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:22.797+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.801005+0000) 2019-10-13T13:01:22.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:22.801+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.801844+0000) 2019-10-13T13:01:22.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:22.801+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.803450+0000) 2019-10-13T13:01:22.807 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:22.801+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.805382+0000) 2019-10-13T13:01:22.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:22.803+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.806653+0000) 2019-10-13T13:01:22.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:22.805+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:22.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:22.807+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:22.811948+0000) 2019-10-13T13:01:24.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:24.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:24.255346+0000) 2019-10-13T13:01:24.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:24.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:24.539822+0000) 2019-10-13T13:01:26.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:26.435+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:26.437735+0000) 2019-10-13T13:01:27.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:27.808+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.811199+0000) 2019-10-13T13:01:27.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:27.808+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.812067+0000) 2019-10-13T13:01:27.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:27.808+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.812414+0000) 2019-10-13T13:01:27.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:27.812+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.814001+0000) 2019-10-13T13:01:27.816 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:27.812+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.813825+0000) 2019-10-13T13:01:27.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:27.811+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.814968+0000) 2019-10-13T13:01:27.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:27.812+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.815045+0000) 2019-10-13T13:01:27.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:27.816+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:27.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:27.819+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.821357+0000) 2019-10-13T13:01:27.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:27.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:27.911645+0000) 2019-10-13T13:01:28.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:28.132+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:28.136826+0000) 2019-10-13T13:01:28.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:28.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:28.157733+0000) 2019-10-13T13:01:28.679 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:28.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:28.677953+0000) 2019-10-13T13:01:31.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:31.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:31.928663+0000) 2019-10-13T13:01:32.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:32.823+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.824585+0000) 2019-10-13T13:01:32.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:32.820+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.824229+0000) 2019-10-13T13:01:32.827 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:32.820+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.825124+0000) 2019-10-13T13:01:32.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:32.820+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.825517+0000) 2019-10-13T13:01:32.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:32.824+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.826582+0000) 2019-10-13T13:01:32.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:32.824+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.827052+0000) 2019-10-13T13:01:32.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:32.824+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.826027+0000) 2019-10-13T13:01:32.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:32.827+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:32.829663+0000) 2019-10-13T13:01:32.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:32.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:34.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:34.252+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:34.255579+0000) 2019-10-13T13:01:34.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:34.536+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:34.540014+0000) 2019-10-13T13:01:36.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:36.443+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:36.444262+0000) 2019-10-13T13:01:37.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:37.832+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.835198+0000) 2019-10-13T13:01:37.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:37.832+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.835583+0000) 2019-10-13T13:01:37.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:37.832+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.835883+0000) 2019-10-13T13:01:37.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:37.832+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.836331+0000) 2019-10-13T13:01:37.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:37.835+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.837136+0000) 2019-10-13T13:01:37.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:37.832+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.837210+0000) 2019-10-13T13:01:37.839 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:37.836+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.837833+0000) 2019-10-13T13:01:37.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:37.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:37.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:37.843+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.844584+0000) 2019-10-13T13:01:37.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:37.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:37.911778+0000) 2019-10-13T13:01:38.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:38.136+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:38.137000+0000) 2019-10-13T13:01:38.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:38.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:38.157984+0000) 2019-10-13T13:01:38.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:38.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:38.678135+0000) 2019-10-13T13:01:41.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:41.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:41.928865+0000) 2019-10-13T13:01:42.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:42.843+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.847647+0000) 2019-10-13T13:01:42.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:42.844+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.848231+0000) 2019-10-13T13:01:42.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:42.844+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.848827+0000) 2019-10-13T13:01:42.851 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:42.844+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.849176+0000) 2019-10-13T13:01:42.852 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:42.848+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.850630+0000) 2019-10-13T13:01:42.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:42.848+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.851013+0000) 2019-10-13T13:01:42.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:42.848+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.851459+0000) 2019-10-13T13:01:42.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:42.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:42.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:42.855+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:42.858887+0000) 2019-10-13T13:01:44.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:44.252+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:44.255779+0000) 2019-10-13T13:01:44.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:44.536+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:44.540245+0000) 2019-10-13T13:01:46.026 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:01:46.026 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:46.030 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:01:46.030 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:01:46.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:46.447+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:46.450948+0000) 2019-10-13T13:01:47.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:47.856+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.860078+0000) 2019-10-13T13:01:47.862 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:47.856+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.860963+0000) 2019-10-13T13:01:47.865 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:47.860+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.863298+0000) 2019-10-13T13:01:47.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:47.863+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.864141+0000) 2019-10-13T13:01:47.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:47.864+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.866001+0000) 2019-10-13T13:01:47.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:47.864+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.866266+0000) 2019-10-13T13:01:47.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:47.864+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.867040+0000) 2019-10-13T13:01:47.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:47.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:47.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:47.871+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.872235+0000) 2019-10-13T13:01:47.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:47.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:47.911982+0000) 2019-10-13T13:01:48.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:48.132+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:48.137173+0000) 2019-10-13T13:01:48.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:48.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:48.158188+0000) 2019-10-13T13:01:48.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:48.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:48.678348+0000) 2019-10-13T13:01:51.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:51.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:51.929056+0000) 2019-10-13T13:01:52.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:52.871+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.873332+0000) 2019-10-13T13:01:52.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:52.868+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.873520+0000) 2019-10-13T13:01:52.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:52.872+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.874081+0000) 2019-10-13T13:01:52.876 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:52.872+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.874450+0000) 2019-10-13T13:01:52.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:52.872+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.874747+0000) 2019-10-13T13:01:52.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:52.872+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.876782+0000) 2019-10-13T13:01:52.878 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:52.872+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.876940+0000) 2019-10-13T13:01:52.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:52.879+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:52.881847+0000) 2019-10-13T13:01:52.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:52.880+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173664 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:54.257 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:54.252+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:54.255979+0000) 2019-10-13T13:01:54.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:54.536+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:54.540473+0000) 2019-10-13T13:01:56.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:56.455+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:56.457543+0000) 2019-10-13T13:01:57.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:01:57.883+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.886927+0000) 2019-10-13T13:01:57.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:57.884+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.886012+0000) 2019-10-13T13:01:57.891 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:01:57.884+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.887273+0000) 2019-10-13T13:01:57.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:57.884+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.886994+0000) 2019-10-13T13:01:57.891 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:57.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.887711+0000) 2019-10-13T13:01:57.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:57.884+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.888117+0000) 2019-10-13T13:01:57.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:01:57.888+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.890524+0000) 2019-10-13T13:01:57.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:01:57.891+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.894936+0000) 2019-10-13T13:01:57.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:01:57.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173696 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:01:57.913 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:01:57.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:57.912207+0000) 2019-10-13T13:01:58.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:01:58.132+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:58.137318+0000) 2019-10-13T13:01:58.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:01:58.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:58.158368+0000) 2019-10-13T13:01:58.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:01:58.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:01:58.678526+0000) 2019-10-13T13:02:01.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:01.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:01.929249+0000) 2019-10-13T13:02:02.900 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:02.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.898095+0000) 2019-10-13T13:02:02.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:02.896+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.898421+0000) 2019-10-13T13:02:02.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:02.896+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.898942+0000) 2019-10-13T13:02:02.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:02.896+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.899292+0000) 2019-10-13T13:02:02.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:02.899+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.900669+0000) 2019-10-13T13:02:02.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:02.896+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.900740+0000) 2019-10-13T13:02:02.903 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:02.896+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.901161+0000) 2019-10-13T13:02:02.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:02.903+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:02.907021+0000) 2019-10-13T13:02:02.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:02.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173725 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:04.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:04.252+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:04.256172+0000) 2019-10-13T13:02:04.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:04.536+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:04.540688+0000) 2019-10-13T13:02:06.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:06.459+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:06.464049+0000) 2019-10-13T13:02:07.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:07.908+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.910606+0000) 2019-10-13T13:02:07.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:07.908+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.911189+0000) 2019-10-13T13:02:07.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:07.908+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.911670+0000) 2019-10-13T13:02:07.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:07.908+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.912087+0000) 2019-10-13T13:02:07.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:07.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.912339+0000) 2019-10-13T13:02:07.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:07.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.912975+0000) 2019-10-13T13:02:07.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:07.908+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.913390+0000) 2019-10-13T13:02:07.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:07.912+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.914019+0000) 2019-10-13T13:02:07.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:07.915+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:07.919886+0000) 2019-10-13T13:02:07.922 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:07.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:08.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:08.132+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:08.137488+0000) 2019-10-13T13:02:08.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:08.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:08.158547+0000) 2019-10-13T13:02:08.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:08.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:08.678700+0000) 2019-10-13T13:02:11.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:11.927+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:11.929448+0000) 2019-10-13T13:02:12.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:12.920+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.923567+0000) 2019-10-13T13:02:12.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:12.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.924550+0000) 2019-10-13T13:02:12.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:12.920+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.924760+0000) 2019-10-13T13:02:12.927 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:12.920+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.925464+0000) 2019-10-13T13:02:12.928 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:12.922+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.927244+0000) 2019-10-13T13:02:12.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:12.924+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.928614+0000) 2019-10-13T13:02:12.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:12.924+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.928902+0000) 2019-10-13T13:02:12.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:12.926+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:12.931810+0000) 2019-10-13T13:02:12.934 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:12.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173785 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:14.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:14.252+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:14.256370+0000) 2019-10-13T13:02:14.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:14.540+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:14.540922+0000) 2019-10-13T13:02:16.141 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:02:16.141 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:16.145 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:02:16.145 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:16.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:16.466+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:16.470641+0000) 2019-10-13T13:02:17.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:17.908+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.912567+0000) 2019-10-13T13:02:17.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:17.934+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.936402+0000) 2019-10-13T13:02:17.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:17.932+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.937118+0000) 2019-10-13T13:02:17.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:17.932+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.936717+0000) 2019-10-13T13:02:17.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:17.932+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.937660+0000) 2019-10-13T13:02:17.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:17.936+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.938259+0000) 2019-10-13T13:02:17.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:17.936+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.939026+0000) 2019-10-13T13:02:17.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:17.936+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.939807+0000) 2019-10-13T13:02:17.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:17.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:17.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:17.942+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:17.947584+0000) 2019-10-13T13:02:18.139 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:18.136+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:18.137695+0000) 2019-10-13T13:02:18.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:18.156+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:18.158723+0000) 2019-10-13T13:02:18.680 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:18.676+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:18.678891+0000) 2019-10-13T13:02:21.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:21.926+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:21.929654+0000) 2019-10-13T13:02:22.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:22.943+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.947582+0000) 2019-10-13T13:02:22.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:22.943+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.947895+0000) 2019-10-13T13:02:22.950 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:22.943+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.948924+0000) 2019-10-13T13:02:22.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:22.947+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.950057+0000) 2019-10-13T13:02:22.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:22.947+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.950305+0000) 2019-10-13T13:02:22.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:22.946+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.950905+0000) 2019-10-13T13:02:22.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:22.947+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.950867+0000) 2019-10-13T13:02:22.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:22.950+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:22.955916+0000) 2019-10-13T13:02:22.957 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:22.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:24.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:24.251+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:24.256601+0000) 2019-10-13T13:02:24.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:24.539+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:24.541122+0000) 2019-10-13T13:02:26.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:26.474+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:26.476763+0000) 2019-10-13T13:02:27.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:27.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.912784+0000) 2019-10-13T13:02:27.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:27.955+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.959291+0000) 2019-10-13T13:02:27.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:27.955+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.959735+0000) 2019-10-13T13:02:27.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:27.955+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.961406+0000) 2019-10-13T13:02:27.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:27.958+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.961857+0000) 2019-10-13T13:02:27.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:27.959+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.961991+0000) 2019-10-13T13:02:27.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:27.959+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.962476+0000) 2019-10-13T13:02:27.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:27.959+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.963957+0000) 2019-10-13T13:02:27.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:27.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:27.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:27.966+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:27.969243+0000) 2019-10-13T13:02:28.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:28.135+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:28.137906+0000) 2019-10-13T13:02:28.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:28.155+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:28.158933+0000) 2019-10-13T13:02:28.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:28.675+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:28.679066+0000) 2019-10-13T13:02:31.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:31.926+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:31.929882+0000) 2019-10-13T13:02:32.972 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:32.967+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.970531+0000) 2019-10-13T13:02:32.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:32.966+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.971581+0000) 2019-10-13T13:02:32.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:32.967+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.971866+0000) 2019-10-13T13:02:32.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:32.967+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.972174+0000) 2019-10-13T13:02:32.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:32.967+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.972761+0000) 2019-10-13T13:02:32.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:32.967+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.973354+0000) 2019-10-13T13:02:32.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:32.971+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.973885+0000) 2019-10-13T13:02:32.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:32.974+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:32.978990+0000) 2019-10-13T13:02:32.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:32.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:34.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:34.251+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:34.256798+0000) 2019-10-13T13:02:34.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:34.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:34.541322+0000) 2019-10-13T13:02:36.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:36.482+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:36.483252+0000) 2019-10-13T13:02:37.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:37.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.912994+0000) 2019-10-13T13:02:37.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:37.979+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.984081+0000) 2019-10-13T13:02:37.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:37.982+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.984576+0000) 2019-10-13T13:02:37.987 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:37.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.984603+0000) 2019-10-13T13:02:37.987 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:37.979+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.985636+0000) 2019-10-13T13:02:37.988 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:37.983+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.986159+0000) 2019-10-13T13:02:37.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:37.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.986517+0000) 2019-10-13T13:02:37.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:37.983+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.986668+0000) 2019-10-13T13:02:37.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:37.986+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:37.990722+0000) 2019-10-13T13:02:37.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:37.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:38.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:38.135+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:38.138089+0000) 2019-10-13T13:02:38.160 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:38.155+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:38.159111+0000) 2019-10-13T13:02:38.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:38.675+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:38.679276+0000) 2019-10-13T13:02:41.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:41.926+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:41.930106+0000) 2019-10-13T13:02:42.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:42.991+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.996058+0000) 2019-10-13T13:02:42.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:42.991+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.997196+0000) 2019-10-13T13:02:42.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:42.994+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.997779+0000) 2019-10-13T13:02:43.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:42.995+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.997815+0000) 2019-10-13T13:02:43.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:42.995+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.998882+0000) 2019-10-13T13:02:43.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:42.995+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.999453+0000) 2019-10-13T13:02:43.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:42.995+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:42.999530+0000) 2019-10-13T13:02:43.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:42.999+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:43.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:43.006+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:43.008943+0000) 2019-10-13T13:02:44.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:44.251+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:44.256993+0000) 2019-10-13T13:02:44.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:44.535+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:44.541547+0000) 2019-10-13T13:02:46.213 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:02:46.214 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:46.217 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:02:46.217 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:02:46.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:46.486+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:46.489717+0000) 2019-10-13T13:02:47.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:47.907+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:47.913221+0000) 2019-10-13T13:02:48.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:48.003+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.009161+0000) 2019-10-13T13:02:48.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:48.003+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.009692+0000) 2019-10-13T13:02:48.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:48.007+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.009723+0000) 2019-10-13T13:02:48.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:48.007+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.010244+0000) 2019-10-13T13:02:48.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:48.007+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.010994+0000) 2019-10-13T13:02:48.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:48.006+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.011980+0000) 2019-10-13T13:02:48.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:48.007+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.011889+0000) 2019-10-13T13:02:48.019 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:48.011+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173992 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:48.022 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:48.018+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.020300+0000) 2019-10-13T13:02:48.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:48.135+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.138285+0000) 2019-10-13T13:02:48.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:48.155+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.159287+0000) 2019-10-13T13:02:48.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:48.675+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:48.679453+0000) 2019-10-13T13:02:51.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:51.926+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:51.930301+0000) 2019-10-13T13:02:53.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:53.018+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.021351+0000) 2019-10-13T13:02:53.023 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:53.015+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.021571+0000) 2019-10-13T13:02:53.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:53.019+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.023346+0000) 2019-10-13T13:02:53.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:53.019+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.023542+0000) 2019-10-13T13:02:53.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:53.019+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.025074+0000) 2019-10-13T13:02:53.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:53.019+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.025371+0000) 2019-10-13T13:02:53.029 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:53.023+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.027114+0000) 2019-10-13T13:02:53.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:53.023+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174023 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:53.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:53.030+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:53.032878+0000) 2019-10-13T13:02:54.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:54.251+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:54.257188+0000) 2019-10-13T13:02:54.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:54.539+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:54.541765+0000) 2019-10-13T13:02:56.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:56.494+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:56.496394+0000) 2019-10-13T13:02:57.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:57.911+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:57.913449+0000) 2019-10-13T13:02:58.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:58.027+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.032931+0000) 2019-10-13T13:02:58.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:02:58.031+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.034947+0000) 2019-10-13T13:02:58.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:58.031+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.035182+0000) 2019-10-13T13:02:58.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:02:58.034+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.037884+0000) 2019-10-13T13:02:58.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:02:58.031+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.037558+0000) 2019-10-13T13:02:58.040 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:02:58.035+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.037908+0000) 2019-10-13T13:02:58.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:58.035+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.039025+0000) 2019-10-13T13:02:58.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:02:58.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:02:58.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:02:58.042+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.044275+0000) 2019-10-13T13:02:58.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:02:58.135+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.138464+0000) 2019-10-13T13:02:58.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:02:58.159+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.159523+0000) 2019-10-13T13:02:58.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:02:58.675+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:02:58.679625+0000) 2019-10-13T13:03:01.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:01.926+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:01.930529+0000) 2019-10-13T13:03:03.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:03.039+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.045250+0000) 2019-10-13T13:03:03.048 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:03.043+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.046449+0000) 2019-10-13T13:03:03.050 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:03.043+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.048365+0000) 2019-10-13T13:03:03.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:03.046+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.048728+0000) 2019-10-13T13:03:03.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:03.047+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.051079+0000) 2019-10-13T13:03:03.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:03.047+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.053166+0000) 2019-10-13T13:03:03.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:03.051+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174080 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:03.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:03.051+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.054806+0000) 2019-10-13T13:03:03.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:03.058+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:03.061014+0000) 2019-10-13T13:03:04.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:04.251+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:04.257405+0000) 2019-10-13T13:03:04.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:04.539+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:04.541992+0000) 2019-10-13T13:03:06.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:06.498+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:06.502893+0000) 2019-10-13T13:03:07.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:07.911+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:07.913685+0000) 2019-10-13T13:03:08.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:08.054+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.057739+0000) 2019-10-13T13:03:08.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:08.055+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.058169+0000) 2019-10-13T13:03:08.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:08.055+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.060438+0000) 2019-10-13T13:03:08.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:08.055+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.060906+0000) 2019-10-13T13:03:08.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:08.059+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.062082+0000) 2019-10-13T13:03:08.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:08.059+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.062653+0000) 2019-10-13T13:03:08.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:08.059+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.063919+0000) 2019-10-13T13:03:08.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:08.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:08.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:08.066+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.069408+0000) 2019-10-13T13:03:08.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:08.135+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.138646+0000) 2019-10-13T13:03:08.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:08.159+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.159737+0000) 2019-10-13T13:03:08.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:08.679+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:08.679803+0000) 2019-10-13T13:03:11.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:11.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:11.930752+0000) 2019-10-13T13:03:13.071 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:13.063+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.069569+0000) 2019-10-13T13:03:13.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:13.066+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.070395+0000) 2019-10-13T13:03:13.072 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:13.067+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.070367+0000) 2019-10-13T13:03:13.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:13.067+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.071163+0000) 2019-10-13T13:03:13.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:13.067+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.071728+0000) 2019-10-13T13:03:13.074 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:13.067+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.072289+0000) 2019-10-13T13:03:13.075 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:13.067+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.073133+0000) 2019-10-13T13:03:13.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:13.075+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:13.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:13.078+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:13.080532+0000) 2019-10-13T13:03:14.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:14.255+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:14.257603+0000) 2019-10-13T13:03:14.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:14.539+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:14.542208+0000) 2019-10-13T13:03:16.346 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:03:16.347 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:16.350 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:03:16.350 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:16.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:16.506+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:16.511959+0000) 2019-10-13T13:03:17.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:17.911+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:17.913946+0000) 2019-10-13T13:03:18.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:18.074+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.081500+0000) 2019-10-13T13:03:18.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:18.078+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.083030+0000) 2019-10-13T13:03:18.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:18.078+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.083413+0000) 2019-10-13T13:03:18.085 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:18.078+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.083785+0000) 2019-10-13T13:03:18.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:18.081+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.084881+0000) 2019-10-13T13:03:18.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:18.078+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.084374+0000) 2019-10-13T13:03:18.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:18.078+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.085007+0000) 2019-10-13T13:03:18.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:18.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:18.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:18.089+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.092626+0000) 2019-10-13T13:03:18.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:18.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.138826+0000) 2019-10-13T13:03:18.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:18.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.159916+0000) 2019-10-13T13:03:18.681 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:18.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:18.679980+0000) 2019-10-13T13:03:21.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:21.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:21.930965+0000) 2019-10-13T13:03:23.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:23.086+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.093707+0000) 2019-10-13T13:03:23.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:23.090+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.095409+0000) 2019-10-13T13:03:23.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:23.090+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.095649+0000) 2019-10-13T13:03:23.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:23.090+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.096637+0000) 2019-10-13T13:03:23.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:23.090+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.097394+0000) 2019-10-13T13:03:23.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:23.094+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.098061+0000) 2019-10-13T13:03:23.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:23.093+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.099440+0000) 2019-10-13T13:03:23.104 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:23.098+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:23.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:23.101+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:23.106838+0000) 2019-10-13T13:03:24.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:24.254+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:24.257809+0000) 2019-10-13T13:03:24.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:24.538+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:24.542405+0000) 2019-10-13T13:03:26.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:26.517+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:26.523554+0000) 2019-10-13T13:03:27.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:27.910+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:27.914103+0000) 2019-10-13T13:03:28.108 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:28.102+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.106327+0000) 2019-10-13T13:03:28.109 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:28.102+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.107077+0000) 2019-10-13T13:03:28.109 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:28.101+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.107627+0000) 2019-10-13T13:03:28.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:28.102+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.108068+0000) 2019-10-13T13:03:28.110 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:28.102+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.108478+0000) 2019-10-13T13:03:28.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:28.102+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.109074+0000) 2019-10-13T13:03:28.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:28.102+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.109437+0000) 2019-10-13T13:03:28.115 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:28.109+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.114307+0000) 2019-10-13T13:03:28.116 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:28.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:28.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:28.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.139042+0000) 2019-10-13T13:03:28.162 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:28.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.160114+0000) 2019-10-13T13:03:28.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:28.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:28.680209+0000) 2019-10-13T13:03:31.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:31.929+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:31.931183+0000) 2019-10-13T13:03:33.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:33.110+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.117693+0000) 2019-10-13T13:03:33.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:33.113+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.119806+0000) 2019-10-13T13:03:33.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:33.114+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.120057+0000) 2019-10-13T13:03:33.124 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:33.118+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.122665+0000) 2019-10-13T13:03:33.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:33.118+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.122984+0000) 2019-10-13T13:03:33.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:33.118+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.124848+0000) 2019-10-13T13:03:33.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:33.118+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.124972+0000) 2019-10-13T13:03:33.128 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:33.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:33.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:33.129+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:33.134852+0000) 2019-10-13T13:03:34.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:34.254+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:34.257979+0000) 2019-10-13T13:03:34.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:34.538+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:34.542616+0000) 2019-10-13T13:03:36.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:36.525+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:36.530152+0000) 2019-10-13T13:03:37.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:37.910+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:37.914311+0000) 2019-10-13T13:03:38.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:38.126+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.129980+0000) 2019-10-13T13:03:38.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:38.126+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.130430+0000) 2019-10-13T13:03:38.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:38.126+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.130668+0000) 2019-10-13T13:03:38.134 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:38.125+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.131516+0000) 2019-10-13T13:03:38.134 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:38.126+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.131562+0000) 2019-10-13T13:03:38.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:38.126+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.132781+0000) 2019-10-13T13:03:38.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:38.126+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.133277+0000) 2019-10-13T13:03:38.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:38.134+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:38.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:38.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.139206+0000) 2019-10-13T13:03:38.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:38.133+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.139813+0000) 2019-10-13T13:03:38.162 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:38.154+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.160319+0000) 2019-10-13T13:03:38.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:38.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:38.680417+0000) 2019-10-13T13:03:41.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:41.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:41.931379+0000) 2019-10-13T13:03:43.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:43.138+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.142125+0000) 2019-10-13T13:03:43.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:43.138+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.144375+0000) 2019-10-13T13:03:43.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:43.142+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.145988+0000) 2019-10-13T13:03:43.148 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:43.142+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.146501+0000) 2019-10-13T13:03:43.149 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:43.141+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.146952+0000) 2019-10-13T13:03:43.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:43.142+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.147100+0000) 2019-10-13T13:03:43.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:43.142+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.148015+0000) 2019-10-13T13:03:43.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:43.146+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:43.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:43.149+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:43.154041+0000) 2019-10-13T13:03:44.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:44.254+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:44.258188+0000) 2019-10-13T13:03:44.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:44.542+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:44.542864+0000) 2019-10-13T13:03:46.503 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:03:46.504 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:46.508 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:03:46.508 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:03:46.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:46.537+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:46.541756+0000) 2019-10-13T13:03:47.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:47.914+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:47.914510+0000) 2019-10-13T13:03:48.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:48.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.139398+0000) 2019-10-13T13:03:48.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:48.153+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.156145+0000) 2019-10-13T13:03:48.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:48.150+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.156158+0000) 2019-10-13T13:03:48.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:48.150+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.157188+0000) 2019-10-13T13:03:48.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:48.154+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.159431+0000) 2019-10-13T13:03:48.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:48.154+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.161106+0000) 2019-10-13T13:03:48.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:48.158+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.162886+0000) 2019-10-13T13:03:48.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:48.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.163184+0000) 2019-10-13T13:03:48.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:48.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174346 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:48.166 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:48.158+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.163649+0000) 2019-10-13T13:03:48.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:48.674+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:48.680604+0000) 2019-10-13T13:03:51.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:51.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:51.931573+0000) 2019-10-13T13:03:53.168 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:53.161+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.166729+0000) 2019-10-13T13:03:53.169 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:53.162+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.167385+0000) 2019-10-13T13:03:53.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:53.162+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.168107+0000) 2019-10-13T13:03:53.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:53.162+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.168368+0000) 2019-10-13T13:03:53.171 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:53.162+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.169064+0000) 2019-10-13T13:03:53.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:53.162+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.169445+0000) 2019-10-13T13:03:53.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:53.166+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.169825+0000) 2019-10-13T13:03:53.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:53.169+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:53.173692+0000) 2019-10-13T13:03:53.177 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:53.170+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:54.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:54.254+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:54.258426+0000) 2019-10-13T13:03:54.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:54.538+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:54.543026+0000) 2019-10-13T13:03:54.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:54.906+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:54.910937+0000) 2019-10-13T13:03:56.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:56.545+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:56.548626+0000) 2019-10-13T13:03:57.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:57.910+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:57.914705+0000) 2019-10-13T13:03:58.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:58.134+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.139565+0000) 2019-10-13T13:03:58.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:03:58.173+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.178935+0000) 2019-10-13T13:03:58.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:03:58.174+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.179318+0000) 2019-10-13T13:03:58.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:58.174+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.179842+0000) 2019-10-13T13:03:58.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:03:58.174+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.180687+0000) 2019-10-13T13:03:58.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:03:58.174+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.181023+0000) 2019-10-13T13:03:58.185 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:03:58.174+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.181347+0000) 2019-10-13T13:03:58.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:03:58.178+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.181990+0000) 2019-10-13T13:03:58.189 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:03:58.182+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:03:58.193 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:03:58.189+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.192435+0000) 2019-10-13T13:03:58.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:03:58.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:03:58.680782+0000) 2019-10-13T13:04:01.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:01.929+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:01.931801+0000) 2019-10-13T13:04:03.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:03.185+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.191637+0000) 2019-10-13T13:04:03.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:03.186+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.193662+0000) 2019-10-13T13:04:03.195 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:03.190+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.194102+0000) 2019-10-13T13:04:03.198 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:03.190+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.195068+0000) 2019-10-13T13:04:03.198 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:03.190+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.195320+0000) 2019-10-13T13:04:03.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:03.190+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.197086+0000) 2019-10-13T13:04:03.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:03.190+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.197377+0000) 2019-10-13T13:04:03.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:03.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:03.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:03.197+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:03.203799+0000) 2019-10-13T13:04:04.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:04.258+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:04.258659+0000) 2019-10-13T13:04:04.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:04.542+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:04.543220+0000) 2019-10-13T13:04:04.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:04.910+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:04.911139+0000) 2019-10-13T13:04:06.561 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:06.553+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:06.559503+0000) 2019-10-13T13:04:07.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:07.914+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:07.914925+0000) 2019-10-13T13:04:08.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:08.138+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.139725+0000) 2019-10-13T13:04:08.205 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:08.198+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.203714+0000) 2019-10-13T13:04:08.206 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:08.198+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.204427+0000) 2019-10-13T13:04:08.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:08.201+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.205157+0000) 2019-10-13T13:04:08.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:08.202+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.206830+0000) 2019-10-13T13:04:08.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:08.202+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.207029+0000) 2019-10-13T13:04:08.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:08.202+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.207778+0000) 2019-10-13T13:04:08.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:08.202+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.208119+0000) 2019-10-13T13:04:08.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:08.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:08.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:08.209+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.214617+0000) 2019-10-13T13:04:08.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:08.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:08.680953+0000) 2019-10-13T13:04:11.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:11.925+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:11.931998+0000) 2019-10-13T13:04:13.218 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:13.210+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.216732+0000) 2019-10-13T13:04:13.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:13.210+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.217074+0000) 2019-10-13T13:04:13.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:13.210+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.217186+0000) 2019-10-13T13:04:13.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:13.210+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.217672+0000) 2019-10-13T13:04:13.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:13.214+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.218445+0000) 2019-10-13T13:04:13.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:13.213+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.219411+0000) 2019-10-13T13:04:13.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:13.214+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.219555+0000) 2019-10-13T13:04:13.226 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:13.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:13.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:13.221+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:13.225723+0000) 2019-10-13T13:04:14.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:14.253+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:14.258869+0000) 2019-10-13T13:04:14.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:14.537+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:14.543430+0000) 2019-10-13T13:04:14.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:14.905+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:14.911339+0000) 2019-10-13T13:04:16.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:16.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:16.566027+0000) 2019-10-13T13:04:16.635 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:04:16.635 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:16.639 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:04:16.639 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:17.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:17.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:17.915150+0000) 2019-10-13T13:04:18.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:18.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.139958+0000) 2019-10-13T13:04:18.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:18.221+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.228434+0000) 2019-10-13T13:04:18.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:18.221+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.228727+0000) 2019-10-13T13:04:18.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:18.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.231023+0000) 2019-10-13T13:04:18.233 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:18.225+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.231087+0000) 2019-10-13T13:04:18.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:18.225+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.232719+0000) 2019-10-13T13:04:18.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:18.225+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.233019+0000) 2019-10-13T13:04:18.235 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:18.229+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.234070+0000) 2019-10-13T13:04:18.238 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:18.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174525 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:18.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:18.233+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.239542+0000) 2019-10-13T13:04:18.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:18.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:18.681125+0000) 2019-10-13T13:04:21.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:21.928+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:21.932198+0000) 2019-10-13T13:04:23.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:23.233+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.239566+0000) 2019-10-13T13:04:23.243 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:23.236+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.241750+0000) 2019-10-13T13:04:23.243 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:23.237+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.241794+0000) 2019-10-13T13:04:23.245 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:23.237+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.243407+0000) 2019-10-13T13:04:23.245 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:23.237+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.243939+0000) 2019-10-13T13:04:23.247 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:23.241+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.245991+0000) 2019-10-13T13:04:23.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:23.241+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.248166+0000) 2019-10-13T13:04:23.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:23.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:23.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:23.244+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:23.251026+0000) 2019-10-13T13:04:24.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:24.257+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:24.259079+0000) 2019-10-13T13:04:24.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:24.541+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:24.543651+0000) 2019-10-13T13:04:24.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:24.909+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:24.911582+0000) 2019-10-13T13:04:26.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:26.568+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:26.572723+0000) 2019-10-13T13:04:27.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:27.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:27.915350+0000) 2019-10-13T13:04:28.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:28.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.140157+0000) 2019-10-13T13:04:28.254 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:28.249+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.252757+0000) 2019-10-13T13:04:28.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:28.249+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.253243+0000) 2019-10-13T13:04:28.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:28.249+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.253636+0000) 2019-10-13T13:04:28.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:28.253+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.255099+0000) 2019-10-13T13:04:28.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:28.252+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.258100+0000) 2019-10-13T13:04:28.261 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:28.257+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.259972+0000) 2019-10-13T13:04:28.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:28.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:28.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:28.257+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.261592+0000) 2019-10-13T13:04:28.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:28.264+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.268744+0000) 2019-10-13T13:04:28.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:28.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:28.681327+0000) 2019-10-13T13:04:31.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:31.928+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:31.932395+0000) 2019-10-13T13:04:33.267 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:33.261+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.264791+0000) 2019-10-13T13:04:33.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:33.261+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.265137+0000) 2019-10-13T13:04:33.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:33.265+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.266206+0000) 2019-10-13T13:04:33.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:33.265+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.266477+0000) 2019-10-13T13:04:33.268 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:33.265+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.266842+0000) 2019-10-13T13:04:33.269 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:33.260+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.267504+0000) 2019-10-13T13:04:33.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:33.265+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.268197+0000) 2019-10-13T13:04:33.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:33.268+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:33.272435+0000) 2019-10-13T13:04:33.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:33.269+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:34.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:34.257+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:34.259288+0000) 2019-10-13T13:04:34.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:34.541+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:34.543855+0000) 2019-10-13T13:04:34.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:34.909+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:34.911783+0000) 2019-10-13T13:04:36.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:36.580+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:36.584222+0000) 2019-10-13T13:04:37.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:37.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:37.915557+0000) 2019-10-13T13:04:38.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:38.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.140351+0000) 2019-10-13T13:04:38.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:38.272+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.276711+0000) 2019-10-13T13:04:38.279 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:38.273+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.276808+0000) 2019-10-13T13:04:38.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:38.273+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.277524+0000) 2019-10-13T13:04:38.280 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:38.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.278822+0000) 2019-10-13T13:04:38.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:38.277+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.279436+0000) 2019-10-13T13:04:38.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:38.277+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.279886+0000) 2019-10-13T13:04:38.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:38.277+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.280537+0000) 2019-10-13T13:04:38.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:38.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174645 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:38.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:38.284+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.289233+0000) 2019-10-13T13:04:38.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:38.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:38.681499+0000) 2019-10-13T13:04:41.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:41.928+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:41.932622+0000) 2019-10-13T13:04:43.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:43.285+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.288296+0000) 2019-10-13T13:04:43.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:43.285+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.289277+0000) 2019-10-13T13:04:43.292 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:43.289+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.290940+0000) 2019-10-13T13:04:43.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:43.289+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.293547+0000) 2019-10-13T13:04:43.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:43.288+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.295028+0000) 2019-10-13T13:04:43.298 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:43.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:43.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:43.293+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.297127+0000) 2019-10-13T13:04:43.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:43.292+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.297924+0000) 2019-10-13T13:04:43.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:43.293+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:43.297673+0000) 2019-10-13T13:04:44.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:44.257+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:44.259531+0000) 2019-10-13T13:04:44.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:44.541+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:44.544078+0000) 2019-10-13T13:04:44.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:44.909+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:44.911987+0000) 2019-10-13T13:04:46.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:46.584+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:46.590848+0000) 2019-10-13T13:04:46.867 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:04:46.868 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:46.871 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:04:46.872 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:04:47.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:47.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:47.915716+0000) 2019-10-13T13:04:48.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:48.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.140543+0000) 2019-10-13T13:04:48.302 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:48.297+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.300313+0000) 2019-10-13T13:04:48.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:48.297+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.301661+0000) 2019-10-13T13:04:48.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:48.301+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.301977+0000) 2019-10-13T13:04:48.304 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:48.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.302347+0000) 2019-10-13T13:04:48.304 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:48.301+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.302816+0000) 2019-10-13T13:04:48.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:48.300+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.305697+0000) 2019-10-13T13:04:48.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:48.305+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.305843+0000) 2019-10-13T13:04:48.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:48.304+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.309935+0000) 2019-10-13T13:04:48.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:48.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:48.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:48.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:48.681678+0000) 2019-10-13T13:04:51.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:51.932+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:51.932816+0000) 2019-10-13T13:04:53.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:53.309+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.313154+0000) 2019-10-13T13:04:53.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:53.309+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.313720+0000) 2019-10-13T13:04:53.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:53.313+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.316376+0000) 2019-10-13T13:04:53.319 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:53.317+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.318150+0000) 2019-10-13T13:04:53.320 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:53.317+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.318524+0000) 2019-10-13T13:04:53.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:53.317+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.321292+0000) 2019-10-13T13:04:53.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:53.316+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.323234+0000) 2019-10-13T13:04:53.325 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:53.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174733 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:53.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:53.320+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:53.325109+0000) 2019-10-13T13:04:54.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:54.257+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:54.259735+0000) 2019-10-13T13:04:54.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:54.541+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:54.544270+0000) 2019-10-13T13:04:54.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:54.909+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:54.912190+0000) 2019-10-13T13:04:56.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:56.592+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:56.597385+0000) 2019-10-13T13:04:57.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:57.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:57.915951+0000) 2019-10-13T13:04:58.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:58.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.140744+0000) 2019-10-13T13:04:58.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:04:58.325+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.327678+0000) 2019-10-13T13:04:58.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:04:58.325+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.328324+0000) 2019-10-13T13:04:58.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:04:58.325+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.329124+0000) 2019-10-13T13:04:58.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:04:58.325+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.329674+0000) 2019-10-13T13:04:58.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:58.329+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.329946+0000) 2019-10-13T13:04:58.332 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:04:58.324+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.330845+0000) 2019-10-13T13:04:58.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:04:58.329+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.330875+0000) 2019-10-13T13:04:58.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:04:58.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:04:58.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:04:58.332+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.336948+0000) 2019-10-13T13:04:58.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:04:58.681+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:04:58.681854+0000) 2019-10-13T13:05:01.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:01.932+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:01.933041+0000) 2019-10-13T13:05:03.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:03.337+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.339266+0000) 2019-10-13T13:05:03.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:03.337+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.340615+0000) 2019-10-13T13:05:03.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:03.337+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.340947+0000) 2019-10-13T13:05:03.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:03.337+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.341492+0000) 2019-10-13T13:05:03.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:03.341+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.341919+0000) 2019-10-13T13:05:03.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:03.341+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.342888+0000) 2019-10-13T13:05:03.345 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:03.336+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.343403+0000) 2019-10-13T13:05:03.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:03.340+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:03.347399+0000) 2019-10-13T13:05:03.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:03.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:04.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:04.257+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:04.259886+0000) 2019-10-13T13:05:04.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:04.541+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:04.544473+0000) 2019-10-13T13:05:04.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:04.909+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:04.912387+0000) 2019-10-13T13:05:06.605 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:06.596+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:06.603831+0000) 2019-10-13T13:05:07.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:07.913+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:07.916167+0000) 2019-10-13T13:05:08.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:08.137+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.140980+0000) 2019-10-13T13:05:08.354 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:08.349+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.352555+0000) 2019-10-13T13:05:08.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:08.348+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.353445+0000) 2019-10-13T13:05:08.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:08.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.354048+0000) 2019-10-13T13:05:08.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:08.353+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.354621+0000) 2019-10-13T13:05:08.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:08.353+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.356649+0000) 2019-10-13T13:05:08.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:08.353+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.356741+0000) 2019-10-13T13:05:08.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:08.353+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.357368+0000) 2019-10-13T13:05:08.362 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:08.357+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:08.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:08.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.364792+0000) 2019-10-13T13:05:08.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:08.681+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:08.682027+0000) 2019-10-13T13:05:11.934 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:11.928+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:11.933243+0000) 2019-10-13T13:05:13.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:13.360+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.364419+0000) 2019-10-13T13:05:13.366 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:13.360+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.364764+0000) 2019-10-13T13:05:13.367 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:13.360+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.365564+0000) 2019-10-13T13:05:13.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:13.364+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.369508+0000) 2019-10-13T13:05:13.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:13.364+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.369435+0000) 2019-10-13T13:05:13.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:13.368+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.371518+0000) 2019-10-13T13:05:13.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:13.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:13.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:13.372+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.374316+0000) 2019-10-13T13:05:13.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:13.376+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:13.381221+0000) 2019-10-13T13:05:14.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:14.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:14.260088+0000) 2019-10-13T13:05:14.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:14.540+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:14.544709+0000) 2019-10-13T13:05:14.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:14.908+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:14.912588+0000) 2019-10-13T13:05:16.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:16.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:16.610301+0000) 2019-10-13T13:05:17.050 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:05:17.050 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:17.054 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:05:17.054 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:17.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:17.912+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:17.916363+0000) 2019-10-13T13:05:18.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:18.136+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.141178+0000) 2019-10-13T13:05:18.379 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:18.372+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.377374+0000) 2019-10-13T13:05:18.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:18.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.377865+0000) 2019-10-13T13:05:18.380 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:18.376+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.378047+0000) 2019-10-13T13:05:18.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:18.376+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.379220+0000) 2019-10-13T13:05:18.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:18.376+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.379715+0000) 2019-10-13T13:05:18.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:18.376+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.379938+0000) 2019-10-13T13:05:18.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:18.376+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.380388+0000) 2019-10-13T13:05:18.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:18.380+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:18.387 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:18.380+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.385825+0000) 2019-10-13T13:05:18.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:18.680+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:18.682247+0000) 2019-10-13T13:05:21.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:21.932+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:21.933441+0000) 2019-10-13T13:05:23.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:23.384+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.388531+0000) 2019-10-13T13:05:23.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:23.388+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.390548+0000) 2019-10-13T13:05:23.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:23.388+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.390947+0000) 2019-10-13T13:05:23.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:23.392+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.393174+0000) 2019-10-13T13:05:23.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:23.388+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.393418+0000) 2019-10-13T13:05:23.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:23.392+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.393891+0000) 2019-10-13T13:05:23.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:23.392+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.394372+0000) 2019-10-13T13:05:23.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:23.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:23.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:23.396+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:23.400038+0000) 2019-10-13T13:05:24.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:24.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:24.260218+0000) 2019-10-13T13:05:24.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:24.544+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:24.544919+0000) 2019-10-13T13:05:24.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:24.908+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:24.912795+0000) 2019-10-13T13:05:26.618 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:26.615+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:26.616861+0000) 2019-10-13T13:05:27.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:27.912+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:27.916562+0000) 2019-10-13T13:05:28.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:28.140+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.141388+0000) 2019-10-13T13:05:28.402 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:28.396+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.400326+0000) 2019-10-13T13:05:28.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:28.396+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.400764+0000) 2019-10-13T13:05:28.404 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:28.400+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.402581+0000) 2019-10-13T13:05:28.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:28.400+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.402948+0000) 2019-10-13T13:05:28.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:28.400+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.403229+0000) 2019-10-13T13:05:28.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:28.399+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.403746+0000) 2019-10-13T13:05:28.406 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:28.400+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.404055+0000) 2019-10-13T13:05:28.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:28.407+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.408200+0000) 2019-10-13T13:05:28.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:28.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174943 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:28.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:28.680+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:28.682443+0000) 2019-10-13T13:05:31.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:31.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:31.933631+0000) 2019-10-13T13:05:33.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:33.408+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.412845+0000) 2019-10-13T13:05:33.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:33.412+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.414547+0000) 2019-10-13T13:05:33.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:33.412+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.414846+0000) 2019-10-13T13:05:33.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:33.411+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.415521+0000) 2019-10-13T13:05:33.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:33.412+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.415866+0000) 2019-10-13T13:05:33.418 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:33.412+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.416759+0000) 2019-10-13T13:05:33.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:33.412+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.417211+0000) 2019-10-13T13:05:33.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:33.419+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:33.422087+0000) 2019-10-13T13:05:33.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:33.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:34.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:34.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:34.260398+0000) 2019-10-13T13:05:34.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:34.540+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:34.545112+0000) 2019-10-13T13:05:34.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:34.908+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:34.913007+0000) 2019-10-13T13:05:36.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:36.619+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:36.623362+0000) 2019-10-13T13:05:37.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:37.916+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:37.916760+0000) 2019-10-13T13:05:38.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:38.136+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.141603+0000) 2019-10-13T13:05:38.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:38.424+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.425767+0000) 2019-10-13T13:05:38.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:38.424+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.427235+0000) 2019-10-13T13:05:38.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:38.424+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.428204+0000) 2019-10-13T13:05:38.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:38.427+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.428463+0000) 2019-10-13T13:05:38.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:38.424+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.428665+0000) 2019-10-13T13:05:38.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:38.424+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.429020+0000) 2019-10-13T13:05:38.431 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:38.424+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.429280+0000) 2019-10-13T13:05:38.436 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:38.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:38.438 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:38.435+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.436491+0000) 2019-10-13T13:05:38.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:38.680+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:38.682683+0000) 2019-10-13T13:05:41.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:41.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:41.933859+0000) 2019-10-13T13:05:43.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:43.432+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.437474+0000) 2019-10-13T13:05:43.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:43.436+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.439546+0000) 2019-10-13T13:05:43.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:43.436+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.439901+0000) 2019-10-13T13:05:43.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:43.439+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.441863+0000) 2019-10-13T13:05:43.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:43.436+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.441694+0000) 2019-10-13T13:05:43.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:43.440+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.443499+0000) 2019-10-13T13:05:43.445 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:43.440+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.444151+0000) 2019-10-13T13:05:43.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:43.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:43.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:43.451+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:43.452712+0000) 2019-10-13T13:05:44.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:44.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:44.260625+0000) 2019-10-13T13:05:44.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:44.544+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:44.545335+0000) 2019-10-13T13:05:44.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:44.908+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:44.913204+0000) 2019-10-13T13:05:46.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:46.627+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:46.629883+0000) 2019-10-13T13:05:47.098 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:05:47.099 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:47.102 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:05:47.102 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:05:47.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:47.912+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:47.916959+0000) 2019-10-13T13:05:48.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:48.140+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.141826+0000) 2019-10-13T13:05:48.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:48.444+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.449691+0000) 2019-10-13T13:05:48.452 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:48.448+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.450199+0000) 2019-10-13T13:05:48.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:48.451+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.452870+0000) 2019-10-13T13:05:48.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:48.448+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.453023+0000) 2019-10-13T13:05:48.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:48.452+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.455012+0000) 2019-10-13T13:05:48.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:48.452+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.455050+0000) 2019-10-13T13:05:48.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:48.452+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.457551+0000) 2019-10-13T13:05:48.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:48.456+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:48.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:48.459+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.462327+0000) 2019-10-13T13:05:48.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:48.680+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:48.682859+0000) 2019-10-13T13:05:51.935 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:51.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:51.934085+0000) 2019-10-13T13:05:53.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:53.456+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.461392+0000) 2019-10-13T13:05:53.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:53.459+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.462754+0000) 2019-10-13T13:05:53.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:53.464+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.466991+0000) 2019-10-13T13:05:53.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:53.464+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.467869+0000) 2019-10-13T13:05:53.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:53.464+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.468167+0000) 2019-10-13T13:05:53.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:53.464+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.468364+0000) 2019-10-13T13:05:53.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:53.464+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.469073+0000) 2019-10-13T13:05:53.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:53.467+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:53.469587+0000) 2019-10-13T13:05:53.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:53.468+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:54.262 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:54.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:54.260804+0000) 2019-10-13T13:05:54.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:54.544+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:54.545533+0000) 2019-10-13T13:05:54.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:54.908+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:54.913417+0000) 2019-10-13T13:05:56.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:56.635+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:56.636366+0000) 2019-10-13T13:05:57.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:57.912+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:57.917154+0000) 2019-10-13T13:05:58.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:58.140+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.142016+0000) 2019-10-13T13:05:58.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:58.476+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.478518+0000) 2019-10-13T13:05:58.482 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:05:58.476+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.480960+0000) 2019-10-13T13:05:58.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:05:58.480+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.482711+0000) 2019-10-13T13:05:58.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:05:58.480+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.483160+0000) 2019-10-13T13:05:58.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:05:58.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:05:58.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:05:58.483+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.484179+0000) 2019-10-13T13:05:58.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:05:58.480+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.483726+0000) 2019-10-13T13:05:58.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:05:58.480+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.484374+0000) 2019-10-13T13:05:58.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:05:58.483+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.485098+0000) 2019-10-13T13:05:58.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:05:58.680+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:05:58.683052+0000) 2019-10-13T13:06:01.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:01.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:01.934276+0000) 2019-10-13T13:06:03.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:03.484+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.485962+0000) 2019-10-13T13:06:03.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:03.484+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.487470+0000) 2019-10-13T13:06:03.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:03.484+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.488160+0000) 2019-10-13T13:06:03.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:03.484+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.488826+0000) 2019-10-13T13:06:03.490 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:03.484+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.489027+0000) 2019-10-13T13:06:03.492 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:03.488+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.490285+0000) 2019-10-13T13:06:03.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:03.487+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.490978+0000) 2019-10-13T13:06:03.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:03.491+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:03.494464+0000) 2019-10-13T13:06:03.497 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:03.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:04.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:04.256+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:04.260988+0000) 2019-10-13T13:06:04.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:04.544+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:04.545789+0000) 2019-10-13T13:06:04.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:04.907+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:04.913650+0000) 2019-10-13T13:06:06.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:06.639+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:06.642891+0000) 2019-10-13T13:06:07.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:07.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:07.917353+0000) 2019-10-13T13:06:08.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:08.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.142229+0000) 2019-10-13T13:06:08.502 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:08.495+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.499559+0000) 2019-10-13T13:06:08.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:08.495+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.500110+0000) 2019-10-13T13:06:08.503 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:08.495+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.500532+0000) 2019-10-13T13:06:08.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:08.495+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.500949+0000) 2019-10-13T13:06:08.504 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:08.495+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.501218+0000) 2019-10-13T13:06:08.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:08.499+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.502246+0000) 2019-10-13T13:06:08.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:08.499+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.503492+0000) 2019-10-13T13:06:08.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:08.503+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.508056+0000) 2019-10-13T13:06:08.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:08.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:08.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:08.679+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:08.683193+0000) 2019-10-13T13:06:11.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:11.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:11.934470+0000) 2019-10-13T13:06:13.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:13.507+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.511824+0000) 2019-10-13T13:06:13.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:13.507+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.512503+0000) 2019-10-13T13:06:13.517 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:13.511+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.515670+0000) 2019-10-13T13:06:13.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:13.511+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.516315+0000) 2019-10-13T13:06:13.518 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:13.511+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.515959+0000) 2019-10-13T13:06:13.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:13.511+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.516665+0000) 2019-10-13T13:06:13.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:13.515+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.517417+0000) 2019-10-13T13:06:13.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:13.515+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:13.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:13.519+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:13.523617+0000) 2019-10-13T13:06:14.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:14.255+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:14.261160+0000) 2019-10-13T13:06:14.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:14.543+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:14.546028+0000) 2019-10-13T13:06:14.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:14.911+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:14.913849+0000) 2019-10-13T13:06:16.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:16.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:16.649444+0000) 2019-10-13T13:06:17.201 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:06:17.201 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:17.204 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:06:17.204 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:17.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:17.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:17.917572+0000) 2019-10-13T13:06:18.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:18.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.142445+0000) 2019-10-13T13:06:18.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:18.519+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.525289+0000) 2019-10-13T13:06:18.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:18.519+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.525637+0000) 2019-10-13T13:06:18.529 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:18.523+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.527410+0000) 2019-10-13T13:06:18.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:18.523+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.527885+0000) 2019-10-13T13:06:18.530 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:18.523+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.528758+0000) 2019-10-13T13:06:18.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:18.527+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.529453+0000) 2019-10-13T13:06:18.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:18.527+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.530178+0000) 2019-10-13T13:06:18.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:18.527+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:18.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:18.535+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.536357+0000) 2019-10-13T13:06:18.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:18.679+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:18.683388+0000) 2019-10-13T13:06:21.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:21.931+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:21.934680+0000) 2019-10-13T13:06:23.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:23.531+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.536211+0000) 2019-10-13T13:06:23.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:23.531+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.536769+0000) 2019-10-13T13:06:23.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:23.535+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.538748+0000) 2019-10-13T13:06:23.540 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:23.535+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.538695+0000) 2019-10-13T13:06:23.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:23.535+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.539171+0000) 2019-10-13T13:06:23.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:23.535+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.539674+0000) 2019-10-13T13:06:23.543 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:23.535+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.541034+0000) 2019-10-13T13:06:23.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:23.539+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:23.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:23.543+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:23.547660+0000) 2019-10-13T13:06:24.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:24.259+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:24.261339+0000) 2019-10-13T13:06:24.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:24.543+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:24.546258+0000) 2019-10-13T13:06:24.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:24.911+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:24.914047+0000) 2019-10-13T13:06:26.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:26.651+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:26.655983+0000) 2019-10-13T13:06:27.919 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:27.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:27.917843+0000) 2019-10-13T13:06:28.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:28.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.142682+0000) 2019-10-13T13:06:28.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:28.543+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.549102+0000) 2019-10-13T13:06:28.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:28.547+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.549966+0000) 2019-10-13T13:06:28.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:28.547+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.550278+0000) 2019-10-13T13:06:28.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:28.547+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.550938+0000) 2019-10-13T13:06:28.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:28.547+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.551244+0000) 2019-10-13T13:06:28.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:28.551+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.553384+0000) 2019-10-13T13:06:28.555 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:28.547+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.553544+0000) 2019-10-13T13:06:28.558 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:28.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:28.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:28.559+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.561837+0000) 2019-10-13T13:06:28.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:28.679+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:28.683564+0000) 2019-10-13T13:06:31.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:31.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:31.934884+0000) 2019-10-13T13:06:33.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:33.559+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.561833+0000) 2019-10-13T13:06:33.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:33.559+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.562848+0000) 2019-10-13T13:06:33.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:33.559+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.563130+0000) 2019-10-13T13:06:33.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:33.559+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.563426+0000) 2019-10-13T13:06:33.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:33.559+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.564068+0000) 2019-10-13T13:06:33.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:33.562+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.564759+0000) 2019-10-13T13:06:33.567 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:33.559+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.565683+0000) 2019-10-13T13:06:33.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:33.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:33.573 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:33.566+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:33.571572+0000) 2019-10-13T13:06:34.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:34.259+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:34.261514+0000) 2019-10-13T13:06:34.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:34.543+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:34.546460+0000) 2019-10-13T13:06:34.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:34.911+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:34.914249+0000) 2019-10-13T13:06:36.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:36.658+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:36.662599+0000) 2019-10-13T13:06:37.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:37.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:37.918040+0000) 2019-10-13T13:06:38.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:38.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.142889+0000) 2019-10-13T13:06:38.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:38.567+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.572394+0000) 2019-10-13T13:06:38.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:38.570+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.574184+0000) 2019-10-13T13:06:38.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:38.571+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.575597+0000) 2019-10-13T13:06:38.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:38.571+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.577606+0000) 2019-10-13T13:06:38.580 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:38.575+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.578077+0000) 2019-10-13T13:06:38.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:38.575+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.578471+0000) 2019-10-13T13:06:38.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:38.575+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.579042+0000) 2019-10-13T13:06:38.583 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:38.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:38.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:38.578+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.582672+0000) 2019-10-13T13:06:38.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:38.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:38.683740+0000) 2019-10-13T13:06:41.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:41.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:41.935089+0000) 2019-10-13T13:06:43.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:43.583+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.585882+0000) 2019-10-13T13:06:43.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:43.583+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.586317+0000) 2019-10-13T13:06:43.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:43.583+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.586784+0000) 2019-10-13T13:06:43.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:43.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.586980+0000) 2019-10-13T13:06:43.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:43.583+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.587471+0000) 2019-10-13T13:06:43.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:43.583+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.587967+0000) 2019-10-13T13:06:43.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:43.586+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.590438+0000) 2019-10-13T13:06:43.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:43.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:43.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:43.594+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:43.598223+0000) 2019-10-13T13:06:44.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:44.255+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:44.261701+0000) 2019-10-13T13:06:44.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:44.543+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:44.546637+0000) 2019-10-13T13:06:44.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:44.911+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:44.914448+0000) 2019-10-13T13:06:46.670 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:46.666+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:46.669105+0000) 2019-10-13T13:06:47.356 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:06:47.357 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:47.360 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:06:47.360 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:06:47.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:47.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:47.918240+0000) 2019-10-13T13:06:48.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:48.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.143069+0000) 2019-10-13T13:06:48.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:48.591+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.597495+0000) 2019-10-13T13:06:48.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:48.595+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.599628+0000) 2019-10-13T13:06:48.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:48.595+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.600358+0000) 2019-10-13T13:06:48.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:48.595+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.600708+0000) 2019-10-13T13:06:48.603 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:48.595+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.601099+0000) 2019-10-13T13:06:48.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:48.598+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.602226+0000) 2019-10-13T13:06:48.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:48.599+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.602140+0000) 2019-10-13T13:06:48.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:48.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:48.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:48.606+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.608233+0000) 2019-10-13T13:06:48.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:48.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:48.683913+0000) 2019-10-13T13:06:51.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:51.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:51.935283+0000) 2019-10-13T13:06:53.612 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:53.606+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.610343+0000) 2019-10-13T13:06:53.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:53.607+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.610442+0000) 2019-10-13T13:06:53.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:53.607+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.610502+0000) 2019-10-13T13:06:53.613 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:53.607+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.610789+0000) 2019-10-13T13:06:53.613 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:53.607+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.611187+0000) 2019-10-13T13:06:53.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:53.607+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.612404+0000) 2019-10-13T13:06:53.614 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:53.607+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.612980+0000) 2019-10-13T13:06:53.620 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:53.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:53.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:53.614+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:53.619570+0000) 2019-10-13T13:06:54.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:54.259+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:54.261888+0000) 2019-10-13T13:06:54.548 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:54.543+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:54.546814+0000) 2019-10-13T13:06:54.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:54.911+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:54.914651+0000) 2019-10-13T13:06:56.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:56.670+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:56.675650+0000) 2019-10-13T13:06:57.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:57.915+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:57.918473+0000) 2019-10-13T13:06:58.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:58.139+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.143294+0000) 2019-10-13T13:06:58.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:06:58.619+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.622062+0000) 2019-10-13T13:06:58.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:06:58.619+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.622225+0000) 2019-10-13T13:06:58.625 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:06:58.619+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.622811+0000) 2019-10-13T13:06:58.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:06:58.619+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.623214+0000) 2019-10-13T13:06:58.626 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:06:58.619+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.624183+0000) 2019-10-13T13:06:58.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:06:58.622+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.625414+0000) 2019-10-13T13:06:58.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:58.619+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.625443+0000) 2019-10-13T13:06:58.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:06:58.626+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.630366+0000) 2019-10-13T13:06:58.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:06:58.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:06:58.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:06:58.679+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:06:58.684091+0000) 2019-10-13T13:07:01.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:01.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:01.935496+0000) 2019-10-13T13:07:03.639 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:03.630+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.637197+0000) 2019-10-13T13:07:03.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:03.634+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.638634+0000) 2019-10-13T13:07:03.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:03.634+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.639084+0000) 2019-10-13T13:07:03.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:03.634+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.639198+0000) 2019-10-13T13:07:03.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:03.634+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.640032+0000) 2019-10-13T13:07:03.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:03.634+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.641130+0000) 2019-10-13T13:07:03.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:03.634+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.641581+0000) 2019-10-13T13:07:03.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:03.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:03.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:03.646+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:03.649859+0000) 2019-10-13T13:07:04.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:04.258+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:04.262075+0000) 2019-10-13T13:07:04.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:04.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:04.546983+0000) 2019-10-13T13:07:04.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:04.910+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:04.914855+0000) 2019-10-13T13:07:06.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:06.678+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:06.681592+0000) 2019-10-13T13:07:07.920 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:07.914+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:07.918639+0000) 2019-10-13T13:07:08.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:08.142+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.143496+0000) 2019-10-13T13:07:08.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:08.642+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.648119+0000) 2019-10-13T13:07:08.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:08.642+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.648308+0000) 2019-10-13T13:07:08.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:08.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.650005+0000) 2019-10-13T13:07:08.652 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:08.646+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.650623+0000) 2019-10-13T13:07:08.654 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:08.646+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.652241+0000) 2019-10-13T13:07:08.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:08.646+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.652437+0000) 2019-10-13T13:07:08.655 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:08.650+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.653897+0000) 2019-10-13T13:07:08.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:08.654+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.656655+0000) 2019-10-13T13:07:08.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:08.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:08.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:08.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:08.684266+0000) 2019-10-13T13:07:11.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:11.930+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:11.935712+0000) 2019-10-13T13:07:13.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:13.654+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.660771+0000) 2019-10-13T13:07:13.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:13.658+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.662411+0000) 2019-10-13T13:07:13.664 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:13.658+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.663047+0000) 2019-10-13T13:07:13.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:13.658+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.663470+0000) 2019-10-13T13:07:13.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:13.662+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.664751+0000) 2019-10-13T13:07:13.666 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:13.658+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.664910+0000) 2019-10-13T13:07:13.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:13.658+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.664662+0000) 2019-10-13T13:07:13.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:13.666+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:13.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:13.666+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:13.671140+0000) 2019-10-13T13:07:14.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:14.258+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:14.262263+0000) 2019-10-13T13:07:14.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:14.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:14.547164+0000) 2019-10-13T13:07:14.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:14.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:14.915086+0000) 2019-10-13T13:07:16.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:16.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:16.688176+0000) 2019-10-13T13:07:17.428 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:07:17.429 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:17.432 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:07:17.433 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:17.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:17.918+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:17.918796+0000) 2019-10-13T13:07:18.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:18.142+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.143693+0000) 2019-10-13T13:07:18.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:18.666+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.673351+0000) 2019-10-13T13:07:18.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:18.670+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.674226+0000) 2019-10-13T13:07:18.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:18.670+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.675432+0000) 2019-10-13T13:07:18.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:18.670+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.675493+0000) 2019-10-13T13:07:18.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:18.670+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.675734+0000) 2019-10-13T13:07:18.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:18.670+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.676214+0000) 2019-10-13T13:07:18.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:18.670+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.676397+0000) 2019-10-13T13:07:18.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:18.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:18.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:18.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.684404+0000) 2019-10-13T13:07:18.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:18.682+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:18.684981+0000) 2019-10-13T13:07:21.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:21.934+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:21.935904+0000) 2019-10-13T13:07:23.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:23.678+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.684911+0000) 2019-10-13T13:07:23.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:23.678+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.685317+0000) 2019-10-13T13:07:23.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:23.678+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.685593+0000) 2019-10-13T13:07:23.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:23.682+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.687608+0000) 2019-10-13T13:07:23.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:23.682+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.687853+0000) 2019-10-13T13:07:23.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:23.682+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.689220+0000) 2019-10-13T13:07:23.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:23.686+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.691080+0000) 2019-10-13T13:07:23.695 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:23.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:23.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:23.694+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:23.698383+0000) 2019-10-13T13:07:24.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:24.258+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:24.262488+0000) 2019-10-13T13:07:24.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:24.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:24.547365+0000) 2019-10-13T13:07:24.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:24.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:24.915326+0000) 2019-10-13T13:07:26.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:26.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:26.694685+0000) 2019-10-13T13:07:27.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:27.918+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:27.919000+0000) 2019-10-13T13:07:28.145 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:28.142+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.143891+0000) 2019-10-13T13:07:28.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:28.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.684616+0000) 2019-10-13T13:07:28.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:28.690+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.697134+0000) 2019-10-13T13:07:28.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:28.694+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.697825+0000) 2019-10-13T13:07:28.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:28.694+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.698483+0000) 2019-10-13T13:07:28.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:28.694+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.698690+0000) 2019-10-13T13:07:28.701 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:28.694+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.698947+0000) 2019-10-13T13:07:28.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:28.694+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.699604+0000) 2019-10-13T13:07:28.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:28.694+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.700240+0000) 2019-10-13T13:07:28.706 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:28.702+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:28.704468+0000) 2019-10-13T13:07:28.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:28.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175678 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:31.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:31.934+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:31.936124+0000) 2019-10-13T13:07:33.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:33.702+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.708174+0000) 2019-10-13T13:07:33.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:33.706+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.709009+0000) 2019-10-13T13:07:33.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:33.706+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.710319+0000) 2019-10-13T13:07:33.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:33.706+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.710577+0000) 2019-10-13T13:07:33.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:33.706+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.710949+0000) 2019-10-13T13:07:33.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:33.706+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.711326+0000) 2019-10-13T13:07:33.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:33.706+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.711908+0000) 2019-10-13T13:07:33.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:33.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:33.716307+0000) 2019-10-13T13:07:33.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:33.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175707 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:34.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:34.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:34.262677+0000) 2019-10-13T13:07:34.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:34.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:34.547547+0000) 2019-10-13T13:07:34.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:34.910+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:34.915509+0000) 2019-10-13T13:07:36.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:36.697+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:36.701459+0000) 2019-10-13T13:07:37.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:37.914+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:37.919257+0000) 2019-10-13T13:07:38.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:38.142+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.144089+0000) 2019-10-13T13:07:38.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:38.682+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.684872+0000) 2019-10-13T13:07:38.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:38.714+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.720273+0000) 2019-10-13T13:07:38.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:38.718+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.722103+0000) 2019-10-13T13:07:38.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:38.717+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.722787+0000) 2019-10-13T13:07:38.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:38.718+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.722790+0000) 2019-10-13T13:07:38.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:38.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.723202+0000) 2019-10-13T13:07:38.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:38.718+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.724457+0000) 2019-10-13T13:07:38.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:38.722+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.725810+0000) 2019-10-13T13:07:38.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:38.722+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:38.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:38.725+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:38.729998+0000) 2019-10-13T13:07:41.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:41.933+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:41.936324+0000) 2019-10-13T13:07:43.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:43.726+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.732719+0000) 2019-10-13T13:07:43.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:43.726+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.733314+0000) 2019-10-13T13:07:43.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:43.729+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.735050+0000) 2019-10-13T13:07:43.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:43.730+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.735344+0000) 2019-10-13T13:07:43.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:43.730+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.735493+0000) 2019-10-13T13:07:43.737 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:43.730+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.736024+0000) 2019-10-13T13:07:43.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:43.730+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.736695+0000) 2019-10-13T13:07:43.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:43.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175767 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:43.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:43.737+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:43.742898+0000) 2019-10-13T13:07:44.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:44.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:44.262863+0000) 2019-10-13T13:07:44.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:44.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:44.547746+0000) 2019-10-13T13:07:44.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:44.910+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:44.915724+0000) 2019-10-13T13:07:46.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:46.701+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:46.707980+0000) 2019-10-13T13:07:47.759 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:07:47.759 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:47.762 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:07:47.763 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:07:47.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:47.914+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:47.919525+0000) 2019-10-13T13:07:48.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:48.142+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.144287+0000) 2019-10-13T13:07:48.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:48.678+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.685074+0000) 2019-10-13T13:07:48.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:48.738+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.745030+0000) 2019-10-13T13:07:48.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:48.738+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.745310+0000) 2019-10-13T13:07:48.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:48.742+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.746196+0000) 2019-10-13T13:07:48.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:48.742+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.746619+0000) 2019-10-13T13:07:48.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:48.741+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.747373+0000) 2019-10-13T13:07:48.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:48.742+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.747525+0000) 2019-10-13T13:07:48.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:48.742+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.747865+0000) 2019-10-13T13:07:48.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:48.750+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:48.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:48.753+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:48.756365+0000) 2019-10-13T13:07:51.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:51.933+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:51.936519+0000) 2019-10-13T13:07:53.759 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:53.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.757131+0000) 2019-10-13T13:07:53.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:53.750+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.757423+0000) 2019-10-13T13:07:53.760 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:53.754+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.758341+0000) 2019-10-13T13:07:53.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:53.754+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.759055+0000) 2019-10-13T13:07:53.761 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:53.754+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.759529+0000) 2019-10-13T13:07:53.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:53.754+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.759831+0000) 2019-10-13T13:07:53.762 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:53.757+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.760477+0000) 2019-10-13T13:07:53.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:53.761+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:53.764283+0000) 2019-10-13T13:07:53.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:53.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:07:54.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:54.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:54.263072+0000) 2019-10-13T13:07:54.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:54.542+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:54.547946+0000) 2019-10-13T13:07:54.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:54.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:54.915940+0000) 2019-10-13T13:07:56.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:56.709+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:56.714566+0000) 2019-10-13T13:07:57.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:57.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:57.919769+0000) 2019-10-13T13:07:58.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:58.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.144492+0000) 2019-10-13T13:07:58.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:58.677+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.685303+0000) 2019-10-13T13:07:58.770 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:07:58.761+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.768577+0000) 2019-10-13T13:07:58.771 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:07:58.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.770132+0000) 2019-10-13T13:07:58.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:07:58.765+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.770405+0000) 2019-10-13T13:07:58.772 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:07:58.765+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.770929+0000) 2019-10-13T13:07:58.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:07:58.765+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.771357+0000) 2019-10-13T13:07:58.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:07:58.765+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.771929+0000) 2019-10-13T13:07:58.774 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:07:58.769+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.772415+0000) 2019-10-13T13:07:58.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:07:58.773+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:07:58.776428+0000) 2019-10-13T13:07:58.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:07:58.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:01.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:01.933+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:01.936726+0000) 2019-10-13T13:08:03.783 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:03.777+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.781289+0000) 2019-10-13T13:08:03.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:03.777+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.782299+0000) 2019-10-13T13:08:03.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:03.777+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.781858+0000) 2019-10-13T13:08:03.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:03.777+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.782590+0000) 2019-10-13T13:08:03.785 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:03.777+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.783014+0000) 2019-10-13T13:08:03.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:03.777+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.783679+0000) 2019-10-13T13:08:03.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:03.777+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.784568+0000) 2019-10-13T13:08:03.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:03.785+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:03.790531+0000) 2019-10-13T13:08:03.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:03.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:04.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:04.261+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:04.263269+0000) 2019-10-13T13:08:04.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:04.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:04.548131+0000) 2019-10-13T13:08:04.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:04.913+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:04.916152+0000) 2019-10-13T13:08:06.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:06.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:06.721053+0000) 2019-10-13T13:08:07.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:07.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:07.919963+0000) 2019-10-13T13:08:08.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:08.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.144724+0000) 2019-10-13T13:08:08.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:08.681+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.685497+0000) 2019-10-13T13:08:08.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:08.789+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.793515+0000) 2019-10-13T13:08:08.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:08.789+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.794386+0000) 2019-10-13T13:08:08.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:08.789+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.794857+0000) 2019-10-13T13:08:08.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:08.789+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.795475+0000) 2019-10-13T13:08:08.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:08.789+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.795795+0000) 2019-10-13T13:08:08.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:08.789+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.796141+0000) 2019-10-13T13:08:08.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:08.789+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.796437+0000) 2019-10-13T13:08:08.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:08.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:08.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:08.801+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:08.804681+0000) 2019-10-13T13:08:11.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:11.933+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:11.936923+0000) 2019-10-13T13:08:13.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:13.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.807371+0000) 2019-10-13T13:08:13.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:13.801+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.807940+0000) 2019-10-13T13:08:13.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:13.805+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.808025+0000) 2019-10-13T13:08:13.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:13.805+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.808897+0000) 2019-10-13T13:08:13.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:13.805+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.809357+0000) 2019-10-13T13:08:13.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:13.809+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.810023+0000) 2019-10-13T13:08:13.813 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:13.809+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.810187+0000) 2019-10-13T13:08:13.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:13.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:13.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:13.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:13.815063+0000) 2019-10-13T13:08:14.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:14.261+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:14.263459+0000) 2019-10-13T13:08:14.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:14.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:14.548327+0000) 2019-10-13T13:08:14.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:14.913+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:14.916349+0000) 2019-10-13T13:08:16.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:16.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:16.726738+0000) 2019-10-13T13:08:17.873 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:08:17.873 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:17.877 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:08:17.877 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:17.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:17.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:17.920165+0000) 2019-10-13T13:08:18.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:18.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.144927+0000) 2019-10-13T13:08:18.687 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:18.681+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.685713+0000) 2019-10-13T13:08:18.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:18.817+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.818232+0000) 2019-10-13T13:08:18.821 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:18.817+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.818922+0000) 2019-10-13T13:08:18.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:18.817+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.819926+0000) 2019-10-13T13:08:18.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:18.817+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.820533+0000) 2019-10-13T13:08:18.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:18.817+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.820576+0000) 2019-10-13T13:08:18.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:18.817+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.820827+0000) 2019-10-13T13:08:18.823 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:18.817+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.820593+0000) 2019-10-13T13:08:18.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:18.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:18.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:18.829+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:18.832900+0000) 2019-10-13T13:08:21.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:21.933+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:21.937134+0000) 2019-10-13T13:08:23.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:23.825+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.828938+0000) 2019-10-13T13:08:23.831 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:23.829+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.829833+0000) 2019-10-13T13:08:23.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:23.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.830453+0000) 2019-10-13T13:08:23.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:23.825+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.831544+0000) 2019-10-13T13:08:23.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:23.829+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.831690+0000) 2019-10-13T13:08:23.834 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:23.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.832104+0000) 2019-10-13T13:08:23.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:23.829+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.832858+0000) 2019-10-13T13:08:23.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:23.833+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:23.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:23.833+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:23.839204+0000) 2019-10-13T13:08:24.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:24.261+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:24.263644+0000) 2019-10-13T13:08:24.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:24.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:24.548507+0000) 2019-10-13T13:08:24.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:24.913+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:24.916593+0000) 2019-10-13T13:08:26.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:26.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:26.733336+0000) 2019-10-13T13:08:27.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:27.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:27.920395+0000) 2019-10-13T13:08:28.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:28.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.145089+0000) 2019-10-13T13:08:28.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:28.685+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.685933+0000) 2019-10-13T13:08:28.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:28.837+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.839724+0000) 2019-10-13T13:08:28.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:28.837+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.840321+0000) 2019-10-13T13:08:28.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:28.837+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.840753+0000) 2019-10-13T13:08:28.844 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:28.841+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.842218+0000) 2019-10-13T13:08:28.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:28.837+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.843136+0000) 2019-10-13T13:08:28.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:28.841+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.843208+0000) 2019-10-13T13:08:28.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:28.841+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.843036+0000) 2019-10-13T13:08:28.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:28.845+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:28.850384+0000) 2019-10-13T13:08:28.852 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:28.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:31.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:31.937+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:31.937329+0000) 2019-10-13T13:08:33.855 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:33.853+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.853836+0000) 2019-10-13T13:08:33.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:33.853+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.856485+0000) 2019-10-13T13:08:33.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:33.853+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.856472+0000) 2019-10-13T13:08:33.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:33.853+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.856826+0000) 2019-10-13T13:08:33.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:33.853+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.857205+0000) 2019-10-13T13:08:33.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:33.857+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.857944+0000) 2019-10-13T13:08:33.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:33.857+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.858712+0000) 2019-10-13T13:08:33.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:33.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:33.865 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:33.857+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:33.863860+0000) 2019-10-13T13:08:34.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:34.261+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:34.263835+0000) 2019-10-13T13:08:34.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:34.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:34.548690+0000) 2019-10-13T13:08:34.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:34.913+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:34.916749+0000) 2019-10-13T13:08:36.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:36.733+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:36.739782+0000) 2019-10-13T13:08:37.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:37.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:37.920588+0000) 2019-10-13T13:08:38.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:38.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.145303+0000) 2019-10-13T13:08:38.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:38.685+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.686046+0000) 2019-10-13T13:08:38.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:38.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.865007+0000) 2019-10-13T13:08:38.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:38.861+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.865368+0000) 2019-10-13T13:08:38.869 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:38.865+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.867026+0000) 2019-10-13T13:08:38.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:38.865+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.868897+0000) 2019-10-13T13:08:38.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:38.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.870402+0000) 2019-10-13T13:08:38.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:38.869+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.870379+0000) 2019-10-13T13:08:38.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:38.869+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.870712+0000) 2019-10-13T13:08:38.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:38.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176105 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:38.878 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:38.872+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:38.877214+0000) 2019-10-13T13:08:41.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:41.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:41.937531+0000) 2019-10-13T13:08:43.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:43.873+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.877622+0000) 2019-10-13T13:08:43.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:43.872+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.878496+0000) 2019-10-13T13:08:43.880 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:43.877+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.878569+0000) 2019-10-13T13:08:43.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:43.877+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.880314+0000) 2019-10-13T13:08:43.882 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:43.877+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.880964+0000) 2019-10-13T13:08:43.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:43.881+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.882985+0000) 2019-10-13T13:08:43.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:43.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.884517+0000) 2019-10-13T13:08:43.888 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:43.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:43.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:43.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:43.893204+0000) 2019-10-13T13:08:44.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:44.261+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:44.264024+0000) 2019-10-13T13:08:44.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:44.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:44.548902+0000) 2019-10-13T13:08:44.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:44.913+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:44.916977+0000) 2019-10-13T13:08:46.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:46.740+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:46.746042+0000) 2019-10-13T13:08:47.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:47.917+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:47.920801+0000) 2019-10-13T13:08:48.015 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:08:48.016 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:48.019 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:08:48.019 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:08:48.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:48.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.145538+0000) 2019-10-13T13:08:48.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:48.685+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.686261+0000) 2019-10-13T13:08:48.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:48.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.890471+0000) 2019-10-13T13:08:48.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:48.889+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.890522+0000) 2019-10-13T13:08:48.893 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:48.889+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.891117+0000) 2019-10-13T13:08:48.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:48.889+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.891752+0000) 2019-10-13T13:08:48.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:48.889+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.892011+0000) 2019-10-13T13:08:48.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:48.889+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.891785+0000) 2019-10-13T13:08:48.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:48.889+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.892557+0000) 2019-10-13T13:08:48.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:48.897+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176164 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:48.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:48.896+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:48.900288+0000) 2019-10-13T13:08:51.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:51.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:51.937764+0000) 2019-10-13T13:08:53.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:53.896+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.902702+0000) 2019-10-13T13:08:53.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:53.900+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.903383+0000) 2019-10-13T13:08:53.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:53.900+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.903605+0000) 2019-10-13T13:08:53.906 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:53.900+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.904006+0000) 2019-10-13T13:08:53.907 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:53.900+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.905703+0000) 2019-10-13T13:08:53.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:53.904+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.907650+0000) 2019-10-13T13:08:53.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:53.908+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.910414+0000) 2019-10-13T13:08:53.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:53.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:08:53.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:53.912+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:53.919181+0000) 2019-10-13T13:08:54.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:54.260+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:54.264246+0000) 2019-10-13T13:08:54.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:54.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:54.549081+0000) 2019-10-13T13:08:54.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:54.912+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:54.917179+0000) 2019-10-13T13:08:56.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:56.748+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:56.752413+0000) 2019-10-13T13:08:57.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:57.916+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:57.921013+0000) 2019-10-13T13:08:58.147 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:58.144+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.145767+0000) 2019-10-13T13:08:58.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:58.684+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.686455+0000) 2019-10-13T13:08:58.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:08:58.912+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.914810+0000) 2019-10-13T13:08:58.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:08:58.912+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.915814+0000) 2019-10-13T13:08:58.918 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:08:58.912+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.917077+0000) 2019-10-13T13:08:58.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:08:58.912+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.917662+0000) 2019-10-13T13:08:58.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:08:58.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.917724+0000) 2019-10-13T13:08:58.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:08:58.916+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.918577+0000) 2019-10-13T13:08:58.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:08:58.916+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.918287+0000) 2019-10-13T13:08:58.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:08:58.916+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:08:58.923465+0000) 2019-10-13T13:08:58.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:08:58.920+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:01.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:01.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:01.937986+0000) 2019-10-13T13:09:03.929 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:03.924+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.927358+0000) 2019-10-13T13:09:03.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:03.924+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.928320+0000) 2019-10-13T13:09:03.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:03.924+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.928374+0000) 2019-10-13T13:09:03.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:03.924+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.928953+0000) 2019-10-13T13:09:03.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:03.924+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.929640+0000) 2019-10-13T13:09:03.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:03.928+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.929968+0000) 2019-10-13T13:09:03.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:03.928+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.930713+0000) 2019-10-13T13:09:03.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:03.932+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:03.937199+0000) 2019-10-13T13:09:03.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:03.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:04.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:04.260+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:04.264430+0000) 2019-10-13T13:09:04.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:04.544+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:04.549292+0000) 2019-10-13T13:09:04.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:04.912+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:04.917414+0000) 2019-10-13T13:09:06.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:06.756+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:06.759090+0000) 2019-10-13T13:09:07.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:07.916+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:07.921213+0000) 2019-10-13T13:09:08.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:08.144+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.145966+0000) 2019-10-13T13:09:08.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:08.684+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.686671+0000) 2019-10-13T13:09:08.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:08.936+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.940073+0000) 2019-10-13T13:09:08.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:08.936+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.942396+0000) 2019-10-13T13:09:08.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:08.940+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.943620+0000) 2019-10-13T13:09:08.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:08.940+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.944672+0000) 2019-10-13T13:09:08.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:08.940+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.945548+0000) 2019-10-13T13:09:08.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:08.940+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.945597+0000) 2019-10-13T13:09:08.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:08.944+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.946350+0000) 2019-10-13T13:09:08.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:08.944+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:08.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:08.944+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:08.950642+0000) 2019-10-13T13:09:11.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:11.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:11.938180+0000) 2019-10-13T13:09:13.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:13.948+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.952057+0000) 2019-10-13T13:09:13.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:13.948+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.952589+0000) 2019-10-13T13:09:13.955 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:13.948+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.952929+0000) 2019-10-13T13:09:13.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:13.948+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.954091+0000) 2019-10-13T13:09:13.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:13.952+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.954651+0000) 2019-10-13T13:09:13.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:13.952+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.955012+0000) 2019-10-13T13:09:13.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:13.952+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.955331+0000) 2019-10-13T13:09:13.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:13.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:13.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:13.956+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:13.962686+0000) 2019-10-13T13:09:14.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:14.260+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:14.264573+0000) 2019-10-13T13:09:14.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:14.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:14.549500+0000) 2019-10-13T13:09:14.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:14.912+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:14.917589+0000) 2019-10-13T13:09:16.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:16.768+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:16.770503+0000) 2019-10-13T13:09:17.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:17.916+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:17.921387+0000) 2019-10-13T13:09:18.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:18.144+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.146173+0000) 2019-10-13T13:09:18.231 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:09:18.231 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:18.234 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:09:18.235 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:18.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:18.684+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.686873+0000) 2019-10-13T13:09:18.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:18.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.964486+0000) 2019-10-13T13:09:18.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:18.964+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.966032+0000) 2019-10-13T13:09:18.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:18.964+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.966469+0000) 2019-10-13T13:09:18.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:18.964+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.967644+0000) 2019-10-13T13:09:18.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:18.964+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.968311+0000) 2019-10-13T13:09:18.970 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:18.964+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.968569+0000) 2019-10-13T13:09:18.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:18.964+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.969124+0000) 2019-10-13T13:09:18.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:18.968+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:18.973483+0000) 2019-10-13T13:09:18.975 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:18.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:21.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:21.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:21.938428+0000) 2019-10-13T13:09:23.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:23.972+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.976861+0000) 2019-10-13T13:09:23.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:23.972+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.977714+0000) 2019-10-13T13:09:23.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:23.976+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.978721+0000) 2019-10-13T13:09:23.981 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:23.976+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.979497+0000) 2019-10-13T13:09:23.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:23.972+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.980050+0000) 2019-10-13T13:09:23.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:23.976+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.980142+0000) 2019-10-13T13:09:23.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:23.976+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.981273+0000) 2019-10-13T13:09:23.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:23.980+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:23.985648+0000) 2019-10-13T13:09:23.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:23.984+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176371 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:24.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:24.264+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:24.264756+0000) 2019-10-13T13:09:24.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:24.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:24.549678+0000) 2019-10-13T13:09:24.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:24.916+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:24.917780+0000) 2019-10-13T13:09:26.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:26.776+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:26.777126+0000) 2019-10-13T13:09:27.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:27.916+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:27.921603+0000) 2019-10-13T13:09:28.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:28.144+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.146414+0000) 2019-10-13T13:09:28.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:28.684+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.687074+0000) 2019-10-13T13:09:28.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:28.988+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.990581+0000) 2019-10-13T13:09:28.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:28.988+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.990900+0000) 2019-10-13T13:09:28.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:28.988+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.991454+0000) 2019-10-13T13:09:28.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:28.988+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.992943+0000) 2019-10-13T13:09:28.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:28.988+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.993459+0000) 2019-10-13T13:09:28.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:28.988+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.994753+0000) 2019-10-13T13:09:28.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:28.992+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:28.994759+0000) 2019-10-13T13:09:29.000 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:28.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:29.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:28.996+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:29.001236+0000) 2019-10-13T13:09:31.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:31.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:31.938616+0000) 2019-10-13T13:09:34.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:33.996+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.001574+0000) 2019-10-13T13:09:34.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:34.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.003090+0000) 2019-10-13T13:09:34.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:34.000+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.003052+0000) 2019-10-13T13:09:34.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:34.000+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.003460+0000) 2019-10-13T13:09:34.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:34.000+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.004419+0000) 2019-10-13T13:09:34.006 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:34.000+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.003818+0000) 2019-10-13T13:09:34.006 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:34.000+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.004637+0000) 2019-10-13T13:09:34.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:34.004+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.009629+0000) 2019-10-13T13:09:34.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:34.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:34.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:34.264+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.264937+0000) 2019-10-13T13:09:34.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:34.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.549858+0000) 2019-10-13T13:09:34.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:34.916+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:34.917989+0000) 2019-10-13T13:09:36.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:36.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:36.783642+0000) 2019-10-13T13:09:37.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:37.920+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:37.921835+0000) 2019-10-13T13:09:38.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:38.144+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:38.146653+0000) 2019-10-13T13:09:38.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:38.684+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:38.687275+0000) 2019-10-13T13:09:39.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:39.012+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.013864+0000) 2019-10-13T13:09:39.019 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:39.012+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.016989+0000) 2019-10-13T13:09:39.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:39.012+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.017711+0000) 2019-10-13T13:09:39.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:39.012+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.017513+0000) 2019-10-13T13:09:39.020 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:39.016+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.018963+0000) 2019-10-13T13:09:39.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:39.016+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.019616+0000) 2019-10-13T13:09:39.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:39.016+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.019584+0000) 2019-10-13T13:09:39.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:39.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:39.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:39.024+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:39.026019+0000) 2019-10-13T13:09:41.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:41.936+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:41.938840+0000) 2019-10-13T13:09:44.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:44.024+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.028556+0000) 2019-10-13T13:09:44.031 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:44.024+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.029295+0000) 2019-10-13T13:09:44.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:44.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.029613+0000) 2019-10-13T13:09:44.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:44.027+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.030162+0000) 2019-10-13T13:09:44.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:44.028+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.031180+0000) 2019-10-13T13:09:44.033 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:44.028+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.031695+0000) 2019-10-13T13:09:44.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:44.028+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.032592+0000) 2019-10-13T13:09:44.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:44.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:44.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:44.035+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.037699+0000) 2019-10-13T13:09:44.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:44.260+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.265117+0000) 2019-10-13T13:09:44.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:44.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.550044+0000) 2019-10-13T13:09:44.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:44.916+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:44.918175+0000) 2019-10-13T13:09:46.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:46.787+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:46.790191+0000) 2019-10-13T13:09:47.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:47.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:47.922049+0000) 2019-10-13T13:09:48.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:48.143+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:48.146875+0000) 2019-10-13T13:09:48.347 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:09:48.347 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:48.351 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:09:48.351 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:09:48.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:48.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:48.687474+0000) 2019-10-13T13:09:49.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:49.035+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.039693+0000) 2019-10-13T13:09:49.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:49.035+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.040195+0000) 2019-10-13T13:09:49.042 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:49.035+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.040506+0000) 2019-10-13T13:09:49.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:49.035+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.041059+0000) 2019-10-13T13:09:49.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:49.035+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.041384+0000) 2019-10-13T13:09:49.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:49.039+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.042327+0000) 2019-10-13T13:09:49.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:49.039+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.042214+0000) 2019-10-13T13:09:49.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:49.043+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:49.045938+0000) 2019-10-13T13:09:49.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:49.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176528 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:51.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:51.935+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:51.939049+0000) 2019-10-13T13:09:54.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:54.047+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.051840+0000) 2019-10-13T13:09:54.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:54.047+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.052421+0000) 2019-10-13T13:09:54.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:54.047+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.053006+0000) 2019-10-13T13:09:54.055 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:54.051+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.053872+0000) 2019-10-13T13:09:54.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:54.051+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.054065+0000) 2019-10-13T13:09:54.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:54.051+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.054462+0000) 2019-10-13T13:09:54.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:54.051+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.054849+0000) 2019-10-13T13:09:54.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:54.055+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.058225+0000) 2019-10-13T13:09:54.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:54.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:54.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:54.259+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.265293+0000) 2019-10-13T13:09:54.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:54.547+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.550260+0000) 2019-10-13T13:09:54.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:54.915+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:54.918393+0000) 2019-10-13T13:09:56.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:56.795+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:56.797185+0000) 2019-10-13T13:09:57.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:57.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:57.922252+0000) 2019-10-13T13:09:58.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:58.143+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:58.147075+0000) 2019-10-13T13:09:58.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:58.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:58.687713+0000) 2019-10-13T13:09:59.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:09:59.059+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.062884+0000) 2019-10-13T13:09:59.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:09:59.059+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.064677+0000) 2019-10-13T13:09:59.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:09:59.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.065163+0000) 2019-10-13T13:09:59.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:09:59.063+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.065863+0000) 2019-10-13T13:09:59.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:09:59.063+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.065806+0000) 2019-10-13T13:09:59.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:09:59.063+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.066212+0000) 2019-10-13T13:09:59.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:09:59.063+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.068158+0000) 2019-10-13T13:09:59.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:09:59.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:09:59.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:09:59.071+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:09:59.074604+0000) 2019-10-13T13:10:01.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:01.935+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:01.939245+0000) 2019-10-13T13:10:04.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:04.071+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.075852+0000) 2019-10-13T13:10:04.078 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:04.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.077097+0000) 2019-10-13T13:10:04.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:04.075+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.078881+0000) 2019-10-13T13:10:04.081 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:04.075+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.079614+0000) 2019-10-13T13:10:04.082 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:04.079+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.081066+0000) 2019-10-13T13:10:04.083 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:04.075+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.081123+0000) 2019-10-13T13:10:04.083 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:04.075+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.081362+0000) 2019-10-13T13:10:04.087 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:04.083+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:04.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:04.083+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.087493+0000) 2019-10-13T13:10:04.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:04.263+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.265496+0000) 2019-10-13T13:10:04.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:04.547+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.550451+0000) 2019-10-13T13:10:04.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:04.915+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:04.918567+0000) 2019-10-13T13:10:06.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:06.803+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:06.803941+0000) 2019-10-13T13:10:07.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:07.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:07.922481+0000) 2019-10-13T13:10:08.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:08.143+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:08.147303+0000) 2019-10-13T13:10:08.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:08.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:08.687913+0000) 2019-10-13T13:10:09.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:09.083+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.089041+0000) 2019-10-13T13:10:09.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:09.083+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.089650+0000) 2019-10-13T13:10:09.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:09.087+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.090447+0000) 2019-10-13T13:10:09.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:09.087+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.091170+0000) 2019-10-13T13:10:09.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:09.087+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.091347+0000) 2019-10-13T13:10:09.093 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:09.087+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.091654+0000) 2019-10-13T13:10:09.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:09.087+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.093005+0000) 2019-10-13T13:10:09.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:09.095+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:09.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:09.099+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:09.102283+0000) 2019-10-13T13:10:11.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:11.935+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:11.939439+0000) 2019-10-13T13:10:14.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:14.099+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.102263+0000) 2019-10-13T13:10:14.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:14.099+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.102445+0000) 2019-10-13T13:10:14.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:14.099+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.103205+0000) 2019-10-13T13:10:14.105 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:14.099+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.103688+0000) 2019-10-13T13:10:14.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:14.099+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.105498+0000) 2019-10-13T13:10:14.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:14.103+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.105973+0000) 2019-10-13T13:10:14.108 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:14.103+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.106597+0000) 2019-10-13T13:10:14.112 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:14.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:14.115 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:14.111+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.113549+0000) 2019-10-13T13:10:14.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:14.259+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.265678+0000) 2019-10-13T13:10:14.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:14.547+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.550625+0000) 2019-10-13T13:10:14.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:14.915+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:14.918758+0000) 2019-10-13T13:10:16.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:16.807+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:16.810476+0000) 2019-10-13T13:10:17.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:17.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:17.922694+0000) 2019-10-13T13:10:18.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:18.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:18.147504+0000) 2019-10-13T13:10:18.444 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:10:18.444 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:18.448 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:10:18.448 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:18.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:18.687+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:18.688141+0000) 2019-10-13T13:10:19.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:19.115+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.119096+0000) 2019-10-13T13:10:19.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:19.111+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.114383+0000) 2019-10-13T13:10:19.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:19.111+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.114621+0000) 2019-10-13T13:10:19.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:19.111+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.116316+0000) 2019-10-13T13:10:19.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:19.115+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.118110+0000) 2019-10-13T13:10:19.122 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:19.115+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.119501+0000) 2019-10-13T13:10:19.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:19.115+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.121314+0000) 2019-10-13T13:10:19.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:19.119+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:19.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:19.127+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:19.128773+0000) 2019-10-13T13:10:21.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:21.935+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:21.939634+0000) 2019-10-13T13:10:24.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:24.123+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.127345+0000) 2019-10-13T13:10:24.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:24.123+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.127982+0000) 2019-10-13T13:10:24.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:24.123+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.128078+0000) 2019-10-13T13:10:24.133 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:24.127+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.131468+0000) 2019-10-13T13:10:24.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:24.127+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.131631+0000) 2019-10-13T13:10:24.135 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:24.131+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.133454+0000) 2019-10-13T13:10:24.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:24.131+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.134746+0000) 2019-10-13T13:10:24.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:24.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:24.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:24.135+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.137018+0000) 2019-10-13T13:10:24.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:24.263+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.265856+0000) 2019-10-13T13:10:24.552 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:24.547+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.550860+0000) 2019-10-13T13:10:24.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:24.915+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:24.918940+0000) 2019-10-13T13:10:26.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:26.815+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:26.817142+0000) 2019-10-13T13:10:27.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:27.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:27.922914+0000) 2019-10-13T13:10:28.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:28.143+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:28.147765+0000) 2019-10-13T13:10:28.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:28.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:28.688377+0000) 2019-10-13T13:10:29.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:29.135+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.141270+0000) 2019-10-13T13:10:29.144 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:29.139+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.141965+0000) 2019-10-13T13:10:29.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:29.135+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.141662+0000) 2019-10-13T13:10:29.145 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:29.139+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.143762+0000) 2019-10-13T13:10:29.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:29.139+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.144031+0000) 2019-10-13T13:10:29.146 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:29.139+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.144731+0000) 2019-10-13T13:10:29.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:29.143+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.145931+0000) 2019-10-13T13:10:29.148 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:29.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:29.146113+0000) 2019-10-13T13:10:29.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:29.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:31.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:31.939+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:31.939833+0000) 2019-10-13T13:10:34.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:34.147+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.149990+0000) 2019-10-13T13:10:34.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:34.147+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.150514+0000) 2019-10-13T13:10:34.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:34.147+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.153230+0000) 2019-10-13T13:10:34.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:34.147+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.152633+0000) 2019-10-13T13:10:34.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:34.151+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.153750+0000) 2019-10-13T13:10:34.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:34.151+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.154175+0000) 2019-10-13T13:10:34.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:34.151+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.154576+0000) 2019-10-13T13:10:34.160 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:34.155+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176799 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:34.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:34.159+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.160224+0000) 2019-10-13T13:10:34.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:34.263+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.266033+0000) 2019-10-13T13:10:34.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:34.547+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.551036+0000) 2019-10-13T13:10:34.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:34.915+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:34.919114+0000) 2019-10-13T13:10:36.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:36.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:36.823564+0000) 2019-10-13T13:10:37.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:37.919+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:37.923117+0000) 2019-10-13T13:10:38.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:38.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:38.147962+0000) 2019-10-13T13:10:38.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:38.683+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:38.688571+0000) 2019-10-13T13:10:39.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:39.159+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.162560+0000) 2019-10-13T13:10:39.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:39.159+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.164127+0000) 2019-10-13T13:10:39.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:39.163+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.164757+0000) 2019-10-13T13:10:39.166 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:39.159+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.165025+0000) 2019-10-13T13:10:39.167 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:39.159+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.165354+0000) 2019-10-13T13:10:39.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:39.163+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.165784+0000) 2019-10-13T13:10:39.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:39.163+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.166332+0000) 2019-10-13T13:10:39.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:39.167+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:39.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:39.171+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:39.173049+0000) 2019-10-13T13:10:41.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:41.939+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:41.940024+0000) 2019-10-13T13:10:44.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:44.170+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.173909+0000) 2019-10-13T13:10:44.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:44.170+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.174476+0000) 2019-10-13T13:10:44.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:44.170+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.176590+0000) 2019-10-13T13:10:44.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:44.170+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.176777+0000) 2019-10-13T13:10:44.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:44.174+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.179008+0000) 2019-10-13T13:10:44.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:44.179+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.180161+0000) 2019-10-13T13:10:44.182 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:44.174+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.180227+0000) 2019-10-13T13:10:44.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:44.178+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:44.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:44.187+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.189339+0000) 2019-10-13T13:10:44.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:44.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.266223+0000) 2019-10-13T13:10:44.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:44.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.551214+0000) 2019-10-13T13:10:44.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:44.918+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:44.919300+0000) 2019-10-13T13:10:46.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:46.827+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:46.830204+0000) 2019-10-13T13:10:47.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:47.918+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:47.923313+0000) 2019-10-13T13:10:48.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:48.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:48.148167+0000) 2019-10-13T13:10:48.501 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:10:48.501 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:48.505 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:10:48.505 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:10:48.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:48.686+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:48.688786+0000) 2019-10-13T13:10:49.188 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:49.182+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.186472+0000) 2019-10-13T13:10:49.189 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:49.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.187451+0000) 2019-10-13T13:10:49.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:49.182+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.189081+0000) 2019-10-13T13:10:49.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:49.182+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.189466+0000) 2019-10-13T13:10:49.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:49.186+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.190070+0000) 2019-10-13T13:10:49.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:49.186+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.191588+0000) 2019-10-13T13:10:49.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:49.186+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.193034+0000) 2019-10-13T13:10:49.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:49.190+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:49.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:49.198+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:49.200914+0000) 2019-10-13T13:10:51.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:51.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:51.940260+0000) 2019-10-13T13:10:54.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:54.194+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.199734+0000) 2019-10-13T13:10:54.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:54.194+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.199823+0000) 2019-10-13T13:10:54.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:54.194+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.201109+0000) 2019-10-13T13:10:54.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:54.198+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.201867+0000) 2019-10-13T13:10:54.204 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:54.198+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.202592+0000) 2019-10-13T13:10:54.205 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:54.198+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.203484+0000) 2019-10-13T13:10:54.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:54.198+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.203959+0000) 2019-10-13T13:10:54.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:54.206+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.208162+0000) 2019-10-13T13:10:54.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:54.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176921 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:54.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:54.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.266405+0000) 2019-10-13T13:10:54.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:54.550+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.551418+0000) 2019-10-13T13:10:54.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:54.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:54.919474+0000) 2019-10-13T13:10:56.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:56.834+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:56.836843+0000) 2019-10-13T13:10:57.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:57.918+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:57.923511+0000) 2019-10-13T13:10:58.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:58.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:58.148395+0000) 2019-10-13T13:10:58.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:58.686+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:58.688986+0000) 2019-10-13T13:10:59.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:10:59.206+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.212234+0000) 2019-10-13T13:10:59.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:10:59.210+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.214001+0000) 2019-10-13T13:10:59.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:10:59.210+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.214740+0000) 2019-10-13T13:10:59.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:10:59.210+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.215073+0000) 2019-10-13T13:10:59.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:10:59.210+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.216554+0000) 2019-10-13T13:10:59.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:10:59.214+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.217454+0000) 2019-10-13T13:10:59.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:10:59.210+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.217721+0000) 2019-10-13T13:10:59.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:10:59.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:10:59.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:10:59.218+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:10:59.222553+0000) 2019-10-13T13:11:01.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:01.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:01.940458+0000) 2019-10-13T13:11:04.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:04.218+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.223474+0000) 2019-10-13T13:11:04.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:04.222+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.224595+0000) 2019-10-13T13:11:04.226 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:04.218+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.224093+0000) 2019-10-13T13:11:04.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:04.218+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.225178+0000) 2019-10-13T13:11:04.227 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:04.218+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.225485+0000) 2019-10-13T13:11:04.227 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:04.222+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.225862+0000) 2019-10-13T13:11:04.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:04.222+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.226502+0000) 2019-10-13T13:11:04.232 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:04.226+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.231214+0000) 2019-10-13T13:11:04.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:04.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:04.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:04.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.266563+0000) 2019-10-13T13:11:04.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:04.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.551588+0000) 2019-10-13T13:11:04.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:04.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:04.919654+0000) 2019-10-13T13:11:06.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:06.842+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:06.843357+0000) 2019-10-13T13:11:07.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:07.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:07.923704+0000) 2019-10-13T13:11:08.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:08.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:08.148589+0000) 2019-10-13T13:11:08.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:08.686+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:08.689194+0000) 2019-10-13T13:11:09.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:09.230+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.235384+0000) 2019-10-13T13:11:09.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:09.230+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.235776+0000) 2019-10-13T13:11:09.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:09.230+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.236479+0000) 2019-10-13T13:11:09.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:09.234+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.237081+0000) 2019-10-13T13:11:09.240 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:09.234+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.238367+0000) 2019-10-13T13:11:09.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:09.234+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.239010+0000) 2019-10-13T13:11:09.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:09.234+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.238863+0000) 2019-10-13T13:11:09.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:09.242+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:09.244651+0000) 2019-10-13T13:11:09.246 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:09.238+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177015 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:11.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:11.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:11.940654+0000) 2019-10-13T13:11:14.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:14.242+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.248722+0000) 2019-10-13T13:11:14.252 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:14.246+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.249902+0000) 2019-10-13T13:11:14.252 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:14.246+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.250028+0000) 2019-10-13T13:11:14.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:14.246+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.250872+0000) 2019-10-13T13:11:14.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:14.246+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.250879+0000) 2019-10-13T13:11:14.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:14.246+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.251324+0000) 2019-10-13T13:11:14.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:14.246+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.251591+0000) 2019-10-13T13:11:14.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:14.250+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.254928+0000) 2019-10-13T13:11:14.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:14.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177042 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:14.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:14.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.266676+0000) 2019-10-13T13:11:14.553 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:14.546+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.551769+0000) 2019-10-13T13:11:14.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:14.914+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:14.919829+0000) 2019-10-13T13:11:16.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:16.846+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:16.849421+0000) 2019-10-13T13:11:17.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:17.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:17.923918+0000) 2019-10-13T13:11:18.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:18.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:18.148795+0000) 2019-10-13T13:11:18.604 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:11:18.605 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:18.608 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:11:18.608 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:18.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:18.682+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:18.689392+0000) 2019-10-13T13:11:19.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:19.254+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.259999+0000) 2019-10-13T13:11:19.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:19.258+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.260973+0000) 2019-10-13T13:11:19.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:19.254+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.261040+0000) 2019-10-13T13:11:19.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:19.254+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.261322+0000) 2019-10-13T13:11:19.264 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:19.258+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.261885+0000) 2019-10-13T13:11:19.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:19.258+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.262579+0000) 2019-10-13T13:11:19.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:19.258+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.262660+0000) 2019-10-13T13:11:19.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:19.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:19.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:19.266+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:19.269801+0000) 2019-10-13T13:11:21.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:21.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:21.940849+0000) 2019-10-13T13:11:24.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:24.262+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.266835+0000) 2019-10-13T13:11:24.274 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:24.270+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.272490+0000) 2019-10-13T13:11:24.274 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:24.266+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.272717+0000) 2019-10-13T13:11:24.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:24.266+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.273633+0000) 2019-10-13T13:11:24.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:24.270+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.274082+0000) 2019-10-13T13:11:24.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:24.270+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.274565+0000) 2019-10-13T13:11:24.277 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:24.270+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.274851+0000) 2019-10-13T13:11:24.277 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:24.270+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.275282+0000) 2019-10-13T13:11:24.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:24.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:24.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:24.278+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.283730+0000) 2019-10-13T13:11:24.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:24.550+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.551944+0000) 2019-10-13T13:11:24.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:24.918+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:24.920035+0000) 2019-10-13T13:11:26.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:26.850+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:26.856008+0000) 2019-10-13T13:11:27.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:27.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:27.924132+0000) 2019-10-13T13:11:28.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:28.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:28.148994+0000) 2019-10-13T13:11:28.691 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:28.686+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:28.689591+0000) 2019-10-13T13:11:29.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:29.278+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.284560+0000) 2019-10-13T13:11:29.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:29.278+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.285226+0000) 2019-10-13T13:11:29.289 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:29.282+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.287882+0000) 2019-10-13T13:11:29.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:29.282+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.287984+0000) 2019-10-13T13:11:29.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:29.282+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.288301+0000) 2019-10-13T13:11:29.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:29.282+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.288593+0000) 2019-10-13T13:11:29.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:29.286+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.290327+0000) 2019-10-13T13:11:29.294 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:29.290+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:29.293282+0000) 2019-10-13T13:11:29.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:29.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177134 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:31.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:31.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:31.941069+0000) 2019-10-13T13:11:34.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:34.266+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.267033+0000) 2019-10-13T13:11:34.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:34.290+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.297478+0000) 2019-10-13T13:11:34.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:34.294+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.297873+0000) 2019-10-13T13:11:34.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:34.294+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.298660+0000) 2019-10-13T13:11:34.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:34.294+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.299179+0000) 2019-10-13T13:11:34.301 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:34.294+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.299457+0000) 2019-10-13T13:11:34.302 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:34.298+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.300140+0000) 2019-10-13T13:11:34.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:34.294+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.300047+0000) 2019-10-13T13:11:34.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:34.302+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.304183+0000) 2019-10-13T13:11:34.306 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:34.298+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177162 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:34.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:34.550+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.552138+0000) 2019-10-13T13:11:34.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:34.918+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:34.920205+0000) 2019-10-13T13:11:36.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:36.858+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:36.862488+0000) 2019-10-13T13:11:37.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:37.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:37.924345+0000) 2019-10-13T13:11:38.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:38.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:38.149228+0000) 2019-10-13T13:11:38.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:38.685+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:38.689785+0000) 2019-10-13T13:11:39.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:39.301+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.307316+0000) 2019-10-13T13:11:39.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:39.301+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.308977+0000) 2019-10-13T13:11:39.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:39.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.309207+0000) 2019-10-13T13:11:39.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:39.306+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.311317+0000) 2019-10-13T13:11:39.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:39.305+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.311477+0000) 2019-10-13T13:11:39.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:39.305+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.312115+0000) 2019-10-13T13:11:39.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:39.305+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.312890+0000) 2019-10-13T13:11:39.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:39.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:39.322 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:39.318+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:39.320440+0000) 2019-10-13T13:11:41.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:41.938+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:41.941288+0000) 2019-10-13T13:11:44.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:44.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.267229+0000) 2019-10-13T13:11:44.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:44.313+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.319691+0000) 2019-10-13T13:11:44.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:44.313+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.321612+0000) 2019-10-13T13:11:44.324 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:44.317+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.321883+0000) 2019-10-13T13:11:44.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:44.317+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.322348+0000) 2019-10-13T13:11:44.325 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:44.317+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.324067+0000) 2019-10-13T13:11:44.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:44.322+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.324650+0000) 2019-10-13T13:11:44.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:44.317+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.325026+0000) 2019-10-13T13:11:44.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:44.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:44.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:44.326+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.331269+0000) 2019-10-13T13:11:44.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:44.545+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.552347+0000) 2019-10-13T13:11:44.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:44.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:44.920406+0000) 2019-10-13T13:11:46.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:46.866+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:46.868372+0000) 2019-10-13T13:11:47.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:47.921+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:47.924549+0000) 2019-10-13T13:11:48.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:48.141+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:48.149421+0000) 2019-10-13T13:11:48.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:48.689+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:48.689947+0000) 2019-10-13T13:11:48.710 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:11:48.710 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:48.714 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:11:48.714 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:11:49.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:49.325+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.331151+0000) 2019-10-13T13:11:49.334 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:49.325+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.332655+0000) 2019-10-13T13:11:49.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:49.329+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.334143+0000) 2019-10-13T13:11:49.336 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:49.329+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.334976+0000) 2019-10-13T13:11:49.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:49.329+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.335604+0000) 2019-10-13T13:11:49.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:49.329+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.335870+0000) 2019-10-13T13:11:49.338 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:49.334+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.336699+0000) 2019-10-13T13:11:49.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:49.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:49.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:49.338+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:49.341042+0000) 2019-10-13T13:11:51.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:51.937+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:51.941481+0000) 2019-10-13T13:11:54.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:54.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.267449+0000) 2019-10-13T13:11:54.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:54.341+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.342990+0000) 2019-10-13T13:11:54.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:54.345+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.345887+0000) 2019-10-13T13:11:54.348 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:54.345+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.346128+0000) 2019-10-13T13:11:54.349 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:54.341+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.347851+0000) 2019-10-13T13:11:54.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:54.345+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.347853+0000) 2019-10-13T13:11:54.350 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:54.345+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.348716+0000) 2019-10-13T13:11:54.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:54.345+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.349146+0000) 2019-10-13T13:11:54.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:54.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:54.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:54.349+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.353934+0000) 2019-10-13T13:11:54.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:54.549+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.552554+0000) 2019-10-13T13:11:54.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:54.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:54.920605+0000) 2019-10-13T13:11:56.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:56.869+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:56.874724+0000) 2019-10-13T13:11:57.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:57.921+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:57.924740+0000) 2019-10-13T13:11:58.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:58.145+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:58.149619+0000) 2019-10-13T13:11:58.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:58.689+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:58.690155+0000) 2019-10-13T13:11:59.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:11:59.353+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.356037+0000) 2019-10-13T13:11:59.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:11:59.353+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.356409+0000) 2019-10-13T13:11:59.359 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:11:59.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.356870+0000) 2019-10-13T13:11:59.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:11:59.353+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.357700+0000) 2019-10-13T13:11:59.360 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:11:59.357+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.358006+0000) 2019-10-13T13:11:59.360 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:11:59.357+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.358974+0000) 2019-10-13T13:11:59.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:11:59.357+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.359778+0000) 2019-10-13T13:11:59.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:11:59.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:11:59.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:11:59.361+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:11:59.366208+0000) 2019-10-13T13:12:01.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:01.937+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:01.941687+0000) 2019-10-13T13:12:04.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:04.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.267683+0000) 2019-10-13T13:12:04.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:04.365+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.368728+0000) 2019-10-13T13:12:04.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:04.365+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.369212+0000) 2019-10-13T13:12:04.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:04.369+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.370424+0000) 2019-10-13T13:12:04.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:04.369+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.370700+0000) 2019-10-13T13:12:04.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:04.369+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.371810+0000) 2019-10-13T13:12:04.374 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:04.369+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.372607+0000) 2019-10-13T13:12:04.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:04.369+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.372897+0000) 2019-10-13T13:12:04.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:04.373+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:04.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:04.377+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.382101+0000) 2019-10-13T13:12:04.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:04.549+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.552730+0000) 2019-10-13T13:12:04.922 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:04.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:04.920769+0000) 2019-10-13T13:12:06.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:06.881+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:06.883884+0000) 2019-10-13T13:12:07.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:07.921+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:07.924973+0000) 2019-10-13T13:12:08.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:08.149+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:08.149835+0000) 2019-10-13T13:12:08.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:08.689+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:08.690379+0000) 2019-10-13T13:12:09.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:09.377+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.380910+0000) 2019-10-13T13:12:09.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:09.377+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.381136+0000) 2019-10-13T13:12:09.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:09.381+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.383265+0000) 2019-10-13T13:12:09.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:09.377+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.383824+0000) 2019-10-13T13:12:09.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:09.385+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.387937+0000) 2019-10-13T13:12:09.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:09.385+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.389552+0000) 2019-10-13T13:12:09.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:09.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:09.392 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:09.389+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.390534+0000) 2019-10-13T13:12:09.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:09.385+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:09.391741+0000) 2019-10-13T13:12:11.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:11.937+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:11.941906+0000) 2019-10-13T13:12:14.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:14.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.267893+0000) 2019-10-13T13:12:14.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:14.393+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.393876+0000) 2019-10-13T13:12:14.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:14.393+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.395055+0000) 2019-10-13T13:12:14.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:14.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.395255+0000) 2019-10-13T13:12:14.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:14.389+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.396006+0000) 2019-10-13T13:12:14.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:14.393+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.395385+0000) 2019-10-13T13:12:14.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:14.393+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.396923+0000) 2019-10-13T13:12:14.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:14.397+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.397834+0000) 2019-10-13T13:12:14.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:14.397+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.401131+0000) 2019-10-13T13:12:14.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:14.401+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:14.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:14.549+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.552917+0000) 2019-10-13T13:12:14.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:14.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:14.920993+0000) 2019-10-13T13:12:16.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:16.889+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:16.890422+0000) 2019-10-13T13:12:17.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:17.921+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:17.925169+0000) 2019-10-13T13:12:18.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:18.149+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:18.150035+0000) 2019-10-13T13:12:18.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:18.689+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:18.690592+0000) 2019-10-13T13:12:18.837 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:12:18.837 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:18.841 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:12:18.841 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:19.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:19.405+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.405874+0000) 2019-10-13T13:12:19.409 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:19.405+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.407438+0000) 2019-10-13T13:12:19.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:19.409+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.409791+0000) 2019-10-13T13:12:19.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:19.409+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.410006+0000) 2019-10-13T13:12:19.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:19.409+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.412214+0000) 2019-10-13T13:12:19.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:19.409+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.414445+0000) 2019-10-13T13:12:19.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:19.413+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.414288+0000) 2019-10-13T13:12:19.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:19.409+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:19.415250+0000) 2019-10-13T13:12:19.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:19.413+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:21.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:21.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:21.942097+0000) 2019-10-13T13:12:24.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:24.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.268111+0000) 2019-10-13T13:12:24.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:24.417+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.418921+0000) 2019-10-13T13:12:24.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:24.417+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.419699+0000) 2019-10-13T13:12:24.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:24.417+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.420647+0000) 2019-10-13T13:12:24.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:24.417+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.421001+0000) 2019-10-13T13:12:24.423 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:24.417+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.421331+0000) 2019-10-13T13:12:24.424 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:24.417+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.422820+0000) 2019-10-13T13:12:24.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:24.421+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.422228+0000) 2019-10-13T13:12:24.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:24.421+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.426713+0000) 2019-10-13T13:12:24.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:24.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:24.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:24.549+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.553092+0000) 2019-10-13T13:12:24.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:24.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:24.921173+0000) 2019-10-13T13:12:26.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:26.893+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:26.897176+0000) 2019-10-13T13:12:27.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:27.925+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:27.925366+0000) 2019-10-13T13:12:28.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:28.149+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:28.150262+0000) 2019-10-13T13:12:28.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:28.689+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:28.690784+0000) 2019-10-13T13:12:29.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:29.429+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.431739+0000) 2019-10-13T13:12:29.434 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:29.429+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.432034+0000) 2019-10-13T13:12:29.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:29.429+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.432894+0000) 2019-10-13T13:12:29.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:29.433+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.434018+0000) 2019-10-13T13:12:29.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:29.433+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.434753+0000) 2019-10-13T13:12:29.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:29.433+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.435269+0000) 2019-10-13T13:12:29.437 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:29.429+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.436034+0000) 2019-10-13T13:12:29.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:29.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:29.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:29.437+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:29.441723+0000) 2019-10-13T13:12:31.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:31.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:31.942299+0000) 2019-10-13T13:12:34.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:34.264+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.268341+0000) 2019-10-13T13:12:34.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:34.440+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.443813+0000) 2019-10-13T13:12:34.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:34.440+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.444127+0000) 2019-10-13T13:12:34.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:34.440+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.445502+0000) 2019-10-13T13:12:34.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:34.440+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.445638+0000) 2019-10-13T13:12:34.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:34.444+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.446038+0000) 2019-10-13T13:12:34.451 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:34.448+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.449962+0000) 2019-10-13T13:12:34.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:34.445+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.450617+0000) 2019-10-13T13:12:34.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:34.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177521 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:34.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:34.449+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.455510+0000) 2019-10-13T13:12:34.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:34.552+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.553272+0000) 2019-10-13T13:12:34.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:34.916+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:34.921343+0000) 2019-10-13T13:12:36.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:36.901+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:36.903720+0000) 2019-10-13T13:12:37.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:37.920+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:37.925558+0000) 2019-10-13T13:12:38.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:38.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:38.150489+0000) 2019-10-13T13:12:38.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:38.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:38.691011+0000) 2019-10-13T13:12:39.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:39.452+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.455765+0000) 2019-10-13T13:12:39.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:39.452+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.455412+0000) 2019-10-13T13:12:39.460 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:39.456+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.458171+0000) 2019-10-13T13:12:39.460 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:39.456+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.458606+0000) 2019-10-13T13:12:39.461 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:39.453+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.459501+0000) 2019-10-13T13:12:39.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:39.456+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.459579+0000) 2019-10-13T13:12:39.462 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:39.456+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.460126+0000) 2019-10-13T13:12:39.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:39.460+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:39.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:39.465+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:39.468644+0000) 2019-10-13T13:12:41.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:41.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:41.942482+0000) 2019-10-13T13:12:44.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:44.264+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.268546+0000) 2019-10-13T13:12:44.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:44.464+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.467605+0000) 2019-10-13T13:12:44.470 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:44.464+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.467940+0000) 2019-10-13T13:12:44.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:44.465+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.469704+0000) 2019-10-13T13:12:44.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:44.464+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.469583+0000) 2019-10-13T13:12:44.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:44.468+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.469987+0000) 2019-10-13T13:12:44.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:44.468+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.470546+0000) 2019-10-13T13:12:44.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:44.468+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.470904+0000) 2019-10-13T13:12:44.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:44.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:44.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:44.469+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.475389+0000) 2019-10-13T13:12:44.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:44.552+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.553462+0000) 2019-10-13T13:12:44.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:44.920+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:44.921555+0000) 2019-10-13T13:12:46.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:46.909+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:46.910253+0000) 2019-10-13T13:12:47.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:47.924+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:47.925771+0000) 2019-10-13T13:12:48.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:48.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:48.150725+0000) 2019-10-13T13:12:48.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:48.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:48.691201+0000) 2019-10-13T13:12:49.161 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:12:49.161 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:49.165 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:12:49.165 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:12:49.482 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:49.477+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.481042+0000) 2019-10-13T13:12:49.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:49.476+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.481209+0000) 2019-10-13T13:12:49.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:49.476+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.481492+0000) 2019-10-13T13:12:49.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:49.480+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.482482+0000) 2019-10-13T13:12:49.484 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:49.480+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.481931+0000) 2019-10-13T13:12:49.485 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:49.480+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.483351+0000) 2019-10-13T13:12:49.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:49.480+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.483663+0000) 2019-10-13T13:12:49.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:49.481+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:49.486987+0000) 2019-10-13T13:12:49.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:49.484+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177614 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:51.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:51.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:51.942673+0000) 2019-10-13T13:12:54.270 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:54.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.268809+0000) 2019-10-13T13:12:54.493 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:54.488+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.491396+0000) 2019-10-13T13:12:54.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:54.488+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.491676+0000) 2019-10-13T13:12:54.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:54.489+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.492265+0000) 2019-10-13T13:12:54.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:54.488+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.492794+0000) 2019-10-13T13:12:54.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:54.488+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.493502+0000) 2019-10-13T13:12:54.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:54.492+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.494255+0000) 2019-10-13T13:12:54.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:54.492+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.494847+0000) 2019-10-13T13:12:54.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:54.496+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:54.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:54.497+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.501976+0000) 2019-10-13T13:12:54.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:54.548+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.553658+0000) 2019-10-13T13:12:54.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:54.920+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:54.921730+0000) 2019-10-13T13:12:56.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:56.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:56.916870+0000) 2019-10-13T13:12:57.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:57.924+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:57.925974+0000) 2019-10-13T13:12:58.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:58.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:58.150922+0000) 2019-10-13T13:12:58.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:58.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:58.691393+0000) 2019-10-13T13:12:59.506 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:12:59.500+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.504093+0000) 2019-10-13T13:12:59.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:12:59.500+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.504956+0000) 2019-10-13T13:12:59.507 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:12:59.500+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.505536+0000) 2019-10-13T13:12:59.508 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:12:59.504+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.505890+0000) 2019-10-13T13:12:59.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:12:59.504+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.506137+0000) 2019-10-13T13:12:59.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:12:59.504+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.506881+0000) 2019-10-13T13:12:59.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:12:59.504+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.507169+0000) 2019-10-13T13:12:59.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:12:59.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:12:59.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:12:59.508+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:12:59.513257+0000) 2019-10-13T13:13:01.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:01.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:01.942874+0000) 2019-10-13T13:13:04.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:04.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.268973+0000) 2019-10-13T13:13:04.519 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:04.512+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.517306+0000) 2019-10-13T13:13:04.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:04.516+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.519768+0000) 2019-10-13T13:13:04.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:04.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.520470+0000) 2019-10-13T13:13:04.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:04.516+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.520950+0000) 2019-10-13T13:13:04.523 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:04.516+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.521184+0000) 2019-10-13T13:13:04.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:04.520+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.521928+0000) 2019-10-13T13:13:04.524 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:04.520+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.522475+0000) 2019-10-13T13:13:04.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:04.520+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.524789+0000) 2019-10-13T13:13:04.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:04.524+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:04.556 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:04.552+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.553867+0000) 2019-10-13T13:13:04.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:04.920+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:04.921937+0000) 2019-10-13T13:13:06.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:06.920+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:06.923102+0000) 2019-10-13T13:13:07.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:07.924+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:07.926174+0000) 2019-10-13T13:13:08.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:08.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:08.151124+0000) 2019-10-13T13:13:08.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:08.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:08.691592+0000) 2019-10-13T13:13:09.531 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:09.524+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.529091+0000) 2019-10-13T13:13:09.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:09.528+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.529850+0000) 2019-10-13T13:13:09.532 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:09.528+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.530314+0000) 2019-10-13T13:13:09.532 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:09.528+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.530796+0000) 2019-10-13T13:13:09.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:09.524+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.531468+0000) 2019-10-13T13:13:09.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:09.528+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.531421+0000) 2019-10-13T13:13:09.534 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:09.528+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.532741+0000) 2019-10-13T13:13:09.541 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:09.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177734 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:09.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:09.532+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:09.539668+0000) 2019-10-13T13:13:11.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:11.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:11.943071+0000) 2019-10-13T13:13:14.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:14.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.269207+0000) 2019-10-13T13:13:14.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:14.540+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.542545+0000) 2019-10-13T13:13:14.545 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:14.540+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.543070+0000) 2019-10-13T13:13:14.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:14.540+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.543392+0000) 2019-10-13T13:13:14.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:14.540+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.544066+0000) 2019-10-13T13:13:14.546 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:14.540+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.544679+0000) 2019-10-13T13:13:14.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:14.540+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.545145+0000) 2019-10-13T13:13:14.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:14.540+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.547963+0000) 2019-10-13T13:13:14.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:14.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177761 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:14.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:14.548+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.554536+0000) 2019-10-13T13:13:14.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:14.552+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.555111+0000) 2019-10-13T13:13:14.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:14.920+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:14.922111+0000) 2019-10-13T13:13:16.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:16.928+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:16.929565+0000) 2019-10-13T13:13:17.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:17.924+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:17.926365+0000) 2019-10-13T13:13:18.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:18.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:18.151357+0000) 2019-10-13T13:13:18.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:18.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:18.691819+0000) 2019-10-13T13:13:19.285 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:13:19.285 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:19.288 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:13:19.288 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:19.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:19.552+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.554827+0000) 2019-10-13T13:13:19.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:19.552+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.555552+0000) 2019-10-13T13:13:19.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:19.552+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.555304+0000) 2019-10-13T13:13:19.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:19.552+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.555901+0000) 2019-10-13T13:13:19.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:19.556+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.557989+0000) 2019-10-13T13:13:19.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:19.560+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.562605+0000) 2019-10-13T13:13:19.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:19.556+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.560434+0000) 2019-10-13T13:13:19.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:19.556+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:19.562783+0000) 2019-10-13T13:13:19.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:19.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:21.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:21.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:21.943301+0000) 2019-10-13T13:13:24.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:24.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.269396+0000) 2019-10-13T13:13:24.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:24.552+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.555299+0000) 2019-10-13T13:13:24.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:24.564+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.567751+0000) 2019-10-13T13:13:24.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:24.564+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.568363+0000) 2019-10-13T13:13:24.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:24.564+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.569074+0000) 2019-10-13T13:13:24.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:24.564+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.569545+0000) 2019-10-13T13:13:24.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:24.568+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.570033+0000) 2019-10-13T13:13:24.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:24.564+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.571046+0000) 2019-10-13T13:13:24.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:24.568+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.570950+0000) 2019-10-13T13:13:24.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:24.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:24.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:24.572+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.576331+0000) 2019-10-13T13:13:24.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:24.920+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:24.922247+0000) 2019-10-13T13:13:26.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:26.928+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:26.936015+0000) 2019-10-13T13:13:27.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:27.924+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:27.926564+0000) 2019-10-13T13:13:28.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:28.148+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:28.151536+0000) 2019-10-13T13:13:28.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:28.688+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:28.692042+0000) 2019-10-13T13:13:29.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:29.576+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.580064+0000) 2019-10-13T13:13:29.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:29.576+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.580665+0000) 2019-10-13T13:13:29.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:29.576+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.583921+0000) 2019-10-13T13:13:29.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:29.580+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.587700+0000) 2019-10-13T13:13:29.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:29.576+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.581648+0000) 2019-10-13T13:13:29.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:29.576+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.581666+0000) 2019-10-13T13:13:29.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:29.580+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.583433+0000) 2019-10-13T13:13:29.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:29.580+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:29.584986+0000) 2019-10-13T13:13:29.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:29.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177855 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:31.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:31.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:31.943519+0000) 2019-10-13T13:13:34.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:34.263+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.269591+0000) 2019-10-13T13:13:34.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:34.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.555501+0000) 2019-10-13T13:13:34.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:34.587+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.593424+0000) 2019-10-13T13:13:34.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:34.591+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.593715+0000) 2019-10-13T13:13:34.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:34.588+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.594555+0000) 2019-10-13T13:13:34.596 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:34.591+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.594738+0000) 2019-10-13T13:13:34.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:34.591+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.594970+0000) 2019-10-13T13:13:34.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:34.591+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.596464+0000) 2019-10-13T13:13:34.598 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:34.591+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.597112+0000) 2019-10-13T13:13:34.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:34.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:34.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:34.596+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.602178+0000) 2019-10-13T13:13:34.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:34.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:34.922466+0000) 2019-10-13T13:13:36.944 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:36.940+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:36.942398+0000) 2019-10-13T13:13:37.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:37.923+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:37.926792+0000) 2019-10-13T13:13:38.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:38.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:38.151734+0000) 2019-10-13T13:13:38.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:38.691+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:38.692237+0000) 2019-10-13T13:13:39.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:39.599+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.604078+0000) 2019-10-13T13:13:39.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:39.600+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.605344+0000) 2019-10-13T13:13:39.607 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:39.599+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.605513+0000) 2019-10-13T13:13:39.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:39.603+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.606227+0000) 2019-10-13T13:13:39.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:39.603+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.606830+0000) 2019-10-13T13:13:39.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:39.603+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.606773+0000) 2019-10-13T13:13:39.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:39.603+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.608311+0000) 2019-10-13T13:13:39.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:39.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:39.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:39.608+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:39.615687+0000) 2019-10-13T13:13:41.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:41.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:41.943677+0000) 2019-10-13T13:13:44.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:44.267+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.269817+0000) 2019-10-13T13:13:44.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:44.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.555703+0000) 2019-10-13T13:13:44.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:44.611+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.616483+0000) 2019-10-13T13:13:44.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:44.611+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.617246+0000) 2019-10-13T13:13:44.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:44.615+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.617736+0000) 2019-10-13T13:13:44.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:44.615+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.618139+0000) 2019-10-13T13:13:44.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:44.612+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.619550+0000) 2019-10-13T13:13:44.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:44.615+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.619666+0000) 2019-10-13T13:13:44.622 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:44.615+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.620008+0000) 2019-10-13T13:13:44.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:44.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:44.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:44.620+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.627119+0000) 2019-10-13T13:13:44.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:44.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:44.922647+0000) 2019-10-13T13:13:46.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:46.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:46.953565+0000) 2019-10-13T13:13:47.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:47.923+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:47.927005+0000) 2019-10-13T13:13:48.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:48.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:48.151967+0000) 2019-10-13T13:13:48.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:48.687+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:48.692436+0000) 2019-10-13T13:13:49.406 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:13:49.407 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:49.411 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:13:49.411 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:13:49.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:49.623+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.628925+0000) 2019-10-13T13:13:49.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:49.627+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.630349+0000) 2019-10-13T13:13:49.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:49.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.631454+0000) 2019-10-13T13:13:49.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:49.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.631535+0000) 2019-10-13T13:13:49.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:49.627+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.631677+0000) 2019-10-13T13:13:49.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:49.627+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.632056+0000) 2019-10-13T13:13:49.635 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:49.627+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.632154+0000) 2019-10-13T13:13:49.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:49.632+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:49.637352+0000) 2019-10-13T13:13:49.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:49.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:51.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:51.940+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:51.943891+0000) 2019-10-13T13:13:54.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:54.267+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.270019+0000) 2019-10-13T13:13:54.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:54.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.555901+0000) 2019-10-13T13:13:54.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:54.635+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.640802+0000) 2019-10-13T13:13:54.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:54.635+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.641475+0000) 2019-10-13T13:13:54.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:54.639+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.642765+0000) 2019-10-13T13:13:54.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:54.639+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.643138+0000) 2019-10-13T13:13:54.645 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:54.636+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.643814+0000) 2019-10-13T13:13:54.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:54.639+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.643813+0000) 2019-10-13T13:13:54.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:54.639+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.644623+0000) 2019-10-13T13:13:54.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:54.644+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.648867+0000) 2019-10-13T13:13:54.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:54.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178004 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:54.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:54.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:54.922842+0000) 2019-10-13T13:13:56.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:56.960+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:56.960630+0000) 2019-10-13T13:13:57.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:57.923+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:57.927195+0000) 2019-10-13T13:13:58.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:58.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:58.152165+0000) 2019-10-13T13:13:58.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:58.691+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:58.692658+0000) 2019-10-13T13:13:59.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:13:59.651+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.654866+0000) 2019-10-13T13:13:59.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:13:59.651+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.656242+0000) 2019-10-13T13:13:59.659 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:13:59.651+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.656934+0000) 2019-10-13T13:13:59.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:13:59.651+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.657304+0000) 2019-10-13T13:13:59.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:13:59.651+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.657620+0000) 2019-10-13T13:13:59.660 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:13:59.655+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.658236+0000) 2019-10-13T13:13:59.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:13:59.656+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.659128+0000) 2019-10-13T13:13:59.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:13:59.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:13:59.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:13:59.664+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:13:59.664795+0000) 2019-10-13T13:14:01.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:01.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:01.944127+0000) 2019-10-13T13:14:04.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:04.267+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.270281+0000) 2019-10-13T13:14:04.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:04.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.556098+0000) 2019-10-13T13:14:04.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:04.663+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.666910+0000) 2019-10-13T13:14:04.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:04.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.667520+0000) 2019-10-13T13:14:04.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:04.663+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.667792+0000) 2019-10-13T13:14:04.670 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:04.663+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.668175+0000) 2019-10-13T13:14:04.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:04.663+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.668496+0000) 2019-10-13T13:14:04.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:04.663+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.669259+0000) 2019-10-13T13:14:04.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:04.667+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.670355+0000) 2019-10-13T13:14:04.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:04.671+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.673318+0000) 2019-10-13T13:14:04.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:04.671+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:04.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:04.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:04.923058+0000) 2019-10-13T13:14:06.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:06.963+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:06.967718+0000) 2019-10-13T13:14:07.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:07.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:07.927418+0000) 2019-10-13T13:14:08.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:08.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:08.152373+0000) 2019-10-13T13:14:08.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:08.691+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:08.692853+0000) 2019-10-13T13:14:09.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:09.675+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.678922+0000) 2019-10-13T13:14:09.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:09.675+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.679345+0000) 2019-10-13T13:14:09.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:09.675+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.680054+0000) 2019-10-13T13:14:09.684 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:09.679+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.682358+0000) 2019-10-13T13:14:09.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:09.679+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.685075+0000) 2019-10-13T13:14:09.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:09.683+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.687483+0000) 2019-10-13T13:14:09.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:09.683+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178099 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:09.691 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:09.683+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.689043+0000) 2019-10-13T13:14:09.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:09.691+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:09.692704+0000) 2019-10-13T13:14:11.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:11.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:11.944317+0000) 2019-10-13T13:14:14.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:14.267+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.270499+0000) 2019-10-13T13:14:14.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:14.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.556298+0000) 2019-10-13T13:14:14.693 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:14.687+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.691840+0000) 2019-10-13T13:14:14.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:14.691+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.694217+0000) 2019-10-13T13:14:14.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:14.691+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.694771+0000) 2019-10-13T13:14:14.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:14.691+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.695009+0000) 2019-10-13T13:14:14.699 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:14.691+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.697163+0000) 2019-10-13T13:14:14.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:14.691+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.697261+0000) 2019-10-13T13:14:14.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:14.695+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.700224+0000) 2019-10-13T13:14:14.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:14.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178126 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:14.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:14.703+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.706042+0000) 2019-10-13T13:14:14.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:14.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:14.923237+0000) 2019-10-13T13:14:16.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:16.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:16.974803+0000) 2019-10-13T13:14:17.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:17.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:17.927646+0000) 2019-10-13T13:14:18.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:18.147+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.152598+0000) 2019-10-13T13:14:18.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:18.687+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:18.693048+0000) 2019-10-13T13:14:19.495 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:14:19.495 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:19.499 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:14:19.499 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:19.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:19.699+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.705102+0000) 2019-10-13T13:14:19.708 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:19.703+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.706244+0000) 2019-10-13T13:14:19.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:19.703+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.706923+0000) 2019-10-13T13:14:19.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:19.703+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.707774+0000) 2019-10-13T13:14:19.710 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:19.703+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.708332+0000) 2019-10-13T13:14:19.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:19.703+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.708503+0000) 2019-10-13T13:14:19.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:19.707+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.709212+0000) 2019-10-13T13:14:19.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:19.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178158 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:19.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:19.711+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:19.714596+0000) 2019-10-13T13:14:21.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:21.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:21.944517+0000) 2019-10-13T13:14:24.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:24.267+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.270715+0000) 2019-10-13T13:14:24.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:24.551+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.556460+0000) 2019-10-13T13:14:24.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:24.711+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.717035+0000) 2019-10-13T13:14:24.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:24.715+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.718133+0000) 2019-10-13T13:14:24.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:24.715+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.719333+0000) 2019-10-13T13:14:24.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:24.715+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.718733+0000) 2019-10-13T13:14:24.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:24.715+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.719522+0000) 2019-10-13T13:14:24.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:24.715+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.719770+0000) 2019-10-13T13:14:24.723 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:24.715+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.720290+0000) 2019-10-13T13:14:24.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:24.719+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:24.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:24.723+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.725064+0000) 2019-10-13T13:14:24.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:24.919+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:24.923451+0000) 2019-10-13T13:14:26.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:26.979+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:26.981188+0000) 2019-10-13T13:14:27.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:27.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:27.927874+0000) 2019-10-13T13:14:28.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:28.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.152795+0000) 2019-10-13T13:14:28.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:28.690+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:28.693259+0000) 2019-10-13T13:14:29.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:29.722+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.729322+0000) 2019-10-13T13:14:29.732 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:29.726+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.729991+0000) 2019-10-13T13:14:29.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:29.727+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.730689+0000) 2019-10-13T13:14:29.733 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:29.726+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.731109+0000) 2019-10-13T13:14:29.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:29.726+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.731133+0000) 2019-10-13T13:14:29.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:29.726+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.733594+0000) 2019-10-13T13:14:29.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:29.730+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.736366+0000) 2019-10-13T13:14:29.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:29.735+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:29.736819+0000) 2019-10-13T13:14:29.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:29.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:31.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:31.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:31.944712+0000) 2019-10-13T13:14:34.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:34.270+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.270919+0000) 2019-10-13T13:14:34.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:34.554+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.556666+0000) 2019-10-13T13:14:34.742 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:34.734+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.740600+0000) 2019-10-13T13:14:34.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:34.738+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.742155+0000) 2019-10-13T13:14:34.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:34.739+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.742860+0000) 2019-10-13T13:14:34.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:34.738+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.743314+0000) 2019-10-13T13:14:34.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:34.738+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.742980+0000) 2019-10-13T13:14:34.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:34.738+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.743729+0000) 2019-10-13T13:14:34.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:34.738+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.744411+0000) 2019-10-13T13:14:34.750 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:34.742+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:34.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:34.751+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.753287+0000) 2019-10-13T13:14:34.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:34.922+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:34.923633+0000) 2019-10-13T13:14:36.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:36.987+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:36.988361+0000) 2019-10-13T13:14:37.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:37.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:37.928041+0000) 2019-10-13T13:14:38.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:38.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.152997+0000) 2019-10-13T13:14:38.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:38.690+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:38.693445+0000) 2019-10-13T13:14:39.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:39.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.752269+0000) 2019-10-13T13:14:39.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:39.746+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.752471+0000) 2019-10-13T13:14:39.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:39.746+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.753099+0000) 2019-10-13T13:14:39.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:39.750+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.755363+0000) 2019-10-13T13:14:39.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:39.750+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.756016+0000) 2019-10-13T13:14:39.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:39.755+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.756559+0000) 2019-10-13T13:14:39.759 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:39.750+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.757003+0000) 2019-10-13T13:14:39.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:39.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:39.764 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:39.759+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:39.762545+0000) 2019-10-13T13:14:41.946 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:41.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:41.944926+0000) 2019-10-13T13:14:44.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:44.266+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.271146+0000) 2019-10-13T13:14:44.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:44.554+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.556885+0000) 2019-10-13T13:14:44.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:44.758+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.763541+0000) 2019-10-13T13:14:44.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:44.758+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.764837+0000) 2019-10-13T13:14:44.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:44.758+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.765267+0000) 2019-10-13T13:14:44.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:44.762+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.765734+0000) 2019-10-13T13:14:44.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:44.763+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.766416+0000) 2019-10-13T13:14:44.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:44.762+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.766588+0000) 2019-10-13T13:14:44.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:44.762+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.767880+0000) 2019-10-13T13:14:44.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:44.767+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.770282+0000) 2019-10-13T13:14:44.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:44.766+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178308 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:44.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:44.918+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:44.923811+0000) 2019-10-13T13:14:46.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:46.995+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:46.995523+0000) 2019-10-13T13:14:47.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:47.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:47.928254+0000) 2019-10-13T13:14:48.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:48.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.153149+0000) 2019-10-13T13:14:48.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:48.690+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:48.693643+0000) 2019-10-13T13:14:49.600 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:14:49.600 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:49.604 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:14:49.604 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:14:49.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:49.770+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.776035+0000) 2019-10-13T13:14:49.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:49.770+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.776707+0000) 2019-10-13T13:14:49.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:49.770+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.777349+0000) 2019-10-13T13:14:49.780 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:49.770+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.777691+0000) 2019-10-13T13:14:49.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:49.775+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.779375+0000) 2019-10-13T13:14:49.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:49.774+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.779445+0000) 2019-10-13T13:14:49.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:49.774+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.779482+0000) 2019-10-13T13:14:49.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:49.778+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:49.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:49.783+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:49.787155+0000) 2019-10-13T13:14:51.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:51.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:51.945162+0000) 2019-10-13T13:14:54.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:54.270+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.271335+0000) 2019-10-13T13:14:54.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:54.550+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.557083+0000) 2019-10-13T13:14:54.790 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:54.782+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.788766+0000) 2019-10-13T13:14:54.791 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:54.782+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.789335+0000) 2019-10-13T13:14:54.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:54.786+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.792202+0000) 2019-10-13T13:14:54.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:54.790+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.794300+0000) 2019-10-13T13:14:54.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:54.795+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.796591+0000) 2019-10-13T13:14:54.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:54.795+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.797566+0000) 2019-10-13T13:14:54.799 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:54.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:14:54.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:54.794+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.797983+0000) 2019-10-13T13:14:54.800 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:54.794+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.798395+0000) 2019-10-13T13:14:54.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:54.918+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:54.923984+0000) 2019-10-13T13:14:57.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:56.999+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:57.002717+0000) 2019-10-13T13:14:57.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:57.922+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:57.928448+0000) 2019-10-13T13:14:58.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:58.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.153377+0000) 2019-10-13T13:14:58.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:58.690+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:58.693840+0000) 2019-10-13T13:14:59.803 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:14:59.794+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.801487+0000) 2019-10-13T13:14:59.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:14:59.799+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.803555+0000) 2019-10-13T13:14:59.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:14:59.798+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.803493+0000) 2019-10-13T13:14:59.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:14:59.798+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.803151+0000) 2019-10-13T13:14:59.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:14:59.798+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.804215+0000) 2019-10-13T13:14:59.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:14:59.798+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.803924+0000) 2019-10-13T13:14:59.807 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:14:59.798+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.804621+0000) 2019-10-13T13:14:59.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:14:59.803+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:14:59.806909+0000) 2019-10-13T13:14:59.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:14:59.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:01.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:01.943+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:01.945398+0000) 2019-10-13T13:15:04.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:04.266+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.271529+0000) 2019-10-13T13:15:04.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:04.554+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.557287+0000) 2019-10-13T13:15:04.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:04.806+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.813182+0000) 2019-10-13T13:15:04.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:04.810+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.814486+0000) 2019-10-13T13:15:04.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:04.810+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.814826+0000) 2019-10-13T13:15:04.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:04.811+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.815587+0000) 2019-10-13T13:15:04.817 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:04.810+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.815855+0000) 2019-10-13T13:15:04.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:04.810+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.816483+0000) 2019-10-13T13:15:04.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:04.810+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.817120+0000) 2019-10-13T13:15:04.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:04.819+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.822760+0000) 2019-10-13T13:15:04.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:04.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:04.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:04.922+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:04.924173+0000) 2019-10-13T13:15:07.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:07.006+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:07.009826+0000) 2019-10-13T13:15:07.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:07.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:07.928649+0000) 2019-10-13T13:15:08.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:08.146+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.153597+0000) 2019-10-13T13:15:08.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:08.690+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:08.694032+0000) 2019-10-13T13:15:09.828 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:09.822+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.826449+0000) 2019-10-13T13:15:09.829 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:09.822+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.827063+0000) 2019-10-13T13:15:09.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:09.822+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.827536+0000) 2019-10-13T13:15:09.830 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:09.826+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.828779+0000) 2019-10-13T13:15:09.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:09.822+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.829034+0000) 2019-10-13T13:15:09.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:09.822+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.829265+0000) 2019-10-13T13:15:09.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:09.826+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.829992+0000) 2019-10-13T13:15:09.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:09.830+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:09.835371+0000) 2019-10-13T13:15:09.838 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:09.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:11.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:11.942+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:11.945599+0000) 2019-10-13T13:15:14.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:14.266+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.271741+0000) 2019-10-13T13:15:14.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:14.554+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.557493+0000) 2019-10-13T13:15:14.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:14.834+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.838797+0000) 2019-10-13T13:15:14.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:14.834+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.839651+0000) 2019-10-13T13:15:14.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:14.838+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.840222+0000) 2019-10-13T13:15:14.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:14.834+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.841291+0000) 2019-10-13T13:15:14.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:14.838+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.841729+0000) 2019-10-13T13:15:14.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:14.838+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.841799+0000) 2019-10-13T13:15:14.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:14.838+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.842001+0000) 2019-10-13T13:15:14.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:14.842+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.845909+0000) 2019-10-13T13:15:14.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:14.842+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:14.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:14.922+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:14.924367+0000) 2019-10-13T13:15:17.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:17.014+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:17.016970+0000) 2019-10-13T13:15:17.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:17.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:17.928846+0000) 2019-10-13T13:15:18.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:18.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.153805+0000) 2019-10-13T13:15:18.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:18.694+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:18.694231+0000) 2019-10-13T13:15:19.835 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:15:19.835 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:19.839 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:15:19.839 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:19.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:19.846+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.850703+0000) 2019-10-13T13:15:19.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:19.846+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.851140+0000) 2019-10-13T13:15:19.854 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:19.846+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.851255+0000) 2019-10-13T13:15:19.854 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:19.850+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.852442+0000) 2019-10-13T13:15:19.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:19.846+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.851896+0000) 2019-10-13T13:15:19.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:19.846+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.853711+0000) 2019-10-13T13:15:19.856 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:19.850+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.854284+0000) 2019-10-13T13:15:19.860 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:19.854+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:19.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:19.858+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:19.861470+0000) 2019-10-13T13:15:21.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:21.942+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:21.945794+0000) 2019-10-13T13:15:24.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:24.265+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.271941+0000) 2019-10-13T13:15:24.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:24.553+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.557699+0000) 2019-10-13T13:15:24.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:24.853+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.861415+0000) 2019-10-13T13:15:24.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:24.857+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.861869+0000) 2019-10-13T13:15:24.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:24.857+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.862185+0000) 2019-10-13T13:15:24.865 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:24.858+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.863402+0000) 2019-10-13T13:15:24.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:24.857+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.863755+0000) 2019-10-13T13:15:24.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:24.857+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.863938+0000) 2019-10-13T13:15:24.866 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:24.857+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.864284+0000) 2019-10-13T13:15:24.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:24.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:24.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:24.870+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.873302+0000) 2019-10-13T13:15:24.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:24.917+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:24.924569+0000) 2019-10-13T13:15:27.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:27.022+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:27.024046+0000) 2019-10-13T13:15:27.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:27.925+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:27.929049+0000) 2019-10-13T13:15:28.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:28.149+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.154002+0000) 2019-10-13T13:15:28.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:28.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:28.694430+0000) 2019-10-13T13:15:29.877 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:29.870+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.875860+0000) 2019-10-13T13:15:29.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:29.869+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.875777+0000) 2019-10-13T13:15:29.879 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:29.869+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.876196+0000) 2019-10-13T13:15:29.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:29.869+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.876394+0000) 2019-10-13T13:15:29.879 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:29.869+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.876922+0000) 2019-10-13T13:15:29.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:29.869+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.877654+0000) 2019-10-13T13:15:29.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:29.873+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.878018+0000) 2019-10-13T13:15:29.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:29.878+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:29.881637+0000) 2019-10-13T13:15:29.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:29.877+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178587 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:31.947 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:31.942+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:31.945990+0000) 2019-10-13T13:15:34.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:34.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.272139+0000) 2019-10-13T13:15:34.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:34.557+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.557932+0000) 2019-10-13T13:15:34.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:34.886+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.890312+0000) 2019-10-13T13:15:34.894 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:34.881+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.888309+0000) 2019-10-13T13:15:34.894 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:34.881+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.889058+0000) 2019-10-13T13:15:34.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:34.881+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.889614+0000) 2019-10-13T13:15:34.895 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:34.881+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.889639+0000) 2019-10-13T13:15:34.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:34.885+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.890950+0000) 2019-10-13T13:15:34.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:34.885+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.891524+0000) 2019-10-13T13:15:34.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:34.894+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.897022+0000) 2019-10-13T13:15:34.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:34.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:34.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:34.921+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:34.924786+0000) 2019-10-13T13:15:37.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:37.026+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:37.031175+0000) 2019-10-13T13:15:37.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:37.925+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:37.929248+0000) 2019-10-13T13:15:38.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:38.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.154253+0000) 2019-10-13T13:15:38.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:38.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:38.694630+0000) 2019-10-13T13:15:39.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:39.897+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.900666+0000) 2019-10-13T13:15:39.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:39.901+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.901775+0000) 2019-10-13T13:15:39.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:39.901+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.902548+0000) 2019-10-13T13:15:39.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:39.901+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.902342+0000) 2019-10-13T13:15:39.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:39.898+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.903645+0000) 2019-10-13T13:15:39.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:39.901+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.903557+0000) 2019-10-13T13:15:39.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:39.901+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.903937+0000) 2019-10-13T13:15:39.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:39.902+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:39.905828+0000) 2019-10-13T13:15:39.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:39.905+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:41.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:41.942+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:41.946223+0000) 2019-10-13T13:15:44.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:44.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.272335+0000) 2019-10-13T13:15:44.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:44.557+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.558123+0000) 2019-10-13T13:15:44.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:44.910+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.913433+0000) 2019-10-13T13:15:44.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:44.909+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.912743+0000) 2019-10-13T13:15:44.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:44.913+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.914081+0000) 2019-10-13T13:15:44.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:44.913+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.914648+0000) 2019-10-13T13:15:44.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:44.913+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.915793+0000) 2019-10-13T13:15:44.918 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:44.913+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.915970+0000) 2019-10-13T13:15:44.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:44.913+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.916378+0000) 2019-10-13T13:15:44.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:44.918+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.921810+0000) 2019-10-13T13:15:44.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:44.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178677 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:44.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:44.921+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:44.924938+0000) 2019-10-13T13:15:47.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:47.034+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:47.038309+0000) 2019-10-13T13:15:47.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:47.925+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:47.929450+0000) 2019-10-13T13:15:48.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:48.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.154454+0000) 2019-10-13T13:15:48.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:48.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:48.694842+0000) 2019-10-13T13:15:49.900 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:15:49.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:49.903 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:15:49.904 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:15:49.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:49.922+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.924631+0000) 2019-10-13T13:15:49.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:49.921+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.924757+0000) 2019-10-13T13:15:49.927 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:49.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.925281+0000) 2019-10-13T13:15:49.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:49.921+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.925186+0000) 2019-10-13T13:15:49.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:49.921+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.925675+0000) 2019-10-13T13:15:49.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:49.925+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.926130+0000) 2019-10-13T13:15:49.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:49.925+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.926650+0000) 2019-10-13T13:15:49.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:49.930+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:49.932797+0000) 2019-10-13T13:15:49.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:49.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:51.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:51.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:51.946406+0000) 2019-10-13T13:15:54.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:54.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.272550+0000) 2019-10-13T13:15:54.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:54.557+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.558349+0000) 2019-10-13T13:15:54.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:54.925+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.926238+0000) 2019-10-13T13:15:54.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:54.933+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.936710+0000) 2019-10-13T13:15:54.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:54.934+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.937762+0000) 2019-10-13T13:15:54.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:54.933+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.937340+0000) 2019-10-13T13:15:54.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:54.937+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.938550+0000) 2019-10-13T13:15:54.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:54.937+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.938871+0000) 2019-10-13T13:15:54.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:54.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.939315+0000) 2019-10-13T13:15:54.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:54.937+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.939876+0000) 2019-10-13T13:15:54.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:54.938+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:54.943365+0000) 2019-10-13T13:15:54.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:54.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:15:57.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:57.042+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:57.045150+0000) 2019-10-13T13:15:57.931 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:57.925+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:57.929672+0000) 2019-10-13T13:15:58.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:58.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.154702+0000) 2019-10-13T13:15:58.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:58.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:58.695034+0000) 2019-10-13T13:15:59.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:15:59.946+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.950734+0000) 2019-10-13T13:15:59.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:15:59.949+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.950797+0000) 2019-10-13T13:15:59.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:15:59.949+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.951242+0000) 2019-10-13T13:15:59.954 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:15:59.949+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.951793+0000) 2019-10-13T13:15:59.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:15:59.949+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.952205+0000) 2019-10-13T13:15:59.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:15:59.949+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.952463+0000) 2019-10-13T13:15:59.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:15:59.949+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.953184+0000) 2019-10-13T13:15:59.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:15:59.954+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:15:59.957819+0000) 2019-10-13T13:15:59.960 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:15:59.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178771 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:01.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:01.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:01.946596+0000) 2019-10-13T13:16:04.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:04.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.272794+0000) 2019-10-13T13:16:04.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:04.557+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.558580+0000) 2019-10-13T13:16:04.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:04.925+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.926471+0000) 2019-10-13T13:16:04.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:04.961+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.962688+0000) 2019-10-13T13:16:04.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:04.962+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.964758+0000) 2019-10-13T13:16:04.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:04.961+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.964337+0000) 2019-10-13T13:16:04.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:04.961+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.965359+0000) 2019-10-13T13:16:04.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:04.965+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.966119+0000) 2019-10-13T13:16:04.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:04.965+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.966569+0000) 2019-10-13T13:16:04.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:04.965+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.967034+0000) 2019-10-13T13:16:04.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:04.969+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178802 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:04.972 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:04.966+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:04.971161+0000) 2019-10-13T13:16:07.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:07.054+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:07.054183+0000) 2019-10-13T13:16:07.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:07.929+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:07.929878+0000) 2019-10-13T13:16:08.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:08.153+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.154902+0000) 2019-10-13T13:16:08.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:08.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:08.695227+0000) 2019-10-13T13:16:09.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:09.969+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.972557+0000) 2019-10-13T13:16:09.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:09.973+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.973836+0000) 2019-10-13T13:16:09.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:09.973+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.974228+0000) 2019-10-13T13:16:09.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:09.973+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.974866+0000) 2019-10-13T13:16:09.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:09.969+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.975515+0000) 2019-10-13T13:16:09.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:09.973+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.976205+0000) 2019-10-13T13:16:09.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:09.973+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.976728+0000) 2019-10-13T13:16:09.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:09.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:09.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:09.977+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:09.982731+0000) 2019-10-13T13:16:11.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:11.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:11.946788+0000) 2019-10-13T13:16:14.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:14.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.273015+0000) 2019-10-13T13:16:14.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:14.557+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.558807+0000) 2019-10-13T13:16:14.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:14.925+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.926673+0000) 2019-10-13T13:16:14.986 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:14.981+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.984596+0000) 2019-10-13T13:16:14.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:14.985+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.986403+0000) 2019-10-13T13:16:14.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:14.985+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.987281+0000) 2019-10-13T13:16:14.990 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:14.985+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.988171+0000) 2019-10-13T13:16:14.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:14.985+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.988150+0000) 2019-10-13T13:16:14.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:14.985+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.988744+0000) 2019-10-13T13:16:14.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:14.985+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.989356+0000) 2019-10-13T13:16:14.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:14.993+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178865 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:14.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:14.993+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:14.996898+0000) 2019-10-13T13:16:17.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:17.057+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:17.060893+0000) 2019-10-13T13:16:17.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:17.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:17.930086+0000) 2019-10-13T13:16:18.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:18.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.155119+0000) 2019-10-13T13:16:18.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:18.692+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:18.695419+0000) 2019-10-13T13:16:19.997 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:16:19.997 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:20.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:19.996+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:19.998175+0000) 2019-10-13T13:16:20.001 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:16:20.001 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:20.002 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:19.996+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:19.998977+0000) 2019-10-13T13:16:20.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:19.997+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.001135+0000) 2019-10-13T13:16:20.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:19.996+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:19.999877+0000) 2019-10-13T13:16:20.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:19.996+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.001477+0000) 2019-10-13T13:16:20.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:20.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.002623+0000) 2019-10-13T13:16:20.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:20.000+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.003426+0000) 2019-10-13T13:16:20.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:20.004+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:20.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:20.001+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:20.007827+0000) 2019-10-13T13:16:21.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:21.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:21.946997+0000) 2019-10-13T13:16:24.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:24.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:24.273218+0000) 2019-10-13T13:16:24.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:24.556+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:24.559003+0000) 2019-10-13T13:16:24.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:24.924+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:24.926886+0000) 2019-10-13T13:16:25.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:25.008+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.010807+0000) 2019-10-13T13:16:25.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:25.008+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.011819+0000) 2019-10-13T13:16:25.016 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:25.008+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.012321+0000) 2019-10-13T13:16:25.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:25.008+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.012806+0000) 2019-10-13T13:16:25.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:25.008+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.012846+0000) 2019-10-13T13:16:25.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:25.008+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.013518+0000) 2019-10-13T13:16:25.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:25.009+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.013604+0000) 2019-10-13T13:16:25.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:25.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178928 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:25.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:25.013+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:25.019761+0000) 2019-10-13T13:16:27.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:27.061+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:27.067285+0000) 2019-10-13T13:16:27.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:27.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:27.930284+0000) 2019-10-13T13:16:28.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:28.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.155342+0000) 2019-10-13T13:16:28.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:28.692+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:28.695615+0000) 2019-10-13T13:16:30.023 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:30.016+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.021645+0000) 2019-10-13T13:16:30.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:30.020+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.022139+0000) 2019-10-13T13:16:30.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:30.020+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.022661+0000) 2019-10-13T13:16:30.025 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:30.020+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.022672+0000) 2019-10-13T13:16:30.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:30.017+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.023316+0000) 2019-10-13T13:16:30.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:30.020+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.023854+0000) 2019-10-13T13:16:30.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:30.020+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.023420+0000) 2019-10-13T13:16:30.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:30.025+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:30.028570+0000) 2019-10-13T13:16:30.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:30.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178958 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:31.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:31.945+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:31.947195+0000) 2019-10-13T13:16:34.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:34.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:34.273415+0000) 2019-10-13T13:16:34.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:34.556+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:34.559202+0000) 2019-10-13T13:16:34.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:34.924+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:34.927075+0000) 2019-10-13T13:16:35.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:35.033+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.036608+0000) 2019-10-13T13:16:35.039 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:35.032+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.036507+0000) 2019-10-13T13:16:35.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:35.032+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.036800+0000) 2019-10-13T13:16:35.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:35.032+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.037313+0000) 2019-10-13T13:16:35.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:35.036+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.038013+0000) 2019-10-13T13:16:35.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:35.036+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.038609+0000) 2019-10-13T13:16:35.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:35.036+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.039584+0000) 2019-10-13T13:16:35.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:35.037+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:35.041852+0000) 2019-10-13T13:16:35.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:35.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:37.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:37.073+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:37.073722+0000) 2019-10-13T13:16:37.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:37.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:37.930497+0000) 2019-10-13T13:16:38.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:38.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.155569+0000) 2019-10-13T13:16:38.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:38.692+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:38.695838+0000) 2019-10-13T13:16:40.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:40.044+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.047664+0000) 2019-10-13T13:16:40.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:40.044+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.048310+0000) 2019-10-13T13:16:40.050 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:40.044+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.049027+0000) 2019-10-13T13:16:40.051 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:40.045+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.049533+0000) 2019-10-13T13:16:40.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:40.048+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.049827+0000) 2019-10-13T13:16:40.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:40.048+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.050498+0000) 2019-10-13T13:16:40.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:40.048+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.050593+0000) 2019-10-13T13:16:40.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:40.052+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:40.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:40.057+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:40.060679+0000) 2019-10-13T13:16:41.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:41.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:41.947403+0000) 2019-10-13T13:16:44.275 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:44.268+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:44.273623+0000) 2019-10-13T13:16:44.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:44.556+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:44.559417+0000) 2019-10-13T13:16:44.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:44.924+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:44.927295+0000) 2019-10-13T13:16:45.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:45.056+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.059921+0000) 2019-10-13T13:16:45.063 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:45.056+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.060967+0000) 2019-10-13T13:16:45.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:45.056+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.060695+0000) 2019-10-13T13:16:45.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:45.057+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.062048+0000) 2019-10-13T13:16:45.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:45.060+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.062418+0000) 2019-10-13T13:16:45.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:45.060+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.062879+0000) 2019-10-13T13:16:45.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:45.060+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.063449+0000) 2019-10-13T13:16:45.071 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:45.064+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179051 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:45.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:45.065+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:45.070219+0000) 2019-10-13T13:16:47.082 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:47.077+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:47.080278+0000) 2019-10-13T13:16:47.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:47.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:47.930717+0000) 2019-10-13T13:16:48.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:48.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.155782+0000) 2019-10-13T13:16:48.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:48.692+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:48.696060+0000) 2019-10-13T13:16:50.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:50.068+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.072989+0000) 2019-10-13T13:16:50.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:50.068+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.073003+0000) 2019-10-13T13:16:50.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:50.068+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.073384+0000) 2019-10-13T13:16:50.077 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:50.069+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.074375+0000) 2019-10-13T13:16:50.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:50.072+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.073936+0000) 2019-10-13T13:16:50.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:50.072+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.074272+0000) 2019-10-13T13:16:50.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:50.072+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.074736+0000) 2019-10-13T13:16:50.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:50.073+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:50.079765+0000) 2019-10-13T13:16:50.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:50.076+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:50.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:16:50.105 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:50.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:16:50.109 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:16:51.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:51.945+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:51.947629+0000) 2019-10-13T13:16:54.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:54.272+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:54.273825+0000) 2019-10-13T13:16:54.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:54.556+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:54.559649+0000) 2019-10-13T13:16:54.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:54.924+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:54.927498+0000) 2019-10-13T13:16:55.086 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:16:55.080+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.084327+0000) 2019-10-13T13:16:55.087 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:16:55.080+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.084834+0000) 2019-10-13T13:16:55.089 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:55.084+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.087352+0000) 2019-10-13T13:16:55.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:55.084+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.088038+0000) 2019-10-13T13:16:55.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:55.084+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.088462+0000) 2019-10-13T13:16:55.090 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:16:55.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.089139+0000) 2019-10-13T13:16:55.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:16:55.084+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.089327+0000) 2019-10-13T13:16:55.095 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:16:55.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:16:55.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:55.089+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:55.094199+0000) 2019-10-13T13:16:57.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:16:57.081+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:57.086748+0000) 2019-10-13T13:16:57.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:16:57.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:57.930919+0000) 2019-10-13T13:16:58.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:16:58.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.155985+0000) 2019-10-13T13:16:58.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:16:58.692+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:16:58.696283+0000) 2019-10-13T13:17:00.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:00.092+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.096408+0000) 2019-10-13T13:17:00.100 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:00.096+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.098554+0000) 2019-10-13T13:17:00.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:00.096+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.098930+0000) 2019-10-13T13:17:00.101 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:00.096+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.099424+0000) 2019-10-13T13:17:00.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:00.096+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.099891+0000) 2019-10-13T13:17:00.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:00.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.100483+0000) 2019-10-13T13:17:00.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:00.097+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.101234+0000) 2019-10-13T13:17:00.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:00.101+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:00.104523+0000) 2019-10-13T13:17:00.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:00.100+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:01.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:01.945+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:01.947834+0000) 2019-10-13T13:17:04.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:04.272+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:04.274024+0000) 2019-10-13T13:17:04.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:04.556+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:04.559852+0000) 2019-10-13T13:17:04.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:04.924+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:04.927711+0000) 2019-10-13T13:17:05.110 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:05.104+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.108510+0000) 2019-10-13T13:17:05.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:05.104+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.109276+0000) 2019-10-13T13:17:05.111 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:05.105+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.110041+0000) 2019-10-13T13:17:05.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:05.108+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.110174+0000) 2019-10-13T13:17:05.112 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:05.108+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.110994+0000) 2019-10-13T13:17:05.113 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:05.108+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.111264+0000) 2019-10-13T13:17:05.113 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:05.108+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.111769+0000) 2019-10-13T13:17:05.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:05.113+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:05.117145+0000) 2019-10-13T13:17:05.119 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:05.112+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:07.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:07.093+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:07.093205+0000) 2019-10-13T13:17:07.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:07.928+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:07.931142+0000) 2019-10-13T13:17:08.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:08.152+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.156200+0000) 2019-10-13T13:17:08.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:08.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:08.696490+0000) 2019-10-13T13:17:10.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:10.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.120567+0000) 2019-10-13T13:17:10.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:10.120+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.121828+0000) 2019-10-13T13:17:10.124 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:10.120+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.122362+0000) 2019-10-13T13:17:10.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:10.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.123958+0000) 2019-10-13T13:17:10.126 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:10.120+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.124151+0000) 2019-10-13T13:17:10.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:10.120+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.124904+0000) 2019-10-13T13:17:10.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:10.120+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.125435+0000) 2019-10-13T13:17:10.128 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:10.121+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:10.127256+0000) 2019-10-13T13:17:10.131 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:10.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:11.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:11.941+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:11.948054+0000) 2019-10-13T13:17:14.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:14.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:14.274245+0000) 2019-10-13T13:17:14.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:14.555+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:14.560073+0000) 2019-10-13T13:17:14.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:14.923+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:14.927908+0000) 2019-10-13T13:17:15.135 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:15.127+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.133375+0000) 2019-10-13T13:17:15.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:15.128+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.134055+0000) 2019-10-13T13:17:15.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:15.127+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.133725+0000) 2019-10-13T13:17:15.136 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:15.131+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.134783+0000) 2019-10-13T13:17:15.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:15.131+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.135129+0000) 2019-10-13T13:17:15.137 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:15.131+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.135924+0000) 2019-10-13T13:17:15.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:15.131+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.136266+0000) 2019-10-13T13:17:15.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:15.139+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:15.145 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:15.136+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:15.143373+0000) 2019-10-13T13:17:17.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:17.100+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:17.104394+0000) 2019-10-13T13:17:17.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:17.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:17.931345+0000) 2019-10-13T13:17:18.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:18.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.156396+0000) 2019-10-13T13:17:18.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:18.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:18.696692+0000) 2019-10-13T13:17:20.147 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:20.139+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.145443+0000) 2019-10-13T13:17:20.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:20.143+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.146142+0000) 2019-10-13T13:17:20.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:20.143+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.146687+0000) 2019-10-13T13:17:20.149 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:20.143+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.147181+0000) 2019-10-13T13:17:20.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:20.143+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.147645+0000) 2019-10-13T13:17:20.150 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:20.143+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.148161+0000) 2019-10-13T13:17:20.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:20.144+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.148713+0000) 2019-10-13T13:17:20.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:20.151+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:20.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:20.152+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:20.157412+0000) 2019-10-13T13:17:20.236 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:17:20.237 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:20.240 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:17:20.240 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:21.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:21.944+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:21.948294+0000) 2019-10-13T13:17:24.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:24.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:24.274460+0000) 2019-10-13T13:17:24.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:24.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:24.560274+0000) 2019-10-13T13:17:24.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:24.927+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:24.928109+0000) 2019-10-13T13:17:25.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:25.151+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.157150+0000) 2019-10-13T13:17:25.159 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:25.155+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.157908+0000) 2019-10-13T13:17:25.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:25.155+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.158650+0000) 2019-10-13T13:17:25.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:25.155+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.159404+0000) 2019-10-13T13:17:25.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:25.155+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.161325+0000) 2019-10-13T13:17:25.163 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:25.155+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.161467+0000) 2019-10-13T13:17:25.164 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:25.156+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.162733+0000) 2019-10-13T13:17:25.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:25.163+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:25.170 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:25.164+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:25.168758+0000) 2019-10-13T13:17:27.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:27.104+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:27.110924+0000) 2019-10-13T13:17:27.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:27.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:27.931580+0000) 2019-10-13T13:17:28.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:28.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.156573+0000) 2019-10-13T13:17:28.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:28.691+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:28.696886+0000) 2019-10-13T13:17:30.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:30.167+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.170379+0000) 2019-10-13T13:17:30.174 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:30.168+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.173052+0000) 2019-10-13T13:17:30.175 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:30.167+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.173048+0000) 2019-10-13T13:17:30.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:30.171+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.174575+0000) 2019-10-13T13:17:30.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:30.171+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.174926+0000) 2019-10-13T13:17:30.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:30.171+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.175464+0000) 2019-10-13T13:17:30.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:30.171+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.176331+0000) 2019-10-13T13:17:30.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:30.172+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:30.177728+0000) 2019-10-13T13:17:30.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:30.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179330 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:31.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:31.944+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:31.948487+0000) 2019-10-13T13:17:34.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:34.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:34.274658+0000) 2019-10-13T13:17:34.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:34.555+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:34.560480+0000) 2019-10-13T13:17:34.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:34.923+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:34.928306+0000) 2019-10-13T13:17:35.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:35.179+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.183449+0000) 2019-10-13T13:17:35.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:35.179+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.185336+0000) 2019-10-13T13:17:35.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:35.183+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.187347+0000) 2019-10-13T13:17:35.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:35.180+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.187531+0000) 2019-10-13T13:17:35.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:35.183+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.188104+0000) 2019-10-13T13:17:35.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:35.183+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.188394+0000) 2019-10-13T13:17:35.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:35.183+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.188985+0000) 2019-10-13T13:17:35.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:35.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:35.195 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:35.188+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:35.193519+0000) 2019-10-13T13:17:37.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:37.116+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:37.117999+0000) 2019-10-13T13:17:37.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:37.931+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:37.931740+0000) 2019-10-13T13:17:38.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:38.151+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.156802+0000) 2019-10-13T13:17:38.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:38.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:38.697081+0000) 2019-10-13T13:17:40.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:40.188+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.196044+0000) 2019-10-13T13:17:40.198 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:40.191+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.196624+0000) 2019-10-13T13:17:40.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:40.191+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.196958+0000) 2019-10-13T13:17:40.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:40.191+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.197311+0000) 2019-10-13T13:17:40.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:40.195+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.198197+0000) 2019-10-13T13:17:40.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:40.195+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.198381+0000) 2019-10-13T13:17:40.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:40.195+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.199493+0000) 2019-10-13T13:17:40.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:40.196+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:40.202797+0000) 2019-10-13T13:17:40.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:40.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:41.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:41.944+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:41.948700+0000) 2019-10-13T13:17:44.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:44.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:44.274873+0000) 2019-10-13T13:17:44.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:44.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:44.560682+0000) 2019-10-13T13:17:44.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:44.923+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:44.928501+0000) 2019-10-13T13:17:45.211 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:45.203+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.209054+0000) 2019-10-13T13:17:45.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:45.207+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.212358+0000) 2019-10-13T13:17:45.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:45.207+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.213594+0000) 2019-10-13T13:17:45.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:45.208+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.214167+0000) 2019-10-13T13:17:45.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:45.211+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.214320+0000) 2019-10-13T13:17:45.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:45.211+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.214709+0000) 2019-10-13T13:17:45.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:45.212+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.216622+0000) 2019-10-13T13:17:45.218 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:45.211+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:45.216326+0000) 2019-10-13T13:17:45.219 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:45.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:47.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:47.120+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:47.124522+0000) 2019-10-13T13:17:47.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:47.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:47.931968+0000) 2019-10-13T13:17:48.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:48.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.157001+0000) 2019-10-13T13:17:48.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:48.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:48.697237+0000) 2019-10-13T13:17:50.224 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:50.219+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.221853+0000) 2019-10-13T13:17:50.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:50.219+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.222135+0000) 2019-10-13T13:17:50.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:50.219+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.222865+0000) 2019-10-13T13:17:50.225 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:50.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.223212+0000) 2019-10-13T13:17:50.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:50.220+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.224439+0000) 2019-10-13T13:17:50.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:50.219+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.224617+0000) 2019-10-13T13:17:50.228 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:50.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.226612+0000) 2019-10-13T13:17:50.232 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:50.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:50.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:50.232+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:50.236206+0000) 2019-10-13T13:17:50.410 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:17:50.410 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:50.414 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:17:50.415 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:17:51.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:51.944+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:51.948911+0000) 2019-10-13T13:17:54.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:54.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:54.275147+0000) 2019-10-13T13:17:54.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:54.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:54.560880+0000) 2019-10-13T13:17:54.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:54.923+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:54.928651+0000) 2019-10-13T13:17:55.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:55.227+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.233527+0000) 2019-10-13T13:17:55.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:17:55.231+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.234010+0000) 2019-10-13T13:17:55.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:55.231+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.234350+0000) 2019-10-13T13:17:55.237 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:17:55.231+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.235002+0000) 2019-10-13T13:17:55.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:17:55.231+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.236856+0000) 2019-10-13T13:17:55.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:17:55.232+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.238597+0000) 2019-10-13T13:17:55.240 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:55.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.238662+0000) 2019-10-13T13:17:55.244 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:17:55.239+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:17:55.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:55.240+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:55.244417+0000) 2019-10-13T13:17:57.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:17:57.128+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:57.131041+0000) 2019-10-13T13:17:57.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:17:57.927+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:57.932193+0000) 2019-10-13T13:17:58.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:17:58.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.157200+0000) 2019-10-13T13:17:58.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:17:58.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:17:58.697446+0000) 2019-10-13T13:18:00.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:00.243+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.246000+0000) 2019-10-13T13:18:00.248 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:00.243+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.246387+0000) 2019-10-13T13:18:00.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:00.243+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.247099+0000) 2019-10-13T13:18:00.250 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:00.244+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.248992+0000) 2019-10-13T13:18:00.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:00.243+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.249476+0000) 2019-10-13T13:18:00.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:00.247+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.249806+0000) 2019-10-13T13:18:00.253 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:00.247+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.251130+0000) 2019-10-13T13:18:00.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:00.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179516 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:00.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:00.252+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:00.257670+0000) 2019-10-13T13:18:01.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:01.948+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:01.949140+0000) 2019-10-13T13:18:04.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:04.271+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:04.275371+0000) 2019-10-13T13:18:04.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:04.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:04.561086+0000) 2019-10-13T13:18:04.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:04.927+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:04.928822+0000) 2019-10-13T13:18:05.260 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:05.252+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.258975+0000) 2019-10-13T13:18:05.261 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:05.255+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.259397+0000) 2019-10-13T13:18:05.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:05.255+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.259835+0000) 2019-10-13T13:18:05.262 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:05.255+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.260130+0000) 2019-10-13T13:18:05.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:05.255+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.260617+0000) 2019-10-13T13:18:05.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:05.255+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.261204+0000) 2019-10-13T13:18:05.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:05.255+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.261708+0000) 2019-10-13T13:18:05.269 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:05.263+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:05.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:05.264+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:05.269419+0000) 2019-10-13T13:18:07.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:07.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:07.137567+0000) 2019-10-13T13:18:07.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:07.931+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:07.932349+0000) 2019-10-13T13:18:08.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:08.150+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.157407+0000) 2019-10-13T13:18:08.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:08.694+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:08.697656+0000) 2019-10-13T13:18:10.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:10.266+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.270613+0000) 2019-10-13T13:18:10.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:10.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.271262+0000) 2019-10-13T13:18:10.273 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:10.266+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.271641+0000) 2019-10-13T13:18:10.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:10.266+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.272207+0000) 2019-10-13T13:18:10.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:10.268+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.273650+0000) 2019-10-13T13:18:10.276 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:10.270+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.274427+0000) 2019-10-13T13:18:10.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:10.270+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.274986+0000) 2019-10-13T13:18:10.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:10.276+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:10.280409+0000) 2019-10-13T13:18:10.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:10.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179580 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:11.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:11.948+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:11.949348+0000) 2019-10-13T13:18:14.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:14.274+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:14.275623+0000) 2019-10-13T13:18:14.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:14.554+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:14.561303+0000) 2019-10-13T13:18:14.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:14.926+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:14.929030+0000) 2019-10-13T13:18:15.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:15.278+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.284165+0000) 2019-10-13T13:18:15.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:15.278+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.284796+0000) 2019-10-13T13:18:15.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:15.284+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.285942+0000) 2019-10-13T13:18:15.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:15.282+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.286005+0000) 2019-10-13T13:18:15.288 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:15.282+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.286461+0000) 2019-10-13T13:18:15.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:15.282+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.286844+0000) 2019-10-13T13:18:15.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:15.282+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.287002+0000) 2019-10-13T13:18:15.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:15.288+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:15.291872+0000) 2019-10-13T13:18:15.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:15.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:17.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:17.140+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:17.144033+0000) 2019-10-13T13:18:17.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:17.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:17.932543+0000) 2019-10-13T13:18:18.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:18.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.157602+0000) 2019-10-13T13:18:18.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:18.694+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:18.697851+0000) 2019-10-13T13:18:20.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:20.290+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.296072+0000) 2019-10-13T13:18:20.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:20.290+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.297216+0000) 2019-10-13T13:18:20.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:20.290+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.297345+0000) 2019-10-13T13:18:20.300 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:20.294+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.298178+0000) 2019-10-13T13:18:20.300 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:20.295+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.299029+0000) 2019-10-13T13:18:20.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:20.294+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.299068+0000) 2019-10-13T13:18:20.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:20.294+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.299287+0000) 2019-10-13T13:18:20.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:20.303+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:20.304923+0000) 2019-10-13T13:18:20.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:20.302+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:20.508 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:18:20.508 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:20.512 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:18:20.512 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:21.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:21.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:21.949582+0000) 2019-10-13T13:18:24.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:24.270+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:24.275819+0000) 2019-10-13T13:18:24.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:24.558+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:24.561524+0000) 2019-10-13T13:18:24.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:24.926+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:24.929245+0000) 2019-10-13T13:18:25.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:25.302+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.309102+0000) 2019-10-13T13:18:25.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:25.302+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.309509+0000) 2019-10-13T13:18:25.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:25.307+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.311482+0000) 2019-10-13T13:18:25.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:25.306+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.312567+0000) 2019-10-13T13:18:25.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:25.306+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.313174+0000) 2019-10-13T13:18:25.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:25.306+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.313627+0000) 2019-10-13T13:18:25.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:25.310+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.315564+0000) 2019-10-13T13:18:25.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:25.315+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:25.319666+0000) 2019-10-13T13:18:25.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:25.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:27.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:27.147+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:27.149984+0000) 2019-10-13T13:18:27.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:27.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:27.932771+0000) 2019-10-13T13:18:28.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:28.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.157805+0000) 2019-10-13T13:18:28.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:28.694+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:28.698043+0000) 2019-10-13T13:18:30.324 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:30.319+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.322763+0000) 2019-10-13T13:18:30.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:30.318+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.323763+0000) 2019-10-13T13:18:30.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:30.318+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.323611+0000) 2019-10-13T13:18:30.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:30.318+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.323695+0000) 2019-10-13T13:18:30.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:30.318+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.324183+0000) 2019-10-13T13:18:30.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:30.322+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.326485+0000) 2019-10-13T13:18:30.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:30.322+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.327063+0000) 2019-10-13T13:18:30.332 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:30.327+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:30.331007+0000) 2019-10-13T13:18:30.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:30.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:31.951 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:31.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:31.949814+0000) 2019-10-13T13:18:34.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:34.270+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:34.276052+0000) 2019-10-13T13:18:34.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:34.558+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:34.561779+0000) 2019-10-13T13:18:34.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:34.926+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:34.929421+0000) 2019-10-13T13:18:35.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:35.330+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.335464+0000) 2019-10-13T13:18:35.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:35.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.336603+0000) 2019-10-13T13:18:35.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:35.330+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.337305+0000) 2019-10-13T13:18:35.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:35.334+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.337804+0000) 2019-10-13T13:18:35.340 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:35.334+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.338216+0000) 2019-10-13T13:18:35.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:35.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.338673+0000) 2019-10-13T13:18:35.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:35.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.339560+0000) 2019-10-13T13:18:35.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:35.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:35.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:35.343+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:35.345841+0000) 2019-10-13T13:18:37.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:37.159+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:37.160938+0000) 2019-10-13T13:18:37.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:37.930+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:37.932969+0000) 2019-10-13T13:18:38.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:38.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.158029+0000) 2019-10-13T13:18:38.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:38.694+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:38.698266+0000) 2019-10-13T13:18:40.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:40.342+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.348175+0000) 2019-10-13T13:18:40.350 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:40.342+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.348694+0000) 2019-10-13T13:18:40.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:40.342+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.349193+0000) 2019-10-13T13:18:40.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:40.346+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.351541+0000) 2019-10-13T13:18:40.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:40.346+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.351994+0000) 2019-10-13T13:18:40.356 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:40.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.354574+0000) 2019-10-13T13:18:40.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:40.355+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.356976+0000) 2019-10-13T13:18:40.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:40.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:40.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:40.355+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:40.358633+0000) 2019-10-13T13:18:41.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:41.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:41.950023+0000) 2019-10-13T13:18:44.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:44.274+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:44.276273+0000) 2019-10-13T13:18:44.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:44.558+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:44.562001+0000) 2019-10-13T13:18:44.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:44.926+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:44.929625+0000) 2019-10-13T13:18:45.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:45.354+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.361468+0000) 2019-10-13T13:18:45.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:45.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.361905+0000) 2019-10-13T13:18:45.365 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:45.358+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.362790+0000) 2019-10-13T13:18:45.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:45.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.364815+0000) 2019-10-13T13:18:45.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:45.358+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.365236+0000) 2019-10-13T13:18:45.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:45.362+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.367423+0000) 2019-10-13T13:18:45.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:45.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.368630+0000) 2019-10-13T13:18:45.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:45.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:45.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:45.371+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:45.372741+0000) 2019-10-13T13:18:47.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:47.163+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:47.167555+0000) 2019-10-13T13:18:47.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:47.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:47.933205+0000) 2019-10-13T13:18:48.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:48.154+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.158243+0000) 2019-10-13T13:18:48.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:48.698+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:48.698473+0000) 2019-10-13T13:18:50.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:50.370+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.373596+0000) 2019-10-13T13:18:50.376 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:50.370+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.374047+0000) 2019-10-13T13:18:50.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:50.370+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.375460+0000) 2019-10-13T13:18:50.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:50.374+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.377782+0000) 2019-10-13T13:18:50.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:50.375+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.379155+0000) 2019-10-13T13:18:50.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:50.374+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.379054+0000) 2019-10-13T13:18:50.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:50.374+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.379816+0000) 2019-10-13T13:18:50.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:50.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:50.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:50.383+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:50.384592+0000) 2019-10-13T13:18:50.619 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:18:50.619 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:50.623 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:18:50.623 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:18:51.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:51.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:51.950219+0000) 2019-10-13T13:18:54.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:54.274+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:54.276470+0000) 2019-10-13T13:18:54.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:54.562+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:54.562204+0000) 2019-10-13T13:18:54.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:54.926+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:54.929842+0000) 2019-10-13T13:18:55.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:18:55.383+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.387347+0000) 2019-10-13T13:18:55.389 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:18:55.382+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.387394+0000) 2019-10-13T13:18:55.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:55.386+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.389945+0000) 2019-10-13T13:18:55.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:18:55.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.390427+0000) 2019-10-13T13:18:55.394 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:55.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.392127+0000) 2019-10-13T13:18:55.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:55.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.392515+0000) 2019-10-13T13:18:55.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:18:55.390+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.394472+0000) 2019-10-13T13:18:55.397 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:18:55.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:18:55.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:55.395+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:55.396308+0000) 2019-10-13T13:18:57.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:18:57.171+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:57.174118+0000) 2019-10-13T13:18:57.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:18:57.926+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:57.933399+0000) 2019-10-13T13:18:58.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:18:58.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.158437+0000) 2019-10-13T13:18:58.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:18:58.698+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:18:58.698671+0000) 2019-10-13T13:19:00.401 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:00.394+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.399202+0000) 2019-10-13T13:19:00.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:00.399+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.401529+0000) 2019-10-13T13:19:00.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:00.398+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.401846+0000) 2019-10-13T13:19:00.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:00.398+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.402317+0000) 2019-10-13T13:19:00.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:00.398+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.402914+0000) 2019-10-13T13:19:00.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:00.398+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.404992+0000) 2019-10-13T13:19:00.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:00.398+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.405706+0000) 2019-10-13T13:19:00.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:00.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:00.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:00.407+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:00.411663+0000) 2019-10-13T13:19:01.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:01.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:01.950431+0000) 2019-10-13T13:19:04.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:04.269+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:04.276669+0000) 2019-10-13T13:19:04.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:04.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:04.562404+0000) 2019-10-13T13:19:04.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:04.925+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:04.930014+0000) 2019-10-13T13:19:05.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:05.405+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.410558+0000) 2019-10-13T13:19:05.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:05.405+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.412711+0000) 2019-10-13T13:19:05.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:05.405+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.413418+0000) 2019-10-13T13:19:05.415 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:05.405+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.413677+0000) 2019-10-13T13:19:05.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:05.411+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.414285+0000) 2019-10-13T13:19:05.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:05.409+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.414706+0000) 2019-10-13T13:19:05.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:05.409+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.414948+0000) 2019-10-13T13:19:05.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:05.413+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:05.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:05.419+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:05.420838+0000) 2019-10-13T13:19:07.182 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:07.179+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:07.180736+0000) 2019-10-13T13:19:07.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:07.929+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:07.933598+0000) 2019-10-13T13:19:08.160 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:08.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.158668+0000) 2019-10-13T13:19:08.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:08.693+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:08.698865+0000) 2019-10-13T13:19:10.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:10.417+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.423580+0000) 2019-10-13T13:19:10.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:10.417+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.424036+0000) 2019-10-13T13:19:10.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:10.417+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.425136+0000) 2019-10-13T13:19:10.427 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:10.423+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.426221+0000) 2019-10-13T13:19:10.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:10.421+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.426608+0000) 2019-10-13T13:19:10.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:10.421+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.427236+0000) 2019-10-13T13:19:10.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:10.421+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.428013+0000) 2019-10-13T13:19:10.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:10.427+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:10.432053+0000) 2019-10-13T13:19:10.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:10.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:11.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:11.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:11.950620+0000) 2019-10-13T13:19:14.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:14.273+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:14.276920+0000) 2019-10-13T13:19:14.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:14.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:14.562612+0000) 2019-10-13T13:19:14.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:14.929+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:14.930218+0000) 2019-10-13T13:19:15.438 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:15.429+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.436161+0000) 2019-10-13T13:19:15.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:15.429+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.436622+0000) 2019-10-13T13:19:15.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:15.429+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.437632+0000) 2019-10-13T13:19:15.440 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:15.433+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.438577+0000) 2019-10-13T13:19:15.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:15.433+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.439034+0000) 2019-10-13T13:19:15.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:15.433+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.439671+0000) 2019-10-13T13:19:15.442 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:15.439+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.440307+0000) 2019-10-13T13:19:15.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:15.443+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:15.444432+0000) 2019-10-13T13:19:15.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:15.441+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:17.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:17.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:17.187347+0000) 2019-10-13T13:19:17.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:17.933+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:17.933837+0000) 2019-10-13T13:19:18.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:18.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.158866+0000) 2019-10-13T13:19:18.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:18.697+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:18.699107+0000) 2019-10-13T13:19:20.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:20.445+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.449477+0000) 2019-10-13T13:19:20.452 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:20.449+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.450233+0000) 2019-10-13T13:19:20.452 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:20.449+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.450442+0000) 2019-10-13T13:19:20.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:20.449+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.452628+0000) 2019-10-13T13:19:20.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:20.449+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.452929+0000) 2019-10-13T13:19:20.455 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:20.449+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.453237+0000) 2019-10-13T13:19:20.455 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:20.451+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.453929+0000) 2019-10-13T13:19:20.460 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:20.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:20.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:20.459+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:20.461503+0000) 2019-10-13T13:19:20.741 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:19:20.742 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:20.745 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:19:20.745 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:21.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:21.947+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:21.950813+0000) 2019-10-13T13:19:24.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:24.273+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:24.277145+0000) 2019-10-13T13:19:24.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:24.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:24.562829+0000) 2019-10-13T13:19:24.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:24.929+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:24.930446+0000) 2019-10-13T13:19:25.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:25.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.462340+0000) 2019-10-13T13:19:25.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:25.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.465276+0000) 2019-10-13T13:19:25.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:25.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.463490+0000) 2019-10-13T13:19:25.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:25.461+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.463919+0000) 2019-10-13T13:19:25.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:25.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.464619+0000) 2019-10-13T13:19:25.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:25.461+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.465709+0000) 2019-10-13T13:19:25.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:25.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.466114+0000) 2019-10-13T13:19:25.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:25.466+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:25.469649+0000) 2019-10-13T13:19:25.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:25.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:27.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:27.194+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:27.197678+0000) 2019-10-13T13:19:27.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:27.933+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:27.934045+0000) 2019-10-13T13:19:28.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:28.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.159068+0000) 2019-10-13T13:19:28.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:28.697+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:28.699299+0000) 2019-10-13T13:19:30.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:30.473+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.474954+0000) 2019-10-13T13:19:30.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:30.473+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.475991+0000) 2019-10-13T13:19:30.478 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:30.474+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.476469+0000) 2019-10-13T13:19:30.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:30.473+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.476527+0000) 2019-10-13T13:19:30.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:30.473+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.476829+0000) 2019-10-13T13:19:30.479 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:30.473+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.477231+0000) 2019-10-13T13:19:30.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:30.473+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.477522+0000) 2019-10-13T13:19:30.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:30.478+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:30.480831+0000) 2019-10-13T13:19:30.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:30.481+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:31.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:31.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:31.951013+0000) 2019-10-13T13:19:34.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:34.273+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:34.277359+0000) 2019-10-13T13:19:34.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:34.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:34.563040+0000) 2019-10-13T13:19:34.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:34.929+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:34.930633+0000) 2019-10-13T13:19:35.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:35.485+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.487154+0000) 2019-10-13T13:19:35.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:35.485+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.487779+0000) 2019-10-13T13:19:35.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:35.485+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.488567+0000) 2019-10-13T13:19:35.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:35.489+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.490402+0000) 2019-10-13T13:19:35.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:35.486+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.491096+0000) 2019-10-13T13:19:35.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:35.489+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.491221+0000) 2019-10-13T13:19:35.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:35.489+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.493114+0000) 2019-10-13T13:19:35.497 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:35.493+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:35.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:35.494+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:35.499402+0000) 2019-10-13T13:19:37.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:37.202+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:37.204289+0000) 2019-10-13T13:19:37.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:37.933+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:37.934244+0000) 2019-10-13T13:19:38.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:38.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.159268+0000) 2019-10-13T13:19:38.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:38.697+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:38.699493+0000) 2019-10-13T13:19:40.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:40.497+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.498796+0000) 2019-10-13T13:19:40.501 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:40.497+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.499198+0000) 2019-10-13T13:19:40.502 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:40.497+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.499653+0000) 2019-10-13T13:19:40.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:40.498+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.501255+0000) 2019-10-13T13:19:40.503 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:40.497+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.501168+0000) 2019-10-13T13:19:40.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:40.501+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.503765+0000) 2019-10-13T13:19:40.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:40.501+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.503944+0000) 2019-10-13T13:19:40.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:40.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:40.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:40.506+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:40.510336+0000) 2019-10-13T13:19:41.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:41.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:41.951207+0000) 2019-10-13T13:19:44.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:44.277+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:44.277572+0000) 2019-10-13T13:19:44.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:44.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:44.563279+0000) 2019-10-13T13:19:44.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:44.929+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:44.930833+0000) 2019-10-13T13:19:45.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:45.509+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.511066+0000) 2019-10-13T13:19:45.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:45.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.511755+0000) 2019-10-13T13:19:45.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:45.509+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.512153+0000) 2019-10-13T13:19:45.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:45.513+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.514600+0000) 2019-10-13T13:19:45.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:45.513+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.515122+0000) 2019-10-13T13:19:45.518 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:45.513+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.516172+0000) 2019-10-13T13:19:45.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:45.513+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.517228+0000) 2019-10-13T13:19:45.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:45.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:45.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:45.518+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:45.520889+0000) 2019-10-13T13:19:47.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:47.210+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:47.210680+0000) 2019-10-13T13:19:47.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:47.933+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:47.934479+0000) 2019-10-13T13:19:48.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:48.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.159483+0000) 2019-10-13T13:19:48.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:48.697+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:48.699719+0000) 2019-10-13T13:19:50.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:50.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.523874+0000) 2019-10-13T13:19:50.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:50.522+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.524966+0000) 2019-10-13T13:19:50.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:50.521+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.524385+0000) 2019-10-13T13:19:50.527 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:50.521+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.525476+0000) 2019-10-13T13:19:50.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:50.525+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.526341+0000) 2019-10-13T13:19:50.528 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:50.525+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.526603+0000) 2019-10-13T13:19:50.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:50.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.527628+0000) 2019-10-13T13:19:50.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:50.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:50.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:50.530+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:50.533806+0000) 2019-10-13T13:19:50.861 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:19:50.861 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:50.865 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:19:50.865 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:19:51.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:51.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:51.951401+0000) 2019-10-13T13:19:54.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:54.277+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:54.277806+0000) 2019-10-13T13:19:54.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:54.561+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:54.563498+0000) 2019-10-13T13:19:54.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:54.929+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:54.931032+0000) 2019-10-13T13:19:55.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:55.533+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.536760+0000) 2019-10-13T13:19:55.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:55.533+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.537649+0000) 2019-10-13T13:19:55.540 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:19:55.534+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.538295+0000) 2019-10-13T13:19:55.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:19:55.537+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.538454+0000) 2019-10-13T13:19:55.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:19:55.537+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.538800+0000) 2019-10-13T13:19:55.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:19:55.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.538472+0000) 2019-10-13T13:19:55.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:55.537+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.540402+0000) 2019-10-13T13:19:55.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:19:55.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:19:55.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:55.542+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:55.547617+0000) 2019-10-13T13:19:57.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:19:57.218+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:57.222330+0000) 2019-10-13T13:19:57.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:19:57.933+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:57.934677+0000) 2019-10-13T13:19:58.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:19:58.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.159702+0000) 2019-10-13T13:19:58.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:19:58.697+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:19:58.699917+0000) 2019-10-13T13:20:00.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:00.544+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.549437+0000) 2019-10-13T13:20:00.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:00.548+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.550051+0000) 2019-10-13T13:20:00.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:00.548+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.552300+0000) 2019-10-13T13:20:00.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:00.550+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.553801+0000) 2019-10-13T13:20:00.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:00.552+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.555302+0000) 2019-10-13T13:20:00.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:00.556+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.558248+0000) 2019-10-13T13:20:00.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:00.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:00.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:00.556+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.558888+0000) 2019-10-13T13:20:00.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:00.562+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:00.565799+0000) 2019-10-13T13:20:01.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:01.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:01.951619+0000) 2019-10-13T13:20:04.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:04.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:04.278075+0000) 2019-10-13T13:20:04.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:04.560+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:04.563714+0000) 2019-10-13T13:20:04.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:04.928+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:04.931262+0000) 2019-10-13T13:20:05.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:05.556+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.561548+0000) 2019-10-13T13:20:05.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:05.560+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.562406+0000) 2019-10-13T13:20:05.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:05.560+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.563093+0000) 2019-10-13T13:20:05.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:05.560+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.563707+0000) 2019-10-13T13:20:05.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:05.560+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.564032+0000) 2019-10-13T13:20:05.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:05.562+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.566962+0000) 2019-10-13T13:20:05.570 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:05.564+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.568655+0000) 2019-10-13T13:20:05.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:05.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:05.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:05.574+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:05.579731+0000) 2019-10-13T13:20:07.230 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:07.226+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:07.228840+0000) 2019-10-13T13:20:07.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:07.932+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:07.934905+0000) 2019-10-13T13:20:08.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:08.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.159901+0000) 2019-10-13T13:20:08.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:08.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:08.700105+0000) 2019-10-13T13:20:10.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:10.572+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.575481+0000) 2019-10-13T13:20:10.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:10.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.576337+0000) 2019-10-13T13:20:10.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:10.572+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.576382+0000) 2019-10-13T13:20:10.579 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:10.572+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.576664+0000) 2019-10-13T13:20:10.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:10.572+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.577302+0000) 2019-10-13T13:20:10.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:10.576+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.578368+0000) 2019-10-13T13:20:10.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:10.576+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.579259+0000) 2019-10-13T13:20:10.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:10.582+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:10.584288+0000) 2019-10-13T13:20:10.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:10.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:11.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:11.946+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:11.951810+0000) 2019-10-13T13:20:14.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:14.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:14.278331+0000) 2019-10-13T13:20:14.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:14.560+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:14.563915+0000) 2019-10-13T13:20:14.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:14.928+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:14.931418+0000) 2019-10-13T13:20:15.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:15.584+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.588028+0000) 2019-10-13T13:20:15.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:15.584+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.588561+0000) 2019-10-13T13:20:15.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:15.584+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.589635+0000) 2019-10-13T13:20:15.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:15.588+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.590012+0000) 2019-10-13T13:20:15.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:15.588+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.590400+0000) 2019-10-13T13:20:15.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:15.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.591242+0000) 2019-10-13T13:20:15.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:15.588+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.591956+0000) 2019-10-13T13:20:15.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:15.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180366 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:15.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:15.594+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:15.597361+0000) 2019-10-13T13:20:17.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:17.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:17.235249+0000) 2019-10-13T13:20:17.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:17.932+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:17.935114+0000) 2019-10-13T13:20:18.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:18.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.160095+0000) 2019-10-13T13:20:18.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:18.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:18.700305+0000) 2019-10-13T13:20:20.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:20.596+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.600266+0000) 2019-10-13T13:20:20.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:20.596+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.600788+0000) 2019-10-13T13:20:20.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:20.598+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.601605+0000) 2019-10-13T13:20:20.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:20.596+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.601488+0000) 2019-10-13T13:20:20.604 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:20.600+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.601891+0000) 2019-10-13T13:20:20.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:20.600+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.602540+0000) 2019-10-13T13:20:20.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:20.600+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.603606+0000) 2019-10-13T13:20:20.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:20.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180396 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:20.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:20.606+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:20.611542+0000) 2019-10-13T13:20:20.985 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:20:20.985 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:20.988 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:20:20.989 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:21.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:21.950+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:21.952007+0000) 2019-10-13T13:20:24.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:24.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:24.278538+0000) 2019-10-13T13:20:24.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:24.560+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:24.564116+0000) 2019-10-13T13:20:24.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:24.928+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:24.931618+0000) 2019-10-13T13:20:25.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:25.610+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.612288+0000) 2019-10-13T13:20:25.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:25.608+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.612791+0000) 2019-10-13T13:20:25.615 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:25.608+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.613007+0000) 2019-10-13T13:20:25.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:25.608+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.613383+0000) 2019-10-13T13:20:25.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:25.612+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.614246+0000) 2019-10-13T13:20:25.617 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:25.612+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.614949+0000) 2019-10-13T13:20:25.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:25.612+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.615511+0000) 2019-10-13T13:20:25.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:25.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:25.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:25.622+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:25.624648+0000) 2019-10-13T13:20:27.243 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:27.237+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:27.241847+0000) 2019-10-13T13:20:27.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:27.932+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:27.935328+0000) 2019-10-13T13:20:28.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:28.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.160352+0000) 2019-10-13T13:20:28.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:28.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:28.700524+0000) 2019-10-13T13:20:30.625 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:30.620+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.623366+0000) 2019-10-13T13:20:30.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:30.617+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.624043+0000) 2019-10-13T13:20:30.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:30.620+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.624848+0000) 2019-10-13T13:20:30.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:30.620+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.624691+0000) 2019-10-13T13:20:30.627 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:30.620+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.625389+0000) 2019-10-13T13:20:30.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:30.620+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.625654+0000) 2019-10-13T13:20:30.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:30.624+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.626927+0000) 2019-10-13T13:20:30.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:30.625+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:30.630735+0000) 2019-10-13T13:20:30.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:30.628+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:31.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:31.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:31.952167+0000) 2019-10-13T13:20:34.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:34.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:34.278739+0000) 2019-10-13T13:20:34.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:34.560+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:34.564322+0000) 2019-10-13T13:20:34.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:34.928+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:34.931816+0000) 2019-10-13T13:20:35.637 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:35.629+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.635211+0000) 2019-10-13T13:20:35.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:35.632+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.635813+0000) 2019-10-13T13:20:35.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:35.632+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.636623+0000) 2019-10-13T13:20:35.639 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:35.632+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.637195+0000) 2019-10-13T13:20:35.639 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:35.632+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.637378+0000) 2019-10-13T13:20:35.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:35.636+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.638377+0000) 2019-10-13T13:20:35.640 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:35.636+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.638670+0000) 2019-10-13T13:20:35.646 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:35.640+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:35.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:35.641+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:35.646830+0000) 2019-10-13T13:20:37.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:37.245+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:37.248553+0000) 2019-10-13T13:20:37.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:37.932+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:37.935525+0000) 2019-10-13T13:20:38.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:38.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.160553+0000) 2019-10-13T13:20:38.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:38.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:38.700713+0000) 2019-10-13T13:20:40.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:40.644+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.647933+0000) 2019-10-13T13:20:40.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:40.644+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.649301+0000) 2019-10-13T13:20:40.653 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:40.648+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.651431+0000) 2019-10-13T13:20:40.653 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:40.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.651948+0000) 2019-10-13T13:20:40.654 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:40.648+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.652885+0000) 2019-10-13T13:20:40.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:40.648+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.653584+0000) 2019-10-13T13:20:40.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:40.652+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.654477+0000) 2019-10-13T13:20:40.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:40.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:40.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:40.653+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:40.659035+0000) 2019-10-13T13:20:41.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:41.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:41.952378+0000) 2019-10-13T13:20:44.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:44.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:44.278941+0000) 2019-10-13T13:20:44.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:44.564+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:44.564523+0000) 2019-10-13T13:20:44.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:44.928+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:44.932009+0000) 2019-10-13T13:20:45.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:45.656+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.661173+0000) 2019-10-13T13:20:45.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:45.656+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.661577+0000) 2019-10-13T13:20:45.664 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:45.660+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.661982+0000) 2019-10-13T13:20:45.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:45.660+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.663127+0000) 2019-10-13T13:20:45.665 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:45.660+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.663720+0000) 2019-10-13T13:20:45.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:45.661+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.664274+0000) 2019-10-13T13:20:45.666 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:45.660+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.664306+0000) 2019-10-13T13:20:45.671 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:45.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:45.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:45.665+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:45.670458+0000) 2019-10-13T13:20:47.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:47.249+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:47.255180+0000) 2019-10-13T13:20:47.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:47.932+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:47.935755+0000) 2019-10-13T13:20:48.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:48.156+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.160701+0000) 2019-10-13T13:20:48.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:48.696+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:48.700902+0000) 2019-10-13T13:20:50.675 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:50.668+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.673005+0000) 2019-10-13T13:20:50.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:50.672+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.675126+0000) 2019-10-13T13:20:50.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:50.672+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.675679+0000) 2019-10-13T13:20:50.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:50.672+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.676308+0000) 2019-10-13T13:20:50.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:50.676+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.677919+0000) 2019-10-13T13:20:50.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:50.673+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.678488+0000) 2019-10-13T13:20:50.682 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:50.676+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.680203+0000) 2019-10-13T13:20:50.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:50.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:50.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:50.681+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:50.685798+0000) 2019-10-13T13:20:51.076 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:20:51.076 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:51.080 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:20:51.080 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:20:51.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:51.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:51.952576+0000) 2019-10-13T13:20:54.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:54.276+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:54.279181+0000) 2019-10-13T13:20:54.566 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:54.564+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:54.564757+0000) 2019-10-13T13:20:54.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:54.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:54.932213+0000) 2019-10-13T13:20:55.687 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:20:55.679+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.684995+0000) 2019-10-13T13:20:55.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:55.679+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.685560+0000) 2019-10-13T13:20:55.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:20:55.683+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.686054+0000) 2019-10-13T13:20:55.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:20:55.681+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.686771+0000) 2019-10-13T13:20:55.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:20:55.683+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.687800+0000) 2019-10-13T13:20:55.690 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:55.683+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.688284+0000) 2019-10-13T13:20:55.690 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:55.683+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.688739+0000) 2019-10-13T13:20:55.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:20:55.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180617 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:20:55.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:55.693+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:55.697040+0000) 2019-10-13T13:20:57.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:20:57.261+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:57.266605+0000) 2019-10-13T13:20:57.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:20:57.931+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:57.935997+0000) 2019-10-13T13:20:58.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:20:58.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.160918+0000) 2019-10-13T13:20:58.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:20:58.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:20:58.701099+0000) 2019-10-13T13:21:00.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:00.695+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.697982+0000) 2019-10-13T13:21:00.701 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:00.695+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.698423+0000) 2019-10-13T13:21:00.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:00.695+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.699111+0000) 2019-10-13T13:21:00.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:00.693+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.699765+0000) 2019-10-13T13:21:00.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:00.695+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.699659+0000) 2019-10-13T13:21:00.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:00.695+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.700141+0000) 2019-10-13T13:21:00.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:00.695+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.700637+0000) 2019-10-13T13:21:00.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:00.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:00.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:00.701+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:00.706872+0000) 2019-10-13T13:21:01.954 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:01.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:01.952809+0000) 2019-10-13T13:21:04.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:04.275+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:04.279351+0000) 2019-10-13T13:21:04.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:04.563+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:04.564956+0000) 2019-10-13T13:21:04.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:04.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:04.932408+0000) 2019-10-13T13:21:05.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:05.703+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.709239+0000) 2019-10-13T13:21:05.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:05.705+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.711205+0000) 2019-10-13T13:21:05.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:05.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.713194+0000) 2019-10-13T13:21:05.716 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:05.711+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.714665+0000) 2019-10-13T13:21:05.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:05.711+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.715113+0000) 2019-10-13T13:21:05.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:05.711+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.715841+0000) 2019-10-13T13:21:05.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:05.711+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.716071+0000) 2019-10-13T13:21:05.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:05.717+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:05.721822+0000) 2019-10-13T13:21:05.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:05.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:07.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:07.269+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:07.273100+0000) 2019-10-13T13:21:07.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:07.935+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:07.936205+0000) 2019-10-13T13:21:08.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:08.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.161126+0000) 2019-10-13T13:21:08.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:08.699+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:08.701296+0000) 2019-10-13T13:21:10.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:10.715+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.721690+0000) 2019-10-13T13:21:10.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:10.719+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.722286+0000) 2019-10-13T13:21:10.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:10.719+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.724570+0000) 2019-10-13T13:21:10.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:10.723+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.725818+0000) 2019-10-13T13:21:10.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:10.721+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.726342+0000) 2019-10-13T13:21:10.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:10.723+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.726494+0000) 2019-10-13T13:21:10.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:10.723+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.727056+0000) 2019-10-13T13:21:10.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:10.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:10.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:10.725+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:10.731668+0000) 2019-10-13T13:21:11.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:11.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:11.953010+0000) 2019-10-13T13:21:14.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:14.275+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:14.279576+0000) 2019-10-13T13:21:14.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:14.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:14.565154+0000) 2019-10-13T13:21:14.934 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:14.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:14.932609+0000) 2019-10-13T13:21:15.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:15.731+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.735138+0000) 2019-10-13T13:21:15.737 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:15.729+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.735698+0000) 2019-10-13T13:21:15.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:15.731+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.736050+0000) 2019-10-13T13:21:15.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:15.731+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.736825+0000) 2019-10-13T13:21:15.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:15.731+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.737023+0000) 2019-10-13T13:21:15.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:15.731+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.737442+0000) 2019-10-13T13:21:15.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:15.731+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.737670+0000) 2019-10-13T13:21:15.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:15.737+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:15.742783+0000) 2019-10-13T13:21:15.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:15.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:17.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:17.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:17.279630+0000) 2019-10-13T13:21:17.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:17.935+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:17.936482+0000) 2019-10-13T13:21:18.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:18.155+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:18.161345+0000) 2019-10-13T13:21:18.703 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:18.699+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:18.701492+0000) 2019-10-13T13:21:20.749 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:20.743+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.746861+0000) 2019-10-13T13:21:20.749 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:20.741+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.747631+0000) 2019-10-13T13:21:20.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:20.743+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.748234+0000) 2019-10-13T13:21:20.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:20.743+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.748841+0000) 2019-10-13T13:21:20.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:20.743+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.748924+0000) 2019-10-13T13:21:20.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:20.747+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.749824+0000) 2019-10-13T13:21:20.752 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:20.747+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.750670+0000) 2019-10-13T13:21:20.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:20.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:20.758 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:20.753+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:20.756696+0000) 2019-10-13T13:21:21.215 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:21:21.216 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:21.219 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:21:21.219 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:21.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:21.949+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:21.953163+0000) 2019-10-13T13:21:24.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:24.275+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.279773+0000) 2019-10-13T13:21:24.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:24.563+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.565357+0000) 2019-10-13T13:21:24.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:24.927+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:24.932828+0000) 2019-10-13T13:21:25.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:25.755+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.758911+0000) 2019-10-13T13:21:25.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:25.757+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.761510+0000) 2019-10-13T13:21:25.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:25.755+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.761646+0000) 2019-10-13T13:21:25.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:25.759+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.764090+0000) 2019-10-13T13:21:25.767 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:25.759+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.764848+0000) 2019-10-13T13:21:25.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:25.759+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.765066+0000) 2019-10-13T13:21:25.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:25.759+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.765319+0000) 2019-10-13T13:21:25.769 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:25.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:25.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:25.765+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:25.770386+0000) 2019-10-13T13:21:27.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:27.289+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:27.291195+0000) 2019-10-13T13:21:27.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:27.935+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:27.936679+0000) 2019-10-13T13:21:28.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:28.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:28.161552+0000) 2019-10-13T13:21:28.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:28.695+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:28.701684+0000) 2019-10-13T13:21:30.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:30.767+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.771272+0000) 2019-10-13T13:21:30.774 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:30.767+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.771377+0000) 2019-10-13T13:21:30.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:30.767+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.771838+0000) 2019-10-13T13:21:30.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:30.769+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.774011+0000) 2019-10-13T13:21:30.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:30.771+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.774129+0000) 2019-10-13T13:21:30.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:30.773+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.777116+0000) 2019-10-13T13:21:30.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:30.771+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.776923+0000) 2019-10-13T13:21:30.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:30.775+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:30.779113+0000) 2019-10-13T13:21:30.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:30.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:31.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:31.948+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:31.953394+0000) 2019-10-13T13:21:34.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:34.275+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.279968+0000) 2019-10-13T13:21:34.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:34.559+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.565531+0000) 2019-10-13T13:21:34.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:34.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:34.933060+0000) 2019-10-13T13:21:35.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:35.780+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.785058+0000) 2019-10-13T13:21:35.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:35.779+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.785136+0000) 2019-10-13T13:21:35.787 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:35.779+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.785320+0000) 2019-10-13T13:21:35.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:35.779+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.785671+0000) 2019-10-13T13:21:35.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:35.783+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.786206+0000) 2019-10-13T13:21:35.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:35.783+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.786493+0000) 2019-10-13T13:21:35.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:35.783+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.786359+0000) 2019-10-13T13:21:35.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:35.784+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:35.789302+0000) 2019-10-13T13:21:35.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:35.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:37.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:37.292+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:37.297651+0000) 2019-10-13T13:21:37.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:37.935+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:37.936895+0000) 2019-10-13T13:21:38.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:38.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:38.161791+0000) 2019-10-13T13:21:38.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:38.699+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:38.701911+0000) 2019-10-13T13:21:40.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:40.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.796185+0000) 2019-10-13T13:21:40.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:40.791+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.796576+0000) 2019-10-13T13:21:40.801 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:40.795+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.799179+0000) 2019-10-13T13:21:40.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:40.795+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.800591+0000) 2019-10-13T13:21:40.804 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:40.799+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.802291+0000) 2019-10-13T13:21:40.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:40.796+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.803615+0000) 2019-10-13T13:21:40.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:40.799+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.804245+0000) 2019-10-13T13:21:40.807 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:40.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:40.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:40.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:40.805639+0000) 2019-10-13T13:21:41.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:41.948+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:41.953585+0000) 2019-10-13T13:21:44.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:44.279+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.280177+0000) 2019-10-13T13:21:44.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:44.563+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.565789+0000) 2019-10-13T13:21:44.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:44.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:44.933257+0000) 2019-10-13T13:21:45.811 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:45.803+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.808504+0000) 2019-10-13T13:21:45.812 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:45.807+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.809965+0000) 2019-10-13T13:21:45.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:45.807+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.810514+0000) 2019-10-13T13:21:45.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:45.807+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.812473+0000) 2019-10-13T13:21:45.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:45.807+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.812748+0000) 2019-10-13T13:21:45.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:45.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.812794+0000) 2019-10-13T13:21:45.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:45.807+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.813302+0000) 2019-10-13T13:21:45.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:45.808+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:45.815172+0000) 2019-10-13T13:21:45.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:45.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:47.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:47.300+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:47.304378+0000) 2019-10-13T13:21:47.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:47.931+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:47.937106+0000) 2019-10-13T13:21:48.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:48.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:48.162034+0000) 2019-10-13T13:21:48.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:48.699+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:48.702132+0000) 2019-10-13T13:21:50.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:50.816+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.820523+0000) 2019-10-13T13:21:50.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:50.814+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.821347+0000) 2019-10-13T13:21:50.823 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:50.814+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.821659+0000) 2019-10-13T13:21:50.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:50.818+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.822410+0000) 2019-10-13T13:21:50.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:50.818+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.822849+0000) 2019-10-13T13:21:50.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:50.818+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.824933+0000) 2019-10-13T13:21:50.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:50.822+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.827162+0000) 2019-10-13T13:21:50.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:50.824+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:50.828606+0000) 2019-10-13T13:21:50.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:50.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:51.317 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:21:51.317 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:51.321 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:21:51.321 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:21:51.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:51.952+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:51.953782+0000) 2019-10-13T13:21:54.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:54.278+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.280409+0000) 2019-10-13T13:21:54.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:54.562+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.566006+0000) 2019-10-13T13:21:54.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:54.930+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:54.933483+0000) 2019-10-13T13:21:55.835 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:21:55.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.833346+0000) 2019-10-13T13:21:55.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:55.826+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.833492+0000) 2019-10-13T13:21:55.836 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:21:55.830+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.834230+0000) 2019-10-13T13:21:55.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:21:55.830+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.834933+0000) 2019-10-13T13:21:55.837 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:55.830+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.834952+0000) 2019-10-13T13:21:55.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:21:55.830+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.835163+0000) 2019-10-13T13:21:55.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:55.830+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.836226+0000) 2019-10-13T13:21:55.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:21:55.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:21:55.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:55.840+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:55.844925+0000) 2019-10-13T13:21:57.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:21:57.308+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:57.310846+0000) 2019-10-13T13:21:57.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:21:57.930+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:57.937353+0000) 2019-10-13T13:21:58.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:21:58.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:58.162275+0000) 2019-10-13T13:21:58.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:21:58.698+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:21:58.702357+0000) 2019-10-13T13:22:00.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:00.838+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.845606+0000) 2019-10-13T13:22:00.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:00.842+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.846498+0000) 2019-10-13T13:22:00.849 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:00.842+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.847270+0000) 2019-10-13T13:22:00.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:00.840+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.847866+0000) 2019-10-13T13:22:00.850 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:00.842+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.848501+0000) 2019-10-13T13:22:00.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:00.842+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.848246+0000) 2019-10-13T13:22:00.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:00.842+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.848811+0000) 2019-10-13T13:22:00.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:00.848+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:00.853700+0000) 2019-10-13T13:22:00.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:00.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181026 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:01.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:01.952+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:01.953981+0000) 2019-10-13T13:22:04.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:04.278+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.280612+0000) 2019-10-13T13:22:04.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:04.562+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.566240+0000) 2019-10-13T13:22:04.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:04.930+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:04.933802+0000) 2019-10-13T13:22:05.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:05.854+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.859557+0000) 2019-10-13T13:22:05.862 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:05.854+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.859907+0000) 2019-10-13T13:22:05.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:05.854+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.860535+0000) 2019-10-13T13:22:05.863 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:05.854+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.860869+0000) 2019-10-13T13:22:05.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:05.856+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.861134+0000) 2019-10-13T13:22:05.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:05.854+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.861614+0000) 2019-10-13T13:22:05.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:05.858+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.862004+0000) 2019-10-13T13:22:05.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:05.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:05.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:05.864+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:05.868617+0000) 2019-10-13T13:22:07.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:07.320+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:07.322155+0000) 2019-10-13T13:22:07.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:07.934+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:07.937564+0000) 2019-10-13T13:22:08.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:08.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:08.162491+0000) 2019-10-13T13:22:08.704 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:08.698+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:08.702585+0000) 2019-10-13T13:22:10.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:10.866+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.870268+0000) 2019-10-13T13:22:10.873 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:10.866+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.871286+0000) 2019-10-13T13:22:10.875 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:10.866+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.873152+0000) 2019-10-13T13:22:10.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:10.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.874113+0000) 2019-10-13T13:22:10.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:10.870+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.874508+0000) 2019-10-13T13:22:10.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:10.870+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.874828+0000) 2019-10-13T13:22:10.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:10.870+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.875017+0000) 2019-10-13T13:22:10.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:10.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181090 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:10.882 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:10.876+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:10.880909+0000) 2019-10-13T13:22:11.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:11.952+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:11.954196+0000) 2019-10-13T13:22:14.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:14.274+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.280841+0000) 2019-10-13T13:22:14.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:14.562+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.566457+0000) 2019-10-13T13:22:14.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:14.930+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:14.934000+0000) 2019-10-13T13:22:15.885 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:15.878+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.883537+0000) 2019-10-13T13:22:15.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:15.878+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.883679+0000) 2019-10-13T13:22:15.889 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:15.882+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.887024+0000) 2019-10-13T13:22:15.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:15.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.890671+0000) 2019-10-13T13:22:15.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:15.884+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.890831+0000) 2019-10-13T13:22:15.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:15.886+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.891191+0000) 2019-10-13T13:22:15.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:15.886+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.891932+0000) 2019-10-13T13:22:15.894 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:15.886+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:15.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:15.886+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:15.893446+0000) 2019-10-13T13:22:17.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:17.332+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:17.333699+0000) 2019-10-13T13:22:17.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:17.934+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:17.937775+0000) 2019-10-13T13:22:18.164 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:18.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:18.162705+0000) 2019-10-13T13:22:18.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:18.702+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:18.702779+0000) 2019-10-13T13:22:20.900 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:20.894+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.898099+0000) 2019-10-13T13:22:20.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:20.894+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.898367+0000) 2019-10-13T13:22:20.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:20.894+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.899054+0000) 2019-10-13T13:22:20.902 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:20.894+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.898559+0000) 2019-10-13T13:22:20.902 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:20.894+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.899625+0000) 2019-10-13T13:22:20.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:20.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.899289+0000) 2019-10-13T13:22:20.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:20.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.900394+0000) 2019-10-13T13:22:20.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:20.900+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:20.904457+0000) 2019-10-13T13:22:20.908 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:20.902+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181152 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:21.432 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:22:21.432 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:21.435 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:22:21.436 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:21.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:21.948+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:21.954422+0000) 2019-10-13T13:22:24.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:24.278+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.281043+0000) 2019-10-13T13:22:24.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:24.566+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.566657+0000) 2019-10-13T13:22:24.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:24.930+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:24.934191+0000) 2019-10-13T13:22:25.911 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:25.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.909278+0000) 2019-10-13T13:22:25.912 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:25.906+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.909787+0000) 2019-10-13T13:22:25.912 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:25.904+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.910826+0000) 2019-10-13T13:22:25.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:25.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.911114+0000) 2019-10-13T13:22:25.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:25.906+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.912093+0000) 2019-10-13T13:22:25.914 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:25.906+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.911950+0000) 2019-10-13T13:22:25.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:25.906+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.912410+0000) 2019-10-13T13:22:25.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:25.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:25.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:25.912+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:25.919606+0000) 2019-10-13T13:22:27.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:27.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:27.340319+0000) 2019-10-13T13:22:27.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:27.934+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:27.937974+0000) 2019-10-13T13:22:28.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:28.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:28.162959+0000) 2019-10-13T13:22:28.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:28.702+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:28.702972+0000) 2019-10-13T13:22:30.923 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:30.920+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.921940+0000) 2019-10-13T13:22:30.924 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:30.918+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.922399+0000) 2019-10-13T13:22:30.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:30.918+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.922754+0000) 2019-10-13T13:22:30.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:30.918+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.924548+0000) 2019-10-13T13:22:30.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:30.918+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.925181+0000) 2019-10-13T13:22:30.928 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:30.918+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.925036+0000) 2019-10-13T13:22:30.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:30.918+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.925459+0000) 2019-10-13T13:22:30.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:30.926+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:30.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:30.932+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:30.932945+0000) 2019-10-13T13:22:31.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:31.952+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:31.954616+0000) 2019-10-13T13:22:34.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:34.278+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.281238+0000) 2019-10-13T13:22:34.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:34.562+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.566855+0000) 2019-10-13T13:22:34.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:34.934+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:34.934407+0000) 2019-10-13T13:22:35.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:35.932+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.934887+0000) 2019-10-13T13:22:35.937 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:35.930+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.935437+0000) 2019-10-13T13:22:35.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:35.930+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.935647+0000) 2019-10-13T13:22:35.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:35.930+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.937476+0000) 2019-10-13T13:22:35.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:35.934+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.938428+0000) 2019-10-13T13:22:35.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:35.934+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.939166+0000) 2019-10-13T13:22:35.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:35.934+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.939459+0000) 2019-10-13T13:22:35.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:35.944+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:35.944938+0000) 2019-10-13T13:22:35.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:35.938+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:37.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:37.343+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:37.346858+0000) 2019-10-13T13:22:37.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:37.934+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:37.938178+0000) 2019-10-13T13:22:38.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:38.158+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:38.163151+0000) 2019-10-13T13:22:38.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:38.702+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:38.703187+0000) 2019-10-13T13:22:40.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:40.942+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.948214+0000) 2019-10-13T13:22:40.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:40.946+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.950287+0000) 2019-10-13T13:22:40.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:40.946+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.950528+0000) 2019-10-13T13:22:40.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:40.946+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.951021+0000) 2019-10-13T13:22:40.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:40.947+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.951855+0000) 2019-10-13T13:22:40.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:40.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.954275+0000) 2019-10-13T13:22:40.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:40.951+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.954990+0000) 2019-10-13T13:22:40.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:40.950+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:40.956343+0000) 2019-10-13T13:22:40.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:40.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:41.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:41.951+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:41.954803+0000) 2019-10-13T13:22:44.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:44.278+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.281435+0000) 2019-10-13T13:22:44.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:44.566+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.567010+0000) 2019-10-13T13:22:44.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:44.934+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:44.934638+0000) 2019-10-13T13:22:45.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:45.954+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.960815+0000) 2019-10-13T13:22:45.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:45.954+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.961627+0000) 2019-10-13T13:22:45.965 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:45.958+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.963597+0000) 2019-10-13T13:22:45.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:45.958+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.964562+0000) 2019-10-13T13:22:45.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:45.958+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.965120+0000) 2019-10-13T13:22:45.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:45.963+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.965640+0000) 2019-10-13T13:22:45.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:45.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.965866+0000) 2019-10-13T13:22:45.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:45.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:45.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:45.971+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:45.972377+0000) 2019-10-13T13:22:47.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:47.351+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:47.353453+0000) 2019-10-13T13:22:47.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:47.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:47.938383+0000) 2019-10-13T13:22:48.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:48.157+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:48.163355+0000) 2019-10-13T13:22:48.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:48.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:48.703408+0000) 2019-10-13T13:22:50.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:50.969+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.974222+0000) 2019-10-13T13:22:50.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:50.969+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.974692+0000) 2019-10-13T13:22:50.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:50.969+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.976382+0000) 2019-10-13T13:22:50.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:50.975+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.976935+0000) 2019-10-13T13:22:50.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:50.973+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.978812+0000) 2019-10-13T13:22:50.981 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:50.973+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.979655+0000) 2019-10-13T13:22:50.982 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:50.973+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.979916+0000) 2019-10-13T13:22:50.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:50.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:50.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:50.983+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:50.984853+0000) 2019-10-13T13:22:51.503 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:22:51.504 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:51.508 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:22:51.508 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:22:51.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:51.951+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:51.955022+0000) 2019-10-13T13:22:54.283 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:54.277+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.281639+0000) 2019-10-13T13:22:54.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:54.565+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.567172+0000) 2019-10-13T13:22:54.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:54.933+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:54.934835+0000) 2019-10-13T13:22:55.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:22:55.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.986350+0000) 2019-10-13T13:22:55.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:55.981+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.986487+0000) 2019-10-13T13:22:55.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:22:55.987+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.988528+0000) 2019-10-13T13:22:55.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:22:55.985+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.990225+0000) 2019-10-13T13:22:55.992 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:55.985+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.990856+0000) 2019-10-13T13:22:55.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:55.985+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.991279+0000) 2019-10-13T13:22:55.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:22:55.985+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.993429+0000) 2019-10-13T13:22:55.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:22:55.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:22:55.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:55.995+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:55.997305+0000) 2019-10-13T13:22:57.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:22:57.355+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:57.359969+0000) 2019-10-13T13:22:57.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:22:57.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:57.938615+0000) 2019-10-13T13:22:58.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:22:58.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:58.163571+0000) 2019-10-13T13:22:58.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:22:58.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:22:58.703603+0000) 2019-10-13T13:23:01.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:00.997+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.000978+0000) 2019-10-13T13:23:01.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:00.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.001737+0000) 2019-10-13T13:23:01.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:01.001+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.001834+0000) 2019-10-13T13:23:01.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:01.001+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.002598+0000) 2019-10-13T13:23:01.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:01.001+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.002647+0000) 2019-10-13T13:23:01.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:01.001+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.003039+0000) 2019-10-13T13:23:01.005 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:01.001+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.003546+0000) 2019-10-13T13:23:01.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:01.003+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.005711+0000) 2019-10-13T13:23:01.010 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:01.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:01.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:01.951+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:01.955195+0000) 2019-10-13T13:23:04.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:04.281+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.281850+0000) 2019-10-13T13:23:04.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:04.565+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.567378+0000) 2019-10-13T13:23:04.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:04.933+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:04.935034+0000) 2019-10-13T13:23:06.014 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:06.009+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.012431+0000) 2019-10-13T13:23:06.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:06.009+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.012712+0000) 2019-10-13T13:23:06.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:06.009+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.013005+0000) 2019-10-13T13:23:06.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:06.015+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.016217+0000) 2019-10-13T13:23:06.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:06.013+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.016344+0000) 2019-10-13T13:23:06.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:06.017+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.018314+0000) 2019-10-13T13:23:06.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:06.017+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.018475+0000) 2019-10-13T13:23:06.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:06.019+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:06.021852+0000) 2019-10-13T13:23:06.024 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:06.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181436 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:07.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:07.363+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:07.366449+0000) 2019-10-13T13:23:07.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:07.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:07.938830+0000) 2019-10-13T13:23:08.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:08.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:08.163782+0000) 2019-10-13T13:23:08.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:08.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:08.703792+0000) 2019-10-13T13:23:11.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:11.023+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.024442+0000) 2019-10-13T13:23:11.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:11.021+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.024822+0000) 2019-10-13T13:23:11.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:11.021+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.025090+0000) 2019-10-13T13:23:11.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:11.025+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.026261+0000) 2019-10-13T13:23:11.028 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:11.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.026742+0000) 2019-10-13T13:23:11.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:11.025+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.027101+0000) 2019-10-13T13:23:11.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:11.025+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.027881+0000) 2019-10-13T13:23:11.036 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:11.031+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.035124+0000) 2019-10-13T13:23:11.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:11.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:11.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:11.951+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:11.955373+0000) 2019-10-13T13:23:14.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:14.281+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.282038+0000) 2019-10-13T13:23:14.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:14.565+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.567628+0000) 2019-10-13T13:23:14.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:14.933+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:14.935248+0000) 2019-10-13T13:23:16.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:16.037+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.039284+0000) 2019-10-13T13:23:16.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:16.037+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.039768+0000) 2019-10-13T13:23:16.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:16.037+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.040451+0000) 2019-10-13T13:23:16.043 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:16.039+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.041548+0000) 2019-10-13T13:23:16.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:16.037+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.041294+0000) 2019-10-13T13:23:16.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:16.041+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.041922+0000) 2019-10-13T13:23:16.044 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:16.041+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.042352+0000) 2019-10-13T13:23:16.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:16.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:16.047955+0000) 2019-10-13T13:23:16.050 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:16.045+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:17.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:17.371+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:17.372953+0000) 2019-10-13T13:23:17.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:17.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:17.939025+0000) 2019-10-13T13:23:18.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:18.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:18.163971+0000) 2019-10-13T13:23:18.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:18.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:18.703986+0000) 2019-10-13T13:23:21.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:21.047+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.051322+0000) 2019-10-13T13:23:21.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:21.049+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.051670+0000) 2019-10-13T13:23:21.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:21.049+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.052510+0000) 2019-10-13T13:23:21.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:21.053+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.054183+0000) 2019-10-13T13:23:21.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:21.053+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.055708+0000) 2019-10-13T13:23:21.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:21.053+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.056302+0000) 2019-10-13T13:23:21.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:21.053+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.056660+0000) 2019-10-13T13:23:21.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:21.057+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:21.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:21.059+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.061595+0000) 2019-10-13T13:23:21.633 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:23:21.634 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:21.637 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:23:21.637 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:21.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:21.951+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:21.955575+0000) 2019-10-13T13:23:24.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:24.281+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.282234+0000) 2019-10-13T13:23:24.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:24.565+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.567821+0000) 2019-10-13T13:23:24.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:24.933+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:24.935470+0000) 2019-10-13T13:23:26.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:26.061+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.064919+0000) 2019-10-13T13:23:26.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:26.061+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.065161+0000) 2019-10-13T13:23:26.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:26.061+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.065538+0000) 2019-10-13T13:23:26.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:26.065+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.067552+0000) 2019-10-13T13:23:26.070 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:26.065+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.067859+0000) 2019-10-13T13:23:26.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:26.067+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.069408+0000) 2019-10-13T13:23:26.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:26.069+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.071408+0000) 2019-10-13T13:23:26.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:26.069+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:26.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:26.075+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:26.076536+0000) 2019-10-13T13:23:27.382 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:27.375+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:27.379802+0000) 2019-10-13T13:23:27.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:27.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:27.939235+0000) 2019-10-13T13:23:28.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:28.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:28.164166+0000) 2019-10-13T13:23:28.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:28.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:28.704185+0000) 2019-10-13T13:23:31.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:31.073+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.076604+0000) 2019-10-13T13:23:31.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:31.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.077148+0000) 2019-10-13T13:23:31.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:31.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.077496+0000) 2019-10-13T13:23:31.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:31.077+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.077871+0000) 2019-10-13T13:23:31.084 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:31.079+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.081383+0000) 2019-10-13T13:23:31.084 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:31.077+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.078996+0000) 2019-10-13T13:23:31.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:31.077+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.079584+0000) 2019-10-13T13:23:31.088 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:31.085+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:31.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:31.083+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.086743+0000) 2019-10-13T13:23:31.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:31.955+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:31.955783+0000) 2019-10-13T13:23:34.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:34.281+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.282439+0000) 2019-10-13T13:23:34.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:34.565+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.568050+0000) 2019-10-13T13:23:34.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:34.933+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:34.935681+0000) 2019-10-13T13:23:36.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:36.089+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.090618+0000) 2019-10-13T13:23:36.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:36.089+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.090766+0000) 2019-10-13T13:23:36.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:36.089+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.091340+0000) 2019-10-13T13:23:36.095 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:36.089+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.093057+0000) 2019-10-13T13:23:36.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:36.093+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.095152+0000) 2019-10-13T13:23:36.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:36.091+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.095668+0000) 2019-10-13T13:23:36.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:36.093+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.097597+0000) 2019-10-13T13:23:36.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:36.097+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:36.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:36.095+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:36.100057+0000) 2019-10-13T13:23:37.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:37.383+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:37.386306+0000) 2019-10-13T13:23:37.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:37.937+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:37.939430+0000) 2019-10-13T13:23:38.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:38.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:38.164333+0000) 2019-10-13T13:23:38.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:38.701+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:38.704398+0000) 2019-10-13T13:23:41.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:41.101+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.102233+0000) 2019-10-13T13:23:41.106 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:41.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.104432+0000) 2019-10-13T13:23:41.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:41.101+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.105029+0000) 2019-10-13T13:23:41.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:41.101+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.105540+0000) 2019-10-13T13:23:41.108 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:41.105+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.106061+0000) 2019-10-13T13:23:41.108 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:41.105+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.106377+0000) 2019-10-13T13:23:41.108 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:41.102+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.106924+0000) 2019-10-13T13:23:41.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:41.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:41.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:41.110+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.112386+0000) 2019-10-13T13:23:41.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:41.950+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:41.955950+0000) 2019-10-13T13:23:44.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:44.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.282654+0000) 2019-10-13T13:23:44.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:44.564+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.568206+0000) 2019-10-13T13:23:44.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:44.932+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:44.935883+0000) 2019-10-13T13:23:46.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:46.112+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.115113+0000) 2019-10-13T13:23:46.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:46.112+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.115906+0000) 2019-10-13T13:23:46.118 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:46.112+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.116266+0000) 2019-10-13T13:23:46.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:46.112+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.116722+0000) 2019-10-13T13:23:46.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:46.112+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.117078+0000) 2019-10-13T13:23:46.119 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:46.112+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.117538+0000) 2019-10-13T13:23:46.120 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:46.114+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.118388+0000) 2019-10-13T13:23:46.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:46.120+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:46.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:46.122+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:46.125005+0000) 2019-10-13T13:23:47.394 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:47.390+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:47.392449+0000) 2019-10-13T13:23:47.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:47.936+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:47.939652+0000) 2019-10-13T13:23:48.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:48.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:48.164548+0000) 2019-10-13T13:23:48.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:48.700+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:48.704591+0000) 2019-10-13T13:23:51.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:51.124+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.126762+0000) 2019-10-13T13:23:51.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:51.124+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.127815+0000) 2019-10-13T13:23:51.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:51.124+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.128536+0000) 2019-10-13T13:23:51.131 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:51.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.129973+0000) 2019-10-13T13:23:51.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:51.128+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.130412+0000) 2019-10-13T13:23:51.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:51.128+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.132222+0000) 2019-10-13T13:23:51.135 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:51.130+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.133327+0000) 2019-10-13T13:23:51.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:51.134+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.136211+0000) 2019-10-13T13:23:51.140 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:51.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:51.781 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:23:51.782 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:51.786 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:23:51.786 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:23:51.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:51.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:51.956147+0000) 2019-10-13T13:23:54.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:54.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.282856+0000) 2019-10-13T13:23:54.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:54.564+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.568432+0000) 2019-10-13T13:23:54.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:54.932+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:54.936082+0000) 2019-10-13T13:23:56.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:23:56.136+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.141400+0000) 2019-10-13T13:23:56.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:23:56.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.143823+0000) 2019-10-13T13:23:56.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:23:56.140+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.143403+0000) 2019-10-13T13:23:56.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:56.144+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.146126+0000) 2019-10-13T13:23:56.148 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:56.144+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.146545+0000) 2019-10-13T13:23:56.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:23:56.144+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.147004+0000) 2019-10-13T13:23:56.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:56.144+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.147254+0000) 2019-10-13T13:23:56.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:23:56.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181750 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:23:56.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:56.150+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:56.154049+0000) 2019-10-13T13:23:57.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:23:57.394+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:57.399109+0000) 2019-10-13T13:23:57.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:23:57.936+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:57.939896+0000) 2019-10-13T13:23:58.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:23:58.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:58.164746+0000) 2019-10-13T13:23:58.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:23:58.700+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:23:58.704785+0000) 2019-10-13T13:24:01.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:01.152+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.153949+0000) 2019-10-13T13:24:01.156 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:01.152+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.154306+0000) 2019-10-13T13:24:01.157 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:01.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.155922+0000) 2019-10-13T13:24:01.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:01.152+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.156448+0000) 2019-10-13T13:24:01.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:01.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.157425+0000) 2019-10-13T13:24:01.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:01.152+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.157461+0000) 2019-10-13T13:24:01.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:01.156+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.158848+0000) 2019-10-13T13:24:01.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:01.158+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.162468+0000) 2019-10-13T13:24:01.166 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:01.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:01.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:01.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:01.956334+0000) 2019-10-13T13:24:04.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:04.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.283063+0000) 2019-10-13T13:24:04.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:04.564+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.568629+0000) 2019-10-13T13:24:04.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:04.932+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:04.936289+0000) 2019-10-13T13:24:06.169 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:06.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.167779+0000) 2019-10-13T13:24:06.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:06.164+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.167867+0000) 2019-10-13T13:24:06.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:06.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.168503+0000) 2019-10-13T13:24:06.172 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:06.168+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.170387+0000) 2019-10-13T13:24:06.173 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:06.168+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.171024+0000) 2019-10-13T13:24:06.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:06.168+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.171457+0000) 2019-10-13T13:24:06.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:06.168+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.171631+0000) 2019-10-13T13:24:06.178 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:06.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:06.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:06.178+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:06.181742+0000) 2019-10-13T13:24:07.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:07.402+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:07.405803+0000) 2019-10-13T13:24:07.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:07.936+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:07.940154+0000) 2019-10-13T13:24:08.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:08.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:08.165009+0000) 2019-10-13T13:24:08.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:08.704+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:08.704980+0000) 2019-10-13T13:24:11.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:11.176+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.180209+0000) 2019-10-13T13:24:11.184 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:11.180+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.182023+0000) 2019-10-13T13:24:11.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:11.180+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.183242+0000) 2019-10-13T13:24:11.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:11.180+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.183513+0000) 2019-10-13T13:24:11.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:11.180+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.183871+0000) 2019-10-13T13:24:11.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:11.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.184479+0000) 2019-10-13T13:24:11.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:11.180+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.184752+0000) 2019-10-13T13:24:11.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:11.184+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:11.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:11.186+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.188972+0000) 2019-10-13T13:24:11.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:11.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:11.956507+0000) 2019-10-13T13:24:14.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:14.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.283249+0000) 2019-10-13T13:24:14.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:14.568+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.568835+0000) 2019-10-13T13:24:14.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:14.936+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:14.936494+0000) 2019-10-13T13:24:16.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:16.188+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.192471+0000) 2019-10-13T13:24:16.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:16.190+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.193483+0000) 2019-10-13T13:24:16.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:16.188+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.192943+0000) 2019-10-13T13:24:16.196 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:16.188+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.193662+0000) 2019-10-13T13:24:16.196 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:16.192+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.194315+0000) 2019-10-13T13:24:16.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:16.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.194785+0000) 2019-10-13T13:24:16.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:16.192+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.195098+0000) 2019-10-13T13:24:16.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:16.196+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:16.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:16.198+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:16.200457+0000) 2019-10-13T13:24:17.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:17.410+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:17.412454+0000) 2019-10-13T13:24:17.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:17.936+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:17.940354+0000) 2019-10-13T13:24:18.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:18.160+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:18.165212+0000) 2019-10-13T13:24:18.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:18.700+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:18.705167+0000) 2019-10-13T13:24:21.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:21.200+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.204009+0000) 2019-10-13T13:24:21.208 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:21.200+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.205578+0000) 2019-10-13T13:24:21.208 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:21.204+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.205817+0000) 2019-10-13T13:24:21.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:21.204+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.206815+0000) 2019-10-13T13:24:21.209 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:21.202+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.207448+0000) 2019-10-13T13:24:21.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:21.204+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.207683+0000) 2019-10-13T13:24:21.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:21.204+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.208417+0000) 2019-10-13T13:24:21.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:21.208+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:21.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:21.210+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.213738+0000) 2019-10-13T13:24:21.912 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:24:21.912 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:21.916 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:24:21.916 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:21.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:21.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:21.956679+0000) 2019-10-13T13:24:24.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:24.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.283446+0000) 2019-10-13T13:24:24.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:24.568+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.569071+0000) 2019-10-13T13:24:24.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:24.932+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:24.936717+0000) 2019-10-13T13:24:26.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:26.212+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.216202+0000) 2019-10-13T13:24:26.219 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:26.212+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.216960+0000) 2019-10-13T13:24:26.219 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:26.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.217499+0000) 2019-10-13T13:24:26.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:26.214+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.218783+0000) 2019-10-13T13:24:26.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:26.216+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.220419+0000) 2019-10-13T13:24:26.223 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:26.216+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.220560+0000) 2019-10-13T13:24:26.223 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:26.216+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.221010+0000) 2019-10-13T13:24:26.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:26.222+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:26.224370+0000) 2019-10-13T13:24:26.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:26.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:27.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:27.414+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:27.419116+0000) 2019-10-13T13:24:27.942 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:27.940+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:27.940554+0000) 2019-10-13T13:24:28.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:28.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:28.165447+0000) 2019-10-13T13:24:28.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:28.704+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:28.705358+0000) 2019-10-13T13:24:31.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:31.228+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.230314+0000) 2019-10-13T13:24:31.233 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:31.228+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.230706+0000) 2019-10-13T13:24:31.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:31.228+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.231404+0000) 2019-10-13T13:24:31.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:31.226+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.232058+0000) 2019-10-13T13:24:31.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:31.228+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.232181+0000) 2019-10-13T13:24:31.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:31.228+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.232498+0000) 2019-10-13T13:24:31.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:31.228+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.233551+0000) 2019-10-13T13:24:31.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:31.236+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:31.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:31.238+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.240319+0000) 2019-10-13T13:24:31.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:31.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:31.956856+0000) 2019-10-13T13:24:34.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:34.280+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.283642+0000) 2019-10-13T13:24:34.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:34.568+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.569317+0000) 2019-10-13T13:24:34.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:34.932+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:34.936951+0000) 2019-10-13T13:24:36.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:36.236+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.241310+0000) 2019-10-13T13:24:36.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:36.240+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.241864+0000) 2019-10-13T13:24:36.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:36.240+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.244309+0000) 2019-10-13T13:24:36.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:36.240+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.244726+0000) 2019-10-13T13:24:36.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:36.240+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.245298+0000) 2019-10-13T13:24:36.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:36.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.245870+0000) 2019-10-13T13:24:36.248 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:36.244+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.246406+0000) 2019-10-13T13:24:36.251 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:36.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:36.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:36.246+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:36.250465+0000) 2019-10-13T13:24:37.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:37.422+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:37.425791+0000) 2019-10-13T13:24:37.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:37.939+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:37.940752+0000) 2019-10-13T13:24:38.167 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:38.159+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:38.165636+0000) 2019-10-13T13:24:38.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:38.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:38.705516+0000) 2019-10-13T13:24:41.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:41.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.253032+0000) 2019-10-13T13:24:41.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:41.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.253531+0000) 2019-10-13T13:24:41.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:41.251+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.254140+0000) 2019-10-13T13:24:41.256 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:41.251+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.254196+0000) 2019-10-13T13:24:41.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:41.251+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.254951+0000) 2019-10-13T13:24:41.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:41.250+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.255739+0000) 2019-10-13T13:24:41.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:41.251+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.256638+0000) 2019-10-13T13:24:41.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:41.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:41.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:41.258+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.264061+0000) 2019-10-13T13:24:41.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:41.954+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:41.957025+0000) 2019-10-13T13:24:44.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:44.279+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.283844+0000) 2019-10-13T13:24:44.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:44.567+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.569525+0000) 2019-10-13T13:24:44.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:44.931+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:44.937184+0000) 2019-10-13T13:24:46.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:46.263+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.266163+0000) 2019-10-13T13:24:46.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:46.263+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.266641+0000) 2019-10-13T13:24:46.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:46.263+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.268494+0000) 2019-10-13T13:24:46.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:46.267+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.269777+0000) 2019-10-13T13:24:46.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:46.263+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.269207+0000) 2019-10-13T13:24:46.272 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:46.265+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.270639+0000) 2019-10-13T13:24:46.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:46.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.270665+0000) 2019-10-13T13:24:46.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:46.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:46.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:46.273+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:46.276425+0000) 2019-10-13T13:24:47.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:47.429+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:47.432628+0000) 2019-10-13T13:24:47.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:47.939+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:47.940958+0000) 2019-10-13T13:24:48.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:48.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:48.165832+0000) 2019-10-13T13:24:48.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:48.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:48.705771+0000) 2019-10-13T13:24:51.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:51.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.278885+0000) 2019-10-13T13:24:51.283 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:51.275+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.280870+0000) 2019-10-13T13:24:51.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:51.275+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.281210+0000) 2019-10-13T13:24:51.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:51.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.281780+0000) 2019-10-13T13:24:51.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:51.279+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.282341+0000) 2019-10-13T13:24:51.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:51.279+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.282518+0000) 2019-10-13T13:24:51.285 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:51.277+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.283188+0000) 2019-10-13T13:24:51.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:51.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:51.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:51.285+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.289045+0000) 2019-10-13T13:24:51.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:51.953+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:51.957220+0000) 2019-10-13T13:24:52.045 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:24:52.045 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:52.049 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:24:52.050 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:24:54.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:54.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.284057+0000) 2019-10-13T13:24:54.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:54.567+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.569798+0000) 2019-10-13T13:24:54.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:54.935+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:54.937418+0000) 2019-10-13T13:24:56.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:24:56.287+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.291567+0000) 2019-10-13T13:24:56.294 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:56.287+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.292307+0000) 2019-10-13T13:24:56.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:56.287+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.293048+0000) 2019-10-13T13:24:56.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:24:56.287+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.293329+0000) 2019-10-13T13:24:56.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:24:56.287+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.293566+0000) 2019-10-13T13:24:56.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:56.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.294562+0000) 2019-10-13T13:24:56.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:24:56.289+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.295500+0000) 2019-10-13T13:24:56.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:24:56.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:24:56.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:56.297+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:56.302529+0000) 2019-10-13T13:24:57.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:24:57.437+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:57.439177+0000) 2019-10-13T13:24:57.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:24:57.939+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:57.941156+0000) 2019-10-13T13:24:58.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:24:58.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:58.166066+0000) 2019-10-13T13:24:58.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:24:58.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:24:58.705995+0000) 2019-10-13T13:25:01.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:01.299+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.304188+0000) 2019-10-13T13:25:01.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:01.299+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.304680+0000) 2019-10-13T13:25:01.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:01.299+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.305384+0000) 2019-10-13T13:25:01.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:01.299+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.305651+0000) 2019-10-13T13:25:01.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:01.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.306225+0000) 2019-10-13T13:25:01.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:01.301+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.306954+0000) 2019-10-13T13:25:01.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:01.303+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.307673+0000) 2019-10-13T13:25:01.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:01.309+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.313049+0000) 2019-10-13T13:25:01.315 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:01.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:01.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:01.953+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:01.957388+0000) 2019-10-13T13:25:04.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:04.279+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.284256+0000) 2019-10-13T13:25:04.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:04.567+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.570001+0000) 2019-10-13T13:25:04.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:04.935+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:04.937650+0000) 2019-10-13T13:25:06.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:06.311+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.316052+0000) 2019-10-13T13:25:06.319 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:06.311+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.317372+0000) 2019-10-13T13:25:06.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:06.315+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.317910+0000) 2019-10-13T13:25:06.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:06.315+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.318069+0000) 2019-10-13T13:25:06.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:06.315+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.318400+0000) 2019-10-13T13:25:06.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:06.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.320448+0000) 2019-10-13T13:25:06.324 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:06.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.321967+0000) 2019-10-13T13:25:06.327 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:06.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:06.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:06.325+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:06.329374+0000) 2019-10-13T13:25:07.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:07.441+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:07.445749+0000) 2019-10-13T13:25:07.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:07.935+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:07.941354+0000) 2019-10-13T13:25:08.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:08.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:08.166267+0000) 2019-10-13T13:25:08.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:08.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:08.706232+0000) 2019-10-13T13:25:11.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:11.323+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.328410+0000) 2019-10-13T13:25:11.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:11.323+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.328995+0000) 2019-10-13T13:25:11.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:11.327+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.329904+0000) 2019-10-13T13:25:11.333 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:11.327+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.331776+0000) 2019-10-13T13:25:11.334 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:11.327+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.332309+0000) 2019-10-13T13:25:11.335 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:11.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.332952+0000) 2019-10-13T13:25:11.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:11.327+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.332889+0000) 2019-10-13T13:25:11.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:11.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:11.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:11.337+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.340739+0000) 2019-10-13T13:25:11.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:11.953+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:11.957593+0000) 2019-10-13T13:25:14.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:14.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.284454+0000) 2019-10-13T13:25:14.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:14.567+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.570218+0000) 2019-10-13T13:25:14.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:14.935+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:14.937855+0000) 2019-10-13T13:25:16.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:16.335+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.340719+0000) 2019-10-13T13:25:16.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:16.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.341360+0000) 2019-10-13T13:25:16.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:16.339+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.342100+0000) 2019-10-13T13:25:16.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:16.339+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.342227+0000) 2019-10-13T13:25:16.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:16.339+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.342638+0000) 2019-10-13T13:25:16.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:16.337+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.343703+0000) 2019-10-13T13:25:16.346 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:16.339+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.342916+0000) 2019-10-13T13:25:16.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:16.345+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:16.348201+0000) 2019-10-13T13:25:16.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:16.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:17.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:17.449+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:17.452411+0000) 2019-10-13T13:25:17.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:17.939+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:17.941553+0000) 2019-10-13T13:25:18.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:18.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:18.166471+0000) 2019-10-13T13:25:18.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:18.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:18.706447+0000) 2019-10-13T13:25:21.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:21.347+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.352339+0000) 2019-10-13T13:25:21.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:21.347+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.352688+0000) 2019-10-13T13:25:21.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:21.351+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.354708+0000) 2019-10-13T13:25:21.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:21.351+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.357070+0000) 2019-10-13T13:25:21.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:21.353+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.357597+0000) 2019-10-13T13:25:21.361 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:21.355+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.358683+0000) 2019-10-13T13:25:21.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:21.355+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.358707+0000) 2019-10-13T13:25:21.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:21.355+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182283 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:21.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:21.365+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.368419+0000) 2019-10-13T13:25:21.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:21.953+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:21.957780+0000) 2019-10-13T13:25:22.147 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:25:22.147 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:22.152 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:25:22.152 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:24.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:24.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.284668+0000) 2019-10-13T13:25:24.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:24.567+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.570425+0000) 2019-10-13T13:25:24.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:24.935+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:24.938076+0000) 2019-10-13T13:25:26.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:26.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.366182+0000) 2019-10-13T13:25:26.370 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:26.363+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.368471+0000) 2019-10-13T13:25:26.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:26.365+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.369086+0000) 2019-10-13T13:25:26.371 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:26.363+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.369266+0000) 2019-10-13T13:25:26.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:26.367+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.370947+0000) 2019-10-13T13:25:26.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:26.367+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.371064+0000) 2019-10-13T13:25:26.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:26.367+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.371826+0000) 2019-10-13T13:25:26.376 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:26.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:26.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:26.373+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:26.378739+0000) 2019-10-13T13:25:27.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:27.457+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:27.458877+0000) 2019-10-13T13:25:27.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:27.939+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:27.941778+0000) 2019-10-13T13:25:28.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:28.163+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:28.166683+0000) 2019-10-13T13:25:28.708 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:28.703+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:28.706641+0000) 2019-10-13T13:25:31.379 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:31.371+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.377426+0000) 2019-10-13T13:25:31.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:31.375+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.378828+0000) 2019-10-13T13:25:31.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:31.375+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.379037+0000) 2019-10-13T13:25:31.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:31.377+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.380437+0000) 2019-10-13T13:25:31.382 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:31.375+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.380680+0000) 2019-10-13T13:25:31.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:31.375+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.381462+0000) 2019-10-13T13:25:31.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:31.375+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.381670+0000) 2019-10-13T13:25:31.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:31.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:31.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:31.385+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.388739+0000) 2019-10-13T13:25:31.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:31.957+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:31.957951+0000) 2019-10-13T13:25:34.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:34.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.284885+0000) 2019-10-13T13:25:34.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:34.566+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.570656+0000) 2019-10-13T13:25:34.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:34.934+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:34.938299+0000) 2019-10-13T13:25:36.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:36.386+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.389962+0000) 2019-10-13T13:25:36.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:36.385+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.390554+0000) 2019-10-13T13:25:36.393 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:36.386+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.390495+0000) 2019-10-13T13:25:36.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:36.386+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.391453+0000) 2019-10-13T13:25:36.395 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:36.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.392243+0000) 2019-10-13T13:25:36.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:36.386+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.392591+0000) 2019-10-13T13:25:36.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:36.386+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.392943+0000) 2019-10-13T13:25:36.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:36.393+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:36.396473+0000) 2019-10-13T13:25:36.401 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:36.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:37.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:37.461+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:37.465419+0000) 2019-10-13T13:25:37.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:37.938+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:37.941972+0000) 2019-10-13T13:25:38.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:38.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:38.166907+0000) 2019-10-13T13:25:38.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:38.702+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:38.706840+0000) 2019-10-13T13:25:41.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:41.401+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.405486+0000) 2019-10-13T13:25:41.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:41.398+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.405557+0000) 2019-10-13T13:25:41.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:41.402+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.405818+0000) 2019-10-13T13:25:41.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:41.402+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.406032+0000) 2019-10-13T13:25:41.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:41.402+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.406822+0000) 2019-10-13T13:25:41.411 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:41.402+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.409233+0000) 2019-10-13T13:25:41.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:41.406+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.409940+0000) 2019-10-13T13:25:41.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:41.406+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:41.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:41.409+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.413162+0000) 2019-10-13T13:25:41.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:41.957+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:41.958125+0000) 2019-10-13T13:25:44.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:44.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.285125+0000) 2019-10-13T13:25:44.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:44.570+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.570862+0000) 2019-10-13T13:25:44.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:44.934+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:44.938498+0000) 2019-10-13T13:25:46.418 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:46.410+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.415869+0000) 2019-10-13T13:25:46.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:46.410+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.416232+0000) 2019-10-13T13:25:46.418 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:46.410+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.416699+0000) 2019-10-13T13:25:46.419 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:46.413+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.417437+0000) 2019-10-13T13:25:46.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:46.410+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.417598+0000) 2019-10-13T13:25:46.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:46.414+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.418642+0000) 2019-10-13T13:25:46.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:46.414+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.418231+0000) 2019-10-13T13:25:46.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:46.417+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:46.421932+0000) 2019-10-13T13:25:46.426 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:46.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:47.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:47.469+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:47.471419+0000) 2019-10-13T13:25:47.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:47.938+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:47.942178+0000) 2019-10-13T13:25:48.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:48.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:48.167164+0000) 2019-10-13T13:25:48.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:48.702+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:48.707089+0000) 2019-10-13T13:25:51.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:51.422+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.427227+0000) 2019-10-13T13:25:51.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:51.422+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.428403+0000) 2019-10-13T13:25:51.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:51.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.429639+0000) 2019-10-13T13:25:51.432 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:51.426+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.429950+0000) 2019-10-13T13:25:51.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:51.426+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.430364+0000) 2019-10-13T13:25:51.433 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:51.426+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.430961+0000) 2019-10-13T13:25:51.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:51.424+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.431416+0000) 2019-10-13T13:25:51.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:51.432+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.436161+0000) 2019-10-13T13:25:51.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:51.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:51.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:51.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:51.958251+0000) 2019-10-13T13:25:52.283 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:25:52.284 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:52.287 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:25:52.288 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:25:54.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:54.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.285355+0000) 2019-10-13T13:25:54.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:54.570+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.571097+0000) 2019-10-13T13:25:54.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:54.938+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:54.938700+0000) 2019-10-13T13:25:56.444 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:25:56.438+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.442196+0000) 2019-10-13T13:25:56.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:56.438+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.442566+0000) 2019-10-13T13:25:56.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:56.438+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.444001+0000) 2019-10-13T13:25:56.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:56.438+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.444264+0000) 2019-10-13T13:25:56.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:25:56.438+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.444187+0000) 2019-10-13T13:25:56.447 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:25:56.440+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.445289+0000) 2019-10-13T13:25:56.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:25:56.438+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.444650+0000) 2019-10-13T13:25:56.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:25:56.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:25:56.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:56.444+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:56.451785+0000) 2019-10-13T13:25:57.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:25:57.476+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:57.478453+0000) 2019-10-13T13:25:57.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:25:57.938+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:57.942378+0000) 2019-10-13T13:25:58.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:25:58.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:58.167363+0000) 2019-10-13T13:25:58.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:25:58.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:25:58.707276+0000) 2019-10-13T13:26:01.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:01.446+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.453300+0000) 2019-10-13T13:26:01.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:01.448+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.453870+0000) 2019-10-13T13:26:01.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:01.450+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.454651+0000) 2019-10-13T13:26:01.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:01.450+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.454949+0000) 2019-10-13T13:26:01.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:01.450+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.455525+0000) 2019-10-13T13:26:01.459 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:01.450+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.457050+0000) 2019-10-13T13:26:01.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:01.450+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.456272+0000) 2019-10-13T13:26:01.464 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:01.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:01.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:01.456+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.463700+0000) 2019-10-13T13:26:01.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:01.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:01.958422+0000) 2019-10-13T13:26:04.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:04.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.285553+0000) 2019-10-13T13:26:04.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:04.570+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.571299+0000) 2019-10-13T13:26:04.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:04.938+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:04.938929+0000) 2019-10-13T13:26:06.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:06.462+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.466775+0000) 2019-10-13T13:26:06.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:06.462+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.467380+0000) 2019-10-13T13:26:06.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:06.462+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.467614+0000) 2019-10-13T13:26:06.471 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:06.466+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.469884+0000) 2019-10-13T13:26:06.474 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:06.468+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.472587+0000) 2019-10-13T13:26:06.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:06.468+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.474338+0000) 2019-10-13T13:26:06.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:06.470+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.474183+0000) 2019-10-13T13:26:06.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:06.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182561 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:06.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:06.470+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:06.475319+0000) 2019-10-13T13:26:07.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:07.480+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:07.486094+0000) 2019-10-13T13:26:07.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:07.942+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:07.942576+0000) 2019-10-13T13:26:08.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:08.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:08.167554+0000) 2019-10-13T13:26:08.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:08.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:08.707504+0000) 2019-10-13T13:26:11.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:11.474+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.479538+0000) 2019-10-13T13:26:11.482 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:11.474+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.480281+0000) 2019-10-13T13:26:11.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:11.474+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.481130+0000) 2019-10-13T13:26:11.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:11.478+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.481942+0000) 2019-10-13T13:26:11.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:11.478+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.482386+0000) 2019-10-13T13:26:11.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:11.478+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.482804+0000) 2019-10-13T13:26:11.486 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:11.480+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.484763+0000) 2019-10-13T13:26:11.490 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:11.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:11.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:11.484+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.489497+0000) 2019-10-13T13:26:11.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:11.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:11.958616+0000) 2019-10-13T13:26:14.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:14.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.285783+0000) 2019-10-13T13:26:14.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:14.570+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.571528+0000) 2019-10-13T13:26:14.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:14.938+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:14.939156+0000) 2019-10-13T13:26:16.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:16.486+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.491179+0000) 2019-10-13T13:26:16.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:16.486+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.491595+0000) 2019-10-13T13:26:16.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:16.486+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.492203+0000) 2019-10-13T13:26:16.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:16.486+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.492645+0000) 2019-10-13T13:26:16.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:16.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.493158+0000) 2019-10-13T13:26:16.496 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:16.488+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.494429+0000) 2019-10-13T13:26:16.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:16.490+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.494327+0000) 2019-10-13T13:26:16.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:16.492+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:16.499618+0000) 2019-10-13T13:26:16.503 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:16.494+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:17.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:17.488+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:17.492694+0000) 2019-10-13T13:26:17.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:17.942+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:17.942775+0000) 2019-10-13T13:26:18.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:18.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:18.167751+0000) 2019-10-13T13:26:18.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:18.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:18.707736+0000) 2019-10-13T13:26:21.505 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:21.498+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.503383+0000) 2019-10-13T13:26:21.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:21.498+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.504616+0000) 2019-10-13T13:26:21.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:21.500+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.505690+0000) 2019-10-13T13:26:21.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:21.502+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.505811+0000) 2019-10-13T13:26:21.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:21.502+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.506338+0000) 2019-10-13T13:26:21.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:21.502+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.507251+0000) 2019-10-13T13:26:21.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:21.502+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.507305+0000) 2019-10-13T13:26:21.516 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:21.508+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.514023+0000) 2019-10-13T13:26:21.516 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:21.510+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:21.960 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:21.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:21.958802+0000) 2019-10-13T13:26:22.407 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:26:22.407 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:22.411 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:26:22.411 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:24.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:24.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.285961+0000) 2019-10-13T13:26:24.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:24.570+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.571730+0000) 2019-10-13T13:26:24.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:24.934+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:24.939344+0000) 2019-10-13T13:26:26.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:26.514+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.517238+0000) 2019-10-13T13:26:26.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:26.514+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.518312+0000) 2019-10-13T13:26:26.520 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:26.514+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.518628+0000) 2019-10-13T13:26:26.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:26.512+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.519370+0000) 2019-10-13T13:26:26.521 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:26.514+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.519627+0000) 2019-10-13T13:26:26.522 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:26.514+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.519670+0000) 2019-10-13T13:26:26.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:26.514+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.520289+0000) 2019-10-13T13:26:26.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:26.520+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:26.526643+0000) 2019-10-13T13:26:26.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:26.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:27.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:27.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:27.499278+0000) 2019-10-13T13:26:27.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:27.942+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:27.942978+0000) 2019-10-13T13:26:28.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:28.162+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:28.167948+0000) 2019-10-13T13:26:28.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:28.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:28.707925+0000) 2019-10-13T13:26:31.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:31.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.530692+0000) 2019-10-13T13:26:31.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:31.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.531436+0000) 2019-10-13T13:26:31.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:31.525+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.531891+0000) 2019-10-13T13:26:31.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:31.525+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.531684+0000) 2019-10-13T13:26:31.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:31.525+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.532203+0000) 2019-10-13T13:26:31.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:31.525+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.532689+0000) 2019-10-13T13:26:31.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:31.525+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.533483+0000) 2019-10-13T13:26:31.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:31.532+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.537221+0000) 2019-10-13T13:26:31.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:31.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:31.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:31.952+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:31.959014+0000) 2019-10-13T13:26:34.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:34.281+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.286119+0000) 2019-10-13T13:26:34.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:34.569+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.571973+0000) 2019-10-13T13:26:34.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:34.937+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:34.939544+0000) 2019-10-13T13:26:36.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:36.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.544039+0000) 2019-10-13T13:26:36.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:36.537+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.544706+0000) 2019-10-13T13:26:36.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:36.540+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.546089+0000) 2019-10-13T13:26:36.548 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:36.541+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.546184+0000) 2019-10-13T13:26:36.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:36.541+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.546542+0000) 2019-10-13T13:26:36.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:36.541+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.546672+0000) 2019-10-13T13:26:36.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:36.541+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.547134+0000) 2019-10-13T13:26:36.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:36.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182746 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:36.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:36.548+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:36.553888+0000) 2019-10-13T13:26:37.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:37.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:37.505881+0000) 2019-10-13T13:26:37.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:37.941+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:37.943211+0000) 2019-10-13T13:26:38.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:38.161+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:38.168145+0000) 2019-10-13T13:26:38.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:38.705+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:38.708124+0000) 2019-10-13T13:26:41.557 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:41.549+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.555067+0000) 2019-10-13T13:26:41.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:41.548+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.555912+0000) 2019-10-13T13:26:41.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:41.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.556479+0000) 2019-10-13T13:26:41.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:41.549+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.557190+0000) 2019-10-13T13:26:41.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:41.553+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.558019+0000) 2019-10-13T13:26:41.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:41.553+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.558377+0000) 2019-10-13T13:26:41.561 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:41.553+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.559054+0000) 2019-10-13T13:26:41.564 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:41.556+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.562256+0000) 2019-10-13T13:26:41.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:41.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182780 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:41.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:41.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:41.959187+0000) 2019-10-13T13:26:44.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:44.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.286336+0000) 2019-10-13T13:26:44.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:44.569+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.572228+0000) 2019-10-13T13:26:44.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:44.937+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:44.939755+0000) 2019-10-13T13:26:46.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:46.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.567431+0000) 2019-10-13T13:26:46.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:46.565+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.567664+0000) 2019-10-13T13:26:46.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:46.565+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.568264+0000) 2019-10-13T13:26:46.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:46.565+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.569520+0000) 2019-10-13T13:26:46.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:46.569+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.570008+0000) 2019-10-13T13:26:46.573 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:46.569+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.570239+0000) 2019-10-13T13:26:46.573 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:46.569+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.570900+0000) 2019-10-13T13:26:46.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:46.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182809 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:46.577 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:46.572+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:46.575827+0000) 2019-10-13T13:26:47.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:47.512+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:47.512375+0000) 2019-10-13T13:26:47.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:47.941+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:47.943415+0000) 2019-10-13T13:26:48.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:48.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:48.168356+0000) 2019-10-13T13:26:48.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:48.705+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:48.708319+0000) 2019-10-13T13:26:51.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:51.577+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.578746+0000) 2019-10-13T13:26:51.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:51.577+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.579516+0000) 2019-10-13T13:26:51.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:51.576+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.579986+0000) 2019-10-13T13:26:51.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:51.577+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.580290+0000) 2019-10-13T13:26:51.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:51.577+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.580104+0000) 2019-10-13T13:26:51.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:51.577+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.581436+0000) 2019-10-13T13:26:51.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:51.581+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.582061+0000) 2019-10-13T13:26:51.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:51.588+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.588207+0000) 2019-10-13T13:26:51.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:51.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:51.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:51.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:51.959390+0000) 2019-10-13T13:26:52.522 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:26:52.522 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:52.525 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:26:52.526 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:26:54.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:54.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.286518+0000) 2019-10-13T13:26:54.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:54.569+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.572447+0000) 2019-10-13T13:26:54.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:54.937+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:54.939969+0000) 2019-10-13T13:26:56.594 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:26:56.589+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.591646+0000) 2019-10-13T13:26:56.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:56.589+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.592085+0000) 2019-10-13T13:26:56.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:26:56.589+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.593593+0000) 2019-10-13T13:26:56.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:56.593+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.594696+0000) 2019-10-13T13:26:56.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:56.593+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.596517+0000) 2019-10-13T13:26:56.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:26:56.595+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.597341+0000) 2019-10-13T13:26:56.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:26:56.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.597531+0000) 2019-10-13T13:26:56.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:26:56.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:26:56.603 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:56.599+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:56.601540+0000) 2019-10-13T13:26:57.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:26:57.515+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:57.518332+0000) 2019-10-13T13:26:57.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:26:57.941+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:57.943631+0000) 2019-10-13T13:26:58.170 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:26:58.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:58.168605+0000) 2019-10-13T13:26:58.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:26:58.705+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:26:58.708509+0000) 2019-10-13T13:27:01.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:01.601+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.604516+0000) 2019-10-13T13:27:01.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:01.605+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.605903+0000) 2019-10-13T13:27:01.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:01.605+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.606688+0000) 2019-10-13T13:27:01.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:01.605+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.607076+0000) 2019-10-13T13:27:01.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:01.605+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.607378+0000) 2019-10-13T13:27:01.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:01.605+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.608170+0000) 2019-10-13T13:27:01.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:01.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.609419+0000) 2019-10-13T13:27:01.614 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:01.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182904 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:01.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:01.611+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.615792+0000) 2019-10-13T13:27:01.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:01.955+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:01.959583+0000) 2019-10-13T13:27:04.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:04.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.286737+0000) 2019-10-13T13:27:04.574 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:04.569+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.572662+0000) 2019-10-13T13:27:04.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:04.937+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:04.940170+0000) 2019-10-13T13:27:06.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:06.613+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.616159+0000) 2019-10-13T13:27:06.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:06.613+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.616478+0000) 2019-10-13T13:27:06.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:06.617+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.618553+0000) 2019-10-13T13:27:06.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:06.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.619026+0000) 2019-10-13T13:27:06.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:06.617+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.619416+0000) 2019-10-13T13:27:06.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:06.617+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.619740+0000) 2019-10-13T13:27:06.625 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:06.619+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.623852+0000) 2019-10-13T13:27:06.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:06.621+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182932 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:06.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:06.627+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:06.628958+0000) 2019-10-13T13:27:07.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:07.523+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:07.527596+0000) 2019-10-13T13:27:07.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:07.941+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:07.943832+0000) 2019-10-13T13:27:08.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:08.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:08.168799+0000) 2019-10-13T13:27:08.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:08.705+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:08.708704+0000) 2019-10-13T13:27:11.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:11.625+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.627534+0000) 2019-10-13T13:27:11.630 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:11.625+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.628192+0000) 2019-10-13T13:27:11.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:11.625+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.628569+0000) 2019-10-13T13:27:11.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:11.625+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.629067+0000) 2019-10-13T13:27:11.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:11.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.629326+0000) 2019-10-13T13:27:11.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:11.627+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.630674+0000) 2019-10-13T13:27:11.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:11.629+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.630667+0000) 2019-10-13T13:27:11.637 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:11.635+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.636106+0000) 2019-10-13T13:27:11.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:11.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:11.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:11.955+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:11.959733+0000) 2019-10-13T13:27:14.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:14.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.286934+0000) 2019-10-13T13:27:14.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:14.569+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.572864+0000) 2019-10-13T13:27:14.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:14.937+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:14.940383+0000) 2019-10-13T13:27:16.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:16.637+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.638843+0000) 2019-10-13T13:27:16.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:16.637+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.640344+0000) 2019-10-13T13:27:16.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:16.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.641036+0000) 2019-10-13T13:27:16.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:16.641+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.643311+0000) 2019-10-13T13:27:16.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:16.641+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.643230+0000) 2019-10-13T13:27:16.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:16.643+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.645568+0000) 2019-10-13T13:27:16.649 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:16.645+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.647276+0000) 2019-10-13T13:27:16.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:16.643+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:16.647648+0000) 2019-10-13T13:27:16.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:16.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:17.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:17.531+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:17.534181+0000) 2019-10-13T13:27:17.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:17.941+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:17.944067+0000) 2019-10-13T13:27:18.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:18.165+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:18.168998+0000) 2019-10-13T13:27:18.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:18.705+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:18.708906+0000) 2019-10-13T13:27:21.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:21.649+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.651202+0000) 2019-10-13T13:27:21.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:21.649+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.651654+0000) 2019-10-13T13:27:21.654 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:21.649+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.651853+0000) 2019-10-13T13:27:21.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:21.649+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.653326+0000) 2019-10-13T13:27:21.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:21.653+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.654082+0000) 2019-10-13T13:27:21.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:21.651+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.654799+0000) 2019-10-13T13:27:21.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:21.653+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.655066+0000) 2019-10-13T13:27:21.661 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:21.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:21.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:21.659+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.661525+0000) 2019-10-13T13:27:21.961 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:21.955+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:21.959902+0000) 2019-10-13T13:27:22.635 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:27:22.635 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:22.639 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:27:22.639 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:24.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:24.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.287138+0000) 2019-10-13T13:27:24.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:24.572+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.573071+0000) 2019-10-13T13:27:24.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:24.936+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:24.940609+0000) 2019-10-13T13:27:26.665 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:26.660+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.663629+0000) 2019-10-13T13:27:26.666 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:26.660+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.664625+0000) 2019-10-13T13:27:26.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:26.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.664890+0000) 2019-10-13T13:27:26.667 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:26.660+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.665098+0000) 2019-10-13T13:27:26.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:26.663+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.666027+0000) 2019-10-13T13:27:26.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:26.664+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.666117+0000) 2019-10-13T13:27:26.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:26.664+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.666382+0000) 2019-10-13T13:27:26.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:26.667+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:26.670689+0000) 2019-10-13T13:27:26.673 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:26.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:27.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:27.539+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:27.540928+0000) 2019-10-13T13:27:27.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:27.940+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:27.944283+0000) 2019-10-13T13:27:28.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:28.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:28.169198+0000) 2019-10-13T13:27:28.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:28.708+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:28.709099+0000) 2019-10-13T13:27:31.677 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:31.672+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.674893+0000) 2019-10-13T13:27:31.679 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:31.675+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.677179+0000) 2019-10-13T13:27:31.680 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:31.676+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.678061+0000) 2019-10-13T13:27:31.680 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:31.676+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.678218+0000) 2019-10-13T13:27:31.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:31.676+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.680320+0000) 2019-10-13T13:27:31.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:31.676+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.680492+0000) 2019-10-13T13:27:31.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:31.676+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.681309+0000) 2019-10-13T13:27:31.686 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:31.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:31.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:31.683+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.686609+0000) 2019-10-13T13:27:31.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:31.959+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:31.960072+0000) 2019-10-13T13:27:34.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:34.284+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.287340+0000) 2019-10-13T13:27:34.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:34.572+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.573317+0000) 2019-10-13T13:27:34.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:34.936+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:34.940813+0000) 2019-10-13T13:27:36.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:36.684+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.687291+0000) 2019-10-13T13:27:36.690 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:36.684+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.687697+0000) 2019-10-13T13:27:36.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:36.687+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.688788+0000) 2019-10-13T13:27:36.691 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:36.684+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.688817+0000) 2019-10-13T13:27:36.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:36.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.689777+0000) 2019-10-13T13:27:36.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:36.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.690221+0000) 2019-10-13T13:27:36.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:36.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.690323+0000) 2019-10-13T13:27:36.698 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:36.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:36.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:36.695+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:36.697695+0000) 2019-10-13T13:27:37.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:37.543+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:37.547485+0000) 2019-10-13T13:27:37.946 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:37.940+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:37.944481+0000) 2019-10-13T13:27:38.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:38.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:38.169387+0000) 2019-10-13T13:27:38.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:38.704+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:38.709313+0000) 2019-10-13T13:27:41.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:41.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.699438+0000) 2019-10-13T13:27:41.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:41.696+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.700273+0000) 2019-10-13T13:27:41.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:41.700+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.702557+0000) 2019-10-13T13:27:41.705 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:41.699+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.703865+0000) 2019-10-13T13:27:41.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:41.700+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.703876+0000) 2019-10-13T13:27:41.706 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:41.700+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.704236+0000) 2019-10-13T13:27:41.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:41.700+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.704639+0000) 2019-10-13T13:27:41.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:41.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183151 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:41.713 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:41.707+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.711770+0000) 2019-10-13T13:27:41.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:41.959+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:41.960304+0000) 2019-10-13T13:27:44.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:44.284+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.287540+0000) 2019-10-13T13:27:44.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:44.568+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.573550+0000) 2019-10-13T13:27:44.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:44.936+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:44.941047+0000) 2019-10-13T13:27:46.715 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:46.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.712932+0000) 2019-10-13T13:27:46.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:46.712+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.714194+0000) 2019-10-13T13:27:46.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:46.712+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.714564+0000) 2019-10-13T13:27:46.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:46.711+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.715352+0000) 2019-10-13T13:27:46.717 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:46.712+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.715428+0000) 2019-10-13T13:27:46.718 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:46.712+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.715770+0000) 2019-10-13T13:27:46.718 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:46.712+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.716958+0000) 2019-10-13T13:27:46.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:46.719+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:46.720408+0000) 2019-10-13T13:27:46.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:46.720+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183180 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:47.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:47.551+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:47.553944+0000) 2019-10-13T13:27:47.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:47.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:47.944678+0000) 2019-10-13T13:27:48.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:48.164+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:48.169589+0000) 2019-10-13T13:27:48.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:48.704+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:48.709508+0000) 2019-10-13T13:27:51.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:51.720+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.725573+0000) 2019-10-13T13:27:51.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:51.724+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.726109+0000) 2019-10-13T13:27:51.729 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:51.724+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.727013+0000) 2019-10-13T13:27:51.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:51.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.727584+0000) 2019-10-13T13:27:51.730 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:51.727+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.728159+0000) 2019-10-13T13:27:51.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:51.724+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.728317+0000) 2019-10-13T13:27:51.731 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:51.724+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.729225+0000) 2019-10-13T13:27:51.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:51.731+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.733030+0000) 2019-10-13T13:27:51.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:51.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:51.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:51.959+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:51.960533+0000) 2019-10-13T13:27:53.008 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:27:53.008 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:53.013 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:27:53.013 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:27:54.289 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:54.284+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.287684+0000) 2019-10-13T13:27:54.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:54.572+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.573769+0000) 2019-10-13T13:27:54.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:54.936+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:54.941247+0000) 2019-10-13T13:27:56.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:56.736+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.738536+0000) 2019-10-13T13:27:56.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:27:56.735+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.739238+0000) 2019-10-13T13:27:56.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:27:56.736+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.740741+0000) 2019-10-13T13:27:56.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:27:56.736+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.740982+0000) 2019-10-13T13:27:56.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:56.736+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.741433+0000) 2019-10-13T13:27:56.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:56.740+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.741824+0000) 2019-10-13T13:27:56.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:27:56.740+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.742410+0000) 2019-10-13T13:27:56.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:27:56.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:27:56.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:56.747+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:56.749957+0000) 2019-10-13T13:27:57.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:27:57.555+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:57.559963+0000) 2019-10-13T13:27:57.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:27:57.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:57.944873+0000) 2019-10-13T13:27:58.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:27:58.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:58.169782+0000) 2019-10-13T13:27:58.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:27:58.704+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:27:58.709700+0000) 2019-10-13T13:28:01.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:01.748+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.751352+0000) 2019-10-13T13:28:01.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:01.748+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.751748+0000) 2019-10-13T13:28:01.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:01.748+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.752946+0000) 2019-10-13T13:28:01.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:01.748+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.753540+0000) 2019-10-13T13:28:01.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:01.750+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.753789+0000) 2019-10-13T13:28:01.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:01.752+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.754312+0000) 2019-10-13T13:28:01.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:01.752+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.754782+0000) 2019-10-13T13:28:01.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:01.754+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.757407+0000) 2019-10-13T13:28:01.761 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:01.756+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183275 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:01.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:01.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:01.960720+0000) 2019-10-13T13:28:04.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:04.284+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.287820+0000) 2019-10-13T13:28:04.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:04.572+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.573974+0000) 2019-10-13T13:28:04.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:04.940+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:04.941452+0000) 2019-10-13T13:28:06.764 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:06.760+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.762584+0000) 2019-10-13T13:28:06.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:06.760+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.764723+0000) 2019-10-13T13:28:06.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:06.760+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.765220+0000) 2019-10-13T13:28:06.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:06.760+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.765427+0000) 2019-10-13T13:28:06.768 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:06.764+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.765884+0000) 2019-10-13T13:28:06.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:06.764+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.767943+0000) 2019-10-13T13:28:06.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:06.766+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.768450+0000) 2019-10-13T13:28:06.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:06.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:06.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:06.778+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:06.782471+0000) 2019-10-13T13:28:07.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:07.562+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:07.565953+0000) 2019-10-13T13:28:07.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:07.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:07.945079+0000) 2019-10-13T13:28:08.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:08.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:08.170006+0000) 2019-10-13T13:28:08.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:08.708+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:08.709932+0000) 2019-10-13T13:28:11.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:11.772+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.776147+0000) 2019-10-13T13:28:11.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:11.772+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.776523+0000) 2019-10-13T13:28:11.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:11.774+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.777515+0000) 2019-10-13T13:28:11.780 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:11.776+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.778403+0000) 2019-10-13T13:28:11.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:11.776+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.779049+0000) 2019-10-13T13:28:11.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:11.776+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.779112+0000) 2019-10-13T13:28:11.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:11.776+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.779398+0000) 2019-10-13T13:28:11.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:11.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183336 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:11.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:11.782+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.786559+0000) 2019-10-13T13:28:11.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:11.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:11.960907+0000) 2019-10-13T13:28:14.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:14.284+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:14.288024+0000) 2019-10-13T13:28:14.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:14.572+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:14.574195+0000) 2019-10-13T13:28:14.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:14.940+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:14.941655+0000) 2019-10-13T13:28:16.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:16.786+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.788371+0000) 2019-10-13T13:28:16.793 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:16.788+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.791757+0000) 2019-10-13T13:28:16.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:16.788+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.791433+0000) 2019-10-13T13:28:16.794 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:16.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.792294+0000) 2019-10-13T13:28:16.795 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:16.788+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.792972+0000) 2019-10-13T13:28:16.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:16.792+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.795232+0000) 2019-10-13T13:28:16.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:16.792+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.795809+0000) 2019-10-13T13:28:16.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:16.796+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:16.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:16.798+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:16.803584+0000) 2019-10-13T13:28:17.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:17.570+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:17.572387+0000) 2019-10-13T13:28:17.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:17.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:17.945309+0000) 2019-10-13T13:28:18.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:18.168+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:18.170237+0000) 2019-10-13T13:28:18.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:18.708+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:18.710130+0000) 2019-10-13T13:28:21.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:21.795+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.800961+0000) 2019-10-13T13:28:21.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:21.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.802606+0000) 2019-10-13T13:28:21.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:21.798+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.803102+0000) 2019-10-13T13:28:21.805 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:21.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.803356+0000) 2019-10-13T13:28:21.805 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:21.799+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.803415+0000) 2019-10-13T13:28:21.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:21.799+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.804352+0000) 2019-10-13T13:28:21.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:21.799+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.805449+0000) 2019-10-13T13:28:21.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:21.806+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.810427+0000) 2019-10-13T13:28:21.812 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:21.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183397 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:21.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:21.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:21.961090+0000) 2019-10-13T13:28:23.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:28:23.105 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:23.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:28:23.109 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:24.290 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:24.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:24.288259+0000) 2019-10-13T13:28:24.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:24.571+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:24.574406+0000) 2019-10-13T13:28:24.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:24.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:24.941855+0000) 2019-10-13T13:28:26.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:26.811+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.813856+0000) 2019-10-13T13:28:26.816 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:26.811+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.814182+0000) 2019-10-13T13:28:26.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:26.811+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.815514+0000) 2019-10-13T13:28:26.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:26.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.816028+0000) 2019-10-13T13:28:26.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:26.814+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.817235+0000) 2019-10-13T13:28:26.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:26.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.817294+0000) 2019-10-13T13:28:26.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:26.815+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.818240+0000) 2019-10-13T13:28:26.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:26.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:26.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:26.822+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:26.824413+0000) 2019-10-13T13:28:27.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:27.574+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:27.578877+0000) 2019-10-13T13:28:27.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:27.943+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:27.945537+0000) 2019-10-13T13:28:28.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:28.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:28.170474+0000) 2019-10-13T13:28:28.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:28.707+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:28.710340+0000) 2019-10-13T13:28:31.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:31.823+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.826913+0000) 2019-10-13T13:28:31.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:31.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.828675+0000) 2019-10-13T13:28:31.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:31.827+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.829729+0000) 2019-10-13T13:28:31.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:31.827+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.830769+0000) 2019-10-13T13:28:31.833 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:31.827+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.831380+0000) 2019-10-13T13:28:31.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:31.827+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.831677+0000) 2019-10-13T13:28:31.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:31.830+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.832349+0000) 2019-10-13T13:28:31.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:31.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183457 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:31.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:31.834+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.836381+0000) 2019-10-13T13:28:31.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:31.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:31.961262+0000) 2019-10-13T13:28:34.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:34.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:34.288489+0000) 2019-10-13T13:28:34.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:34.571+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:34.574604+0000) 2019-10-13T13:28:34.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:34.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:34.942055+0000) 2019-10-13T13:28:36.842 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:36.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.839641+0000) 2019-10-13T13:28:36.843 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:36.835+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.839691+0000) 2019-10-13T13:28:36.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:36.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.840251+0000) 2019-10-13T13:28:36.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:36.835+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.840360+0000) 2019-10-13T13:28:36.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:36.835+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.840657+0000) 2019-10-13T13:28:36.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:36.838+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.842389+0000) 2019-10-13T13:28:36.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:36.835+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.841276+0000) 2019-10-13T13:28:36.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:36.842+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:36.845574+0000) 2019-10-13T13:28:36.849 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:36.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:37.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:37.582+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:37.585545+0000) 2019-10-13T13:28:37.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:37.943+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:37.945782+0000) 2019-10-13T13:28:38.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:38.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:38.170711+0000) 2019-10-13T13:28:38.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:38.707+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:38.710536+0000) 2019-10-13T13:28:41.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:41.847+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.850245+0000) 2019-10-13T13:28:41.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:41.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.851225+0000) 2019-10-13T13:28:41.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:41.847+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.851587+0000) 2019-10-13T13:28:41.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:41.847+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.852027+0000) 2019-10-13T13:28:41.855 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:41.847+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.853173+0000) 2019-10-13T13:28:41.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:41.847+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.853502+0000) 2019-10-13T13:28:41.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:41.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.854277+0000) 2019-10-13T13:28:41.859 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:41.854+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.857457+0000) 2019-10-13T13:28:41.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:41.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:41.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:41.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:41.961467+0000) 2019-10-13T13:28:44.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:44.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:44.288685+0000) 2019-10-13T13:28:44.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:44.571+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:44.574837+0000) 2019-10-13T13:28:44.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:44.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:44.942262+0000) 2019-10-13T13:28:46.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:46.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.862670+0000) 2019-10-13T13:28:46.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:46.859+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.863035+0000) 2019-10-13T13:28:46.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:46.858+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.863632+0000) 2019-10-13T13:28:46.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:46.859+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.863755+0000) 2019-10-13T13:28:46.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:46.859+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.864011+0000) 2019-10-13T13:28:46.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:46.859+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.864666+0000) 2019-10-13T13:28:46.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:46.859+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.865161+0000) 2019-10-13T13:28:46.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:46.867+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:46.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:46.866+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:46.871925+0000) 2019-10-13T13:28:47.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:47.590+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:47.592124+0000) 2019-10-13T13:28:47.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:47.943+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:47.945996+0000) 2019-10-13T13:28:48.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:48.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:48.170913+0000) 2019-10-13T13:28:48.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:48.707+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:48.710752+0000) 2019-10-13T13:28:51.876 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:51.871+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.874613+0000) 2019-10-13T13:28:51.877 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:51.871+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.875286+0000) 2019-10-13T13:28:51.878 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:51.871+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.875945+0000) 2019-10-13T13:28:51.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:51.874+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.876957+0000) 2019-10-13T13:28:51.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:51.871+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.876924+0000) 2019-10-13T13:28:51.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:51.871+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.877035+0000) 2019-10-13T13:28:51.880 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:51.875+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.878011+0000) 2019-10-13T13:28:51.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:51.878+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.882802+0000) 2019-10-13T13:28:51.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:51.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:51.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:51.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:51.961683+0000) 2019-10-13T13:28:53.247 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:28:53.248 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:53.258 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:28:53.258 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:28:54.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:54.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:54.288916+0000) 2019-10-13T13:28:54.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:54.571+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:54.575041+0000) 2019-10-13T13:28:54.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:54.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:54.942493+0000) 2019-10-13T13:28:56.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:28:56.883+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.887802+0000) 2019-10-13T13:28:56.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:28:56.883+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.888258+0000) 2019-10-13T13:28:56.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:28:56.886+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.889136+0000) 2019-10-13T13:28:56.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:56.883+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.889520+0000) 2019-10-13T13:28:56.892 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:28:56.887+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.890115+0000) 2019-10-13T13:28:56.892 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:56.887+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.890458+0000) 2019-10-13T13:28:56.893 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:56.887+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.890871+0000) 2019-10-13T13:28:56.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:56.894+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:56.896458+0000) 2019-10-13T13:28:56.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:28:56.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:28:57.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:28:57.598+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:57.598673+0000) 2019-10-13T13:28:57.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:28:57.943+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:57.946215+0000) 2019-10-13T13:28:58.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:28:58.167+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:58.171118+0000) 2019-10-13T13:28:58.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:28:58.707+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:28:58.710968+0000) 2019-10-13T13:29:01.901 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:01.895+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.899538+0000) 2019-10-13T13:29:01.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:01.898+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.900195+0000) 2019-10-13T13:29:01.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:01.895+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.901647+0000) 2019-10-13T13:29:01.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:01.899+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.902435+0000) 2019-10-13T13:29:01.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:01.899+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.903235+0000) 2019-10-13T13:29:01.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:01.899+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.903379+0000) 2019-10-13T13:29:01.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:01.899+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.904201+0000) 2019-10-13T13:29:01.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:01.906+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.909341+0000) 2019-10-13T13:29:01.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:01.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:01.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:01.958+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:01.961861+0000) 2019-10-13T13:29:04.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:04.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:04.289109+0000) 2019-10-13T13:29:04.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:04.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:04.575267+0000) 2019-10-13T13:29:04.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:04.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:04.942707+0000) 2019-10-13T13:29:06.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:06.907+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.912053+0000) 2019-10-13T13:29:06.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:06.909+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.912774+0000) 2019-10-13T13:29:06.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:06.907+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.912865+0000) 2019-10-13T13:29:06.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:06.907+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.913188+0000) 2019-10-13T13:29:06.916 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:06.911+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.913852+0000) 2019-10-13T13:29:06.916 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:06.911+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.914511+0000) 2019-10-13T13:29:06.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:06.911+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.915922+0000) 2019-10-13T13:29:06.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:06.917+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:06.921565+0000) 2019-10-13T13:29:06.924 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:06.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:07.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:07.601+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:07.605279+0000) 2019-10-13T13:29:07.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:07.943+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:07.946416+0000) 2019-10-13T13:29:08.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:08.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:08.171348+0000) 2019-10-13T13:29:08.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:08.707+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:08.711165+0000) 2019-10-13T13:29:11.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:11.921+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.925202+0000) 2019-10-13T13:29:11.927 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:11.919+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.925566+0000) 2019-10-13T13:29:11.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:11.923+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.928219+0000) 2019-10-13T13:29:11.931 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:11.923+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.929152+0000) 2019-10-13T13:29:11.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:11.927+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.930048+0000) 2019-10-13T13:29:11.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:11.927+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.929940+0000) 2019-10-13T13:29:11.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:11.927+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.930994+0000) 2019-10-13T13:29:11.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:11.929+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.933975+0000) 2019-10-13T13:29:11.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:11.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:11.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:11.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:11.962071+0000) 2019-10-13T13:29:14.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:14.283+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:14.289327+0000) 2019-10-13T13:29:14.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:14.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:14.575467+0000) 2019-10-13T13:29:14.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:14.939+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:14.942901+0000) 2019-10-13T13:29:16.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:16.930+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.937641+0000) 2019-10-13T13:29:16.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:16.934+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.938115+0000) 2019-10-13T13:29:16.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:16.934+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.938742+0000) 2019-10-13T13:29:16.941 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:16.934+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.939490+0000) 2019-10-13T13:29:16.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:16.933+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.939981+0000) 2019-10-13T13:29:16.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:16.934+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.939985+0000) 2019-10-13T13:29:16.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:16.934+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.940768+0000) 2019-10-13T13:29:16.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:16.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:16.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:16.941+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:16.947649+0000) 2019-10-13T13:29:17.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:17.609+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:17.611913+0000) 2019-10-13T13:29:17.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:17.942+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:17.946622+0000) 2019-10-13T13:29:18.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:18.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:18.171558+0000) 2019-10-13T13:29:18.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:18.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:18.711394+0000) 2019-10-13T13:29:21.951 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:21.942+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.949629+0000) 2019-10-13T13:29:21.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:21.946+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.950218+0000) 2019-10-13T13:29:21.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:21.946+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.950928+0000) 2019-10-13T13:29:21.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:21.945+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.951569+0000) 2019-10-13T13:29:21.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:21.946+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.951475+0000) 2019-10-13T13:29:21.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:21.946+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.951946+0000) 2019-10-13T13:29:21.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:21.946+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.953536+0000) 2019-10-13T13:29:21.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:21.954+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:21.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:21.957+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.961981+0000) 2019-10-13T13:29:21.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:21.957+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:21.962193+0000) 2019-10-13T13:29:23.362 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:29:23.362 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:23.366 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:29:23.366 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:24.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:24.282+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:24.289532+0000) 2019-10-13T13:29:24.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:24.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:24.575665+0000) 2019-10-13T13:29:24.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:24.938+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:24.943102+0000) 2019-10-13T13:29:26.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:26.957+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.962739+0000) 2019-10-13T13:29:26.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:26.958+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.963094+0000) 2019-10-13T13:29:26.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:26.958+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.963741+0000) 2019-10-13T13:29:26.966 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:26.958+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.964566+0000) 2019-10-13T13:29:26.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:26.958+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.964996+0000) 2019-10-13T13:29:26.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:26.958+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.965531+0000) 2019-10-13T13:29:26.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:26.962+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.966385+0000) 2019-10-13T13:29:26.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:26.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:26.969203+0000) 2019-10-13T13:29:26.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:26.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:27.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:27.617+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:27.623505+0000) 2019-10-13T13:29:27.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:27.942+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:27.946820+0000) 2019-10-13T13:29:28.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:28.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:28.171807+0000) 2019-10-13T13:29:28.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:28.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:28.711596+0000) 2019-10-13T13:29:31.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:31.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.962418+0000) 2019-10-13T13:29:31.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:31.970+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.974576+0000) 2019-10-13T13:29:31.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:31.973+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.976317+0000) 2019-10-13T13:29:31.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:31.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.976564+0000) 2019-10-13T13:29:31.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:31.970+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.976689+0000) 2019-10-13T13:29:31.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:31.970+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.977249+0000) 2019-10-13T13:29:31.980 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:31.974+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.978391+0000) 2019-10-13T13:29:31.981 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:31.974+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.978793+0000) 2019-10-13T13:29:31.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:31.977+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:31.983525+0000) 2019-10-13T13:29:31.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:31.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:34.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:34.286+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:34.289727+0000) 2019-10-13T13:29:34.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:34.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:34.575894+0000) 2019-10-13T13:29:34.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:34.942+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:34.943320+0000) 2019-10-13T13:29:36.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:36.982+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.988922+0000) 2019-10-13T13:29:36.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:36.985+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.989984+0000) 2019-10-13T13:29:36.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:36.986+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.990024+0000) 2019-10-13T13:29:36.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:36.986+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.990374+0000) 2019-10-13T13:29:36.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:36.986+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.990821+0000) 2019-10-13T13:29:36.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:36.986+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.991292+0000) 2019-10-13T13:29:36.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:36.986+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.992174+0000) 2019-10-13T13:29:36.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:36.989+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:36.995083+0000) 2019-10-13T13:29:36.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:36.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:37.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:37.625+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:37.629992+0000) 2019-10-13T13:29:37.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:37.946+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:37.946995+0000) 2019-10-13T13:29:38.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:38.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:38.172022+0000) 2019-10-13T13:29:38.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:38.710+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:38.711794+0000) 2019-10-13T13:29:41.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:41.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:41.962620+0000) 2019-10-13T13:29:42.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:41.998+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.004650+0000) 2019-10-13T13:29:42.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:42.001+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.005744+0000) 2019-10-13T13:29:42.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:42.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.006074+0000) 2019-10-13T13:29:42.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:42.002+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.006238+0000) 2019-10-13T13:29:42.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:42.002+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.006768+0000) 2019-10-13T13:29:42.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:42.002+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.007176+0000) 2019-10-13T13:29:42.009 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:42.002+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.007636+0000) 2019-10-13T13:29:42.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:42.005+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:42.009715+0000) 2019-10-13T13:29:42.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:42.006+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:44.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:44.286+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:44.289972+0000) 2019-10-13T13:29:44.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:44.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:44.576131+0000) 2019-10-13T13:29:44.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:44.938+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:44.943521+0000) 2019-10-13T13:29:47.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:47.010+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.014735+0000) 2019-10-13T13:29:47.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:47.009+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.015485+0000) 2019-10-13T13:29:47.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:47.010+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.015696+0000) 2019-10-13T13:29:47.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:47.010+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.015931+0000) 2019-10-13T13:29:47.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:47.010+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.017411+0000) 2019-10-13T13:29:47.020 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:47.014+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.018538+0000) 2019-10-13T13:29:47.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:47.014+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.020145+0000) 2019-10-13T13:29:47.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:47.017+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.022028+0000) 2019-10-13T13:29:47.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:47.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:47.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:47.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.636561+0000) 2019-10-13T13:29:47.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:47.946+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:47.947180+0000) 2019-10-13T13:29:48.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:48.166+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:48.172241+0000) 2019-10-13T13:29:48.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:48.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:48.712005+0000) 2019-10-13T13:29:51.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:51.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:51.962836+0000) 2019-10-13T13:29:52.029 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:52.022+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.026705+0000) 2019-10-13T13:29:52.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:52.022+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.027032+0000) 2019-10-13T13:29:52.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:52.025+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.028312+0000) 2019-10-13T13:29:52.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:52.022+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.028581+0000) 2019-10-13T13:29:52.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:52.022+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.029592+0000) 2019-10-13T13:29:52.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:52.026+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.029810+0000) 2019-10-13T13:29:52.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:52.026+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.030418+0000) 2019-10-13T13:29:52.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:52.029+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:52.035464+0000) 2019-10-13T13:29:52.037 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:52.030+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183944 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:53.480 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:29:53.480 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:53.483 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:29:53.483 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:29:54.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:54.286+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:54.290213+0000) 2019-10-13T13:29:54.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:54.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:54.576358+0000) 2019-10-13T13:29:54.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:54.942+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:54.943727+0000) 2019-10-13T13:29:57.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:57.034+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.039133+0000) 2019-10-13T13:29:57.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:57.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.039745+0000) 2019-10-13T13:29:57.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:29:57.037+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.040373+0000) 2019-10-13T13:29:57.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:29:57.034+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.041031+0000) 2019-10-13T13:29:57.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:29:57.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.041505+0000) 2019-10-13T13:29:57.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:57.038+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.041902+0000) 2019-10-13T13:29:57.044 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:29:57.038+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.042352+0000) 2019-10-13T13:29:57.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:29:57.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183975 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:29:57.050 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:57.045+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.048956+0000) 2019-10-13T13:29:57.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:29:57.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.643033+0000) 2019-10-13T13:29:57.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:29:57.946+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:57.947393+0000) 2019-10-13T13:29:58.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:29:58.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:58.172473+0000) 2019-10-13T13:29:58.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:29:58.706+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:29:58.712203+0000) 2019-10-13T13:30:01.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:01.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:01.963037+0000) 2019-10-13T13:30:02.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:02.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.051918+0000) 2019-10-13T13:30:02.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:02.046+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.051965+0000) 2019-10-13T13:30:02.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:02.046+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.052616+0000) 2019-10-13T13:30:02.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:02.050+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.055379+0000) 2019-10-13T13:30:02.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:02.053+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.057356+0000) 2019-10-13T13:30:02.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:02.050+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.057416+0000) 2019-10-13T13:30:02.060 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:02.054+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.057849+0000) 2019-10-13T13:30:02.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:02.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184006 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:02.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:02.061+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:02.065152+0000) 2019-10-13T13:30:04.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:04.290+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:04.290443+0000) 2019-10-13T13:30:04.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:04.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:04.576560+0000) 2019-10-13T13:30:04.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:04.942+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:04.943986+0000) 2019-10-13T13:30:07.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:07.058+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.063611+0000) 2019-10-13T13:30:07.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:07.058+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.064335+0000) 2019-10-13T13:30:07.067 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:07.061+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.064992+0000) 2019-10-13T13:30:07.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:07.058+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.065014+0000) 2019-10-13T13:30:07.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:07.062+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.066044+0000) 2019-10-13T13:30:07.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:07.062+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.066520+0000) 2019-10-13T13:30:07.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:07.062+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.068184+0000) 2019-10-13T13:30:07.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:07.069+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.072145+0000) 2019-10-13T13:30:07.074 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:07.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:07.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:07.644+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.649644+0000) 2019-10-13T13:30:07.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:07.946+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:07.947602+0000) 2019-10-13T13:30:08.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:08.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:08.172700+0000) 2019-10-13T13:30:08.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:08.710+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:08.712401+0000) 2019-10-13T13:30:11.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:11.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:11.963261+0000) 2019-10-13T13:30:12.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:12.069+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.075505+0000) 2019-10-13T13:30:12.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:12.072+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.076127+0000) 2019-10-13T13:30:12.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:12.069+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.076964+0000) 2019-10-13T13:30:12.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:12.073+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.078462+0000) 2019-10-13T13:30:12.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:12.073+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.078594+0000) 2019-10-13T13:30:12.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:12.073+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.081288+0000) 2019-10-13T13:30:12.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:12.077+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.082360+0000) 2019-10-13T13:30:12.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:12.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:12.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:12.080+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:12.086459+0000) 2019-10-13T13:30:14.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:14.285+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:14.290682+0000) 2019-10-13T13:30:14.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:14.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:14.576762+0000) 2019-10-13T13:30:14.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:14.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:14.944190+0000) 2019-10-13T13:30:17.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:17.081+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.089102+0000) 2019-10-13T13:30:17.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:17.085+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.090186+0000) 2019-10-13T13:30:17.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:17.085+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.090622+0000) 2019-10-13T13:30:17.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:17.085+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.092570+0000) 2019-10-13T13:30:17.095 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:17.088+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.093328+0000) 2019-10-13T13:30:17.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:17.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.093519+0000) 2019-10-13T13:30:17.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:17.089+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.094043+0000) 2019-10-13T13:30:17.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:17.089+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:17.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:17.092+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.098907+0000) 2019-10-13T13:30:17.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:17.652+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.656214+0000) 2019-10-13T13:30:17.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:17.945+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:17.947787+0000) 2019-10-13T13:30:18.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:18.169+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:18.172897+0000) 2019-10-13T13:30:18.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:18.709+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:18.712598+0000) 2019-10-13T13:30:21.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:21.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:21.963476+0000) 2019-10-13T13:30:22.102 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:22.096+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.100662+0000) 2019-10-13T13:30:22.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:22.093+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.100558+0000) 2019-10-13T13:30:22.103 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:22.093+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.101033+0000) 2019-10-13T13:30:22.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:22.093+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.101600+0000) 2019-10-13T13:30:22.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:22.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.102179+0000) 2019-10-13T13:30:22.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:22.097+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.102753+0000) 2019-10-13T13:30:22.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:22.097+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.102923+0000) 2019-10-13T13:30:22.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:22.101+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:22.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:22.104+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:22.109801+0000) 2019-10-13T13:30:23.535 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:30:23.535 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:23.539 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:30:23.539 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:24.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:24.289+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:24.290903+0000) 2019-10-13T13:30:24.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:24.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:24.576974+0000) 2019-10-13T13:30:24.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:24.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:24.944340+0000) 2019-10-13T13:30:27.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:27.109+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.111832+0000) 2019-10-13T13:30:27.115 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:27.108+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.113199+0000) 2019-10-13T13:30:27.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:27.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.113262+0000) 2019-10-13T13:30:27.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:27.113+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.113958+0000) 2019-10-13T13:30:27.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:27.113+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.114323+0000) 2019-10-13T13:30:27.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:27.113+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.114860+0000) 2019-10-13T13:30:27.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:27.113+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.115200+0000) 2019-10-13T13:30:27.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:27.112+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.119356+0000) 2019-10-13T13:30:27.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:27.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:27.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:27.656+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.662785+0000) 2019-10-13T13:30:27.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:27.945+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:27.947983+0000) 2019-10-13T13:30:28.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:28.169+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:28.173109+0000) 2019-10-13T13:30:28.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:28.709+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:28.712779+0000) 2019-10-13T13:30:31.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:31.960+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:31.963698+0000) 2019-10-13T13:30:32.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:32.121+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.124807+0000) 2019-10-13T13:30:32.127 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:32.121+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.125572+0000) 2019-10-13T13:30:32.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:32.120+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.126416+0000) 2019-10-13T13:30:32.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:32.125+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.127057+0000) 2019-10-13T13:30:32.129 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:32.125+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.127182+0000) 2019-10-13T13:30:32.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:32.125+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.127368+0000) 2019-10-13T13:30:32.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:32.125+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.125855+0000) 2019-10-13T13:30:32.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:32.124+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:32.130877+0000) 2019-10-13T13:30:32.134 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:32.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:34.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:34.289+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:34.291155+0000) 2019-10-13T13:30:34.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:34.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:34.577171+0000) 2019-10-13T13:30:34.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:34.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:34.944539+0000) 2019-10-13T13:30:37.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:37.133+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.135946+0000) 2019-10-13T13:30:37.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:37.133+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.136447+0000) 2019-10-13T13:30:37.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:37.133+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.136908+0000) 2019-10-13T13:30:37.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:37.133+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.137389+0000) 2019-10-13T13:30:37.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:37.132+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.137870+0000) 2019-10-13T13:30:37.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:37.137+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.138556+0000) 2019-10-13T13:30:37.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:37.137+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.138930+0000) 2019-10-13T13:30:37.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:37.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:37.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:37.140+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.144583+0000) 2019-10-13T13:30:37.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:37.668+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.669152+0000) 2019-10-13T13:30:37.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:37.945+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:37.948178+0000) 2019-10-13T13:30:38.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:38.169+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:38.173323+0000) 2019-10-13T13:30:38.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:38.709+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:38.712992+0000) 2019-10-13T13:30:41.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:41.956+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:41.963892+0000) 2019-10-13T13:30:42.150 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:42.145+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.147325+0000) 2019-10-13T13:30:42.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:42.145+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.147513+0000) 2019-10-13T13:30:42.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:42.145+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.148011+0000) 2019-10-13T13:30:42.152 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:42.144+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.150502+0000) 2019-10-13T13:30:42.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:42.149+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.150682+0000) 2019-10-13T13:30:42.153 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:42.149+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.151164+0000) 2019-10-13T13:30:42.156 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:42.153+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.154507+0000) 2019-10-13T13:30:42.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:42.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:42.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:42.152+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:42.159398+0000) 2019-10-13T13:30:44.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:44.289+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:44.291389+0000) 2019-10-13T13:30:44.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:44.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:44.577371+0000) 2019-10-13T13:30:44.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:44.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:44.944740+0000) 2019-10-13T13:30:47.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:47.157+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.160369+0000) 2019-10-13T13:30:47.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:47.157+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.161359+0000) 2019-10-13T13:30:47.165 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:47.161+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.163017+0000) 2019-10-13T13:30:47.166 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:47.161+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.163927+0000) 2019-10-13T13:30:47.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:47.160+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.164674+0000) 2019-10-13T13:30:47.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:47.161+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.164861+0000) 2019-10-13T13:30:47.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:47.161+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.165543+0000) 2019-10-13T13:30:47.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:47.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:47.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:47.168+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.173319+0000) 2019-10-13T13:30:47.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:47.672+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.678723+0000) 2019-10-13T13:30:47.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:47.945+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:47.948362+0000) 2019-10-13T13:30:48.175 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:48.169+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:48.173519+0000) 2019-10-13T13:30:48.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:48.709+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:48.713186+0000) 2019-10-13T13:30:51.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:51.960+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:51.964113+0000) 2019-10-13T13:30:52.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:52.169+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.172565+0000) 2019-10-13T13:30:52.177 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:52.173+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.175545+0000) 2019-10-13T13:30:52.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:52.173+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.175150+0000) 2019-10-13T13:30:52.179 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:52.173+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.176678+0000) 2019-10-13T13:30:52.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:52.173+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.176933+0000) 2019-10-13T13:30:52.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:52.173+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.177496+0000) 2019-10-13T13:30:52.180 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:52.172+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.178283+0000) 2019-10-13T13:30:52.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:52.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:52.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:52.180+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:52.185796+0000) 2019-10-13T13:30:53.632 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:30:53.632 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:53.639 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:30:53.639 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:30:54.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:54.289+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:54.291604+0000) 2019-10-13T13:30:54.579 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:54.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:54.577573+0000) 2019-10-13T13:30:54.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:54.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:54.944935+0000) 2019-10-13T13:30:57.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:30:57.185+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.186992+0000) 2019-10-13T13:30:57.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:57.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.187300+0000) 2019-10-13T13:30:57.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:30:57.185+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.187687+0000) 2019-10-13T13:30:57.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:57.185+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.188707+0000) 2019-10-13T13:30:57.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:30:57.184+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.189592+0000) 2019-10-13T13:30:57.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:30:57.185+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.189471+0000) 2019-10-13T13:30:57.192 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:57.185+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.189464+0000) 2019-10-13T13:30:57.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:57.188+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.195346+0000) 2019-10-13T13:30:57.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:30:57.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:30:57.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:30:57.684+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.689993+0000) 2019-10-13T13:30:57.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:30:57.945+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:57.948558+0000) 2019-10-13T13:30:58.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:30:58.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:58.173721+0000) 2019-10-13T13:30:58.715 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:30:58.713+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:30:58.713380+0000) 2019-10-13T13:31:01.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:01.964+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:01.964346+0000) 2019-10-13T13:31:02.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:02.197+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.198550+0000) 2019-10-13T13:31:02.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:02.197+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.199065+0000) 2019-10-13T13:31:02.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:02.197+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.200759+0000) 2019-10-13T13:31:02.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:02.197+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.201628+0000) 2019-10-13T13:31:02.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:02.201+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.202154+0000) 2019-10-13T13:31:02.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:02.196+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.203005+0000) 2019-10-13T13:31:02.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:02.201+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.202921+0000) 2019-10-13T13:31:02.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:02.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:02.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:02.204+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:02.210636+0000) 2019-10-13T13:31:04.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:04.289+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:04.291841+0000) 2019-10-13T13:31:04.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:04.577+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:04.577818+0000) 2019-10-13T13:31:04.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:04.941+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:04.945142+0000) 2019-10-13T13:31:07.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:07.208+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.210364+0000) 2019-10-13T13:31:07.214 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:07.208+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.212263+0000) 2019-10-13T13:31:07.215 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:07.208+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.212539+0000) 2019-10-13T13:31:07.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:07.212+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.214228+0000) 2019-10-13T13:31:07.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:07.212+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.215399+0000) 2019-10-13T13:31:07.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:07.212+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.215530+0000) 2019-10-13T13:31:07.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:07.216+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.216245+0000) 2019-10-13T13:31:07.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:07.216+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:07.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:07.220+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.220862+0000) 2019-10-13T13:31:07.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:07.692+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.695952+0000) 2019-10-13T13:31:07.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:07.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:07.948761+0000) 2019-10-13T13:31:08.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:08.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:08.173985+0000) 2019-10-13T13:31:08.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:08.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:08.713583+0000) 2019-10-13T13:31:11.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:11.964+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:11.964524+0000) 2019-10-13T13:31:12.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:12.220+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.223274+0000) 2019-10-13T13:31:12.227 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:12.220+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.224831+0000) 2019-10-13T13:31:12.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:12.220+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.224688+0000) 2019-10-13T13:31:12.227 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:12.220+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.225158+0000) 2019-10-13T13:31:12.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:12.223+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.225746+0000) 2019-10-13T13:31:12.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:12.224+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.225819+0000) 2019-10-13T13:31:12.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:12.224+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.226245+0000) 2019-10-13T13:31:12.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:12.231+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:12.232685+0000) 2019-10-13T13:31:12.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:12.228+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:14.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:14.288+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:14.292044+0000) 2019-10-13T13:31:14.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:14.576+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:14.578020+0000) 2019-10-13T13:31:14.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:14.944+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:14.945345+0000) 2019-10-13T13:31:17.238 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:17.232+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.235820+0000) 2019-10-13T13:31:17.238 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:17.232+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.236155+0000) 2019-10-13T13:31:17.239 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:17.232+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.237176+0000) 2019-10-13T13:31:17.239 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:17.232+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.236819+0000) 2019-10-13T13:31:17.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:17.236+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.238772+0000) 2019-10-13T13:31:17.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:17.236+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.239259+0000) 2019-10-13T13:31:17.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:17.235+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.240033+0000) 2019-10-13T13:31:17.247 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:17.240+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184463 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:17.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:17.243+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.246578+0000) 2019-10-13T13:31:17.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:17.699+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.702552+0000) 2019-10-13T13:31:17.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:17.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:17.948934+0000) 2019-10-13T13:31:18.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:18.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:18.174203+0000) 2019-10-13T13:31:18.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:18.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:18.713775+0000) 2019-10-13T13:31:21.966 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:21.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:21.964747+0000) 2019-10-13T13:31:22.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:22.244+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.249272+0000) 2019-10-13T13:31:22.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:22.244+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.249549+0000) 2019-10-13T13:31:22.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:22.248+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.250060+0000) 2019-10-13T13:31:22.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:22.248+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.250552+0000) 2019-10-13T13:31:22.253 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:22.248+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.251481+0000) 2019-10-13T13:31:22.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:22.251+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.252337+0000) 2019-10-13T13:31:22.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:22.248+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.252526+0000) 2019-10-13T13:31:22.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:22.251+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:22.255515+0000) 2019-10-13T13:31:22.260 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:22.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:23.790 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:31:23.790 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:23.793 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:31:23.794 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:24.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:24.288+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:24.292267+0000) 2019-10-13T13:31:24.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:24.576+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:24.578216+0000) 2019-10-13T13:31:24.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:24.940+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:24.945544+0000) 2019-10-13T13:31:27.265 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:27.260+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.262935+0000) 2019-10-13T13:31:27.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:27.260+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.263714+0000) 2019-10-13T13:31:27.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:27.260+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.264155+0000) 2019-10-13T13:31:27.266 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:27.260+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.264561+0000) 2019-10-13T13:31:27.267 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:27.260+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.265527+0000) 2019-10-13T13:31:27.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:27.260+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.265549+0000) 2019-10-13T13:31:27.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:27.263+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.266539+0000) 2019-10-13T13:31:27.273 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:27.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:27.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:27.271+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.272855+0000) 2019-10-13T13:31:27.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:27.707+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.709251+0000) 2019-10-13T13:31:27.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:27.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:27.949104+0000) 2019-10-13T13:31:28.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:28.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:28.174396+0000) 2019-10-13T13:31:28.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:28.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:28.714003+0000) 2019-10-13T13:31:31.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:31.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:31.964944+0000) 2019-10-13T13:31:32.276 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:32.272+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.273815+0000) 2019-10-13T13:31:32.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:32.272+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.274663+0000) 2019-10-13T13:31:32.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:32.272+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.275092+0000) 2019-10-13T13:31:32.278 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:32.272+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.276316+0000) 2019-10-13T13:31:32.279 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:32.272+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.277019+0000) 2019-10-13T13:31:32.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:32.272+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.277659+0000) 2019-10-13T13:31:32.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:32.275+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.278468+0000) 2019-10-13T13:31:32.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:32.279+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:32.283135+0000) 2019-10-13T13:31:32.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:32.280+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184554 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:34.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:34.288+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:34.292529+0000) 2019-10-13T13:31:34.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:34.576+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:34.578424+0000) 2019-10-13T13:31:34.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:34.944+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:34.945777+0000) 2019-10-13T13:31:37.290 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:37.284+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.287790+0000) 2019-10-13T13:31:37.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:37.284+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.288119+0000) 2019-10-13T13:31:37.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:37.288+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.290771+0000) 2019-10-13T13:31:37.293 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:37.288+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.290970+0000) 2019-10-13T13:31:37.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:37.288+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.291227+0000) 2019-10-13T13:31:37.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:37.292+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.295579+0000) 2019-10-13T13:31:37.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:37.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:37.300 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:37.295+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.298853+0000) 2019-10-13T13:31:37.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:37.303+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.304300+0000) 2019-10-13T13:31:37.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:37.711+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.715727+0000) 2019-10-13T13:31:37.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:37.944+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:37.949282+0000) 2019-10-13T13:31:38.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:38.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:38.174572+0000) 2019-10-13T13:31:38.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:38.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:38.714203+0000) 2019-10-13T13:31:41.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:41.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:41.965152+0000) 2019-10-13T13:31:42.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:42.296+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.300804+0000) 2019-10-13T13:31:42.303 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:42.296+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.301352+0000) 2019-10-13T13:31:42.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:42.300+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.302373+0000) 2019-10-13T13:31:42.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:42.303+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.304388+0000) 2019-10-13T13:31:42.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:42.300+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.303730+0000) 2019-10-13T13:31:42.309 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:42.304+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.306102+0000) 2019-10-13T13:31:42.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:42.307+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.308622+0000) 2019-10-13T13:31:42.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:42.304+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:42.308581+0000) 2019-10-13T13:31:42.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:42.308+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:44.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:44.292+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:44.292732+0000) 2019-10-13T13:31:44.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:44.576+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:44.578644+0000) 2019-10-13T13:31:44.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:44.944+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:44.945999+0000) 2019-10-13T13:31:47.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:47.312+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.315505+0000) 2019-10-13T13:31:47.318 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:47.312+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.315760+0000) 2019-10-13T13:31:47.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:47.312+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.316788+0000) 2019-10-13T13:31:47.320 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:47.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.318108+0000) 2019-10-13T13:31:47.320 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:47.316+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.318447+0000) 2019-10-13T13:31:47.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:47.316+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.319010+0000) 2019-10-13T13:31:47.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:47.316+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.319553+0000) 2019-10-13T13:31:47.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:47.319+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.323996+0000) 2019-10-13T13:31:47.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:47.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:47.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:47.719+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.722210+0000) 2019-10-13T13:31:47.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:47.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:47.949464+0000) 2019-10-13T13:31:48.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:48.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:48.174750+0000) 2019-10-13T13:31:48.716 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:48.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:48.714401+0000) 2019-10-13T13:31:51.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:51.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:51.965403+0000) 2019-10-13T13:31:52.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:52.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.328184+0000) 2019-10-13T13:31:52.332 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:52.328+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.330578+0000) 2019-10-13T13:31:52.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:52.328+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.331377+0000) 2019-10-13T13:31:52.334 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:52.328+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.331657+0000) 2019-10-13T13:31:52.334 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:52.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.332426+0000) 2019-10-13T13:31:52.334 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:52.328+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.332765+0000) 2019-10-13T13:31:52.335 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:52.328+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.333187+0000) 2019-10-13T13:31:52.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:52.335+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:52.337044+0000) 2019-10-13T13:31:52.339 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:52.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184676 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:53.846 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:31:53.846 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:53.849 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:31:53.850 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:31:54.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:54.288+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:54.292943+0000) 2019-10-13T13:31:54.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:54.576+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:54.578852+0000) 2019-10-13T13:31:54.948 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:54.944+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:54.946202+0000) 2019-10-13T13:31:57.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:31:57.336+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.340460+0000) 2019-10-13T13:31:57.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:31:57.336+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.340900+0000) 2019-10-13T13:31:57.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:31:57.339+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.341918+0000) 2019-10-13T13:31:57.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:31:57.340+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.342289+0000) 2019-10-13T13:31:57.345 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:57.340+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.342492+0000) 2019-10-13T13:31:57.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:57.340+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.343028+0000) 2019-10-13T13:31:57.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:57.340+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.343674+0000) 2019-10-13T13:31:57.351 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:57.347+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.348817+0000) 2019-10-13T13:31:57.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:31:57.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:31:57.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:31:57.727+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.728735+0000) 2019-10-13T13:31:57.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:31:57.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:57.949633+0000) 2019-10-13T13:31:58.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:31:58.172+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:58.174952+0000) 2019-10-13T13:31:58.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:31:58.712+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:31:58.714595+0000) 2019-10-13T13:32:01.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:01.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:01.965598+0000) 2019-10-13T13:32:02.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:02.351+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.352596+0000) 2019-10-13T13:32:02.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:02.348+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.352861+0000) 2019-10-13T13:32:02.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:02.348+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.353200+0000) 2019-10-13T13:32:02.356 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:02.348+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.353475+0000) 2019-10-13T13:32:02.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:02.352+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.354887+0000) 2019-10-13T13:32:02.359 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:02.352+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.355722+0000) 2019-10-13T13:32:02.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:02.352+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.357428+0000) 2019-10-13T13:32:02.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:02.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184738 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:02.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:02.363+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:02.365512+0000) 2019-10-13T13:32:04.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:04.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:04.293184+0000) 2019-10-13T13:32:04.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:04.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:04.579048+0000) 2019-10-13T13:32:04.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:04.943+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:04.946403+0000) 2019-10-13T13:32:07.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:07.359+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.365601+0000) 2019-10-13T13:32:07.369 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:07.363+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.367307+0000) 2019-10-13T13:32:07.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:07.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.367649+0000) 2019-10-13T13:32:07.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:07.367+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.368542+0000) 2019-10-13T13:32:07.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:07.363+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.367895+0000) 2019-10-13T13:32:07.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:07.363+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.368821+0000) 2019-10-13T13:32:07.372 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:07.363+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.369177+0000) 2019-10-13T13:32:07.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:07.371+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:07.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:07.375+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.376887+0000) 2019-10-13T13:32:07.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:07.735+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.739906+0000) 2019-10-13T13:32:07.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:07.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:07.949816+0000) 2019-10-13T13:32:08.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:08.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:08.175128+0000) 2019-10-13T13:32:08.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:08.711+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:08.714789+0000) 2019-10-13T13:32:11.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:11.963+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:11.965821+0000) 2019-10-13T13:32:12.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:12.375+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.379126+0000) 2019-10-13T13:32:12.381 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:12.375+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.379492+0000) 2019-10-13T13:32:12.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:12.379+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.380214+0000) 2019-10-13T13:32:12.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:12.375+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.380292+0000) 2019-10-13T13:32:12.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:12.375+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.381595+0000) 2019-10-13T13:32:12.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:12.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.382266+0000) 2019-10-13T13:32:12.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:12.379+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.384177+0000) 2019-10-13T13:32:12.390 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:12.383+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:12.392 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:12.387+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:12.390475+0000) 2019-10-13T13:32:14.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:14.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:14.293388+0000) 2019-10-13T13:32:14.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:14.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:14.579244+0000) 2019-10-13T13:32:14.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:14.943+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:14.946617+0000) 2019-10-13T13:32:17.395 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:17.387+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.393074+0000) 2019-10-13T13:32:17.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:17.391+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.394001+0000) 2019-10-13T13:32:17.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:17.391+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.394664+0000) 2019-10-13T13:32:17.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:17.391+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.395097+0000) 2019-10-13T13:32:17.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:17.391+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.395912+0000) 2019-10-13T13:32:17.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:17.394+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.396560+0000) 2019-10-13T13:32:17.399 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:17.391+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.396494+0000) 2019-10-13T13:32:17.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:17.398+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.400796+0000) 2019-10-13T13:32:17.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:17.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:17.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:17.742+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.746525+0000) 2019-10-13T13:32:17.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:17.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:17.949998+0000) 2019-10-13T13:32:18.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:18.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:18.175338+0000) 2019-10-13T13:32:18.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:18.711+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:18.714996+0000) 2019-10-13T13:32:21.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:21.962+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:21.966040+0000) 2019-10-13T13:32:22.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:22.399+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.405133+0000) 2019-10-13T13:32:22.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:22.399+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.405466+0000) 2019-10-13T13:32:22.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:22.403+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.406427+0000) 2019-10-13T13:32:22.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:22.403+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.407894+0000) 2019-10-13T13:32:22.410 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:22.406+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.408850+0000) 2019-10-13T13:32:22.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:22.403+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.409140+0000) 2019-10-13T13:32:22.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:22.403+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.409363+0000) 2019-10-13T13:32:22.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:22.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:22.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:22.414+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:22.419644+0000) 2019-10-13T13:32:24.098 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:32:24.098 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:24.102 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:32:24.102 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:24.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:24.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:24.293629+0000) 2019-10-13T13:32:24.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:24.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:24.579441+0000) 2019-10-13T13:32:24.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:24.943+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:24.946817+0000) 2019-10-13T13:32:27.421 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:27.415+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.418704+0000) 2019-10-13T13:32:27.421 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:27.415+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.419381+0000) 2019-10-13T13:32:27.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:27.415+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.420454+0000) 2019-10-13T13:32:27.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:27.415+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.420624+0000) 2019-10-13T13:32:27.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:27.415+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.421103+0000) 2019-10-13T13:32:27.424 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:27.418+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.422455+0000) 2019-10-13T13:32:27.425 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:27.419+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.423379+0000) 2019-10-13T13:32:27.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:27.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:27.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:27.426+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.428748+0000) 2019-10-13T13:32:27.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:27.750+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.752574+0000) 2019-10-13T13:32:27.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:27.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:27.950211+0000) 2019-10-13T13:32:28.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:28.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:28.175510+0000) 2019-10-13T13:32:28.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:28.711+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:28.715237+0000) 2019-10-13T13:32:31.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:31.962+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:31.966251+0000) 2019-10-13T13:32:32.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:32.427+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.430071+0000) 2019-10-13T13:32:32.433 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:32.427+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.430602+0000) 2019-10-13T13:32:32.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:32.427+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.433386+0000) 2019-10-13T13:32:32.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:32.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.435026+0000) 2019-10-13T13:32:32.438 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:32.431+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.436076+0000) 2019-10-13T13:32:32.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:32.434+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.437477+0000) 2019-10-13T13:32:32.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:32.434+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.437762+0000) 2019-10-13T13:32:32.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:32.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:32.439684+0000) 2019-10-13T13:32:32.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:32.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:34.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:34.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:34.293902+0000) 2019-10-13T13:32:34.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:34.575+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:34.579647+0000) 2019-10-13T13:32:34.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:34.943+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:34.947051+0000) 2019-10-13T13:32:37.445 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:37.439+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.442978+0000) 2019-10-13T13:32:37.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:37.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.443788+0000) 2019-10-13T13:32:37.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:37.442+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.445054+0000) 2019-10-13T13:32:37.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:37.439+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.444918+0000) 2019-10-13T13:32:37.447 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:37.439+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.445168+0000) 2019-10-13T13:32:37.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:37.443+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.446474+0000) 2019-10-13T13:32:37.449 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:37.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.446910+0000) 2019-10-13T13:32:37.454 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:37.447+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:37.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:37.450+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.454697+0000) 2019-10-13T13:32:37.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:37.754+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.759039+0000) 2019-10-13T13:32:37.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:37.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:37.950343+0000) 2019-10-13T13:32:38.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:38.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:38.175717+0000) 2019-10-13T13:32:38.717 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:38.711+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:38.715433+0000) 2019-10-13T13:32:41.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:41.962+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:41.966460+0000) 2019-10-13T13:32:42.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:42.451+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.456320+0000) 2019-10-13T13:32:42.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:42.451+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.457275+0000) 2019-10-13T13:32:42.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:42.459+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.462276+0000) 2019-10-13T13:32:42.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:42.459+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.464224+0000) 2019-10-13T13:32:42.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:42.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:42.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:42.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.466362+0000) 2019-10-13T13:32:42.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:42.462+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.466610+0000) 2019-10-13T13:32:42.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:42.463+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.466278+0000) 2019-10-13T13:32:42.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:42.463+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:42.466367+0000) 2019-10-13T13:32:44.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:44.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:44.294132+0000) 2019-10-13T13:32:44.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:44.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:44.579853+0000) 2019-10-13T13:32:44.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:44.943+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:44.947245+0000) 2019-10-13T13:32:47.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:47.463+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.468817+0000) 2019-10-13T13:32:47.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:47.467+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.470090+0000) 2019-10-13T13:32:47.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:47.467+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.470776+0000) 2019-10-13T13:32:47.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:47.467+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.473046+0000) 2019-10-13T13:32:47.477 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:47.470+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.475865+0000) 2019-10-13T13:32:47.478 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:47.471+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.476131+0000) 2019-10-13T13:32:47.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:47.474+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.476890+0000) 2019-10-13T13:32:47.479 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:47.471+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.477015+0000) 2019-10-13T13:32:47.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:47.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:47.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:47.762+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.765668+0000) 2019-10-13T13:32:47.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:47.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:47.950553+0000) 2019-10-13T13:32:48.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:48.171+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:48.175899+0000) 2019-10-13T13:32:48.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:48.715+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:48.715633+0000) 2019-10-13T13:32:51.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:51.962+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:51.966679+0000) 2019-10-13T13:32:52.483 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:52.475+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.480872+0000) 2019-10-13T13:32:52.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:52.475+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.481708+0000) 2019-10-13T13:32:52.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:52.479+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.482847+0000) 2019-10-13T13:32:52.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:52.479+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.484001+0000) 2019-10-13T13:32:52.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:52.479+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.484001+0000) 2019-10-13T13:32:52.487 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:52.479+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.484271+0000) 2019-10-13T13:32:52.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:52.482+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.484907+0000) 2019-10-13T13:32:52.491 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:52.486+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:52.489995+0000) 2019-10-13T13:32:52.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:52.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:54.209 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:32:54.209 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:54.213 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:32:54.213 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:32:54.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:54.291+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:54.294331+0000) 2019-10-13T13:32:54.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:54.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:54.580053+0000) 2019-10-13T13:32:54.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:54.947+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:54.947450+0000) 2019-10-13T13:32:57.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:32:57.491+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.494355+0000) 2019-10-13T13:32:57.497 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:57.491+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.495052+0000) 2019-10-13T13:32:57.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:32:57.491+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.495643+0000) 2019-10-13T13:32:57.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:57.491+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.497663+0000) 2019-10-13T13:32:57.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:57.495+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.497972+0000) 2019-10-13T13:32:57.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:32:57.495+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.500298+0000) 2019-10-13T13:32:57.503 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:32:57.498+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.500969+0000) 2019-10-13T13:32:57.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:32:57.499+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:32:57.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:57.502+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.506023+0000) 2019-10-13T13:32:57.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:32:57.770+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.772276+0000) 2019-10-13T13:32:57.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:32:57.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:57.950730+0000) 2019-10-13T13:32:58.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:32:58.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:58.176106+0000) 2019-10-13T13:32:58.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:32:58.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:32:58.715831+0000) 2019-10-13T13:33:01.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:01.966+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:01.966919+0000) 2019-10-13T13:33:02.509 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:02.502+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.506844+0000) 2019-10-13T13:33:02.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:02.502+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.507880+0000) 2019-10-13T13:33:02.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:02.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.508825+0000) 2019-10-13T13:33:02.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:02.502+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.508254+0000) 2019-10-13T13:33:02.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:02.502+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.508873+0000) 2019-10-13T13:33:02.512 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:02.502+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.509214+0000) 2019-10-13T13:33:02.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:02.506+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.510045+0000) 2019-10-13T13:33:02.519 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:02.514+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:02.516970+0000) 2019-10-13T13:33:02.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:02.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185102 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:04.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:04.290+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:04.294541+0000) 2019-10-13T13:33:04.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:04.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:04.580287+0000) 2019-10-13T13:33:04.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:04.946+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:04.947656+0000) 2019-10-13T13:33:07.525 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:07.518+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.523023+0000) 2019-10-13T13:33:07.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:07.518+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.523855+0000) 2019-10-13T13:33:07.527 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:07.518+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.524523+0000) 2019-10-13T13:33:07.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:07.518+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.525205+0000) 2019-10-13T13:33:07.527 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:07.518+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.524954+0000) 2019-10-13T13:33:07.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:07.522+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.527609+0000) 2019-10-13T13:33:07.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:07.526+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.530157+0000) 2019-10-13T13:33:07.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:07.526+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:07.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:07.534+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.537742+0000) 2019-10-13T13:33:07.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:07.778+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.778777+0000) 2019-10-13T13:33:07.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:07.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:07.950931+0000) 2019-10-13T13:33:08.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:08.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:08.176291+0000) 2019-10-13T13:33:08.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:08.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:08.716027+0000) 2019-10-13T13:33:11.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:11.966+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:11.967114+0000) 2019-10-13T13:33:12.538 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:12.530+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.536140+0000) 2019-10-13T13:33:12.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:12.530+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.536207+0000) 2019-10-13T13:33:12.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:12.530+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.536434+0000) 2019-10-13T13:33:12.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:12.534+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.538252+0000) 2019-10-13T13:33:12.541 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:12.534+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.538875+0000) 2019-10-13T13:33:12.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:12.534+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.539498+0000) 2019-10-13T13:33:12.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:12.534+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.539870+0000) 2019-10-13T13:33:12.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:12.538+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:12.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:12.542+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:12.546776+0000) 2019-10-13T13:33:14.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:14.290+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:14.294785+0000) 2019-10-13T13:33:14.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:14.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:14.580486+0000) 2019-10-13T13:33:14.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:14.946+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:14.947896+0000) 2019-10-13T13:33:17.551 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:17.542+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.548959+0000) 2019-10-13T13:33:17.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:17.542+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.549457+0000) 2019-10-13T13:33:17.552 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:17.542+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.549691+0000) 2019-10-13T13:33:17.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:17.546+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.553485+0000) 2019-10-13T13:33:17.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:17.550+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.556462+0000) 2019-10-13T13:33:17.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:17.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.557843+0000) 2019-10-13T13:33:17.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:17.554+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.558429+0000) 2019-10-13T13:33:17.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:17.554+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185192 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:17.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:17.554+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.559322+0000) 2019-10-13T13:33:17.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:17.782+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.784847+0000) 2019-10-13T13:33:17.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:17.946+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:17.951085+0000) 2019-10-13T13:33:18.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:18.170+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:18.176475+0000) 2019-10-13T13:33:18.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:18.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:18.716226+0000) 2019-10-13T13:33:21.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:21.965+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:21.967283+0000) 2019-10-13T13:33:22.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:22.557+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.563233+0000) 2019-10-13T13:33:22.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:22.558+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.562797+0000) 2019-10-13T13:33:22.566 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:22.558+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.563756+0000) 2019-10-13T13:33:22.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:22.558+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.564500+0000) 2019-10-13T13:33:22.567 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:22.558+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.565286+0000) 2019-10-13T13:33:22.568 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:22.562+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.565936+0000) 2019-10-13T13:33:22.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:22.562+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.566882+0000) 2019-10-13T13:33:22.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:22.566+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:22.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:22.569+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:22.572751+0000) 2019-10-13T13:33:24.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:24.290+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:24.294984+0000) 2019-10-13T13:33:24.342 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:33:24.342 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:24.346 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:33:24.346 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:24.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:24.574+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:24.580686+0000) 2019-10-13T13:33:24.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:24.942+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:24.948049+0000) 2019-10-13T13:33:27.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:27.570+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.575851+0000) 2019-10-13T13:33:27.578 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:27.570+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.576233+0000) 2019-10-13T13:33:27.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:27.574+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.579176+0000) 2019-10-13T13:33:27.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:27.577+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.581444+0000) 2019-10-13T13:33:27.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:27.577+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.583832+0000) 2019-10-13T13:33:27.586 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:27.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:27.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:27.578+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.584497+0000) 2019-10-13T13:33:27.587 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:27.578+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.585407+0000) 2019-10-13T13:33:27.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:27.578+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.585334+0000) 2019-10-13T13:33:27.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:27.789+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.791920+0000) 2019-10-13T13:33:27.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:27.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:27.951261+0000) 2019-10-13T13:33:28.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:28.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:28.176689+0000) 2019-10-13T13:33:28.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:28.710+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:28.716404+0000) 2019-10-13T13:33:31.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:31.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:31.967474+0000) 2019-10-13T13:33:32.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:32.582+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.587466+0000) 2019-10-13T13:33:32.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:32.582+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.587921+0000) 2019-10-13T13:33:32.591 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:32.582+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.588945+0000) 2019-10-13T13:33:32.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:32.586+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.590573+0000) 2019-10-13T13:33:32.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:32.586+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.593614+0000) 2019-10-13T13:33:32.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:32.590+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:32.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:32.594+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.598263+0000) 2019-10-13T13:33:32.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:32.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.598815+0000) 2019-10-13T13:33:32.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:32.597+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:32.602196+0000) 2019-10-13T13:33:34.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:34.290+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:34.295227+0000) 2019-10-13T13:33:34.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:34.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:34.580920+0000) 2019-10-13T13:33:34.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:34.942+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:34.948244+0000) 2019-10-13T13:33:37.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:37.594+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.600881+0000) 2019-10-13T13:33:37.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:37.594+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.601281+0000) 2019-10-13T13:33:37.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:37.598+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.601787+0000) 2019-10-13T13:33:37.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:37.598+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.602184+0000) 2019-10-13T13:33:37.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:37.598+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.602721+0000) 2019-10-13T13:33:37.606 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:37.598+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.604053+0000) 2019-10-13T13:33:37.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:37.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.604650+0000) 2019-10-13T13:33:37.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:37.605+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.610120+0000) 2019-10-13T13:33:37.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:37.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185312 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:37.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:37.793+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.797244+0000) 2019-10-13T13:33:37.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:37.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:37.951435+0000) 2019-10-13T13:33:38.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:38.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:38.176901+0000) 2019-10-13T13:33:38.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:38.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:38.716612+0000) 2019-10-13T13:33:41.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:41.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:41.967666+0000) 2019-10-13T13:33:42.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:42.610+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.613632+0000) 2019-10-13T13:33:42.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:42.610+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.615781+0000) 2019-10-13T13:33:42.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:42.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.617843+0000) 2019-10-13T13:33:42.620 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:42.610+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.617400+0000) 2019-10-13T13:33:42.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:42.614+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.617825+0000) 2019-10-13T13:33:42.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:42.614+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.618286+0000) 2019-10-13T13:33:42.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:42.614+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.618734+0000) 2019-10-13T13:33:42.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:42.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:42.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:42.617+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:42.623904+0000) 2019-10-13T13:33:44.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:44.294+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:44.295459+0000) 2019-10-13T13:33:44.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:44.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:44.581153+0000) 2019-10-13T13:33:44.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:44.946+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:44.948445+0000) 2019-10-13T13:33:47.630 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:47.622+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.627635+0000) 2019-10-13T13:33:47.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:47.622+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.627895+0000) 2019-10-13T13:33:47.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:47.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.629538+0000) 2019-10-13T13:33:47.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:47.626+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.629965+0000) 2019-10-13T13:33:47.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:47.625+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.630838+0000) 2019-10-13T13:33:47.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:47.626+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.631217+0000) 2019-10-13T13:33:47.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:47.626+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.631829+0000) 2019-10-13T13:33:47.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:47.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185371 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:47.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:47.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.637097+0000) 2019-10-13T13:33:47.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:47.801+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.804593+0000) 2019-10-13T13:33:47.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:47.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:47.951627+0000) 2019-10-13T13:33:48.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:48.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:48.177117+0000) 2019-10-13T13:33:48.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:48.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:48.716813+0000) 2019-10-13T13:33:51.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:51.965+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:51.967865+0000) 2019-10-13T13:33:52.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:52.634+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.639837+0000) 2019-10-13T13:33:52.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:52.634+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.640795+0000) 2019-10-13T13:33:52.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:52.634+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.641614+0000) 2019-10-13T13:33:52.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:52.638+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.641986+0000) 2019-10-13T13:33:52.645 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:52.638+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.642013+0000) 2019-10-13T13:33:52.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:52.638+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.642413+0000) 2019-10-13T13:33:52.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:52.637+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.642977+0000) 2019-10-13T13:33:52.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:52.645+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:52.649008+0000) 2019-10-13T13:33:52.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:52.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:54.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:54.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:54.295690+0000) 2019-10-13T13:33:54.447 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:33:54.447 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:54.451 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:33:54.451 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:33:54.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:54.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:54.581353+0000) 2019-10-13T13:33:54.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:54.945+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:54.948661+0000) 2019-10-13T13:33:57.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:57.649+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.653896+0000) 2019-10-13T13:33:57.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:33:57.649+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.654615+0000) 2019-10-13T13:33:57.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:33:57.649+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.654779+0000) 2019-10-13T13:33:57.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:33:57.649+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.655903+0000) 2019-10-13T13:33:57.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:33:57.649+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.655253+0000) 2019-10-13T13:33:57.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:57.649+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.656183+0000) 2019-10-13T13:33:57.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:57.649+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.656536+0000) 2019-10-13T13:33:57.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:57.657+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.660742+0000) 2019-10-13T13:33:57.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:33:57.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:33:57.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:33:57.809+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.811889+0000) 2019-10-13T13:33:57.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:33:57.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:57.951834+0000) 2019-10-13T13:33:58.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:33:58.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:58.177297+0000) 2019-10-13T13:33:58.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:33:58.709+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:33:58.717047+0000) 2019-10-13T13:34:01.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:01.961+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:01.968047+0000) 2019-10-13T13:34:02.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:02.661+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.665277+0000) 2019-10-13T13:34:02.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:02.661+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.665794+0000) 2019-10-13T13:34:02.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:02.661+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.666024+0000) 2019-10-13T13:34:02.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:02.661+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.666297+0000) 2019-10-13T13:34:02.669 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:02.661+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.667056+0000) 2019-10-13T13:34:02.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:02.661+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.667418+0000) 2019-10-13T13:34:02.670 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:02.661+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.668701+0000) 2019-10-13T13:34:02.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:02.669+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:02.674966+0000) 2019-10-13T13:34:02.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:02.669+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:04.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:04.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:04.295897+0000) 2019-10-13T13:34:04.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:04.573+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:04.581547+0000) 2019-10-13T13:34:04.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:04.945+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:04.948898+0000) 2019-10-13T13:34:07.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:07.673+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.678193+0000) 2019-10-13T13:34:07.681 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:07.673+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.679263+0000) 2019-10-13T13:34:07.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:07.673+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.679309+0000) 2019-10-13T13:34:07.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:07.673+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.679907+0000) 2019-10-13T13:34:07.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:07.673+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.680672+0000) 2019-10-13T13:34:07.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:07.673+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.680810+0000) 2019-10-13T13:34:07.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:07.673+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.681672+0000) 2019-10-13T13:34:07.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:07.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185491 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:07.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:07.681+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.687937+0000) 2019-10-13T13:34:07.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:07.817+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.818954+0000) 2019-10-13T13:34:07.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:07.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:07.952015+0000) 2019-10-13T13:34:08.179 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:08.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:08.177485+0000) 2019-10-13T13:34:08.719 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:08.713+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:08.717264+0000) 2019-10-13T13:34:11.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:11.965+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:11.968254+0000) 2019-10-13T13:34:12.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:12.689+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.691386+0000) 2019-10-13T13:34:12.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:12.689+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.691907+0000) 2019-10-13T13:34:12.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:12.689+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.692404+0000) 2019-10-13T13:34:12.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:12.689+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.693149+0000) 2019-10-13T13:34:12.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:12.689+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.693780+0000) 2019-10-13T13:34:12.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:12.693+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.693958+0000) 2019-10-13T13:34:12.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:12.693+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.694755+0000) 2019-10-13T13:34:12.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:12.697+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:12.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:12.697+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:12.703596+0000) 2019-10-13T13:34:14.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:14.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:14.296121+0000) 2019-10-13T13:34:14.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:14.581+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:14.581782+0000) 2019-10-13T13:34:14.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:14.945+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:14.949115+0000) 2019-10-13T13:34:17.706 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:17.701+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.703567+0000) 2019-10-13T13:34:17.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:17.701+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.703819+0000) 2019-10-13T13:34:17.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:17.705+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.706615+0000) 2019-10-13T13:34:17.713 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:17.705+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.711290+0000) 2019-10-13T13:34:17.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:17.705+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.711372+0000) 2019-10-13T13:34:17.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:17.709+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:17.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:17.709+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.713399+0000) 2019-10-13T13:34:17.716 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:17.709+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.713699+0000) 2019-10-13T13:34:17.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:17.713+0000 7f0edc51c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.714523+0000) 2019-10-13T13:34:17.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:17.821+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.826133+0000) 2019-10-13T13:34:17.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:17.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:17.952148+0000) 2019-10-13T13:34:18.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:18.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:18.177659+0000) 2019-10-13T13:34:18.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:18.713+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:18.717486+0000) 2019-10-13T13:34:21.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:21.965+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:21.968403+0000) 2019-10-13T13:34:22.718 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:22.713+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.715457+0000) 2019-10-13T13:34:22.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:22.713+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.717557+0000) 2019-10-13T13:34:22.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:22.717+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.719749+0000) 2019-10-13T13:34:22.722 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:22.717+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.720568+0000) 2019-10-13T13:34:22.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:22.721+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.722145+0000) 2019-10-13T13:34:22.725 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:22.721+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.722356+0000) 2019-10-13T13:34:22.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:22.721+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.724149+0000) 2019-10-13T13:34:22.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:22.721+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:22.726457+0000) 2019-10-13T13:34:22.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:22.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:24.298 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:24.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:24.296327+0000) 2019-10-13T13:34:24.579 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:34:24.579 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:24.583 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:34:24.583 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:24.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:24.581+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:24.581972+0000) 2019-10-13T13:34:24.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:24.945+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:24.949311+0000) 2019-10-13T13:34:27.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:27.729+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.730452+0000) 2019-10-13T13:34:27.733 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:27.724+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.730974+0000) 2019-10-13T13:34:27.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:27.729+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.731440+0000) 2019-10-13T13:34:27.734 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:27.729+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.731576+0000) 2019-10-13T13:34:27.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:27.729+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.732209+0000) 2019-10-13T13:34:27.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:27.729+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.732588+0000) 2019-10-13T13:34:27.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:27.729+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.733436+0000) 2019-10-13T13:34:27.742 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:27.737+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:27.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:27.736+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.740516+0000) 2019-10-13T13:34:27.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:27.832+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.836966+0000) 2019-10-13T13:34:27.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:27.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:27.952307+0000) 2019-10-13T13:34:28.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:28.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:28.177835+0000) 2019-10-13T13:34:28.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:28.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:28.717705+0000) 2019-10-13T13:34:31.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:31.964+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:31.968629+0000) 2019-10-13T13:34:32.745 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:32.741+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.743071+0000) 2019-10-13T13:34:32.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:32.741+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.743835+0000) 2019-10-13T13:34:32.747 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:32.741+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.744484+0000) 2019-10-13T13:34:32.747 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:32.741+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.745167+0000) 2019-10-13T13:34:32.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:32.740+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.745604+0000) 2019-10-13T13:34:32.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:32.741+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.745609+0000) 2019-10-13T13:34:32.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:32.745+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.746011+0000) 2019-10-13T13:34:32.755 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:32.749+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:32.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:32.748+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:32.753557+0000) 2019-10-13T13:34:34.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:34.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:34.296519+0000) 2019-10-13T13:34:34.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:34.581+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:34.582226+0000) 2019-10-13T13:34:34.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:34.945+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:34.949510+0000) 2019-10-13T13:34:37.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:37.753+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.755699+0000) 2019-10-13T13:34:37.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:37.753+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.756229+0000) 2019-10-13T13:34:37.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:37.753+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.756416+0000) 2019-10-13T13:34:37.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:37.753+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.756739+0000) 2019-10-13T13:34:37.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:37.753+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.757332+0000) 2019-10-13T13:34:37.770 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:37.757+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.758686+0000) 2019-10-13T13:34:37.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:37.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185671 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:37.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:37.752+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.757987+0000) 2019-10-13T13:34:37.771 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:37.760+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.764731+0000) 2019-10-13T13:34:37.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:37.840+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.843572+0000) 2019-10-13T13:34:37.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:37.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:37.952477+0000) 2019-10-13T13:34:38.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:38.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:38.178010+0000) 2019-10-13T13:34:38.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:38.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:38.717925+0000) 2019-10-13T13:34:41.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:41.968+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:41.968832+0000) 2019-10-13T13:34:42.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:42.765+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.768947+0000) 2019-10-13T13:34:42.775 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:42.764+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.772052+0000) 2019-10-13T13:34:42.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:42.769+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.771527+0000) 2019-10-13T13:34:42.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:42.769+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.772082+0000) 2019-10-13T13:34:42.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:42.769+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.772515+0000) 2019-10-13T13:34:42.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:42.773+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.774691+0000) 2019-10-13T13:34:42.777 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:42.773+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.774748+0000) 2019-10-13T13:34:42.781 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:42.777+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:42.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:42.772+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:42.779277+0000) 2019-10-13T13:34:44.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:44.293+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:44.296751+0000) 2019-10-13T13:34:44.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:44.581+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:44.582427+0000) 2019-10-13T13:34:44.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:44.949+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:44.949711+0000) 2019-10-13T13:34:47.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:47.781+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.782062+0000) 2019-10-13T13:34:47.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:47.781+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.782528+0000) 2019-10-13T13:34:47.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:47.776+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.783437+0000) 2019-10-13T13:34:47.786 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:47.781+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.783591+0000) 2019-10-13T13:34:47.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:47.781+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.785414+0000) 2019-10-13T13:34:47.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:47.781+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.785614+0000) 2019-10-13T13:34:47.788 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:47.785+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.786791+0000) 2019-10-13T13:34:47.793 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:47.789+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185732 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:47.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:47.788+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.792905+0000) 2019-10-13T13:34:47.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:47.840+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.847980+0000) 2019-10-13T13:34:47.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:47.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:47.952654+0000) 2019-10-13T13:34:48.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:48.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:48.178212+0000) 2019-10-13T13:34:48.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:48.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:48.718105+0000) 2019-10-13T13:34:51.971 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:51.964+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:51.969053+0000) 2019-10-13T13:34:52.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:52.792+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.794097+0000) 2019-10-13T13:34:52.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:52.792+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.794325+0000) 2019-10-13T13:34:52.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:52.792+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.795424+0000) 2019-10-13T13:34:52.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:52.792+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.795965+0000) 2019-10-13T13:34:52.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:52.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.796372+0000) 2019-10-13T13:34:52.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:52.792+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.797709+0000) 2019-10-13T13:34:52.800 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:52.796+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.797770+0000) 2019-10-13T13:34:52.806 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:52.800+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:52.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:52.800+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:52.804606+0000) 2019-10-13T13:34:54.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:54.292+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:54.296971+0000) 2019-10-13T13:34:54.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:54.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:54.582627+0000) 2019-10-13T13:34:54.685 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:34:54.685 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:54.693 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:34:54.693 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:34:54.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:54.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:54.949941+0000) 2019-10-13T13:34:57.810 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:57.804+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.807657+0000) 2019-10-13T13:34:57.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:34:57.804+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.808079+0000) 2019-10-13T13:34:57.811 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:57.804+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.808486+0000) 2019-10-13T13:34:57.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:34:57.808+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.810937+0000) 2019-10-13T13:34:57.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:57.808+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.811742+0000) 2019-10-13T13:34:57.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:34:57.808+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.812424+0000) 2019-10-13T13:34:57.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:34:57.808+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.813039+0000) 2019-10-13T13:34:57.818 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:34:57.812+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185793 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:34:57.819 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:57.812+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.816806+0000) 2019-10-13T13:34:57.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:34:57.852+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.854428+0000) 2019-10-13T13:34:57.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:34:57.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:57.952806+0000) 2019-10-13T13:34:58.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:34:58.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:58.178397+0000) 2019-10-13T13:34:58.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:34:58.716+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:34:58.718283+0000) 2019-10-13T13:35:01.971 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:01.968+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:01.969282+0000) 2019-10-13T13:35:02.822 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:02.816+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.819537+0000) 2019-10-13T13:35:02.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:02.816+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.819921+0000) 2019-10-13T13:35:02.823 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:02.816+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.820623+0000) 2019-10-13T13:35:02.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:02.816+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.821486+0000) 2019-10-13T13:35:02.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:02.816+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.822134+0000) 2019-10-13T13:35:02.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:02.820+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.823902+0000) 2019-10-13T13:35:02.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:02.820+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.824247+0000) 2019-10-13T13:35:02.831 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:02.824+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:02.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:02.824+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:02.830113+0000) 2019-10-13T13:35:04.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:04.292+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:04.297171+0000) 2019-10-13T13:35:04.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:04.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:04.582837+0000) 2019-10-13T13:35:04.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:04.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:04.950137+0000) 2019-10-13T13:35:07.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:07.828+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.833041+0000) 2019-10-13T13:35:07.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:07.832+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.836552+0000) 2019-10-13T13:35:07.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:07.836+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.838984+0000) 2019-10-13T13:35:07.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:07.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.841419+0000) 2019-10-13T13:35:07.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:07.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:07.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:07.840+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.842739+0000) 2019-10-13T13:35:07.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:07.840+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.843064+0000) 2019-10-13T13:35:07.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:07.840+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.842806+0000) 2019-10-13T13:35:07.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:07.836+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.843799+0000) 2019-10-13T13:35:07.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:07.856+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.860329+0000) 2019-10-13T13:35:07.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:07.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:07.952989+0000) 2019-10-13T13:35:08.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:08.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:08.178578+0000) 2019-10-13T13:35:08.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:08.716+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:08.718451+0000) 2019-10-13T13:35:11.971 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:11.968+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:11.969475+0000) 2019-10-13T13:35:12.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:12.844+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.845946+0000) 2019-10-13T13:35:12.852 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:12.848+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.850295+0000) 2019-10-13T13:35:12.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:12.848+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.850778+0000) 2019-10-13T13:35:12.854 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:12.848+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.852267+0000) 2019-10-13T13:35:12.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:12.852+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.854161+0000) 2019-10-13T13:35:12.856 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:12.848+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.854757+0000) 2019-10-13T13:35:12.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:12.852+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.854739+0000) 2019-10-13T13:35:12.857 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:12.852+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:12.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:12.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:12.855749+0000) 2019-10-13T13:35:14.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:14.292+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:14.297365+0000) 2019-10-13T13:35:14.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:14.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:14.583042+0000) 2019-10-13T13:35:14.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:14.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:14.950370+0000) 2019-10-13T13:35:17.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:17.856+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.860214+0000) 2019-10-13T13:35:17.863 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:17.856+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.860673+0000) 2019-10-13T13:35:17.863 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:17.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.861289+0000) 2019-10-13T13:35:17.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:17.860+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.861892+0000) 2019-10-13T13:35:17.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:17.860+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.863152+0000) 2019-10-13T13:35:17.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:17.860+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.863807+0000) 2019-10-13T13:35:17.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:17.860+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.863588+0000) 2019-10-13T13:35:17.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:17.860+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.865095+0000) 2019-10-13T13:35:17.869 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:17.860+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.867588+0000) 2019-10-13T13:35:17.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:17.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:17.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:17.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:17.953171+0000) 2019-10-13T13:35:18.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:18.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:18.178755+0000) 2019-10-13T13:35:18.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:18.716+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:18.718633+0000) 2019-10-13T13:35:21.971 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:21.968+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:21.969684+0000) 2019-10-13T13:35:22.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:22.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.872485+0000) 2019-10-13T13:35:22.875 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:22.868+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.872575+0000) 2019-10-13T13:35:22.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:22.872+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.874811+0000) 2019-10-13T13:35:22.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:22.872+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.875112+0000) 2019-10-13T13:35:22.879 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:22.872+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.877081+0000) 2019-10-13T13:35:22.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:22.876+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.878712+0000) 2019-10-13T13:35:22.881 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:22.876+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.879100+0000) 2019-10-13T13:35:22.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:22.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:22.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:22.876+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:22.880808+0000) 2019-10-13T13:35:24.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:24.296+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:24.297590+0000) 2019-10-13T13:35:24.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:24.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:24.583237+0000) 2019-10-13T13:35:24.813 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:35:24.813 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:24.823 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:35:24.823 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:24.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:24.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:24.950595+0000) 2019-10-13T13:35:27.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:27.868+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.871674+0000) 2019-10-13T13:35:27.886 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:27.880+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.883341+0000) 2019-10-13T13:35:27.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:27.880+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.883901+0000) 2019-10-13T13:35:27.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:27.880+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.884533+0000) 2019-10-13T13:35:27.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:27.880+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.884893+0000) 2019-10-13T13:35:27.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:27.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.885560+0000) 2019-10-13T13:35:27.889 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:27.884+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.886551+0000) 2019-10-13T13:35:27.889 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:27.884+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.887594+0000) 2019-10-13T13:35:27.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:27.892+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.893649+0000) 2019-10-13T13:35:27.897 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:27.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:27.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:27.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:27.953375+0000) 2019-10-13T13:35:28.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:28.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:28.178938+0000) 2019-10-13T13:35:28.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:28.716+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:28.718759+0000) 2019-10-13T13:35:31.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:31.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:31.969909+0000) 2019-10-13T13:35:32.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:32.899+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.902130+0000) 2019-10-13T13:35:32.906 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:32.896+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.898899+0000) 2019-10-13T13:35:32.906 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:32.896+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.899782+0000) 2019-10-13T13:35:32.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:32.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.900256+0000) 2019-10-13T13:35:32.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:32.896+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.900468+0000) 2019-10-13T13:35:32.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:32.896+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.900827+0000) 2019-10-13T13:35:32.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:32.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.902368+0000) 2019-10-13T13:35:32.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:32.903+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:32.904750+0000) 2019-10-13T13:35:32.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:32.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:34.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:34.296+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:34.297797+0000) 2019-10-13T13:35:34.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:34.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:34.583433+0000) 2019-10-13T13:35:34.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:34.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:34.950825+0000) 2019-10-13T13:35:37.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:37.875+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.878077+0000) 2019-10-13T13:35:37.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:37.908+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.911758+0000) 2019-10-13T13:35:37.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:37.908+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.912159+0000) 2019-10-13T13:35:37.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:37.908+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.912588+0000) 2019-10-13T13:35:37.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:37.908+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.913265+0000) 2019-10-13T13:35:37.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:37.912+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.914357+0000) 2019-10-13T13:35:37.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:37.911+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.915741+0000) 2019-10-13T13:35:37.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:37.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.916144+0000) 2019-10-13T13:35:37.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:37.919+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.921113+0000) 2019-10-13T13:35:37.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:37.916+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:37.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:37.948+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:37.953540+0000) 2019-10-13T13:35:38.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:38.176+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:38.179151+0000) 2019-10-13T13:35:38.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:38.716+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:38.718907+0000) 2019-10-13T13:35:41.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:41.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:41.970124+0000) 2019-10-13T13:35:42.928 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:42.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.925530+0000) 2019-10-13T13:35:42.928 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:42.924+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.925709+0000) 2019-10-13T13:35:42.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:42.924+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.926355+0000) 2019-10-13T13:35:42.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:42.924+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.926471+0000) 2019-10-13T13:35:42.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:42.924+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.927005+0000) 2019-10-13T13:35:42.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:42.924+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.927465+0000) 2019-10-13T13:35:42.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:42.927+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.929052+0000) 2019-10-13T13:35:42.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:42.931+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:42.932475+0000) 2019-10-13T13:35:42.935 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:42.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:44.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:44.296+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:44.298008+0000) 2019-10-13T13:35:44.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:44.580+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:44.583625+0000) 2019-10-13T13:35:44.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:44.948+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:44.951028+0000) 2019-10-13T13:35:47.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:47.883+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.884555+0000) 2019-10-13T13:35:47.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:47.931+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.937322+0000) 2019-10-13T13:35:47.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:47.935+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.938129+0000) 2019-10-13T13:35:47.941 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:47.935+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.938642+0000) 2019-10-13T13:35:47.941 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:47.935+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.939038+0000) 2019-10-13T13:35:47.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:47.935+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.939279+0000) 2019-10-13T13:35:47.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:47.935+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.939699+0000) 2019-10-13T13:35:47.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:47.935+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.940235+0000) 2019-10-13T13:35:47.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:47.939+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.943070+0000) 2019-10-13T13:35:47.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:47.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:47.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:47.947+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:47.953662+0000) 2019-10-13T13:35:48.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:48.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:48.179330+0000) 2019-10-13T13:35:48.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:48.715+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:48.719035+0000) 2019-10-13T13:35:51.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:51.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:51.970357+0000) 2019-10-13T13:35:52.952 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:52.943+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.949552+0000) 2019-10-13T13:35:52.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:52.947+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.950160+0000) 2019-10-13T13:35:52.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:52.947+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.950722+0000) 2019-10-13T13:35:52.955 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:52.951+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.952178+0000) 2019-10-13T13:35:52.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:52.947+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.952423+0000) 2019-10-13T13:35:52.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:52.947+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.953315+0000) 2019-10-13T13:35:52.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:52.947+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.953527+0000) 2019-10-13T13:35:52.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:52.955+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:52.957332+0000) 2019-10-13T13:35:52.961 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:52.955+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:54.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:54.295+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:54.298224+0000) 2019-10-13T13:35:54.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:54.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:54.583841+0000) 2019-10-13T13:35:54.948 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:35:54.948 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:54.951 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:35:54.952 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:35:54.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:54.947+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:54.951210+0000) 2019-10-13T13:35:57.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:57.887+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.891248+0000) 2019-10-13T13:35:57.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:57.951+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.953783+0000) 2019-10-13T13:35:57.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:35:57.959+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.962909+0000) 2019-10-13T13:35:57.966 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:57.959+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.963130+0000) 2019-10-13T13:35:57.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:35:57.959+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.964065+0000) 2019-10-13T13:35:57.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:35:57.963+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.964948+0000) 2019-10-13T13:35:57.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:35:57.959+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.964906+0000) 2019-10-13T13:35:57.970 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:57.963+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.966216+0000) 2019-10-13T13:35:57.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:35:57.963+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.966528+0000) 2019-10-13T13:35:57.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:35:57.967+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:57.969157+0000) 2019-10-13T13:35:57.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:35:57.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:35:58.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:35:58.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:58.179505+0000) 2019-10-13T13:35:58.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:35:58.715+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:35:58.719203+0000) 2019-10-13T13:36:01.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:01.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:01.970504+0000) 2019-10-13T13:36:02.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:02.971+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.974436+0000) 2019-10-13T13:36:02.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:02.975+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.977986+0000) 2019-10-13T13:36:02.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:02.979+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.981959+0000) 2019-10-13T13:36:02.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:02.979+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.983261+0000) 2019-10-13T13:36:02.985 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:02.979+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.983517+0000) 2019-10-13T13:36:02.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:02.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186181 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:02.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:02.979+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.984530+0000) 2019-10-13T13:36:02.987 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:02.979+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.984698+0000) 2019-10-13T13:36:02.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:02.991+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:02.992290+0000) 2019-10-13T13:36:04.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:04.295+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:04.298431+0000) 2019-10-13T13:36:04.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:04.583+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:04.584049+0000) 2019-10-13T13:36:04.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:04.947+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:04.951444+0000) 2019-10-13T13:36:07.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:07.899+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.902865+0000) 2019-10-13T13:36:07.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:07.951+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.954040+0000) 2019-10-13T13:36:07.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:07.983+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.988861+0000) 2019-10-13T13:36:07.992 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:07.987+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.989914+0000) 2019-10-13T13:36:07.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:07.987+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.990141+0000) 2019-10-13T13:36:07.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:07.987+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.990873+0000) 2019-10-13T13:36:07.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:07.987+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.990914+0000) 2019-10-13T13:36:07.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:07.987+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.991441+0000) 2019-10-13T13:36:07.995 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:07.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.992649+0000) 2019-10-13T13:36:07.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:07.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:07.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:07.995+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:07.997824+0000) 2019-10-13T13:36:08.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:08.179+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:08.179689+0000) 2019-10-13T13:36:08.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:08.715+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:08.719400+0000) 2019-10-13T13:36:11.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:11.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:11.970690+0000) 2019-10-13T13:36:13.003 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:12.995+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.000948+0000) 2019-10-13T13:36:13.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:12.999+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.001873+0000) 2019-10-13T13:36:13.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:12.999+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.002016+0000) 2019-10-13T13:36:13.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:12.999+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.002318+0000) 2019-10-13T13:36:13.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:12.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.003799+0000) 2019-10-13T13:36:13.006 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:12.999+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.003885+0000) 2019-10-13T13:36:13.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:12.999+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.004272+0000) 2019-10-13T13:36:13.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:13.007+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:13.009426+0000) 2019-10-13T13:36:13.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:13.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:14.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:14.295+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:14.298628+0000) 2019-10-13T13:36:14.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:14.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:14.584248+0000) 2019-10-13T13:36:14.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:14.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:14.951652+0000) 2019-10-13T13:36:17.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:17.907+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:17.909412+0000) 2019-10-13T13:36:17.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:17.951+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:17.954242+0000) 2019-10-13T13:36:18.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:18.007+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.012612+0000) 2019-10-13T13:36:18.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:18.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.014021+0000) 2019-10-13T13:36:18.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:18.011+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.015720+0000) 2019-10-13T13:36:18.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:18.011+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.015923+0000) 2019-10-13T13:36:18.019 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:18.011+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.016984+0000) 2019-10-13T13:36:18.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:18.015+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.017719+0000) 2019-10-13T13:36:18.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:18.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.018595+0000) 2019-10-13T13:36:18.023 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:18.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:18.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:18.023+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.027949+0000) 2019-10-13T13:36:18.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:18.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.179868+0000) 2019-10-13T13:36:18.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:18.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:18.719607+0000) 2019-10-13T13:36:21.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:21.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:21.970883+0000) 2019-10-13T13:36:23.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:23.023+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.025893+0000) 2019-10-13T13:36:23.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:23.023+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.026229+0000) 2019-10-13T13:36:23.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:23.023+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.026245+0000) 2019-10-13T13:36:23.029 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:23.023+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.027225+0000) 2019-10-13T13:36:23.030 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:23.023+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.028196+0000) 2019-10-13T13:36:23.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:23.027+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.028674+0000) 2019-10-13T13:36:23.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:23.023+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.028751+0000) 2019-10-13T13:36:23.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:23.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186301 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:23.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:23.035+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:23.036805+0000) 2019-10-13T13:36:24.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:24.295+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:24.298844+0000) 2019-10-13T13:36:24.586 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:24.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:24.584445+0000) 2019-10-13T13:36:24.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:24.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:24.951849+0000) 2019-10-13T13:36:25.066 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:36:25.066 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:25.070 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:36:25.070 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:27.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:27.915+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:27.915827+0000) 2019-10-13T13:36:27.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:27.951+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:27.954444+0000) 2019-10-13T13:36:28.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:28.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.039178+0000) 2019-10-13T13:36:28.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:28.035+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.039667+0000) 2019-10-13T13:36:28.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:28.039+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.040822+0000) 2019-10-13T13:36:28.043 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:28.035+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.040837+0000) 2019-10-13T13:36:28.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:28.035+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.041282+0000) 2019-10-13T13:36:28.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:28.035+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.041579+0000) 2019-10-13T13:36:28.044 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:28.039+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.042327+0000) 2019-10-13T13:36:28.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:28.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.046774+0000) 2019-10-13T13:36:28.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:28.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:28.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:28.179+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.180049+0000) 2019-10-13T13:36:28.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:28.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:28.719780+0000) 2019-10-13T13:36:31.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:31.967+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:31.971070+0000) 2019-10-13T13:36:33.053 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:33.047+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.050555+0000) 2019-10-13T13:36:33.053 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:33.047+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.051420+0000) 2019-10-13T13:36:33.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:33.047+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.051723+0000) 2019-10-13T13:36:33.054 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:33.047+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.052270+0000) 2019-10-13T13:36:33.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:33.047+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.052761+0000) 2019-10-13T13:36:33.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:33.047+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.053702+0000) 2019-10-13T13:36:33.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:33.051+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.053899+0000) 2019-10-13T13:36:33.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:33.059+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:33.060142+0000) 2019-10-13T13:36:33.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:33.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:34.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:34.295+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:34.299038+0000) 2019-10-13T13:36:34.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:34.579+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:34.584671+0000) 2019-10-13T13:36:34.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:34.947+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:34.952045+0000) 2019-10-13T13:36:37.924 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:37.918+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:37.922376+0000) 2019-10-13T13:36:37.957 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:37.951+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:37.954657+0000) 2019-10-13T13:36:38.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:38.059+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.063462+0000) 2019-10-13T13:36:38.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:38.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.064228+0000) 2019-10-13T13:36:38.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:38.059+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.064499+0000) 2019-10-13T13:36:38.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:38.059+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.064735+0000) 2019-10-13T13:36:38.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:38.063+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.065914+0000) 2019-10-13T13:36:38.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:38.062+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.066616+0000) 2019-10-13T13:36:38.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:38.063+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.067288+0000) 2019-10-13T13:36:38.074 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:38.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:38.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:38.070+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.073468+0000) 2019-10-13T13:36:38.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:38.175+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.180261+0000) 2019-10-13T13:36:38.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:38.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:38.719971+0000) 2019-10-13T13:36:41.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:41.970+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:41.971263+0000) 2019-10-13T13:36:43.078 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:43.074+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.076174+0000) 2019-10-13T13:36:43.079 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:43.070+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.076390+0000) 2019-10-13T13:36:43.079 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:43.070+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.077077+0000) 2019-10-13T13:36:43.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:43.074+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.078114+0000) 2019-10-13T13:36:43.080 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:43.074+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.078320+0000) 2019-10-13T13:36:43.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:43.074+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.078633+0000) 2019-10-13T13:36:43.081 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:43.074+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.079246+0000) 2019-10-13T13:36:43.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:43.078+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:43.084020+0000) 2019-10-13T13:36:43.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:43.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186418 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:44.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:44.294+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:44.299250+0000) 2019-10-13T13:36:44.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:44.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:44.584896+0000) 2019-10-13T13:36:44.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:44.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:44.952202+0000) 2019-10-13T13:36:47.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:47.930+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:47.931587+0000) 2019-10-13T13:36:47.957 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:47.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:47.954852+0000) 2019-10-13T13:36:48.090 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:48.082+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.087595+0000) 2019-10-13T13:36:48.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:48.082+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.087959+0000) 2019-10-13T13:36:48.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:48.082+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.088725+0000) 2019-10-13T13:36:48.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:48.086+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.089195+0000) 2019-10-13T13:36:48.092 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:48.082+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.089204+0000) 2019-10-13T13:36:48.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:48.082+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.089550+0000) 2019-10-13T13:36:48.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:48.086+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.090124+0000) 2019-10-13T13:36:48.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:48.094+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.096175+0000) 2019-10-13T13:36:48.098 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:48.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:48.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:48.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.180485+0000) 2019-10-13T13:36:48.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:48.718+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:48.720147+0000) 2019-10-13T13:36:51.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:51.970+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:51.971460+0000) 2019-10-13T13:36:53.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:53.094+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.100027+0000) 2019-10-13T13:36:53.104 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:53.098+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.102761+0000) 2019-10-13T13:36:53.105 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:53.098+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.102795+0000) 2019-10-13T13:36:53.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:53.098+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.103497+0000) 2019-10-13T13:36:53.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:53.098+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.104000+0000) 2019-10-13T13:36:53.106 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:53.098+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.104269+0000) 2019-10-13T13:36:53.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:53.098+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.105124+0000) 2019-10-13T13:36:53.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:53.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:53.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:53.110+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:53.112192+0000) 2019-10-13T13:36:54.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:54.294+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:54.299453+0000) 2019-10-13T13:36:54.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:54.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:54.585154+0000) 2019-10-13T13:36:54.954 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:54.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:54.952409+0000) 2019-10-13T13:36:55.404 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:36:55.405 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:55.408 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:36:55.408 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:36:57.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:57.934+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:57.938108+0000) 2019-10-13T13:36:57.957 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:57.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:57.955077+0000) 2019-10-13T13:36:58.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:36:58.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.112733+0000) 2019-10-13T13:36:58.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:36:58.110+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.114583+0000) 2019-10-13T13:36:58.117 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:58.110+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.114850+0000) 2019-10-13T13:36:58.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:36:58.114+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.116841+0000) 2019-10-13T13:36:58.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:58.110+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.117193+0000) 2019-10-13T13:36:58.120 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:36:58.114+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.118029+0000) 2019-10-13T13:36:58.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:36:58.114+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.118545+0000) 2019-10-13T13:36:58.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:36:58.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:36:58.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:36:58.122+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.124155+0000) 2019-10-13T13:36:58.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:36:58.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.180666+0000) 2019-10-13T13:36:58.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:36:58.718+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:36:58.720320+0000) 2019-10-13T13:37:01.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:01.966+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:01.971654+0000) 2019-10-13T13:37:03.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:03.122+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.126938+0000) 2019-10-13T13:37:03.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:03.122+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.127296+0000) 2019-10-13T13:37:03.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:03.126+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.128268+0000) 2019-10-13T13:37:03.130 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:03.122+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.127721+0000) 2019-10-13T13:37:03.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:03.122+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.128603+0000) 2019-10-13T13:37:03.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:03.122+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.128298+0000) 2019-10-13T13:37:03.131 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:03.122+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.128921+0000) 2019-10-13T13:37:03.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:03.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:03.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:03.138+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:03.140950+0000) 2019-10-13T13:37:04.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:04.294+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:04.299662+0000) 2019-10-13T13:37:04.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:04.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:04.585348+0000) 2019-10-13T13:37:04.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:04.946+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:04.952609+0000) 2019-10-13T13:37:07.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:07.942+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:07.944551+0000) 2019-10-13T13:37:07.957 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:07.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:07.955247+0000) 2019-10-13T13:37:08.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:08.134+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.138323+0000) 2019-10-13T13:37:08.142 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:08.134+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.139818+0000) 2019-10-13T13:37:08.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:08.134+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.140113+0000) 2019-10-13T13:37:08.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:08.134+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.140703+0000) 2019-10-13T13:37:08.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:08.134+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.140942+0000) 2019-10-13T13:37:08.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:08.138+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.141703+0000) 2019-10-13T13:37:08.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:08.138+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.142224+0000) 2019-10-13T13:37:08.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:08.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.148257+0000) 2019-10-13T13:37:08.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:08.142+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:08.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:08.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.180826+0000) 2019-10-13T13:37:08.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:08.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:08.720523+0000) 2019-10-13T13:37:11.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:11.966+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:11.971843+0000) 2019-10-13T13:37:13.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:13.146+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.151894+0000) 2019-10-13T13:37:13.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:13.146+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.152721+0000) 2019-10-13T13:37:13.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:13.146+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.153489+0000) 2019-10-13T13:37:13.156 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:13.150+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.154167+0000) 2019-10-13T13:37:13.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:13.150+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.154419+0000) 2019-10-13T13:37:13.157 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:13.150+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.154730+0000) 2019-10-13T13:37:13.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:13.150+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.155163+0000) 2019-10-13T13:37:13.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:13.154+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:13.158940+0000) 2019-10-13T13:37:13.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:13.154+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:14.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:14.298+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:14.299897+0000) 2019-10-13T13:37:14.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:14.578+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:14.585551+0000) 2019-10-13T13:37:14.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:14.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:14.952830+0000) 2019-10-13T13:37:17.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:17.946+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:17.951060+0000) 2019-10-13T13:37:17.957 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:17.950+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:17.955442+0000) 2019-10-13T13:37:18.166 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:18.158+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.163943+0000) 2019-10-13T13:37:18.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:18.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.164627+0000) 2019-10-13T13:37:18.167 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:18.158+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.165434+0000) 2019-10-13T13:37:18.168 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:18.162+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.165923+0000) 2019-10-13T13:37:18.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:18.162+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.166181+0000) 2019-10-13T13:37:18.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:18.162+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.166586+0000) 2019-10-13T13:37:18.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:18.162+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.167313+0000) 2019-10-13T13:37:18.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:18.170+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.172386+0000) 2019-10-13T13:37:18.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:18.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:18.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:18.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.180945+0000) 2019-10-13T13:37:18.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:18.718+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:18.720715+0000) 2019-10-13T13:37:21.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:21.966+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:21.972031+0000) 2019-10-13T13:37:23.180 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:23.170+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.177658+0000) 2019-10-13T13:37:23.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:23.174+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.178053+0000) 2019-10-13T13:37:23.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:23.174+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.178731+0000) 2019-10-13T13:37:23.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:23.174+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.179027+0000) 2019-10-13T13:37:23.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:23.174+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.179718+0000) 2019-10-13T13:37:23.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:23.174+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.180070+0000) 2019-10-13T13:37:23.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:23.174+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.179719+0000) 2019-10-13T13:37:23.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:23.178+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:23.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:23.182+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:23.187292+0000) 2019-10-13T13:37:24.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:24.298+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:24.300120+0000) 2019-10-13T13:37:24.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:24.582+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:24.585770+0000) 2019-10-13T13:37:24.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:24.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:24.953029+0000) 2019-10-13T13:37:25.450 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:37:25.450 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:25.454 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:37:25.454 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:27.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:27.954+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:27.955650+0000) 2019-10-13T13:37:27.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:27.954+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:27.957556+0000) 2019-10-13T13:37:28.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:28.174+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.181110+0000) 2019-10-13T13:37:28.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:28.182+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.188662+0000) 2019-10-13T13:37:28.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:28.182+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.189718+0000) 2019-10-13T13:37:28.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:28.186+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.191405+0000) 2019-10-13T13:37:28.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:28.186+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.191587+0000) 2019-10-13T13:37:28.195 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:28.186+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.193254+0000) 2019-10-13T13:37:28.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:28.190+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.195015+0000) 2019-10-13T13:37:28.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:28.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.196324+0000) 2019-10-13T13:37:28.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:28.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:28.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:28.194+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.199997+0000) 2019-10-13T13:37:28.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:28.714+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:28.720904+0000) 2019-10-13T13:37:31.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:31.970+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:31.972253+0000) 2019-10-13T13:37:33.205 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:33.198+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.202670+0000) 2019-10-13T13:37:33.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:33.198+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.203229+0000) 2019-10-13T13:37:33.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:33.198+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.203881+0000) 2019-10-13T13:37:33.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:33.198+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.204577+0000) 2019-10-13T13:37:33.207 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:33.198+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.204678+0000) 2019-10-13T13:37:33.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:33.202+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.205290+0000) 2019-10-13T13:37:33.208 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:33.198+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.205191+0000) 2019-10-13T13:37:33.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:33.206+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:33.209414+0000) 2019-10-13T13:37:33.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:33.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186713 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:34.302 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:34.294+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:34.300358+0000) 2019-10-13T13:37:34.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:34.582+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:34.585970+0000) 2019-10-13T13:37:34.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:34.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:34.953224+0000) 2019-10-13T13:37:37.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:37.949+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:37.955885+0000) 2019-10-13T13:37:37.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:37.962+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:37.964120+0000) 2019-10-13T13:37:38.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:38.173+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.181303+0000) 2019-10-13T13:37:38.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:38.209+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.214732+0000) 2019-10-13T13:37:38.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:38.209+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.215557+0000) 2019-10-13T13:37:38.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:38.209+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.216038+0000) 2019-10-13T13:37:38.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:38.213+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.217972+0000) 2019-10-13T13:37:38.220 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:38.214+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.218369+0000) 2019-10-13T13:37:38.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:38.213+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.219816+0000) 2019-10-13T13:37:38.223 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:38.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.221251+0000) 2019-10-13T13:37:38.226 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:38.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186745 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:38.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:38.222+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.227362+0000) 2019-10-13T13:37:38.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:38.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:38.721073+0000) 2019-10-13T13:37:41.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:41.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:41.972459+0000) 2019-10-13T13:37:43.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:43.221+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.227499+0000) 2019-10-13T13:37:43.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:43.221+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.228229+0000) 2019-10-13T13:37:43.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:43.225+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.230255+0000) 2019-10-13T13:37:43.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:43.225+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.232041+0000) 2019-10-13T13:37:43.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:43.225+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.232622+0000) 2019-10-13T13:37:43.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:43.225+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.232448+0000) 2019-10-13T13:37:43.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:43.225+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.233397+0000) 2019-10-13T13:37:43.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:43.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:43.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:43.237+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:43.242118+0000) 2019-10-13T13:37:44.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:44.297+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:44.300603+0000) 2019-10-13T13:37:44.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:44.581+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:44.586187+0000) 2019-10-13T13:37:44.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:44.949+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:44.953428+0000) 2019-10-13T13:37:47.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:47.953+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:47.956083+0000) 2019-10-13T13:37:47.977 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:47.969+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:47.975562+0000) 2019-10-13T13:37:48.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:48.177+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.181501+0000) 2019-10-13T13:37:48.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:48.233+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.240738+0000) 2019-10-13T13:37:48.243 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:48.237+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.241337+0000) 2019-10-13T13:37:48.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:48.233+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.241596+0000) 2019-10-13T13:37:48.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:48.237+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.243011+0000) 2019-10-13T13:37:48.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:48.237+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.244931+0000) 2019-10-13T13:37:48.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:48.237+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.244935+0000) 2019-10-13T13:37:48.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:48.237+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.245502+0000) 2019-10-13T13:37:48.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:48.245+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.249184+0000) 2019-10-13T13:37:48.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:48.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:48.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:48.713+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:48.721213+0000) 2019-10-13T13:37:51.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:51.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:51.972681+0000) 2019-10-13T13:37:53.255 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:53.249+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.252983+0000) 2019-10-13T13:37:53.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:53.253+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.253902+0000) 2019-10-13T13:37:53.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:53.253+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.254062+0000) 2019-10-13T13:37:53.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:53.253+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.254551+0000) 2019-10-13T13:37:53.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:53.253+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.255015+0000) 2019-10-13T13:37:53.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:53.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.255391+0000) 2019-10-13T13:37:53.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:53.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.255478+0000) 2019-10-13T13:37:53.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:53.261+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:53.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:53.257+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:53.263212+0000) 2019-10-13T13:37:54.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:54.297+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:54.300812+0000) 2019-10-13T13:37:54.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:54.585+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:54.586372+0000) 2019-10-13T13:37:54.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:54.949+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:54.953655+0000) 2019-10-13T13:37:55.800 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:37:55.800 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:55.804 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:37:55.804 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:37:57.958 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:57.953+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:57.956284+0000) 2019-10-13T13:37:57.984 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:57.981+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:57.982128+0000) 2019-10-13T13:37:58.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:58.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.181714+0000) 2019-10-13T13:37:58.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:37:58.265+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.266749+0000) 2019-10-13T13:37:58.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:37:58.265+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.266848+0000) 2019-10-13T13:37:58.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:58.265+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.267237+0000) 2019-10-13T13:37:58.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:37:58.265+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.267725+0000) 2019-10-13T13:37:58.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:37:58.265+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.268125+0000) 2019-10-13T13:37:58.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:37:58.265+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.269314+0000) 2019-10-13T13:37:58.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:37:58.269+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.269862+0000) 2019-10-13T13:37:58.274 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:37:58.269+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.272544+0000) 2019-10-13T13:37:58.276 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:37:58.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:37:58.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:37:58.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:37:58.721383+0000) 2019-10-13T13:38:01.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:01.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:01.972925+0000) 2019-10-13T13:38:03.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:03.273+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.277282+0000) 2019-10-13T13:38:03.280 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:03.273+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.277533+0000) 2019-10-13T13:38:03.281 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:03.277+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.278933+0000) 2019-10-13T13:38:03.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:03.277+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.279252+0000) 2019-10-13T13:38:03.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:03.277+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.280085+0000) 2019-10-13T13:38:03.282 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:03.277+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.280720+0000) 2019-10-13T13:38:03.283 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:03.277+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.280696+0000) 2019-10-13T13:38:03.288 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:03.281+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:03.286368+0000) 2019-10-13T13:38:03.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:03.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:04.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:04.297+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:04.301036+0000) 2019-10-13T13:38:04.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:04.585+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:04.586567+0000) 2019-10-13T13:38:04.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:04.953+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:04.953853+0000) 2019-10-13T13:38:07.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:07.953+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:07.956514+0000) 2019-10-13T13:38:07.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:07.985+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:07.988754+0000) 2019-10-13T13:38:08.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:08.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.181968+0000) 2019-10-13T13:38:08.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:08.285+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.289041+0000) 2019-10-13T13:38:08.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:08.285+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.289417+0000) 2019-10-13T13:38:08.293 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:08.289+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.290699+0000) 2019-10-13T13:38:08.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:08.289+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.291615+0000) 2019-10-13T13:38:08.294 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:08.289+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.291621+0000) 2019-10-13T13:38:08.294 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:08.289+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.292593+0000) 2019-10-13T13:38:08.295 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:08.289+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.293135+0000) 2019-10-13T13:38:08.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:08.293+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.296653+0000) 2019-10-13T13:38:08.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:08.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:08.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:08.717+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:08.721560+0000) 2019-10-13T13:38:11.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:11.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:11.973147+0000) 2019-10-13T13:38:13.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:13.301+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.301986+0000) 2019-10-13T13:38:13.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:13.301+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.303318+0000) 2019-10-13T13:38:13.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:13.301+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.303799+0000) 2019-10-13T13:38:13.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:13.301+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.304320+0000) 2019-10-13T13:38:13.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:13.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.304951+0000) 2019-10-13T13:38:13.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:13.301+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.305478+0000) 2019-10-13T13:38:13.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:13.301+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.305376+0000) 2019-10-13T13:38:13.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:13.305+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:13.309618+0000) 2019-10-13T13:38:13.313 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:13.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:14.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:14.297+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:14.301296+0000) 2019-10-13T13:38:14.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:14.585+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:14.586727+0000) 2019-10-13T13:38:14.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:14.953+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:14.954053+0000) 2019-10-13T13:38:17.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:17.953+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:17.956743+0000) 2019-10-13T13:38:18.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:17.997+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.000246+0000) 2019-10-13T13:38:18.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:18.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.182173+0000) 2019-10-13T13:38:18.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:18.313+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.314576+0000) 2019-10-13T13:38:18.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:18.313+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.315192+0000) 2019-10-13T13:38:18.318 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:18.313+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.316125+0000) 2019-10-13T13:38:18.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:18.313+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.317330+0000) 2019-10-13T13:38:18.320 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:18.313+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.317433+0000) 2019-10-13T13:38:18.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:18.317+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.318645+0000) 2019-10-13T13:38:18.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:18.317+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.319885+0000) 2019-10-13T13:38:18.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:18.317+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.323331+0000) 2019-10-13T13:38:18.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:18.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:18.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:18.721+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:18.721765+0000) 2019-10-13T13:38:21.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:21.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:21.973338+0000) 2019-10-13T13:38:23.330 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:23.325+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.327517+0000) 2019-10-13T13:38:23.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:23.325+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.328099+0000) 2019-10-13T13:38:23.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:23.325+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.328951+0000) 2019-10-13T13:38:23.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:23.325+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.329317+0000) 2019-10-13T13:38:23.334 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:23.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.332269+0000) 2019-10-13T13:38:23.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:23.333+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.334147+0000) 2019-10-13T13:38:23.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:23.333+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.335233+0000) 2019-10-13T13:38:23.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:23.333+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:23.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:23.333+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:23.338502+0000) 2019-10-13T13:38:24.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:24.297+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:24.301479+0000) 2019-10-13T13:38:24.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:24.585+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:24.586937+0000) 2019-10-13T13:38:24.956 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:24.953+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:24.954252+0000) 2019-10-13T13:38:25.921 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:38:25.922 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:25.925 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:38:25.925 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:27.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:27.953+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:27.956963+0000) 2019-10-13T13:38:28.014 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:28.009+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.011708+0000) 2019-10-13T13:38:28.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:28.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.182365+0000) 2019-10-13T13:38:28.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:28.337+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.340936+0000) 2019-10-13T13:38:28.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:28.337+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.341450+0000) 2019-10-13T13:38:28.348 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:28.337+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.343766+0000) 2019-10-13T13:38:28.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:28.341+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.342590+0000) 2019-10-13T13:38:28.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:28.341+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.342931+0000) 2019-10-13T13:38:28.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:28.341+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.342943+0000) 2019-10-13T13:38:28.349 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:28.341+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.344065+0000) 2019-10-13T13:38:28.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:28.345+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.348876+0000) 2019-10-13T13:38:28.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:28.345+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:28.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:28.721+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:28.721967+0000) 2019-10-13T13:38:31.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:31.969+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:31.973559+0000) 2019-10-13T13:38:33.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:33.349+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.352491+0000) 2019-10-13T13:38:33.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:33.349+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.353219+0000) 2019-10-13T13:38:33.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:33.353+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.353965+0000) 2019-10-13T13:38:33.356 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:33.353+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.354501+0000) 2019-10-13T13:38:33.357 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:33.353+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.354987+0000) 2019-10-13T13:38:33.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:33.349+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.355961+0000) 2019-10-13T13:38:33.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:33.353+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.356319+0000) 2019-10-13T13:38:33.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:33.357+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:33.360774+0000) 2019-10-13T13:38:33.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:33.361+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187076 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:34.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:34.296+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:34.301679+0000) 2019-10-13T13:38:34.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:34.584+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:34.587175+0000) 2019-10-13T13:38:34.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:34.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:34.954447+0000) 2019-10-13T13:38:37.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:37.952+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:37.957157+0000) 2019-10-13T13:38:38.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:38.017+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.018233+0000) 2019-10-13T13:38:38.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:38.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.182570+0000) 2019-10-13T13:38:38.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:38.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.366752+0000) 2019-10-13T13:38:38.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:38.364+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.367346+0000) 2019-10-13T13:38:38.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:38.364+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.367657+0000) 2019-10-13T13:38:38.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:38.365+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.370074+0000) 2019-10-13T13:38:38.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:38.368+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.370659+0000) 2019-10-13T13:38:38.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:38.368+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.371120+0000) 2019-10-13T13:38:38.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:38.368+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.371506+0000) 2019-10-13T13:38:38.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:38.369+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.372983+0000) 2019-10-13T13:38:38.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:38.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:38.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:38.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:38.722164+0000) 2019-10-13T13:38:41.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:41.973+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:41.973782+0000) 2019-10-13T13:38:43.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:43.376+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.379402+0000) 2019-10-13T13:38:43.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:43.377+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.380298+0000) 2019-10-13T13:38:43.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:43.376+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.380408+0000) 2019-10-13T13:38:43.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:43.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.381454+0000) 2019-10-13T13:38:43.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:43.380+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.381975+0000) 2019-10-13T13:38:43.385 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:43.380+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.382826+0000) 2019-10-13T13:38:43.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:43.380+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.383087+0000) 2019-10-13T13:38:43.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:43.385+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:43.388923+0000) 2019-10-13T13:38:43.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:43.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187136 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:44.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:44.300+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:44.301865+0000) 2019-10-13T13:38:44.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:44.584+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:44.587380+0000) 2019-10-13T13:38:44.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:44.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:44.954651+0000) 2019-10-13T13:38:47.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:47.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:47.957381+0000) 2019-10-13T13:38:48.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:48.024+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.025075+0000) 2019-10-13T13:38:48.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:48.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.182770+0000) 2019-10-13T13:38:48.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:48.388+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.391832+0000) 2019-10-13T13:38:48.394 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:48.388+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.392098+0000) 2019-10-13T13:38:48.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:48.388+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.392827+0000) 2019-10-13T13:38:48.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:48.388+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.393363+0000) 2019-10-13T13:38:48.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:48.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.393864+0000) 2019-10-13T13:38:48.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:48.388+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.394422+0000) 2019-10-13T13:38:48.398 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:48.392+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.395804+0000) 2019-10-13T13:38:48.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:48.396+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.401433+0000) 2019-10-13T13:38:48.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:48.396+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:48.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:48.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:48.722360+0000) 2019-10-13T13:38:51.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:51.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:51.974007+0000) 2019-10-13T13:38:53.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:53.400+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.404766+0000) 2019-10-13T13:38:53.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:53.400+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.404871+0000) 2019-10-13T13:38:53.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:53.400+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.405417+0000) 2019-10-13T13:38:53.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:53.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.406165+0000) 2019-10-13T13:38:53.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:53.404+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.406636+0000) 2019-10-13T13:38:53.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:53.404+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.406950+0000) 2019-10-13T13:38:53.410 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:53.404+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.407277+0000) 2019-10-13T13:38:53.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:53.408+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:53.413101+0000) 2019-10-13T13:38:53.416 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:53.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:54.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:54.300+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:54.302047+0000) 2019-10-13T13:38:54.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:54.584+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:54.587579+0000) 2019-10-13T13:38:54.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:54.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:54.954854+0000) 2019-10-13T13:38:56.018 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:38:56.018 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:56.022 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:38:56.022 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:38:57.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:57.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:57.957578+0000) 2019-10-13T13:38:58.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:58.024+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.031662+0000) 2019-10-13T13:38:58.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:58.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.182967+0000) 2019-10-13T13:38:58.420 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:38:58.412+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.417626+0000) 2019-10-13T13:38:58.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:38:58.416+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.418602+0000) 2019-10-13T13:38:58.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:38:58.416+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.419380+0000) 2019-10-13T13:38:58.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:38:58.416+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.420084+0000) 2019-10-13T13:38:58.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:38:58.416+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.420089+0000) 2019-10-13T13:38:58.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:38:58.416+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.420663+0000) 2019-10-13T13:38:58.424 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:58.420+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.421844+0000) 2019-10-13T13:38:58.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:38:58.420+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.425688+0000) 2019-10-13T13:38:58.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:38:58.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:38:58.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:38:58.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:38:58.722538+0000) 2019-10-13T13:39:01.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:01.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:01.974199+0000) 2019-10-13T13:39:03.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:03.424+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.430539+0000) 2019-10-13T13:39:03.433 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:03.428+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.430632+0000) 2019-10-13T13:39:03.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:03.428+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.433456+0000) 2019-10-13T13:39:03.436 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:03.432+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.433789+0000) 2019-10-13T13:39:03.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:03.432+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.435770+0000) 2019-10-13T13:39:03.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:03.436+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.437976+0000) 2019-10-13T13:39:03.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:03.436+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.439510+0000) 2019-10-13T13:39:03.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:03.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:03.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:03.440+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:03.445487+0000) 2019-10-13T13:39:04.304 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:04.300+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:04.302237+0000) 2019-10-13T13:39:04.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:04.584+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:04.587796+0000) 2019-10-13T13:39:04.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:04.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:04.955007+0000) 2019-10-13T13:39:07.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:07.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:07.957810+0000) 2019-10-13T13:39:08.040 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:08.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.038276+0000) 2019-10-13T13:39:08.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:08.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.183176+0000) 2019-10-13T13:39:08.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:08.440+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.443210+0000) 2019-10-13T13:39:08.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:08.440+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.443677+0000) 2019-10-13T13:39:08.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:08.440+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.445263+0000) 2019-10-13T13:39:08.448 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:08.444+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.446006+0000) 2019-10-13T13:39:08.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:08.444+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.446501+0000) 2019-10-13T13:39:08.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:08.444+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.446724+0000) 2019-10-13T13:39:08.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:08.444+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.447316+0000) 2019-10-13T13:39:08.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:08.444+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.451327+0000) 2019-10-13T13:39:08.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:08.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:08.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:08.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:08.722706+0000) 2019-10-13T13:39:11.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:11.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:11.974391+0000) 2019-10-13T13:39:13.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:13.452+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.456494+0000) 2019-10-13T13:39:13.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:13.452+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.457126+0000) 2019-10-13T13:39:13.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:13.456+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.457929+0000) 2019-10-13T13:39:13.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:13.456+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.458467+0000) 2019-10-13T13:39:13.461 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:13.452+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.458968+0000) 2019-10-13T13:39:13.461 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:13.456+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.459098+0000) 2019-10-13T13:39:13.462 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:13.456+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.459989+0000) 2019-10-13T13:39:13.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:13.460+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:13.464254+0000) 2019-10-13T13:39:13.468 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:13.464+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187320 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:14.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:14.300+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:14.302447+0000) 2019-10-13T13:39:14.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:14.584+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:14.587955+0000) 2019-10-13T13:39:14.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:14.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:14.955235+0000) 2019-10-13T13:39:17.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:17.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:17.957973+0000) 2019-10-13T13:39:18.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:18.044+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.044814+0000) 2019-10-13T13:39:18.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:18.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.183382+0000) 2019-10-13T13:39:18.472 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:18.464+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.469453+0000) 2019-10-13T13:39:18.472 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:18.464+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.470657+0000) 2019-10-13T13:39:18.473 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:18.468+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.470932+0000) 2019-10-13T13:39:18.474 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:18.468+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.471579+0000) 2019-10-13T13:39:18.474 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:18.468+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.471201+0000) 2019-10-13T13:39:18.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:18.468+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.472360+0000) 2019-10-13T13:39:18.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:18.468+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.472719+0000) 2019-10-13T13:39:18.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:18.472+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.476391+0000) 2019-10-13T13:39:18.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:18.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:18.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:18.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:18.722890+0000) 2019-10-13T13:39:21.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:21.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:21.974600+0000) 2019-10-13T13:39:23.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:23.480+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.482005+0000) 2019-10-13T13:39:23.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:23.480+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.482624+0000) 2019-10-13T13:39:23.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:23.476+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.483432+0000) 2019-10-13T13:39:23.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:23.480+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.483724+0000) 2019-10-13T13:39:23.486 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:23.480+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.484240+0000) 2019-10-13T13:39:23.488 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:23.480+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.485543+0000) 2019-10-13T13:39:23.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:23.480+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.485206+0000) 2019-10-13T13:39:23.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:23.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:23.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:23.488+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:23.492651+0000) 2019-10-13T13:39:24.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:24.300+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:24.302660+0000) 2019-10-13T13:39:24.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:24.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:24.588171+0000) 2019-10-13T13:39:24.957 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:24.952+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:24.955439+0000) 2019-10-13T13:39:26.159 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:39:26.160 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:26.163 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:39:26.163 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:27.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:27.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:27.958205+0000) 2019-10-13T13:39:28.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:28.044+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.051313+0000) 2019-10-13T13:39:28.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:28.180+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.183584+0000) 2019-10-13T13:39:28.497 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:28.492+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.495185+0000) 2019-10-13T13:39:28.498 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:28.492+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.496272+0000) 2019-10-13T13:39:28.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:28.492+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.496408+0000) 2019-10-13T13:39:28.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:28.492+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.497076+0000) 2019-10-13T13:39:28.499 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:28.492+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.497517+0000) 2019-10-13T13:39:28.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:28.496+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.497861+0000) 2019-10-13T13:39:28.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:28.496+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.498614+0000) 2019-10-13T13:39:28.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:28.500+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.504892+0000) 2019-10-13T13:39:28.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:28.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187414 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:28.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:28.720+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:28.723105+0000) 2019-10-13T13:39:31.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:31.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:31.974811+0000) 2019-10-13T13:39:33.511 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:33.503+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.508901+0000) 2019-10-13T13:39:33.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:33.507+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.509804+0000) 2019-10-13T13:39:33.512 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:33.507+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.510125+0000) 2019-10-13T13:39:33.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:33.507+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.510769+0000) 2019-10-13T13:39:33.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:33.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.511275+0000) 2019-10-13T13:39:33.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:33.507+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.511408+0000) 2019-10-13T13:39:33.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:33.507+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.511933+0000) 2019-10-13T13:39:33.519 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:33.511+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:33.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:33.512+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:33.518982+0000) 2019-10-13T13:39:34.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:34.299+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:34.302859+0000) 2019-10-13T13:39:34.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:34.587+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:34.588374+0000) 2019-10-13T13:39:34.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:34.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:34.955638+0000) 2019-10-13T13:39:37.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:37.955+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:37.958407+0000) 2019-10-13T13:39:38.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:38.052+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.057817+0000) 2019-10-13T13:39:38.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:38.179+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.183785+0000) 2019-10-13T13:39:38.523 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:38.515+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.519893+0000) 2019-10-13T13:39:38.523 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:38.515+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.521261+0000) 2019-10-13T13:39:38.524 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:38.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.521880+0000) 2019-10-13T13:39:38.524 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:38.519+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.522161+0000) 2019-10-13T13:39:38.525 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:38.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.523235+0000) 2019-10-13T13:39:38.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:38.519+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.523701+0000) 2019-10-13T13:39:38.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:38.519+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.523834+0000) 2019-10-13T13:39:38.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:38.524+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.530577+0000) 2019-10-13T13:39:38.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:38.527+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:38.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:38.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:38.723276+0000) 2019-10-13T13:39:41.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:41.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:41.975011+0000) 2019-10-13T13:39:43.536 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:43.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.533521+0000) 2019-10-13T13:39:43.537 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:43.531+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.535363+0000) 2019-10-13T13:39:43.538 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:43.531+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.536218+0000) 2019-10-13T13:39:43.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:43.536+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.536917+0000) 2019-10-13T13:39:43.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:43.531+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.537314+0000) 2019-10-13T13:39:43.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:43.535+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.540925+0000) 2019-10-13T13:39:43.543 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:43.535+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.541162+0000) 2019-10-13T13:39:43.546 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:43.539+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187504 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:43.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:43.544+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:43.545651+0000) 2019-10-13T13:39:44.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:44.299+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:44.303041+0000) 2019-10-13T13:39:44.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:44.587+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:44.588601+0000) 2019-10-13T13:39:44.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:44.955+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:44.955871+0000) 2019-10-13T13:39:47.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:47.955+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:47.958605+0000) 2019-10-13T13:39:48.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:48.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.064361+0000) 2019-10-13T13:39:48.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:48.179+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.184004+0000) 2019-10-13T13:39:48.549 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:48.543+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.547294+0000) 2019-10-13T13:39:48.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:48.543+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.547658+0000) 2019-10-13T13:39:48.551 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:48.543+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.549532+0000) 2019-10-13T13:39:48.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:48.547+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.549798+0000) 2019-10-13T13:39:48.554 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:48.551+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.552675+0000) 2019-10-13T13:39:48.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:48.551+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.554174+0000) 2019-10-13T13:39:48.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:48.551+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.556088+0000) 2019-10-13T13:39:48.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:48.555+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.557110+0000) 2019-10-13T13:39:48.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:48.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:48.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:48.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:48.723451+0000) 2019-10-13T13:39:51.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:51.971+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:51.975202+0000) 2019-10-13T13:39:53.562 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:53.555+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.560314+0000) 2019-10-13T13:39:53.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:53.555+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.561637+0000) 2019-10-13T13:39:53.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:53.559+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.562272+0000) 2019-10-13T13:39:53.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:53.559+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.562755+0000) 2019-10-13T13:39:53.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:53.559+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.563301+0000) 2019-10-13T13:39:53.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:53.559+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.563692+0000) 2019-10-13T13:39:53.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:53.559+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.564138+0000) 2019-10-13T13:39:53.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:53.567+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:53.569195+0000) 2019-10-13T13:39:53.572 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:53.567+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187567 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:54.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:54.299+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:54.303240+0000) 2019-10-13T13:39:54.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:54.583+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:54.588808+0000) 2019-10-13T13:39:54.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:54.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:54.956072+0000) 2019-10-13T13:39:56.262 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:39:56.262 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:56.266 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:39:56.266 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:39:57.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:57.955+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:57.958752+0000) 2019-10-13T13:39:58.073 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:58.067+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.070758+0000) 2019-10-13T13:39:58.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:58.179+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.184237+0000) 2019-10-13T13:39:58.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:39:58.571+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.575939+0000) 2019-10-13T13:39:58.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:39:58.571+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.576127+0000) 2019-10-13T13:39:58.580 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:39:58.575+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.577922+0000) 2019-10-13T13:39:58.580 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:39:58.575+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.578030+0000) 2019-10-13T13:39:58.581 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:39:58.575+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.578397+0000) 2019-10-13T13:39:58.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:58.575+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.578877+0000) 2019-10-13T13:39:58.581 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:39:58.575+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.579299+0000) 2019-10-13T13:39:58.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:39:58.579+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:39:58.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:39:58.583+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.586066+0000) 2019-10-13T13:39:58.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:39:58.719+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:39:58.723630+0000) 2019-10-13T13:40:01.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:01.971+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:01.975425+0000) 2019-10-13T13:40:03.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:03.583+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.586370+0000) 2019-10-13T13:40:03.589 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:03.583+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.586554+0000) 2019-10-13T13:40:03.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:03.583+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.587145+0000) 2019-10-13T13:40:03.590 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:03.583+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.587278+0000) 2019-10-13T13:40:03.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:03.583+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.588258+0000) 2019-10-13T13:40:03.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:03.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.588764+0000) 2019-10-13T13:40:03.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:03.587+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.590170+0000) 2019-10-13T13:40:03.598 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:03.595+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:03.596518+0000) 2019-10-13T13:40:03.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:03.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187628 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:04.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:04.299+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:04.303423+0000) 2019-10-13T13:40:04.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:04.583+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:04.589038+0000) 2019-10-13T13:40:04.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:04.955+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:04.956271+0000) 2019-10-13T13:40:07.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:07.955+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:07.958974+0000) 2019-10-13T13:40:08.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:08.079+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.081342+0000) 2019-10-13T13:40:08.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:08.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.184455+0000) 2019-10-13T13:40:08.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:08.595+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.600826+0000) 2019-10-13T13:40:08.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:08.599+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.601910+0000) 2019-10-13T13:40:08.604 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:08.595+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.601432+0000) 2019-10-13T13:40:08.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:08.599+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.602155+0000) 2019-10-13T13:40:08.605 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:08.599+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.602445+0000) 2019-10-13T13:40:08.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:08.599+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.602809+0000) 2019-10-13T13:40:08.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:08.599+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.603781+0000) 2019-10-13T13:40:08.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:08.607+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.609878+0000) 2019-10-13T13:40:08.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:08.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:08.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:08.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:08.723802+0000) 2019-10-13T13:40:11.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:11.971+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:11.975624+0000) 2019-10-13T13:40:13.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:13.611+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.614989+0000) 2019-10-13T13:40:13.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:13.611+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.615474+0000) 2019-10-13T13:40:13.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:13.611+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.615775+0000) 2019-10-13T13:40:13.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:13.611+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.617182+0000) 2019-10-13T13:40:13.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:13.611+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.617312+0000) 2019-10-13T13:40:13.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:13.611+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.617691+0000) 2019-10-13T13:40:13.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:13.615+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.618183+0000) 2019-10-13T13:40:13.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:13.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187691 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:13.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:13.623+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:13.625222+0000) 2019-10-13T13:40:14.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:14.299+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:14.303564+0000) 2019-10-13T13:40:14.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:14.587+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:14.589274+0000) 2019-10-13T13:40:14.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:14.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:14.956486+0000) 2019-10-13T13:40:17.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:17.955+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:17.959178+0000) 2019-10-13T13:40:18.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:18.083+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.087980+0000) 2019-10-13T13:40:18.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:18.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.184654+0000) 2019-10-13T13:40:18.630 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:18.623+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.627442+0000) 2019-10-13T13:40:18.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:18.623+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.627741+0000) 2019-10-13T13:40:18.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:18.627+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.628416+0000) 2019-10-13T13:40:18.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:18.623+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.628908+0000) 2019-10-13T13:40:18.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:18.623+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.629105+0000) 2019-10-13T13:40:18.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:18.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.630754+0000) 2019-10-13T13:40:18.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:18.627+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.631014+0000) 2019-10-13T13:40:18.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:18.631+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.633713+0000) 2019-10-13T13:40:18.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:18.631+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:18.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:18.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:18.724014+0000) 2019-10-13T13:40:21.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:21.975+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:21.975828+0000) 2019-10-13T13:40:23.642 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:23.635+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.639651+0000) 2019-10-13T13:40:23.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:23.635+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.640171+0000) 2019-10-13T13:40:23.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:23.639+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.641449+0000) 2019-10-13T13:40:23.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:23.635+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.641425+0000) 2019-10-13T13:40:23.644 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:23.639+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.641808+0000) 2019-10-13T13:40:23.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:23.639+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.642255+0000) 2019-10-13T13:40:23.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:23.639+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.643002+0000) 2019-10-13T13:40:23.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:23.647+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:23.649939+0000) 2019-10-13T13:40:23.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:23.643+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:24.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:24.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:24.303801+0000) 2019-10-13T13:40:24.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:24.587+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:24.589463+0000) 2019-10-13T13:40:24.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:24.951+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:24.956703+0000) 2019-10-13T13:40:26.398 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:40:26.398 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:26.402 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:40:26.402 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:27.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:27.954+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:27.959369+0000) 2019-10-13T13:40:28.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:28.091+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.094475+0000) 2019-10-13T13:40:28.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:28.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.184883+0000) 2019-10-13T13:40:28.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:28.646+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.653062+0000) 2019-10-13T13:40:28.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:28.646+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.653393+0000) 2019-10-13T13:40:28.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:28.650+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.655394+0000) 2019-10-13T13:40:28.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:28.650+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.655425+0000) 2019-10-13T13:40:28.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:28.655+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.656477+0000) 2019-10-13T13:40:28.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:28.650+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.656618+0000) 2019-10-13T13:40:28.659 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:28.650+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.657097+0000) 2019-10-13T13:40:28.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:28.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:28.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:28.663+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.664138+0000) 2019-10-13T13:40:28.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:28.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:28.724224+0000) 2019-10-13T13:40:31.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:31.975+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:31.976114+0000) 2019-10-13T13:40:33.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:33.658+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.665712+0000) 2019-10-13T13:40:33.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:33.662+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.666275+0000) 2019-10-13T13:40:33.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:33.662+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.666784+0000) 2019-10-13T13:40:33.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:33.662+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.668023+0000) 2019-10-13T13:40:33.671 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:33.667+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.668559+0000) 2019-10-13T13:40:33.672 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:33.666+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.670271+0000) 2019-10-13T13:40:33.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:33.666+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.671834+0000) 2019-10-13T13:40:33.677 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:33.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:33.678 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:33.671+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:33.675435+0000) 2019-10-13T13:40:34.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:34.298+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:34.304046+0000) 2019-10-13T13:40:34.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:34.582+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:34.589636+0000) 2019-10-13T13:40:34.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:34.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:34.956903+0000) 2019-10-13T13:40:37.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:37.958+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:37.959563+0000) 2019-10-13T13:40:38.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:38.099+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.101008+0000) 2019-10-13T13:40:38.187 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:38.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.185094+0000) 2019-10-13T13:40:38.681 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:38.674+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.678252+0000) 2019-10-13T13:40:38.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:38.674+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.679006+0000) 2019-10-13T13:40:38.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:38.674+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.678994+0000) 2019-10-13T13:40:38.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:38.674+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.680219+0000) 2019-10-13T13:40:38.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:38.679+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.680843+0000) 2019-10-13T13:40:38.683 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:38.674+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.681041+0000) 2019-10-13T13:40:38.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:38.678+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.681728+0000) 2019-10-13T13:40:38.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:38.683+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.686310+0000) 2019-10-13T13:40:38.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:38.682+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:38.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:38.718+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:38.724408+0000) 2019-10-13T13:40:41.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:41.975+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:41.976379+0000) 2019-10-13T13:40:43.694 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:43.686+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.691792+0000) 2019-10-13T13:40:43.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:43.690+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.693970+0000) 2019-10-13T13:40:43.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:43.690+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.694219+0000) 2019-10-13T13:40:43.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:43.695+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.696617+0000) 2019-10-13T13:40:43.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:43.695+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.697907+0000) 2019-10-13T13:40:43.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:43.690+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.696672+0000) 2019-10-13T13:40:43.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:43.694+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.699203+0000) 2019-10-13T13:40:43.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:43.694+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:43.699430+0000) 2019-10-13T13:40:43.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:43.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187873 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:44.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:44.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:44.304262+0000) 2019-10-13T13:40:44.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:44.586+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:44.589848+0000) 2019-10-13T13:40:44.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:44.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:44.957106+0000) 2019-10-13T13:40:47.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:47.958+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:47.959797+0000) 2019-10-13T13:40:48.110 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:48.107+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.107524+0000) 2019-10-13T13:40:48.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:48.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.185295+0000) 2019-10-13T13:40:48.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:48.698+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.703967+0000) 2019-10-13T13:40:48.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:48.702+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.705734+0000) 2019-10-13T13:40:48.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:48.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.705998+0000) 2019-10-13T13:40:48.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:48.702+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.706641+0000) 2019-10-13T13:40:48.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:48.702+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.706481+0000) 2019-10-13T13:40:48.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:48.703+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.707138+0000) 2019-10-13T13:40:48.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:48.702+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.707548+0000) 2019-10-13T13:40:48.713 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:48.707+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.711520+0000) 2019-10-13T13:40:48.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:48.706+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:48.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:48.718+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:48.724612+0000) 2019-10-13T13:40:51.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:51.975+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:51.976601+0000) 2019-10-13T13:40:53.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:53.710+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.716860+0000) 2019-10-13T13:40:53.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:53.710+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.717349+0000) 2019-10-13T13:40:53.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:53.714+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.719072+0000) 2019-10-13T13:40:53.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:53.714+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.721409+0000) 2019-10-13T13:40:53.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:53.718+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.722591+0000) 2019-10-13T13:40:53.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:53.718+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.722968+0000) 2019-10-13T13:40:53.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:53.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.723003+0000) 2019-10-13T13:40:53.728 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:53.718+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187935 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:40:53.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:53.726+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:53.728628+0000) 2019-10-13T13:40:54.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:54.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:54.304473+0000) 2019-10-13T13:40:54.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:54.586+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:54.590069+0000) 2019-10-13T13:40:54.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:54.950+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:54.957332+0000) 2019-10-13T13:40:56.498 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:40:56.498 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:56.502 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:40:56.502 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:40:57.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:57.954+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:57.959994+0000) 2019-10-13T13:40:58.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:58.110+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.114269+0000) 2019-10-13T13:40:58.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:58.178+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.185495+0000) 2019-10-13T13:40:58.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:58.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.724734+0000) 2019-10-13T13:40:58.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:40:58.722+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.728247+0000) 2019-10-13T13:40:58.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:40:58.722+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.728884+0000) 2019-10-13T13:40:58.731 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:40:58.726+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.729671+0000) 2019-10-13T13:40:58.732 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:40:58.726+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.730059+0000) 2019-10-13T13:40:58.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:40:58.726+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.730874+0000) 2019-10-13T13:40:58.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:40:58.726+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.731644+0000) 2019-10-13T13:40:58.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:40:58.726+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.732008+0000) 2019-10-13T13:40:58.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:40:58.734+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:40:58.736960+0000) 2019-10-13T13:40:58.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:40:58.730+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:01.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:01.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:01.976808+0000) 2019-10-13T13:41:03.743 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:03.734+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.740776+0000) 2019-10-13T13:41:03.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:03.734+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.741580+0000) 2019-10-13T13:41:03.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:03.738+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.742007+0000) 2019-10-13T13:41:03.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:03.738+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.743994+0000) 2019-10-13T13:41:03.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:03.738+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.743985+0000) 2019-10-13T13:41:03.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:03.742+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.746635+0000) 2019-10-13T13:41:03.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:03.742+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.748644+0000) 2019-10-13T13:41:03.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:03.746+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:03.749846+0000) 2019-10-13T13:41:03.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:03.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:04.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:04.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:04.304657+0000) 2019-10-13T13:41:04.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:04.586+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:04.590247+0000) 2019-10-13T13:41:04.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:04.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:04.957534+0000) 2019-10-13T13:41:07.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:07.958+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:07.960194+0000) 2019-10-13T13:41:08.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:08.122+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.125003+0000) 2019-10-13T13:41:08.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:08.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.185697+0000) 2019-10-13T13:41:08.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:08.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.724939+0000) 2019-10-13T13:41:08.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:08.750+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.754452+0000) 2019-10-13T13:41:08.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:08.750+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.755316+0000) 2019-10-13T13:41:08.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:08.750+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.755310+0000) 2019-10-13T13:41:08.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:08.750+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.755820+0000) 2019-10-13T13:41:08.760 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:08.754+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.757905+0000) 2019-10-13T13:41:08.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:08.754+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.757906+0000) 2019-10-13T13:41:08.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:08.754+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.758676+0000) 2019-10-13T13:41:08.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:08.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:08.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:08.762+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:08.766056+0000) 2019-10-13T13:41:11.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:11.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:11.977034+0000) 2019-10-13T13:41:13.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:13.762+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.766985+0000) 2019-10-13T13:41:13.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:13.762+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.768034+0000) 2019-10-13T13:41:13.771 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:13.762+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.767767+0000) 2019-10-13T13:41:13.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:13.762+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.768133+0000) 2019-10-13T13:41:13.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:13.766+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.769379+0000) 2019-10-13T13:41:13.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:13.762+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.769652+0000) 2019-10-13T13:41:13.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:13.766+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.769944+0000) 2019-10-13T13:41:13.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:13.774+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:13.776409+0000) 2019-10-13T13:41:13.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:13.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188057 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:14.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:14.298+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:14.304842+0000) 2019-10-13T13:41:14.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:14.586+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:14.590420+0000) 2019-10-13T13:41:14.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:14.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:14.957777+0000) 2019-10-13T13:41:17.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:17.954+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:17.960389+0000) 2019-10-13T13:41:18.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:18.130+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.131557+0000) 2019-10-13T13:41:18.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:18.182+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.185953+0000) 2019-10-13T13:41:18.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:18.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.725129+0000) 2019-10-13T13:41:18.782 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:18.774+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.779827+0000) 2019-10-13T13:41:18.782 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:18.774+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.780487+0000) 2019-10-13T13:41:18.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:18.778+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.783140+0000) 2019-10-13T13:41:18.786 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:18.778+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.783575+0000) 2019-10-13T13:41:18.786 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:18.778+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.784194+0000) 2019-10-13T13:41:18.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:18.782+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.786239+0000) 2019-10-13T13:41:18.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:18.786+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.788313+0000) 2019-10-13T13:41:18.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:18.782+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:18.795 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:18.790+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:18.792901+0000) 2019-10-13T13:41:21.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:21.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:21.977256+0000) 2019-10-13T13:41:23.795 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:23.785+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.792718+0000) 2019-10-13T13:41:23.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:23.785+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.793547+0000) 2019-10-13T13:41:23.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:23.790+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.794148+0000) 2019-10-13T13:41:23.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:23.789+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.794203+0000) 2019-10-13T13:41:23.798 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:23.789+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.795785+0000) 2019-10-13T13:41:23.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:23.789+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.797421+0000) 2019-10-13T13:41:23.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:23.793+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.797786+0000) 2019-10-13T13:41:23.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:23.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:23.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:23.798+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:23.802663+0000) 2019-10-13T13:41:24.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:24.301+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:24.305064+0000) 2019-10-13T13:41:24.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:24.585+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:24.590633+0000) 2019-10-13T13:41:24.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:24.953+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:24.957974+0000) 2019-10-13T13:41:26.636 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:41:26.636 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:26.639 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:41:26.639 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:27.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:27.957+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:27.960626+0000) 2019-10-13T13:41:28.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:28.134+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.138112+0000) 2019-10-13T13:41:28.188 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:28.181+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.186184+0000) 2019-10-13T13:41:28.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:28.721+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.725359+0000) 2019-10-13T13:41:28.807 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:28.797+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.805081+0000) 2019-10-13T13:41:28.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:28.802+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.805730+0000) 2019-10-13T13:41:28.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:28.801+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.805994+0000) 2019-10-13T13:41:28.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:28.801+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.806574+0000) 2019-10-13T13:41:28.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:28.801+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.807186+0000) 2019-10-13T13:41:28.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:28.801+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.807624+0000) 2019-10-13T13:41:28.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:28.801+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.808625+0000) 2019-10-13T13:41:28.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:28.810+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:28.813893+0000) 2019-10-13T13:41:28.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:28.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188149 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:31.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:31.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:31.977453+0000) 2019-10-13T13:41:33.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:33.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.818390+0000) 2019-10-13T13:41:33.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:33.813+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.818334+0000) 2019-10-13T13:41:33.822 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:33.813+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.818993+0000) 2019-10-13T13:41:33.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:33.813+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.819499+0000) 2019-10-13T13:41:33.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:33.818+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.820570+0000) 2019-10-13T13:41:33.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:33.813+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.820621+0000) 2019-10-13T13:41:33.823 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:33.813+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.821136+0000) 2019-10-13T13:41:33.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:33.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:33.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:33.826+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:33.828715+0000) 2019-10-13T13:41:34.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:34.301+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:34.305259+0000) 2019-10-13T13:41:34.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:34.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:34.590829+0000) 2019-10-13T13:41:34.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:34.957+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:34.958197+0000) 2019-10-13T13:41:37.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:37.957+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:37.960864+0000) 2019-10-13T13:41:38.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:38.142+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.144805+0000) 2019-10-13T13:41:38.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:38.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.186336+0000) 2019-10-13T13:41:38.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:38.721+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.725553+0000) 2019-10-13T13:41:38.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:38.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.829882+0000) 2019-10-13T13:41:38.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:38.829+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.830364+0000) 2019-10-13T13:41:38.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:38.829+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.831113+0000) 2019-10-13T13:41:38.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:38.829+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.831508+0000) 2019-10-13T13:41:38.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:38.830+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.832712+0000) 2019-10-13T13:41:38.835 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:38.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.832825+0000) 2019-10-13T13:41:38.835 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:38.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.832755+0000) 2019-10-13T13:41:38.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:38.834+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:38.839794+0000) 2019-10-13T13:41:38.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:38.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188211 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:41.980 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:41.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:41.977665+0000) 2019-10-13T13:41:43.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:43.841+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.843581+0000) 2019-10-13T13:41:43.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:43.841+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.844918+0000) 2019-10-13T13:41:43.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:43.841+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.845477+0000) 2019-10-13T13:41:43.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:43.845+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.847318+0000) 2019-10-13T13:41:43.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:43.845+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.849314+0000) 2019-10-13T13:41:43.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:43.846+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.850510+0000) 2019-10-13T13:41:43.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:43.849+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.850870+0000) 2019-10-13T13:41:43.854 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:43.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:43.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:43.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:43.852961+0000) 2019-10-13T13:41:44.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:44.301+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:44.305449+0000) 2019-10-13T13:41:44.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:44.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:44.591006+0000) 2019-10-13T13:41:44.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:44.957+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:44.958417+0000) 2019-10-13T13:41:47.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:47.957+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:47.961086+0000) 2019-10-13T13:41:48.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:48.154+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.155294+0000) 2019-10-13T13:41:48.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:48.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.186566+0000) 2019-10-13T13:41:48.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:48.725+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.725770+0000) 2019-10-13T13:41:48.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:48.853+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.856173+0000) 2019-10-13T13:41:48.859 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:48.853+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.856578+0000) 2019-10-13T13:41:48.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:48.853+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.857029+0000) 2019-10-13T13:41:48.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:48.853+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.857434+0000) 2019-10-13T13:41:48.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:48.857+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.857815+0000) 2019-10-13T13:41:48.861 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:48.857+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.858655+0000) 2019-10-13T13:41:48.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:48.854+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.859149+0000) 2019-10-13T13:41:48.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:48.861+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188271 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:48.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:48.862+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:48.865143+0000) 2019-10-13T13:41:51.980 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:51.974+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:51.977875+0000) 2019-10-13T13:41:53.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:53.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.868032+0000) 2019-10-13T13:41:53.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:53.869+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.869915+0000) 2019-10-13T13:41:53.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:53.869+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.870550+0000) 2019-10-13T13:41:53.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:53.869+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.871050+0000) 2019-10-13T13:41:53.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:53.870+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.872151+0000) 2019-10-13T13:41:53.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:53.869+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.872260+0000) 2019-10-13T13:41:53.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:53.869+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.872624+0000) 2019-10-13T13:41:53.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:53.874+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:53.877832+0000) 2019-10-13T13:41:53.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:53.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188300 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:41:54.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:54.305+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:54.305652+0000) 2019-10-13T13:41:54.593 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:54.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:54.591180+0000) 2019-10-13T13:41:54.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:54.957+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:54.958590+0000) 2019-10-13T13:41:56.751 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:41:56.751 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:56.755 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:41:56.755 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:41:57.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:57.957+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:57.961281+0000) 2019-10-13T13:41:58.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:58.157+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.161887+0000) 2019-10-13T13:41:58.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:58.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.186762+0000) 2019-10-13T13:41:58.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:58.725+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.725965+0000) 2019-10-13T13:41:58.883 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:41:58.877+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.880596+0000) 2019-10-13T13:41:58.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:41:58.877+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.881075+0000) 2019-10-13T13:41:58.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:41:58.881+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.882041+0000) 2019-10-13T13:41:58.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:41:58.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.884075+0000) 2019-10-13T13:41:58.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:41:58.881+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.884736+0000) 2019-10-13T13:41:58.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:41:58.881+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.885945+0000) 2019-10-13T13:41:58.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:41:58.885+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.886921+0000) 2019-10-13T13:41:58.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:41:58.885+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:41:58.889955+0000) 2019-10-13T13:41:58.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:41:58.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188333 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:01.980 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:01.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:01.978082+0000) 2019-10-13T13:42:03.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:03.893+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.893992+0000) 2019-10-13T13:42:03.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:03.893+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.894018+0000) 2019-10-13T13:42:03.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:03.893+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.894888+0000) 2019-10-13T13:42:03.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:03.893+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.895422+0000) 2019-10-13T13:42:03.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:03.893+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.896071+0000) 2019-10-13T13:42:03.899 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:03.893+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.896694+0000) 2019-10-13T13:42:03.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:03.893+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.896810+0000) 2019-10-13T13:42:03.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:03.897+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:03.900942+0000) 2019-10-13T13:42:03.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:03.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188360 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:04.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:04.305+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:04.305865+0000) 2019-10-13T13:42:04.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:04.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:04.591358+0000) 2019-10-13T13:42:04.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:04.957+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:04.958790+0000) 2019-10-13T13:42:07.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:07.961+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:07.961498+0000) 2019-10-13T13:42:08.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:08.169+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.172674+0000) 2019-10-13T13:42:08.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:08.185+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.186992+0000) 2019-10-13T13:42:08.728 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:08.725+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.726165+0000) 2019-10-13T13:42:08.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:08.905+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.906135+0000) 2019-10-13T13:42:08.909 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:08.905+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.906428+0000) 2019-10-13T13:42:08.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:08.905+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.907117+0000) 2019-10-13T13:42:08.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:08.909+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.911055+0000) 2019-10-13T13:42:08.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:08.905+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.911448+0000) 2019-10-13T13:42:08.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:08.909+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.912119+0000) 2019-10-13T13:42:08.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:08.909+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.912277+0000) 2019-10-13T13:42:08.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:08.909+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:08.916038+0000) 2019-10-13T13:42:08.918 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:08.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:11.980 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:11.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:11.978279+0000) 2019-10-13T13:42:13.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:13.917+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.920296+0000) 2019-10-13T13:42:13.923 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:13.917+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.920513+0000) 2019-10-13T13:42:13.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:13.917+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.921374+0000) 2019-10-13T13:42:13.924 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:13.921+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.921874+0000) 2019-10-13T13:42:13.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:13.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.922505+0000) 2019-10-13T13:42:13.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:13.921+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.923333+0000) 2019-10-13T13:42:13.926 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:13.921+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.923632+0000) 2019-10-13T13:42:13.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:13.925+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:13.929607+0000) 2019-10-13T13:42:13.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:13.925+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:14.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:14.305+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:14.306050+0000) 2019-10-13T13:42:14.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:14.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:14.591537+0000) 2019-10-13T13:42:14.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:14.957+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:14.958986+0000) 2019-10-13T13:42:17.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:17.956+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:17.961694+0000) 2019-10-13T13:42:18.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:18.173+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.179103+0000) 2019-10-13T13:42:18.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:18.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.187160+0000) 2019-10-13T13:42:18.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:18.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.726364+0000) 2019-10-13T13:42:18.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:18.928+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.933651+0000) 2019-10-13T13:42:18.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:18.929+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.934211+0000) 2019-10-13T13:42:18.937 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:18.932+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.935053+0000) 2019-10-13T13:42:18.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:18.932+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.935798+0000) 2019-10-13T13:42:18.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:18.932+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.936389+0000) 2019-10-13T13:42:18.939 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:18.932+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.936513+0000) 2019-10-13T13:42:18.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:18.932+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.937209+0000) 2019-10-13T13:42:18.941 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:18.933+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:18.938964+0000) 2019-10-13T13:42:18.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:18.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188454 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:21.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:21.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:21.978509+0000) 2019-10-13T13:42:23.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:23.940+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.945337+0000) 2019-10-13T13:42:23.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:23.944+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.945942+0000) 2019-10-13T13:42:23.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:23.941+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.946465+0000) 2019-10-13T13:42:23.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:23.944+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.949198+0000) 2019-10-13T13:42:23.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:23.944+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.949350+0000) 2019-10-13T13:42:23.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:23.948+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.949859+0000) 2019-10-13T13:42:23.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:23.948+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.950420+0000) 2019-10-13T13:42:23.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:23.949+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:23.953655+0000) 2019-10-13T13:42:23.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:23.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:24.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:24.304+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:24.306241+0000) 2019-10-13T13:42:24.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:24.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:24.591711+0000) 2019-10-13T13:42:24.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:24.956+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:24.959183+0000) 2019-10-13T13:42:26.874 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:42:26.874 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:26.878 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:42:26.878 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:27.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:27.960+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:27.961950+0000) 2019-10-13T13:42:28.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:28.185+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.185579+0000) 2019-10-13T13:42:28.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:28.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.187366+0000) 2019-10-13T13:42:28.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:28.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.726583+0000) 2019-10-13T13:42:28.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:28.956+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.958779+0000) 2019-10-13T13:42:28.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:28.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.960916+0000) 2019-10-13T13:42:28.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:28.956+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.961131+0000) 2019-10-13T13:42:28.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:28.960+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.963354+0000) 2019-10-13T13:42:28.966 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:28.960+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.963608+0000) 2019-10-13T13:42:28.967 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:28.960+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.964375+0000) 2019-10-13T13:42:28.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:28.960+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.964736+0000) 2019-10-13T13:42:28.971 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:28.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188513 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:28.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:28.965+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:28.968962+0000) 2019-10-13T13:42:31.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:31.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:31.978699+0000) 2019-10-13T13:42:33.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:33.968+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.972102+0000) 2019-10-13T13:42:33.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:33.968+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.972927+0000) 2019-10-13T13:42:33.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:33.968+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.973351+0000) 2019-10-13T13:42:33.976 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:33.969+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.974228+0000) 2019-10-13T13:42:33.977 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:33.972+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.974469+0000) 2019-10-13T13:42:33.977 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:33.972+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.974864+0000) 2019-10-13T13:42:33.977 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:33.972+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.975166+0000) 2019-10-13T13:42:33.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:33.973+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:33.978851+0000) 2019-10-13T13:42:33.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:33.976+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188542 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:34.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:34.304+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:34.306430+0000) 2019-10-13T13:42:34.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:34.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:34.591890+0000) 2019-10-13T13:42:34.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:34.956+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:34.959402+0000) 2019-10-13T13:42:37.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:37.960+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:37.962102+0000) 2019-10-13T13:42:38.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:38.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.187625+0000) 2019-10-13T13:42:38.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:38.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.196634+0000) 2019-10-13T13:42:38.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:38.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.726780+0000) 2019-10-13T13:42:38.986 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:38.980+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.983776+0000) 2019-10-13T13:42:38.986 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:38.980+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.984031+0000) 2019-10-13T13:42:38.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:38.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.984580+0000) 2019-10-13T13:42:38.987 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:38.980+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.984669+0000) 2019-10-13T13:42:38.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:38.980+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.985634+0000) 2019-10-13T13:42:38.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:38.984+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.987840+0000) 2019-10-13T13:42:38.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:38.985+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.988373+0000) 2019-10-13T13:42:38.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:38.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:38.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:38.989+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:38.994833+0000) 2019-10-13T13:42:41.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:41.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:41.978902+0000) 2019-10-13T13:42:43.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:43.992+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:43.996775+0000) 2019-10-13T13:42:44.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:43.996+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:43.997837+0000) 2019-10-13T13:42:44.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:43.996+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:43.999502+0000) 2019-10-13T13:42:44.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:43.996+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:43.999945+0000) 2019-10-13T13:42:44.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:43.997+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.000465+0000) 2019-10-13T13:42:44.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:43.996+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.001703+0000) 2019-10-13T13:42:44.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:44.000+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.003528+0000) 2019-10-13T13:42:44.009 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:44.004+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:44.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:44.005+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.008414+0000) 2019-10-13T13:42:44.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:44.304+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.306647+0000) 2019-10-13T13:42:44.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:44.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.592069+0000) 2019-10-13T13:42:44.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:44.956+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:44.959612+0000) 2019-10-13T13:42:47.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:47.960+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:47.962379+0000) 2019-10-13T13:42:48.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:48.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:48.187824+0000) 2019-10-13T13:42:48.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:48.197+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:48.203242+0000) 2019-10-13T13:42:48.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:48.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:48.727000+0000) 2019-10-13T13:42:49.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:49.008+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.010057+0000) 2019-10-13T13:42:49.013 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:49.008+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.010771+0000) 2019-10-13T13:42:49.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:49.005+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.011620+0000) 2019-10-13T13:42:49.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:49.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.012050+0000) 2019-10-13T13:42:49.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:49.008+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.012185+0000) 2019-10-13T13:42:49.016 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:49.012+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.013771+0000) 2019-10-13T13:42:49.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:49.012+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.015534+0000) 2019-10-13T13:42:49.022 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:49.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:49.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:49.021+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:49.025664+0000) 2019-10-13T13:42:51.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:51.977+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:51.979125+0000) 2019-10-13T13:42:54.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:54.020+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.022684+0000) 2019-10-13T13:42:54.026 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:54.020+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.023715+0000) 2019-10-13T13:42:54.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:54.020+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.023787+0000) 2019-10-13T13:42:54.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:54.020+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.024641+0000) 2019-10-13T13:42:54.028 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:54.024+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.025724+0000) 2019-10-13T13:42:54.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:54.021+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.026486+0000) 2019-10-13T13:42:54.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:54.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.027567+0000) 2019-10-13T13:42:54.033 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:54.025+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.031005+0000) 2019-10-13T13:42:54.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:54.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188661 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:42:54.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:54.304+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.306838+0000) 2019-10-13T13:42:54.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:54.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.592224+0000) 2019-10-13T13:42:54.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:54.956+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:54.959824+0000) 2019-10-13T13:42:57.131 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:42:57.131 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:57.135 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:42:57.135 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:42:57.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:57.960+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:57.962578+0000) 2019-10-13T13:42:58.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:58.184+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:58.188044+0000) 2019-10-13T13:42:58.212 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:58.209+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:58.209859+0000) 2019-10-13T13:42:58.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:58.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:58.727194+0000) 2019-10-13T13:42:59.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:42:59.032+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.036003+0000) 2019-10-13T13:42:59.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:42:59.033+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.036963+0000) 2019-10-13T13:42:59.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:42:59.032+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.037338+0000) 2019-10-13T13:42:59.041 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:42:59.036+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.038773+0000) 2019-10-13T13:42:59.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:42:59.036+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.039381+0000) 2019-10-13T13:42:59.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:42:59.036+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.039582+0000) 2019-10-13T13:42:59.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:42:59.036+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.040363+0000) 2019-10-13T13:42:59.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:42:59.037+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:42:59.043363+0000) 2019-10-13T13:42:59.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:42:59.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:01.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:01.972+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:01.979317+0000) 2019-10-13T13:43:04.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:04.044+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.047938+0000) 2019-10-13T13:43:04.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:04.044+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.048478+0000) 2019-10-13T13:43:04.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:04.044+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.048700+0000) 2019-10-13T13:43:04.052 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:04.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.049800+0000) 2019-10-13T13:43:04.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:04.048+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.049821+0000) 2019-10-13T13:43:04.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:04.048+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.050423+0000) 2019-10-13T13:43:04.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:04.048+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.051162+0000) 2019-10-13T13:43:04.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:04.052+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:04.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:04.052+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.059517+0000) 2019-10-13T13:43:04.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:04.304+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.307034+0000) 2019-10-13T13:43:04.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:04.588+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.592421+0000) 2019-10-13T13:43:04.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:04.956+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:04.960007+0000) 2019-10-13T13:43:07.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:07.960+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:07.962788+0000) 2019-10-13T13:43:08.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:08.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:08.188244+0000) 2019-10-13T13:43:08.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:08.212+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:08.216399+0000) 2019-10-13T13:43:08.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:08.724+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:08.727394+0000) 2019-10-13T13:43:09.063 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:09.056+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.060258+0000) 2019-10-13T13:43:09.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:09.056+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.061037+0000) 2019-10-13T13:43:09.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:09.060+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.062417+0000) 2019-10-13T13:43:09.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:09.060+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.063149+0000) 2019-10-13T13:43:09.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:09.060+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.064168+0000) 2019-10-13T13:43:09.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:09.060+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.064421+0000) 2019-10-13T13:43:09.067 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:09.060+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.064686+0000) 2019-10-13T13:43:09.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:09.068+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:09.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:09.068+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:09.073543+0000) 2019-10-13T13:43:11.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:11.976+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:11.979513+0000) 2019-10-13T13:43:14.076 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:14.068+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.073625+0000) 2019-10-13T13:43:14.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:14.071+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.073818+0000) 2019-10-13T13:43:14.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:14.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.074507+0000) 2019-10-13T13:43:14.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:14.071+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.074721+0000) 2019-10-13T13:43:14.077 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:14.071+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.075068+0000) 2019-10-13T13:43:14.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:14.071+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.075901+0000) 2019-10-13T13:43:14.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:14.071+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.076646+0000) 2019-10-13T13:43:14.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:14.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:14.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:14.080+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.085418+0000) 2019-10-13T13:43:14.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:14.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.307225+0000) 2019-10-13T13:43:14.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:14.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.592631+0000) 2019-10-13T13:43:14.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:14.959+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:14.960201+0000) 2019-10-13T13:43:17.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:17.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:17.962970+0000) 2019-10-13T13:43:18.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:18.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:18.188446+0000) 2019-10-13T13:43:18.225 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:18.220+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:18.222934+0000) 2019-10-13T13:43:18.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:18.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:18.727547+0000) 2019-10-13T13:43:19.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:19.083+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.085902+0000) 2019-10-13T13:43:19.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:19.083+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.088051+0000) 2019-10-13T13:43:19.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:19.084+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.088628+0000) 2019-10-13T13:43:19.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:19.083+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.089094+0000) 2019-10-13T13:43:19.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:19.083+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.089255+0000) 2019-10-13T13:43:19.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:19.087+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.089823+0000) 2019-10-13T13:43:19.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:19.083+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.089700+0000) 2019-10-13T13:43:19.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:19.091+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:19.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:19.092+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:19.096645+0000) 2019-10-13T13:43:21.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:21.976+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:21.979714+0000) 2019-10-13T13:43:24.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:24.092+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.098529+0000) 2019-10-13T13:43:24.101 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:24.095+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.098870+0000) 2019-10-13T13:43:24.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:24.095+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.099280+0000) 2019-10-13T13:43:24.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:24.095+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.100086+0000) 2019-10-13T13:43:24.103 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:24.095+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.100415+0000) 2019-10-13T13:43:24.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:24.095+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.101343+0000) 2019-10-13T13:43:24.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:24.095+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.101673+0000) 2019-10-13T13:43:24.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:24.100+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.107645+0000) 2019-10-13T13:43:24.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:24.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:24.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:24.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.307416+0000) 2019-10-13T13:43:24.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:24.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.592814+0000) 2019-10-13T13:43:24.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:24.955+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:24.960402+0000) 2019-10-13T13:43:27.253 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:43:27.254 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:27.257 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:43:27.257 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:27.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:27.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:27.963168+0000) 2019-10-13T13:43:28.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:28.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:28.188649+0000) 2019-10-13T13:43:28.232 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:28.228+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:28.229591+0000) 2019-10-13T13:43:28.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:28.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:28.727741+0000) 2019-10-13T13:43:29.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:29.107+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.111365+0000) 2019-10-13T13:43:29.114 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:29.107+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.112459+0000) 2019-10-13T13:43:29.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:29.107+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.113258+0000) 2019-10-13T13:43:29.116 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:29.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.113983+0000) 2019-10-13T13:43:29.116 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:29.111+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.114018+0000) 2019-10-13T13:43:29.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:29.111+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.114585+0000) 2019-10-13T13:43:29.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:29.111+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.114855+0000) 2019-10-13T13:43:29.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:29.112+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:29.119365+0000) 2019-10-13T13:43:29.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:29.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188878 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:31.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:31.976+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:31.979916+0000) 2019-10-13T13:43:34.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:34.119+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.123724+0000) 2019-10-13T13:43:34.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:34.119+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.125414+0000) 2019-10-13T13:43:34.128 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:34.123+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.126153+0000) 2019-10-13T13:43:34.130 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:34.123+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.127835+0000) 2019-10-13T13:43:34.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:34.123+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.128608+0000) 2019-10-13T13:43:34.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:34.124+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.130592+0000) 2019-10-13T13:43:34.133 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:34.127+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.131039+0000) 2019-10-13T13:43:34.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:34.127+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188905 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:34.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:34.132+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.137162+0000) 2019-10-13T13:43:34.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:34.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.307584+0000) 2019-10-13T13:43:34.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:34.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.593028+0000) 2019-10-13T13:43:34.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:34.959+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:34.960574+0000) 2019-10-13T13:43:37.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:37.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:37.963368+0000) 2019-10-13T13:43:38.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:38.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:38.188879+0000) 2019-10-13T13:43:38.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:38.232+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:38.235522+0000) 2019-10-13T13:43:38.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:38.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:38.727957+0000) 2019-10-13T13:43:39.139 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:39.131+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.137171+0000) 2019-10-13T13:43:39.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:39.131+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.137441+0000) 2019-10-13T13:43:39.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:39.135+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.138288+0000) 2019-10-13T13:43:39.142 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:39.135+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.137910+0000) 2019-10-13T13:43:39.142 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:39.135+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.140098+0000) 2019-10-13T13:43:39.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:39.139+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.142385+0000) 2019-10-13T13:43:39.146 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:39.140+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.144545+0000) 2019-10-13T13:43:39.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:39.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:39.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:39.140+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:39.147305+0000) 2019-10-13T13:43:41.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:41.976+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:41.980140+0000) 2019-10-13T13:43:44.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:44.147+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.149980+0000) 2019-10-13T13:43:44.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:44.147+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.150691+0000) 2019-10-13T13:43:44.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:44.147+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.151195+0000) 2019-10-13T13:43:44.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:44.147+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.151504+0000) 2019-10-13T13:43:44.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:44.148+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.152220+0000) 2019-10-13T13:43:44.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:44.147+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.152220+0000) 2019-10-13T13:43:44.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:44.147+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.153451+0000) 2019-10-13T13:43:44.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:44.152+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.158829+0000) 2019-10-13T13:43:44.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:44.155+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188965 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:44.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:44.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.307764+0000) 2019-10-13T13:43:44.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:44.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.593241+0000) 2019-10-13T13:43:44.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:44.959+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:44.960754+0000) 2019-10-13T13:43:47.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:47.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:47.963617+0000) 2019-10-13T13:43:48.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:48.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:48.189078+0000) 2019-10-13T13:43:48.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:48.236+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:48.242115+0000) 2019-10-13T13:43:48.730 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:48.723+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:48.728148+0000) 2019-10-13T13:43:49.165 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:49.159+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.162547+0000) 2019-10-13T13:43:49.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:49.159+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.162885+0000) 2019-10-13T13:43:49.166 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:49.159+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.163483+0000) 2019-10-13T13:43:49.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:49.160+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.164031+0000) 2019-10-13T13:43:49.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:49.159+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.164423+0000) 2019-10-13T13:43:49.167 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:49.159+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.164214+0000) 2019-10-13T13:43:49.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:49.159+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.164943+0000) 2019-10-13T13:43:49.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:49.167+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:49.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:49.168+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:49.172222+0000) 2019-10-13T13:43:51.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:51.980+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:51.980365+0000) 2019-10-13T13:43:54.177 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:54.171+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.174707+0000) 2019-10-13T13:43:54.178 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:54.168+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.175496+0000) 2019-10-13T13:43:54.178 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:54.171+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.175747+0000) 2019-10-13T13:43:54.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:54.171+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.176352+0000) 2019-10-13T13:43:54.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:54.175+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.178079+0000) 2019-10-13T13:43:54.181 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:54.175+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.178811+0000) 2019-10-13T13:43:54.182 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:54.175+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.179335+0000) 2019-10-13T13:43:54.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:54.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:43:54.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:54.180+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.184206+0000) 2019-10-13T13:43:54.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:54.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.307996+0000) 2019-10-13T13:43:54.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:54.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.593420+0000) 2019-10-13T13:43:54.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:54.959+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:54.960959+0000) 2019-10-13T13:43:57.368 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:43:57.368 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:57.377 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:43:57.377 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:43:57.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:57.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:57.963779+0000) 2019-10-13T13:43:58.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:58.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:58.189282+0000) 2019-10-13T13:43:58.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:58.248+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:58.248798+0000) 2019-10-13T13:43:58.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:58.727+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:58.728339+0000) 2019-10-13T13:43:59.189 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:43:59.184+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.187222+0000) 2019-10-13T13:43:59.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:43:59.183+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.187517+0000) 2019-10-13T13:43:59.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:43:59.183+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.188223+0000) 2019-10-13T13:43:59.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:43:59.183+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.188856+0000) 2019-10-13T13:43:59.191 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:43:59.183+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.189016+0000) 2019-10-13T13:43:59.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:43:59.183+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.189407+0000) 2019-10-13T13:43:59.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:43:59.183+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.189659+0000) 2019-10-13T13:43:59.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:43:59.188+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:43:59.194390+0000) 2019-10-13T13:43:59.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:43:59.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:01.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:01.980+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:01.980616+0000) 2019-10-13T13:44:04.202 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:04.195+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.199846+0000) 2019-10-13T13:44:04.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:04.195+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.200597+0000) 2019-10-13T13:44:04.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:04.195+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.201647+0000) 2019-10-13T13:44:04.204 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:04.199+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.202107+0000) 2019-10-13T13:44:04.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:04.200+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.202562+0000) 2019-10-13T13:44:04.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:04.199+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.204438+0000) 2019-10-13T13:44:04.207 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:04.199+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.205031+0000) 2019-10-13T13:44:04.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:04.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189088 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:04.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:04.212+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.213449+0000) 2019-10-13T13:44:04.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:04.303+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.308177+0000) 2019-10-13T13:44:04.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:04.587+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.593597+0000) 2019-10-13T13:44:04.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:04.955+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:04.961137+0000) 2019-10-13T13:44:07.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:07.959+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:07.963977+0000) 2019-10-13T13:44:08.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:08.183+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:08.189492+0000) 2019-10-13T13:44:08.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:08.251+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:08.255216+0000) 2019-10-13T13:44:08.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:08.727+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:08.728540+0000) 2019-10-13T13:44:09.214 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:09.207+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.212086+0000) 2019-10-13T13:44:09.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:09.207+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.212424+0000) 2019-10-13T13:44:09.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:09.211+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.213867+0000) 2019-10-13T13:44:09.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:09.211+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.214423+0000) 2019-10-13T13:44:09.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:09.211+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.214758+0000) 2019-10-13T13:44:09.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:09.211+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.214715+0000) 2019-10-13T13:44:09.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:09.211+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.216019+0000) 2019-10-13T13:44:09.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:09.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:09.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:09.219+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:09.222240+0000) 2019-10-13T13:44:11.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:11.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:11.980809+0000) 2019-10-13T13:44:14.227 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:14.223+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.225137+0000) 2019-10-13T13:44:14.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:14.218+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.225433+0000) 2019-10-13T13:44:14.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:14.222+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.225819+0000) 2019-10-13T13:44:14.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:14.222+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.226549+0000) 2019-10-13T13:44:14.229 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:14.222+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.226839+0000) 2019-10-13T13:44:14.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:14.222+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.226990+0000) 2019-10-13T13:44:14.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:14.222+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.228531+0000) 2019-10-13T13:44:14.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:14.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189150 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:14.236 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:14.231+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.234562+0000) 2019-10-13T13:44:14.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:14.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.308375+0000) 2019-10-13T13:44:14.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:14.590+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.593804+0000) 2019-10-13T13:44:14.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:14.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:14.961311+0000) 2019-10-13T13:44:17.966 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:17.962+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:17.964180+0000) 2019-10-13T13:44:18.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:18.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:18.189729+0000) 2019-10-13T13:44:18.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:18.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:18.261947+0000) 2019-10-13T13:44:18.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:18.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:18.728740+0000) 2019-10-13T13:44:19.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:19.230+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.237653+0000) 2019-10-13T13:44:19.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:19.235+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.238541+0000) 2019-10-13T13:44:19.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:19.234+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.238134+0000) 2019-10-13T13:44:19.241 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:19.234+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.238669+0000) 2019-10-13T13:44:19.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:19.234+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.239006+0000) 2019-10-13T13:44:19.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:19.234+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.239286+0000) 2019-10-13T13:44:19.242 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:19.234+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.239634+0000) 2019-10-13T13:44:19.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:19.243+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:19.246413+0000) 2019-10-13T13:44:19.249 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:19.242+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:21.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:21.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:21.981020+0000) 2019-10-13T13:44:24.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:24.242+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.249339+0000) 2019-10-13T13:44:24.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:24.246+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.250367+0000) 2019-10-13T13:44:24.253 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:24.246+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.251057+0000) 2019-10-13T13:44:24.253 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:24.247+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.251720+0000) 2019-10-13T13:44:24.254 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:24.246+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.251792+0000) 2019-10-13T13:44:24.255 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:24.246+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.252637+0000) 2019-10-13T13:44:24.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:24.246+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.252981+0000) 2019-10-13T13:44:24.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:24.255+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.259867+0000) 2019-10-13T13:44:24.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:24.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189210 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:24.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:24.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.308560+0000) 2019-10-13T13:44:24.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:24.590+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.593985+0000) 2019-10-13T13:44:24.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:24.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:24.961491+0000) 2019-10-13T13:44:27.543 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:44:27.543 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:27.547 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:44:27.547 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:27.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:27.962+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:27.964381+0000) 2019-10-13T13:44:28.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:28.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:28.190016+0000) 2019-10-13T13:44:28.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:28.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:28.267829+0000) 2019-10-13T13:44:28.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:28.726+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:28.728932+0000) 2019-10-13T13:44:29.265 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:29.258+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.262705+0000) 2019-10-13T13:44:29.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:29.258+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.264859+0000) 2019-10-13T13:44:29.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:29.258+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.265593+0000) 2019-10-13T13:44:29.268 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:29.262+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.266333+0000) 2019-10-13T13:44:29.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:29.262+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.266910+0000) 2019-10-13T13:44:29.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:29.263+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.267447+0000) 2019-10-13T13:44:29.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:29.262+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.267902+0000) 2019-10-13T13:44:29.274 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:29.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189244 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:29.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:29.271+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:29.274220+0000) 2019-10-13T13:44:31.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:31.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:31.981168+0000) 2019-10-13T13:44:34.277 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:34.271+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.275449+0000) 2019-10-13T13:44:34.279 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:34.270+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.276347+0000) 2019-10-13T13:44:34.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:34.274+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.278171+0000) 2019-10-13T13:44:34.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:34.274+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.279506+0000) 2019-10-13T13:44:34.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:34.278+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.281883+0000) 2019-10-13T13:44:34.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:34.278+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.282843+0000) 2019-10-13T13:44:34.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:34.278+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.283461+0000) 2019-10-13T13:44:34.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:34.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:34.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:34.283+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.284781+0000) 2019-10-13T13:44:34.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:34.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.308742+0000) 2019-10-13T13:44:34.596 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:34.590+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.594159+0000) 2019-10-13T13:44:34.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:34.954+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:34.961662+0000) 2019-10-13T13:44:37.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:37.962+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:37.964578+0000) 2019-10-13T13:44:38.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:38.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:38.190236+0000) 2019-10-13T13:44:38.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:38.271+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:38.274471+0000) 2019-10-13T13:44:38.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:38.722+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:38.729087+0000) 2019-10-13T13:44:39.290 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:39.282+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.287941+0000) 2019-10-13T13:44:39.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:39.282+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.288286+0000) 2019-10-13T13:44:39.291 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:39.282+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.288836+0000) 2019-10-13T13:44:39.292 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:39.282+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.289440+0000) 2019-10-13T13:44:39.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:39.287+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.290883+0000) 2019-10-13T13:44:39.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:39.286+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.291032+0000) 2019-10-13T13:44:39.293 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:39.286+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.291118+0000) 2019-10-13T13:44:39.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:39.295+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:39.296394+0000) 2019-10-13T13:44:39.299 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:39.290+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:41.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:41.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:41.981385+0000) 2019-10-13T13:44:44.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:44.298+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.301924+0000) 2019-10-13T13:44:44.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:44.298+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.302281+0000) 2019-10-13T13:44:44.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:44.299+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.302904+0000) 2019-10-13T13:44:44.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:44.298+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.302048+0000) 2019-10-13T13:44:44.306 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:44.298+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.303199+0000) 2019-10-13T13:44:44.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:44.298+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.303572+0000) 2019-10-13T13:44:44.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:44.298+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.304638+0000) 2019-10-13T13:44:44.310 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:44.303+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.307760+0000) 2019-10-13T13:44:44.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:44.302+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:44.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:44.302+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.308963+0000) 2019-10-13T13:44:44.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:44.590+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.594336+0000) 2019-10-13T13:44:44.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:44.958+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:44.961840+0000) 2019-10-13T13:44:47.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:47.962+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:47.964777+0000) 2019-10-13T13:44:48.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:48.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:48.190460+0000) 2019-10-13T13:44:48.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:48.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:48.281146+0000) 2019-10-13T13:44:48.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:48.726+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:48.729295+0000) 2019-10-13T13:44:49.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:49.306+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.312171+0000) 2019-10-13T13:44:49.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:49.310+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.313726+0000) 2019-10-13T13:44:49.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:49.310+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.314655+0000) 2019-10-13T13:44:49.317 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:49.310+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.315029+0000) 2019-10-13T13:44:49.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:49.310+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.315907+0000) 2019-10-13T13:44:49.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:49.315+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.316927+0000) 2019-10-13T13:44:49.319 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:49.310+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.317195+0000) 2019-10-13T13:44:49.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:49.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189364 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:49.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:49.319+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:49.320808+0000) 2019-10-13T13:44:51.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:51.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:51.981581+0000) 2019-10-13T13:44:54.311 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:54.306+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.309182+0000) 2019-10-13T13:44:54.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:54.318+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.323744+0000) 2019-10-13T13:44:54.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:54.318+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.323844+0000) 2019-10-13T13:44:54.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:54.318+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.325153+0000) 2019-10-13T13:44:54.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:54.323+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.325650+0000) 2019-10-13T13:44:54.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:54.322+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.326122+0000) 2019-10-13T13:44:54.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:54.322+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.326714+0000) 2019-10-13T13:44:54.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:54.322+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.326994+0000) 2019-10-13T13:44:54.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:54.327+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.331270+0000) 2019-10-13T13:44:54.335 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:54.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189392 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:54.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:54.594+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.594513+0000) 2019-10-13T13:44:54.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:54.958+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:54.962014+0000) 2019-10-13T13:44:57.590 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:44:57.590 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:57.594 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:44:57.594 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:44:57.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:57.962+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:57.964980+0000) 2019-10-13T13:44:58.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:58.186+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:58.190674+0000) 2019-10-13T13:44:58.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:58.283+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:58.287842+0000) 2019-10-13T13:44:58.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:58.726+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:58.729489+0000) 2019-10-13T13:44:59.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:44:59.330+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.336424+0000) 2019-10-13T13:44:59.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:44:59.330+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.336970+0000) 2019-10-13T13:44:59.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:44:59.335+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.338223+0000) 2019-10-13T13:44:59.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:44:59.334+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.338392+0000) 2019-10-13T13:44:59.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:44:59.334+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.338662+0000) 2019-10-13T13:44:59.341 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:44:59.334+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.339461+0000) 2019-10-13T13:44:59.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:44:59.334+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.340141+0000) 2019-10-13T13:44:59.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:44:59.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:44:59.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:44:59.343+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:44:59.346896+0000) 2019-10-13T13:45:01.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:01.979+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:01.981771+0000) 2019-10-13T13:45:04.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:04.306+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.309394+0000) 2019-10-13T13:45:04.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:04.346+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.349914+0000) 2019-10-13T13:45:04.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:04.346+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.350348+0000) 2019-10-13T13:45:04.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:04.346+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.350876+0000) 2019-10-13T13:45:04.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:04.346+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.351662+0000) 2019-10-13T13:45:04.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:04.346+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.351982+0000) 2019-10-13T13:45:04.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:04.351+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.352685+0000) 2019-10-13T13:45:04.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:04.346+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.353030+0000) 2019-10-13T13:45:04.359 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:04.355+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.357532+0000) 2019-10-13T13:45:04.360 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:04.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:04.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:04.594+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.594733+0000) 2019-10-13T13:45:04.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:04.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:04.962187+0000) 2019-10-13T13:45:07.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:07.961+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:07.965208+0000) 2019-10-13T13:45:08.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:08.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:08.190874+0000) 2019-10-13T13:45:08.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:08.291+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:08.294433+0000) 2019-10-13T13:45:08.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:08.725+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:08.729641+0000) 2019-10-13T13:45:09.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:09.357+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.362059+0000) 2019-10-13T13:45:09.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:09.359+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.362915+0000) 2019-10-13T13:45:09.365 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:09.357+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.362957+0000) 2019-10-13T13:45:09.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:09.357+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.365162+0000) 2019-10-13T13:45:09.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:09.357+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.365132+0000) 2019-10-13T13:45:09.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:09.357+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.365547+0000) 2019-10-13T13:45:09.369 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:09.361+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.366904+0000) 2019-10-13T13:45:09.373 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:09.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:09.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:09.371+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:09.372607+0000) 2019-10-13T13:45:11.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:11.978+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:11.981968+0000) 2019-10-13T13:45:14.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:14.305+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.309623+0000) 2019-10-13T13:45:14.376 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:14.369+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.373689+0000) 2019-10-13T13:45:14.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:14.369+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.375761+0000) 2019-10-13T13:45:14.379 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:14.369+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.377019+0000) 2019-10-13T13:45:14.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:14.369+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.377415+0000) 2019-10-13T13:45:14.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:14.374+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.378378+0000) 2019-10-13T13:45:14.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:14.373+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.380069+0000) 2019-10-13T13:45:14.382 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:14.373+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.380343+0000) 2019-10-13T13:45:14.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:14.377+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:14.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:14.382+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.386038+0000) 2019-10-13T13:45:14.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:14.589+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.594940+0000) 2019-10-13T13:45:14.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:14.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:14.962362+0000) 2019-10-13T13:45:17.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:17.961+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:17.965362+0000) 2019-10-13T13:45:18.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:18.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:18.191082+0000) 2019-10-13T13:45:18.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:18.302+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:18.305944+0000) 2019-10-13T13:45:18.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:18.729+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:18.729835+0000) 2019-10-13T13:45:19.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:19.381+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.387612+0000) 2019-10-13T13:45:19.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:19.381+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.388147+0000) 2019-10-13T13:45:19.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:19.381+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.388703+0000) 2019-10-13T13:45:19.391 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:19.381+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.389430+0000) 2019-10-13T13:45:19.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:19.385+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.389809+0000) 2019-10-13T13:45:19.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:19.385+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.390462+0000) 2019-10-13T13:45:19.393 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:19.386+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.391577+0000) 2019-10-13T13:45:19.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:19.394+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:19.396867+0000) 2019-10-13T13:45:19.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:19.389+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189542 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:21.984 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:21.978+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:21.982191+0000) 2019-10-13T13:45:24.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:24.309+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.309819+0000) 2019-10-13T13:45:24.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:24.397+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.400168+0000) 2019-10-13T13:45:24.403 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:24.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.401139+0000) 2019-10-13T13:45:24.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:24.397+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.401559+0000) 2019-10-13T13:45:24.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:24.397+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.401315+0000) 2019-10-13T13:45:24.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:24.401+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.401965+0000) 2019-10-13T13:45:24.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:24.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.402503+0000) 2019-10-13T13:45:24.405 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:24.401+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.402725+0000) 2019-10-13T13:45:24.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:24.402+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.406480+0000) 2019-10-13T13:45:24.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:24.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:24.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:24.593+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.595121+0000) 2019-10-13T13:45:24.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:24.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:24.962570+0000) 2019-10-13T13:45:27.696 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:45:27.697 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:27.700 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:45:27.700 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:27.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:27.961+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:27.965559+0000) 2019-10-13T13:45:28.193 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:28.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:28.191312+0000) 2019-10-13T13:45:28.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:28.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:28.312450+0000) 2019-10-13T13:45:28.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:28.729+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:28.730025+0000) 2019-10-13T13:45:29.416 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:29.409+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.413313+0000) 2019-10-13T13:45:29.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:29.409+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.413711+0000) 2019-10-13T13:45:29.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:29.413+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.414562+0000) 2019-10-13T13:45:29.417 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:29.410+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.415109+0000) 2019-10-13T13:45:29.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:29.413+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.415233+0000) 2019-10-13T13:45:29.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:29.413+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.416731+0000) 2019-10-13T13:45:29.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:29.417+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.419040+0000) 2019-10-13T13:45:29.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:29.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189603 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:29.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:29.422+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:29.425363+0000) 2019-10-13T13:45:31.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:31.978+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:31.982405+0000) 2019-10-13T13:45:34.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:34.309+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.310033+0000) 2019-10-13T13:45:34.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:34.421+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.425575+0000) 2019-10-13T13:45:34.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:34.425+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.425988+0000) 2019-10-13T13:45:34.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:34.425+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.426568+0000) 2019-10-13T13:45:34.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:34.425+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.426905+0000) 2019-10-13T13:45:34.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:34.425+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.427709+0000) 2019-10-13T13:45:34.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:34.426+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.428424+0000) 2019-10-13T13:45:34.431 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:34.425+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.429340+0000) 2019-10-13T13:45:34.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:34.433+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189632 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:34.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:34.430+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.435125+0000) 2019-10-13T13:45:34.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:34.593+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.595328+0000) 2019-10-13T13:45:34.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:34.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:34.962744+0000) 2019-10-13T13:45:37.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:37.965+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:37.965807+0000) 2019-10-13T13:45:38.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:38.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:38.191514+0000) 2019-10-13T13:45:38.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:38.314+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:38.318945+0000) 2019-10-13T13:45:38.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:38.729+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:38.730227+0000) 2019-10-13T13:45:39.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:39.433+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.437481+0000) 2019-10-13T13:45:39.440 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:39.437+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.438147+0000) 2019-10-13T13:45:39.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:39.437+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.438770+0000) 2019-10-13T13:45:39.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:39.437+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.440137+0000) 2019-10-13T13:45:39.443 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:39.437+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.441054+0000) 2019-10-13T13:45:39.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:39.441+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.441732+0000) 2019-10-13T13:45:39.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:39.438+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.442453+0000) 2019-10-13T13:45:39.450 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:39.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:39.451 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:39.446+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:39.449015+0000) 2019-10-13T13:45:41.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:41.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:41.982622+0000) 2019-10-13T13:45:44.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:44.309+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.310267+0000) 2019-10-13T13:45:44.455 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:44.449+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.451961+0000) 2019-10-13T13:45:44.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:44.449+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.452346+0000) 2019-10-13T13:45:44.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:44.449+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.452850+0000) 2019-10-13T13:45:44.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:44.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.453402+0000) 2019-10-13T13:45:44.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:44.449+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.453633+0000) 2019-10-13T13:45:44.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:44.453+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.454089+0000) 2019-10-13T13:45:44.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:44.453+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.455630+0000) 2019-10-13T13:45:44.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:44.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:44.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:44.458+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.463126+0000) 2019-10-13T13:45:44.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:44.593+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.595538+0000) 2019-10-13T13:45:44.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:44.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:44.962919+0000) 2019-10-13T13:45:47.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:47.965+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:47.966039+0000) 2019-10-13T13:45:48.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:48.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:48.191710+0000) 2019-10-13T13:45:48.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:48.322+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:48.325540+0000) 2019-10-13T13:45:48.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:48.729+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:48.730422+0000) 2019-10-13T13:45:49.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:49.461+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.464481+0000) 2019-10-13T13:45:49.467 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:49.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.464948+0000) 2019-10-13T13:45:49.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:49.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.464963+0000) 2019-10-13T13:45:49.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:49.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.465460+0000) 2019-10-13T13:45:49.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:49.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.466935+0000) 2019-10-13T13:45:49.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:49.465+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.467069+0000) 2019-10-13T13:45:49.469 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:49.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.467393+0000) 2019-10-13T13:45:49.475 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:49.470+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:49.472659+0000) 2019-10-13T13:45:49.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:49.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189723 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:51.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:51.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:51.982816+0000) 2019-10-13T13:45:54.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:54.309+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.310427+0000) 2019-10-13T13:45:54.481 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:54.477+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.478295+0000) 2019-10-13T13:45:54.481 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:54.477+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.478929+0000) 2019-10-13T13:45:54.482 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:54.474+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.479873+0000) 2019-10-13T13:45:54.483 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:54.477+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.480717+0000) 2019-10-13T13:45:54.484 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:54.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.481517+0000) 2019-10-13T13:45:54.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:54.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.481701+0000) 2019-10-13T13:45:54.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:54.481+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.482198+0000) 2019-10-13T13:45:54.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:54.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:54.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:54.482+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.486799+0000) 2019-10-13T13:45:54.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:54.593+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.595713+0000) 2019-10-13T13:45:54.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:54.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:54.963097+0000) 2019-10-13T13:45:57.846 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:45:57.846 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:57.850 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:45:57.850 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:45:57.968 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:57.965+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:57.966210+0000) 2019-10-13T13:45:58.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:58.189+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:58.191912+0000) 2019-10-13T13:45:58.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:58.330+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:58.331910+0000) 2019-10-13T13:45:58.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:58.729+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:58.730594+0000) 2019-10-13T13:45:59.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:45:59.486+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.490384+0000) 2019-10-13T13:45:59.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:45:59.489+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.491292+0000) 2019-10-13T13:45:59.494 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:45:59.489+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.491728+0000) 2019-10-13T13:45:59.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:45:59.489+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.492679+0000) 2019-10-13T13:45:59.495 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:45:59.489+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.493483+0000) 2019-10-13T13:45:59.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:45:59.489+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.493690+0000) 2019-10-13T13:45:59.497 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:45:59.493+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.494768+0000) 2019-10-13T13:45:59.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:45:59.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:45:59.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:45:59.498+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:45:59.500742+0000) 2019-10-13T13:46:01.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:01.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:01.983012+0000) 2019-10-13T13:46:04.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:04.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.310625+0000) 2019-10-13T13:46:04.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:04.500+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.502917+0000) 2019-10-13T13:46:04.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:04.500+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.503475+0000) 2019-10-13T13:46:04.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:04.500+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.503668+0000) 2019-10-13T13:46:04.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:04.500+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.504403+0000) 2019-10-13T13:46:04.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:04.502+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.506237+0000) 2019-10-13T13:46:04.509 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:04.504+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.506761+0000) 2019-10-13T13:46:04.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:04.504+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.506754+0000) 2019-10-13T13:46:04.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:04.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:04.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:04.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.515761+0000) 2019-10-13T13:46:04.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:04.592+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.595900+0000) 2019-10-13T13:46:04.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:04.960+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:04.963272+0000) 2019-10-13T13:46:07.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:07.964+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:07.966414+0000) 2019-10-13T13:46:08.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:08.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:08.192127+0000) 2019-10-13T13:46:08.341 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:08.338+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:08.338451+0000) 2019-10-13T13:46:08.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:08.728+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:08.730797+0000) 2019-10-13T13:46:09.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:09.512+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.515670+0000) 2019-10-13T13:46:09.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:09.512+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.516464+0000) 2019-10-13T13:46:09.519 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:09.512+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.517124+0000) 2019-10-13T13:46:09.521 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:09.516+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.518521+0000) 2019-10-13T13:46:09.521 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:09.516+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.518797+0000) 2019-10-13T13:46:09.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:09.514+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.519751+0000) 2019-10-13T13:46:09.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:09.516+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.520385+0000) 2019-10-13T13:46:09.527 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:09.522+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:09.525643+0000) 2019-10-13T13:46:09.528 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:09.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:11.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:11.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:11.983209+0000) 2019-10-13T13:46:14.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:14.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.310858+0000) 2019-10-13T13:46:14.531 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:14.524+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.528837+0000) 2019-10-13T13:46:14.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:14.525+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.529618+0000) 2019-10-13T13:46:14.534 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:14.528+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.531044+0000) 2019-10-13T13:46:14.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:14.528+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.531105+0000) 2019-10-13T13:46:14.534 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:14.528+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.531356+0000) 2019-10-13T13:46:14.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:14.528+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.532294+0000) 2019-10-13T13:46:14.535 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:14.528+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.532720+0000) 2019-10-13T13:46:14.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:14.533+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.536328+0000) 2019-10-13T13:46:14.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:14.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:14.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:14.592+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.596108+0000) 2019-10-13T13:46:14.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:14.960+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:14.963452+0000) 2019-10-13T13:46:17.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:17.964+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:17.966655+0000) 2019-10-13T13:46:18.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:18.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:18.192322+0000) 2019-10-13T13:46:18.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:18.341+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:18.345008+0000) 2019-10-13T13:46:18.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:18.728+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:18.730990+0000) 2019-10-13T13:46:19.547 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:19.540+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.544285+0000) 2019-10-13T13:46:19.548 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:19.541+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.545130+0000) 2019-10-13T13:46:19.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:19.540+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.544696+0000) 2019-10-13T13:46:19.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:19.540+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.544519+0000) 2019-10-13T13:46:19.548 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:19.540+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.545479+0000) 2019-10-13T13:46:19.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:19.544+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.546157+0000) 2019-10-13T13:46:19.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:19.544+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.546648+0000) 2019-10-13T13:46:19.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:19.545+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:19.548707+0000) 2019-10-13T13:46:19.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:19.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:21.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:21.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:21.983403+0000) 2019-10-13T13:46:24.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:24.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.311054+0000) 2019-10-13T13:46:24.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:24.552+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.554600+0000) 2019-10-13T13:46:24.557 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:24.552+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.554997+0000) 2019-10-13T13:46:24.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:24.552+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.555261+0000) 2019-10-13T13:46:24.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:24.552+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.556039+0000) 2019-10-13T13:46:24.559 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:24.552+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.557066+0000) 2019-10-13T13:46:24.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:24.553+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.557434+0000) 2019-10-13T13:46:24.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:24.552+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.557603+0000) 2019-10-13T13:46:24.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:24.557+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.560827+0000) 2019-10-13T13:46:24.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:24.560+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:24.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:24.592+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.596255+0000) 2019-10-13T13:46:24.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:24.960+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:24.963635+0000) 2019-10-13T13:46:27.960 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:46:27.961 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:27.964 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:46:27.964 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:27.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:27.964+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:27.966844+0000) 2019-10-13T13:46:28.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:28.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:28.192555+0000) 2019-10-13T13:46:28.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:28.345+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:28.351566+0000) 2019-10-13T13:46:28.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:28.728+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:28.731182+0000) 2019-10-13T13:46:29.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:29.564+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.566720+0000) 2019-10-13T13:46:29.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:29.564+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.566950+0000) 2019-10-13T13:46:29.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:29.564+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.567962+0000) 2019-10-13T13:46:29.571 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:29.564+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.568463+0000) 2019-10-13T13:46:29.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:29.564+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.568989+0000) 2019-10-13T13:46:29.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:29.568+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.569858+0000) 2019-10-13T13:46:29.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:29.565+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.570624+0000) 2019-10-13T13:46:29.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:29.572+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:29.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:29.573+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:29.577545+0000) 2019-10-13T13:46:31.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:31.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:31.983628+0000) 2019-10-13T13:46:34.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:34.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.311227+0000) 2019-10-13T13:46:34.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:34.576+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.578833+0000) 2019-10-13T13:46:34.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:34.573+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.579490+0000) 2019-10-13T13:46:34.582 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:34.576+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.579454+0000) 2019-10-13T13:46:34.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:34.576+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.580311+0000) 2019-10-13T13:46:34.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:34.576+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.580725+0000) 2019-10-13T13:46:34.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:34.576+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.581229+0000) 2019-10-13T13:46:34.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:34.576+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.581591+0000) 2019-10-13T13:46:34.588 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:34.581+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.586256+0000) 2019-10-13T13:46:34.590 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:34.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:34.598 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:34.592+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.596400+0000) 2019-10-13T13:46:34.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:34.960+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:34.963809+0000) 2019-10-13T13:46:37.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:37.964+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:37.967000+0000) 2019-10-13T13:46:38.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:38.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:38.192749+0000) 2019-10-13T13:46:38.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:38.353+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:38.358029+0000) 2019-10-13T13:46:38.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:38.728+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:38.731383+0000) 2019-10-13T13:46:39.594 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:39.588+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.591572+0000) 2019-10-13T13:46:39.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:39.588+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.592309+0000) 2019-10-13T13:46:39.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:39.588+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.592281+0000) 2019-10-13T13:46:39.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:39.589+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.593341+0000) 2019-10-13T13:46:39.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:39.592+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.593929+0000) 2019-10-13T13:46:39.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:39.592+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.594404+0000) 2019-10-13T13:46:39.597 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:39.592+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.594694+0000) 2019-10-13T13:46:39.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:39.593+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:39.599238+0000) 2019-10-13T13:46:39.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:39.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:41.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:41.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:41.983833+0000) 2019-10-13T13:46:44.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:44.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.311424+0000) 2019-10-13T13:46:44.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:44.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.596606+0000) 2019-10-13T13:46:44.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:44.600+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.602439+0000) 2019-10-13T13:46:44.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:44.600+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.603954+0000) 2019-10-13T13:46:44.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:44.600+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.604580+0000) 2019-10-13T13:46:44.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:44.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.607392+0000) 2019-10-13T13:46:44.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:44.604+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.607606+0000) 2019-10-13T13:46:44.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:44.604+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.608264+0000) 2019-10-13T13:46:44.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:44.604+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.609541+0000) 2019-10-13T13:46:44.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:44.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:44.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:44.609+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.615166+0000) 2019-10-13T13:46:44.966 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:44.960+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:44.963989+0000) 2019-10-13T13:46:47.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:47.964+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:47.967194+0000) 2019-10-13T13:46:48.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:48.188+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:48.192975+0000) 2019-10-13T13:46:48.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:48.361+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:48.364596+0000) 2019-10-13T13:46:48.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:48.728+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:48.731588+0000) 2019-10-13T13:46:49.620 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:49.612+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.617617+0000) 2019-10-13T13:46:49.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:49.616+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.618195+0000) 2019-10-13T13:46:49.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:49.616+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.618473+0000) 2019-10-13T13:46:49.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:49.616+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.619009+0000) 2019-10-13T13:46:49.623 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:49.616+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.620768+0000) 2019-10-13T13:46:49.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:49.616+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.621241+0000) 2019-10-13T13:46:49.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:49.617+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.621769+0000) 2019-10-13T13:46:49.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:49.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:49.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:49.625+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:49.628991+0000) 2019-10-13T13:46:51.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:51.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:51.984025+0000) 2019-10-13T13:46:54.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:54.308+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.311624+0000) 2019-10-13T13:46:54.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:54.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.596839+0000) 2019-10-13T13:46:54.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:54.625+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.629602+0000) 2019-10-13T13:46:54.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:54.628+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.630394+0000) 2019-10-13T13:46:54.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:54.628+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.631104+0000) 2019-10-13T13:46:54.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:54.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.631256+0000) 2019-10-13T13:46:54.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:54.628+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.631715+0000) 2019-10-13T13:46:54.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:54.628+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.632342+0000) 2019-10-13T13:46:54.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:54.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.632876+0000) 2019-10-13T13:46:54.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:54.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:54.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:54.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.640163+0000) 2019-10-13T13:46:54.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:54.964+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:54.964175+0000) 2019-10-13T13:46:57.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:57.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:57.967391+0000) 2019-10-13T13:46:58.084 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:46:58.084 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:58.088 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:46:58.088 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:46:58.195 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:58.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:58.193175+0000) 2019-10-13T13:46:58.373 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:58.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:58.371006+0000) 2019-10-13T13:46:58.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:58.727+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:58.731813+0000) 2019-10-13T13:46:59.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:46:59.639+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.641987+0000) 2019-10-13T13:46:59.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:46:59.639+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.643452+0000) 2019-10-13T13:46:59.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:46:59.639+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.645053+0000) 2019-10-13T13:46:59.648 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:46:59.639+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.645339+0000) 2019-10-13T13:46:59.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:46:59.643+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.647077+0000) 2019-10-13T13:46:59.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:46:59.643+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.647666+0000) 2019-10-13T13:46:59.650 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:46:59.645+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.648200+0000) 2019-10-13T13:46:59.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:46:59.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190143 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:46:59.657 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:46:59.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:46:59.654746+0000) 2019-10-13T13:47:01.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:01.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:01.984290+0000) 2019-10-13T13:47:04.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:04.307+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.311832+0000) 2019-10-13T13:47:04.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:04.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.597038+0000) 2019-10-13T13:47:04.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:04.651+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.656726+0000) 2019-10-13T13:47:04.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:04.651+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.657314+0000) 2019-10-13T13:47:04.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:04.651+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.657673+0000) 2019-10-13T13:47:04.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:04.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.658397+0000) 2019-10-13T13:47:04.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:04.655+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.658589+0000) 2019-10-13T13:47:04.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:04.655+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.659204+0000) 2019-10-13T13:47:04.662 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:04.655+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.659314+0000) 2019-10-13T13:47:04.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:04.657+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.662960+0000) 2019-10-13T13:47:04.667 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:04.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190172 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:04.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:04.963+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:04.964372+0000) 2019-10-13T13:47:07.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:07.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:07.967585+0000) 2019-10-13T13:47:08.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:08.187+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:08.193374+0000) 2019-10-13T13:47:08.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:08.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:08.377502+0000) 2019-10-13T13:47:08.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:08.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:08.732010+0000) 2019-10-13T13:47:09.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:09.663+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.668288+0000) 2019-10-13T13:47:09.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:09.663+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.669028+0000) 2019-10-13T13:47:09.672 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:09.665+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.670224+0000) 2019-10-13T13:47:09.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:09.667+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.670367+0000) 2019-10-13T13:47:09.673 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:09.667+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.670665+0000) 2019-10-13T13:47:09.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:09.667+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.671023+0000) 2019-10-13T13:47:09.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:09.667+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.672002+0000) 2019-10-13T13:47:09.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:09.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:09.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:09.673+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:09.678855+0000) 2019-10-13T13:47:11.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:11.981+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:11.984520+0000) 2019-10-13T13:47:14.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:14.311+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.311986+0000) 2019-10-13T13:47:14.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:14.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.597251+0000) 2019-10-13T13:47:14.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:14.679+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.682140+0000) 2019-10-13T13:47:14.686 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:14.679+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.683199+0000) 2019-10-13T13:47:14.686 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:14.679+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.683526+0000) 2019-10-13T13:47:14.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:14.681+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.684482+0000) 2019-10-13T13:47:14.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:14.679+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.683992+0000) 2019-10-13T13:47:14.687 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:14.679+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.684591+0000) 2019-10-13T13:47:14.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:14.679+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.685303+0000) 2019-10-13T13:47:14.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:14.685+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.688799+0000) 2019-10-13T13:47:14.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:14.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:14.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:14.963+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:14.964570+0000) 2019-10-13T13:47:17.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:17.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:17.967785+0000) 2019-10-13T13:47:18.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:18.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:18.193608+0000) 2019-10-13T13:47:18.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:18.381+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:18.384027+0000) 2019-10-13T13:47:18.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:18.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:18.732224+0000) 2019-10-13T13:47:19.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:19.691+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.695328+0000) 2019-10-13T13:47:19.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:19.691+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.695835+0000) 2019-10-13T13:47:19.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:19.691+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.697538+0000) 2019-10-13T13:47:19.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:19.695+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.697998+0000) 2019-10-13T13:47:19.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:19.695+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.698266+0000) 2019-10-13T13:47:19.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:19.695+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.699294+0000) 2019-10-13T13:47:19.702 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:19.692+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.699849+0000) 2019-10-13T13:47:19.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:19.696+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:19.703057+0000) 2019-10-13T13:47:19.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:19.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:21.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:21.980+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:21.984686+0000) 2019-10-13T13:47:24.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:24.307+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.312136+0000) 2019-10-13T13:47:24.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:24.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.597481+0000) 2019-10-13T13:47:24.710 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:24.703+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.707964+0000) 2019-10-13T13:47:24.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:24.704+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.708365+0000) 2019-10-13T13:47:24.711 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:24.703+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.708644+0000) 2019-10-13T13:47:24.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:24.703+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.709208+0000) 2019-10-13T13:47:24.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:24.707+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.710102+0000) 2019-10-13T13:47:24.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:24.707+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.710064+0000) 2019-10-13T13:47:24.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:24.707+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.711080+0000) 2019-10-13T13:47:24.719 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:24.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:24.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:24.712+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.717136+0000) 2019-10-13T13:47:24.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:24.963+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:24.964747+0000) 2019-10-13T13:47:27.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:27.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:27.967984+0000) 2019-10-13T13:47:28.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:28.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:28.193809+0000) 2019-10-13T13:47:28.202 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:47:28.202 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:28.205 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:47:28.205 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:28.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:28.384+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:28.390483+0000) 2019-10-13T13:47:28.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:28.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:28.732421+0000) 2019-10-13T13:47:29.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:29.715+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.720410+0000) 2019-10-13T13:47:29.723 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:29.715+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.720926+0000) 2019-10-13T13:47:29.724 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:29.715+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.721103+0000) 2019-10-13T13:47:29.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:29.715+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.721580+0000) 2019-10-13T13:47:29.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:29.719+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.723572+0000) 2019-10-13T13:47:29.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:29.720+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.724198+0000) 2019-10-13T13:47:29.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:29.719+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.724337+0000) 2019-10-13T13:47:29.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:29.727+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:29.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:29.724+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:29.731673+0000) 2019-10-13T13:47:31.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:31.980+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:31.984835+0000) 2019-10-13T13:47:34.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:34.311+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.312345+0000) 2019-10-13T13:47:34.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:34.591+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.597676+0000) 2019-10-13T13:47:34.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:34.727+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.732649+0000) 2019-10-13T13:47:34.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:34.728+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.734216+0000) 2019-10-13T13:47:34.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:34.727+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.733593+0000) 2019-10-13T13:47:34.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:34.731+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.735055+0000) 2019-10-13T13:47:34.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:34.731+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.737006+0000) 2019-10-13T13:47:34.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:34.731+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.737510+0000) 2019-10-13T13:47:34.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:34.735+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.737895+0000) 2019-10-13T13:47:34.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:34.739+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:34.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:34.736+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.742800+0000) 2019-10-13T13:47:34.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:34.963+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:34.964920+0000) 2019-10-13T13:47:37.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:37.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:37.968176+0000) 2019-10-13T13:47:38.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:38.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:38.194008+0000) 2019-10-13T13:47:38.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:38.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:38.399952+0000) 2019-10-13T13:47:38.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:38.727+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:38.732578+0000) 2019-10-13T13:47:39.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:39.739+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.745462+0000) 2019-10-13T13:47:39.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:39.743+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.745839+0000) 2019-10-13T13:47:39.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:39.743+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.746305+0000) 2019-10-13T13:47:39.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:39.743+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.747116+0000) 2019-10-13T13:47:39.751 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:39.743+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.749318+0000) 2019-10-13T13:47:39.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:39.744+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.751657+0000) 2019-10-13T13:47:39.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:39.747+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.752361+0000) 2019-10-13T13:47:39.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:39.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:39.758 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:39.752+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:39.756165+0000) 2019-10-13T13:47:41.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:41.984+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:41.985083+0000) 2019-10-13T13:47:44.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:44.307+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.312522+0000) 2019-10-13T13:47:44.600 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:44.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.597906+0000) 2019-10-13T13:47:44.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:44.755+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.758529+0000) 2019-10-13T13:47:44.761 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:44.755+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.758933+0000) 2019-10-13T13:47:44.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:44.755+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.759564+0000) 2019-10-13T13:47:44.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:44.755+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.759993+0000) 2019-10-13T13:47:44.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:44.755+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.760222+0000) 2019-10-13T13:47:44.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:44.756+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.761024+0000) 2019-10-13T13:47:44.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:44.755+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.761290+0000) 2019-10-13T13:47:44.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:44.760+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.767563+0000) 2019-10-13T13:47:44.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:44.763+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:44.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:44.959+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:44.965108+0000) 2019-10-13T13:47:47.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:47.963+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:47.968435+0000) 2019-10-13T13:47:48.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:48.191+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:48.194208+0000) 2019-10-13T13:47:48.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:48.404+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:48.406603+0000) 2019-10-13T13:47:48.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:48.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:48.732811+0000) 2019-10-13T13:47:49.774 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:49.767+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.770964+0000) 2019-10-13T13:47:49.774 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:49.767+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.771144+0000) 2019-10-13T13:47:49.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:49.767+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.772037+0000) 2019-10-13T13:47:49.775 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:49.767+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.772522+0000) 2019-10-13T13:47:49.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:49.768+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.773626+0000) 2019-10-13T13:47:49.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:49.767+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.773042+0000) 2019-10-13T13:47:49.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:49.771+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.774154+0000) 2019-10-13T13:47:49.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:49.776+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:49.781209+0000) 2019-10-13T13:47:49.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:49.775+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:51.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:51.980+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:51.985319+0000) 2019-10-13T13:47:54.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:54.310+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.312677+0000) 2019-10-13T13:47:54.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:54.594+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.598138+0000) 2019-10-13T13:47:54.787 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:54.778+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.784614+0000) 2019-10-13T13:47:54.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:54.778+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.785171+0000) 2019-10-13T13:47:54.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:54.782+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.786786+0000) 2019-10-13T13:47:54.790 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:54.782+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.787195+0000) 2019-10-13T13:47:54.790 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:54.782+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.787993+0000) 2019-10-13T13:47:54.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:54.782+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.788949+0000) 2019-10-13T13:47:54.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:54.784+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.789404+0000) 2019-10-13T13:47:54.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:54.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190477 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:47:54.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:54.788+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.795028+0000) 2019-10-13T13:47:54.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:54.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:54.965294+0000) 2019-10-13T13:47:57.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:57.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:57.968631+0000) 2019-10-13T13:47:58.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:58.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:58.194401+0000) 2019-10-13T13:47:58.398 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:47:58.399 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:58.402 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:47:58.402 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:47:58.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:58.408+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:58.413294+0000) 2019-10-13T13:47:58.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:58.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:58.733008+0000) 2019-10-13T13:47:59.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:47:59.790+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.796998+0000) 2019-10-13T13:47:59.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:47:59.794+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.798078+0000) 2019-10-13T13:47:59.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:47:59.792+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.798404+0000) 2019-10-13T13:47:59.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:47:59.794+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.798728+0000) 2019-10-13T13:47:59.802 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:47:59.794+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.799213+0000) 2019-10-13T13:47:59.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:47:59.794+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.799323+0000) 2019-10-13T13:47:59.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:47:59.794+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.799896+0000) 2019-10-13T13:47:59.809 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:47:59.800+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:47:59.806730+0000) 2019-10-13T13:47:59.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:47:59.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190509 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:01.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:01.984+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:01.985467+0000) 2019-10-13T13:48:04.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:04.310+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.312823+0000) 2019-10-13T13:48:04.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:04.594+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.598359+0000) 2019-10-13T13:48:04.814 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:04.806+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.811162+0000) 2019-10-13T13:48:04.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:04.806+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.811734+0000) 2019-10-13T13:48:04.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:04.806+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.812220+0000) 2019-10-13T13:48:04.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:04.806+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.812582+0000) 2019-10-13T13:48:04.817 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:04.808+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.815150+0000) 2019-10-13T13:48:04.819 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:04.810+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.816825+0000) 2019-10-13T13:48:04.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:04.814+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.818246+0000) 2019-10-13T13:48:04.822 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:04.814+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:04.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:04.816+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.821009+0000) 2019-10-13T13:48:04.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:04.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:04.965476+0000) 2019-10-13T13:48:07.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:07.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:07.968845+0000) 2019-10-13T13:48:08.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:08.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:08.194604+0000) 2019-10-13T13:48:08.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:08.420+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:08.420541+0000) 2019-10-13T13:48:08.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:08.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:08.733235+0000) 2019-10-13T13:48:09.827 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:09.818+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.825103+0000) 2019-10-13T13:48:09.828 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:09.822+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.825862+0000) 2019-10-13T13:48:09.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:09.822+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.825948+0000) 2019-10-13T13:48:09.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:09.822+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.826734+0000) 2019-10-13T13:48:09.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:09.822+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.827055+0000) 2019-10-13T13:48:09.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:09.824+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.829289+0000) 2019-10-13T13:48:09.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:09.822+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.829653+0000) 2019-10-13T13:48:09.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:09.824+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:09.831175+0000) 2019-10-13T13:48:09.836 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:09.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:11.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:11.984+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:11.985660+0000) 2019-10-13T13:48:14.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:14.306+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.312999+0000) 2019-10-13T13:48:14.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:14.594+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.598568+0000) 2019-10-13T13:48:14.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:14.830+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.837188+0000) 2019-10-13T13:48:14.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:14.834+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.837778+0000) 2019-10-13T13:48:14.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:14.834+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.838569+0000) 2019-10-13T13:48:14.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:14.834+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.840145+0000) 2019-10-13T13:48:14.844 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:14.838+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.842021+0000) 2019-10-13T13:48:14.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:14.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.842836+0000) 2019-10-13T13:48:14.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:14.838+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.842895+0000) 2019-10-13T13:48:14.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:14.842+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190599 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:14.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:14.844+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.848571+0000) 2019-10-13T13:48:14.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:14.958+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:14.965681+0000) 2019-10-13T13:48:17.971 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:17.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:17.969040+0000) 2019-10-13T13:48:18.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:18.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:18.194800+0000) 2019-10-13T13:48:18.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:18.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:18.427192+0000) 2019-10-13T13:48:18.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:18.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:18.733430+0000) 2019-10-13T13:48:19.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:19.846+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.851661+0000) 2019-10-13T13:48:19.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:19.850+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.854034+0000) 2019-10-13T13:48:19.859 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:19.850+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.856294+0000) 2019-10-13T13:48:19.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:19.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.856400+0000) 2019-10-13T13:48:19.861 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:19.854+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.858391+0000) 2019-10-13T13:48:19.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:19.854+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.860103+0000) 2019-10-13T13:48:19.863 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:19.860+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.860641+0000) 2019-10-13T13:48:19.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:19.854+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:19.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:19.854+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:19.861323+0000) 2019-10-13T13:48:21.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:21.984+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:21.985887+0000) 2019-10-13T13:48:24.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:24.306+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.313197+0000) 2019-10-13T13:48:24.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:24.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.598800+0000) 2019-10-13T13:48:24.867 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:24.858+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.864160+0000) 2019-10-13T13:48:24.868 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:24.858+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.865465+0000) 2019-10-13T13:48:24.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:24.862+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.866024+0000) 2019-10-13T13:48:24.870 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:24.862+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.867604+0000) 2019-10-13T13:48:24.870 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:24.862+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.868060+0000) 2019-10-13T13:48:24.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:24.862+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.869336+0000) 2019-10-13T13:48:24.872 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:24.867+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.870193+0000) 2019-10-13T13:48:24.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:24.867+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.871826+0000) 2019-10-13T13:48:24.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:24.870+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:24.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:24.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:24.965854+0000) 2019-10-13T13:48:27.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:27.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:27.969244+0000) 2019-10-13T13:48:28.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:28.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.195017+0000) 2019-10-13T13:48:28.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:28.431+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.433262+0000) 2019-10-13T13:48:28.520 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:48:28.520 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:28.524 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:48:28.524 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:28.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:28.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:28.733693+0000) 2019-10-13T13:48:29.880 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:29.870+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.877122+0000) 2019-10-13T13:48:29.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:29.874+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.878137+0000) 2019-10-13T13:48:29.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:29.874+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.879081+0000) 2019-10-13T13:48:29.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:29.874+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.879352+0000) 2019-10-13T13:48:29.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:29.879+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.880714+0000) 2019-10-13T13:48:29.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:29.874+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.879683+0000) 2019-10-13T13:48:29.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:29.874+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.880414+0000) 2019-10-13T13:48:29.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:29.882+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:29.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:29.883+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:29.887211+0000) 2019-10-13T13:48:31.988 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:31.983+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:31.986099+0000) 2019-10-13T13:48:34.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:34.310+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.313397+0000) 2019-10-13T13:48:34.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:34.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.598993+0000) 2019-10-13T13:48:34.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:34.886+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.891192+0000) 2019-10-13T13:48:34.894 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:34.886+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.891433+0000) 2019-10-13T13:48:34.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:34.886+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.893195+0000) 2019-10-13T13:48:34.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:34.886+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.893516+0000) 2019-10-13T13:48:34.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:34.890+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.894968+0000) 2019-10-13T13:48:34.898 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:34.890+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.895349+0000) 2019-10-13T13:48:34.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:34.895+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.896485+0000) 2019-10-13T13:48:34.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:34.894+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:34.905 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:34.899+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.902741+0000) 2019-10-13T13:48:34.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:34.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:34.966052+0000) 2019-10-13T13:48:37.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:37.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:37.969440+0000) 2019-10-13T13:48:38.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:38.190+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.195249+0000) 2019-10-13T13:48:38.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:38.443+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.444666+0000) 2019-10-13T13:48:38.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:38.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:38.733875+0000) 2019-10-13T13:48:39.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:39.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.903433+0000) 2019-10-13T13:48:39.906 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:39.898+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.903776+0000) 2019-10-13T13:48:39.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:39.898+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.904107+0000) 2019-10-13T13:48:39.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:39.902+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.906183+0000) 2019-10-13T13:48:39.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:39.902+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.906348+0000) 2019-10-13T13:48:39.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:39.902+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.906738+0000) 2019-10-13T13:48:39.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:39.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.908422+0000) 2019-10-13T13:48:39.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:39.907+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:39.909276+0000) 2019-10-13T13:48:39.914 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:39.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:41.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:41.983+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:41.986321+0000) 2019-10-13T13:48:44.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:44.310+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.313607+0000) 2019-10-13T13:48:44.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:44.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.599212+0000) 2019-10-13T13:48:44.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:44.910+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.915240+0000) 2019-10-13T13:48:44.918 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:44.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.915857+0000) 2019-10-13T13:48:44.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:44.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.916362+0000) 2019-10-13T13:48:44.919 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:44.910+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.916574+0000) 2019-10-13T13:48:44.920 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:44.914+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.917985+0000) 2019-10-13T13:48:44.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:44.914+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.917731+0000) 2019-10-13T13:48:44.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:44.914+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.919221+0000) 2019-10-13T13:48:44.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:44.919+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.923755+0000) 2019-10-13T13:48:44.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:44.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:44.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:44.962+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:44.966248+0000) 2019-10-13T13:48:47.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:47.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:47.969646+0000) 2019-10-13T13:48:48.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:48.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.195452+0000) 2019-10-13T13:48:48.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:48.455+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.456209+0000) 2019-10-13T13:48:48.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:48.734+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:48.734113+0000) 2019-10-13T13:48:49.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:49.921+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.928844+0000) 2019-10-13T13:48:49.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:49.925+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.930516+0000) 2019-10-13T13:48:49.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:49.927+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.930924+0000) 2019-10-13T13:48:49.934 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:49.925+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.932041+0000) 2019-10-13T13:48:49.936 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:49.925+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.933584+0000) 2019-10-13T13:48:49.938 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:49.929+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.935651+0000) 2019-10-13T13:48:49.938 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:49.929+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.935884+0000) 2019-10-13T13:48:49.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:49.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190815 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:49.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:49.939+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:49.940437+0000) 2019-10-13T13:48:51.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:51.983+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:51.986515+0000) 2019-10-13T13:48:54.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:54.309+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.313804+0000) 2019-10-13T13:48:54.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:54.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.599413+0000) 2019-10-13T13:48:54.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:54.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.942009+0000) 2019-10-13T13:48:54.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:54.937+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.942672+0000) 2019-10-13T13:48:54.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:54.937+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.943330+0000) 2019-10-13T13:48:54.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:54.937+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.944478+0000) 2019-10-13T13:48:54.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:54.937+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.944682+0000) 2019-10-13T13:48:54.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:54.943+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.945944+0000) 2019-10-13T13:48:54.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:54.941+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.946284+0000) 2019-10-13T13:48:54.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:54.947+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.950391+0000) 2019-10-13T13:48:54.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:54.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:54.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:54.961+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:54.966425+0000) 2019-10-13T13:48:57.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:57.969+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:57.969916+0000) 2019-10-13T13:48:58.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:58.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.195654+0000) 2019-10-13T13:48:58.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:58.459+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.462748+0000) 2019-10-13T13:48:58.634 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:48:58.635 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:58.638 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:48:58.638 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:48:58.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:58.733+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:58.734342+0000) 2019-10-13T13:48:59.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:48:59.949+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.955130+0000) 2019-10-13T13:48:59.959 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:48:59.949+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.956808+0000) 2019-10-13T13:48:59.960 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:48:59.949+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.957692+0000) 2019-10-13T13:48:59.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:48:59.953+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.958745+0000) 2019-10-13T13:48:59.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:48:59.953+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.959095+0000) 2019-10-13T13:48:59.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:48:59.953+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.959671+0000) 2019-10-13T13:48:59.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:48:59.955+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.958709+0000) 2019-10-13T13:48:59.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:48:59.957+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:48:59.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:48:59.963+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:48:59.964119+0000) 2019-10-13T13:49:01.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:01.983+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:01.986754+0000) 2019-10-13T13:49:04.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:04.313+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.314004+0000) 2019-10-13T13:49:04.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:04.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.599610+0000) 2019-10-13T13:49:04.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:04.963+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.966559+0000) 2019-10-13T13:49:04.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:04.965+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.967280+0000) 2019-10-13T13:49:04.971 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:04.965+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.968175+0000) 2019-10-13T13:49:04.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:04.965+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.969538+0000) 2019-10-13T13:49:04.972 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:04.969+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.969813+0000) 2019-10-13T13:49:04.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:04.969+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.970196+0000) 2019-10-13T13:49:04.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:04.969+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.970539+0000) 2019-10-13T13:49:04.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:04.971+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:04.974347+0000) 2019-10-13T13:49:04.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:04.973+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:07.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:07.969+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:07.970144+0000) 2019-10-13T13:49:08.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:08.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.195849+0000) 2019-10-13T13:49:08.475 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:08.471+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.472970+0000) 2019-10-13T13:49:08.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:08.733+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:08.734565+0000) 2019-10-13T13:49:09.983 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:09.977+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.980233+0000) 2019-10-13T13:49:09.983 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:09.977+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.980573+0000) 2019-10-13T13:49:09.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:09.979+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.983276+0000) 2019-10-13T13:49:09.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:09.981+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.984231+0000) 2019-10-13T13:49:09.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:09.983+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.984841+0000) 2019-10-13T13:49:09.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:09.981+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.985136+0000) 2019-10-13T13:49:09.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:09.985+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.985864+0000) 2019-10-13T13:49:09.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:09.985+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:09.987043+0000) 2019-10-13T13:49:09.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:09.985+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190939 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:11.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:11.713+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:11.717118+0000) 2019-10-13T13:49:11.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:11.983+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:11.986909+0000) 2019-10-13T13:49:14.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:14.313+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.314219+0000) 2019-10-13T13:49:14.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:14.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.599847+0000) 2019-10-13T13:49:14.994 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:14.989+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.991851+0000) 2019-10-13T13:49:14.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:14.991+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.992949+0000) 2019-10-13T13:49:14.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:14.993+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.994858+0000) 2019-10-13T13:49:14.998 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:14.993+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.995746+0000) 2019-10-13T13:49:14.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:14.993+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.996103+0000) 2019-10-13T13:49:14.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:14.993+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.996941+0000) 2019-10-13T13:49:15.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:14.993+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:14.997528+0000) 2019-10-13T13:49:15.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:14.999+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:15.001831+0000) 2019-10-13T13:49:15.004 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:15.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:17.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:17.969+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:17.970338+0000) 2019-10-13T13:49:18.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:18.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.196048+0000) 2019-10-13T13:49:18.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:18.483+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.484445+0000) 2019-10-13T13:49:18.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:18.733+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:18.734797+0000) 2019-10-13T13:49:20.010 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:20.005+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.007594+0000) 2019-10-13T13:49:20.010 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:20.005+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.007785+0000) 2019-10-13T13:49:20.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:20.005+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.008593+0000) 2019-10-13T13:49:20.012 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:20.005+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.009621+0000) 2019-10-13T13:49:20.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:20.007+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.010145+0000) 2019-10-13T13:49:20.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:20.009+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.011873+0000) 2019-10-13T13:49:20.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:20.009+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.012163+0000) 2019-10-13T13:49:20.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:20.013+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:20.022 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:20.015+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:20.019778+0000) 2019-10-13T13:49:21.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:21.713+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:21.717346+0000) 2019-10-13T13:49:21.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:21.987+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:21.987144+0000) 2019-10-13T13:49:24.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:24.313+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:24.314415+0000) 2019-10-13T13:49:24.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:24.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:24.600046+0000) 2019-10-13T13:49:25.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:25.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.019164+0000) 2019-10-13T13:49:25.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:25.017+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.021253+0000) 2019-10-13T13:49:25.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:25.017+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.021243+0000) 2019-10-13T13:49:25.024 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:25.021+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.021877+0000) 2019-10-13T13:49:25.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:25.021+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.022230+0000) 2019-10-13T13:49:25.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:25.021+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.024182+0000) 2019-10-13T13:49:25.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:25.025+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.026594+0000) 2019-10-13T13:49:25.031 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:25.025+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:25.036 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:25.031+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:25.034387+0000) 2019-10-13T13:49:27.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:27.969+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:27.970554+0000) 2019-10-13T13:49:28.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:28.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.196265+0000) 2019-10-13T13:49:28.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:28.490+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.493274+0000) 2019-10-13T13:49:28.737 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:28.733+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:28.734995+0000) 2019-10-13T13:49:28.739 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:49:28.739 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:28.743 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:49:28.743 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:30.034 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:30.029+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.032017+0000) 2019-10-13T13:49:30.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:30.029+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.032931+0000) 2019-10-13T13:49:30.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:30.030+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.035174+0000) 2019-10-13T13:49:30.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:30.033+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.035452+0000) 2019-10-13T13:49:30.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:30.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.037927+0000) 2019-10-13T13:49:30.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:30.037+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.040006+0000) 2019-10-13T13:49:30.044 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:30.041+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.041849+0000) 2019-10-13T13:49:30.045 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:30.041+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:30.052 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:30.046+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:30.049803+0000) 2019-10-13T13:49:31.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:31.717+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:31.717588+0000) 2019-10-13T13:49:31.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:31.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:31.987340+0000) 2019-10-13T13:49:34.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:34.313+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:34.314645+0000) 2019-10-13T13:49:34.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:34.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:34.600267+0000) 2019-10-13T13:49:35.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:35.045+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.046322+0000) 2019-10-13T13:49:35.049 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:35.045+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.046962+0000) 2019-10-13T13:49:35.050 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:35.045+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.047959+0000) 2019-10-13T13:49:35.051 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:35.045+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.048870+0000) 2019-10-13T13:49:35.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:35.045+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.049343+0000) 2019-10-13T13:49:35.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:35.045+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.049664+0000) 2019-10-13T13:49:35.052 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:35.046+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.050289+0000) 2019-10-13T13:49:35.058 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:35.053+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:35.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:35.054+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:35.057878+0000) 2019-10-13T13:49:37.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:37.969+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:37.970725+0000) 2019-10-13T13:49:38.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:38.193+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.196465+0000) 2019-10-13T13:49:38.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:38.498+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.499878+0000) 2019-10-13T13:49:38.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:38.733+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:38.735194+0000) 2019-10-13T13:49:40.062 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:40.057+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.059226+0000) 2019-10-13T13:49:40.063 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:40.057+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.060877+0000) 2019-10-13T13:49:40.064 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:40.057+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.061376+0000) 2019-10-13T13:49:40.064 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:40.057+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.061577+0000) 2019-10-13T13:49:40.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:40.061+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.062202+0000) 2019-10-13T13:49:40.065 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:40.058+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.062702+0000) 2019-10-13T13:49:40.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:40.061+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.063050+0000) 2019-10-13T13:49:40.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:40.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:40.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:40.066+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:40.070181+0000) 2019-10-13T13:49:41.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:41.717+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:41.717794+0000) 2019-10-13T13:49:41.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:41.986+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:41.987533+0000) 2019-10-13T13:49:44.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:44.313+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:44.314847+0000) 2019-10-13T13:49:44.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:44.597+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:44.600465+0000) 2019-10-13T13:49:45.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:45.069+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.071525+0000) 2019-10-13T13:49:45.075 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:45.069+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.072110+0000) 2019-10-13T13:49:45.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:45.070+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.072634+0000) 2019-10-13T13:49:45.076 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:45.069+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.072660+0000) 2019-10-13T13:49:45.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:45.069+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.073278+0000) 2019-10-13T13:49:45.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:45.069+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.073581+0000) 2019-10-13T13:49:45.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:45.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.075345+0000) 2019-10-13T13:49:45.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:45.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:45.085 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:45.078+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:45.082622+0000) 2019-10-13T13:49:47.973 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:47.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:47.970920+0000) 2019-10-13T13:49:48.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:48.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.196678+0000) 2019-10-13T13:49:48.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:48.502+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.506758+0000) 2019-10-13T13:49:48.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:48.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:48.735393+0000) 2019-10-13T13:49:50.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:50.080+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.083961+0000) 2019-10-13T13:49:50.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:50.082+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.084303+0000) 2019-10-13T13:49:50.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:50.080+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.085032+0000) 2019-10-13T13:49:50.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:50.080+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.085281+0000) 2019-10-13T13:49:50.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:50.084+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.086042+0000) 2019-10-13T13:49:50.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:50.084+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.085984+0000) 2019-10-13T13:49:50.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:50.084+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.086382+0000) 2019-10-13T13:49:50.095 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:50.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:50.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:50.090+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:50.094548+0000) 2019-10-13T13:49:51.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:51.716+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:51.718049+0000) 2019-10-13T13:49:51.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:51.986+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:51.987754+0000) 2019-10-13T13:49:54.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:54.312+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:54.315053+0000) 2019-10-13T13:49:54.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:54.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:54.600663+0000) 2019-10-13T13:49:55.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:55.092+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.095478+0000) 2019-10-13T13:49:55.098 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:49:55.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.096366+0000) 2019-10-13T13:49:55.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:49:55.092+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.096988+0000) 2019-10-13T13:49:55.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:49:55.094+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.097891+0000) 2019-10-13T13:49:55.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:55.096+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.098153+0000) 2019-10-13T13:49:55.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:55.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.098785+0000) 2019-10-13T13:49:55.101 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:49:55.096+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.099049+0000) 2019-10-13T13:49:55.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:55.098+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:55.103990+0000) 2019-10-13T13:49:55.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:49:55.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:49:57.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:49:57.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:57.971072+0000) 2019-10-13T13:49:58.199 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:49:58.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.196878+0000) 2019-10-13T13:49:58.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:49:58.510+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.512636+0000) 2019-10-13T13:49:58.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:49:58.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:49:58.735586+0000) 2019-10-13T13:49:58.790 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:49:58.790 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:49:58.794 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:49:58.794 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:00.111 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:00.104+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.108861+0000) 2019-10-13T13:50:00.112 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:00.106+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.110064+0000) 2019-10-13T13:50:00.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:00.104+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.109569+0000) 2019-10-13T13:50:00.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:00.108+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.110980+0000) 2019-10-13T13:50:00.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:00.108+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.111496+0000) 2019-10-13T13:50:00.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:00.108+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.111310+0000) 2019-10-13T13:50:00.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:00.108+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.112283+0000) 2019-10-13T13:50:00.119 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:00.114+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:00.116927+0000) 2019-10-13T13:50:00.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:00.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:01.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:01.716+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:01.718206+0000) 2019-10-13T13:50:01.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:01.982+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:01.987902+0000) 2019-10-13T13:50:04.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:04.312+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:04.315313+0000) 2019-10-13T13:50:04.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:04.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:04.600859+0000) 2019-10-13T13:50:05.125 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:05.118+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.123227+0000) 2019-10-13T13:50:05.126 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:05.120+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.123443+0000) 2019-10-13T13:50:05.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:05.120+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.124222+0000) 2019-10-13T13:50:05.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:05.120+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.125133+0000) 2019-10-13T13:50:05.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:05.120+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.125457+0000) 2019-10-13T13:50:05.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:05.124+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.126718+0000) 2019-10-13T13:50:05.129 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:05.124+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.127217+0000) 2019-10-13T13:50:05.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:05.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:05.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:05.130+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:05.134324+0000) 2019-10-13T13:50:07.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:07.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:07.971271+0000) 2019-10-13T13:50:08.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:08.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.197076+0000) 2019-10-13T13:50:08.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:08.514+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.519447+0000) 2019-10-13T13:50:08.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:08.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:08.735782+0000) 2019-10-13T13:50:10.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:10.132+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.136924+0000) 2019-10-13T13:50:10.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:10.132+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.137276+0000) 2019-10-13T13:50:10.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:10.132+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.137617+0000) 2019-10-13T13:50:10.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:10.136+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.138687+0000) 2019-10-13T13:50:10.141 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:10.136+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.139105+0000) 2019-10-13T13:50:10.142 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:10.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.139858+0000) 2019-10-13T13:50:10.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:10.136+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.140905+0000) 2019-10-13T13:50:10.149 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:10.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:10.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:10.146+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:10.148341+0000) 2019-10-13T13:50:11.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:11.716+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:11.718405+0000) 2019-10-13T13:50:11.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:11.986+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:11.988083+0000) 2019-10-13T13:50:14.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:14.312+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:14.315510+0000) 2019-10-13T13:50:14.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:14.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:14.601055+0000) 2019-10-13T13:50:15.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:15.148+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.150677+0000) 2019-10-13T13:50:15.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:15.146+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.151336+0000) 2019-10-13T13:50:15.154 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:15.148+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.151858+0000) 2019-10-13T13:50:15.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:15.148+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.152196+0000) 2019-10-13T13:50:15.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:15.148+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.152815+0000) 2019-10-13T13:50:15.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:15.148+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.153295+0000) 2019-10-13T13:50:15.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:15.148+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.153652+0000) 2019-10-13T13:50:15.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:15.154+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:15.158090+0000) 2019-10-13T13:50:15.162 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:15.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191340 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:17.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:17.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:17.971475+0000) 2019-10-13T13:50:18.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:18.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.197275+0000) 2019-10-13T13:50:18.533 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:18.526+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.530949+0000) 2019-10-13T13:50:18.738 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:18.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:18.735974+0000) 2019-10-13T13:50:20.165 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:20.160+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.162892+0000) 2019-10-13T13:50:20.166 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:20.160+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.163771+0000) 2019-10-13T13:50:20.167 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:20.160+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.164603+0000) 2019-10-13T13:50:20.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:20.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.165467+0000) 2019-10-13T13:50:20.168 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:20.164+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.166136+0000) 2019-10-13T13:50:20.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:20.164+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.166046+0000) 2019-10-13T13:50:20.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:20.164+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.167299+0000) 2019-10-13T13:50:20.174 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:20.166+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:20.172018+0000) 2019-10-13T13:50:20.174 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:20.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:21.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:21.716+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:21.718609+0000) 2019-10-13T13:50:21.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:21.986+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:21.988309+0000) 2019-10-13T13:50:24.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:24.312+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:24.315745+0000) 2019-10-13T13:50:24.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:24.596+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:24.601250+0000) 2019-10-13T13:50:25.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:25.172+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.175413+0000) 2019-10-13T13:50:25.179 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:25.174+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.177433+0000) 2019-10-13T13:50:25.180 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:25.172+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.177674+0000) 2019-10-13T13:50:25.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:25.176+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.178531+0000) 2019-10-13T13:50:25.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:25.176+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.178830+0000) 2019-10-13T13:50:25.181 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:25.176+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.179199+0000) 2019-10-13T13:50:25.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:25.176+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.179777+0000) 2019-10-13T13:50:25.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:25.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191402 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:25.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:25.182+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:25.187434+0000) 2019-10-13T13:50:27.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:27.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:27.971691+0000) 2019-10-13T13:50:28.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:28.192+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.197495+0000) 2019-10-13T13:50:28.540 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:28.534+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.537474+0000) 2019-10-13T13:50:28.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:28.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:28.736213+0000) 2019-10-13T13:50:28.900 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:50:28.900 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:28.904 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:50:28.904 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:30.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:30.184+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.188460+0000) 2019-10-13T13:50:30.192 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:30.188+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.189931+0000) 2019-10-13T13:50:30.193 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:30.188+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.190400+0000) 2019-10-13T13:50:30.194 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:30.186+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.190952+0000) 2019-10-13T13:50:30.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:30.188+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.191448+0000) 2019-10-13T13:50:30.195 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:30.188+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.191597+0000) 2019-10-13T13:50:30.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:30.188+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.192046+0000) 2019-10-13T13:50:30.199 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:30.190+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:30.196074+0000) 2019-10-13T13:50:30.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:30.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:31.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:31.716+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:31.718812+0000) 2019-10-13T13:50:31.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:31.986+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:31.988502+0000) 2019-10-13T13:50:34.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:34.312+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:34.315944+0000) 2019-10-13T13:50:34.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:34.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:34.601449+0000) 2019-10-13T13:50:35.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:35.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.201656+0000) 2019-10-13T13:50:35.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:35.200+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.202667+0000) 2019-10-13T13:50:35.205 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:35.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.203079+0000) 2019-10-13T13:50:35.206 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:35.200+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.203226+0000) 2019-10-13T13:50:35.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:35.200+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.203698+0000) 2019-10-13T13:50:35.206 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:35.200+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.203969+0000) 2019-10-13T13:50:35.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:35.200+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.204468+0000) 2019-10-13T13:50:35.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:35.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191463 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:35.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:35.209+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:35.212272+0000) 2019-10-13T13:50:37.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:37.968+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:37.971871+0000) 2019-10-13T13:50:38.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:38.196+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.197694+0000) 2019-10-13T13:50:38.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:38.537+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.543988+0000) 2019-10-13T13:50:38.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:38.732+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:38.736440+0000) 2019-10-13T13:50:40.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:40.208+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.213053+0000) 2019-10-13T13:50:40.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:40.212+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.214514+0000) 2019-10-13T13:50:40.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:40.209+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.215656+0000) 2019-10-13T13:50:40.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:40.212+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.215685+0000) 2019-10-13T13:50:40.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:40.212+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.216020+0000) 2019-10-13T13:50:40.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:40.212+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.216633+0000) 2019-10-13T13:50:40.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:40.212+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.216795+0000) 2019-10-13T13:50:40.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:40.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:40.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:40.221+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:40.224136+0000) 2019-10-13T13:50:41.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:41.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:41.719041+0000) 2019-10-13T13:50:41.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:41.985+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:41.988699+0000) 2019-10-13T13:50:44.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:44.315+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:44.316142+0000) 2019-10-13T13:50:44.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:44.595+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:44.601679+0000) 2019-10-13T13:50:45.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:45.223+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.227696+0000) 2019-10-13T13:50:45.231 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:45.223+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.228239+0000) 2019-10-13T13:50:45.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:45.223+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.229127+0000) 2019-10-13T13:50:45.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:45.225+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.229666+0000) 2019-10-13T13:50:45.232 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:45.223+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.229625+0000) 2019-10-13T13:50:45.233 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:45.227+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.230017+0000) 2019-10-13T13:50:45.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:45.227+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.230602+0000) 2019-10-13T13:50:45.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:45.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:45.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:45.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:45.236907+0000) 2019-10-13T13:50:47.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:47.971+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:47.972126+0000) 2019-10-13T13:50:48.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:48.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.197931+0000) 2019-10-13T13:50:48.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:48.545+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.550601+0000) 2019-10-13T13:50:48.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:48.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:48.736669+0000) 2019-10-13T13:50:50.240 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:50.231+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.237422+0000) 2019-10-13T13:50:50.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:50.235+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.238445+0000) 2019-10-13T13:50:50.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:50.233+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.239717+0000) 2019-10-13T13:50:50.242 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:50.235+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.239775+0000) 2019-10-13T13:50:50.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:50.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.240420+0000) 2019-10-13T13:50:50.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:50.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.240570+0000) 2019-10-13T13:50:50.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:50.235+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.241200+0000) 2019-10-13T13:50:50.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:50.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:50.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:50.245+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:50.248132+0000) 2019-10-13T13:50:51.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:51.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:51.719232+0000) 2019-10-13T13:50:51.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:51.985+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:51.988904+0000) 2019-10-13T13:50:54.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:54.315+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:54.316349+0000) 2019-10-13T13:50:54.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:54.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:54.601932+0000) 2019-10-13T13:50:55.253 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:50:55.245+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.251075+0000) 2019-10-13T13:50:55.254 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:50:55.247+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.251274+0000) 2019-10-13T13:50:55.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:50:55.247+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.251748+0000) 2019-10-13T13:50:55.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:55.247+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.251828+0000) 2019-10-13T13:50:55.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:55.247+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.252658+0000) 2019-10-13T13:50:55.256 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:50:55.247+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.253306+0000) 2019-10-13T13:50:55.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:55.247+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.253359+0000) 2019-10-13T13:50:55.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:55.253+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:55.258993+0000) 2019-10-13T13:50:55.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:50:55.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191587 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:50:57.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:50:57.971+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:57.972290+0000) 2019-10-13T13:50:58.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:50:58.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.198128+0000) 2019-10-13T13:50:58.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:50:58.553+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.557125+0000) 2019-10-13T13:50:58.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:50:58.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:50:58.736862+0000) 2019-10-13T13:50:58.997 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:50:58.998 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:50:59.001 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:50:59.002 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:00.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:00.259+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.263768+0000) 2019-10-13T13:51:00.267 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:00.259+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.264178+0000) 2019-10-13T13:51:00.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:00.259+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.264679+0000) 2019-10-13T13:51:00.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:00.261+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.265192+0000) 2019-10-13T13:51:00.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:00.259+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.265254+0000) 2019-10-13T13:51:00.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:00.259+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.265655+0000) 2019-10-13T13:51:00.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:00.263+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.266380+0000) 2019-10-13T13:51:00.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:00.265+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:00.269735+0000) 2019-10-13T13:51:00.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:00.267+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:01.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:01.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:01.719452+0000) 2019-10-13T13:51:01.991 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:01.985+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:01.989098+0000) 2019-10-13T13:51:04.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:04.315+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:04.316564+0000) 2019-10-13T13:51:04.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:04.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:04.602135+0000) 2019-10-13T13:51:05.279 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:05.271+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.276843+0000) 2019-10-13T13:51:05.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:05.271+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.277237+0000) 2019-10-13T13:51:05.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:05.275+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.277727+0000) 2019-10-13T13:51:05.280 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:05.275+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.277953+0000) 2019-10-13T13:51:05.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:05.275+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.279938+0000) 2019-10-13T13:51:05.283 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:05.277+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.280725+0000) 2019-10-13T13:51:05.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:05.275+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.281636+0000) 2019-10-13T13:51:05.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:05.277+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:05.282676+0000) 2019-10-13T13:51:05.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:05.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:07.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:07.971+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:07.972492+0000) 2019-10-13T13:51:08.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:08.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.198322+0000) 2019-10-13T13:51:08.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:08.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.563675+0000) 2019-10-13T13:51:08.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:08.731+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:08.737080+0000) 2019-10-13T13:51:10.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:10.283+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.289689+0000) 2019-10-13T13:51:10.293 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:10.287+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.290325+0000) 2019-10-13T13:51:10.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:10.287+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.290647+0000) 2019-10-13T13:51:10.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:10.287+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.291296+0000) 2019-10-13T13:51:10.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:10.285+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.291727+0000) 2019-10-13T13:51:10.295 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:10.287+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.291956+0000) 2019-10-13T13:51:10.295 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:10.287+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.292862+0000) 2019-10-13T13:51:10.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:10.295+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:10.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:10.297+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:10.300327+0000) 2019-10-13T13:51:11.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:11.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:11.719651+0000) 2019-10-13T13:51:11.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:11.985+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:11.989248+0000) 2019-10-13T13:51:14.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:14.311+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:14.316761+0000) 2019-10-13T13:51:14.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:14.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:14.602336+0000) 2019-10-13T13:51:15.305 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:15.299+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.302740+0000) 2019-10-13T13:51:15.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:15.299+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.303387+0000) 2019-10-13T13:51:15.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:15.299+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.304422+0000) 2019-10-13T13:51:15.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:15.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.305251+0000) 2019-10-13T13:51:15.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:15.303+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.305903+0000) 2019-10-13T13:51:15.309 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:15.301+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.306644+0000) 2019-10-13T13:51:15.309 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:15.303+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.307219+0000) 2019-10-13T13:51:15.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:15.305+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:15.310234+0000) 2019-10-13T13:51:15.315 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:15.307+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:17.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:17.967+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:17.972643+0000) 2019-10-13T13:51:18.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:18.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.198543+0000) 2019-10-13T13:51:18.572 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:18.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.570177+0000) 2019-10-13T13:51:18.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:18.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:18.737229+0000) 2019-10-13T13:51:20.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:20.311+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.316622+0000) 2019-10-13T13:51:20.319 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:20.311+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.317127+0000) 2019-10-13T13:51:20.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:20.315+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.318149+0000) 2019-10-13T13:51:20.321 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:20.315+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.318538+0000) 2019-10-13T13:51:20.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:20.315+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.318779+0000) 2019-10-13T13:51:20.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:20.313+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.319433+0000) 2019-10-13T13:51:20.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:20.315+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.319563+0000) 2019-10-13T13:51:20.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:20.317+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:20.322786+0000) 2019-10-13T13:51:20.328 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:20.319+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191743 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:21.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:21.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:21.719845+0000) 2019-10-13T13:51:21.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:21.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:21.989441+0000) 2019-10-13T13:51:24.319 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:24.311+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:24.316975+0000) 2019-10-13T13:51:24.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:24.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:24.602530+0000) 2019-10-13T13:51:25.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:25.323+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.327844+0000) 2019-10-13T13:51:25.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:25.323+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.329515+0000) 2019-10-13T13:51:25.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:25.327+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.330275+0000) 2019-10-13T13:51:25.333 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:25.327+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.330635+0000) 2019-10-13T13:51:25.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:25.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.333833+0000) 2019-10-13T13:51:25.336 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:25.327+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.333359+0000) 2019-10-13T13:51:25.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:25.331+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.334471+0000) 2019-10-13T13:51:25.341 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:25.335+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191776 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:25.342 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:25.333+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:25.339788+0000) 2019-10-13T13:51:27.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:27.967+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:27.972852+0000) 2019-10-13T13:51:28.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:28.195+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.198773+0000) 2019-10-13T13:51:28.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:28.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.576718+0000) 2019-10-13T13:51:28.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:28.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:28.737424+0000) 2019-10-13T13:51:29.193 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:51:29.193 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:29.197 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:51:29.197 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:30.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:30.339+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.342598+0000) 2019-10-13T13:51:30.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:30.339+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.343231+0000) 2019-10-13T13:51:30.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:30.339+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.343806+0000) 2019-10-13T13:51:30.346 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:30.341+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.344207+0000) 2019-10-13T13:51:30.348 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:30.339+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.345403+0000) 2019-10-13T13:51:30.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:30.339+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.345684+0000) 2019-10-13T13:51:30.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:30.339+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.345271+0000) 2019-10-13T13:51:30.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:30.347+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:30.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:30.349+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:30.352478+0000) 2019-10-13T13:51:31.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:31.715+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:31.720059+0000) 2019-10-13T13:51:31.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:31.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:31.989627+0000) 2019-10-13T13:51:34.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:34.315+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:34.317217+0000) 2019-10-13T13:51:34.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:34.599+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:34.602739+0000) 2019-10-13T13:51:35.357 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:35.351+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.354867+0000) 2019-10-13T13:51:35.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:35.351+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.355295+0000) 2019-10-13T13:51:35.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:35.351+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.357454+0000) 2019-10-13T13:51:35.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:35.353+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.358592+0000) 2019-10-13T13:51:35.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:35.355+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.358664+0000) 2019-10-13T13:51:35.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:35.355+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.359518+0000) 2019-10-13T13:51:35.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:35.355+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.359942+0000) 2019-10-13T13:51:35.366 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:35.359+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:35.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:35.361+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:35.367660+0000) 2019-10-13T13:51:37.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:37.970+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:37.973054+0000) 2019-10-13T13:51:38.201 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:38.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.198990+0000) 2019-10-13T13:51:38.586 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:38.580+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.583303+0000) 2019-10-13T13:51:38.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:38.730+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:38.737617+0000) 2019-10-13T13:51:40.369 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:40.362+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.366237+0000) 2019-10-13T13:51:40.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:40.362+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.366950+0000) 2019-10-13T13:51:40.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:40.362+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.368449+0000) 2019-10-13T13:51:40.371 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:40.362+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.368903+0000) 2019-10-13T13:51:40.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:40.362+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.369508+0000) 2019-10-13T13:51:40.374 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:40.364+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.371581+0000) 2019-10-13T13:51:40.374 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:40.366+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.371606+0000) 2019-10-13T13:51:40.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:40.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:40.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:40.376+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:40.383658+0000) 2019-10-13T13:51:41.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:41.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:41.720258+0000) 2019-10-13T13:51:41.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:41.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:41.989858+0000) 2019-10-13T13:51:44.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:44.314+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:44.317456+0000) 2019-10-13T13:51:44.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:44.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:44.602898+0000) 2019-10-13T13:51:45.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:45.374+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.380232+0000) 2019-10-13T13:51:45.383 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:45.374+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.380928+0000) 2019-10-13T13:51:45.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:45.374+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.381294+0000) 2019-10-13T13:51:45.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:45.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.381968+0000) 2019-10-13T13:51:45.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:45.378+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.382286+0000) 2019-10-13T13:51:45.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:45.376+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.383112+0000) 2019-10-13T13:51:45.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:45.378+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.383663+0000) 2019-10-13T13:51:45.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:45.384+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:45.388128+0000) 2019-10-13T13:51:45.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:45.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:47.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:47.966+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:47.973271+0000) 2019-10-13T13:51:48.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:48.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.199199+0000) 2019-10-13T13:51:48.597 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:48.588+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.594647+0000) 2019-10-13T13:51:48.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:48.734+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:48.737807+0000) 2019-10-13T13:51:50.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:50.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.393483+0000) 2019-10-13T13:51:50.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:50.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.393783+0000) 2019-10-13T13:51:50.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:50.388+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.394161+0000) 2019-10-13T13:51:50.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:50.390+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.394952+0000) 2019-10-13T13:51:50.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:50.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.395252+0000) 2019-10-13T13:51:50.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:50.390+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.395776+0000) 2019-10-13T13:51:50.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:50.390+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.396689+0000) 2019-10-13T13:51:50.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:50.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:50.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:50.396+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:50.403192+0000) 2019-10-13T13:51:51.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:51.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:51.720479+0000) 2019-10-13T13:51:51.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:51.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:51.990043+0000) 2019-10-13T13:51:54.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:54.314+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:54.317676+0000) 2019-10-13T13:51:54.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:54.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:54.603128+0000) 2019-10-13T13:51:55.409 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:55.402+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.406260+0000) 2019-10-13T13:51:55.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:51:55.402+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.406629+0000) 2019-10-13T13:51:55.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:55.402+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.407424+0000) 2019-10-13T13:51:55.410 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:51:55.400+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.407868+0000) 2019-10-13T13:51:55.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:51:55.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.408682+0000) 2019-10-13T13:51:55.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:55.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.408989+0000) 2019-10-13T13:51:55.412 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:51:55.406+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.409850+0000) 2019-10-13T13:51:55.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:55.408+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:55.413910+0000) 2019-10-13T13:51:55.419 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:51:55.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:51:57.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:51:57.970+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:57.973466+0000) 2019-10-13T13:51:58.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:51:58.194+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.199418+0000) 2019-10-13T13:51:58.604 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:51:58.600+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.601294+0000) 2019-10-13T13:51:58.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:51:58.734+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:51:58.738034+0000) 2019-10-13T13:51:59.333 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:51:59.333 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:51:59.337 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:51:59.337 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:00.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:00.416+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.419354+0000) 2019-10-13T13:52:00.422 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:00.414+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.419914+0000) 2019-10-13T13:52:00.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:00.414+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.420054+0000) 2019-10-13T13:52:00.423 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:00.414+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.420719+0000) 2019-10-13T13:52:00.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:00.414+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.421152+0000) 2019-10-13T13:52:00.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:00.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.421939+0000) 2019-10-13T13:52:00.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:00.418+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.422287+0000) 2019-10-13T13:52:00.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:00.424+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:00.428799+0000) 2019-10-13T13:52:00.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:00.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:01.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:01.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:01.720670+0000) 2019-10-13T13:52:01.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:01.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:01.990243+0000) 2019-10-13T13:52:04.320 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:04.314+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:04.317950+0000) 2019-10-13T13:52:04.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:04.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:04.603319+0000) 2019-10-13T13:52:05.435 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:05.426+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.432861+0000) 2019-10-13T13:52:05.436 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:05.426+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.433341+0000) 2019-10-13T13:52:05.436 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:05.428+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.434109+0000) 2019-10-13T13:52:05.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:05.430+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.434314+0000) 2019-10-13T13:52:05.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:05.430+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.436586+0000) 2019-10-13T13:52:05.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:05.430+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.437040+0000) 2019-10-13T13:52:05.441 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:05.434+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.438502+0000) 2019-10-13T13:52:05.446 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:05.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:05.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:05.440+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:05.447276+0000) 2019-10-13T13:52:07.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:07.970+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:07.973662+0000) 2019-10-13T13:52:08.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:08.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.199630+0000) 2019-10-13T13:52:08.610 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:08.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.607656+0000) 2019-10-13T13:52:08.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:08.734+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:08.738244+0000) 2019-10-13T13:52:10.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:10.442+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.446658+0000) 2019-10-13T13:52:10.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:10.442+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.447157+0000) 2019-10-13T13:52:10.450 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:10.442+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.447893+0000) 2019-10-13T13:52:10.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:10.442+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.448487+0000) 2019-10-13T13:52:10.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:10.442+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.449227+0000) 2019-10-13T13:52:10.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:10.444+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.450133+0000) 2019-10-13T13:52:10.453 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:10.446+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.450371+0000) 2019-10-13T13:52:10.458 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:10.448+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:10.455649+0000) 2019-10-13T13:52:10.459 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:10.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:11.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:11.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:11.720874+0000) 2019-10-13T13:52:11.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:11.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:11.990438+0000) 2019-10-13T13:52:14.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:14.314+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:14.318190+0000) 2019-10-13T13:52:14.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:14.598+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:14.603516+0000) 2019-10-13T13:52:15.462 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:15.454+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.459471+0000) 2019-10-13T13:52:15.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:15.454+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.460472+0000) 2019-10-13T13:52:15.463 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:15.454+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.459867+0000) 2019-10-13T13:52:15.463 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:15.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.461078+0000) 2019-10-13T13:52:15.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:15.456+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.461805+0000) 2019-10-13T13:52:15.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:15.458+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.462414+0000) 2019-10-13T13:52:15.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:15.458+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.463054+0000) 2019-10-13T13:52:15.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:15.464+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:15.468554+0000) 2019-10-13T13:52:15.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:15.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:17.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:17.970+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:17.973862+0000) 2019-10-13T13:52:18.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:18.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.199849+0000) 2019-10-13T13:52:18.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:18.612+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.614271+0000) 2019-10-13T13:52:18.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:18.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:18.738441+0000) 2019-10-13T13:52:20.475 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:20.466+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.472793+0000) 2019-10-13T13:52:20.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:20.466+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.473540+0000) 2019-10-13T13:52:20.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:20.470+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.474023+0000) 2019-10-13T13:52:20.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:20.470+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.474633+0000) 2019-10-13T13:52:20.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:20.470+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.474857+0000) 2019-10-13T13:52:20.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:20.470+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.475640+0000) 2019-10-13T13:52:20.478 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:20.472+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.476388+0000) 2019-10-13T13:52:20.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:20.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:20.488 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:20.480+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:20.485499+0000) 2019-10-13T13:52:21.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:21.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:21.721105+0000) 2019-10-13T13:52:21.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:21.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:21.990626+0000) 2019-10-13T13:52:24.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:24.314+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:24.318404+0000) 2019-10-13T13:52:24.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:24.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:24.603723+0000) 2019-10-13T13:52:25.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:25.482+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.486692+0000) 2019-10-13T13:52:25.489 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:25.482+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.486810+0000) 2019-10-13T13:52:25.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:25.482+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.487649+0000) 2019-10-13T13:52:25.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:25.482+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.487984+0000) 2019-10-13T13:52:25.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:25.482+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.488408+0000) 2019-10-13T13:52:25.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:25.482+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.489025+0000) 2019-10-13T13:52:25.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:25.484+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.489886+0000) 2019-10-13T13:52:25.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:25.488+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:25.493942+0000) 2019-10-13T13:52:25.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:25.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:27.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:27.970+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:27.974061+0000) 2019-10-13T13:52:28.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:28.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.200084+0000) 2019-10-13T13:52:28.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:28.620+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.620745+0000) 2019-10-13T13:52:28.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:28.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:28.738634+0000) 2019-10-13T13:52:29.446 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:52:29.446 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:29.450 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:52:29.450 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:30.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:30.494+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.499517+0000) 2019-10-13T13:52:30.502 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:30.494+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.499869+0000) 2019-10-13T13:52:30.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:30.494+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.500867+0000) 2019-10-13T13:52:30.504 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:30.494+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.501228+0000) 2019-10-13T13:52:30.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:30.496+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.503329+0000) 2019-10-13T13:52:30.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:30.498+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.502891+0000) 2019-10-13T13:52:30.506 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:30.498+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.502563+0000) 2019-10-13T13:52:30.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:30.502+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192185 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:30.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:30.504+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:30.509201+0000) 2019-10-13T13:52:31.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:31.714+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:31.721344+0000) 2019-10-13T13:52:31.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:31.988+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:31.990853+0000) 2019-10-13T13:52:34.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:34.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:34.318605+0000) 2019-10-13T13:52:34.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:34.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:34.603937+0000) 2019-10-13T13:52:35.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:35.505+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.512124+0000) 2019-10-13T13:52:35.515 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:35.505+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.512560+0000) 2019-10-13T13:52:35.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:35.505+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.513379+0000) 2019-10-13T13:52:35.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:35.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.515930+0000) 2019-10-13T13:52:35.519 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:35.516+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.516589+0000) 2019-10-13T13:52:35.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:35.513+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.517722+0000) 2019-10-13T13:52:35.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:35.513+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.519808+0000) 2019-10-13T13:52:35.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:35.520+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:35.522593+0000) 2019-10-13T13:52:35.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:35.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:37.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:37.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:37.974260+0000) 2019-10-13T13:52:38.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:38.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.200282+0000) 2019-10-13T13:52:38.630 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:38.624+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.627273+0000) 2019-10-13T13:52:38.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:38.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:38.738825+0000) 2019-10-13T13:52:40.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:40.521+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.525990+0000) 2019-10-13T13:52:40.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:40.521+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.526831+0000) 2019-10-13T13:52:40.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:40.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.527547+0000) 2019-10-13T13:52:40.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:40.521+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.528215+0000) 2019-10-13T13:52:40.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:40.528+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.528784+0000) 2019-10-13T13:52:40.532 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:40.521+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.529210+0000) 2019-10-13T13:52:40.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:40.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.530342+0000) 2019-10-13T13:52:40.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:40.532+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:40.533130+0000) 2019-10-13T13:52:40.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:40.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192246 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:41.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:41.713+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:41.721562+0000) 2019-10-13T13:52:41.993 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:41.987+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:41.991075+0000) 2019-10-13T13:52:44.321 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:44.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:44.318818+0000) 2019-10-13T13:52:44.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:44.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:44.604083+0000) 2019-10-13T13:52:45.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:45.533+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.539701+0000) 2019-10-13T13:52:45.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:45.533+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.539987+0000) 2019-10-13T13:52:45.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:45.537+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.543594+0000) 2019-10-13T13:52:45.546 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:45.539+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.543772+0000) 2019-10-13T13:52:45.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:45.537+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.543896+0000) 2019-10-13T13:52:45.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:45.541+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.546532+0000) 2019-10-13T13:52:45.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:45.541+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.548439+0000) 2019-10-13T13:52:45.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:45.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:45.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:45.547+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:45.550890+0000) 2019-10-13T13:52:47.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:47.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:47.974462+0000) 2019-10-13T13:52:48.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:48.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.200484+0000) 2019-10-13T13:52:48.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:48.631+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.633853+0000) 2019-10-13T13:52:48.741 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:48.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:48.739062+0000) 2019-10-13T13:52:50.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:50.551+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.554252+0000) 2019-10-13T13:52:50.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:50.553+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.554924+0000) 2019-10-13T13:52:50.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:50.553+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.555704+0000) 2019-10-13T13:52:50.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:50.553+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.556364+0000) 2019-10-13T13:52:50.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:50.553+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.556076+0000) 2019-10-13T13:52:50.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:50.557+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.558145+0000) 2019-10-13T13:52:50.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:50.557+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.559160+0000) 2019-10-13T13:52:50.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:50.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:50.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:50.563+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:50.564844+0000) 2019-10-13T13:52:51.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:51.721+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:51.721788+0000) 2019-10-13T13:52:51.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:51.987+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:51.991278+0000) 2019-10-13T13:52:54.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:54.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:54.319017+0000) 2019-10-13T13:52:54.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:54.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:54.604285+0000) 2019-10-13T13:52:55.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:55.565+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.568172+0000) 2019-10-13T13:52:55.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:52:55.565+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.568238+0000) 2019-10-13T13:52:55.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:55.565+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.568689+0000) 2019-10-13T13:52:55.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:52:55.567+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.569608+0000) 2019-10-13T13:52:55.572 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:55.565+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.569067+0000) 2019-10-13T13:52:55.573 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:52:55.569+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.570170+0000) 2019-10-13T13:52:55.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:52:55.569+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.570366+0000) 2019-10-13T13:52:55.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:55.575+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:55.576187+0000) 2019-10-13T13:52:55.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:52:55.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:52:57.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:52:57.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:57.974639+0000) 2019-10-13T13:52:58.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:52:58.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.200688+0000) 2019-10-13T13:52:58.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:52:58.635+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.639814+0000) 2019-10-13T13:52:58.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:52:58.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:52:58.739284+0000) 2019-10-13T13:52:59.512 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:52:59.512 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:52:59.515 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:52:59.515 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:00.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:00.577+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.580060+0000) 2019-10-13T13:53:00.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:00.577+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.580642+0000) 2019-10-13T13:53:00.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:00.579+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.581225+0000) 2019-10-13T13:53:00.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:00.581+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.582115+0000) 2019-10-13T13:53:00.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:00.581+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.582786+0000) 2019-10-13T13:53:00.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:00.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.583046+0000) 2019-10-13T13:53:00.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:00.581+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.583466+0000) 2019-10-13T13:53:00.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:00.587+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:00.589066+0000) 2019-10-13T13:53:00.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:00.585+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:01.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:01.721+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:01.721948+0000) 2019-10-13T13:53:01.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:01.987+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:01.991479+0000) 2019-10-13T13:53:04.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:04.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:04.319216+0000) 2019-10-13T13:53:04.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:04.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:04.604426+0000) 2019-10-13T13:53:05.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:05.589+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.592971+0000) 2019-10-13T13:53:05.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:05.593+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.595096+0000) 2019-10-13T13:53:05.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:05.591+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.595876+0000) 2019-10-13T13:53:05.599 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:05.593+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.596456+0000) 2019-10-13T13:53:05.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:05.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.596899+0000) 2019-10-13T13:53:05.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:05.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.597864+0000) 2019-10-13T13:53:05.600 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:05.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.597668+0000) 2019-10-13T13:53:05.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:05.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192405 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:05.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:05.603+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:05.605701+0000) 2019-10-13T13:53:07.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:07.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:07.974815+0000) 2019-10-13T13:53:08.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:08.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.200890+0000) 2019-10-13T13:53:08.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:08.643+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.646441+0000) 2019-10-13T13:53:08.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:08.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:08.739485+0000) 2019-10-13T13:53:10.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:10.605+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.606389+0000) 2019-10-13T13:53:10.609 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:10.605+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.607072+0000) 2019-10-13T13:53:10.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:10.605+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.607630+0000) 2019-10-13T13:53:10.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:10.605+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.608093+0000) 2019-10-13T13:53:10.611 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:10.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.608444+0000) 2019-10-13T13:53:10.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:10.605+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.608778+0000) 2019-10-13T13:53:10.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:10.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.611121+0000) 2019-10-13T13:53:10.617 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:10.611+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:10.614554+0000) 2019-10-13T13:53:10.618 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:10.613+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:11.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:11.721+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:11.722148+0000) 2019-10-13T13:53:11.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:11.991+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:11.991696+0000) 2019-10-13T13:53:14.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:14.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:14.319442+0000) 2019-10-13T13:53:14.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:14.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:14.604626+0000) 2019-10-13T13:53:15.621 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:15.615+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.618861+0000) 2019-10-13T13:53:15.622 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:15.617+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.619131+0000) 2019-10-13T13:53:15.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:15.617+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.620295+0000) 2019-10-13T13:53:15.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:15.617+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.620688+0000) 2019-10-13T13:53:15.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:15.617+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.621040+0000) 2019-10-13T13:53:15.624 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:15.621+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.621795+0000) 2019-10-13T13:53:15.624 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:15.621+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.622029+0000) 2019-10-13T13:53:15.631 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:15.627+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:15.628880+0000) 2019-10-13T13:53:15.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:15.625+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:17.977 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:17.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:17.974968+0000) 2019-10-13T13:53:18.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:18.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.201120+0000) 2019-10-13T13:53:18.655 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:18.651+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.653050+0000) 2019-10-13T13:53:18.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:18.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:18.739708+0000) 2019-10-13T13:53:20.635 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:20.629+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.632325+0000) 2019-10-13T13:53:20.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:20.633+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.633954+0000) 2019-10-13T13:53:20.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:20.633+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.634375+0000) 2019-10-13T13:53:20.637 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:20.633+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.634214+0000) 2019-10-13T13:53:20.638 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:20.631+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.634277+0000) 2019-10-13T13:53:20.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:20.633+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.634773+0000) 2019-10-13T13:53:20.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:20.633+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.634566+0000) 2019-10-13T13:53:20.643 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:20.639+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:20.640742+0000) 2019-10-13T13:53:20.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:20.637+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192499 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:21.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:21.721+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:21.722345+0000) 2019-10-13T13:53:21.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:21.987+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:21.991904+0000) 2019-10-13T13:53:24.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:24.317+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:24.319640+0000) 2019-10-13T13:53:24.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:24.601+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:24.604821+0000) 2019-10-13T13:53:25.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:25.643+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.644409+0000) 2019-10-13T13:53:25.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:25.641+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.644552+0000) 2019-10-13T13:53:25.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:25.645+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.646646+0000) 2019-10-13T13:53:25.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:25.645+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.646960+0000) 2019-10-13T13:53:25.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:25.645+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.647648+0000) 2019-10-13T13:53:25.650 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:25.645+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.647679+0000) 2019-10-13T13:53:25.651 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:25.645+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.647981+0000) 2019-10-13T13:53:25.655 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:25.651+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:25.652589+0000) 2019-10-13T13:53:25.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:25.649+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192531 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:27.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:27.973+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:27.975124+0000) 2019-10-13T13:53:28.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:28.197+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.201322+0000) 2019-10-13T13:53:28.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:28.663+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.664424+0000) 2019-10-13T13:53:28.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:28.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:28.739901+0000) 2019-10-13T13:53:29.625 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:53:29.625 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:29.629 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:53:29.629 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:30.660 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:30.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.657450+0000) 2019-10-13T13:53:30.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:30.652+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.657645+0000) 2019-10-13T13:53:30.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:30.656+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.660209+0000) 2019-10-13T13:53:30.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:30.656+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.660254+0000) 2019-10-13T13:53:30.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:30.656+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.660856+0000) 2019-10-13T13:53:30.664 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:30.656+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.661519+0000) 2019-10-13T13:53:30.665 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:30.660+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.662159+0000) 2019-10-13T13:53:30.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:30.663+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:30.664510+0000) 2019-10-13T13:53:30.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:30.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:31.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:31.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:31.722589+0000) 2019-10-13T13:53:31.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:31.991+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:31.992139+0000) 2019-10-13T13:53:34.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:34.316+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:34.319846+0000) 2019-10-13T13:53:34.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:34.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:34.605019+0000) 2019-10-13T13:53:35.673 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:35.668+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.670512+0000) 2019-10-13T13:53:35.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:35.667+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.671751+0000) 2019-10-13T13:53:35.674 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:35.668+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.671899+0000) 2019-10-13T13:53:35.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:35.668+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.671894+0000) 2019-10-13T13:53:35.675 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:35.668+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.672697+0000) 2019-10-13T13:53:35.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:35.668+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.673337+0000) 2019-10-13T13:53:35.676 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:35.668+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.673681+0000) 2019-10-13T13:53:35.681 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:35.675+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:35.678754+0000) 2019-10-13T13:53:35.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:35.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:37.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:37.972+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:37.975316+0000) 2019-10-13T13:53:38.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:38.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.201524+0000) 2019-10-13T13:53:38.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:38.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.670967+0000) 2019-10-13T13:53:38.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:38.736+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:38.740097+0000) 2019-10-13T13:53:40.686 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:40.679+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.683649+0000) 2019-10-13T13:53:40.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:40.680+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.684127+0000) 2019-10-13T13:53:40.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:40.680+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.684678+0000) 2019-10-13T13:53:40.689 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:40.684+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.686678+0000) 2019-10-13T13:53:40.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:40.684+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.686851+0000) 2019-10-13T13:53:40.691 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:40.684+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.688793+0000) 2019-10-13T13:53:40.693 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:40.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.690486+0000) 2019-10-13T13:53:40.695 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:40.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192625 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:40.698 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:40.691+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:40.695463+0000) 2019-10-13T13:53:41.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:41.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:41.722741+0000) 2019-10-13T13:53:41.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:41.991+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:41.992360+0000) 2019-10-13T13:53:44.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:44.316+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:44.320049+0000) 2019-10-13T13:53:44.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:44.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:44.605210+0000) 2019-10-13T13:53:45.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:45.694+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.696400+0000) 2019-10-13T13:53:45.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:45.692+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.696888+0000) 2019-10-13T13:53:45.700 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:45.692+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.697363+0000) 2019-10-13T13:53:45.700 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:45.692+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.697695+0000) 2019-10-13T13:53:45.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:45.696+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.698309+0000) 2019-10-13T13:53:45.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:45.696+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.699848+0000) 2019-10-13T13:53:45.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:45.696+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.698762+0000) 2019-10-13T13:53:45.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:45.702+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:45.705075+0000) 2019-10-13T13:53:45.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:45.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:47.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:47.972+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:47.975471+0000) 2019-10-13T13:53:48.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:48.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.201774+0000) 2019-10-13T13:53:48.680 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:48.674+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.677554+0000) 2019-10-13T13:53:48.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:48.736+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:48.740290+0000) 2019-10-13T13:53:50.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:50.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.709082+0000) 2019-10-13T13:53:50.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:50.704+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.709501+0000) 2019-10-13T13:53:50.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:50.706+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.710007+0000) 2019-10-13T13:53:50.713 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:50.708+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.710324+0000) 2019-10-13T13:53:50.714 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:50.708+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.711251+0000) 2019-10-13T13:53:50.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:50.708+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.711576+0000) 2019-10-13T13:53:50.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:50.708+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.712142+0000) 2019-10-13T13:53:50.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:50.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:50.718266+0000) 2019-10-13T13:53:50.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:50.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:51.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:51.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:51.722978+0000) 2019-10-13T13:53:51.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:51.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:51.992576+0000) 2019-10-13T13:53:54.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:54.316+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:54.320253+0000) 2019-10-13T13:53:54.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:54.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:54.605369+0000) 2019-10-13T13:53:55.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:55.720+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.722487+0000) 2019-10-13T13:53:55.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:53:55.718+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.723203+0000) 2019-10-13T13:53:55.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:55.720+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.724081+0000) 2019-10-13T13:53:55.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:55.720+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.724923+0000) 2019-10-13T13:53:55.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:53:55.720+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.725513+0000) 2019-10-13T13:53:55.728 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:53:55.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.725527+0000) 2019-10-13T13:53:55.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:53:55.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.726173+0000) 2019-10-13T13:53:55.732 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:55.726+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:55.730234+0000) 2019-10-13T13:53:55.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:53:55.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:53:57.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:53:57.972+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:57.975715+0000) 2019-10-13T13:53:58.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:53:58.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.201966+0000) 2019-10-13T13:53:58.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:53:58.678+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.684014+0000) 2019-10-13T13:53:58.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:53:58.736+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:53:58.740481+0000) 2019-10-13T13:53:59.759 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:53:59.759 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:53:59.763 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:53:59.763 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:00.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:00.730+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.735540+0000) 2019-10-13T13:54:00.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:00.732+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.735617+0000) 2019-10-13T13:54:00.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:00.732+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.735890+0000) 2019-10-13T13:54:00.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:00.732+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.736248+0000) 2019-10-13T13:54:00.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:00.732+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.737036+0000) 2019-10-13T13:54:00.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:00.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.738230+0000) 2019-10-13T13:54:00.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:00.736+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.738518+0000) 2019-10-13T13:54:00.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:00.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:00.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:00.742+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:00.746105+0000) 2019-10-13T13:54:01.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:01.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:01.723177+0000) 2019-10-13T13:54:01.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:01.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:01.992772+0000) 2019-10-13T13:54:04.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:04.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:04.320462+0000) 2019-10-13T13:54:04.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:04.600+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:04.605602+0000) 2019-10-13T13:54:05.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:05.744+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.747012+0000) 2019-10-13T13:54:05.750 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:05.744+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.747519+0000) 2019-10-13T13:54:05.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:05.744+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.747895+0000) 2019-10-13T13:54:05.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:05.744+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.748703+0000) 2019-10-13T13:54:05.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:05.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.749390+0000) 2019-10-13T13:54:05.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:05.744+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.749440+0000) 2019-10-13T13:54:05.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:05.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.750290+0000) 2019-10-13T13:54:05.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:05.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:05.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:05.754+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:05.756764+0000) 2019-10-13T13:54:07.978 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:07.972+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:07.975914+0000) 2019-10-13T13:54:08.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:08.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.202164+0000) 2019-10-13T13:54:08.693 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:08.686+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.690682+0000) 2019-10-13T13:54:08.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:08.736+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:08.740625+0000) 2019-10-13T13:54:10.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:10.756+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.759655+0000) 2019-10-13T13:54:10.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:10.758+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.760678+0000) 2019-10-13T13:54:10.763 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:10.756+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.761115+0000) 2019-10-13T13:54:10.764 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:10.756+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.761327+0000) 2019-10-13T13:54:10.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:10.760+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.762053+0000) 2019-10-13T13:54:10.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:10.760+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.762292+0000) 2019-10-13T13:54:10.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:10.760+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.762762+0000) 2019-10-13T13:54:10.771 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:10.764+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:10.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:10.766+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:10.770029+0000) 2019-10-13T13:54:11.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:11.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:11.723354+0000) 2019-10-13T13:54:11.995 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:11.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:11.992981+0000) 2019-10-13T13:54:14.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:14.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:14.320688+0000) 2019-10-13T13:54:14.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:14.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:14.605795+0000) 2019-10-13T13:54:15.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:15.768+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.771750+0000) 2019-10-13T13:54:15.775 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:15.768+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.772793+0000) 2019-10-13T13:54:15.776 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:15.768+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.773437+0000) 2019-10-13T13:54:15.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:15.770+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.774014+0000) 2019-10-13T13:54:15.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:15.772+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.774737+0000) 2019-10-13T13:54:15.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:15.772+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.775894+0000) 2019-10-13T13:54:15.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:15.772+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.776783+0000) 2019-10-13T13:54:15.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:15.774+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:15.777893+0000) 2019-10-13T13:54:15.784 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:15.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:17.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:17.972+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:17.976115+0000) 2019-10-13T13:54:18.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:18.200+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.202392+0000) 2019-10-13T13:54:18.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:18.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.697455+0000) 2019-10-13T13:54:18.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:18.736+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:18.740810+0000) 2019-10-13T13:54:20.787 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:20.780+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.784606+0000) 2019-10-13T13:54:20.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:20.784+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.786890+0000) 2019-10-13T13:54:20.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:20.784+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.787186+0000) 2019-10-13T13:54:20.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:20.784+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.788214+0000) 2019-10-13T13:54:20.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:20.784+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.789643+0000) 2019-10-13T13:54:20.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:20.786+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.790388+0000) 2019-10-13T13:54:20.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:20.788+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.791543+0000) 2019-10-13T13:54:20.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:20.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:20.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:20.798+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:20.801006+0000) 2019-10-13T13:54:21.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:21.720+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:21.723532+0000) 2019-10-13T13:54:21.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:21.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:21.993201+0000) 2019-10-13T13:54:24.323 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:24.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:24.320894+0000) 2019-10-13T13:54:24.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:24.604+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:24.605989+0000) 2019-10-13T13:54:25.800 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:25.795+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.797973+0000) 2019-10-13T13:54:25.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:25.794+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.798734+0000) 2019-10-13T13:54:25.802 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:25.795+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.798757+0000) 2019-10-13T13:54:25.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:25.795+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.799143+0000) 2019-10-13T13:54:25.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:25.795+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.800260+0000) 2019-10-13T13:54:25.803 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:25.795+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.801022+0000) 2019-10-13T13:54:25.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:25.795+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.801646+0000) 2019-10-13T13:54:25.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:25.802+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:25.806092+0000) 2019-10-13T13:54:25.810 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:25.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192909 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:27.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:27.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:27.976313+0000) 2019-10-13T13:54:28.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:28.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.202617+0000) 2019-10-13T13:54:28.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:28.706+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.708820+0000) 2019-10-13T13:54:28.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:28.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:28.741043+0000) 2019-10-13T13:54:29.839 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:54:29.839 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:29.843 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:54:29.843 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:30.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:30.807+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.811262+0000) 2019-10-13T13:54:30.815 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:30.807+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.811963+0000) 2019-10-13T13:54:30.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:30.810+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.812768+0000) 2019-10-13T13:54:30.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:30.807+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.813482+0000) 2019-10-13T13:54:30.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:30.811+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.814218+0000) 2019-10-13T13:54:30.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:30.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.814277+0000) 2019-10-13T13:54:30.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:30.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.814493+0000) 2019-10-13T13:54:30.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:30.818+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:30.820819+0000) 2019-10-13T13:54:30.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:30.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:31.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:31.719+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:31.723663+0000) 2019-10-13T13:54:31.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:31.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:31.993393+0000) 2019-10-13T13:54:34.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:34.319+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:34.321093+0000) 2019-10-13T13:54:34.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:34.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:34.606200+0000) 2019-10-13T13:54:35.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:35.819+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.824603+0000) 2019-10-13T13:54:35.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:35.819+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.825060+0000) 2019-10-13T13:54:35.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:35.822+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.826379+0000) 2019-10-13T13:54:35.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:35.823+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.826861+0000) 2019-10-13T13:54:35.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:35.823+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.826691+0000) 2019-10-13T13:54:35.830 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:35.823+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.827505+0000) 2019-10-13T13:54:35.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:35.823+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.827834+0000) 2019-10-13T13:54:35.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:35.830+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:35.832113+0000) 2019-10-13T13:54:35.837 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:35.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:37.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:37.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:37.976520+0000) 2019-10-13T13:54:38.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:38.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.202853+0000) 2019-10-13T13:54:38.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:38.710+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.715343+0000) 2019-10-13T13:54:38.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:38.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:38.741254+0000) 2019-10-13T13:54:40.840 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:40.834+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.837422+0000) 2019-10-13T13:54:40.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:40.835+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.837725+0000) 2019-10-13T13:54:40.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:40.835+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.839231+0000) 2019-10-13T13:54:40.842 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:40.835+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.839888+0000) 2019-10-13T13:54:40.843 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:40.835+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.840731+0000) 2019-10-13T13:54:40.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:40.835+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.841067+0000) 2019-10-13T13:54:40.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:40.839+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.842073+0000) 2019-10-13T13:54:40.850 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:40.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:40.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:40.842+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:40.847977+0000) 2019-10-13T13:54:41.726 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:41.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:41.723841+0000) 2019-10-13T13:54:41.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:41.990+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:41.993589+0000) 2019-10-13T13:54:44.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:44.319+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:44.321247+0000) 2019-10-13T13:54:44.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:44.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:44.606448+0000) 2019-10-13T13:54:45.854 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:45.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.851306+0000) 2019-10-13T13:54:45.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:45.847+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.852343+0000) 2019-10-13T13:54:45.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:45.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.853021+0000) 2019-10-13T13:54:45.855 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:45.847+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.853165+0000) 2019-10-13T13:54:45.856 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:45.851+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.854142+0000) 2019-10-13T13:54:45.857 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:45.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.854765+0000) 2019-10-13T13:54:45.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:45.851+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.854953+0000) 2019-10-13T13:54:45.862 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:45.858+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:45.860356+0000) 2019-10-13T13:54:45.864 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:45.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:47.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:47.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:47.976728+0000) 2019-10-13T13:54:48.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:48.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.203070+0000) 2019-10-13T13:54:48.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:48.718+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.721915+0000) 2019-10-13T13:54:48.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:48.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:48.741439+0000) 2019-10-13T13:54:50.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:50.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.865032+0000) 2019-10-13T13:54:50.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:50.863+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.865940+0000) 2019-10-13T13:54:50.869 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:50.863+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.866872+0000) 2019-10-13T13:54:50.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:50.861+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.867397+0000) 2019-10-13T13:54:50.870 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:50.863+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.867867+0000) 2019-10-13T13:54:50.871 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:50.863+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.868341+0000) 2019-10-13T13:54:50.871 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:50.863+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.868845+0000) 2019-10-13T13:54:50.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:50.865+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:50.871867+0000) 2019-10-13T13:54:50.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:50.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193066 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:51.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:51.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:51.724019+0000) 2019-10-13T13:54:51.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:51.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:51.993830+0000) 2019-10-13T13:54:54.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:54.319+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:54.321444+0000) 2019-10-13T13:54:54.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:54.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:54.606647+0000) 2019-10-13T13:54:55.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:55.875+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.878793+0000) 2019-10-13T13:54:55.882 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:55.875+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.879037+0000) 2019-10-13T13:54:55.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:54:55.875+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.879528+0000) 2019-10-13T13:54:55.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:54:55.875+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.880606+0000) 2019-10-13T13:54:55.883 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:54:55.877+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.881247+0000) 2019-10-13T13:54:55.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:55.875+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.881331+0000) 2019-10-13T13:54:55.884 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:54:55.875+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.881625+0000) 2019-10-13T13:54:55.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:55.881+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:55.886257+0000) 2019-10-13T13:54:55.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:54:55.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:54:57.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:54:57.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:57.976967+0000) 2019-10-13T13:54:58.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:54:58.199+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.203292+0000) 2019-10-13T13:54:58.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:54:58.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.728504+0000) 2019-10-13T13:54:58.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:54:58.735+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:54:58.741631+0000) 2019-10-13T13:54:59.973 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:54:59.974 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:54:59.977 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:54:59.977 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:00.895 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:00.887+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.892127+0000) 2019-10-13T13:55:00.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:00.889+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.892694+0000) 2019-10-13T13:55:00.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:00.887+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.893011+0000) 2019-10-13T13:55:00.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:00.891+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.893842+0000) 2019-10-13T13:55:00.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:00.891+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.894030+0000) 2019-10-13T13:55:00.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:00.891+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.894781+0000) 2019-10-13T13:55:00.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:00.891+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.894633+0000) 2019-10-13T13:55:00.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:00.893+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:00.898149+0000) 2019-10-13T13:55:00.904 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:00.895+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:01.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:01.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:01.724202+0000) 2019-10-13T13:55:01.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:01.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:01.994019+0000) 2019-10-13T13:55:04.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:04.315+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.321643+0000) 2019-10-13T13:55:04.609 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:04.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:04.606859+0000) 2019-10-13T13:55:05.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:05.901+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.904925+0000) 2019-10-13T13:55:05.908 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:05.899+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.905165+0000) 2019-10-13T13:55:05.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:05.903+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.906271+0000) 2019-10-13T13:55:05.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:05.903+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.906844+0000) 2019-10-13T13:55:05.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:05.903+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.907438+0000) 2019-10-13T13:55:05.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:05.903+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.908096+0000) 2019-10-13T13:55:05.911 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:05.903+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.908378+0000) 2019-10-13T13:55:05.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:05.909+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:05.912222+0000) 2019-10-13T13:55:05.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:05.911+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:07.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:07.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:07.977170+0000) 2019-10-13T13:55:08.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:08.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:08.203471+0000) 2019-10-13T13:55:08.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:08.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:08.735264+0000) 2019-10-13T13:55:08.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:08.739+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:08.741847+0000) 2019-10-13T13:55:10.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:10.911+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.917683+0000) 2019-10-13T13:55:10.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:10.915+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.918159+0000) 2019-10-13T13:55:10.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:10.915+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.918501+0000) 2019-10-13T13:55:10.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:10.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.918856+0000) 2019-10-13T13:55:10.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:10.915+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.919012+0000) 2019-10-13T13:55:10.923 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:10.915+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.920819+0000) 2019-10-13T13:55:10.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:10.915+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.921009+0000) 2019-10-13T13:55:10.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:10.923+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193192 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:10.931 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:10.921+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:10.928033+0000) 2019-10-13T13:55:11.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:11.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:11.724396+0000) 2019-10-13T13:55:11.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:11.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:11.994256+0000) 2019-10-13T13:55:14.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:14.319+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.321835+0000) 2019-10-13T13:55:14.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:14.603+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:14.607103+0000) 2019-10-13T13:55:15.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:15.927+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.931158+0000) 2019-10-13T13:55:15.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:15.927+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.931458+0000) 2019-10-13T13:55:15.935 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:15.927+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.931953+0000) 2019-10-13T13:55:15.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:15.927+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.932107+0000) 2019-10-13T13:55:15.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:15.929+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.933082+0000) 2019-10-13T13:55:15.936 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:15.927+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.933152+0000) 2019-10-13T13:55:15.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:15.931+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.934115+0000) 2019-10-13T13:55:15.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:15.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:15.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:15.937+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:15.940730+0000) 2019-10-13T13:55:17.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:17.971+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:17.977321+0000) 2019-10-13T13:55:18.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:18.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:18.203670+0000) 2019-10-13T13:55:18.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:18.737+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:18.741733+0000) 2019-10-13T13:55:18.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:18.739+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:18.742033+0000) 2019-10-13T13:55:20.946 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:20.938+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.943292+0000) 2019-10-13T13:55:20.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:20.938+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.943826+0000) 2019-10-13T13:55:20.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:20.938+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.944268+0000) 2019-10-13T13:55:20.948 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:20.938+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.945215+0000) 2019-10-13T13:55:20.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:20.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.946051+0000) 2019-10-13T13:55:20.949 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:20.941+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.946373+0000) 2019-10-13T13:55:20.949 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:20.942+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.946652+0000) 2019-10-13T13:55:20.954 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:20.945+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:20.951344+0000) 2019-10-13T13:55:20.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:20.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:21.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:21.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:21.724574+0000) 2019-10-13T13:55:21.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:21.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:21.994476+0000) 2019-10-13T13:55:24.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:24.318+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.321986+0000) 2019-10-13T13:55:24.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:24.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:24.607315+0000) 2019-10-13T13:55:25.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:25.950+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.956336+0000) 2019-10-13T13:55:25.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:25.950+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.956728+0000) 2019-10-13T13:55:25.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:25.954+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.958612+0000) 2019-10-13T13:55:25.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:25.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.961222+0000) 2019-10-13T13:55:25.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:25.958+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.962426+0000) 2019-10-13T13:55:25.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:25.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.963277+0000) 2019-10-13T13:55:25.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:25.958+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.964135+0000) 2019-10-13T13:55:25.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:25.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:25.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:25.961+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:25.966293+0000) 2019-10-13T13:55:27.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:27.974+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:27.977512+0000) 2019-10-13T13:55:28.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:28.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:28.203873+0000) 2019-10-13T13:55:28.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:28.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:28.742270+0000) 2019-10-13T13:55:28.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:28.745+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:28.748190+0000) 2019-10-13T13:55:30.056 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:55:30.056 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:30.060 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:55:30.060 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:55:30.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:30.962+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.969294+0000) 2019-10-13T13:55:30.972 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:30.962+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.969674+0000) 2019-10-13T13:55:30.974 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:30.966+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.971629+0000) 2019-10-13T13:55:30.977 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:30.970+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.974631+0000) 2019-10-13T13:55:30.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:30.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.976124+0000) 2019-10-13T13:55:30.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:30.973+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.976951+0000) 2019-10-13T13:55:30.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:30.970+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.977446+0000) 2019-10-13T13:55:30.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:30.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193318 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:30.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:30.977+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:30.980712+0000) 2019-10-13T13:55:31.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:31.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:31.724746+0000) 2019-10-13T13:55:31.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:31.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:31.994670+0000) 2019-10-13T13:55:34.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:34.318+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.322188+0000) 2019-10-13T13:55:34.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:34.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:34.607515+0000) 2019-10-13T13:55:35.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:35.978+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.981998+0000) 2019-10-13T13:55:35.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:35.978+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.982691+0000) 2019-10-13T13:55:35.987 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:35.978+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.984452+0000) 2019-10-13T13:55:35.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:35.982+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.986524+0000) 2019-10-13T13:55:35.990 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:35.982+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.987457+0000) 2019-10-13T13:55:35.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:35.981+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.987712+0000) 2019-10-13T13:55:35.991 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:35.982+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.988293+0000) 2019-10-13T13:55:35.994 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:35.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193348 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:35.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:35.989+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:35.992612+0000) 2019-10-13T13:55:37.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:37.974+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:37.977708+0000) 2019-10-13T13:55:38.206 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:38.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:38.204027+0000) 2019-10-13T13:55:38.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:38.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:38.742486+0000) 2019-10-13T13:55:38.757 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:38.749+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:38.754594+0000) 2019-10-13T13:55:40.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:40.990+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.993946+0000) 2019-10-13T13:55:40.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:40.989+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.994830+0000) 2019-10-13T13:55:40.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:40.990+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.996372+0000) 2019-10-13T13:55:40.999 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:40.990+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.996785+0000) 2019-10-13T13:55:40.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:40.990+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.997320+0000) 2019-10-13T13:55:41.000 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:40.994+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.997809+0000) 2019-10-13T13:55:41.000 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:40.994+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:40.998082+0000) 2019-10-13T13:55:41.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:40.997+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:41.001914+0000) 2019-10-13T13:55:41.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:40.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193380 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:41.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:41.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:41.724930+0000) 2019-10-13T13:55:41.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:41.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:41.994865+0000) 2019-10-13T13:55:44.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:44.318+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.322400+0000) 2019-10-13T13:55:44.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:44.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:44.607744+0000) 2019-10-13T13:55:46.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:46.002+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.006760+0000) 2019-10-13T13:55:46.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:46.002+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.008016+0000) 2019-10-13T13:55:46.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:46.002+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.008383+0000) 2019-10-13T13:55:46.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:46.005+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.008824+0000) 2019-10-13T13:55:46.012 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:46.002+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.009621+0000) 2019-10-13T13:55:46.013 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:46.006+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.010371+0000) 2019-10-13T13:55:46.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:46.006+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.011024+0000) 2019-10-13T13:55:46.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:46.013+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:46.016791+0000) 2019-10-13T13:55:46.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:46.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193412 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:47.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:47.974+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:47.977941+0000) 2019-10-13T13:55:48.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:48.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:48.204221+0000) 2019-10-13T13:55:48.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:48.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:48.742694+0000) 2019-10-13T13:55:48.764 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:48.757+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:48.761133+0000) 2019-10-13T13:55:51.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:51.014+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.020428+0000) 2019-10-13T13:55:51.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:51.017+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.021049+0000) 2019-10-13T13:55:51.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:51.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.021531+0000) 2019-10-13T13:55:51.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:51.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.022152+0000) 2019-10-13T13:55:51.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:51.018+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.022521+0000) 2019-10-13T13:55:51.026 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:51.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.023156+0000) 2019-10-13T13:55:51.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:51.018+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.023491+0000) 2019-10-13T13:55:51.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:51.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:51.034 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:51.025+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.031250+0000) 2019-10-13T13:55:51.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:51.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.725104+0000) 2019-10-13T13:55:51.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:51.989+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:51.995055+0000) 2019-10-13T13:55:54.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:54.322+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.322603+0000) 2019-10-13T13:55:54.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:54.602+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:54.607953+0000) 2019-10-13T13:55:56.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:56.030+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.034464+0000) 2019-10-13T13:55:56.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:55:56.032+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.037448+0000) 2019-10-13T13:55:56.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:55:56.030+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.037681+0000) 2019-10-13T13:55:56.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:55:56.034+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.038114+0000) 2019-10-13T13:55:56.043 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:56.034+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.040525+0000) 2019-10-13T13:55:56.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:56.038+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.042531+0000) 2019-10-13T13:55:56.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:55:56.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.043191+0000) 2019-10-13T13:55:56.046 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:55:56.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:55:56.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:56.040+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:56.045478+0000) 2019-10-13T13:55:57.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:55:57.974+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:57.978135+0000) 2019-10-13T13:55:58.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:55:58.198+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:58.204420+0000) 2019-10-13T13:55:58.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:55:58.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:58.742885+0000) 2019-10-13T13:55:58.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:55:58.768+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:55:58.772147+0000) 2019-10-13T13:56:00.190 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:56:00.190 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:00.193 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:56:00.194 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:01.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:01.042+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.047207+0000) 2019-10-13T13:56:01.050 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:01.042+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.047547+0000) 2019-10-13T13:56:01.052 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:01.042+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.048876+0000) 2019-10-13T13:56:01.052 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:01.044+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.049283+0000) 2019-10-13T13:56:01.052 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:01.046+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.050188+0000) 2019-10-13T13:56:01.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:01.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.050579+0000) 2019-10-13T13:56:01.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:01.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.050695+0000) 2019-10-13T13:56:01.057 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:01.048+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.055123+0000) 2019-10-13T13:56:01.059 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:01.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:01.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:01.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.725242+0000) 2019-10-13T13:56:01.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:01.992+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:01.995235+0000) 2019-10-13T13:56:04.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:04.318+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.322803+0000) 2019-10-13T13:56:04.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:04.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:04.608179+0000) 2019-10-13T13:56:06.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:06.054+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.059583+0000) 2019-10-13T13:56:06.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:06.054+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.059926+0000) 2019-10-13T13:56:06.063 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:06.056+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.060420+0000) 2019-10-13T13:56:06.063 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:06.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.060676+0000) 2019-10-13T13:56:06.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:06.054+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.060872+0000) 2019-10-13T13:56:06.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:06.058+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.062293+0000) 2019-10-13T13:56:06.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:06.058+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.062577+0000) 2019-10-13T13:56:06.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:06.060+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:06.067882+0000) 2019-10-13T13:56:06.073 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:06.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:07.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:07.978+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:07.978331+0000) 2019-10-13T13:56:08.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:08.202+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:08.204648+0000) 2019-10-13T13:56:08.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:08.738+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:08.743111+0000) 2019-10-13T13:56:08.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:08.776+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:08.777993+0000) 2019-10-13T13:56:11.077 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:11.070+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.074821+0000) 2019-10-13T13:56:11.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:11.070+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.075663+0000) 2019-10-13T13:56:11.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:11.070+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.075963+0000) 2019-10-13T13:56:11.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:11.070+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.076268+0000) 2019-10-13T13:56:11.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:11.070+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.077017+0000) 2019-10-13T13:56:11.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:11.072+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.077564+0000) 2019-10-13T13:56:11.080 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:11.074+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.077794+0000) 2019-10-13T13:56:11.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:11.078+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193569 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:11.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:11.080+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.084490+0000) 2019-10-13T13:56:11.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:11.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.725374+0000) 2019-10-13T13:56:11.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:11.992+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:11.995420+0000) 2019-10-13T13:56:14.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:14.322+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.323004+0000) 2019-10-13T13:56:14.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:14.606+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:14.608382+0000) 2019-10-13T13:56:16.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:16.082+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.088907+0000) 2019-10-13T13:56:16.092 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:16.082+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.089590+0000) 2019-10-13T13:56:16.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:16.086+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.090057+0000) 2019-10-13T13:56:16.093 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:16.086+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.090937+0000) 2019-10-13T13:56:16.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:16.086+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.091979+0000) 2019-10-13T13:56:16.096 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:16.088+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.094263+0000) 2019-10-13T13:56:16.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:16.090+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.095873+0000) 2019-10-13T13:56:16.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:16.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:16.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:16.100+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:16.104718+0000) 2019-10-13T13:56:17.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:17.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:17.978534+0000) 2019-10-13T13:56:18.207 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:18.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:18.204849+0000) 2019-10-13T13:56:18.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:18.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:18.743286+0000) 2019-10-13T13:56:18.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:18.780+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:18.784494+0000) 2019-10-13T13:56:21.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:21.097+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.101637+0000) 2019-10-13T13:56:21.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:21.097+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.102305+0000) 2019-10-13T13:56:21.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:21.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.102736+0000) 2019-10-13T13:56:21.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:21.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.103666+0000) 2019-10-13T13:56:21.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:21.097+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.103993+0000) 2019-10-13T13:56:21.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:21.100+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.104956+0000) 2019-10-13T13:56:21.108 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:21.097+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.104995+0000) 2019-10-13T13:56:21.113 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:21.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:21.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:21.104+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.110789+0000) 2019-10-13T13:56:21.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:21.721+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.725572+0000) 2019-10-13T13:56:21.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:21.992+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:21.995596+0000) 2019-10-13T13:56:24.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:24.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.323227+0000) 2019-10-13T13:56:24.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:24.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:24.608604+0000) 2019-10-13T13:56:26.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:26.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.113422+0000) 2019-10-13T13:56:26.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:26.109+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.114149+0000) 2019-10-13T13:56:26.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:26.109+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.114713+0000) 2019-10-13T13:56:26.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:26.109+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.115468+0000) 2019-10-13T13:56:26.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:26.109+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.116618+0000) 2019-10-13T13:56:26.119 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:26.112+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.117039+0000) 2019-10-13T13:56:26.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:26.109+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.117133+0000) 2019-10-13T13:56:26.123 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:26.116+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:26.120930+0000) 2019-10-13T13:56:26.126 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:26.117+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:27.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:27.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:27.978731+0000) 2019-10-13T13:56:28.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:28.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:28.205047+0000) 2019-10-13T13:56:28.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:28.737+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:28.743450+0000) 2019-10-13T13:56:28.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:28.788+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:28.791165+0000) 2019-10-13T13:56:30.303 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:56:30.303 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:30.307 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:56:30.307 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:56:31.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:31.121+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.126451+0000) 2019-10-13T13:56:31.132 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:31.121+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.129396+0000) 2019-10-13T13:56:31.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:31.124+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.130312+0000) 2019-10-13T13:56:31.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:31.129+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.130588+0000) 2019-10-13T13:56:31.134 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:31.129+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.130881+0000) 2019-10-13T13:56:31.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:31.129+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.130976+0000) 2019-10-13T13:56:31.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:31.129+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.131286+0000) 2019-10-13T13:56:31.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:31.128+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.133944+0000) 2019-10-13T13:56:31.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:31.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:31.728 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:31.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.725783+0000) 2019-10-13T13:56:31.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:31.992+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:31.995761+0000) 2019-10-13T13:56:34.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:34.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.323430+0000) 2019-10-13T13:56:34.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:34.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:34.608799+0000) 2019-10-13T13:56:36.142 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:36.137+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.139843+0000) 2019-10-13T13:56:36.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:36.137+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.140650+0000) 2019-10-13T13:56:36.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:36.137+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.140972+0000) 2019-10-13T13:56:36.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:36.136+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.141470+0000) 2019-10-13T13:56:36.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:36.141+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.141787+0000) 2019-10-13T13:56:36.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:36.141+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.142899+0000) 2019-10-13T13:56:36.145 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:36.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.142915+0000) 2019-10-13T13:56:36.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:36.144+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:36.148516+0000) 2019-10-13T13:56:36.151 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:36.145+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:37.981 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:37.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:37.978926+0000) 2019-10-13T13:56:38.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:38.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:38.205246+0000) 2019-10-13T13:56:38.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:38.741+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:38.743668+0000) 2019-10-13T13:56:38.800 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:38.796+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:38.797579+0000) 2019-10-13T13:56:41.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:41.149+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.152405+0000) 2019-10-13T13:56:41.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:41.149+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.153214+0000) 2019-10-13T13:56:41.156 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:41.149+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.153601+0000) 2019-10-13T13:56:41.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:41.148+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.154509+0000) 2019-10-13T13:56:41.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:41.153+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.155192+0000) 2019-10-13T13:56:41.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:41.153+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.155676+0000) 2019-10-13T13:56:41.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:41.153+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.156394+0000) 2019-10-13T13:56:41.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:41.156+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.161829+0000) 2019-10-13T13:56:41.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:41.161+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193756 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:41.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:41.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.725961+0000) 2019-10-13T13:56:41.998 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:41.992+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:41.995971+0000) 2019-10-13T13:56:44.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:44.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.323628+0000) 2019-10-13T13:56:44.611 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:44.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:44.608994+0000) 2019-10-13T13:56:46.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:46.165+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.166527+0000) 2019-10-13T13:56:46.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:46.165+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.167032+0000) 2019-10-13T13:56:46.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:46.160+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.167429+0000) 2019-10-13T13:56:46.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:46.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.167528+0000) 2019-10-13T13:56:46.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:46.165+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.167902+0000) 2019-10-13T13:56:46.171 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:46.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.168863+0000) 2019-10-13T13:56:46.173 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:46.169+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.170874+0000) 2019-10-13T13:56:46.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:46.168+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:46.174517+0000) 2019-10-13T13:56:46.177 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:46.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:47.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:47.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:47.979124+0000) 2019-10-13T13:56:48.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:48.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:48.205448+0000) 2019-10-13T13:56:48.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:48.741+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:48.743859+0000) 2019-10-13T13:56:48.806 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:48.800+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:48.804116+0000) 2019-10-13T13:56:51.181 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:51.177+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.178096+0000) 2019-10-13T13:56:51.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:51.177+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.178768+0000) 2019-10-13T13:56:51.182 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:51.177+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.179124+0000) 2019-10-13T13:56:51.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:51.177+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.180015+0000) 2019-10-13T13:56:51.183 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:51.176+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.180670+0000) 2019-10-13T13:56:51.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:51.177+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.180865+0000) 2019-10-13T13:56:51.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:51.181+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.181953+0000) 2019-10-13T13:56:51.190 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:51.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193817 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:51.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:51.180+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.187839+0000) 2019-10-13T13:56:51.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:51.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.726142+0000) 2019-10-13T13:56:51.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:51.996+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:51.996156+0000) 2019-10-13T13:56:54.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:54.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.323860+0000) 2019-10-13T13:56:54.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:54.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:54.609192+0000) 2019-10-13T13:56:56.193 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:56.189+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.190818+0000) 2019-10-13T13:56:56.194 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:56.189+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.191535+0000) 2019-10-13T13:56:56.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:56.189+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.191885+0000) 2019-10-13T13:56:56.195 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:56:56.192+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.192526+0000) 2019-10-13T13:56:56.196 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:56:56.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.193362+0000) 2019-10-13T13:56:56.196 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:56:56.193+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.194190+0000) 2019-10-13T13:56:56.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:56:56.193+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.195099+0000) 2019-10-13T13:56:56.202 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:56:56.197+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193845 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:56:56.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:56.200+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:56.200110+0000) 2019-10-13T13:56:57.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:56:57.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:57.979338+0000) 2019-10-13T13:56:58.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:56:58.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:58.205646+0000) 2019-10-13T13:56:58.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:56:58.741+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:58.744053+0000) 2019-10-13T13:56:58.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:56:58.808+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:56:58.810647+0000) 2019-10-13T13:57:00.434 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:57:00.434 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:00.438 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:57:00.438 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:01.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:01.201+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.203813+0000) 2019-10-13T13:57:01.207 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:01.201+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.204389+0000) 2019-10-13T13:57:01.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:01.201+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.205143+0000) 2019-10-13T13:57:01.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:01.203+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.205732+0000) 2019-10-13T13:57:01.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:01.205+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.205980+0000) 2019-10-13T13:57:01.209 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:01.205+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.206544+0000) 2019-10-13T13:57:01.211 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:01.205+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.208878+0000) 2019-10-13T13:57:01.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:01.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193880 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:01.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:01.211+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.214659+0000) 2019-10-13T13:57:01.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:01.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.726342+0000) 2019-10-13T13:57:01.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:01.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:01.996329+0000) 2019-10-13T13:57:04.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:04.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.324015+0000) 2019-10-13T13:57:04.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:04.605+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:04.609407+0000) 2019-10-13T13:57:06.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:06.213+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.216383+0000) 2019-10-13T13:57:06.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:06.215+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.216773+0000) 2019-10-13T13:57:06.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:06.213+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.217523+0000) 2019-10-13T13:57:06.220 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:06.217+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.217729+0000) 2019-10-13T13:57:06.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:06.217+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.218040+0000) 2019-10-13T13:57:06.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:06.217+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.218471+0000) 2019-10-13T13:57:06.222 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:06.217+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.218800+0000) 2019-10-13T13:57:06.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:06.223+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:06.226003+0000) 2019-10-13T13:57:06.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:06.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:07.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:07.977+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:07.979553+0000) 2019-10-13T13:57:08.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:08.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:08.205858+0000) 2019-10-13T13:57:08.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:08.741+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:08.744235+0000) 2019-10-13T13:57:08.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:08.815+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:08.817120+0000) 2019-10-13T13:57:11.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:11.229+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.229983+0000) 2019-10-13T13:57:11.233 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:11.229+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.230886+0000) 2019-10-13T13:57:11.234 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:11.229+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.231483+0000) 2019-10-13T13:57:11.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:11.229+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.232822+0000) 2019-10-13T13:57:11.237 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:11.231+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.234947+0000) 2019-10-13T13:57:11.238 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:11.233+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.235675+0000) 2019-10-13T13:57:11.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:11.233+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.237470+0000) 2019-10-13T13:57:11.242 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:11.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:11.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:11.239+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.242168+0000) 2019-10-13T13:57:11.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:11.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.726527+0000) 2019-10-13T13:57:11.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:11.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:11.996516+0000) 2019-10-13T13:57:14.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:14.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.324194+0000) 2019-10-13T13:57:14.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:14.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:14.609642+0000) 2019-10-13T13:57:16.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:16.240+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.243822+0000) 2019-10-13T13:57:16.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:16.240+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.244372+0000) 2019-10-13T13:57:16.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:16.243+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.245709+0000) 2019-10-13T13:57:16.248 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:16.244+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.246122+0000) 2019-10-13T13:57:16.249 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:16.244+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.246256+0000) 2019-10-13T13:57:16.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:16.244+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.246439+0000) 2019-10-13T13:57:16.250 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:16.244+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.247664+0000) 2019-10-13T13:57:16.256 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:16.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:16.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:16.251+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:16.253958+0000) 2019-10-13T13:57:17.982 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:17.976+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:17.979747+0000) 2019-10-13T13:57:18.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:18.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:18.206011+0000) 2019-10-13T13:57:18.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:18.740+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:18.744432+0000) 2019-10-13T13:57:18.826 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:18.819+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:18.823420+0000) 2019-10-13T13:57:21.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:21.252+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.256966+0000) 2019-10-13T13:57:21.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:21.252+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.257452+0000) 2019-10-13T13:57:21.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:21.256+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.258067+0000) 2019-10-13T13:57:21.261 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:21.256+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.258339+0000) 2019-10-13T13:57:21.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:21.256+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.258902+0000) 2019-10-13T13:57:21.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:21.255+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.259709+0000) 2019-10-13T13:57:21.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:21.256+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.259984+0000) 2019-10-13T13:57:21.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:21.263+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.266873+0000) 2019-10-13T13:57:21.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:21.264+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:21.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:21.724+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.726745+0000) 2019-10-13T13:57:21.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:21.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:21.996708+0000) 2019-10-13T13:57:24.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:24.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.324414+0000) 2019-10-13T13:57:24.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:24.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:24.609859+0000) 2019-10-13T13:57:26.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:26.268+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.271761+0000) 2019-10-13T13:57:26.275 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:26.268+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.273107+0000) 2019-10-13T13:57:26.278 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:26.272+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.275220+0000) 2019-10-13T13:57:26.278 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:26.272+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.275451+0000) 2019-10-13T13:57:26.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:26.272+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.276633+0000) 2019-10-13T13:57:26.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:26.272+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.277195+0000) 2019-10-13T13:57:26.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:26.275+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.277588+0000) 2019-10-13T13:57:26.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:26.275+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:26.279440+0000) 2019-10-13T13:57:26.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:26.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:27.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:27.976+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:27.980007+0000) 2019-10-13T13:57:28.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:28.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:28.206176+0000) 2019-10-13T13:57:28.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:28.740+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:28.744622+0000) 2019-10-13T13:57:28.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:28.827+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:28.829866+0000) 2019-10-13T13:57:30.505 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:57:30.505 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:30.509 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:57:30.509 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:57:31.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:31.280+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.283368+0000) 2019-10-13T13:57:31.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:31.280+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.284669+0000) 2019-10-13T13:57:31.287 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:31.280+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.284130+0000) 2019-10-13T13:57:31.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:31.284+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.286512+0000) 2019-10-13T13:57:31.290 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:31.283+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.287434+0000) 2019-10-13T13:57:31.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:31.284+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.287753+0000) 2019-10-13T13:57:31.291 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:31.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.288475+0000) 2019-10-13T13:57:31.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:31.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:31.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:31.291+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.293681+0000) 2019-10-13T13:57:31.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:31.724+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.726912+0000) 2019-10-13T13:57:31.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:31.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:31.996872+0000) 2019-10-13T13:57:34.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:34.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.324614+0000) 2019-10-13T13:57:34.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:34.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:34.610073+0000) 2019-10-13T13:57:36.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:36.295+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.296310+0000) 2019-10-13T13:57:36.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:36.292+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.297524+0000) 2019-10-13T13:57:36.301 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:36.296+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.298108+0000) 2019-10-13T13:57:36.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:36.296+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.298395+0000) 2019-10-13T13:57:36.302 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:36.296+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.298820+0000) 2019-10-13T13:57:36.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:36.296+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.298960+0000) 2019-10-13T13:57:36.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:36.296+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.300283+0000) 2019-10-13T13:57:36.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:36.303+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:36.305997+0000) 2019-10-13T13:57:36.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:36.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:37.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:37.976+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:37.980230+0000) 2019-10-13T13:57:38.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:38.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:38.206392+0000) 2019-10-13T13:57:38.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:38.744+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:38.744819+0000) 2019-10-13T13:57:38.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:38.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:38.836460+0000) 2019-10-13T13:57:41.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:41.307+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.310159+0000) 2019-10-13T13:57:41.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:41.308+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.310747+0000) 2019-10-13T13:57:41.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:41.308+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.311671+0000) 2019-10-13T13:57:41.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:41.308+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.311955+0000) 2019-10-13T13:57:41.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:41.308+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.312516+0000) 2019-10-13T13:57:41.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:41.308+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.313005+0000) 2019-10-13T13:57:41.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:41.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.313700+0000) 2019-10-13T13:57:41.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:41.319+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.320621+0000) 2019-10-13T13:57:41.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:41.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194127 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:41.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:41.724+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.727095+0000) 2019-10-13T13:57:42.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:41.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:41.997051+0000) 2019-10-13T13:57:44.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:44.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.324825+0000) 2019-10-13T13:57:44.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:44.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:44.610306+0000) 2019-10-13T13:57:46.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:46.320+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.324443+0000) 2019-10-13T13:57:46.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:46.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.326713+0000) 2019-10-13T13:57:46.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:46.324+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.327300+0000) 2019-10-13T13:57:46.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:46.324+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.327845+0000) 2019-10-13T13:57:46.331 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:46.327+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.328396+0000) 2019-10-13T13:57:46.331 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:46.324+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.328550+0000) 2019-10-13T13:57:46.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:46.324+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.329107+0000) 2019-10-13T13:57:46.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:46.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:46.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:46.331+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:46.336058+0000) 2019-10-13T13:57:47.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:47.976+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:47.980435+0000) 2019-10-13T13:57:48.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:48.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:48.206547+0000) 2019-10-13T13:57:48.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:48.740+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:48.745009+0000) 2019-10-13T13:57:48.845 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:48.839+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:48.842951+0000) 2019-10-13T13:57:51.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:51.332+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.337297+0000) 2019-10-13T13:57:51.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:51.335+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.338106+0000) 2019-10-13T13:57:51.343 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:51.336+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.341131+0000) 2019-10-13T13:57:51.344 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:51.336+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.341546+0000) 2019-10-13T13:57:51.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:51.340+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.341930+0000) 2019-10-13T13:57:51.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:51.340+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.342941+0000) 2019-10-13T13:57:51.346 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:51.340+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.343772+0000) 2019-10-13T13:57:51.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:51.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194188 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:51.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:51.351+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.354616+0000) 2019-10-13T13:57:51.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:51.724+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.727265+0000) 2019-10-13T13:57:52.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:51.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:51.997224+0000) 2019-10-13T13:57:54.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:54.324+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.325050+0000) 2019-10-13T13:57:54.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:54.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:54.610539+0000) 2019-10-13T13:57:56.353 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:56.348+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.350544+0000) 2019-10-13T13:57:56.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:57:56.348+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.350862+0000) 2019-10-13T13:57:56.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:57:56.347+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.351462+0000) 2019-10-13T13:57:56.357 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:56.352+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.355108+0000) 2019-10-13T13:57:56.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:57:56.352+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.357244+0000) 2019-10-13T13:57:56.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:56.355+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.358907+0000) 2019-10-13T13:57:56.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:56.356+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.359031+0000) 2019-10-13T13:57:56.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:57:56.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:57:56.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:57:56.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:56.361560+0000) 2019-10-13T13:57:57.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:57:57.976+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:57.980588+0000) 2019-10-13T13:57:58.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:57:58.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:58.206712+0000) 2019-10-13T13:57:58.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:57:58.740+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:58.745207+0000) 2019-10-13T13:57:58.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:57:58.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:57:58.849351+0000) 2019-10-13T13:58:00.569 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:58:00.569 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:00.574 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:58:00.574 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:01.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:01.363+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.364952+0000) 2019-10-13T13:58:01.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:01.360+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.365078+0000) 2019-10-13T13:58:01.368 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:01.360+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.365390+0000) 2019-10-13T13:58:01.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:01.364+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.366049+0000) 2019-10-13T13:58:01.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:01.364+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.366550+0000) 2019-10-13T13:58:01.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:01.364+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.367312+0000) 2019-10-13T13:58:01.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:01.364+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.368845+0000) 2019-10-13T13:58:01.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:01.371+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.374503+0000) 2019-10-13T13:58:01.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:01.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:01.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:01.724+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.727437+0000) 2019-10-13T13:58:02.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:01.995+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:01.997360+0000) 2019-10-13T13:58:04.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:04.320+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.325265+0000) 2019-10-13T13:58:04.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:04.608+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:04.610729+0000) 2019-10-13T13:58:06.380 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:06.376+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.377724+0000) 2019-10-13T13:58:06.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:06.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.378456+0000) 2019-10-13T13:58:06.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:06.378+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.382620+0000) 2019-10-13T13:58:06.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:06.380+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.383076+0000) 2019-10-13T13:58:06.387 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:06.380+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.384627+0000) 2019-10-13T13:58:06.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:06.380+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.384980+0000) 2019-10-13T13:58:06.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:06.380+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.385455+0000) 2019-10-13T13:58:06.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:06.382+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:06.387767+0000) 2019-10-13T13:58:06.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:06.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194278 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:07.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:07.980+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:07.980795+0000) 2019-10-13T13:58:08.209 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:08.204+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:08.206891+0000) 2019-10-13T13:58:08.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:08.743+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:08.745399+0000) 2019-10-13T13:58:08.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:08.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:08.856021+0000) 2019-10-13T13:58:11.395 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:11.390+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.392523+0000) 2019-10-13T13:58:11.396 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:11.391+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.393935+0000) 2019-10-13T13:58:11.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:11.391+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.394274+0000) 2019-10-13T13:58:11.398 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:11.391+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.395887+0000) 2019-10-13T13:58:11.399 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:11.391+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.396272+0000) 2019-10-13T13:58:11.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:11.391+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.396945+0000) 2019-10-13T13:58:11.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:11.391+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.397107+0000) 2019-10-13T13:58:11.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:11.395+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194310 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:11.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:11.398+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.403379+0000) 2019-10-13T13:58:11.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:11.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.727601+0000) 2019-10-13T13:58:12.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:11.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:11.997590+0000) 2019-10-13T13:58:14.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:14.319+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.325464+0000) 2019-10-13T13:58:14.613 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:14.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:14.610930+0000) 2019-10-13T13:58:16.408 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:16.399+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.405011+0000) 2019-10-13T13:58:16.408 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:16.399+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.405443+0000) 2019-10-13T13:58:16.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:16.406+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.409200+0000) 2019-10-13T13:58:16.419 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:16.410+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.413045+0000) 2019-10-13T13:58:16.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:16.407+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.410846+0000) 2019-10-13T13:58:16.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:16.411+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.414195+0000) 2019-10-13T13:58:16.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:16.411+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:16.420 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:16.411+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.414860+0000) 2019-10-13T13:58:16.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:16.411+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:16.415137+0000) 2019-10-13T13:58:17.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:17.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:17.981034+0000) 2019-10-13T13:58:18.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:18.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:18.207060+0000) 2019-10-13T13:58:18.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:18.739+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:18.745602+0000) 2019-10-13T13:58:18.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:18.862+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:18.867558+0000) 2019-10-13T13:58:21.420 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:21.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.417446+0000) 2019-10-13T13:58:21.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:21.415+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.418217+0000) 2019-10-13T13:58:21.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:21.415+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.418243+0000) 2019-10-13T13:58:21.422 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:21.415+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.419664+0000) 2019-10-13T13:58:21.424 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:21.415+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.421551+0000) 2019-10-13T13:58:21.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:21.418+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.422976+0000) 2019-10-13T13:58:21.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:21.419+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.423138+0000) 2019-10-13T13:58:21.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:21.423+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194372 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:21.431 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:21.422+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.428071+0000) 2019-10-13T13:58:21.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:21.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.727777+0000) 2019-10-13T13:58:22.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:21.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:21.997761+0000) 2019-10-13T13:58:24.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:24.323+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.325671+0000) 2019-10-13T13:58:24.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:24.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:24.611081+0000) 2019-10-13T13:58:26.435 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:26.430+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.433200+0000) 2019-10-13T13:58:26.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:26.427+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.433474+0000) 2019-10-13T13:58:26.437 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:26.431+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.434933+0000) 2019-10-13T13:58:26.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:26.431+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.435612+0000) 2019-10-13T13:58:26.439 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:26.431+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.436265+0000) 2019-10-13T13:58:26.439 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:26.431+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.436416+0000) 2019-10-13T13:58:26.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:26.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.437236+0000) 2019-10-13T13:58:26.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:26.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:26.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:26.438+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:26.442076+0000) 2019-10-13T13:58:27.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:27.979+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:27.981250+0000) 2019-10-13T13:58:28.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:28.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:28.207245+0000) 2019-10-13T13:58:28.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:28.743+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:28.745796+0000) 2019-10-13T13:58:28.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:28.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:28.874024+0000) 2019-10-13T13:58:30.825 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:58:30.825 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:30.829 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:58:30.829 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:58:31.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:31.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.444114+0000) 2019-10-13T13:58:31.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:31.439+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.445317+0000) 2019-10-13T13:58:31.448 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:31.442+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.445974+0000) 2019-10-13T13:58:31.449 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:31.443+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.446577+0000) 2019-10-13T13:58:31.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:31.443+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.448213+0000) 2019-10-13T13:58:31.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:31.443+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.448601+0000) 2019-10-13T13:58:31.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:31.447+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.450137+0000) 2019-10-13T13:58:31.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:31.450+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.452400+0000) 2019-10-13T13:58:31.457 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:31.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:31.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:31.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.727951+0000) 2019-10-13T13:58:32.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:31.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:31.997885+0000) 2019-10-13T13:58:34.328 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:34.323+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.325902+0000) 2019-10-13T13:58:34.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:34.607+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:34.611277+0000) 2019-10-13T13:58:36.461 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:36.455+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.457906+0000) 2019-10-13T13:58:36.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:36.455+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.458694+0000) 2019-10-13T13:58:36.461 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:36.455+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.458948+0000) 2019-10-13T13:58:36.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:36.458+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.461802+0000) 2019-10-13T13:58:36.464 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:36.459+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.461947+0000) 2019-10-13T13:58:36.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:36.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.462288+0000) 2019-10-13T13:58:36.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:36.459+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.462669+0000) 2019-10-13T13:58:36.469 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:36.462+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:36.466553+0000) 2019-10-13T13:58:36.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:36.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:37.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:37.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:37.981441+0000) 2019-10-13T13:58:38.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:38.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:38.207410+0000) 2019-10-13T13:58:38.748 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:38.743+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:38.745996+0000) 2019-10-13T13:58:38.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:38.878+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:38.880564+0000) 2019-10-13T13:58:41.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:41.467+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.471436+0000) 2019-10-13T13:58:41.475 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:41.467+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.472575+0000) 2019-10-13T13:58:41.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:41.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.473236+0000) 2019-10-13T13:58:41.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:41.467+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.473568+0000) 2019-10-13T13:58:41.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:41.470+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.474242+0000) 2019-10-13T13:58:41.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:41.471+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.474334+0000) 2019-10-13T13:58:41.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:41.471+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.474584+0000) 2019-10-13T13:58:41.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:41.474+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.479423+0000) 2019-10-13T13:58:41.484 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:41.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:41.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:41.727+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.728087+0000) 2019-10-13T13:58:42.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:41.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:41.998092+0000) 2019-10-13T13:58:44.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:44.323+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.326091+0000) 2019-10-13T13:58:44.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:44.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:44.611502+0000) 2019-10-13T13:58:46.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:46.479+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.485195+0000) 2019-10-13T13:58:46.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:46.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.485909+0000) 2019-10-13T13:58:46.489 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:46.483+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.486518+0000) 2019-10-13T13:58:46.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:46.483+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.486918+0000) 2019-10-13T13:58:46.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:46.486+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.488185+0000) 2019-10-13T13:58:46.491 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:46.483+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.488256+0000) 2019-10-13T13:58:46.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:46.483+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.488688+0000) 2019-10-13T13:58:46.498 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:46.491+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:46.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:46.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:46.496712+0000) 2019-10-13T13:58:47.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:47.975+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:47.981640+0000) 2019-10-13T13:58:48.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:48.203+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:48.207645+0000) 2019-10-13T13:58:48.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:48.743+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:48.746191+0000) 2019-10-13T13:58:48.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:48.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:48.887244+0000) 2019-10-13T13:58:51.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:51.495+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.499012+0000) 2019-10-13T13:58:51.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:51.495+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.499418+0000) 2019-10-13T13:58:51.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:51.494+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.499986+0000) 2019-10-13T13:58:51.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:51.499+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.502075+0000) 2019-10-13T13:58:51.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:51.499+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.503133+0000) 2019-10-13T13:58:51.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:51.499+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.503527+0000) 2019-10-13T13:58:51.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:51.499+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.503700+0000) 2019-10-13T13:58:51.511 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:51.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194553 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:51.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:51.506+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.509460+0000) 2019-10-13T13:58:51.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:51.723+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.728264+0000) 2019-10-13T13:58:52.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:51.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:51.998263+0000) 2019-10-13T13:58:54.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:54.323+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.326291+0000) 2019-10-13T13:58:54.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:54.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:54.611695+0000) 2019-10-13T13:58:56.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:58:56.506+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.512007+0000) 2019-10-13T13:58:56.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:58:56.507+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.512698+0000) 2019-10-13T13:58:56.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:56.507+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.512846+0000) 2019-10-13T13:58:56.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:56.511+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.513943+0000) 2019-10-13T13:58:56.517 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:58:56.511+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.514692+0000) 2019-10-13T13:58:56.517 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:56.511+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.515106+0000) 2019-10-13T13:58:56.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:58:56.511+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.515518+0000) 2019-10-13T13:58:56.524 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:58:56.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194582 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:58:56.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:56.518+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:56.522411+0000) 2019-10-13T13:58:57.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:58:57.979+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:57.981844+0000) 2019-10-13T13:58:58.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:58:58.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:58.207845+0000) 2019-10-13T13:58:58.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:58:58.743+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:58.746420+0000) 2019-10-13T13:58:58.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:58:58.890+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:58:58.893901+0000) 2019-10-13T13:59:00.906 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:59:00.906 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:00.910 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:59:00.910 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:01.529 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:01.523+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.526208+0000) 2019-10-13T13:59:01.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:01.523+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.526610+0000) 2019-10-13T13:59:01.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:01.522+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.527527+0000) 2019-10-13T13:59:01.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:01.523+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.527627+0000) 2019-10-13T13:59:01.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:01.523+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.527972+0000) 2019-10-13T13:59:01.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:01.523+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.528469+0000) 2019-10-13T13:59:01.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:01.523+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.529229+0000) 2019-10-13T13:59:01.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:01.530+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.533408+0000) 2019-10-13T13:59:01.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:01.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194615 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:01.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:01.727+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.728463+0000) 2019-10-13T13:59:02.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:01.994+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:01.998474+0000) 2019-10-13T13:59:04.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:04.323+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.326455+0000) 2019-10-13T13:59:04.614 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:04.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:04.611854+0000) 2019-10-13T13:59:06.541 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:06.534+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.537841+0000) 2019-10-13T13:59:06.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:06.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.538828+0000) 2019-10-13T13:59:06.542 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:06.534+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.539034+0000) 2019-10-13T13:59:06.542 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:06.534+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.539489+0000) 2019-10-13T13:59:06.543 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:06.534+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.540200+0000) 2019-10-13T13:59:06.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:06.534+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.540877+0000) 2019-10-13T13:59:06.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:06.534+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.540744+0000) 2019-10-13T13:59:06.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:06.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:06.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:06.546+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:06.548545+0000) 2019-10-13T13:59:07.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:07.978+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:07.982044+0000) 2019-10-13T13:59:08.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:08.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:08.208052+0000) 2019-10-13T13:59:08.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:08.742+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:08.746672+0000) 2019-10-13T13:59:08.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:08.898+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:08.900534+0000) 2019-10-13T13:59:11.554 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:11.546+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.551439+0000) 2019-10-13T13:59:11.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:11.546+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.552289+0000) 2019-10-13T13:59:11.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:11.550+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.554562+0000) 2019-10-13T13:59:11.557 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:11.550+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.555170+0000) 2019-10-13T13:59:11.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:11.550+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.555900+0000) 2019-10-13T13:59:11.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:11.553+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.556151+0000) 2019-10-13T13:59:11.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:11.550+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.557055+0000) 2019-10-13T13:59:11.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:11.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:11.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:11.557+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.563815+0000) 2019-10-13T13:59:11.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:11.726+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.728704+0000) 2019-10-13T13:59:12.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:11.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:11.998662+0000) 2019-10-13T13:59:14.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:14.322+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.326619+0000) 2019-10-13T13:59:14.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:14.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:14.612084+0000) 2019-10-13T13:59:16.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:16.558+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.565157+0000) 2019-10-13T13:59:16.568 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:16.558+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.565575+0000) 2019-10-13T13:59:16.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:16.561+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.567245+0000) 2019-10-13T13:59:16.570 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:16.562+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.567342+0000) 2019-10-13T13:59:16.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:16.562+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.567669+0000) 2019-10-13T13:59:16.571 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:16.562+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.567925+0000) 2019-10-13T13:59:16.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:16.562+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.569575+0000) 2019-10-13T13:59:16.577 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:16.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194704 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:16.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:16.569+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:16.575519+0000) 2019-10-13T13:59:17.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:17.978+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:17.982225+0000) 2019-10-13T13:59:18.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:18.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:18.208268+0000) 2019-10-13T13:59:18.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:18.742+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:18.746864+0000) 2019-10-13T13:59:18.909 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:18.901+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:18.907048+0000) 2019-10-13T13:59:21.581 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:21.574+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.578273+0000) 2019-10-13T13:59:21.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:21.574+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.578787+0000) 2019-10-13T13:59:21.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:21.574+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.579245+0000) 2019-10-13T13:59:21.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:21.577+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.580569+0000) 2019-10-13T13:59:21.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:21.574+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.580641+0000) 2019-10-13T13:59:21.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:21.574+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.581172+0000) 2019-10-13T13:59:21.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:21.578+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.582075+0000) 2019-10-13T13:59:21.590 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:21.585+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.588170+0000) 2019-10-13T13:59:21.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:21.582+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:21.731 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:21.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.728915+0000) 2019-10-13T13:59:22.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:21.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:21.998855+0000) 2019-10-13T13:59:24.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:24.326+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.326816+0000) 2019-10-13T13:59:24.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:24.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:24.612277+0000) 2019-10-13T13:59:26.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:26.586+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.591130+0000) 2019-10-13T13:59:26.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:26.586+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.592278+0000) 2019-10-13T13:59:26.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:26.589+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.592890+0000) 2019-10-13T13:59:26.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:26.586+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.593226+0000) 2019-10-13T13:59:26.596 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:26.590+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.593806+0000) 2019-10-13T13:59:26.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:26.590+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.594483+0000) 2019-10-13T13:59:26.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:26.590+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.594935+0000) 2019-10-13T13:59:26.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:26.593+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:26.599376+0000) 2019-10-13T13:59:26.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:26.594+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194765 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:27.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:27.978+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:27.982423+0000) 2019-10-13T13:59:28.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:28.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:28.208490+0000) 2019-10-13T13:59:28.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:28.746+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:28.747070+0000) 2019-10-13T13:59:28.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:28.913+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:28.913901+0000) 2019-10-13T13:59:31.028 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T13:59:31.028 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:31.032 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T13:59:31.032 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T13:59:31.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:31.601+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.604997+0000) 2019-10-13T13:59:31.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:31.602+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.605847+0000) 2019-10-13T13:59:31.609 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:31.602+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.606913+0000) 2019-10-13T13:59:31.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:31.602+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.607200+0000) 2019-10-13T13:59:31.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:31.602+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.607134+0000) 2019-10-13T13:59:31.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:31.602+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.607718+0000) 2019-10-13T13:59:31.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:31.602+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.608149+0000) 2019-10-13T13:59:31.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:31.609+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.615533+0000) 2019-10-13T13:59:31.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:31.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:31.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:31.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.729138+0000) 2019-10-13T13:59:32.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:31.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:31.999045+0000) 2019-10-13T13:59:34.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:34.322+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.327014+0000) 2019-10-13T13:59:34.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:34.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:34.612472+0000) 2019-10-13T13:59:36.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:36.614+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.618363+0000) 2019-10-13T13:59:36.621 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:36.614+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.619125+0000) 2019-10-13T13:59:36.622 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:36.613+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.619855+0000) 2019-10-13T13:59:36.622 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:36.614+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.620213+0000) 2019-10-13T13:59:36.623 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:36.614+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.620570+0000) 2019-10-13T13:59:36.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:36.614+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.621366+0000) 2019-10-13T13:59:36.624 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:36.618+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.621706+0000) 2019-10-13T13:59:36.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:36.621+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:36.625076+0000) 2019-10-13T13:59:36.631 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:36.622+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:37.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:37.982+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:37.982621+0000) 2019-10-13T13:59:38.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:38.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:38.208642+0000) 2019-10-13T13:59:38.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:38.746+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:38.747293+0000) 2019-10-13T13:59:38.923 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:38.917+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:38.920514+0000) 2019-10-13T13:59:41.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:41.629+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.634044+0000) 2019-10-13T13:59:41.637 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:41.630+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.634334+0000) 2019-10-13T13:59:41.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:41.630+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.635094+0000) 2019-10-13T13:59:41.638 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:41.630+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.635559+0000) 2019-10-13T13:59:41.638 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:41.630+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.636078+0000) 2019-10-13T13:59:41.639 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:41.630+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.636553+0000) 2019-10-13T13:59:41.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:41.630+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.636962+0000) 2019-10-13T13:59:41.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:41.634+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:41.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:41.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.642208+0000) 2019-10-13T13:59:41.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:41.726+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.729354+0000) 2019-10-13T13:59:42.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:41.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:41.999214+0000) 2019-10-13T13:59:44.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:44.326+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.327215+0000) 2019-10-13T13:59:44.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:44.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:44.612668+0000) 2019-10-13T13:59:46.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:46.638+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.645634+0000) 2019-10-13T13:59:46.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:46.642+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.646553+0000) 2019-10-13T13:59:46.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:46.641+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.646859+0000) 2019-10-13T13:59:46.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:46.642+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.647033+0000) 2019-10-13T13:59:46.651 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:46.642+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.647611+0000) 2019-10-13T13:59:46.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:46.642+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.647989+0000) 2019-10-13T13:59:46.651 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:46.642+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.648279+0000) 2019-10-13T13:59:46.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:46.649+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:46.653514+0000) 2019-10-13T13:59:46.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:46.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194887 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:47.985 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:47.978+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:47.982839+0000) 2019-10-13T13:59:48.211 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:48.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:48.208837+0000) 2019-10-13T13:59:48.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:48.746+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:48.747495+0000) 2019-10-13T13:59:48.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:48.925+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:48.927124+0000) 2019-10-13T13:59:51.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:51.654+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.658343+0000) 2019-10-13T13:59:51.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:51.654+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.658743+0000) 2019-10-13T13:59:51.662 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:51.654+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.659435+0000) 2019-10-13T13:59:51.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:51.654+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.660085+0000) 2019-10-13T13:59:51.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:51.654+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.660219+0000) 2019-10-13T13:59:51.663 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:51.657+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.661288+0000) 2019-10-13T13:59:51.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:51.654+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.660725+0000) 2019-10-13T13:59:51.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:51.661+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.664313+0000) 2019-10-13T13:59:51.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:51.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:51.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:51.722+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.729530+0000) 2019-10-13T13:59:52.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:51.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:51.999378+0000) 2019-10-13T13:59:54.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:54.322+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.327407+0000) 2019-10-13T13:59:54.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:54.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:54.612865+0000) 2019-10-13T13:59:56.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:56.666+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.671333+0000) 2019-10-13T13:59:56.674 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:56.666+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.671863+0000) 2019-10-13T13:59:56.676 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T13:59:56.666+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.673220+0000) 2019-10-13T13:59:56.676 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T13:59:56.666+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.672805+0000) 2019-10-13T13:59:56.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T13:59:56.666+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.673663+0000) 2019-10-13T13:59:56.677 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:56.670+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.674566+0000) 2019-10-13T13:59:56.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T13:59:56.669+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.675094+0000) 2019-10-13T13:59:56.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:56.673+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:56.679784+0000) 2019-10-13T13:59:56.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T13:59:56.674+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194947 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T13:59:57.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T13:59:57.982+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:57.982992+0000) 2019-10-13T13:59:58.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T13:59:58.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:58.209049+0000) 2019-10-13T13:59:58.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T13:59:58.746+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:58.747688+0000) 2019-10-13T13:59:58.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T13:59:58.933+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T12:59:58.933867+0000) 2019-10-13T14:00:01.142 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:00:01.142 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:01.146 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:00:01.146 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:01.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:01.677+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.683972+0000) 2019-10-13T14:00:01.687 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:01.677+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.684166+0000) 2019-10-13T14:00:01.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:01.677+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.684551+0000) 2019-10-13T14:00:01.689 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:01.681+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.686144+0000) 2019-10-13T14:00:01.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:01.681+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.687067+0000) 2019-10-13T14:00:01.691 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:01.685+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.689188+0000) 2019-10-13T14:00:01.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:01.681+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.689362+0000) 2019-10-13T14:00:01.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:01.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:01.696 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:01.689+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.693550+0000) 2019-10-13T14:00:01.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:01.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.729736+0000) 2019-10-13T14:00:02.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:01.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:01.999564+0000) 2019-10-13T14:00:04.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:04.321+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.327625+0000) 2019-10-13T14:00:04.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:04.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:04.613089+0000) 2019-10-13T14:00:06.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:06.689+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.695979+0000) 2019-10-13T14:00:06.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:06.689+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.697115+0000) 2019-10-13T14:00:06.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:06.689+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.697504+0000) 2019-10-13T14:00:06.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:06.693+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.698002+0000) 2019-10-13T14:00:06.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:06.693+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.698425+0000) 2019-10-13T14:00:06.702 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:06.693+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.699333+0000) 2019-10-13T14:00:06.702 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:06.693+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.699948+0000) 2019-10-13T14:00:06.709 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:06.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:06.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:06.701+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:06.706639+0000) 2019-10-13T14:00:07.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:07.981+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:07.983207+0000) 2019-10-13T14:00:08.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:08.201+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:08.209263+0000) 2019-10-13T14:00:08.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:08.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:08.747882+0000) 2019-10-13T14:00:08.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:08.937+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:08.940523+0000) 2019-10-13T14:00:11.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:11.705+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.709663+0000) 2019-10-13T14:00:11.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:11.705+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.709987+0000) 2019-10-13T14:00:11.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:11.705+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.710678+0000) 2019-10-13T14:00:11.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:11.705+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.710740+0000) 2019-10-13T14:00:11.714 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:11.705+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.711023+0000) 2019-10-13T14:00:11.715 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:11.705+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.711616+0000) 2019-10-13T14:00:11.715 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:11.705+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.712874+0000) 2019-10-13T14:00:11.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:11.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:11.722 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:11.713+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.719771+0000) 2019-10-13T14:00:11.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:11.725+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.729919+0000) 2019-10-13T14:00:12.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:11.993+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:11.999734+0000) 2019-10-13T14:00:14.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:14.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.327821+0000) 2019-10-13T14:00:14.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:14.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:14.613284+0000) 2019-10-13T14:00:16.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:16.721+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.722593+0000) 2019-10-13T14:00:16.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:16.721+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.723007+0000) 2019-10-13T14:00:16.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:16.721+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.723784+0000) 2019-10-13T14:00:16.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:16.721+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.724094+0000) 2019-10-13T14:00:16.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:16.720+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.725213+0000) 2019-10-13T14:00:16.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:16.721+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.725585+0000) 2019-10-13T14:00:16.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:16.725+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.726697+0000) 2019-10-13T14:00:16.734 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:16.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:16.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:16.724+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:16.731881+0000) 2019-10-13T14:00:17.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:17.981+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:17.983465+0000) 2019-10-13T14:00:18.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:18.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:18.209479+0000) 2019-10-13T14:00:18.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:18.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:18.748087+0000) 2019-10-13T14:00:18.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:18.944+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:18.947038+0000) 2019-10-13T14:00:21.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:21.729+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.730084+0000) 2019-10-13T14:00:21.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:21.733+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.735384+0000) 2019-10-13T14:00:21.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:21.733+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.736136+0000) 2019-10-13T14:00:21.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:21.733+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.736367+0000) 2019-10-13T14:00:21.740 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:21.733+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.737315+0000) 2019-10-13T14:00:21.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:21.737+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.738837+0000) 2019-10-13T14:00:21.744 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:21.737+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.741582+0000) 2019-10-13T14:00:21.746 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:21.736+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.744032+0000) 2019-10-13T14:00:21.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:21.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195103 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:21.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:21.740+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.745191+0000) 2019-10-13T14:00:22.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:21.996+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:21.999911+0000) 2019-10-13T14:00:24.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:24.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.328023+0000) 2019-10-13T14:00:24.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:24.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:24.613499+0000) 2019-10-13T14:00:26.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:26.745+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.748647+0000) 2019-10-13T14:00:26.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:26.745+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.749061+0000) 2019-10-13T14:00:26.752 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:26.749+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.749942+0000) 2019-10-13T14:00:26.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:26.749+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.752667+0000) 2019-10-13T14:00:26.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:26.749+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.752334+0000) 2019-10-13T14:00:26.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:26.749+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.753289+0000) 2019-10-13T14:00:26.756 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:26.748+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.753769+0000) 2019-10-13T14:00:26.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:26.752+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:26.756945+0000) 2019-10-13T14:00:26.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:26.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:27.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:27.981+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:27.983668+0000) 2019-10-13T14:00:28.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:28.205+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:28.209675+0000) 2019-10-13T14:00:28.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:28.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:28.748290+0000) 2019-10-13T14:00:28.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:28.952+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:28.953676+0000) 2019-10-13T14:00:31.270 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:00:31.270 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:31.274 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:00:31.274 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:00:31.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:31.729+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.730285+0000) 2019-10-13T14:00:31.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:31.757+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.760816+0000) 2019-10-13T14:00:31.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:31.761+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.761934+0000) 2019-10-13T14:00:31.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:31.761+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.763887+0000) 2019-10-13T14:00:31.767 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:31.761+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.763841+0000) 2019-10-13T14:00:31.769 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:31.765+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.766568+0000) 2019-10-13T14:00:31.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:31.765+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.768412+0000) 2019-10-13T14:00:31.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:31.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.770632+0000) 2019-10-13T14:00:31.773 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:31.764+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:31.771135+0000) 2019-10-13T14:00:31.774 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:31.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195163 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:32.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:31.996+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:32.000079+0000) 2019-10-13T14:00:34.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:34.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.328223+0000) 2019-10-13T14:00:34.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:34.609+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:34.613703+0000) 2019-10-13T14:00:36.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:36.769+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.773505+0000) 2019-10-13T14:00:36.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:36.773+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.775636+0000) 2019-10-13T14:00:36.780 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:36.773+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.777548+0000) 2019-10-13T14:00:36.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:36.777+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.778119+0000) 2019-10-13T14:00:36.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:36.777+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.778764+0000) 2019-10-13T14:00:36.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:36.772+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.779115+0000) 2019-10-13T14:00:36.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:36.781+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.782517+0000) 2019-10-13T14:00:36.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:36.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195191 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:36.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:36.780+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:36.786859+0000) 2019-10-13T14:00:37.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:37.981+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:37.983829+0000) 2019-10-13T14:00:38.212 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:38.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:38.209906+0000) 2019-10-13T14:00:38.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:38.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:38.748487+0000) 2019-10-13T14:00:38.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:38.956+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:38.960043+0000) 2019-10-13T14:00:41.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:41.729+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.730486+0000) 2019-10-13T14:00:41.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:41.785+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.788409+0000) 2019-10-13T14:00:41.791 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:41.785+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.788793+0000) 2019-10-13T14:00:41.793 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:41.789+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.790697+0000) 2019-10-13T14:00:41.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:41.789+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.790459+0000) 2019-10-13T14:00:41.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:41.784+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.791302+0000) 2019-10-13T14:00:41.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:41.789+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.791390+0000) 2019-10-13T14:00:41.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:41.789+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.792430+0000) 2019-10-13T14:00:41.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:41.792+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:41.796868+0000) 2019-10-13T14:00:41.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:41.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:42.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:41.996+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:42.000244+0000) 2019-10-13T14:00:44.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:44.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.328377+0000) 2019-10-13T14:00:44.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:44.613+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:44.613970+0000) 2019-10-13T14:00:46.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:46.797+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.800885+0000) 2019-10-13T14:00:46.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:46.801+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.802629+0000) 2019-10-13T14:00:46.806 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:46.801+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.803249+0000) 2019-10-13T14:00:46.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:46.801+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.803475+0000) 2019-10-13T14:00:46.807 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:46.801+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.804815+0000) 2019-10-13T14:00:46.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:46.801+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.804424+0000) 2019-10-13T14:00:46.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:46.800+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.805109+0000) 2019-10-13T14:00:46.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:46.809+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195253 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:46.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:46.804+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:46.811679+0000) 2019-10-13T14:00:47.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:47.981+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:47.984030+0000) 2019-10-13T14:00:48.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:48.209+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:48.210111+0000) 2019-10-13T14:00:48.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:48.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:48.748699+0000) 2019-10-13T14:00:48.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:48.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:48.966569+0000) 2019-10-13T14:00:51.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:51.729+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.730688+0000) 2019-10-13T14:00:51.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:51.813+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.814541+0000) 2019-10-13T14:00:51.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:51.813+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.815203+0000) 2019-10-13T14:00:51.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:51.813+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.815554+0000) 2019-10-13T14:00:51.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:51.813+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.815854+0000) 2019-10-13T14:00:51.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:51.813+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.816278+0000) 2019-10-13T14:00:51.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:51.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.818660+0000) 2019-10-13T14:00:51.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:51.812+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.819036+0000) 2019-10-13T14:00:51.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:51.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195285 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:51.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:51.820+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:51.825324+0000) 2019-10-13T14:00:52.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:52.000+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:52.000452+0000) 2019-10-13T14:00:54.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:54.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.328588+0000) 2019-10-13T14:00:54.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:54.613+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:54.614197+0000) 2019-10-13T14:00:56.829 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:00:56.824+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.826775+0000) 2019-10-13T14:00:56.830 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:56.824+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.827087+0000) 2019-10-13T14:00:56.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:56.824+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.827896+0000) 2019-10-13T14:00:56.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:00:56.824+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.828680+0000) 2019-10-13T14:00:56.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:00:56.824+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.829340+0000) 2019-10-13T14:00:56.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:00:56.824+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.829704+0000) 2019-10-13T14:00:56.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:56.828+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.830084+0000) 2019-10-13T14:00:56.838 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:56.828+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:56.835350+0000) 2019-10-13T14:00:56.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:00:56.832+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:00:57.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:00:57.980+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:57.984219+0000) 2019-10-13T14:00:58.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:00:58.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:58.210355+0000) 2019-10-13T14:00:58.751 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:00:58.744+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:58.748928+0000) 2019-10-13T14:00:58.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:00:58.972+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:00:58.973028+0000) 2019-10-13T14:01:01.379 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:01:01.379 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:01.383 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:01:01.383 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:01.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:01.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.730886+0000) 2019-10-13T14:01:01.842 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:01.836+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.839210+0000) 2019-10-13T14:01:01.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:01.836+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.841542+0000) 2019-10-13T14:01:01.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:01.840+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.842775+0000) 2019-10-13T14:01:01.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:01.840+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.843312+0000) 2019-10-13T14:01:01.846 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:01.840+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.843611+0000) 2019-10-13T14:01:01.847 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:01.840+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.844018+0000) 2019-10-13T14:01:01.847 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:01.840+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.844720+0000) 2019-10-13T14:01:01.851 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:01.844+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:01.848481+0000) 2019-10-13T14:01:01.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:01.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:02.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:01.996+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:02.000634+0000) 2019-10-13T14:01:04.331 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:04.324+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.328793+0000) 2019-10-13T14:01:04.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:04.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:04.614417+0000) 2019-10-13T14:01:06.855 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:06.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.852756+0000) 2019-10-13T14:01:06.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:06.848+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.853092+0000) 2019-10-13T14:01:06.856 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:06.852+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.853866+0000) 2019-10-13T14:01:06.857 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:06.852+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.854705+0000) 2019-10-13T14:01:06.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:06.852+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.855027+0000) 2019-10-13T14:01:06.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:06.852+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.855337+0000) 2019-10-13T14:01:06.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:06.852+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.856279+0000) 2019-10-13T14:01:06.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:06.860+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195374 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:06.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:06.860+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:06.865227+0000) 2019-10-13T14:01:07.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:07.980+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:07.984450+0000) 2019-10-13T14:01:08.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:08.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:08.210557+0000) 2019-10-13T14:01:08.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:08.748+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:08.749152+0000) 2019-10-13T14:01:08.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:08.976+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:08.979549+0000) 2019-10-13T14:01:11.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:11.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.731101+0000) 2019-10-13T14:01:11.871 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:11.864+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.868584+0000) 2019-10-13T14:01:11.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:11.864+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.869143+0000) 2019-10-13T14:01:11.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:11.864+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.869503+0000) 2019-10-13T14:01:11.873 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:11.864+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.870136+0000) 2019-10-13T14:01:11.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:11.868+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.870346+0000) 2019-10-13T14:01:11.874 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:11.868+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.871555+0000) 2019-10-13T14:01:11.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:11.868+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.871064+0000) 2019-10-13T14:01:11.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:11.872+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:11.876884+0000) 2019-10-13T14:01:11.880 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:11.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195407 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:12.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:12.000+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:12.000831+0000) 2019-10-13T14:01:14.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:14.324+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.329008+0000) 2019-10-13T14:01:14.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:14.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:14.614645+0000) 2019-10-13T14:01:16.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:16.876+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.881181+0000) 2019-10-13T14:01:16.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:16.880+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.882547+0000) 2019-10-13T14:01:16.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:16.880+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.882843+0000) 2019-10-13T14:01:16.886 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:16.880+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.883097+0000) 2019-10-13T14:01:16.886 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:16.880+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.884034+0000) 2019-10-13T14:01:16.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:16.880+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.885015+0000) 2019-10-13T14:01:16.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:16.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.886494+0000) 2019-10-13T14:01:16.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:16.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:16.890096+0000) 2019-10-13T14:01:16.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:16.888+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195434 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:17.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:17.980+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:17.984643+0000) 2019-10-13T14:01:18.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:18.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:18.210806+0000) 2019-10-13T14:01:18.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:18.748+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:18.749354+0000) 2019-10-13T14:01:18.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:18.983+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:18.986148+0000) 2019-10-13T14:01:21.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:21.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.731339+0000) 2019-10-13T14:01:21.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:21.888+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.893430+0000) 2019-10-13T14:01:21.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:21.891+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.893912+0000) 2019-10-13T14:01:21.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:21.892+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.894146+0000) 2019-10-13T14:01:21.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:21.892+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.894385+0000) 2019-10-13T14:01:21.899 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:21.892+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.896279+0000) 2019-10-13T14:01:21.900 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:21.896+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.897791+0000) 2019-10-13T14:01:21.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:21.896+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.898351+0000) 2019-10-13T14:01:21.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:21.900+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:21.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:21.903+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:21.905548+0000) 2019-10-13T14:01:22.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:21.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:22.001028+0000) 2019-10-13T14:01:24.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:24.324+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.329242+0000) 2019-10-13T14:01:24.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:24.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:24.614876+0000) 2019-10-13T14:01:26.909 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:26.904+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.906357+0000) 2019-10-13T14:01:26.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:26.903+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.907414+0000) 2019-10-13T14:01:26.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:26.904+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.907605+0000) 2019-10-13T14:01:26.911 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:26.904+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.908584+0000) 2019-10-13T14:01:26.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:26.908+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.909894+0000) 2019-10-13T14:01:26.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:26.908+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.910233+0000) 2019-10-13T14:01:26.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:26.904+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.909127+0000) 2019-10-13T14:01:26.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:26.911+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:26.915443+0000) 2019-10-13T14:01:26.919 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:26.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:27.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:27.984+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:27.984874+0000) 2019-10-13T14:01:28.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:28.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:28.211020+0000) 2019-10-13T14:01:28.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:28.744+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:28.749544+0000) 2019-10-13T14:01:28.995 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:28.991+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:28.992626+0000) 2019-10-13T14:01:31.485 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:01:31.485 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:31.489 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:01:31.489 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:01:31.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:31.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.731534+0000) 2019-10-13T14:01:31.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:31.916+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.919568+0000) 2019-10-13T14:01:31.923 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:31.916+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.919870+0000) 2019-10-13T14:01:31.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:31.916+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.920559+0000) 2019-10-13T14:01:31.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:31.920+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.922244+0000) 2019-10-13T14:01:31.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:31.920+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.922669+0000) 2019-10-13T14:01:31.926 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:31.919+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.923438+0000) 2019-10-13T14:01:31.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:31.920+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.923542+0000) 2019-10-13T14:01:31.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:31.923+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:31.927536+0000) 2019-10-13T14:01:31.932 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:31.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195529 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:32.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:31.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:32.001195+0000) 2019-10-13T14:01:34.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:34.328+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:34.329475+0000) 2019-10-13T14:01:34.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:34.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:34.615080+0000) 2019-10-13T14:01:36.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:36.928+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.932294+0000) 2019-10-13T14:01:36.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:36.928+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.932770+0000) 2019-10-13T14:01:36.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:36.931+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.934207+0000) 2019-10-13T14:01:36.938 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:36.928+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.933375+0000) 2019-10-13T14:01:36.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:36.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.934049+0000) 2019-10-13T14:01:36.938 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:36.932+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.934363+0000) 2019-10-13T14:01:36.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:36.932+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.935765+0000) 2019-10-13T14:01:36.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:36.939+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:36.941234+0000) 2019-10-13T14:01:36.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:36.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:37.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:37.984+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:37.985111+0000) 2019-10-13T14:01:38.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:38.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:38.211244+0000) 2019-10-13T14:01:38.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:38.748+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:38.749783+0000) 2019-10-13T14:01:39.001 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:38.995+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:38.999014+0000) 2019-10-13T14:01:41.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:41.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.731804+0000) 2019-10-13T14:01:41.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:41.944+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.946947+0000) 2019-10-13T14:01:41.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:41.947+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.949215+0000) 2019-10-13T14:01:41.952 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:41.944+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.947392+0000) 2019-10-13T14:01:41.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:41.944+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.948215+0000) 2019-10-13T14:01:41.953 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:41.944+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.948024+0000) 2019-10-13T14:01:41.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:41.944+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.948652+0000) 2019-10-13T14:01:41.953 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:41.944+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.949276+0000) 2019-10-13T14:01:41.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:41.951+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:41.954998+0000) 2019-10-13T14:01:41.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:41.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195591 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:42.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:41.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:42.001438+0000) 2019-10-13T14:01:44.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:44.328+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:44.329678+0000) 2019-10-13T14:01:44.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:44.612+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:44.615277+0000) 2019-10-13T14:01:46.962 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:46.956+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.959769+0000) 2019-10-13T14:01:46.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:46.956+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.960178+0000) 2019-10-13T14:01:46.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:46.956+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.961074+0000) 2019-10-13T14:01:46.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:46.956+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.961511+0000) 2019-10-13T14:01:46.965 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:46.960+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.963162+0000) 2019-10-13T14:01:46.967 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:46.960+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.965074+0000) 2019-10-13T14:01:46.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:46.963+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.965945+0000) 2019-10-13T14:01:46.969 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:46.963+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:46.966461+0000) 2019-10-13T14:01:46.971 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:46.964+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:47.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:47.980+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:47.985347+0000) 2019-10-13T14:01:48.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:48.208+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:48.211447+0000) 2019-10-13T14:01:48.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:48.748+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:48.750039+0000) 2019-10-13T14:01:49.008 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:49.003+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:49.005516+0000) 2019-10-13T14:01:51.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:51.728+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.732006+0000) 2019-10-13T14:01:51.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:51.968+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.972391+0000) 2019-10-13T14:01:51.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:51.972+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.973922+0000) 2019-10-13T14:01:51.977 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:51.972+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.974219+0000) 2019-10-13T14:01:51.978 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:51.972+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.973957+0000) 2019-10-13T14:01:51.978 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:51.972+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.974639+0000) 2019-10-13T14:01:51.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:51.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.975463+0000) 2019-10-13T14:01:51.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:51.972+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.974998+0000) 2019-10-13T14:01:51.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:51.975+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:51.979722+0000) 2019-10-13T14:01:51.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:51.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195650 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:52.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:51.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:52.001617+0000) 2019-10-13T14:01:54.332 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:54.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:54.329907+0000) 2019-10-13T14:01:54.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:54.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:54.615520+0000) 2019-10-13T14:01:56.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:56.983+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.986365+0000) 2019-10-13T14:01:56.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:56.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.986686+0000) 2019-10-13T14:01:56.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:01:56.987+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.989415+0000) 2019-10-13T14:01:56.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:01:56.983+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.989195+0000) 2019-10-13T14:01:56.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:56.987+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.990255+0000) 2019-10-13T14:01:56.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:01:56.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.991847+0000) 2019-10-13T14:01:56.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:01:56.991+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:56.994000+0000) 2019-10-13T14:01:56.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:01:56.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195679 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:01:57.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:56.999+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:57.001754+0000) 2019-10-13T14:01:57.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:01:57.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:57.985505+0000) 2019-10-13T14:01:58.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:01:58.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:58.211649+0000) 2019-10-13T14:01:58.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:01:58.747+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:58.750230+0000) 2019-10-13T14:01:59.019 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:01:59.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:01:59.016591+0000) 2019-10-13T14:02:01.599 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:02:01.600 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:01.603 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:02:01.603 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:01.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:01.731+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:01.732214+0000) 2019-10-13T14:02:02.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:01.995+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:01.998951+0000) 2019-10-13T14:02:02.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:01.995+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:01.999079+0000) 2019-10-13T14:02:02.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:01.995+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.000366+0000) 2019-10-13T14:02:02.003 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:01.995+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.000615+0000) 2019-10-13T14:02:02.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:01.995+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.001322+0000) 2019-10-13T14:02:02.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:01.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.001772+0000) 2019-10-13T14:02:02.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:01.999+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.002255+0000) 2019-10-13T14:02:02.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:01.999+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.002374+0000) 2019-10-13T14:02:02.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:02.007+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:02.008202+0000) 2019-10-13T14:02:02.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:02.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:04.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:04.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:04.330117+0000) 2019-10-13T14:02:04.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:04.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:04.615724+0000) 2019-10-13T14:02:07.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:07.007+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.012192+0000) 2019-10-13T14:02:07.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:07.011+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.012903+0000) 2019-10-13T14:02:07.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:07.011+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.014730+0000) 2019-10-13T14:02:07.018 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:07.011+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.015120+0000) 2019-10-13T14:02:07.019 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:07.011+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.016936+0000) 2019-10-13T14:02:07.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:07.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.017475+0000) 2019-10-13T14:02:07.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:07.015+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.019750+0000) 2019-10-13T14:02:07.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:07.019+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195741 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:07.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:07.027+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.028818+0000) 2019-10-13T14:02:07.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:07.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:07.985689+0000) 2019-10-13T14:02:08.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:08.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:08.211849+0000) 2019-10-13T14:02:08.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:08.747+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:08.750416+0000) 2019-10-13T14:02:09.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:09.019+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:09.023092+0000) 2019-10-13T14:02:11.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:11.731+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:11.732417+0000) 2019-10-13T14:02:12.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:11.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.001957+0000) 2019-10-13T14:02:12.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:12.019+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.025407+0000) 2019-10-13T14:02:12.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:12.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.026072+0000) 2019-10-13T14:02:12.029 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:12.023+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.026645+0000) 2019-10-13T14:02:12.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:12.023+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.026995+0000) 2019-10-13T14:02:12.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:12.023+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.027828+0000) 2019-10-13T14:02:12.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:12.023+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.028620+0000) 2019-10-13T14:02:12.032 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:12.023+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.029000+0000) 2019-10-13T14:02:12.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:12.035+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:12.036282+0000) 2019-10-13T14:02:12.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:12.031+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:14.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:14.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:14.330346+0000) 2019-10-13T14:02:14.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:14.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:14.615926+0000) 2019-10-13T14:02:17.042 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:17.035+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.039658+0000) 2019-10-13T14:02:17.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:17.035+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.041356+0000) 2019-10-13T14:02:17.045 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:17.039+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.042748+0000) 2019-10-13T14:02:17.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:17.039+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.042082+0000) 2019-10-13T14:02:17.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:17.039+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.043309+0000) 2019-10-13T14:02:17.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:17.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.043775+0000) 2019-10-13T14:02:17.048 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:17.043+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.046186+0000) 2019-10-13T14:02:17.052 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:17.043+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:17.054 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:17.051+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.052206+0000) 2019-10-13T14:02:17.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:17.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:17.985924+0000) 2019-10-13T14:02:18.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:18.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:18.212093+0000) 2019-10-13T14:02:18.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:18.747+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:18.750644+0000) 2019-10-13T14:02:19.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:19.027+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:19.029583+0000) 2019-10-13T14:02:21.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:21.727+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:21.732624+0000) 2019-10-13T14:02:22.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:21.999+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.002149+0000) 2019-10-13T14:02:22.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:22.047+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.052585+0000) 2019-10-13T14:02:22.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:22.047+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.053484+0000) 2019-10-13T14:02:22.058 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:22.051+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.055296+0000) 2019-10-13T14:02:22.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:22.051+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.055453+0000) 2019-10-13T14:02:22.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:22.051+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.055816+0000) 2019-10-13T14:02:22.059 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:22.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.056231+0000) 2019-10-13T14:02:22.059 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:22.051+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.056586+0000) 2019-10-13T14:02:22.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:22.055+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:22.059643+0000) 2019-10-13T14:02:22.065 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:22.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195835 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:24.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:24.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:24.330575+0000) 2019-10-13T14:02:24.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:24.611+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:24.616119+0000) 2019-10-13T14:02:27.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:27.063+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.065871+0000) 2019-10-13T14:02:27.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:27.063+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.066663+0000) 2019-10-13T14:02:27.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:27.063+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.068208+0000) 2019-10-13T14:02:27.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:27.066+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.068673+0000) 2019-10-13T14:02:27.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:27.063+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.068582+0000) 2019-10-13T14:02:27.072 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:27.063+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.069166+0000) 2019-10-13T14:02:27.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:27.063+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.068920+0000) 2019-10-13T14:02:27.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:27.070+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.074466+0000) 2019-10-13T14:02:27.079 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:27.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:27.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:27.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:27.986126+0000) 2019-10-13T14:02:28.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:28.207+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:28.212295+0000) 2019-10-13T14:02:28.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:28.747+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:28.750869+0000) 2019-10-13T14:02:29.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:29.030+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:29.035744+0000) 2019-10-13T14:02:31.646 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:02:31.647 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:31.650 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:02:31.650 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:02:31.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:31.727+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:31.732852+0000) 2019-10-13T14:02:32.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:31.998+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.002299+0000) 2019-10-13T14:02:32.083 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:32.075+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.080251+0000) 2019-10-13T14:02:32.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:32.079+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.082113+0000) 2019-10-13T14:02:32.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:32.079+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.082698+0000) 2019-10-13T14:02:32.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:32.079+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.083385+0000) 2019-10-13T14:02:32.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:32.078+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.083903+0000) 2019-10-13T14:02:32.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:32.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.084224+0000) 2019-10-13T14:02:32.087 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:32.079+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.084843+0000) 2019-10-13T14:02:32.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:32.083+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:32.093 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:32.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:32.091276+0000) 2019-10-13T14:02:34.333 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:34.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:34.330803+0000) 2019-10-13T14:02:34.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:34.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:34.616316+0000) 2019-10-13T14:02:37.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:37.087+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.093426+0000) 2019-10-13T14:02:37.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:37.091+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.093862+0000) 2019-10-13T14:02:37.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:37.091+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.094423+0000) 2019-10-13T14:02:37.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:37.091+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.094978+0000) 2019-10-13T14:02:37.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:37.091+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.095411+0000) 2019-10-13T14:02:37.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:37.090+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.096006+0000) 2019-10-13T14:02:37.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:37.091+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.096708+0000) 2019-10-13T14:02:37.105 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:37.099+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:37.106 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:37.098+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.103051+0000) 2019-10-13T14:02:37.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:37.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:37.986258+0000) 2019-10-13T14:02:38.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:38.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:38.212525+0000) 2019-10-13T14:02:38.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:38.747+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:38.751065+0000) 2019-10-13T14:02:39.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:39.042+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:39.045629+0000) 2019-10-13T14:02:41.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:41.727+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:41.733081+0000) 2019-10-13T14:02:42.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:41.998+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.002496+0000) 2019-10-13T14:02:42.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:42.103+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.106470+0000) 2019-10-13T14:02:42.109 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:42.103+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.107068+0000) 2019-10-13T14:02:42.111 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:42.103+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.108675+0000) 2019-10-13T14:02:42.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:42.103+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.108824+0000) 2019-10-13T14:02:42.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:42.103+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.109272+0000) 2019-10-13T14:02:42.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:42.107+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.109837+0000) 2019-10-13T14:02:42.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:42.106+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.110606+0000) 2019-10-13T14:02:42.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:42.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:42.113734+0000) 2019-10-13T14:02:42.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:42.111+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195959 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:44.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:44.327+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:44.331005+0000) 2019-10-13T14:02:44.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:44.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:44.616516+0000) 2019-10-13T14:02:47.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:47.115+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.119180+0000) 2019-10-13T14:02:47.122 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:47.115+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.119509+0000) 2019-10-13T14:02:47.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:47.115+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.120659+0000) 2019-10-13T14:02:47.124 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:47.118+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.121390+0000) 2019-10-13T14:02:47.124 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:47.119+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.121893+0000) 2019-10-13T14:02:47.125 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:47.119+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.122196+0000) 2019-10-13T14:02:47.125 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:47.119+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.122873+0000) 2019-10-13T14:02:47.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:47.126+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.128082+0000) 2019-10-13T14:02:47.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:47.123+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:47.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:47.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:47.986422+0000) 2019-10-13T14:02:48.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:48.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:48.212745+0000) 2019-10-13T14:02:48.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:48.750+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:48.751277+0000) 2019-10-13T14:02:49.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:49.050+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:49.052224+0000) 2019-10-13T14:02:51.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:51.726+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:51.733283+0000) 2019-10-13T14:02:52.005 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:52.002+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.002688+0000) 2019-10-13T14:02:52.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:52.126+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.132021+0000) 2019-10-13T14:02:52.135 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:52.126+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.132723+0000) 2019-10-13T14:02:52.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:52.126+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.133713+0000) 2019-10-13T14:02:52.138 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:52.130+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.135444+0000) 2019-10-13T14:02:52.138 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:52.130+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.135754+0000) 2019-10-13T14:02:52.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:52.134+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.136383+0000) 2019-10-13T14:02:52.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:52.130+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.137009+0000) 2019-10-13T14:02:52.142 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:52.138+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:52.140263+0000) 2019-10-13T14:02:52.146 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:52.138+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:54.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:54.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:54.331237+0000) 2019-10-13T14:02:54.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:54.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:54.616755+0000) 2019-10-13T14:02:57.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:02:57.142+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.147142+0000) 2019-10-13T14:02:57.151 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:57.142+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.148840+0000) 2019-10-13T14:02:57.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:02:57.142+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.149664+0000) 2019-10-13T14:02:57.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:57.146+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.151796+0000) 2019-10-13T14:02:57.155 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:57.146+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.151985+0000) 2019-10-13T14:02:57.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:02:57.146+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.153409+0000) 2019-10-13T14:02:57.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:57.150+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.153606+0000) 2019-10-13T14:02:57.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:02:57.150+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.155278+0000) 2019-10-13T14:02:57.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:02:57.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196051 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:02:57.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:02:57.982+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:57.986599+0000) 2019-10-13T14:02:58.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:02:58.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:58.212944+0000) 2019-10-13T14:02:58.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:02:58.750+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:58.751463+0000) 2019-10-13T14:02:59.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:02:59.058+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:02:59.058641+0000) 2019-10-13T14:03:01.711 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:03:01.711 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:01.715 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:03:01.715 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:01.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:01.726+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:01.733481+0000) 2019-10-13T14:03:02.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:01.998+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.002890+0000) 2019-10-13T14:03:02.164 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:02.154+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.161270+0000) 2019-10-13T14:03:02.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:02.158+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.162454+0000) 2019-10-13T14:03:02.166 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:02.158+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.163435+0000) 2019-10-13T14:03:02.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:02.162+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.164076+0000) 2019-10-13T14:03:02.167 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:02.158+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.164111+0000) 2019-10-13T14:03:02.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:02.158+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.164994+0000) 2019-10-13T14:03:02.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:02.158+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.165043+0000) 2019-10-13T14:03:02.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:02.166+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:02.169353+0000) 2019-10-13T14:03:02.173 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:02.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196083 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:04.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:04.326+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:04.331453+0000) 2019-10-13T14:03:04.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:04.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:04.616925+0000) 2019-10-13T14:03:07.176 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:07.170+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.173876+0000) 2019-10-13T14:03:07.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:07.170+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.175769+0000) 2019-10-13T14:03:07.179 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:07.170+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.176042+0000) 2019-10-13T14:03:07.180 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:07.174+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.177722+0000) 2019-10-13T14:03:07.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:07.174+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.178593+0000) 2019-10-13T14:03:07.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:07.174+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.179079+0000) 2019-10-13T14:03:07.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:07.174+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.179650+0000) 2019-10-13T14:03:07.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:07.178+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196112 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:07.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:07.182+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.184271+0000) 2019-10-13T14:03:07.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:07.982+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:07.986820+0000) 2019-10-13T14:03:08.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:08.206+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:08.213143+0000) 2019-10-13T14:03:08.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:08.750+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:08.751665+0000) 2019-10-13T14:03:09.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:09.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:09.069680+0000) 2019-10-13T14:03:11.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:11.726+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:11.733706+0000) 2019-10-13T14:03:12.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:11.998+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.003120+0000) 2019-10-13T14:03:12.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:12.182+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.186544+0000) 2019-10-13T14:03:12.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:12.182+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.187518+0000) 2019-10-13T14:03:12.190 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:12.182+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.187638+0000) 2019-10-13T14:03:12.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:12.182+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.187941+0000) 2019-10-13T14:03:12.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:12.182+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.188898+0000) 2019-10-13T14:03:12.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:12.182+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.189352+0000) 2019-10-13T14:03:12.193 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:12.186+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.190328+0000) 2019-10-13T14:03:12.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:12.190+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:12.195557+0000) 2019-10-13T14:03:12.199 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:12.190+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:14.334 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:14.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:14.331658+0000) 2019-10-13T14:03:14.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:14.610+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:14.617141+0000) 2019-10-13T14:03:17.202 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:17.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.199705+0000) 2019-10-13T14:03:17.203 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:17.194+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.200063+0000) 2019-10-13T14:03:17.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:17.194+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.200571+0000) 2019-10-13T14:03:17.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:17.194+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.201627+0000) 2019-10-13T14:03:17.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:17.198+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.202071+0000) 2019-10-13T14:03:17.205 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:17.198+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.202617+0000) 2019-10-13T14:03:17.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:17.198+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.203149+0000) 2019-10-13T14:03:17.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:17.206+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.209064+0000) 2019-10-13T14:03:17.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:17.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196174 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:17.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:17.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:17.986996+0000) 2019-10-13T14:03:18.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:18.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:18.213402+0000) 2019-10-13T14:03:18.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:18.750+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:18.751864+0000) 2019-10-13T14:03:19.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:19.074+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:19.076191+0000) 2019-10-13T14:03:21.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:21.730+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:21.733944+0000) 2019-10-13T14:03:22.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:22.002+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.003282+0000) 2019-10-13T14:03:22.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:22.210+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.212933+0000) 2019-10-13T14:03:22.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:22.210+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.214250+0000) 2019-10-13T14:03:22.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:22.210+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.214611+0000) 2019-10-13T14:03:22.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:22.210+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.215117+0000) 2019-10-13T14:03:22.218 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:22.210+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.215578+0000) 2019-10-13T14:03:22.219 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:22.210+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.216017+0000) 2019-10-13T14:03:22.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:22.210+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.216784+0000) 2019-10-13T14:03:22.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:22.214+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:22.219785+0000) 2019-10-13T14:03:22.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:22.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196205 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:24.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:24.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:24.331854+0000) 2019-10-13T14:03:24.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:24.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:24.617343+0000) 2019-10-13T14:03:27.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:27.222+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.226111+0000) 2019-10-13T14:03:27.230 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:27.222+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.226768+0000) 2019-10-13T14:03:27.230 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:27.222+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.227155+0000) 2019-10-13T14:03:27.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:27.226+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.228180+0000) 2019-10-13T14:03:27.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:27.222+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.227646+0000) 2019-10-13T14:03:27.231 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:27.222+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.228311+0000) 2019-10-13T14:03:27.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:27.222+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.228889+0000) 2019-10-13T14:03:27.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:27.226+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.231740+0000) 2019-10-13T14:03:27.238 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:27.230+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:27.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:27.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:27.987176+0000) 2019-10-13T14:03:28.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:28.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:28.213606+0000) 2019-10-13T14:03:28.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:28.746+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:28.752081+0000) 2019-10-13T14:03:29.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:29.081+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:29.087566+0000) 2019-10-13T14:03:31.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:31.730+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:31.734165+0000) 2019-10-13T14:03:31.826 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:03:31.826 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:31.830 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:03:31.830 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:03:32.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:31.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.003477+0000) 2019-10-13T14:03:32.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:32.234+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.239615+0000) 2019-10-13T14:03:32.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:32.234+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.241435+0000) 2019-10-13T14:03:32.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:32.238+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.241989+0000) 2019-10-13T14:03:32.246 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:32.238+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.243692+0000) 2019-10-13T14:03:32.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:32.241+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.244616+0000) 2019-10-13T14:03:32.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:32.241+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.245428+0000) 2019-10-13T14:03:32.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:32.242+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.247010+0000) 2019-10-13T14:03:32.251 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:32.242+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:32.248945+0000) 2019-10-13T14:03:32.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:32.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:34.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:34.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:34.332079+0000) 2019-10-13T14:03:34.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:34.614+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:34.617558+0000) 2019-10-13T14:03:37.255 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:37.246+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.252769+0000) 2019-10-13T14:03:37.256 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:37.246+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.252946+0000) 2019-10-13T14:03:37.257 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:37.250+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.254148+0000) 2019-10-13T14:03:37.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:37.249+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.255007+0000) 2019-10-13T14:03:37.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:37.250+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.255507+0000) 2019-10-13T14:03:37.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:37.250+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.256083+0000) 2019-10-13T14:03:37.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:37.254+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.257874+0000) 2019-10-13T14:03:37.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:37.257+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.261646+0000) 2019-10-13T14:03:37.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:37.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196298 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:37.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:37.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:37.987355+0000) 2019-10-13T14:03:38.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:38.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:38.213812+0000) 2019-10-13T14:03:38.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:38.750+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:38.752278+0000) 2019-10-13T14:03:39.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:39.089+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:39.094153+0000) 2019-10-13T14:03:41.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:41.730+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:41.734413+0000) 2019-10-13T14:03:42.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:41.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.003676+0000) 2019-10-13T14:03:42.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:42.262+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.266788+0000) 2019-10-13T14:03:42.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:42.262+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.268899+0000) 2019-10-13T14:03:42.272 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:42.265+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.269300+0000) 2019-10-13T14:03:42.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:42.266+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.270550+0000) 2019-10-13T14:03:42.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:42.266+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.271877+0000) 2019-10-13T14:03:42.275 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:42.266+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.272205+0000) 2019-10-13T14:03:42.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:42.270+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.274377+0000) 2019-10-13T14:03:42.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:42.273+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:42.276533+0000) 2019-10-13T14:03:42.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:42.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:44.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:44.325+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:44.332297+0000) 2019-10-13T14:03:44.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:44.613+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:44.617779+0000) 2019-10-13T14:03:47.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:47.273+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.280645+0000) 2019-10-13T14:03:47.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:47.273+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.281657+0000) 2019-10-13T14:03:47.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:47.277+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.282232+0000) 2019-10-13T14:03:47.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:47.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.282450+0000) 2019-10-13T14:03:47.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:47.277+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.283143+0000) 2019-10-13T14:03:47.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:47.277+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.283928+0000) 2019-10-13T14:03:47.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:47.277+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.284494+0000) 2019-10-13T14:03:47.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:47.281+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.288015+0000) 2019-10-13T14:03:47.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:47.285+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:47.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:47.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:47.987542+0000) 2019-10-13T14:03:48.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:48.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:48.213997+0000) 2019-10-13T14:03:48.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:48.745+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:48.752439+0000) 2019-10-13T14:03:49.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:49.101+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:49.105557+0000) 2019-10-13T14:03:51.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:51.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:51.734636+0000) 2019-10-13T14:03:52.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:51.997+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.003900+0000) 2019-10-13T14:03:52.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:52.289+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.294508+0000) 2019-10-13T14:03:52.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:52.289+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.294924+0000) 2019-10-13T14:03:52.298 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:52.289+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.295896+0000) 2019-10-13T14:03:52.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:52.293+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.297939+0000) 2019-10-13T14:03:52.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:52.293+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.298125+0000) 2019-10-13T14:03:52.302 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:52.293+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.299976+0000) 2019-10-13T14:03:52.303 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:52.293+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.300618+0000) 2019-10-13T14:03:52.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:52.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196390 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:52.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:52.305+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:52.308871+0000) 2019-10-13T14:03:54.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:54.329+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:54.332538+0000) 2019-10-13T14:03:54.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:54.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:54.617994+0000) 2019-10-13T14:03:57.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:57.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.308441+0000) 2019-10-13T14:03:57.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:03:57.301+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.309078+0000) 2019-10-13T14:03:57.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:57.305+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.309798+0000) 2019-10-13T14:03:57.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:57.305+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.311576+0000) 2019-10-13T14:03:57.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:03:57.309+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.313888+0000) 2019-10-13T14:03:57.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:03:57.309+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.313987+0000) 2019-10-13T14:03:57.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:03:57.309+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.314908+0000) 2019-10-13T14:03:57.321 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:03:57.313+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:03:57.323 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:57.317+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.321229+0000) 2019-10-13T14:03:57.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:03:57.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:57.987713+0000) 2019-10-13T14:03:58.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:03:58.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:58.214196+0000) 2019-10-13T14:03:58.755 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:03:58.749+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:58.752689+0000) 2019-10-13T14:03:59.114 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:03:59.109+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:03:59.111926+0000) 2019-10-13T14:04:01.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:01.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:01.734854+0000) 2019-10-13T14:04:01.907 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:04:01.907 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:01.911 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:04:01.911 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:02.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:02.001+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.004136+0000) 2019-10-13T14:04:02.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:02.321+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.322924+0000) 2019-10-13T14:04:02.327 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:02.321+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.323362+0000) 2019-10-13T14:04:02.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:02.321+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.324074+0000) 2019-10-13T14:04:02.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:02.321+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.324206+0000) 2019-10-13T14:04:02.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:02.321+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.324710+0000) 2019-10-13T14:04:02.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:02.321+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.325346+0000) 2019-10-13T14:04:02.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:02.321+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.325351+0000) 2019-10-13T14:04:02.333 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:02.329+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:02.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:02.325+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:02.331610+0000) 2019-10-13T14:04:04.335 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:04.329+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:04.332728+0000) 2019-10-13T14:04:04.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:04.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:04.618230+0000) 2019-10-13T14:04:07.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:07.333+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.334624+0000) 2019-10-13T14:04:07.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:07.333+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.335630+0000) 2019-10-13T14:04:07.339 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:07.333+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.336052+0000) 2019-10-13T14:04:07.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:07.333+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.336621+0000) 2019-10-13T14:04:07.339 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:07.333+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.336682+0000) 2019-10-13T14:04:07.340 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:07.333+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.337397+0000) 2019-10-13T14:04:07.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:07.337+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.338091+0000) 2019-10-13T14:04:07.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:07.337+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.341190+0000) 2019-10-13T14:04:07.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:07.341+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:07.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:07.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:07.987928+0000) 2019-10-13T14:04:08.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:08.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:08.214400+0000) 2019-10-13T14:04:08.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:08.749+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:08.752951+0000) 2019-10-13T14:04:09.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:09.117+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:09.117988+0000) 2019-10-13T14:04:11.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:11.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:11.735059+0000) 2019-10-13T14:04:12.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:12.001+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.004375+0000) 2019-10-13T14:04:12.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:12.345+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.348091+0000) 2019-10-13T14:04:12.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:12.345+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.348408+0000) 2019-10-13T14:04:12.352 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:12.345+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.349219+0000) 2019-10-13T14:04:12.352 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:12.345+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.348723+0000) 2019-10-13T14:04:12.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:12.349+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.349564+0000) 2019-10-13T14:04:12.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:12.349+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.350123+0000) 2019-10-13T14:04:12.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:12.349+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.350743+0000) 2019-10-13T14:04:12.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:12.357+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196513 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:12.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:12.357+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:12.360387+0000) 2019-10-13T14:04:14.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:14.329+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:14.332931+0000) 2019-10-13T14:04:14.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:14.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:14.618423+0000) 2019-10-13T14:04:17.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:17.357+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.361940+0000) 2019-10-13T14:04:17.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:17.361+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.362531+0000) 2019-10-13T14:04:17.366 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:17.361+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.363846+0000) 2019-10-13T14:04:17.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:17.361+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.364257+0000) 2019-10-13T14:04:17.369 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:17.365+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.366302+0000) 2019-10-13T14:04:17.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:17.365+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.368331+0000) 2019-10-13T14:04:17.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:17.365+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.369151+0000) 2019-10-13T14:04:17.373 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:17.369+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.370279+0000) 2019-10-13T14:04:17.375 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:17.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:17.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:17.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:17.988118+0000) 2019-10-13T14:04:18.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:18.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:18.214595+0000) 2019-10-13T14:04:18.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:18.749+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:18.753176+0000) 2019-10-13T14:04:19.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:19.121+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:19.124468+0000) 2019-10-13T14:04:21.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:21.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:21.735263+0000) 2019-10-13T14:04:22.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:22.001+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.004611+0000) 2019-10-13T14:04:22.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:22.373+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.376730+0000) 2019-10-13T14:04:22.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:22.373+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.377343+0000) 2019-10-13T14:04:22.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:22.377+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.379654+0000) 2019-10-13T14:04:22.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:22.377+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.380115+0000) 2019-10-13T14:04:22.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:22.381+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.381904+0000) 2019-10-13T14:04:22.386 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:22.377+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.383991+0000) 2019-10-13T14:04:22.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:22.381+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.385331+0000) 2019-10-13T14:04:22.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:22.385+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:22.386780+0000) 2019-10-13T14:04:22.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:22.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196576 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:24.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:24.329+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:24.333133+0000) 2019-10-13T14:04:24.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:24.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:24.618648+0000) 2019-10-13T14:04:27.394 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:27.385+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.391638+0000) 2019-10-13T14:04:27.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:27.389+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.393048+0000) 2019-10-13T14:04:27.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:27.393+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.393899+0000) 2019-10-13T14:04:27.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:27.389+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.393334+0000) 2019-10-13T14:04:27.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:27.393+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.394332+0000) 2019-10-13T14:04:27.398 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:27.393+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.394663+0000) 2019-10-13T14:04:27.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:27.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.395422+0000) 2019-10-13T14:04:27.399 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:27.393+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.397104+0000) 2019-10-13T14:04:27.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:27.397+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:27.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:27.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:27.988332+0000) 2019-10-13T14:04:28.217 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:28.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:28.214812+0000) 2019-10-13T14:04:28.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:28.749+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:28.753386+0000) 2019-10-13T14:04:29.133 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:29.129+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:29.130406+0000) 2019-10-13T14:04:31.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:31.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:31.735461+0000) 2019-10-13T14:04:32.005 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:04:32.006 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:32.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:32.001+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.004811+0000) 2019-10-13T14:04:32.010 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:04:32.010 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:04:32.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:32.401+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.403319+0000) 2019-10-13T14:04:32.407 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:32.401+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.404316+0000) 2019-10-13T14:04:32.407 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:32.401+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.404898+0000) 2019-10-13T14:04:32.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:32.401+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.405573+0000) 2019-10-13T14:04:32.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:32.405+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.406488+0000) 2019-10-13T14:04:32.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:32.405+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.407318+0000) 2019-10-13T14:04:32.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:32.405+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.407945+0000) 2019-10-13T14:04:32.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:32.409+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:32.412286+0000) 2019-10-13T14:04:32.415 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:32.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196637 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:34.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:34.329+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:34.333301+0000) 2019-10-13T14:04:34.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:34.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:34.618868+0000) 2019-10-13T14:04:37.419 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:37.413+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.416684+0000) 2019-10-13T14:04:37.421 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:37.412+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.418418+0000) 2019-10-13T14:04:37.421 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:37.417+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.418531+0000) 2019-10-13T14:04:37.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:37.417+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.418912+0000) 2019-10-13T14:04:37.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:37.417+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.419408+0000) 2019-10-13T14:04:37.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:37.417+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.419726+0000) 2019-10-13T14:04:37.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:37.417+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.420283+0000) 2019-10-13T14:04:37.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:37.416+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.423627+0000) 2019-10-13T14:04:37.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:37.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:37.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:37.985+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:37.988500+0000) 2019-10-13T14:04:38.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:38.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:38.214998+0000) 2019-10-13T14:04:38.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:38.749+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:38.753588+0000) 2019-10-13T14:04:39.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:39.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:39.137038+0000) 2019-10-13T14:04:41.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:41.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:41.735663+0000) 2019-10-13T14:04:42.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:42.004+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.005003+0000) 2019-10-13T14:04:42.434 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:42.428+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.430998+0000) 2019-10-13T14:04:42.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:42.428+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.431609+0000) 2019-10-13T14:04:42.435 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:42.428+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.431983+0000) 2019-10-13T14:04:42.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:42.428+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.432428+0000) 2019-10-13T14:04:42.436 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:42.428+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.433449+0000) 2019-10-13T14:04:42.436 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:42.428+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.433637+0000) 2019-10-13T14:04:42.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:42.432+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.434158+0000) 2019-10-13T14:04:42.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:42.432+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:42.436685+0000) 2019-10-13T14:04:42.442 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:42.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196700 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:44.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:44.328+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:44.333526+0000) 2019-10-13T14:04:44.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:44.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:44.619083+0000) 2019-10-13T14:04:47.448 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:47.440+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.443306+0000) 2019-10-13T14:04:47.449 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:47.440+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.444016+0000) 2019-10-13T14:04:47.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:47.440+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.444282+0000) 2019-10-13T14:04:47.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:47.440+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.444886+0000) 2019-10-13T14:04:47.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:47.440+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.445573+0000) 2019-10-13T14:04:47.449 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:47.440+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.445683+0000) 2019-10-13T14:04:47.450 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:47.440+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.446400+0000) 2019-10-13T14:04:47.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:47.444+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.451088+0000) 2019-10-13T14:04:47.455 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:47.448+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:47.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:47.984+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:47.988697+0000) 2019-10-13T14:04:48.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:48.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:48.215190+0000) 2019-10-13T14:04:48.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:48.752+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:48.753781+0000) 2019-10-13T14:04:49.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:49.136+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:49.143566+0000) 2019-10-13T14:04:51.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:51.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:51.735899+0000) 2019-10-13T14:04:52.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:52.000+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.005225+0000) 2019-10-13T14:04:52.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:52.452+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.455950+0000) 2019-10-13T14:04:52.461 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:52.456+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.458700+0000) 2019-10-13T14:04:52.461 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:52.456+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.458850+0000) 2019-10-13T14:04:52.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:52.456+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.461101+0000) 2019-10-13T14:04:52.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:52.456+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.463482+0000) 2019-10-13T14:04:52.466 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:52.460+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.463931+0000) 2019-10-13T14:04:52.468 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:52.464+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.466163+0000) 2019-10-13T14:04:52.469 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:52.464+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196761 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:52.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:52.460+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:52.467266+0000) 2019-10-13T14:04:54.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:54.332+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:54.333727+0000) 2019-10-13T14:04:54.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:54.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:54.619307+0000) 2019-10-13T14:04:57.472 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:57.468+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.469810+0000) 2019-10-13T14:04:57.474 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:57.468+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.471552+0000) 2019-10-13T14:04:57.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:57.468+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.473653+0000) 2019-10-13T14:04:57.476 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:04:57.472+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.473996+0000) 2019-10-13T14:04:57.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:04:57.472+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.476596+0000) 2019-10-13T14:04:57.480 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:04:57.472+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.477171+0000) 2019-10-13T14:04:57.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:04:57.472+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.476475+0000) 2019-10-13T14:04:57.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:57.472+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.478396+0000) 2019-10-13T14:04:57.483 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:04:57.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:04:57.991 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:04:57.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:57.988905+0000) 2019-10-13T14:04:58.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:04:58.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:58.215368+0000) 2019-10-13T14:04:58.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:04:58.752+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:58.753979+0000) 2019-10-13T14:04:59.153 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:04:59.144+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:04:59.150225+0000) 2019-10-13T14:05:01.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:01.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:01.736113+0000) 2019-10-13T14:05:02.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:02.000+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.005448+0000) 2019-10-13T14:05:02.104 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:05:02.104 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:02.108 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:05:02.108 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:02.486 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:02.480+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.483555+0000) 2019-10-13T14:05:02.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:02.480+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.484417+0000) 2019-10-13T14:05:02.487 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:02.480+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.484682+0000) 2019-10-13T14:05:02.489 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:02.480+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.486938+0000) 2019-10-13T14:05:02.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:02.484+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.487347+0000) 2019-10-13T14:05:02.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:02.484+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.487152+0000) 2019-10-13T14:05:02.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:02.484+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.489637+0000) 2019-10-13T14:05:02.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:02.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:02.498 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:02.488+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:02.495927+0000) 2019-10-13T14:05:04.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:04.332+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:04.333962+0000) 2019-10-13T14:05:04.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:04.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:04.619510+0000) 2019-10-13T14:05:07.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:07.492+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.496828+0000) 2019-10-13T14:05:07.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:07.492+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.497144+0000) 2019-10-13T14:05:07.500 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:07.496+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.497950+0000) 2019-10-13T14:05:07.502 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:07.496+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.499946+0000) 2019-10-13T14:05:07.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:07.496+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.501676+0000) 2019-10-13T14:05:07.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:07.496+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.502120+0000) 2019-10-13T14:05:07.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:07.500+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.502671+0000) 2019-10-13T14:05:07.507 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:07.500+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.504781+0000) 2019-10-13T14:05:07.509 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:07.504+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:07.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:07.984+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:07.989143+0000) 2019-10-13T14:05:08.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:08.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:08.215506+0000) 2019-10-13T14:05:08.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:08.752+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:08.754205+0000) 2019-10-13T14:05:09.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:09.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:09.156956+0000) 2019-10-13T14:05:11.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:11.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:11.736310+0000) 2019-10-13T14:05:12.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:12.000+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.005660+0000) 2019-10-13T14:05:12.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:12.508+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.510372+0000) 2019-10-13T14:05:12.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:12.508+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.511478+0000) 2019-10-13T14:05:12.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:12.508+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.512332+0000) 2019-10-13T14:05:12.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:12.508+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.512500+0000) 2019-10-13T14:05:12.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:12.508+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.512898+0000) 2019-10-13T14:05:12.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:12.508+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.513429+0000) 2019-10-13T14:05:12.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:12.508+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.513851+0000) 2019-10-13T14:05:12.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:12.512+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:12.518857+0000) 2019-10-13T14:05:12.523 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:12.516+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196885 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:14.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:14.332+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:14.334169+0000) 2019-10-13T14:05:14.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:14.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:14.619703+0000) 2019-10-13T14:05:17.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:17.520+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.524412+0000) 2019-10-13T14:05:17.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:17.520+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.525178+0000) 2019-10-13T14:05:17.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:17.520+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.525407+0000) 2019-10-13T14:05:17.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:17.520+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.525637+0000) 2019-10-13T14:05:17.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:17.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.526389+0000) 2019-10-13T14:05:17.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:17.524+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.525944+0000) 2019-10-13T14:05:17.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:17.524+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.526390+0000) 2019-10-13T14:05:17.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:17.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.532853+0000) 2019-10-13T14:05:17.535 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:17.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196916 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:17.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:17.984+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:17.989279+0000) 2019-10-13T14:05:18.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:18.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:18.215677+0000) 2019-10-13T14:05:18.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:18.752+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:18.754438+0000) 2019-10-13T14:05:19.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:19.160+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:19.163610+0000) 2019-10-13T14:05:21.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:21.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:21.736552+0000) 2019-10-13T14:05:22.008 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:22.004+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.005809+0000) 2019-10-13T14:05:22.539 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:22.532+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.536272+0000) 2019-10-13T14:05:22.540 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:22.532+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.536800+0000) 2019-10-13T14:05:22.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:22.536+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.538505+0000) 2019-10-13T14:05:22.541 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:22.536+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.538918+0000) 2019-10-13T14:05:22.542 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:22.536+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.539434+0000) 2019-10-13T14:05:22.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:22.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.540656+0000) 2019-10-13T14:05:22.546 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:22.540+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.543658+0000) 2019-10-13T14:05:22.548 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:22.544+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196948 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:22.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:22.548+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:22.552671+0000) 2019-10-13T14:05:24.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:24.332+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:24.334400+0000) 2019-10-13T14:05:24.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:24.616+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:24.619907+0000) 2019-10-13T14:05:27.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:27.544+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.548938+0000) 2019-10-13T14:05:27.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:27.544+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.549462+0000) 2019-10-13T14:05:27.553 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:27.548+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.550777+0000) 2019-10-13T14:05:27.554 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:27.548+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.551115+0000) 2019-10-13T14:05:27.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:27.548+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.551688+0000) 2019-10-13T14:05:27.555 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:27.548+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.552299+0000) 2019-10-13T14:05:27.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:27.548+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.552206+0000) 2019-10-13T14:05:27.562 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:27.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:27.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:27.552+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.559828+0000) 2019-10-13T14:05:27.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:27.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:27.989491+0000) 2019-10-13T14:05:28.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:28.212+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:28.215910+0000) 2019-10-13T14:05:28.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:28.752+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:28.754638+0000) 2019-10-13T14:05:29.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:29.168+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:29.170040+0000) 2019-10-13T14:05:31.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:31.732+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:31.736776+0000) 2019-10-13T14:05:32.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:32.004+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.006004+0000) 2019-10-13T14:05:32.219 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:05:32.219 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:32.223 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:05:32.223 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:05:32.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:32.560+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.562964+0000) 2019-10-13T14:05:32.566 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:32.560+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.563415+0000) 2019-10-13T14:05:32.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:32.560+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.563921+0000) 2019-10-13T14:05:32.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:32.560+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.564724+0000) 2019-10-13T14:05:32.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:32.560+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.565205+0000) 2019-10-13T14:05:32.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:32.560+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.565678+0000) 2019-10-13T14:05:32.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:32.564+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.566154+0000) 2019-10-13T14:05:32.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:32.568+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:32.571309+0000) 2019-10-13T14:05:32.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:32.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:34.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:34.332+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:34.334601+0000) 2019-10-13T14:05:34.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:34.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:34.620125+0000) 2019-10-13T14:05:37.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:37.576+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.576465+0000) 2019-10-13T14:05:37.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:37.571+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.576895+0000) 2019-10-13T14:05:37.580 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:37.571+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.577608+0000) 2019-10-13T14:05:37.580 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:37.575+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.577736+0000) 2019-10-13T14:05:37.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:37.575+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.578262+0000) 2019-10-13T14:05:37.582 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:37.575+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.578678+0000) 2019-10-13T14:05:37.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:37.575+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.579147+0000) 2019-10-13T14:05:37.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:37.584+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.584928+0000) 2019-10-13T14:05:37.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:37.583+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:37.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:37.983+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:37.989699+0000) 2019-10-13T14:05:38.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:38.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:38.216091+0000) 2019-10-13T14:05:38.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:38.751+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:38.754861+0000) 2019-10-13T14:05:39.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:39.179+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:39.180835+0000) 2019-10-13T14:05:41.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:41.735+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:41.736979+0000) 2019-10-13T14:05:42.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:42.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.006206+0000) 2019-10-13T14:05:42.592 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:42.583+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.589352+0000) 2019-10-13T14:05:42.593 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:42.587+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.590356+0000) 2019-10-13T14:05:42.593 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:42.587+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.590058+0000) 2019-10-13T14:05:42.594 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:42.587+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.591140+0000) 2019-10-13T14:05:42.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:42.587+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.592026+0000) 2019-10-13T14:05:42.595 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:42.587+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.593111+0000) 2019-10-13T14:05:42.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:42.587+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.593603+0000) 2019-10-13T14:05:42.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:42.595+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:42.599373+0000) 2019-10-13T14:05:42.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:42.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:44.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:44.331+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:44.334802+0000) 2019-10-13T14:05:44.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:44.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:44.620357+0000) 2019-10-13T14:05:47.606 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:47.599+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.603129+0000) 2019-10-13T14:05:47.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:47.599+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.604712+0000) 2019-10-13T14:05:47.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:47.599+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.605064+0000) 2019-10-13T14:05:47.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:47.603+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.606052+0000) 2019-10-13T14:05:47.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:47.603+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.606981+0000) 2019-10-13T14:05:47.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:47.603+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.607886+0000) 2019-10-13T14:05:47.611 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:47.603+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.608256+0000) 2019-10-13T14:05:47.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:47.611+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.613181+0000) 2019-10-13T14:05:47.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:47.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:47.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:47.987+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:47.989950+0000) 2019-10-13T14:05:48.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:48.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:48.216312+0000) 2019-10-13T14:05:48.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:48.751+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:48.755054+0000) 2019-10-13T14:05:49.190 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:49.183+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:49.187390+0000) 2019-10-13T14:05:51.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:51.731+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:51.737178+0000) 2019-10-13T14:05:52.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:52.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.006371+0000) 2019-10-13T14:05:52.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:52.611+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.616600+0000) 2019-10-13T14:05:52.620 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:52.611+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.617165+0000) 2019-10-13T14:05:52.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:52.615+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.617606+0000) 2019-10-13T14:05:52.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:52.615+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.617935+0000) 2019-10-13T14:05:52.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:52.615+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.618567+0000) 2019-10-13T14:05:52.621 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:52.615+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.619022+0000) 2019-10-13T14:05:52.622 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:52.615+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.619162+0000) 2019-10-13T14:05:52.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:52.623+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:52.625910+0000) 2019-10-13T14:05:52.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:52.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:54.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:54.331+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:54.335022+0000) 2019-10-13T14:05:54.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:54.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:54.620554+0000) 2019-10-13T14:05:57.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:57.623+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.629476+0000) 2019-10-13T14:05:57.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:57.627+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.629778+0000) 2019-10-13T14:05:57.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:05:57.627+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.630428+0000) 2019-10-13T14:05:57.635 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:05:57.627+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.632476+0000) 2019-10-13T14:05:57.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:57.627+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.633075+0000) 2019-10-13T14:05:57.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:05:57.631+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.634169+0000) 2019-10-13T14:05:57.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:05:57.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.634352+0000) 2019-10-13T14:05:57.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:05:57.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:05:57.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:57.643+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.644247+0000) 2019-10-13T14:05:57.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:05:57.987+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:57.990122+0000) 2019-10-13T14:05:58.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:05:58.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:58.216489+0000) 2019-10-13T14:05:58.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:05:58.751+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:58.755250+0000) 2019-10-13T14:05:59.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:05:59.191+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:05:59.193920+0000) 2019-10-13T14:06:01.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:01.731+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:01.737417+0000) 2019-10-13T14:06:02.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:02.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.006580+0000) 2019-10-13T14:06:02.358 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:06:02.358 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:02.361 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:06:02.362 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:02.645 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:02.639+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.642711+0000) 2019-10-13T14:06:02.646 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:02.639+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.643349+0000) 2019-10-13T14:06:02.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:02.639+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.643754+0000) 2019-10-13T14:06:02.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:02.639+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.644161+0000) 2019-10-13T14:06:02.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:02.639+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.644665+0000) 2019-10-13T14:06:02.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:02.639+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.644992+0000) 2019-10-13T14:06:02.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:02.643+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.646220+0000) 2019-10-13T14:06:02.652 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:02.647+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:02.649368+0000) 2019-10-13T14:06:02.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:02.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197192 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:04.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:04.331+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:04.335255+0000) 2019-10-13T14:06:04.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:04.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:04.620756+0000) 2019-10-13T14:06:07.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:07.651+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.655121+0000) 2019-10-13T14:06:07.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:07.651+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.655496+0000) 2019-10-13T14:06:07.659 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:07.651+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.656096+0000) 2019-10-13T14:06:07.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:07.651+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.657164+0000) 2019-10-13T14:06:07.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:07.655+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.658436+0000) 2019-10-13T14:06:07.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:07.655+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.658425+0000) 2019-10-13T14:06:07.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:07.655+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.659132+0000) 2019-10-13T14:06:07.665 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:07.659+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.662518+0000) 2019-10-13T14:06:07.668 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:07.659+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197220 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:07.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:07.987+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:07.990296+0000) 2019-10-13T14:06:08.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:08.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:08.216689+0000) 2019-10-13T14:06:08.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:08.755+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:08.755475+0000) 2019-10-13T14:06:09.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:09.199+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:09.200505+0000) 2019-10-13T14:06:11.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:11.731+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:11.737624+0000) 2019-10-13T14:06:12.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:12.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.006806+0000) 2019-10-13T14:06:12.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:12.663+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.668390+0000) 2019-10-13T14:06:12.672 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:12.663+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.669155+0000) 2019-10-13T14:06:12.672 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:12.667+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.670015+0000) 2019-10-13T14:06:12.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:12.667+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.669886+0000) 2019-10-13T14:06:12.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:12.667+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.670027+0000) 2019-10-13T14:06:12.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:12.667+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.670689+0000) 2019-10-13T14:06:12.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:12.671+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.672484+0000) 2019-10-13T14:06:12.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:12.675+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:12.676728+0000) 2019-10-13T14:06:12.681 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:12.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:14.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:14.335+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:14.335452+0000) 2019-10-13T14:06:14.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:14.615+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:14.620987+0000) 2019-10-13T14:06:17.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:17.679+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.682011+0000) 2019-10-13T14:06:17.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:17.679+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.682447+0000) 2019-10-13T14:06:17.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:17.679+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.683889+0000) 2019-10-13T14:06:17.687 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:17.679+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.684428+0000) 2019-10-13T14:06:17.687 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:17.679+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.684661+0000) 2019-10-13T14:06:17.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:17.679+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.685237+0000) 2019-10-13T14:06:17.689 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:17.683+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.686365+0000) 2019-10-13T14:06:17.694 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:17.687+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:17.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:17.691+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.692394+0000) 2019-10-13T14:06:17.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:17.987+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:17.990507+0000) 2019-10-13T14:06:18.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:18.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:18.216868+0000) 2019-10-13T14:06:18.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:18.751+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:18.755621+0000) 2019-10-13T14:06:19.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:19.203+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:19.207006+0000) 2019-10-13T14:06:21.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:21.735+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:21.737917+0000) 2019-10-13T14:06:22.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:22.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.007006+0000) 2019-10-13T14:06:22.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:22.691+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.696768+0000) 2019-10-13T14:06:22.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:22.695+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.698589+0000) 2019-10-13T14:06:22.701 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:22.695+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.699016+0000) 2019-10-13T14:06:22.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:22.695+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.699368+0000) 2019-10-13T14:06:22.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:22.695+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.700080+0000) 2019-10-13T14:06:22.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:22.695+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.700372+0000) 2019-10-13T14:06:22.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:22.699+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.701987+0000) 2019-10-13T14:06:22.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:22.699+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197313 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:22.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:22.707+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:22.708832+0000) 2019-10-13T14:06:24.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:24.331+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:24.335622+0000) 2019-10-13T14:06:24.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:24.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:24.621193+0000) 2019-10-13T14:06:27.710 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:27.703+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.707016+0000) 2019-10-13T14:06:27.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:27.703+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.709560+0000) 2019-10-13T14:06:27.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:27.707+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.709792+0000) 2019-10-13T14:06:27.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:27.707+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.710397+0000) 2019-10-13T14:06:27.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:27.707+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.710874+0000) 2019-10-13T14:06:27.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:27.707+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.711011+0000) 2019-10-13T14:06:27.714 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:27.707+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.711402+0000) 2019-10-13T14:06:27.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:27.711+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.714314+0000) 2019-10-13T14:06:27.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:27.711+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:27.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:27.987+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:27.990692+0000) 2019-10-13T14:06:28.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:28.211+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:28.217063+0000) 2019-10-13T14:06:28.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:28.751+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:28.755815+0000) 2019-10-13T14:06:29.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:29.211+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:29.213539+0000) 2019-10-13T14:06:31.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:31.735+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:31.738122+0000) 2019-10-13T14:06:32.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:32.003+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.007228+0000) 2019-10-13T14:06:32.481 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:06:32.481 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:32.485 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:06:32.485 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:06:32.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:32.714+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.719917+0000) 2019-10-13T14:06:32.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:32.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.721441+0000) 2019-10-13T14:06:32.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:32.718+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.723902+0000) 2019-10-13T14:06:32.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:32.718+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.724339+0000) 2019-10-13T14:06:32.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:32.723+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.724824+0000) 2019-10-13T14:06:32.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:32.722+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.726274+0000) 2019-10-13T14:06:32.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:32.726+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.730237+0000) 2019-10-13T14:06:32.734 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:32.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:32.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:32.727+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:32.731820+0000) 2019-10-13T14:06:34.338 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:34.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:34.335842+0000) 2019-10-13T14:06:34.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:34.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:34.621436+0000) 2019-10-13T14:06:37.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:37.730+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.735907+0000) 2019-10-13T14:06:37.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:37.730+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.737014+0000) 2019-10-13T14:06:37.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:37.735+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.737752+0000) 2019-10-13T14:06:37.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:37.734+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.738081+0000) 2019-10-13T14:06:37.741 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:37.734+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.738379+0000) 2019-10-13T14:06:37.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:37.734+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.739121+0000) 2019-10-13T14:06:37.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:37.734+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.740912+0000) 2019-10-13T14:06:37.745 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:37.739+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.742878+0000) 2019-10-13T14:06:37.747 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:37.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:37.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:37.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:37.990840+0000) 2019-10-13T14:06:38.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:38.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:38.217238+0000) 2019-10-13T14:06:38.758 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:38.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:38.756002+0000) 2019-10-13T14:06:39.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:39.219+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:39.220053+0000) 2019-10-13T14:06:41.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:41.734+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:41.738333+0000) 2019-10-13T14:06:42.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:42.007+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.007422+0000) 2019-10-13T14:06:42.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:42.742+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.747403+0000) 2019-10-13T14:06:42.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:42.742+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.747518+0000) 2019-10-13T14:06:42.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:42.742+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.748080+0000) 2019-10-13T14:06:42.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:42.746+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.748986+0000) 2019-10-13T14:06:42.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:42.746+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.749882+0000) 2019-10-13T14:06:42.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:42.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.749779+0000) 2019-10-13T14:06:42.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:42.746+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.750593+0000) 2019-10-13T14:06:42.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:42.746+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:42.751556+0000) 2019-10-13T14:06:42.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:42.750+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:44.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:44.334+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:44.336075+0000) 2019-10-13T14:06:44.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:44.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:44.621669+0000) 2019-10-13T14:06:47.762 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:47.754+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.759238+0000) 2019-10-13T14:06:47.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:47.758+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.761138+0000) 2019-10-13T14:06:47.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:47.754+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.761398+0000) 2019-10-13T14:06:47.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:47.758+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.761732+0000) 2019-10-13T14:06:47.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:47.758+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.762875+0000) 2019-10-13T14:06:47.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:47.758+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.763129+0000) 2019-10-13T14:06:47.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:47.758+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.765386+0000) 2019-10-13T14:06:47.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:47.762+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.766020+0000) 2019-10-13T14:06:47.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:47.766+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197464 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:47.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:47.990+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:47.991023+0000) 2019-10-13T14:06:48.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:48.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:48.217413+0000) 2019-10-13T14:06:48.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:48.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:48.756234+0000) 2019-10-13T14:06:49.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:49.222+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:49.226572+0000) 2019-10-13T14:06:51.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:51.734+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:51.738531+0000) 2019-10-13T14:06:52.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:52.002+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.007570+0000) 2019-10-13T14:06:52.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:52.770+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.775466+0000) 2019-10-13T14:06:52.779 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:52.770+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.775905+0000) 2019-10-13T14:06:52.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:52.770+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.775917+0000) 2019-10-13T14:06:52.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:52.770+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.776297+0000) 2019-10-13T14:06:52.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:52.770+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.776854+0000) 2019-10-13T14:06:52.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:52.770+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.776980+0000) 2019-10-13T14:06:52.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:52.774+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.778466+0000) 2019-10-13T14:06:52.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:52.778+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:52.781435+0000) 2019-10-13T14:06:52.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:52.778+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:54.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:54.334+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:54.336283+0000) 2019-10-13T14:06:54.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:54.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:54.621933+0000) 2019-10-13T14:06:57.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:57.782+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.787507+0000) 2019-10-13T14:06:57.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:06:57.786+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.788255+0000) 2019-10-13T14:06:57.791 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:57.782+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.788444+0000) 2019-10-13T14:06:57.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:06:57.782+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.789350+0000) 2019-10-13T14:06:57.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:06:57.786+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.791572+0000) 2019-10-13T14:06:57.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:57.786+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.791900+0000) 2019-10-13T14:06:57.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:06:57.786+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.792285+0000) 2019-10-13T14:06:57.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:57.790+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.795018+0000) 2019-10-13T14:06:57.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:06:57.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:06:57.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:06:57.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:57.991203+0000) 2019-10-13T14:06:58.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:06:58.210+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:58.217587+0000) 2019-10-13T14:06:58.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:06:58.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:58.756407+0000) 2019-10-13T14:06:59.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:06:59.230+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:06:59.232455+0000) 2019-10-13T14:07:01.741 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:01.734+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:01.738750+0000) 2019-10-13T14:07:02.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:02.006+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.007798+0000) 2019-10-13T14:07:02.529 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:07:02.529 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:02.533 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:07:02.533 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:02.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:02.798+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.801034+0000) 2019-10-13T14:07:02.804 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:02.794+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.801352+0000) 2019-10-13T14:07:02.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:02.798+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.802067+0000) 2019-10-13T14:07:02.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:02.798+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.802582+0000) 2019-10-13T14:07:02.806 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:02.798+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.803227+0000) 2019-10-13T14:07:02.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:02.798+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.803536+0000) 2019-10-13T14:07:02.806 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:02.798+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.803524+0000) 2019-10-13T14:07:02.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:02.806+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:02.809447+0000) 2019-10-13T14:07:02.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:02.806+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197557 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:04.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:04.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:04.336524+0000) 2019-10-13T14:07:04.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:04.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:04.622131+0000) 2019-10-13T14:07:07.816 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:07.810+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.813660+0000) 2019-10-13T14:07:07.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:07.810+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.814126+0000) 2019-10-13T14:07:07.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:07.810+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.814913+0000) 2019-10-13T14:07:07.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:07.810+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.815269+0000) 2019-10-13T14:07:07.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:07.810+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.815898+0000) 2019-10-13T14:07:07.818 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:07.810+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.816154+0000) 2019-10-13T14:07:07.819 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:07.810+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.816437+0000) 2019-10-13T14:07:07.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:07.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:07.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:07.822+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.825443+0000) 2019-10-13T14:07:07.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:07.986+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:07.991373+0000) 2019-10-13T14:07:08.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:08.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:08.217793+0000) 2019-10-13T14:07:08.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:08.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:08.756617+0000) 2019-10-13T14:07:09.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:09.234+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:09.238875+0000) 2019-10-13T14:07:11.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:11.738+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:11.738961+0000) 2019-10-13T14:07:12.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:12.002+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.008022+0000) 2019-10-13T14:07:12.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:12.822+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.827633+0000) 2019-10-13T14:07:12.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:12.822+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.827939+0000) 2019-10-13T14:07:12.832 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:12.822+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.829058+0000) 2019-10-13T14:07:12.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:12.826+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.829789+0000) 2019-10-13T14:07:12.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:12.826+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.829941+0000) 2019-10-13T14:07:12.833 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:12.826+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.830457+0000) 2019-10-13T14:07:12.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:12.826+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.830975+0000) 2019-10-13T14:07:12.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:12.830+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:12.833969+0000) 2019-10-13T14:07:12.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:12.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197617 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:14.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:14.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:14.336724+0000) 2019-10-13T14:07:14.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:14.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:14.622334+0000) 2019-10-13T14:07:17.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:17.834+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.840019+0000) 2019-10-13T14:07:17.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:17.834+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.841095+0000) 2019-10-13T14:07:17.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:17.838+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.842198+0000) 2019-10-13T14:07:17.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:17.838+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.842532+0000) 2019-10-13T14:07:17.845 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:17.838+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.842569+0000) 2019-10-13T14:07:17.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:17.838+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.845642+0000) 2019-10-13T14:07:17.850 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:17.842+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.847891+0000) 2019-10-13T14:07:17.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:17.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:17.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:17.854+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.857603+0000) 2019-10-13T14:07:17.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:17.990+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:17.991516+0000) 2019-10-13T14:07:18.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:18.214+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:18.217993+0000) 2019-10-13T14:07:18.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:18.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:18.756811+0000) 2019-10-13T14:07:19.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:19.242+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:19.245388+0000) 2019-10-13T14:07:21.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:21.738+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:21.739161+0000) 2019-10-13T14:07:22.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:22.006+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.008261+0000) 2019-10-13T14:07:22.856 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:22.850+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.853887+0000) 2019-10-13T14:07:22.857 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:22.850+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.854830+0000) 2019-10-13T14:07:22.858 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:22.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.855453+0000) 2019-10-13T14:07:22.860 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:22.850+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.857275+0000) 2019-10-13T14:07:22.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:22.854+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.857932+0000) 2019-10-13T14:07:22.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:22.854+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.859862+0000) 2019-10-13T14:07:22.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:22.858+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.861437+0000) 2019-10-13T14:07:22.864 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:22.858+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:22.861971+0000) 2019-10-13T14:07:22.866 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:22.858+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197679 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:24.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:24.330+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:24.336935+0000) 2019-10-13T14:07:24.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:24.618+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:24.622525+0000) 2019-10-13T14:07:27.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:27.861+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.866880+0000) 2019-10-13T14:07:27.870 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:27.862+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.867353+0000) 2019-10-13T14:07:27.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:27.861+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.867681+0000) 2019-10-13T14:07:27.871 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:27.861+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.868811+0000) 2019-10-13T14:07:27.872 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:27.861+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.869141+0000) 2019-10-13T14:07:27.872 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:27.861+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.869554+0000) 2019-10-13T14:07:27.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:27.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.870218+0000) 2019-10-13T14:07:27.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:27.870+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.874488+0000) 2019-10-13T14:07:27.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:27.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:27.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:27.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:27.991714+0000) 2019-10-13T14:07:28.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:28.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:28.218172+0000) 2019-10-13T14:07:28.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:28.753+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:28.756941+0000) 2019-10-13T14:07:29.254 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:29.250+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:29.252004+0000) 2019-10-13T14:07:31.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:31.733+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:31.739384+0000) 2019-10-13T14:07:32.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:32.006+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.008490+0000) 2019-10-13T14:07:32.577 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:07:32.577 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:32.581 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:07:32.581 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:07:32.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:32.873+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.880287+0000) 2019-10-13T14:07:32.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:32.873+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.880580+0000) 2019-10-13T14:07:32.884 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:32.873+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.881030+0000) 2019-10-13T14:07:32.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:32.873+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.881428+0000) 2019-10-13T14:07:32.885 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:32.877+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.882342+0000) 2019-10-13T14:07:32.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:32.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.882709+0000) 2019-10-13T14:07:32.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:32.877+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.883450+0000) 2019-10-13T14:07:32.892 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:32.886+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:32.890125+0000) 2019-10-13T14:07:32.893 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:32.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197739 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:34.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:34.333+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:34.337146+0000) 2019-10-13T14:07:34.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:34.617+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:34.622752+0000) 2019-10-13T14:07:37.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:37.890+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.893865+0000) 2019-10-13T14:07:37.897 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:37.889+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.894279+0000) 2019-10-13T14:07:37.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:37.889+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.896411+0000) 2019-10-13T14:07:37.899 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:37.889+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.897046+0000) 2019-10-13T14:07:37.901 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:37.893+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.898837+0000) 2019-10-13T14:07:37.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:37.893+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.898779+0000) 2019-10-13T14:07:37.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:37.893+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.900475+0000) 2019-10-13T14:07:37.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:37.897+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197770 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:37.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:37.902+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.905251+0000) 2019-10-13T14:07:37.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:37.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:37.991908+0000) 2019-10-13T14:07:38.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:38.213+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:38.218360+0000) 2019-10-13T14:07:38.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:38.753+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:38.757124+0000) 2019-10-13T14:07:39.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:39.254+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:39.258351+0000) 2019-10-13T14:07:41.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:41.737+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:41.739583+0000) 2019-10-13T14:07:42.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:42.006+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.008710+0000) 2019-10-13T14:07:42.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:42.902+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.907287+0000) 2019-10-13T14:07:42.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:42.905+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.907993+0000) 2019-10-13T14:07:42.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:42.905+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.908193+0000) 2019-10-13T14:07:42.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:42.909+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.910135+0000) 2019-10-13T14:07:42.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:42.909+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.910956+0000) 2019-10-13T14:07:42.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:42.909+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.911206+0000) 2019-10-13T14:07:42.915 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:42.909+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.912592+0000) 2019-10-13T14:07:42.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:42.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:42.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:42.914+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:42.917675+0000) 2019-10-13T14:07:44.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:44.333+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:44.337369+0000) 2019-10-13T14:07:44.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:44.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:44.622956+0000) 2019-10-13T14:07:47.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:47.917+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.920422+0000) 2019-10-13T14:07:47.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:47.917+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.921072+0000) 2019-10-13T14:07:47.924 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:47.921+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.921936+0000) 2019-10-13T14:07:47.926 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:47.921+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.923519+0000) 2019-10-13T14:07:47.926 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:47.921+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.923670+0000) 2019-10-13T14:07:47.928 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:47.921+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.925723+0000) 2019-10-13T14:07:47.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:47.921+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.926165+0000) 2019-10-13T14:07:47.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:47.925+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.927514+0000) 2019-10-13T14:07:47.934 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:47.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:47.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:47.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:47.992119+0000) 2019-10-13T14:07:48.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:48.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:48.218538+0000) 2019-10-13T14:07:48.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:48.753+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:48.757340+0000) 2019-10-13T14:07:49.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:49.261+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:49.264981+0000) 2019-10-13T14:07:51.742 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:51.737+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:51.739783+0000) 2019-10-13T14:07:52.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:52.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.008907+0000) 2019-10-13T14:07:52.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:52.933+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.934556+0000) 2019-10-13T14:07:52.938 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:52.933+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.935222+0000) 2019-10-13T14:07:52.938 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:52.933+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.935472+0000) 2019-10-13T14:07:52.939 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:52.933+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.935864+0000) 2019-10-13T14:07:52.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:52.933+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.936281+0000) 2019-10-13T14:07:52.940 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:52.933+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.937262+0000) 2019-10-13T14:07:52.941 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:52.937+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.938257+0000) 2019-10-13T14:07:52.947 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:52.941+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:52.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:52.941+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:52.946648+0000) 2019-10-13T14:07:54.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:54.337+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:54.337642+0000) 2019-10-13T14:07:54.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:54.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:54.623158+0000) 2019-10-13T14:07:57.950 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:07:57.945+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.947954+0000) 2019-10-13T14:07:57.951 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:57.945+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.948734+0000) 2019-10-13T14:07:57.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:07:57.945+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.949635+0000) 2019-10-13T14:07:57.952 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:07:57.949+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.950036+0000) 2019-10-13T14:07:57.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:57.949+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.950472+0000) 2019-10-13T14:07:57.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:07:57.945+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.950937+0000) 2019-10-13T14:07:57.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:57.949+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.951180+0000) 2019-10-13T14:07:57.957 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:57.949+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.954714+0000) 2019-10-13T14:07:57.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:07:57.953+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:07:57.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:07:57.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:57.992265+0000) 2019-10-13T14:07:58.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:07:58.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:58.218733+0000) 2019-10-13T14:07:58.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:07:58.757+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:58.757514+0000) 2019-10-13T14:07:59.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:07:59.273+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:07:59.276565+0000) 2019-10-13T14:08:01.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:01.737+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:01.739984+0000) 2019-10-13T14:08:02.011 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:02.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.009111+0000) 2019-10-13T14:08:02.652 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:08:02.652 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:02.656 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:08:02.656 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:02.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:02.957+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.961059+0000) 2019-10-13T14:08:02.964 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:02.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.961599+0000) 2019-10-13T14:08:02.965 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:02.961+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.961998+0000) 2019-10-13T14:08:02.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:02.961+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.962350+0000) 2019-10-13T14:08:02.966 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:02.961+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.962842+0000) 2019-10-13T14:08:02.966 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:02.961+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.963734+0000) 2019-10-13T14:08:02.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:02.961+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.964208+0000) 2019-10-13T14:08:02.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:02.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:02.969216+0000) 2019-10-13T14:08:02.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:02.969+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:04.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:04.337+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:04.337849+0000) 2019-10-13T14:08:04.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:04.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:04.623374+0000) 2019-10-13T14:08:07.976 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:07.969+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.973714+0000) 2019-10-13T14:08:07.977 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:07.973+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.974514+0000) 2019-10-13T14:08:07.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:07.969+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.975299+0000) 2019-10-13T14:08:07.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:07.973+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.975596+0000) 2019-10-13T14:08:07.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:07.973+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.975776+0000) 2019-10-13T14:08:07.979 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:07.973+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.976719+0000) 2019-10-13T14:08:07.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:07.973+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.977373+0000) 2019-10-13T14:08:07.985 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:07.977+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.982630+0000) 2019-10-13T14:08:07.985 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:07.981+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:07.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:07.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:07.992418+0000) 2019-10-13T14:08:08.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:08.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:08.218910+0000) 2019-10-13T14:08:08.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:08.753+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:08.757680+0000) 2019-10-13T14:08:09.286 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:09.277+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:09.283188+0000) 2019-10-13T14:08:11.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:11.737+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:11.740190+0000) 2019-10-13T14:08:12.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:12.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.009342+0000) 2019-10-13T14:08:12.989 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:12.985+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.986419+0000) 2019-10-13T14:08:12.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:12.985+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.987140+0000) 2019-10-13T14:08:12.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:12.985+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.988083+0000) 2019-10-13T14:08:12.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:12.985+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.988838+0000) 2019-10-13T14:08:12.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:12.985+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.988538+0000) 2019-10-13T14:08:12.992 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:12.985+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.989247+0000) 2019-10-13T14:08:12.992 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:12.985+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.989006+0000) 2019-10-13T14:08:12.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:12.993+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:12.996656+0000) 2019-10-13T14:08:12.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:12.993+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:14.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:14.337+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:14.338009+0000) 2019-10-13T14:08:14.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:14.621+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:14.623576+0000) 2019-10-13T14:08:17.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:17.989+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:17.992579+0000) 2019-10-13T14:08:18.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:17.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.000723+0000) 2019-10-13T14:08:18.003 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:17.997+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.000844+0000) 2019-10-13T14:08:18.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:17.997+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.001653+0000) 2019-10-13T14:08:18.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:18.001+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.001840+0000) 2019-10-13T14:08:18.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:18.001+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.002383+0000) 2019-10-13T14:08:18.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:18.001+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.003363+0000) 2019-10-13T14:08:18.006 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:18.001+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.003822+0000) 2019-10-13T14:08:18.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:18.005+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.009197+0000) 2019-10-13T14:08:18.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:18.009+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198014 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:18.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:18.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.219082+0000) 2019-10-13T14:08:18.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:18.757+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:18.757886+0000) 2019-10-13T14:08:19.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:19.285+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:19.289181+0000) 2019-10-13T14:08:21.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:21.737+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:21.740391+0000) 2019-10-13T14:08:22.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:22.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:22.009569+0000) 2019-10-13T14:08:23.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:23.009+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.014116+0000) 2019-10-13T14:08:23.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:23.013+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.014622+0000) 2019-10-13T14:08:23.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:23.013+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.014834+0000) 2019-10-13T14:08:23.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:23.013+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.015303+0000) 2019-10-13T14:08:23.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:23.017+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.017721+0000) 2019-10-13T14:08:23.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:23.017+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.017967+0000) 2019-10-13T14:08:23.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:23.017+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.019622+0000) 2019-10-13T14:08:23.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:23.017+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:23.021774+0000) 2019-10-13T14:08:23.027 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:23.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:24.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:24.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:24.338245+0000) 2019-10-13T14:08:24.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:24.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:24.623773+0000) 2019-10-13T14:08:27.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:27.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:27.992775+0000) 2019-10-13T14:08:28.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:28.024+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.027615+0000) 2019-10-13T14:08:28.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:28.024+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.028109+0000) 2019-10-13T14:08:28.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:28.024+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.028511+0000) 2019-10-13T14:08:28.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:28.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.029172+0000) 2019-10-13T14:08:28.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:28.025+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.029577+0000) 2019-10-13T14:08:28.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:28.028+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.029776+0000) 2019-10-13T14:08:28.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:28.028+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.030838+0000) 2019-10-13T14:08:28.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:28.033+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.036784+0000) 2019-10-13T14:08:28.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:28.032+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:28.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:28.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.219289+0000) 2019-10-13T14:08:28.760 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:28.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:28.758024+0000) 2019-10-13T14:08:29.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:29.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:29.295583+0000) 2019-10-13T14:08:31.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:31.736+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:31.740627+0000) 2019-10-13T14:08:32.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:32.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:32.009825+0000) 2019-10-13T14:08:32.708 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:08:32.708 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:32.712 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:08:32.712 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:08:33.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:33.036+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.040729+0000) 2019-10-13T14:08:33.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:33.036+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.041414+0000) 2019-10-13T14:08:33.045 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:33.040+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.042120+0000) 2019-10-13T14:08:33.045 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:33.040+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.042574+0000) 2019-10-13T14:08:33.047 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:33.041+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.044465+0000) 2019-10-13T14:08:33.048 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:33.040+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.045703+0000) 2019-10-13T14:08:33.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:33.044+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.045854+0000) 2019-10-13T14:08:33.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:33.045+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:33.050660+0000) 2019-10-13T14:08:33.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:33.048+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:34.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:34.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:34.338476+0000) 2019-10-13T14:08:34.626 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:34.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:34.623977+0000) 2019-10-13T14:08:37.995 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:37.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:37.992931+0000) 2019-10-13T14:08:38.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:38.049+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.053718+0000) 2019-10-13T14:08:38.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:38.052+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.054264+0000) 2019-10-13T14:08:38.057 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:38.052+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.054835+0000) 2019-10-13T14:08:38.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:38.052+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.055447+0000) 2019-10-13T14:08:38.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:38.052+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.055000+0000) 2019-10-13T14:08:38.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:38.052+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.057138+0000) 2019-10-13T14:08:38.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:38.056+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.058314+0000) 2019-10-13T14:08:38.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:38.057+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.060734+0000) 2019-10-13T14:08:38.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:38.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:38.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:38.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.219454+0000) 2019-10-13T14:08:38.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:38.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:38.758164+0000) 2019-10-13T14:08:39.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:39.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:39.302198+0000) 2019-10-13T14:08:41.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:41.740+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:41.740857+0000) 2019-10-13T14:08:42.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:42.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:42.010027+0000) 2019-10-13T14:08:43.071 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:43.064+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.068884+0000) 2019-10-13T14:08:43.072 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:43.064+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.069474+0000) 2019-10-13T14:08:43.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:43.068+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.069971+0000) 2019-10-13T14:08:43.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:43.068+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.070400+0000) 2019-10-13T14:08:43.073 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:43.068+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.070832+0000) 2019-10-13T14:08:43.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:43.065+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.071869+0000) 2019-10-13T14:08:43.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:43.068+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.072297+0000) 2019-10-13T14:08:43.081 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:43.073+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:43.078722+0000) 2019-10-13T14:08:43.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:43.076+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:44.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:44.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:44.338675+0000) 2019-10-13T14:08:44.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:44.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:44.624183+0000) 2019-10-13T14:08:47.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:47.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:47.993119+0000) 2019-10-13T14:08:48.085 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:48.080+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.082954+0000) 2019-10-13T14:08:48.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:48.080+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.083723+0000) 2019-10-13T14:08:48.087 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:48.080+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.084078+0000) 2019-10-13T14:08:48.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:48.080+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.084363+0000) 2019-10-13T14:08:48.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:48.080+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.085369+0000) 2019-10-13T14:08:48.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:48.080+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.085518+0000) 2019-10-13T14:08:48.090 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:48.081+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.087830+0000) 2019-10-13T14:08:48.094 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:48.085+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.092076+0000) 2019-10-13T14:08:48.096 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:48.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:48.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:48.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.219661+0000) 2019-10-13T14:08:48.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:48.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:48.758372+0000) 2019-10-13T14:08:49.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:49.305+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:49.308704+0000) 2019-10-13T14:08:51.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:51.736+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:51.741086+0000) 2019-10-13T14:08:52.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:52.005+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:52.010221+0000) 2019-10-13T14:08:53.099 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:53.092+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.096291+0000) 2019-10-13T14:08:53.100 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:53.096+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.098160+0000) 2019-10-13T14:08:53.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:53.096+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.098653+0000) 2019-10-13T14:08:53.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:53.096+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.099486+0000) 2019-10-13T14:08:53.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:53.096+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.099888+0000) 2019-10-13T14:08:53.103 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:53.096+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.099588+0000) 2019-10-13T14:08:53.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:53.096+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.100324+0000) 2019-10-13T14:08:53.108 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:53.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:53.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:53.100+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:53.106911+0000) 2019-10-13T14:08:54.341 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:54.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:54.338903+0000) 2019-10-13T14:08:54.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:54.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:54.624414+0000) 2019-10-13T14:08:57.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:57.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:57.993308+0000) 2019-10-13T14:08:58.112 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:08:58.104+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.109706+0000) 2019-10-13T14:08:58.113 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:58.108+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.110114+0000) 2019-10-13T14:08:58.113 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:08:58.108+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.110631+0000) 2019-10-13T14:08:58.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:58.108+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.111825+0000) 2019-10-13T14:08:58.115 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:08:58.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.112462+0000) 2019-10-13T14:08:58.115 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:08:58.108+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.113264+0000) 2019-10-13T14:08:58.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:08:58.108+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.113710+0000) 2019-10-13T14:08:58.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:58.112+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.118333+0000) 2019-10-13T14:08:58.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:08:58.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198256 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:08:58.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:08:58.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.219842+0000) 2019-10-13T14:08:58.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:08:58.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:58.758568+0000) 2019-10-13T14:08:59.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:08:59.308+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:08:59.315138+0000) 2019-10-13T14:09:01.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:01.736+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:01.741316+0000) 2019-10-13T14:09:02.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:02.004+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:02.010442+0000) 2019-10-13T14:09:02.766 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:09:02.766 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:02.770 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:09:02.770 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:03.126 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:03.120+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.122708+0000) 2019-10-13T14:09:03.127 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:03.120+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.124538+0000) 2019-10-13T14:09:03.127 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:03.120+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.124855+0000) 2019-10-13T14:09:03.128 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:03.120+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.125293+0000) 2019-10-13T14:09:03.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:03.124+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.127556+0000) 2019-10-13T14:09:03.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:03.124+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.128256+0000) 2019-10-13T14:09:03.131 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:03.124+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.128652+0000) 2019-10-13T14:09:03.135 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:03.128+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:03.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:03.128+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:03.133540+0000) 2019-10-13T14:09:04.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:04.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:04.339107+0000) 2019-10-13T14:09:04.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:04.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:04.624612+0000) 2019-10-13T14:09:07.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:07.988+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:07.993497+0000) 2019-10-13T14:09:08.139 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:08.132+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.136199+0000) 2019-10-13T14:09:08.139 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:08.132+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.136553+0000) 2019-10-13T14:09:08.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:08.132+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.136662+0000) 2019-10-13T14:09:08.140 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:08.132+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.137421+0000) 2019-10-13T14:09:08.140 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:08.132+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.137452+0000) 2019-10-13T14:09:08.141 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:08.136+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.138595+0000) 2019-10-13T14:09:08.142 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:08.136+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.139387+0000) 2019-10-13T14:09:08.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:08.136+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.142279+0000) 2019-10-13T14:09:08.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:08.140+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:08.222 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:08.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.220012+0000) 2019-10-13T14:09:08.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:08.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:08.758750+0000) 2019-10-13T14:09:09.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:09.316+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:09.321747+0000) 2019-10-13T14:09:11.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:11.740+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:11.741522+0000) 2019-10-13T14:09:12.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:12.008+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:12.010640+0000) 2019-10-13T14:09:13.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:13.144+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.149021+0000) 2019-10-13T14:09:13.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:13.144+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.149633+0000) 2019-10-13T14:09:13.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:13.148+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.150306+0000) 2019-10-13T14:09:13.153 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:13.148+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.151075+0000) 2019-10-13T14:09:13.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:13.148+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.151465+0000) 2019-10-13T14:09:13.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:13.144+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.152032+0000) 2019-10-13T14:09:13.155 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:13.148+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.153230+0000) 2019-10-13T14:09:13.161 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:13.152+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:13.158625+0000) 2019-10-13T14:09:13.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:13.156+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:14.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:14.336+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:14.339310+0000) 2019-10-13T14:09:14.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:14.620+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:14.624827+0000) 2019-10-13T14:09:17.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:17.992+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:17.993651+0000) 2019-10-13T14:09:18.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:18.156+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.161673+0000) 2019-10-13T14:09:18.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:18.160+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.162623+0000) 2019-10-13T14:09:18.166 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:18.160+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.163332+0000) 2019-10-13T14:09:18.166 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:18.160+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.164005+0000) 2019-10-13T14:09:18.167 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:18.160+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.164414+0000) 2019-10-13T14:09:18.167 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:18.160+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.164676+0000) 2019-10-13T14:09:18.168 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:18.160+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.165684+0000) 2019-10-13T14:09:18.175 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:18.168+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:18.175 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:18.168+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.172952+0000) 2019-10-13T14:09:18.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:18.216+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.220211+0000) 2019-10-13T14:09:18.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:18.756+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:18.758938+0000) 2019-10-13T14:09:19.330 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:19.324+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:19.327548+0000) 2019-10-13T14:09:21.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:21.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:21.741727+0000) 2019-10-13T14:09:22.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:22.004+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:22.010858+0000) 2019-10-13T14:09:23.179 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:23.171+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.176295+0000) 2019-10-13T14:09:23.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:23.171+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.177168+0000) 2019-10-13T14:09:23.180 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:23.171+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.177313+0000) 2019-10-13T14:09:23.181 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:23.175+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.178042+0000) 2019-10-13T14:09:23.181 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:23.175+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.178324+0000) 2019-10-13T14:09:23.182 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:23.172+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.179543+0000) 2019-10-13T14:09:23.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:23.175+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.179633+0000) 2019-10-13T14:09:23.188 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:23.179+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:23.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:23.180+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:23.185907+0000) 2019-10-13T14:09:24.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:24.335+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:24.339509+0000) 2019-10-13T14:09:24.627 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:24.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:24.625027+0000) 2019-10-13T14:09:27.996 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:27.991+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:27.993923+0000) 2019-10-13T14:09:28.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:28.183+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.188620+0000) 2019-10-13T14:09:28.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:28.183+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.188907+0000) 2019-10-13T14:09:28.192 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:28.183+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.189416+0000) 2019-10-13T14:09:28.194 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:28.184+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.191656+0000) 2019-10-13T14:09:28.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:28.187+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.192461+0000) 2019-10-13T14:09:28.196 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:28.187+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.193391+0000) 2019-10-13T14:09:28.196 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:28.191+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.193819+0000) 2019-10-13T14:09:28.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:28.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198435 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:28.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:28.192+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.199630+0000) 2019-10-13T14:09:28.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:28.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.220437+0000) 2019-10-13T14:09:28.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:28.755+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:28.759111+0000) 2019-10-13T14:09:29.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:29.332+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:29.334287+0000) 2019-10-13T14:09:31.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:31.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:31.741968+0000) 2019-10-13T14:09:32.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:32.008+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:32.011093+0000) 2019-10-13T14:09:32.821 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:09:32.822 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:32.825 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:09:32.825 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:09:33.203 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:33.195+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.200339+0000) 2019-10-13T14:09:33.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:33.199+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.201732+0000) 2019-10-13T14:09:33.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:33.195+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.201189+0000) 2019-10-13T14:09:33.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:33.199+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.202689+0000) 2019-10-13T14:09:33.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:33.196+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.203107+0000) 2019-10-13T14:09:33.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:33.199+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.203765+0000) 2019-10-13T14:09:33.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:33.199+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.204041+0000) 2019-10-13T14:09:33.213 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:33.204+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:33.210750+0000) 2019-10-13T14:09:33.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:33.207+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198467 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:34.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:34.335+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:34.339704+0000) 2019-10-13T14:09:34.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:34.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:34.625261+0000) 2019-10-13T14:09:37.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:37.991+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:37.994128+0000) 2019-10-13T14:09:38.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:38.208+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.214180+0000) 2019-10-13T14:09:38.218 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:38.211+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.214379+0000) 2019-10-13T14:09:38.219 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:38.211+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.214788+0000) 2019-10-13T14:09:38.219 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:38.211+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.215256+0000) 2019-10-13T14:09:38.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:38.211+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.215647+0000) 2019-10-13T14:09:38.220 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:38.211+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.216113+0000) 2019-10-13T14:09:38.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:38.215+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.217983+0000) 2019-10-13T14:09:38.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:38.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.220559+0000) 2019-10-13T14:09:38.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:38.216+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.223805+0000) 2019-10-13T14:09:38.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:38.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198496 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:38.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:38.755+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:38.759244+0000) 2019-10-13T14:09:39.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:39.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:39.340807+0000) 2019-10-13T14:09:41.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:41.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:41.742189+0000) 2019-10-13T14:09:42.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:42.008+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:42.011288+0000) 2019-10-13T14:09:43.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:43.224+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.227644+0000) 2019-10-13T14:09:43.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:43.223+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.227862+0000) 2019-10-13T14:09:43.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:43.223+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.228227+0000) 2019-10-13T14:09:43.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:43.223+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.229066+0000) 2019-10-13T14:09:43.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:43.223+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.228824+0000) 2019-10-13T14:09:43.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:43.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.229550+0000) 2019-10-13T14:09:43.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:43.227+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.231190+0000) 2019-10-13T14:09:43.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:43.232+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:43.236802+0000) 2019-10-13T14:09:43.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:43.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:44.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:44.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:44.339907+0000) 2019-10-13T14:09:44.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:44.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:44.625457+0000) 2019-10-13T14:09:47.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:47.991+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:47.994331+0000) 2019-10-13T14:09:48.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:48.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.220741+0000) 2019-10-13T14:09:48.243 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:48.236+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.240378+0000) 2019-10-13T14:09:48.244 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:48.235+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.241584+0000) 2019-10-13T14:09:48.244 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:48.235+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.241311+0000) 2019-10-13T14:09:48.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:48.239+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.242199+0000) 2019-10-13T14:09:48.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:48.239+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.243319+0000) 2019-10-13T14:09:48.246 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:48.239+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.243111+0000) 2019-10-13T14:09:48.247 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:48.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.245298+0000) 2019-10-13T14:09:48.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:48.244+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.249045+0000) 2019-10-13T14:09:48.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:48.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:48.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:48.755+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:48.759399+0000) 2019-10-13T14:09:49.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:49.344+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:49.347346+0000) 2019-10-13T14:09:51.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:51.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:51.742422+0000) 2019-10-13T14:09:52.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:52.008+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:52.011478+0000) 2019-10-13T14:09:53.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:53.251+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.253894+0000) 2019-10-13T14:09:53.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:53.252+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.254755+0000) 2019-10-13T14:09:53.258 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:53.251+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.255334+0000) 2019-10-13T14:09:53.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:53.251+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.255789+0000) 2019-10-13T14:09:53.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:53.251+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.256163+0000) 2019-10-13T14:09:53.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:53.251+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.257240+0000) 2019-10-13T14:09:53.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:53.251+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.256988+0000) 2019-10-13T14:09:53.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:53.260+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:53.260176+0000) 2019-10-13T14:09:53.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:53.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198587 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:54.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:54.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:54.340123+0000) 2019-10-13T14:09:54.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:54.619+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:54.625687+0000) 2019-10-13T14:09:57.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:57.991+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:57.994492+0000) 2019-10-13T14:09:58.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:58.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.220970+0000) 2019-10-13T14:09:58.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:09:58.263+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.266985+0000) 2019-10-13T14:09:58.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:58.263+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.267668+0000) 2019-10-13T14:09:58.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:09:58.267+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.268299+0000) 2019-10-13T14:09:58.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:09:58.263+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.268336+0000) 2019-10-13T14:09:58.272 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:09:58.267+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.269821+0000) 2019-10-13T14:09:58.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:09:58.267+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.271788+0000) 2019-10-13T14:09:58.276 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:09:58.267+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.273689+0000) 2019-10-13T14:09:58.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:09:58.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:09:58.280 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:58.275+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.277349+0000) 2019-10-13T14:09:58.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:09:58.759+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:58.759532+0000) 2019-10-13T14:09:59.356 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:09:59.351+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:09:59.353912+0000) 2019-10-13T14:10:01.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:01.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:01.742623+0000) 2019-10-13T14:10:02.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:02.007+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:02.011634+0000) 2019-10-13T14:10:02.870 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:10:02.871 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:02.875 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:10:02.875 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:03.283 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:03.279+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.280746+0000) 2019-10-13T14:10:03.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:03.275+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.281346+0000) 2019-10-13T14:10:03.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:03.279+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.282118+0000) 2019-10-13T14:10:03.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:03.279+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.282557+0000) 2019-10-13T14:10:03.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:03.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.283055+0000) 2019-10-13T14:10:03.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:03.279+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.283124+0000) 2019-10-13T14:10:03.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:03.279+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.283589+0000) 2019-10-13T14:10:03.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:03.287+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:03.288204+0000) 2019-10-13T14:10:03.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:03.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198646 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:04.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:04.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:04.340334+0000) 2019-10-13T14:10:04.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:04.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:04.625928+0000) 2019-10-13T14:10:07.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:07.991+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:07.994729+0000) 2019-10-13T14:10:08.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:08.215+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.221205+0000) 2019-10-13T14:10:08.296 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:08.287+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.293270+0000) 2019-10-13T14:10:08.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:08.291+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.293916+0000) 2019-10-13T14:10:08.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:08.291+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.294182+0000) 2019-10-13T14:10:08.297 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:08.291+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.295098+0000) 2019-10-13T14:10:08.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:08.291+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.295185+0000) 2019-10-13T14:10:08.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:08.291+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.296540+0000) 2019-10-13T14:10:08.299 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:08.291+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.296928+0000) 2019-10-13T14:10:08.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:08.299+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.302102+0000) 2019-10-13T14:10:08.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:08.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:08.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:08.755+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:08.759688+0000) 2019-10-13T14:10:09.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:09.359+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:09.361766+0000) 2019-10-13T14:10:11.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:11.739+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:11.742873+0000) 2019-10-13T14:10:12.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:12.007+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:12.011828+0000) 2019-10-13T14:10:13.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:13.303+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.306830+0000) 2019-10-13T14:10:13.310 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:13.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.307759+0000) 2019-10-13T14:10:13.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:13.303+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.308417+0000) 2019-10-13T14:10:13.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:13.303+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.308784+0000) 2019-10-13T14:10:13.311 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:13.303+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.308998+0000) 2019-10-13T14:10:13.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:13.307+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.309546+0000) 2019-10-13T14:10:13.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:13.303+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.309694+0000) 2019-10-13T14:10:13.318 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:13.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:13.319 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:13.315+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:13.316608+0000) 2019-10-13T14:10:14.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:14.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:14.340533+0000) 2019-10-13T14:10:14.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:14.623+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:14.626171+0000) 2019-10-13T14:10:17.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:17.990+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:17.994962+0000) 2019-10-13T14:10:18.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:18.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.221416+0000) 2019-10-13T14:10:18.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:18.314+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.319305+0000) 2019-10-13T14:10:18.323 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:18.314+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.319671+0000) 2019-10-13T14:10:18.323 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:18.314+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.320177+0000) 2019-10-13T14:10:18.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:18.319+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.321260+0000) 2019-10-13T14:10:18.324 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:18.314+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.321399+0000) 2019-10-13T14:10:18.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:18.318+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.321773+0000) 2019-10-13T14:10:18.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:18.318+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.322330+0000) 2019-10-13T14:10:18.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:18.323+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.327197+0000) 2019-10-13T14:10:18.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:18.322+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:18.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:18.758+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:18.759842+0000) 2019-10-13T14:10:19.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:19.367+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:19.368508+0000) 2019-10-13T14:10:21.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:21.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:21.743111+0000) 2019-10-13T14:10:22.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:22.007+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:22.012025+0000) 2019-10-13T14:10:23.334 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:23.326+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.332029+0000) 2019-10-13T14:10:23.335 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:23.326+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.333219+0000) 2019-10-13T14:10:23.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:23.326+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.333553+0000) 2019-10-13T14:10:23.336 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:23.326+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.333596+0000) 2019-10-13T14:10:23.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:23.330+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.334137+0000) 2019-10-13T14:10:23.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:23.331+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.334691+0000) 2019-10-13T14:10:23.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:23.330+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.335574+0000) 2019-10-13T14:10:23.344 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:23.339+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:23.341754+0000) 2019-10-13T14:10:23.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:23.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:24.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:24.334+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:24.340732+0000) 2019-10-13T14:10:24.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:24.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:24.626401+0000) 2019-10-13T14:10:27.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:27.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:27.995186+0000) 2019-10-13T14:10:28.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:28.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.221618+0000) 2019-10-13T14:10:28.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:28.342+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.346207+0000) 2019-10-13T14:10:28.349 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:28.343+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.346636+0000) 2019-10-13T14:10:28.350 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:28.342+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.347425+0000) 2019-10-13T14:10:28.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:28.342+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.346900+0000) 2019-10-13T14:10:28.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:28.346+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.350021+0000) 2019-10-13T14:10:28.353 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:28.346+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.350090+0000) 2019-10-13T14:10:28.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:28.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.350545+0000) 2019-10-13T14:10:28.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:28.350+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198796 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:28.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:28.355+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.357695+0000) 2019-10-13T14:10:28.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:28.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:28.759989+0000) 2019-10-13T14:10:29.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:29.371+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:29.375115+0000) 2019-10-13T14:10:31.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:31.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:31.743317+0000) 2019-10-13T14:10:32.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:32.011+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:32.012246+0000) 2019-10-13T14:10:32.975 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:10:32.975 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:32.979 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:10:32.979 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:10:33.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:33.355+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.359504+0000) 2019-10-13T14:10:33.362 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:33.354+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.359723+0000) 2019-10-13T14:10:33.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:33.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.360548+0000) 2019-10-13T14:10:33.364 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:33.354+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.361146+0000) 2019-10-13T14:10:33.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:33.354+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.361415+0000) 2019-10-13T14:10:33.364 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:33.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.361938+0000) 2019-10-13T14:10:33.365 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:33.358+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.362145+0000) 2019-10-13T14:10:33.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:33.363+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:33.366656+0000) 2019-10-13T14:10:33.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:33.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:34.343 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:34.338+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:34.340952+0000) 2019-10-13T14:10:34.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:34.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:34.626608+0000) 2019-10-13T14:10:37.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:37.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:37.995391+0000) 2019-10-13T14:10:38.224 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:38.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.221814+0000) 2019-10-13T14:10:38.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:38.366+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.372797+0000) 2019-10-13T14:10:38.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:38.371+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.373656+0000) 2019-10-13T14:10:38.376 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:38.370+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.373918+0000) 2019-10-13T14:10:38.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:38.370+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.374720+0000) 2019-10-13T14:10:38.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:38.370+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.375339+0000) 2019-10-13T14:10:38.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:38.370+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.375600+0000) 2019-10-13T14:10:38.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:38.370+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.376497+0000) 2019-10-13T14:10:38.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:38.379+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.381501+0000) 2019-10-13T14:10:38.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:38.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198858 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:38.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:38.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:38.760208+0000) 2019-10-13T14:10:39.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:39.379+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:39.381612+0000) 2019-10-13T14:10:41.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:41.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:41.743513+0000) 2019-10-13T14:10:42.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:42.011+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:42.012456+0000) 2019-10-13T14:10:43.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:43.382+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.386001+0000) 2019-10-13T14:10:43.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:43.383+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.386848+0000) 2019-10-13T14:10:43.390 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:43.382+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.387254+0000) 2019-10-13T14:10:43.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:43.382+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.388285+0000) 2019-10-13T14:10:43.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:43.382+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.389470+0000) 2019-10-13T14:10:43.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:43.382+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.389175+0000) 2019-10-13T14:10:43.392 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:43.382+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.388634+0000) 2019-10-13T14:10:43.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:43.391+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:43.394210+0000) 2019-10-13T14:10:43.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:43.390+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198890 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:44.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:44.338+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:44.341174+0000) 2019-10-13T14:10:44.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:44.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:44.626832+0000) 2019-10-13T14:10:47.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:47.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:47.995590+0000) 2019-10-13T14:10:48.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:48.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.222010+0000) 2019-10-13T14:10:48.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:48.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.399693+0000) 2019-10-13T14:10:48.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:48.394+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.400499+0000) 2019-10-13T14:10:48.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:48.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.401081+0000) 2019-10-13T14:10:48.404 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:48.394+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.401601+0000) 2019-10-13T14:10:48.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:48.394+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.401703+0000) 2019-10-13T14:10:48.405 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:48.398+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.402954+0000) 2019-10-13T14:10:48.406 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:48.398+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.402684+0000) 2019-10-13T14:10:48.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:48.403+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.405769+0000) 2019-10-13T14:10:48.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:48.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198918 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:48.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:48.758+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:48.760427+0000) 2019-10-13T14:10:49.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:49.387+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:49.388171+0000) 2019-10-13T14:10:51.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:51.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:51.743713+0000) 2019-10-13T14:10:52.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:52.011+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:52.012652+0000) 2019-10-13T14:10:53.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:53.406+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.411732+0000) 2019-10-13T14:10:53.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:53.406+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.412643+0000) 2019-10-13T14:10:53.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:53.410+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.413815+0000) 2019-10-13T14:10:53.417 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:53.410+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.414443+0000) 2019-10-13T14:10:53.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:53.410+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.414658+0000) 2019-10-13T14:10:53.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:53.411+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.415488+0000) 2019-10-13T14:10:53.418 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:53.410+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.416155+0000) 2019-10-13T14:10:53.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:53.418+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:53.427 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:53.423+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:53.425206+0000) 2019-10-13T14:10:54.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:54.338+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:54.341395+0000) 2019-10-13T14:10:54.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:54.622+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:54.627027+0000) 2019-10-13T14:10:57.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:57.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:57.995818+0000) 2019-10-13T14:10:58.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:58.218+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.222204+0000) 2019-10-13T14:10:58.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:10:58.422+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.426105+0000) 2019-10-13T14:10:58.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:10:58.422+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.428273+0000) 2019-10-13T14:10:58.431 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:10:58.422+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.429042+0000) 2019-10-13T14:10:58.432 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:58.422+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.429217+0000) 2019-10-13T14:10:58.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:10:58.426+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.430090+0000) 2019-10-13T14:10:58.434 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:10:58.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.431599+0000) 2019-10-13T14:10:58.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:10:58.426+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.432351+0000) 2019-10-13T14:10:58.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:58.431+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.434042+0000) 2019-10-13T14:10:58.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:10:58.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198980 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:10:58.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:10:58.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:58.760630+0000) 2019-10-13T14:10:59.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:10:59.391+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:10:59.394600+0000) 2019-10-13T14:11:01.746 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:01.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:01.743946+0000) 2019-10-13T14:11:02.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:02.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:02.012849+0000) 2019-10-13T14:11:03.033 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:11:03.034 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:03.037 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:11:03.037 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:03.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:03.434+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.441357+0000) 2019-10-13T14:11:03.445 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:03.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.441949+0000) 2019-10-13T14:11:03.445 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:03.438+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.442244+0000) 2019-10-13T14:11:03.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:03.438+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.444241+0000) 2019-10-13T14:11:03.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:03.438+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.444639+0000) 2019-10-13T14:11:03.449 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:03.442+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.446405+0000) 2019-10-13T14:11:03.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:03.446+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:03.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:03.446+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.450548+0000) 2019-10-13T14:11:03.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:03.450+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:03.455301+0000) 2019-10-13T14:11:04.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:04.338+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:04.341586+0000) 2019-10-13T14:11:04.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:04.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:04.627283+0000) 2019-10-13T14:11:07.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:07.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:07.996020+0000) 2019-10-13T14:11:08.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:08.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.222411+0000) 2019-10-13T14:11:08.457 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:08.450+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.454382+0000) 2019-10-13T14:11:08.458 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:08.450+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.455646+0000) 2019-10-13T14:11:08.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:08.450+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.455836+0000) 2019-10-13T14:11:08.459 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:08.454+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.456275+0000) 2019-10-13T14:11:08.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:08.450+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.456890+0000) 2019-10-13T14:11:08.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:08.450+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.457299+0000) 2019-10-13T14:11:08.460 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:08.454+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.457902+0000) 2019-10-13T14:11:08.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:08.458+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199040 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:08.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:08.462+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.464696+0000) 2019-10-13T14:11:08.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:08.754+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:08.760810+0000) 2019-10-13T14:11:09.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:09.398+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:09.401060+0000) 2019-10-13T14:11:11.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:11.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:11.744144+0000) 2019-10-13T14:11:12.015 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:12.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:12.013048+0000) 2019-10-13T14:11:13.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:13.461+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.466297+0000) 2019-10-13T14:11:13.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:13.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.467219+0000) 2019-10-13T14:11:13.471 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:13.461+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.468399+0000) 2019-10-13T14:11:13.471 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:13.466+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.469139+0000) 2019-10-13T14:11:13.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:13.461+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.469270+0000) 2019-10-13T14:11:13.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:13.465+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.470509+0000) 2019-10-13T14:11:13.474 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:13.465+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.471155+0000) 2019-10-13T14:11:13.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:13.469+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:13.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:13.474+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:13.476636+0000) 2019-10-13T14:11:14.344 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:14.337+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:14.341804+0000) 2019-10-13T14:11:14.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:14.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:14.627517+0000) 2019-10-13T14:11:17.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:17.993+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:17.996235+0000) 2019-10-13T14:11:18.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:18.217+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.222641+0000) 2019-10-13T14:11:18.481 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:18.473+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.478285+0000) 2019-10-13T14:11:18.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:18.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.482103+0000) 2019-10-13T14:11:18.486 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:18.477+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.484008+0000) 2019-10-13T14:11:18.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:18.482+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.487787+0000) 2019-10-13T14:11:18.493 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:18.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199101 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:18.493 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:18.485+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.490624+0000) 2019-10-13T14:11:18.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:18.485+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.491120+0000) 2019-10-13T14:11:18.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:18.485+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.491639+0000) 2019-10-13T14:11:18.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:18.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.497196+0000) 2019-10-13T14:11:18.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:18.753+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:18.760987+0000) 2019-10-13T14:11:19.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:19.406+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:19.407537+0000) 2019-10-13T14:11:21.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:21.741+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:21.744351+0000) 2019-10-13T14:11:22.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:22.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:22.013244+0000) 2019-10-13T14:11:23.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:23.485+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.493603+0000) 2019-10-13T14:11:23.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:23.489+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.494020+0000) 2019-10-13T14:11:23.498 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:23.490+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.495622+0000) 2019-10-13T14:11:23.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:23.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.496517+0000) 2019-10-13T14:11:23.499 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:23.489+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.496820+0000) 2019-10-13T14:11:23.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:23.489+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.497274+0000) 2019-10-13T14:11:23.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:23.489+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.497623+0000) 2019-10-13T14:11:23.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:23.498+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:23.502212+0000) 2019-10-13T14:11:23.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:23.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:24.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:24.341+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:24.342021+0000) 2019-10-13T14:11:24.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:24.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:24.627761+0000) 2019-10-13T14:11:27.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:27.993+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:27.996393+0000) 2019-10-13T14:11:28.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:28.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.222869+0000) 2019-10-13T14:11:28.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:28.505+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.507636+0000) 2019-10-13T14:11:28.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:28.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.510411+0000) 2019-10-13T14:11:28.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:28.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.511190+0000) 2019-10-13T14:11:28.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:28.509+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.511471+0000) 2019-10-13T14:11:28.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:28.509+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.512527+0000) 2019-10-13T14:11:28.515 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:28.509+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.512890+0000) 2019-10-13T14:11:28.516 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:28.509+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.513208+0000) 2019-10-13T14:11:28.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:28.514+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.516307+0000) 2019-10-13T14:11:28.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:28.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199161 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:28.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:28.757+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:28.761170+0000) 2019-10-13T14:11:29.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:29.410+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:29.414117+0000) 2019-10-13T14:11:31.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:31.741+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:31.744552+0000) 2019-10-13T14:11:32.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:32.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:32.013438+0000) 2019-10-13T14:11:33.136 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:11:33.136 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:33.141 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:11:33.141 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:11:33.524 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:33.517+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.521573+0000) 2019-10-13T14:11:33.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:33.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.522017+0000) 2019-10-13T14:11:33.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:33.521+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.523140+0000) 2019-10-13T14:11:33.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:33.521+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.524257+0000) 2019-10-13T14:11:33.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:33.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.524416+0000) 2019-10-13T14:11:33.527 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:33.521+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.524718+0000) 2019-10-13T14:11:33.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:33.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.525843+0000) 2019-10-13T14:11:33.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:33.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:33.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:33.530+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:33.532306+0000) 2019-10-13T14:11:34.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:34.341+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:34.342204+0000) 2019-10-13T14:11:34.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:34.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:34.627976+0000) 2019-10-13T14:11:37.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:37.993+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:37.996615+0000) 2019-10-13T14:11:38.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:38.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.223075+0000) 2019-10-13T14:11:38.538 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:38.533+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.535142+0000) 2019-10-13T14:11:38.538 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:38.533+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.535751+0000) 2019-10-13T14:11:38.539 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:38.534+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.536578+0000) 2019-10-13T14:11:38.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:38.533+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.537384+0000) 2019-10-13T14:11:38.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:38.537+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.538009+0000) 2019-10-13T14:11:38.541 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:38.537+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.538061+0000) 2019-10-13T14:11:38.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:38.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.538476+0000) 2019-10-13T14:11:38.546 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:38.542+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.544247+0000) 2019-10-13T14:11:38.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:38.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:38.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:38.757+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:38.761303+0000) 2019-10-13T14:11:39.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:39.418+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:39.420717+0000) 2019-10-13T14:11:41.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:41.741+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:41.744776+0000) 2019-10-13T14:11:42.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:42.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:42.013640+0000) 2019-10-13T14:11:43.551 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:43.545+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.548517+0000) 2019-10-13T14:11:43.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:43.545+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.549334+0000) 2019-10-13T14:11:43.552 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:43.545+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.549608+0000) 2019-10-13T14:11:43.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:43.549+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.550179+0000) 2019-10-13T14:11:43.553 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:43.546+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.550463+0000) 2019-10-13T14:11:43.553 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:43.549+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.551073+0000) 2019-10-13T14:11:43.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:43.549+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.552241+0000) 2019-10-13T14:11:43.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:43.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:43.556627+0000) 2019-10-13T14:11:43.560 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:43.557+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:44.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:44.341+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:44.342422+0000) 2019-10-13T14:11:44.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:44.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:44.628174+0000) 2019-10-13T14:11:47.999 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:47.993+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:47.996867+0000) 2019-10-13T14:11:48.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:48.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.223276+0000) 2019-10-13T14:11:48.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:48.561+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.561867+0000) 2019-10-13T14:11:48.565 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:48.561+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.562163+0000) 2019-10-13T14:11:48.565 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:48.561+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.563023+0000) 2019-10-13T14:11:48.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:48.558+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.563709+0000) 2019-10-13T14:11:48.566 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:48.561+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.563940+0000) 2019-10-13T14:11:48.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:48.561+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.564304+0000) 2019-10-13T14:11:48.568 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:48.561+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.565431+0000) 2019-10-13T14:11:48.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:48.569+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:48.574 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:48.566+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.571693+0000) 2019-10-13T14:11:48.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:48.761+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:48.761521+0000) 2019-10-13T14:11:49.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:49.422+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:49.427291+0000) 2019-10-13T14:11:51.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:51.741+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:51.744991+0000) 2019-10-13T14:11:52.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:52.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:52.013831+0000) 2019-10-13T14:11:53.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:53.573+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.574477+0000) 2019-10-13T14:11:53.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:53.573+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.574950+0000) 2019-10-13T14:11:53.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:53.573+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.576269+0000) 2019-10-13T14:11:53.579 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:53.573+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.576410+0000) 2019-10-13T14:11:53.579 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:53.573+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.576765+0000) 2019-10-13T14:11:53.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:53.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.577581+0000) 2019-10-13T14:11:53.581 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:53.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.578660+0000) 2019-10-13T14:11:53.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:53.578+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:53.581921+0000) 2019-10-13T14:11:53.587 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:53.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:54.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:54.341+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:54.342612+0000) 2019-10-13T14:11:54.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:54.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:54.628397+0000) 2019-10-13T14:11:58.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:57.993+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:57.997077+0000) 2019-10-13T14:11:58.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:58.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.223493+0000) 2019-10-13T14:11:58.592 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:11:58.585+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.589467+0000) 2019-10-13T14:11:58.592 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:11:58.589+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.590162+0000) 2019-10-13T14:11:58.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:11:58.586+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.591967+0000) 2019-10-13T14:11:58.595 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:11:58.589+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.592425+0000) 2019-10-13T14:11:58.595 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:11:58.589+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.592611+0000) 2019-10-13T14:11:58.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:58.589+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.593249+0000) 2019-10-13T14:11:58.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:11:58.589+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.593701+0000) 2019-10-13T14:11:58.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:58.594+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.599040+0000) 2019-10-13T14:11:58.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:11:58.597+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199347 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:11:58.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:11:58.761+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:58.761703+0000) 2019-10-13T14:11:59.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:11:59.430+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:11:59.433714+0000) 2019-10-13T14:12:01.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:01.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:01.745191+0000) 2019-10-13T14:12:02.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:02.010+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:02.014017+0000) 2019-10-13T14:12:03.173 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:12:03.174 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:03.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:12:03.178 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:03.606 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:03.601+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.602859+0000) 2019-10-13T14:12:03.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:03.601+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.603215+0000) 2019-10-13T14:12:03.607 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:03.602+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.605001+0000) 2019-10-13T14:12:03.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:03.605+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.606714+0000) 2019-10-13T14:12:03.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:03.605+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.606988+0000) 2019-10-13T14:12:03.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:03.605+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.607458+0000) 2019-10-13T14:12:03.610 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:03.605+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.608136+0000) 2019-10-13T14:12:03.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:03.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:03.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:03.610+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:03.613366+0000) 2019-10-13T14:12:04.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:04.341+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:04.342800+0000) 2019-10-13T14:12:04.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:04.625+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:04.628594+0000) 2019-10-13T14:12:08.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:07.996+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:07.997280+0000) 2019-10-13T14:12:08.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:08.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.223687+0000) 2019-10-13T14:12:08.618 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:08.612+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.615654+0000) 2019-10-13T14:12:08.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:08.612+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.617291+0000) 2019-10-13T14:12:08.621 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:08.616+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.617889+0000) 2019-10-13T14:12:08.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:08.616+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.618226+0000) 2019-10-13T14:12:08.622 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:08.616+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.618471+0000) 2019-10-13T14:12:08.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:08.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.620007+0000) 2019-10-13T14:12:08.623 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:08.616+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.619314+0000) 2019-10-13T14:12:08.626 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:08.617+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.623389+0000) 2019-10-13T14:12:08.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:08.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:08.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:08.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:08.761909+0000) 2019-10-13T14:12:09.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:09.437+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:09.440447+0000) 2019-10-13T14:12:11.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:11.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:11.745394+0000) 2019-10-13T14:12:12.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:12.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:12.014174+0000) 2019-10-13T14:12:13.632 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:13.624+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.629080+0000) 2019-10-13T14:12:13.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:13.625+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.629623+0000) 2019-10-13T14:12:13.633 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:13.628+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.630213+0000) 2019-10-13T14:12:13.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:13.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.631224+0000) 2019-10-13T14:12:13.634 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:13.628+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.631851+0000) 2019-10-13T14:12:13.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:13.628+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.631854+0000) 2019-10-13T14:12:13.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:13.628+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.632857+0000) 2019-10-13T14:12:13.642 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:13.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:13.639359+0000) 2019-10-13T14:12:13.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:13.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199441 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:14.345 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:14.340+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:14.343024+0000) 2019-10-13T14:12:14.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:14.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:14.628790+0000) 2019-10-13T14:12:18.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:17.992+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:17.997478+0000) 2019-10-13T14:12:18.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:18.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.223909+0000) 2019-10-13T14:12:18.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:18.640+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.642872+0000) 2019-10-13T14:12:18.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:18.640+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.643096+0000) 2019-10-13T14:12:18.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:18.640+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.644351+0000) 2019-10-13T14:12:18.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:18.641+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.645266+0000) 2019-10-13T14:12:18.648 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:18.640+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.645364+0000) 2019-10-13T14:12:18.649 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:18.644+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.645975+0000) 2019-10-13T14:12:18.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:18.644+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.646140+0000) 2019-10-13T14:12:18.654 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:18.648+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:18.655 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:18.649+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.653424+0000) 2019-10-13T14:12:18.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:18.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:18.762116+0000) 2019-10-13T14:12:19.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:19.445+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:19.446976+0000) 2019-10-13T14:12:21.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:21.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:21.745594+0000) 2019-10-13T14:12:22.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:22.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:22.014365+0000) 2019-10-13T14:12:23.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:23.652+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.656130+0000) 2019-10-13T14:12:23.659 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:23.652+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.656358+0000) 2019-10-13T14:12:23.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:23.656+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.657824+0000) 2019-10-13T14:12:23.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:23.656+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.658508+0000) 2019-10-13T14:12:23.662 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:23.656+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.659462+0000) 2019-10-13T14:12:23.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:23.657+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.660194+0000) 2019-10-13T14:12:23.663 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:23.656+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.660389+0000) 2019-10-13T14:12:23.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:23.661+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:23.664505+0000) 2019-10-13T14:12:23.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:23.664+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199502 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:24.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:24.340+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:24.343235+0000) 2019-10-13T14:12:24.631 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:24.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:24.628962+0000) 2019-10-13T14:12:28.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:27.996+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:27.997779+0000) 2019-10-13T14:12:28.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:28.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.224113+0000) 2019-10-13T14:12:28.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:28.668+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.670110+0000) 2019-10-13T14:12:28.673 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:28.668+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.670730+0000) 2019-10-13T14:12:28.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:28.668+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.671260+0000) 2019-10-13T14:12:28.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:28.668+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.671720+0000) 2019-10-13T14:12:28.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:28.669+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.672958+0000) 2019-10-13T14:12:28.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:28.668+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.673142+0000) 2019-10-13T14:12:28.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:28.672+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.674608+0000) 2019-10-13T14:12:28.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:28.673+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.677157+0000) 2019-10-13T14:12:28.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:28.676+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:28.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:28.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:28.762305+0000) 2019-10-13T14:12:29.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:29.449+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:29.453596+0000) 2019-10-13T14:12:31.748 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:31.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:31.745801+0000) 2019-10-13T14:12:32.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:32.009+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:32.014549+0000) 2019-10-13T14:12:33.221 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:12:33.222 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:33.225 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:12:33.226 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:12:33.687 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:33.680+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.684287+0000) 2019-10-13T14:12:33.688 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:33.680+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.684832+0000) 2019-10-13T14:12:33.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:33.680+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.685229+0000) 2019-10-13T14:12:33.688 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:33.684+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.685924+0000) 2019-10-13T14:12:33.689 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:33.684+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.686753+0000) 2019-10-13T14:12:33.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:33.684+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.687127+0000) 2019-10-13T14:12:33.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:33.681+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.687798+0000) 2019-10-13T14:12:33.694 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:33.685+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:33.691949+0000) 2019-10-13T14:12:33.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:33.692+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199564 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:34.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:34.340+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:34.343423+0000) 2019-10-13T14:12:34.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:34.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:34.629167+0000) 2019-10-13T14:12:38.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:37.996+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:37.997976+0000) 2019-10-13T14:12:38.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:38.220+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.224309+0000) 2019-10-13T14:12:38.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:38.693+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.698232+0000) 2019-10-13T14:12:38.701 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:38.696+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.698678+0000) 2019-10-13T14:12:38.701 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:38.696+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.699019+0000) 2019-10-13T14:12:38.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:38.696+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.699743+0000) 2019-10-13T14:12:38.703 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:38.696+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.701246+0000) 2019-10-13T14:12:38.704 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:38.700+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.701977+0000) 2019-10-13T14:12:38.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:38.700+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.702209+0000) 2019-10-13T14:12:38.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:38.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199594 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:38.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:38.705+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.708663+0000) 2019-10-13T14:12:38.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:38.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:38.762477+0000) 2019-10-13T14:12:39.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:39.461+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:39.462536+0000) 2019-10-13T14:12:41.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:41.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:41.746006+0000) 2019-10-13T14:12:42.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:42.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:42.014742+0000) 2019-10-13T14:12:43.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:43.712+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.714057+0000) 2019-10-13T14:12:43.717 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:43.709+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.714305+0000) 2019-10-13T14:12:43.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:43.712+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.715749+0000) 2019-10-13T14:12:43.718 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:43.712+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.716079+0000) 2019-10-13T14:12:43.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:43.712+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.716597+0000) 2019-10-13T14:12:43.719 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:43.712+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.716980+0000) 2019-10-13T14:12:43.720 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:43.716+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.717827+0000) 2019-10-13T14:12:43.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:43.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:43.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:43.717+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:43.721998+0000) 2019-10-13T14:12:44.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:44.340+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:44.343615+0000) 2019-10-13T14:12:44.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:44.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:44.629346+0000) 2019-10-13T14:12:48.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:47.996+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:47.998169+0000) 2019-10-13T14:12:48.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:48.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.224536+0000) 2019-10-13T14:12:48.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:48.724+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.727291+0000) 2019-10-13T14:12:48.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:48.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.727589+0000) 2019-10-13T14:12:48.732 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:48.724+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.729392+0000) 2019-10-13T14:12:48.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:48.725+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.729504+0000) 2019-10-13T14:12:48.732 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:48.724+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.729707+0000) 2019-10-13T14:12:48.733 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:48.728+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.729961+0000) 2019-10-13T14:12:48.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:48.728+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.730352+0000) 2019-10-13T14:12:48.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:48.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199654 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:48.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:48.733+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.737116+0000) 2019-10-13T14:12:48.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:48.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:48.762639+0000) 2019-10-13T14:12:49.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:49.465+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:49.469024+0000) 2019-10-13T14:12:51.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:51.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:51.746210+0000) 2019-10-13T14:12:52.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:52.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:52.014960+0000) 2019-10-13T14:12:53.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:53.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.738468+0000) 2019-10-13T14:12:53.744 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:53.736+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.739644+0000) 2019-10-13T14:12:53.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:53.736+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.739359+0000) 2019-10-13T14:12:53.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:53.736+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.740831+0000) 2019-10-13T14:12:53.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:53.736+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.741126+0000) 2019-10-13T14:12:53.745 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:53.737+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.741612+0000) 2019-10-13T14:12:53.746 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:53.740+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.742089+0000) 2019-10-13T14:12:53.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:53.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199685 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:53.751 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:53.745+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:53.749076+0000) 2019-10-13T14:12:54.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:54.340+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:54.343755+0000) 2019-10-13T14:12:54.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:54.624+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:54.629522+0000) 2019-10-13T14:12:58.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:57.996+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:57.998368+0000) 2019-10-13T14:12:58.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:58.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.224768+0000) 2019-10-13T14:12:58.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:12:58.748+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.750805+0000) 2019-10-13T14:12:58.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:12:58.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.752622+0000) 2019-10-13T14:12:58.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:12:58.752+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.754120+0000) 2019-10-13T14:12:58.757 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:12:58.749+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.755022+0000) 2019-10-13T14:12:58.758 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:12:58.752+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.755349+0000) 2019-10-13T14:12:58.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:12:58.752+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.755751+0000) 2019-10-13T14:12:58.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:58.752+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.756515+0000) 2019-10-13T14:12:58.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:12:58.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:12:58.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:58.757+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.762546+0000) 2019-10-13T14:12:58.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:12:58.760+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:58.762763+0000) 2019-10-13T14:12:59.478 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:12:59.473+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:12:59.476055+0000) 2019-10-13T14:13:01.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:01.744+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:01.746368+0000) 2019-10-13T14:13:02.017 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:02.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:02.015176+0000) 2019-10-13T14:13:03.283 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:13:03.284 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:03.287 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:13:03.287 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:03.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:03.760+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.765143+0000) 2019-10-13T14:13:03.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:03.761+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.765615+0000) 2019-10-13T14:13:03.769 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:03.764+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.766270+0000) 2019-10-13T14:13:03.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:03.764+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.766957+0000) 2019-10-13T14:13:03.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:03.764+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.767431+0000) 2019-10-13T14:13:03.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:03.764+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.769229+0000) 2019-10-13T14:13:03.774 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:03.768+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.771770+0000) 2019-10-13T14:13:03.778 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:03.772+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:03.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:03.773+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:03.776980+0000) 2019-10-13T14:13:04.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:04.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:04.343932+0000) 2019-10-13T14:13:04.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:04.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:04.629781+0000) 2019-10-13T14:13:08.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:07.995+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:07.998569+0000) 2019-10-13T14:13:08.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:08.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.224961+0000) 2019-10-13T14:13:08.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:08.759+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.762945+0000) 2019-10-13T14:13:08.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:08.775+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.778636+0000) 2019-10-13T14:13:08.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:08.775+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.779068+0000) 2019-10-13T14:13:08.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:08.775+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.779373+0000) 2019-10-13T14:13:08.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:08.775+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.780283+0000) 2019-10-13T14:13:08.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:08.777+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.781025+0000) 2019-10-13T14:13:08.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:08.775+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.781672+0000) 2019-10-13T14:13:08.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:08.779+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.782408+0000) 2019-10-13T14:13:08.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:08.781+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:08.787148+0000) 2019-10-13T14:13:08.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:08.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:09.485 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:09.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:09.483071+0000) 2019-10-13T14:13:11.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:11.743+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:11.746580+0000) 2019-10-13T14:13:12.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:12.012+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:12.015368+0000) 2019-10-13T14:13:13.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:13.787+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.792287+0000) 2019-10-13T14:13:13.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:13.791+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.793924+0000) 2019-10-13T14:13:13.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:13.791+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.794631+0000) 2019-10-13T14:13:13.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:13.791+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.794979+0000) 2019-10-13T14:13:13.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:13.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.795594+0000) 2019-10-13T14:13:13.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:13.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.796007+0000) 2019-10-13T14:13:13.802 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:13.795+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.799399+0000) 2019-10-13T14:13:13.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:13.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199805 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:13.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:13.804+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:13.808175+0000) 2019-10-13T14:13:14.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:14.343+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:14.344154+0000) 2019-10-13T14:13:14.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:14.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:14.629954+0000) 2019-10-13T14:13:18.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:17.995+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:17.998766+0000) 2019-10-13T14:13:18.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:18.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.225157+0000) 2019-10-13T14:13:18.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:18.759+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.763142+0000) 2019-10-13T14:13:18.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:18.803+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.805793+0000) 2019-10-13T14:13:18.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:18.803+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.806396+0000) 2019-10-13T14:13:18.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:18.803+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.807031+0000) 2019-10-13T14:13:18.810 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:18.803+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.807322+0000) 2019-10-13T14:13:18.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:18.800+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.807825+0000) 2019-10-13T14:13:18.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:18.803+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.808049+0000) 2019-10-13T14:13:18.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:18.803+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.809035+0000) 2019-10-13T14:13:18.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:18.808+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:18.814998+0000) 2019-10-13T14:13:18.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:18.811+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:19.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:19.484+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:19.489389+0000) 2019-10-13T14:13:21.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:21.743+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:21.746783+0000) 2019-10-13T14:13:22.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:22.012+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:22.015558+0000) 2019-10-13T14:13:23.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:23.815+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.819658+0000) 2019-10-13T14:13:23.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:23.816+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.821659+0000) 2019-10-13T14:13:23.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:23.819+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.821982+0000) 2019-10-13T14:13:23.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:23.819+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.823194+0000) 2019-10-13T14:13:23.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:23.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.824221+0000) 2019-10-13T14:13:23.827 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:23.819+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.823650+0000) 2019-10-13T14:13:23.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:23.819+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.824922+0000) 2019-10-13T14:13:23.833 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:23.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199864 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:23.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:23.828+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:23.832181+0000) 2019-10-13T14:13:24.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:24.343+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:24.344365+0000) 2019-10-13T14:13:24.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:24.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:24.630166+0000) 2019-10-13T14:13:28.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:27.995+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:27.998992+0000) 2019-10-13T14:13:28.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:28.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.225383+0000) 2019-10-13T14:13:28.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:28.759+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.763363+0000) 2019-10-13T14:13:28.837 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:28.831+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.834403+0000) 2019-10-13T14:13:28.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:28.831+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.835160+0000) 2019-10-13T14:13:28.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:28.831+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.836693+0000) 2019-10-13T14:13:28.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:28.832+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.837072+0000) 2019-10-13T14:13:28.841 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:28.835+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.838380+0000) 2019-10-13T14:13:28.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:28.835+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.838771+0000) 2019-10-13T14:13:28.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:28.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.839676+0000) 2019-10-13T14:13:28.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:28.840+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:28.844410+0000) 2019-10-13T14:13:28.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:28.839+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199893 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:29.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:29.492+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:29.496315+0000) 2019-10-13T14:13:31.749 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:31.743+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:31.746984+0000) 2019-10-13T14:13:32.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:32.008+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:32.015794+0000) 2019-10-13T14:13:33.401 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:13:33.401 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:33.405 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:13:33.405 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:13:33.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:33.844+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.848398+0000) 2019-10-13T14:13:33.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:33.843+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.849324+0000) 2019-10-13T14:13:33.853 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:33.847+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.850902+0000) 2019-10-13T14:13:33.854 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:33.847+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.851162+0000) 2019-10-13T14:13:33.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:33.847+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.851851+0000) 2019-10-13T14:13:33.855 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:33.847+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.852883+0000) 2019-10-13T14:13:33.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:33.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.852752+0000) 2019-10-13T14:13:33.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:33.852+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:33.857741+0000) 2019-10-13T14:13:33.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:33.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:34.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:34.343+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:34.344540+0000) 2019-10-13T14:13:34.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:34.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:34.630337+0000) 2019-10-13T14:13:38.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:37.995+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:37.999232+0000) 2019-10-13T14:13:38.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:38.219+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.225572+0000) 2019-10-13T14:13:38.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:38.759+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.763515+0000) 2019-10-13T14:13:38.864 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:38.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.861809+0000) 2019-10-13T14:13:38.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:38.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.862916+0000) 2019-10-13T14:13:38.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:38.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.863548+0000) 2019-10-13T14:13:38.866 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:38.859+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.863270+0000) 2019-10-13T14:13:38.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:38.859+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.864210+0000) 2019-10-13T14:13:38.867 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:38.859+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.864996+0000) 2019-10-13T14:13:38.868 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:38.859+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.865564+0000) 2019-10-13T14:13:38.873 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:38.867+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199954 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:38.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:38.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:38.871956+0000) 2019-10-13T14:13:39.506 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:39.500+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:39.503399+0000) 2019-10-13T14:13:41.750 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:41.743+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:41.747208+0000) 2019-10-13T14:13:42.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:42.012+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:42.015995+0000) 2019-10-13T14:13:43.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:43.871+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.874275+0000) 2019-10-13T14:13:43.877 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:43.871+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.874436+0000) 2019-10-13T14:13:43.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:43.871+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.875785+0000) 2019-10-13T14:13:43.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:43.871+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.875471+0000) 2019-10-13T14:13:43.879 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:43.871+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.876763+0000) 2019-10-13T14:13:43.880 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:43.871+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.877137+0000) 2019-10-13T14:13:43.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:43.872+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.877623+0000) 2019-10-13T14:13:43.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:43.876+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:43.882492+0000) 2019-10-13T14:13:43.886 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:43.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:44.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:44.339+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:44.344717+0000) 2019-10-13T14:13:44.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:44.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:44.630506+0000) 2019-10-13T14:13:48.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:47.999+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:47.999458+0000) 2019-10-13T14:13:48.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:48.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.225808+0000) 2019-10-13T14:13:48.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:48.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.763738+0000) 2019-10-13T14:13:48.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:48.883+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.887382+0000) 2019-10-13T14:13:48.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:48.883+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.888095+0000) 2019-10-13T14:13:48.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:48.883+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.888486+0000) 2019-10-13T14:13:48.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:48.887+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.890266+0000) 2019-10-13T14:13:48.893 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:48.887+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.890843+0000) 2019-10-13T14:13:48.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:48.887+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.891364+0000) 2019-10-13T14:13:48.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:48.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.891871+0000) 2019-10-13T14:13:48.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:48.892+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:48.897142+0000) 2019-10-13T14:13:48.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:48.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:49.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:49.508+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:49.510555+0000) 2019-10-13T14:13:51.750 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:51.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:51.747416+0000) 2019-10-13T14:13:52.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:52.012+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:52.016192+0000) 2019-10-13T14:13:53.903 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:53.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.900810+0000) 2019-10-13T14:13:53.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:53.895+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.900985+0000) 2019-10-13T14:13:53.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:53.895+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.901699+0000) 2019-10-13T14:13:53.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:53.899+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.902410+0000) 2019-10-13T14:13:53.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:53.899+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.902441+0000) 2019-10-13T14:13:53.906 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:53.899+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.903570+0000) 2019-10-13T14:13:53.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:53.899+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.903947+0000) 2019-10-13T14:13:53.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:53.900+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:53.906211+0000) 2019-10-13T14:13:53.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:53.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200045 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:54.347 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:54.343+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:54.344912+0000) 2019-10-13T14:13:54.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:54.627+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:54.630686+0000) 2019-10-13T14:13:58.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:57.995+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:57.999650+0000) 2019-10-13T14:13:58.228 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:58.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.226012+0000) 2019-10-13T14:13:58.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:58.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.763953+0000) 2019-10-13T14:13:58.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:13:58.911+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.914274+0000) 2019-10-13T14:13:58.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:13:58.908+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.915182+0000) 2019-10-13T14:13:58.918 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:13:58.911+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.915363+0000) 2019-10-13T14:13:58.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:13:58.911+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.917229+0000) 2019-10-13T14:13:58.922 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:13:58.915+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.919629+0000) 2019-10-13T14:13:58.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:13:58.919+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.922009+0000) 2019-10-13T14:13:58.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:58.916+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.923463+0000) 2019-10-13T14:13:58.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:13:58.919+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:58.924499+0000) 2019-10-13T14:13:58.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:13:58.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:13:59.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:13:59.516+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:13:59.517795+0000) 2019-10-13T14:14:01.750 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:01.742+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:01.747633+0000) 2019-10-13T14:14:02.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:02.016+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:02.016398+0000) 2019-10-13T14:14:03.499 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:14:03.499 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:03.503 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:14:03.503 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:03.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:03.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.927871+0000) 2019-10-13T14:14:03.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:03.922+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.929014+0000) 2019-10-13T14:14:03.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:03.924+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.929579+0000) 2019-10-13T14:14:03.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:03.926+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.930126+0000) 2019-10-13T14:14:03.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:03.926+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.930709+0000) 2019-10-13T14:14:03.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:03.926+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.931037+0000) 2019-10-13T14:14:03.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:03.926+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.931783+0000) 2019-10-13T14:14:03.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:03.932+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:03.936976+0000) 2019-10-13T14:14:03.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:03.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200104 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:04.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:04.342+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:04.345097+0000) 2019-10-13T14:14:04.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:04.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:04.630841+0000) 2019-10-13T14:14:08.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:07.998+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:07.999854+0000) 2019-10-13T14:14:08.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:08.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.226213+0000) 2019-10-13T14:14:08.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:08.758+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.764179+0000) 2019-10-13T14:14:08.943 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:08.936+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.940543+0000) 2019-10-13T14:14:08.943 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:08.934+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.940863+0000) 2019-10-13T14:14:08.944 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:08.934+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.941419+0000) 2019-10-13T14:14:08.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:08.938+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.942060+0000) 2019-10-13T14:14:08.945 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:08.938+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.942275+0000) 2019-10-13T14:14:08.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:08.938+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.943081+0000) 2019-10-13T14:14:08.946 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:08.938+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.943420+0000) 2019-10-13T14:14:08.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:08.944+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:08.948144+0000) 2019-10-13T14:14:08.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:08.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:09.527 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:09.524+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:09.524813+0000) 2019-10-13T14:14:11.750 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:11.746+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:11.747851+0000) 2019-10-13T14:14:12.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:12.016+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:12.016588+0000) 2019-10-13T14:14:13.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:13.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.954922+0000) 2019-10-13T14:14:13.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:13.950+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.956367+0000) 2019-10-13T14:14:13.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:13.956+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.956939+0000) 2019-10-13T14:14:13.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:13.950+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.957211+0000) 2019-10-13T14:14:13.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:13.950+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.957532+0000) 2019-10-13T14:14:13.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:13.954+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.959322+0000) 2019-10-13T14:14:13.962 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:13.954+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.959707+0000) 2019-10-13T14:14:13.967 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:13.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:13.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:13.964+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:13.965174+0000) 2019-10-13T14:14:14.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:14.338+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:14.345296+0000) 2019-10-13T14:14:14.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:14.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:14.631023+0000) 2019-10-13T14:14:18.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:17.998+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.000065+0000) 2019-10-13T14:14:18.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:18.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.226429+0000) 2019-10-13T14:14:18.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:18.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.764371+0000) 2019-10-13T14:14:18.971 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:18.962+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.968160+0000) 2019-10-13T14:14:18.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:18.966+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.970064+0000) 2019-10-13T14:14:18.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:18.967+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.970696+0000) 2019-10-13T14:14:18.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:18.966+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.971444+0000) 2019-10-13T14:14:18.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:18.966+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.971795+0000) 2019-10-13T14:14:18.976 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:18.966+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.973340+0000) 2019-10-13T14:14:18.976 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:18.970+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.974187+0000) 2019-10-13T14:14:18.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:18.975+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:18.977152+0000) 2019-10-13T14:14:18.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:18.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200193 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:19.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:19.527+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:19.531879+0000) 2019-10-13T14:14:21.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:21.746+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:21.748054+0000) 2019-10-13T14:14:22.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:22.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:22.016800+0000) 2019-10-13T14:14:23.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:23.978+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.981492+0000) 2019-10-13T14:14:23.985 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:23.978+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.982024+0000) 2019-10-13T14:14:23.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:23.978+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.982681+0000) 2019-10-13T14:14:23.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:23.978+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.983202+0000) 2019-10-13T14:14:23.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:23.979+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.983463+0000) 2019-10-13T14:14:23.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:23.978+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.984276+0000) 2019-10-13T14:14:23.987 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:23.978+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.984841+0000) 2019-10-13T14:14:23.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:23.987+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:23.990316+0000) 2019-10-13T14:14:23.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:23.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200223 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:24.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:24.342+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:24.345522+0000) 2019-10-13T14:14:24.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:24.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:24.631195+0000) 2019-10-13T14:14:28.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:27.994+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.000264+0000) 2019-10-13T14:14:28.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:28.222+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.226675+0000) 2019-10-13T14:14:28.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:28.758+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.764558+0000) 2019-10-13T14:14:28.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:28.990+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.995696+0000) 2019-10-13T14:14:28.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:28.990+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.996354+0000) 2019-10-13T14:14:28.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:28.990+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.996858+0000) 2019-10-13T14:14:29.001 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:28.994+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.998915+0000) 2019-10-13T14:14:29.002 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:28.995+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:28.999364+0000) 2019-10-13T14:14:29.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:28.994+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:29.000777+0000) 2019-10-13T14:14:29.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:28.998+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:29.003207+0000) 2019-10-13T14:14:29.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:28.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:29.014 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:29.011+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:29.012157+0000) 2019-10-13T14:14:29.542 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:29.535+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:29.538970+0000) 2019-10-13T14:14:31.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:31.746+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:31.748236+0000) 2019-10-13T14:14:32.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:32.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:32.016998+0000) 2019-10-13T14:14:33.646 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:14:33.646 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:33.650 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:14:33.650 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:14:34.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:34.002+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.007945+0000) 2019-10-13T14:14:34.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:34.006+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.010721+0000) 2019-10-13T14:14:34.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:34.006+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.012672+0000) 2019-10-13T14:14:34.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:34.010+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.014859+0000) 2019-10-13T14:14:34.017 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:34.010+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.014919+0000) 2019-10-13T14:14:34.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:34.010+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.016854+0000) 2019-10-13T14:14:34.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:34.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.016821+0000) 2019-10-13T14:14:34.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:34.015+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.017262+0000) 2019-10-13T14:14:34.022 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:34.014+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200283 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:34.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:34.342+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.345718+0000) 2019-10-13T14:14:34.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:34.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:34.631372+0000) 2019-10-13T14:14:38.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:37.998+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:38.000460+0000) 2019-10-13T14:14:38.229 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:38.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:38.226906+0000) 2019-10-13T14:14:38.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:38.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:38.764787+0000) 2019-10-13T14:14:39.025 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:39.018+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.022265+0000) 2019-10-13T14:14:39.026 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:39.018+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.023322+0000) 2019-10-13T14:14:39.027 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:39.018+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.024397+0000) 2019-10-13T14:14:39.027 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:39.018+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.024766+0000) 2019-10-13T14:14:39.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:39.018+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.025183+0000) 2019-10-13T14:14:39.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:39.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.025720+0000) 2019-10-13T14:14:39.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:39.022+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.026870+0000) 2019-10-13T14:14:39.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:39.031+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.032624+0000) 2019-10-13T14:14:39.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:39.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:39.548 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:39.543+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:39.546047+0000) 2019-10-13T14:14:41.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:41.746+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:41.748392+0000) 2019-10-13T14:14:42.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:42.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:42.017191+0000) 2019-10-13T14:14:44.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:44.030+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.036575+0000) 2019-10-13T14:14:44.039 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:44.030+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.037074+0000) 2019-10-13T14:14:44.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:44.035+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.037622+0000) 2019-10-13T14:14:44.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:44.034+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.038320+0000) 2019-10-13T14:14:44.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:44.034+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.039105+0000) 2019-10-13T14:14:44.042 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:44.034+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.039612+0000) 2019-10-13T14:14:44.044 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:44.034+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.041399+0000) 2019-10-13T14:14:44.047 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:44.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.044561+0000) 2019-10-13T14:14:44.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:44.042+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:44.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:44.342+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.345929+0000) 2019-10-13T14:14:44.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:44.626+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:44.631551+0000) 2019-10-13T14:14:48.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:47.998+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:48.000674+0000) 2019-10-13T14:14:48.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:48.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:48.227102+0000) 2019-10-13T14:14:48.767 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:48.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:48.764974+0000) 2019-10-13T14:14:49.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:49.046+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.050029+0000) 2019-10-13T14:14:49.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:49.046+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.050339+0000) 2019-10-13T14:14:49.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:49.046+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.050647+0000) 2019-10-13T14:14:49.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:49.046+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.052069+0000) 2019-10-13T14:14:49.055 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:49.051+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.052764+0000) 2019-10-13T14:14:49.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:49.046+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.052800+0000) 2019-10-13T14:14:49.056 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:49.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.053376+0000) 2019-10-13T14:14:49.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:49.055+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.057745+0000) 2019-10-13T14:14:49.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:49.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200369 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:49.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:49.551+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:49.553135+0000) 2019-10-13T14:14:51.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:51.746+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:51.748550+0000) 2019-10-13T14:14:52.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:52.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:52.017406+0000) 2019-10-13T14:14:54.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:54.058+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.062717+0000) 2019-10-13T14:14:54.066 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:54.058+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.063278+0000) 2019-10-13T14:14:54.066 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:54.058+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.064037+0000) 2019-10-13T14:14:54.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:54.058+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.064397+0000) 2019-10-13T14:14:54.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:54.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.065209+0000) 2019-10-13T14:14:54.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:54.063+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.065695+0000) 2019-10-13T14:14:54.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:54.062+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.066005+0000) 2019-10-13T14:14:54.075 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:54.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:54.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:54.071+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.072592+0000) 2019-10-13T14:14:54.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:54.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.346150+0000) 2019-10-13T14:14:54.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:54.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:54.631762+0000) 2019-10-13T14:14:58.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:57.997+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:58.000902+0000) 2019-10-13T14:14:58.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:58.221+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:58.227291+0000) 2019-10-13T14:14:58.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:58.761+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:58.765164+0000) 2019-10-13T14:14:59.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:14:59.069+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.075468+0000) 2019-10-13T14:14:59.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:14:59.069+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.075946+0000) 2019-10-13T14:14:59.080 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:14:59.069+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.077426+0000) 2019-10-13T14:14:59.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:14:59.075+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.078262+0000) 2019-10-13T14:14:59.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:14:59.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.078522+0000) 2019-10-13T14:14:59.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:14:59.073+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.079216+0000) 2019-10-13T14:14:59.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:14:59.073+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.079932+0000) 2019-10-13T14:14:59.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:59.083+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.084184+0000) 2019-10-13T14:14:59.089 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:14:59.081+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200428 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:14:59.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:14:59.559+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:14:59.560181+0000) 2019-10-13T14:15:01.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:01.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:01.748770+0000) 2019-10-13T14:15:02.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:02.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:02.017606+0000) 2019-10-13T14:15:03.747 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:15:03.747 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:03.751 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:15:03.751 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:04.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:04.081+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.089461+0000) 2019-10-13T14:15:04.093 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:04.087+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.090739+0000) 2019-10-13T14:15:04.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:04.085+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.090914+0000) 2019-10-13T14:15:04.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:04.085+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.091119+0000) 2019-10-13T14:15:04.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:04.085+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.091636+0000) 2019-10-13T14:15:04.095 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:04.085+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.093197+0000) 2019-10-13T14:15:04.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:04.085+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.093562+0000) 2019-10-13T14:15:04.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:04.095+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.096697+0000) 2019-10-13T14:15:04.101 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:04.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:04.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:04.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.346363+0000) 2019-10-13T14:15:04.634 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:04.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:04.631896+0000) 2019-10-13T14:15:08.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:07.997+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:08.001125+0000) 2019-10-13T14:15:08.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:08.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:08.227489+0000) 2019-10-13T14:15:08.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:08.761+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:08.765373+0000) 2019-10-13T14:15:09.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:09.097+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.104453+0000) 2019-10-13T14:15:09.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:09.097+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.104974+0000) 2019-10-13T14:15:09.108 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:09.101+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.105833+0000) 2019-10-13T14:15:09.109 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:09.101+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.106748+0000) 2019-10-13T14:15:09.109 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:09.101+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.106943+0000) 2019-10-13T14:15:09.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:09.107+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.108106+0000) 2019-10-13T14:15:09.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:09.101+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.108249+0000) 2019-10-13T14:15:09.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:09.107+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.110121+0000) 2019-10-13T14:15:09.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:09.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200487 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:09.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:09.567+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:09.567234+0000) 2019-10-13T14:15:11.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:11.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:11.748974+0000) 2019-10-13T14:15:12.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:12.015+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:12.017832+0000) 2019-10-13T14:15:14.118 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:14.113+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.115354+0000) 2019-10-13T14:15:14.118 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:14.113+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.116373+0000) 2019-10-13T14:15:14.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:14.113+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.116929+0000) 2019-10-13T14:15:14.121 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:14.117+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.118216+0000) 2019-10-13T14:15:14.121 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:14.117+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.118789+0000) 2019-10-13T14:15:14.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:14.117+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.119134+0000) 2019-10-13T14:15:14.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:14.115+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.119132+0000) 2019-10-13T14:15:14.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:14.119+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.123791+0000) 2019-10-13T14:15:14.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:14.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200520 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:14.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:14.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.346564+0000) 2019-10-13T14:15:14.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:14.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:14.632077+0000) 2019-10-13T14:15:18.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:17.997+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:18.001308+0000) 2019-10-13T14:15:18.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:18.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:18.227717+0000) 2019-10-13T14:15:18.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:18.761+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:18.765559+0000) 2019-10-13T14:15:19.131 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:19.125+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.128528+0000) 2019-10-13T14:15:19.132 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:19.129+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.130190+0000) 2019-10-13T14:15:19.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:19.129+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.132234+0000) 2019-10-13T14:15:19.135 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:19.131+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.132551+0000) 2019-10-13T14:15:19.136 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:19.129+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.133162+0000) 2019-10-13T14:15:19.136 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:19.129+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.133547+0000) 2019-10-13T14:15:19.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:19.133+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.133833+0000) 2019-10-13T14:15:19.141 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:19.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:19.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:19.139+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.140588+0000) 2019-10-13T14:15:19.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:19.571+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:19.573663+0000) 2019-10-13T14:15:21.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:21.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:21.749172+0000) 2019-10-13T14:15:22.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:22.014+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:22.018029+0000) 2019-10-13T14:15:24.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:24.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.142345+0000) 2019-10-13T14:15:24.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:24.141+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.144183+0000) 2019-10-13T14:15:24.147 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:24.141+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.144180+0000) 2019-10-13T14:15:24.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:24.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.144433+0000) 2019-10-13T14:15:24.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:24.145+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.146237+0000) 2019-10-13T14:15:24.151 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:24.145+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.148396+0000) 2019-10-13T14:15:24.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:24.146+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.149951+0000) 2019-10-13T14:15:24.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:24.149+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.150482+0000) 2019-10-13T14:15:24.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:24.149+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:24.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:24.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.346746+0000) 2019-10-13T14:15:24.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:24.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:24.632271+0000) 2019-10-13T14:15:28.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:27.997+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:28.001505+0000) 2019-10-13T14:15:28.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:28.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:28.227920+0000) 2019-10-13T14:15:28.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:28.765+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:28.765772+0000) 2019-10-13T14:15:29.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:29.153+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.155635+0000) 2019-10-13T14:15:29.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:29.153+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.156472+0000) 2019-10-13T14:15:29.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:29.154+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.157185+0000) 2019-10-13T14:15:29.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:29.153+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.157266+0000) 2019-10-13T14:15:29.160 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:29.153+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.157608+0000) 2019-10-13T14:15:29.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:29.157+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.157854+0000) 2019-10-13T14:15:29.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:29.157+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.158454+0000) 2019-10-13T14:15:29.165 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:29.158+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.162486+0000) 2019-10-13T14:15:29.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:29.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:29.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:29.578+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:29.580878+0000) 2019-10-13T14:15:31.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:31.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:31.749382+0000) 2019-10-13T14:15:32.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:32.014+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:32.018260+0000) 2019-10-13T14:15:33.894 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:15:33.894 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:33.897 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:15:33.898 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:15:34.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:34.165+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.169079+0000) 2019-10-13T14:15:34.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:34.169+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.171226+0000) 2019-10-13T14:15:34.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:34.169+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.171865+0000) 2019-10-13T14:15:34.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:34.170+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.172766+0000) 2019-10-13T14:15:34.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:34.169+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.173232+0000) 2019-10-13T14:15:34.176 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:34.169+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.173458+0000) 2019-10-13T14:15:34.176 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:34.173+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.173871+0000) 2019-10-13T14:15:34.181 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:34.177+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:34.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:34.174+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.179482+0000) 2019-10-13T14:15:34.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:34.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.346911+0000) 2019-10-13T14:15:34.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:34.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:34.632459+0000) 2019-10-13T14:15:38.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:37.997+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:38.001703+0000) 2019-10-13T14:15:38.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:38.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:38.228115+0000) 2019-10-13T14:15:38.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:38.765+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:38.765930+0000) 2019-10-13T14:15:39.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:39.181+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.182021+0000) 2019-10-13T14:15:39.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:39.181+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.183093+0000) 2019-10-13T14:15:39.186 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:39.181+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.183419+0000) 2019-10-13T14:15:39.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:39.178+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.183895+0000) 2019-10-13T14:15:39.187 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:39.181+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.184280+0000) 2019-10-13T14:15:39.187 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:39.181+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.185059+0000) 2019-10-13T14:15:39.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:39.181+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.184637+0000) 2019-10-13T14:15:39.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:39.186+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.191520+0000) 2019-10-13T14:15:39.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:39.189+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:39.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:39.590+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:39.592339+0000) 2019-10-13T14:15:41.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:41.745+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:41.749581+0000) 2019-10-13T14:15:42.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:42.014+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:42.018464+0000) 2019-10-13T14:15:44.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:44.193+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.196639+0000) 2019-10-13T14:15:44.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:44.193+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.197714+0000) 2019-10-13T14:15:44.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:44.197+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.198444+0000) 2019-10-13T14:15:44.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:44.197+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.199232+0000) 2019-10-13T14:15:44.202 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:44.197+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.199560+0000) 2019-10-13T14:15:44.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:44.198+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.200601+0000) 2019-10-13T14:15:44.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:44.201+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.202075+0000) 2019-10-13T14:15:44.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:44.202+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.205838+0000) 2019-10-13T14:15:44.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:44.205+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:44.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:44.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.347140+0000) 2019-10-13T14:15:44.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:44.629+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:44.632669+0000) 2019-10-13T14:15:48.004 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:48.001+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:48.001976+0000) 2019-10-13T14:15:48.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:48.225+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:48.228268+0000) 2019-10-13T14:15:48.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:48.765+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:48.766139+0000) 2019-10-13T14:15:49.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:49.205+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.209566+0000) 2019-10-13T14:15:49.213 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:49.209+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.209963+0000) 2019-10-13T14:15:49.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:49.209+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.212727+0000) 2019-10-13T14:15:49.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:49.209+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.212966+0000) 2019-10-13T14:15:49.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:49.209+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.213341+0000) 2019-10-13T14:15:49.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:49.210+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.214750+0000) 2019-10-13T14:15:49.218 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:49.213+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.216176+0000) 2019-10-13T14:15:49.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:49.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200729 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:49.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:49.218+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.221028+0000) 2019-10-13T14:15:49.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:49.594+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:49.598420+0000) 2019-10-13T14:15:51.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:51.749+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:51.749790+0000) 2019-10-13T14:15:52.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:52.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:52.018717+0000) 2019-10-13T14:15:54.226 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:54.220+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.222878+0000) 2019-10-13T14:15:54.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:54.220+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.223299+0000) 2019-10-13T14:15:54.226 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:54.220+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.224388+0000) 2019-10-13T14:15:54.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:54.220+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.225015+0000) 2019-10-13T14:15:54.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:54.220+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.225052+0000) 2019-10-13T14:15:54.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:54.220+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.225678+0000) 2019-10-13T14:15:54.230 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:54.222+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.227960+0000) 2019-10-13T14:15:54.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:54.226+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.231840+0000) 2019-10-13T14:15:54.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:54.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200759 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:54.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:54.344+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.347321+0000) 2019-10-13T14:15:54.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:54.628+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:54.632843+0000) 2019-10-13T14:15:58.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:58.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:58.002167+0000) 2019-10-13T14:15:58.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:58.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:58.228463+0000) 2019-10-13T14:15:58.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:58.764+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:58.766350+0000) 2019-10-13T14:15:59.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:15:59.232+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.237671+0000) 2019-10-13T14:15:59.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:15:59.234+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.238260+0000) 2019-10-13T14:15:59.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:15:59.236+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.238538+0000) 2019-10-13T14:15:59.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:15:59.236+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.239258+0000) 2019-10-13T14:15:59.242 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:15:59.236+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.239521+0000) 2019-10-13T14:15:59.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:15:59.236+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.239955+0000) 2019-10-13T14:15:59.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:15:59.236+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.240271+0000) 2019-10-13T14:15:59.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:59.238+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.243871+0000) 2019-10-13T14:15:59.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:15:59.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200788 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:15:59.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:15:59.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:15:59.604800+0000) 2019-10-13T14:16:01.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:01.748+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:01.750007+0000) 2019-10-13T14:16:02.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:02.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:02.018916+0000) 2019-10-13T14:16:03.999 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:16:03.999 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:04.003 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:16:04.003 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:04.254 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:04.248+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.251152+0000) 2019-10-13T14:16:04.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:04.248+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.251764+0000) 2019-10-13T14:16:04.255 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:04.248+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.253038+0000) 2019-10-13T14:16:04.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:04.248+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.253198+0000) 2019-10-13T14:16:04.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:04.252+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.253724+0000) 2019-10-13T14:16:04.257 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:04.250+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.254515+0000) 2019-10-13T14:16:04.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:04.252+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.254287+0000) 2019-10-13T14:16:04.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:04.254+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.259843+0000) 2019-10-13T14:16:04.263 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:04.256+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200818 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:04.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:04.344+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.347491+0000) 2019-10-13T14:16:04.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:04.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:04.633016+0000) 2019-10-13T14:16:08.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:08.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:08.002400+0000) 2019-10-13T14:16:08.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:08.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:08.228655+0000) 2019-10-13T14:16:08.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:08.764+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:08.766552+0000) 2019-10-13T14:16:09.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:09.260+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.264218+0000) 2019-10-13T14:16:09.267 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:09.260+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.265002+0000) 2019-10-13T14:16:09.268 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:09.260+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.265208+0000) 2019-10-13T14:16:09.269 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:09.262+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.267423+0000) 2019-10-13T14:16:09.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:09.264+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.269557+0000) 2019-10-13T14:16:09.273 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:09.266+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.271147+0000) 2019-10-13T14:16:09.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:09.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.271689+0000) 2019-10-13T14:16:09.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:09.272+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.274347+0000) 2019-10-13T14:16:09.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:09.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200849 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:09.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:09.610+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:09.611305+0000) 2019-10-13T14:16:11.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:11.748+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:11.750269+0000) 2019-10-13T14:16:12.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:12.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:12.019116+0000) 2019-10-13T14:16:14.280 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:14.272+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.277346+0000) 2019-10-13T14:16:14.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:14.278+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.280195+0000) 2019-10-13T14:16:14.283 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:14.276+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.280173+0000) 2019-10-13T14:16:14.283 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:14.276+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.280491+0000) 2019-10-13T14:16:14.284 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:14.278+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.281053+0000) 2019-10-13T14:16:14.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:14.280+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.282008+0000) 2019-10-13T14:16:14.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:14.280+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.282578+0000) 2019-10-13T14:16:14.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:14.280+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.282894+0000) 2019-10-13T14:16:14.291 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:14.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200879 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:14.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:14.344+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.347703+0000) 2019-10-13T14:16:14.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:14.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:14.633198+0000) 2019-10-13T14:16:18.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:18.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:18.002634+0000) 2019-10-13T14:16:18.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:18.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:18.228843+0000) 2019-10-13T14:16:18.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:18.764+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:18.766744+0000) 2019-10-13T14:16:19.295 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:19.290+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.293034+0000) 2019-10-13T14:16:19.296 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:19.288+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.293493+0000) 2019-10-13T14:16:19.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:19.292+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.293939+0000) 2019-10-13T14:16:19.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:19.292+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.294728+0000) 2019-10-13T14:16:19.297 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:19.292+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.295117+0000) 2019-10-13T14:16:19.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:19.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.296564+0000) 2019-10-13T14:16:19.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:19.292+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.297103+0000) 2019-10-13T14:16:19.303 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:19.296+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:19.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:19.298+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.301774+0000) 2019-10-13T14:16:19.620 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:19.614+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:19.617711+0000) 2019-10-13T14:16:21.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:21.748+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:21.750474+0000) 2019-10-13T14:16:22.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:22.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:22.019315+0000) 2019-10-13T14:16:24.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:24.300+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.303603+0000) 2019-10-13T14:16:24.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:24.300+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.304531+0000) 2019-10-13T14:16:24.307 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:24.300+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.305264+0000) 2019-10-13T14:16:24.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:24.302+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.305749+0000) 2019-10-13T14:16:24.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:24.304+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.306064+0000) 2019-10-13T14:16:24.309 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:24.304+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.306164+0000) 2019-10-13T14:16:24.309 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:24.304+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.307291+0000) 2019-10-13T14:16:24.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:24.310+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.313271+0000) 2019-10-13T14:16:24.317 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:24.312+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:24.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:24.344+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.347912+0000) 2019-10-13T14:16:24.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:24.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:24.633369+0000) 2019-10-13T14:16:28.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:28.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:28.002833+0000) 2019-10-13T14:16:28.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:28.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:28.229055+0000) 2019-10-13T14:16:28.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:28.764+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:28.766981+0000) 2019-10-13T14:16:29.321 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:29.316+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.318323+0000) 2019-10-13T14:16:29.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:29.316+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.318854+0000) 2019-10-13T14:16:29.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:29.316+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.319234+0000) 2019-10-13T14:16:29.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:29.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.321277+0000) 2019-10-13T14:16:29.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:29.316+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.321613+0000) 2019-10-13T14:16:29.324 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:29.320+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.322178+0000) 2019-10-13T14:16:29.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:29.320+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.324629+0000) 2019-10-13T14:16:29.331 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:29.324+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200969 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:29.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:29.325+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.328779+0000) 2019-10-13T14:16:29.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:29.621+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:29.624205+0000) 2019-10-13T14:16:31.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:31.748+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:31.750685+0000) 2019-10-13T14:16:32.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:32.017+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:32.019505+0000) 2019-10-13T14:16:34.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:34.332+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.333999+0000) 2019-10-13T14:16:34.337 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:34.332+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.334553+0000) 2019-10-13T14:16:34.337 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:34.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.335228+0000) 2019-10-13T14:16:34.338 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:34.332+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.336352+0000) 2019-10-13T14:16:34.339 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:34.332+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.336624+0000) 2019-10-13T14:16:34.339 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:34.332+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.336881+0000) 2019-10-13T14:16:34.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:34.332+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.337313+0000) 2019-10-13T14:16:34.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:34.337+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.341157+0000) 2019-10-13T14:16:34.345 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:34.340+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200999 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:34.347 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:16:34.348 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:34.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:34.344+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.348041+0000) 2019-10-13T14:16:34.351 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:16:34.351 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:16:34.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:34.628+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:34.633547+0000) 2019-10-13T14:16:38.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:38.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:38.003026+0000) 2019-10-13T14:16:38.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:38.224+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:38.229212+0000) 2019-10-13T14:16:38.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:38.764+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:38.767194+0000) 2019-10-13T14:16:39.349 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:39.344+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.346511+0000) 2019-10-13T14:16:39.349 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:39.344+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.346905+0000) 2019-10-13T14:16:39.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:39.341+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.347830+0000) 2019-10-13T14:16:39.350 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:39.344+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.347558+0000) 2019-10-13T14:16:39.351 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:39.344+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.348809+0000) 2019-10-13T14:16:39.351 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:39.344+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.349322+0000) 2019-10-13T14:16:39.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:39.344+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.349658+0000) 2019-10-13T14:16:39.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:39.349+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.355171+0000) 2019-10-13T14:16:39.358 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:39.352+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:39.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:39.625+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:39.630804+0000) 2019-10-13T14:16:41.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:41.748+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:41.750915+0000) 2019-10-13T14:16:42.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:42.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:42.019717+0000) 2019-10-13T14:16:44.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:44.348+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.348254+0000) 2019-10-13T14:16:44.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:44.353+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.359374+0000) 2019-10-13T14:16:44.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:44.356+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.359410+0000) 2019-10-13T14:16:44.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:44.356+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.359662+0000) 2019-10-13T14:16:44.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:44.356+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.359965+0000) 2019-10-13T14:16:44.363 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:44.356+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.360281+0000) 2019-10-13T14:16:44.364 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:44.356+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.360740+0000) 2019-10-13T14:16:44.364 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:44.356+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.361049+0000) 2019-10-13T14:16:44.369 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:44.361+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.367211+0000) 2019-10-13T14:16:44.371 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:44.364+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201058 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:44.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:44.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:44.633787+0000) 2019-10-13T14:16:48.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:48.000+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:48.003225+0000) 2019-10-13T14:16:48.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:48.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:48.229426+0000) 2019-10-13T14:16:48.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:48.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:48.767390+0000) 2019-10-13T14:16:49.375 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:49.367+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.372472+0000) 2019-10-13T14:16:49.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:49.367+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.373597+0000) 2019-10-13T14:16:49.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:49.369+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.374235+0000) 2019-10-13T14:16:49.377 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:49.371+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.374360+0000) 2019-10-13T14:16:49.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:49.371+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.374907+0000) 2019-10-13T14:16:49.378 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:49.371+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.375221+0000) 2019-10-13T14:16:49.378 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:49.371+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.375644+0000) 2019-10-13T14:16:49.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:49.375+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:49.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:49.377+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.382669+0000) 2019-10-13T14:16:49.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:49.633+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:49.637256+0000) 2019-10-13T14:16:51.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:51.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:51.751150+0000) 2019-10-13T14:16:52.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:52.013+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:52.019925+0000) 2019-10-13T14:16:54.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:54.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.348502+0000) 2019-10-13T14:16:54.387 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:54.381+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.384971+0000) 2019-10-13T14:16:54.387 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:54.379+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.385094+0000) 2019-10-13T14:16:54.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:54.383+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.385800+0000) 2019-10-13T14:16:54.389 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:54.383+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.386093+0000) 2019-10-13T14:16:54.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:54.383+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.387175+0000) 2019-10-13T14:16:54.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:54.383+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.387236+0000) 2019-10-13T14:16:54.390 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:54.383+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.387665+0000) 2019-10-13T14:16:54.394 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:54.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201119 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:54.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:54.389+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.392811+0000) 2019-10-13T14:16:54.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:54.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:54.633958+0000) 2019-10-13T14:16:58.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:57.999+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:58.003424+0000) 2019-10-13T14:16:58.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:58.223+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:58.229625+0000) 2019-10-13T14:16:58.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:58.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:58.767616+0000) 2019-10-13T14:16:59.397 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:16:59.391+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.395041+0000) 2019-10-13T14:16:59.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:16:59.393+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.397341+0000) 2019-10-13T14:16:59.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:16:59.391+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.397447+0000) 2019-10-13T14:16:59.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:16:59.395+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.397851+0000) 2019-10-13T14:16:59.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:16:59.395+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.398489+0000) 2019-10-13T14:16:59.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:16:59.395+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.399603+0000) 2019-10-13T14:16:59.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:16:59.395+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.399983+0000) 2019-10-13T14:16:59.407 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:16:59.399+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201148 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:16:59.408 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:59.401+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.405667+0000) 2019-10-13T14:16:59.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:16:59.641+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:16:59.643736+0000) 2019-10-13T14:17:01.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:01.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:01.751336+0000) 2019-10-13T14:17:02.022 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:02.017+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:02.020123+0000) 2019-10-13T14:17:04.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:04.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.348738+0000) 2019-10-13T14:17:04.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:04.405+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.408280+0000) 2019-10-13T14:17:04.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:04.403+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.409201+0000) 2019-10-13T14:17:04.412 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:04.407+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.409933+0000) 2019-10-13T14:17:04.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:04.407+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.410486+0000) 2019-10-13T14:17:04.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:04.407+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.410374+0000) 2019-10-13T14:17:04.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:04.407+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.411501+0000) 2019-10-13T14:17:04.414 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:04.407+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.411907+0000) 2019-10-13T14:17:04.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:04.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:04.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:04.417+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.421157+0000) 2019-10-13T14:17:04.567 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:17:04.567 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:04.570 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:17:04.571 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:04.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:04.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:04.634107+0000) 2019-10-13T14:17:08.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:07.999+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:08.003626+0000) 2019-10-13T14:17:08.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:08.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:08.229832+0000) 2019-10-13T14:17:08.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:08.767+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:08.767839+0000) 2019-10-13T14:17:09.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:09.419+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.422495+0000) 2019-10-13T14:17:09.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:09.417+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.422994+0000) 2019-10-13T14:17:09.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:09.419+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.425278+0000) 2019-10-13T14:17:09.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:09.419+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.425449+0000) 2019-10-13T14:17:09.428 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:09.423+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.425968+0000) 2019-10-13T14:17:09.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:09.423+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.426354+0000) 2019-10-13T14:17:09.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:09.423+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.427387+0000) 2019-10-13T14:17:09.432 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:09.425+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.429922+0000) 2019-10-13T14:17:09.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:09.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201209 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:09.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:09.645+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:09.650344+0000) 2019-10-13T14:17:11.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:11.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:11.751567+0000) 2019-10-13T14:17:12.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:12.017+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:12.020317+0000) 2019-10-13T14:17:14.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:14.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.348931+0000) 2019-10-13T14:17:14.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:14.431+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.435816+0000) 2019-10-13T14:17:14.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:14.431+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.436208+0000) 2019-10-13T14:17:14.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:14.433+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.436923+0000) 2019-10-13T14:17:14.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:14.431+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.437324+0000) 2019-10-13T14:17:14.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:14.435+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.438524+0000) 2019-10-13T14:17:14.444 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:14.435+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.439097+0000) 2019-10-13T14:17:14.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:14.435+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.439440+0000) 2019-10-13T14:17:14.448 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:14.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201238 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:14.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:14.441+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.447134+0000) 2019-10-13T14:17:14.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:14.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:14.634299+0000) 2019-10-13T14:17:18.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:18.003+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:18.003826+0000) 2019-10-13T14:17:18.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:18.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:18.230055+0000) 2019-10-13T14:17:18.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:18.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:18.768031+0000) 2019-10-13T14:17:19.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:19.445+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.450918+0000) 2019-10-13T14:17:19.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:19.443+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.448758+0000) 2019-10-13T14:17:19.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:19.443+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.449031+0000) 2019-10-13T14:17:19.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:19.447+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.451167+0000) 2019-10-13T14:17:19.457 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:19.447+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.453340+0000) 2019-10-13T14:17:19.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:19.451+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.454924+0000) 2019-10-13T14:17:19.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:19.451+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.456990+0000) 2019-10-13T14:17:19.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:19.453+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.457368+0000) 2019-10-13T14:17:19.461 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:19.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201268 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:19.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:19.653+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:19.656910+0000) 2019-10-13T14:17:21.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:21.751+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:21.751801+0000) 2019-10-13T14:17:22.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:22.017+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:22.020543+0000) 2019-10-13T14:17:24.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:24.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.349092+0000) 2019-10-13T14:17:24.464 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:24.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.461857+0000) 2019-10-13T14:17:24.466 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:24.459+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.463746+0000) 2019-10-13T14:17:24.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:24.461+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.464955+0000) 2019-10-13T14:17:24.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:24.459+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.465713+0000) 2019-10-13T14:17:24.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:24.463+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.466231+0000) 2019-10-13T14:17:24.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:24.463+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.466790+0000) 2019-10-13T14:17:24.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:24.463+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.467432+0000) 2019-10-13T14:17:24.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:24.465+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.470721+0000) 2019-10-13T14:17:24.475 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:24.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:24.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:24.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:24.634485+0000) 2019-10-13T14:17:28.006 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:27.999+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:28.004029+0000) 2019-10-13T14:17:28.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:28.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:28.230272+0000) 2019-10-13T14:17:28.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:28.767+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:28.768254+0000) 2019-10-13T14:17:29.478 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:29.471+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.475855+0000) 2019-10-13T14:17:29.480 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:29.471+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.476876+0000) 2019-10-13T14:17:29.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:29.471+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.477199+0000) 2019-10-13T14:17:29.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:29.475+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.479321+0000) 2019-10-13T14:17:29.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:29.477+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.481747+0000) 2019-10-13T14:17:29.488 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:29.483+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.485929+0000) 2019-10-13T14:17:29.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:29.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.486942+0000) 2019-10-13T14:17:29.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:29.483+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:29.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:29.489+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.493704+0000) 2019-10-13T14:17:29.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:29.661+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:29.663551+0000) 2019-10-13T14:17:31.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:31.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:31.751963+0000) 2019-10-13T14:17:32.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:32.016+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:32.020767+0000) 2019-10-13T14:17:34.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:34.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.349298+0000) 2019-10-13T14:17:34.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:34.487+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.490765+0000) 2019-10-13T14:17:34.494 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:34.484+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.491507+0000) 2019-10-13T14:17:34.494 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:34.487+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.492021+0000) 2019-10-13T14:17:34.495 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:34.487+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.492673+0000) 2019-10-13T14:17:34.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:34.487+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.492876+0000) 2019-10-13T14:17:34.496 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:34.487+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.493368+0000) 2019-10-13T14:17:34.496 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:34.491+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.494172+0000) 2019-10-13T14:17:34.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:34.488+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.494771+0000) 2019-10-13T14:17:34.502 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:34.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:34.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:34.631+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:34.634693+0000) 2019-10-13T14:17:34.691 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:17:34.691 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:34.694 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:17:34.695 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:17:38.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:38.003+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:38.004239+0000) 2019-10-13T14:17:38.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:38.227+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:38.230503+0000) 2019-10-13T14:17:38.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:38.763+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:38.768445+0000) 2019-10-13T14:17:39.506 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:39.499+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.503943+0000) 2019-10-13T14:17:39.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:39.499+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.504372+0000) 2019-10-13T14:17:39.508 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:39.500+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.505327+0000) 2019-10-13T14:17:39.508 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:39.499+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.505094+0000) 2019-10-13T14:17:39.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:39.499+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.505361+0000) 2019-10-13T14:17:39.509 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:39.503+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.505828+0000) 2019-10-13T14:17:39.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:39.503+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.506053+0000) 2019-10-13T14:17:39.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:39.508+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.512283+0000) 2019-10-13T14:17:39.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:39.507+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:39.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:39.668+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:39.669900+0000) 2019-10-13T14:17:41.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:41.747+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:41.752173+0000) 2019-10-13T14:17:42.023 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:42.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:42.021013+0000) 2019-10-13T14:17:44.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:44.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.349536+0000) 2019-10-13T14:17:44.519 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:44.510+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.516400+0000) 2019-10-13T14:17:44.521 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:44.514+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.518597+0000) 2019-10-13T14:17:44.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:44.512+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.518852+0000) 2019-10-13T14:17:44.522 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:44.514+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.519262+0000) 2019-10-13T14:17:44.522 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:44.514+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.519467+0000) 2019-10-13T14:17:44.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:44.518+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.523830+0000) 2019-10-13T14:17:44.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:44.522+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.526568+0000) 2019-10-13T14:17:44.530 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:44.522+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201417 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:44.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:44.524+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.528539+0000) 2019-10-13T14:17:44.637 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:44.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:44.634875+0000) 2019-10-13T14:17:48.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:48.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:48.004453+0000) 2019-10-13T14:17:48.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:48.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:48.230732+0000) 2019-10-13T14:17:48.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:48.766+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:48.768665+0000) 2019-10-13T14:17:49.534 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:49.526+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.531396+0000) 2019-10-13T14:17:49.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:49.526+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.531884+0000) 2019-10-13T14:17:49.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:49.526+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.532602+0000) 2019-10-13T14:17:49.535 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:49.526+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.533142+0000) 2019-10-13T14:17:49.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:49.528+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.533762+0000) 2019-10-13T14:17:49.537 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:49.530+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.533960+0000) 2019-10-13T14:17:49.537 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:49.530+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.534877+0000) 2019-10-13T14:17:49.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:49.536+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.541313+0000) 2019-10-13T14:17:49.544 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:49.534+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201448 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:49.682 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:49.676+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:49.679342+0000) 2019-10-13T14:17:51.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:51.750+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:51.752378+0000) 2019-10-13T14:17:52.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:52.016+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:52.021183+0000) 2019-10-13T14:17:54.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:54.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.349790+0000) 2019-10-13T14:17:54.547 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:54.538+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.544787+0000) 2019-10-13T14:17:54.548 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:54.538+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.545275+0000) 2019-10-13T14:17:54.549 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:54.542+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.546535+0000) 2019-10-13T14:17:54.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:54.542+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.547306+0000) 2019-10-13T14:17:54.550 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:54.542+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.547759+0000) 2019-10-13T14:17:54.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:54.542+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.547912+0000) 2019-10-13T14:17:54.551 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:54.544+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.548401+0000) 2019-10-13T14:17:54.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:54.548+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.553269+0000) 2019-10-13T14:17:54.557 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:54.550+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:54.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:54.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:54.635126+0000) 2019-10-13T14:17:58.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:58.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:58.004656+0000) 2019-10-13T14:17:58.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:58.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:58.230956+0000) 2019-10-13T14:17:58.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:58.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:58.768869+0000) 2019-10-13T14:17:59.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:17:59.554+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.557939+0000) 2019-10-13T14:17:59.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:17:59.554+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.559745+0000) 2019-10-13T14:17:59.563 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:17:59.554+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.560111+0000) 2019-10-13T14:17:59.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:17:59.554+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.560241+0000) 2019-10-13T14:17:59.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:17:59.556+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.560606+0000) 2019-10-13T14:17:59.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:17:59.554+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.560910+0000) 2019-10-13T14:17:59.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:17:59.554+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.561397+0000) 2019-10-13T14:17:59.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:59.560+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.566856+0000) 2019-10-13T14:17:59.571 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:17:59.562+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201507 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:17:59.688 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:17:59.684+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:17:59.685813+0000) 2019-10-13T14:18:01.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:01.750+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:01.752581+0000) 2019-10-13T14:18:02.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:02.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:02.021380+0000) 2019-10-13T14:18:04.352 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:04.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.349951+0000) 2019-10-13T14:18:04.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:04.566+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.572196+0000) 2019-10-13T14:18:04.575 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:04.566+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.573036+0000) 2019-10-13T14:18:04.576 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:04.566+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.573524+0000) 2019-10-13T14:18:04.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:04.568+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.573922+0000) 2019-10-13T14:18:04.577 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:04.570+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.574049+0000) 2019-10-13T14:18:04.577 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:04.570+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.574272+0000) 2019-10-13T14:18:04.577 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:04.570+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.575261+0000) 2019-10-13T14:18:04.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:04.576+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.581499+0000) 2019-10-13T14:18:04.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:04.578+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:04.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:04.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:04.635333+0000) 2019-10-13T14:18:04.758 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:18:04.758 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:04.763 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:18:04.763 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:08.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:08.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:08.004855+0000) 2019-10-13T14:18:08.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:08.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:08.231148+0000) 2019-10-13T14:18:08.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:08.766+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:08.769069+0000) 2019-10-13T14:18:09.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:09.580+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.585561+0000) 2019-10-13T14:18:09.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:09.582+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.585966+0000) 2019-10-13T14:18:09.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:09.582+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.586680+0000) 2019-10-13T14:18:09.589 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:09.582+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.587292+0000) 2019-10-13T14:18:09.590 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:09.582+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.587978+0000) 2019-10-13T14:18:09.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:09.582+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.588529+0000) 2019-10-13T14:18:09.591 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:09.582+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.589135+0000) 2019-10-13T14:18:09.596 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:09.588+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.594080+0000) 2019-10-13T14:18:09.598 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:09.590+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201568 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:09.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:09.688+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:09.692282+0000) 2019-10-13T14:18:11.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:11.750+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:11.752810+0000) 2019-10-13T14:18:12.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:12.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:12.021574+0000) 2019-10-13T14:18:14.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:14.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.350181+0000) 2019-10-13T14:18:14.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:14.594+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.599203+0000) 2019-10-13T14:18:14.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:14.596+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.600365+0000) 2019-10-13T14:18:14.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:14.598+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.602138+0000) 2019-10-13T14:18:14.605 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:14.598+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.602317+0000) 2019-10-13T14:18:14.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:14.598+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.602696+0000) 2019-10-13T14:18:14.608 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:14.602+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.606291+0000) 2019-10-13T14:18:14.609 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:14.602+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.606604+0000) 2019-10-13T14:18:14.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:14.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:14.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:14.604+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.610852+0000) 2019-10-13T14:18:14.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:14.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:14.635542+0000) 2019-10-13T14:18:18.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:18.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:18.005076+0000) 2019-10-13T14:18:18.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:18.230+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:18.231351+0000) 2019-10-13T14:18:18.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:18.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:18.769302+0000) 2019-10-13T14:18:19.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:19.608+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.612976+0000) 2019-10-13T14:18:19.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:19.610+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.614171+0000) 2019-10-13T14:18:19.617 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:19.610+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.614491+0000) 2019-10-13T14:18:19.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:19.610+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.615377+0000) 2019-10-13T14:18:19.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:19.610+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.615646+0000) 2019-10-13T14:18:19.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:19.610+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.615902+0000) 2019-10-13T14:18:19.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:19.610+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.616248+0000) 2019-10-13T14:18:19.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:19.616+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.621404+0000) 2019-10-13T14:18:19.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:19.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:19.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:19.696+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:19.698795+0000) 2019-10-13T14:18:21.755 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:21.750+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:21.753011+0000) 2019-10-13T14:18:22.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:22.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:22.021766+0000) 2019-10-13T14:18:24.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:24.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.350395+0000) 2019-10-13T14:18:24.630 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:24.622+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.627204+0000) 2019-10-13T14:18:24.631 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:24.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.629294+0000) 2019-10-13T14:18:24.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:24.626+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.630599+0000) 2019-10-13T14:18:24.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:24.626+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.631022+0000) 2019-10-13T14:18:24.634 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:24.626+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.631637+0000) 2019-10-13T14:18:24.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:24.626+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.632129+0000) 2019-10-13T14:18:24.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:24.626+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.633087+0000) 2019-10-13T14:18:24.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:24.630+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.635694+0000) 2019-10-13T14:18:24.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:24.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:24.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:24.632+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:24.639268+0000) 2019-10-13T14:18:28.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:28.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:28.005295+0000) 2019-10-13T14:18:28.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:28.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:28.231544+0000) 2019-10-13T14:18:28.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:28.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:28.769515+0000) 2019-10-13T14:18:29.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:29.634+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.640043+0000) 2019-10-13T14:18:29.643 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:29.634+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.640783+0000) 2019-10-13T14:18:29.644 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:29.634+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.641223+0000) 2019-10-13T14:18:29.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:29.640+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.641928+0000) 2019-10-13T14:18:29.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:29.638+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.642609+0000) 2019-10-13T14:18:29.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:29.638+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.643087+0000) 2019-10-13T14:18:29.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:29.638+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.643884+0000) 2019-10-13T14:18:29.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:29.644+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.645895+0000) 2019-10-13T14:18:29.652 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:29.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201690 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:29.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:29.704+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:29.705390+0000) 2019-10-13T14:18:31.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:31.750+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:31.753207+0000) 2019-10-13T14:18:32.024 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:32.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:32.021960+0000) 2019-10-13T14:18:34.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:34.346+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.350586+0000) 2019-10-13T14:18:34.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:34.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.635883+0000) 2019-10-13T14:18:34.655 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:34.652+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.652869+0000) 2019-10-13T14:18:34.656 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:34.646+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.653393+0000) 2019-10-13T14:18:34.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:34.650+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.654179+0000) 2019-10-13T14:18:34.657 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:34.650+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.654603+0000) 2019-10-13T14:18:34.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:34.650+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.655081+0000) 2019-10-13T14:18:34.658 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:34.650+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.656189+0000) 2019-10-13T14:18:34.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:34.650+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.656709+0000) 2019-10-13T14:18:34.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:34.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201718 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:34.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:34.660+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:34.663519+0000) 2019-10-13T14:18:34.815 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:18:34.815 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:34.818 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:18:34.819 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:18:38.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:38.002+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:38.005448+0000) 2019-10-13T14:18:38.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:38.226+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:38.231741+0000) 2019-10-13T14:18:38.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:38.762+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:38.769711+0000) 2019-10-13T14:18:39.669 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:39.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.667023+0000) 2019-10-13T14:18:39.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:39.662+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.667441+0000) 2019-10-13T14:18:39.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:39.662+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.668843+0000) 2019-10-13T14:18:39.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:39.666+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.669948+0000) 2019-10-13T14:18:39.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:39.666+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.670219+0000) 2019-10-13T14:18:39.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:39.666+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.670668+0000) 2019-10-13T14:18:39.674 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:39.666+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.671581+0000) 2019-10-13T14:18:39.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:39.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:39.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:39.675+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.676132+0000) 2019-10-13T14:18:39.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:39.715+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:39.716189+0000) 2019-10-13T14:18:41.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:41.749+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:41.753441+0000) 2019-10-13T14:18:42.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:42.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:42.022156+0000) 2019-10-13T14:18:44.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:44.345+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.350779+0000) 2019-10-13T14:18:44.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:44.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.636143+0000) 2019-10-13T14:18:44.683 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:44.673+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.680374+0000) 2019-10-13T14:18:44.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:44.673+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.680858+0000) 2019-10-13T14:18:44.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:44.673+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.681315+0000) 2019-10-13T14:18:44.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:44.673+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.681489+0000) 2019-10-13T14:18:44.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:44.679+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.681958+0000) 2019-10-13T14:18:44.685 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:44.677+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.683055+0000) 2019-10-13T14:18:44.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:44.677+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.683924+0000) 2019-10-13T14:18:44.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:44.687+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:44.688293+0000) 2019-10-13T14:18:44.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:44.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:48.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:48.001+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:48.005645+0000) 2019-10-13T14:18:48.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:48.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:48.231965+0000) 2019-10-13T14:18:48.772 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:48.765+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:48.769902+0000) 2019-10-13T14:18:49.697 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:49.689+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.693827+0000) 2019-10-13T14:18:49.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:49.685+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.693494+0000) 2019-10-13T14:18:49.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:49.689+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.694364+0000) 2019-10-13T14:18:49.697 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:49.691+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.694878+0000) 2019-10-13T14:18:49.698 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:49.689+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.695426+0000) 2019-10-13T14:18:49.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:49.689+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.695848+0000) 2019-10-13T14:18:49.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:49.689+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.697400+0000) 2019-10-13T14:18:49.705 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:49.697+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201811 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:49.708 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:49.703+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.706108+0000) 2019-10-13T14:18:49.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:49.723+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:49.725764+0000) 2019-10-13T14:18:51.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:51.749+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:51.753639+0000) 2019-10-13T14:18:52.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:52.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:52.022353+0000) 2019-10-13T14:18:54.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:54.349+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.351011+0000) 2019-10-13T14:18:54.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:54.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.636340+0000) 2019-10-13T14:18:54.709 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:54.701+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.705823+0000) 2019-10-13T14:18:54.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:54.701+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.706240+0000) 2019-10-13T14:18:54.710 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:54.701+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.707938+0000) 2019-10-13T14:18:54.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:54.707+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.708457+0000) 2019-10-13T14:18:54.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:54.701+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.708920+0000) 2019-10-13T14:18:54.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:54.701+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.709421+0000) 2019-10-13T14:18:54.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:54.709+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.709861+0000) 2019-10-13T14:18:54.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:54.711+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:54.713982+0000) 2019-10-13T14:18:54.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:54.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:58.008 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:58.005+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:58.005877+0000) 2019-10-13T14:18:58.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:58.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:58.232164+0000) 2019-10-13T14:18:58.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:58.769+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:58.770105+0000) 2019-10-13T14:18:59.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:18:59.717+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.719179+0000) 2019-10-13T14:18:59.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:18:59.717+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.719531+0000) 2019-10-13T14:18:59.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:18:59.717+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.720086+0000) 2019-10-13T14:18:59.723 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:18:59.717+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.720437+0000) 2019-10-13T14:18:59.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:18:59.717+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.720551+0000) 2019-10-13T14:18:59.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:18:59.717+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.721350+0000) 2019-10-13T14:18:59.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:18:59.719+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.721931+0000) 2019-10-13T14:18:59.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:59.727+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.729301+0000) 2019-10-13T14:18:59.732 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:18:59.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201871 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:18:59.735 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:18:59.731+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:18:59.733071+0000) 2019-10-13T14:19:01.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:01.753+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:01.753814+0000) 2019-10-13T14:19:02.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:02.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:02.022546+0000) 2019-10-13T14:19:04.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:04.349+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.351203+0000) 2019-10-13T14:19:04.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:04.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.636549+0000) 2019-10-13T14:19:04.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:04.729+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.732955+0000) 2019-10-13T14:19:04.736 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:04.731+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.733778+0000) 2019-10-13T14:19:04.736 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:04.729+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.733569+0000) 2019-10-13T14:19:04.737 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:04.733+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.734597+0000) 2019-10-13T14:19:04.737 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:04.733+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.735291+0000) 2019-10-13T14:19:04.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:04.733+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.736053+0000) 2019-10-13T14:19:04.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:04.733+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.736362+0000) 2019-10-13T14:19:04.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:04.739+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:04.741275+0000) 2019-10-13T14:19:04.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:04.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:04.870 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:19:04.871 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:04.874 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:19:04.875 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:08.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:08.005+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:08.006037+0000) 2019-10-13T14:19:08.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:08.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:08.232370+0000) 2019-10-13T14:19:08.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:08.769+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:08.770346+0000) 2019-10-13T14:19:09.743 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:09.739+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.740307+0000) 2019-10-13T14:19:09.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:09.745+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.747688+0000) 2019-10-13T14:19:09.752 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:09.745+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.749359+0000) 2019-10-13T14:19:09.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:09.747+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.749885+0000) 2019-10-13T14:19:09.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:09.749+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.750282+0000) 2019-10-13T14:19:09.754 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:09.749+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.752278+0000) 2019-10-13T14:19:09.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:09.749+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.752627+0000) 2019-10-13T14:19:09.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:09.753+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.753850+0000) 2019-10-13T14:19:09.759 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:09.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:09.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:09.755+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:09.757664+0000) 2019-10-13T14:19:11.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:11.753+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:11.754027+0000) 2019-10-13T14:19:12.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:12.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:12.022741+0000) 2019-10-13T14:19:14.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:14.349+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.351438+0000) 2019-10-13T14:19:14.639 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:14.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.636781+0000) 2019-10-13T14:19:14.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:14.757+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.760892+0000) 2019-10-13T14:19:14.764 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:14.757+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.761584+0000) 2019-10-13T14:19:14.764 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:14.761+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.762265+0000) 2019-10-13T14:19:14.765 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:14.759+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.762751+0000) 2019-10-13T14:19:14.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:14.761+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.763588+0000) 2019-10-13T14:19:14.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:14.761+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.763988+0000) 2019-10-13T14:19:14.767 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:14.761+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.764445+0000) 2019-10-13T14:19:14.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:14.767+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:14.768308+0000) 2019-10-13T14:19:14.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:14.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:18.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:18.005+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:18.006239+0000) 2019-10-13T14:19:18.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:18.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:18.232570+0000) 2019-10-13T14:19:18.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:18.769+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:18.770575+0000) 2019-10-13T14:19:19.750 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:19.747+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.747436+0000) 2019-10-13T14:19:19.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:19.773+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.774407+0000) 2019-10-13T14:19:19.777 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:19.773+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.775362+0000) 2019-10-13T14:19:19.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:19.771+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.775755+0000) 2019-10-13T14:19:19.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:19.773+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.776128+0000) 2019-10-13T14:19:19.779 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:19.773+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.776281+0000) 2019-10-13T14:19:19.780 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:19.773+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.777692+0000) 2019-10-13T14:19:19.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:19.777+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.777795+0000) 2019-10-13T14:19:19.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:19.779+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:19.783571+0000) 2019-10-13T14:19:19.788 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:19.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:21.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:21.753+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:21.754252+0000) 2019-10-13T14:19:22.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:22.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:22.022948+0000) 2019-10-13T14:19:24.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:24.349+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.351636+0000) 2019-10-13T14:19:24.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:24.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.636975+0000) 2019-10-13T14:19:24.791 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:24.787+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.788940+0000) 2019-10-13T14:19:24.793 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:24.789+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.791083+0000) 2019-10-13T14:19:24.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:24.789+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.791561+0000) 2019-10-13T14:19:24.794 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:24.789+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.791767+0000) 2019-10-13T14:19:24.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:24.793+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.795123+0000) 2019-10-13T14:19:24.800 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:24.793+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.797478+0000) 2019-10-13T14:19:24.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:24.795+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.798964+0000) 2019-10-13T14:19:24.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:24.797+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:24.799115+0000) 2019-10-13T14:19:24.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:24.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:28.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:28.005+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:28.006467+0000) 2019-10-13T14:19:28.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:28.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:28.232801+0000) 2019-10-13T14:19:28.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:28.769+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:28.770771+0000) 2019-10-13T14:19:29.756 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:29.751+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.754016+0000) 2019-10-13T14:19:29.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:29.801+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.803903+0000) 2019-10-13T14:19:29.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:29.805+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.808545+0000) 2019-10-13T14:19:29.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:29.807+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.808530+0000) 2019-10-13T14:19:29.813 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:29.807+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.810518+0000) 2019-10-13T14:19:29.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:29.809+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.811026+0000) 2019-10-13T14:19:29.814 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:29.809+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.811482+0000) 2019-10-13T14:19:29.815 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:29.809+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.812631+0000) 2019-10-13T14:19:29.815 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:29.809+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:29.813327+0000) 2019-10-13T14:19:29.816 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:29.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202054 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:31.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:31.753+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:31.754423+0000) 2019-10-13T14:19:32.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:32.019+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:32.023142+0000) 2019-10-13T14:19:34.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:34.349+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.351792+0000) 2019-10-13T14:19:34.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:34.633+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.637168+0000) 2019-10-13T14:19:34.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:34.813+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.817071+0000) 2019-10-13T14:19:34.821 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:34.815+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.818647+0000) 2019-10-13T14:19:34.821 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:34.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.819145+0000) 2019-10-13T14:19:34.822 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:34.817+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.819937+0000) 2019-10-13T14:19:34.822 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:34.817+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.820227+0000) 2019-10-13T14:19:34.823 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:34.817+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.820820+0000) 2019-10-13T14:19:34.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:34.817+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.821681+0000) 2019-10-13T14:19:34.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:34.823+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:34.826064+0000) 2019-10-13T14:19:34.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:34.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:34.927 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:19:34.927 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:34.931 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:19:34.931 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:19:38.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:38.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:38.006633+0000) 2019-10-13T14:19:38.235 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:38.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:38.232970+0000) 2019-10-13T14:19:38.773 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:38.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:38.770926+0000) 2019-10-13T14:19:39.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:39.763+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.764887+0000) 2019-10-13T14:19:39.832 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:39.824+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.829691+0000) 2019-10-13T14:19:39.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:39.828+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.831052+0000) 2019-10-13T14:19:39.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:39.827+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.831472+0000) 2019-10-13T14:19:39.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:39.828+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.831873+0000) 2019-10-13T14:19:39.838 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:39.828+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.831929+0000) 2019-10-13T14:19:39.838 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:39.828+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.832319+0000) 2019-10-13T14:19:39.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:39.828+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.832541+0000) 2019-10-13T14:19:39.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:39.835+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:39.839469+0000) 2019-10-13T14:19:39.843 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:39.836+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:41.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:41.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:41.754638+0000) 2019-10-13T14:19:42.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:42.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:42.023332+0000) 2019-10-13T14:19:44.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:44.348+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.352024+0000) 2019-10-13T14:19:44.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:44.632+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.637363+0000) 2019-10-13T14:19:44.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:44.838+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.843649+0000) 2019-10-13T14:19:44.846 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:44.840+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.844242+0000) 2019-10-13T14:19:44.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:44.840+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.844802+0000) 2019-10-13T14:19:44.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:44.840+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.844686+0000) 2019-10-13T14:19:44.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:44.840+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.845256+0000) 2019-10-13T14:19:44.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:44.844+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.846265+0000) 2019-10-13T14:19:44.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:44.844+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.846818+0000) 2019-10-13T14:19:44.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:44.850+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:44.853037+0000) 2019-10-13T14:19:44.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:44.848+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:48.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:48.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:48.006857+0000) 2019-10-13T14:19:48.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:48.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:48.233166+0000) 2019-10-13T14:19:48.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:48.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:48.771129+0000) 2019-10-13T14:19:49.779 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:49.774+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.776397+0000) 2019-10-13T14:19:49.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:49.852+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.856797+0000) 2019-10-13T14:19:49.860 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:49.854+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.857212+0000) 2019-10-13T14:19:49.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:49.852+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.857555+0000) 2019-10-13T14:19:49.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:49.856+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.857844+0000) 2019-10-13T14:19:49.861 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:49.856+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.858260+0000) 2019-10-13T14:19:49.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:49.856+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.859152+0000) 2019-10-13T14:19:49.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:49.856+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.859497+0000) 2019-10-13T14:19:49.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:49.858+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:49.862348+0000) 2019-10-13T14:19:49.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:49.864+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202176 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:51.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:51.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:51.754884+0000) 2019-10-13T14:19:52.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:52.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:52.023542+0000) 2019-10-13T14:19:54.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:54.348+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.352236+0000) 2019-10-13T14:19:54.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:54.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.637558+0000) 2019-10-13T14:19:54.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:54.868+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.871018+0000) 2019-10-13T14:19:54.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:54.868+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.871441+0000) 2019-10-13T14:19:54.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:54.870+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.873508+0000) 2019-10-13T14:19:54.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:54.872+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.873779+0000) 2019-10-13T14:19:54.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:54.872+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.876647+0000) 2019-10-13T14:19:54.880 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:54.876+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.878236+0000) 2019-10-13T14:19:54.881 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:54.876+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.879239+0000) 2019-10-13T14:19:54.883 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:54.876+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:54.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:54.878+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:54.883072+0000) 2019-10-13T14:19:58.009 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:58.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:58.007087+0000) 2019-10-13T14:19:58.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:58.228+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:58.233378+0000) 2019-10-13T14:19:58.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:58.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:58.771386+0000) 2019-10-13T14:19:59.785 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:59.782+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.782873+0000) 2019-10-13T14:19:59.886 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:19:59.880+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.883978+0000) 2019-10-13T14:19:59.887 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:19:59.880+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.884475+0000) 2019-10-13T14:19:59.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:19:59.880+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.885124+0000) 2019-10-13T14:19:59.888 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:19:59.882+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.885975+0000) 2019-10-13T14:19:59.889 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:19:59.884+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.886793+0000) 2019-10-13T14:19:59.890 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:19:59.884+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.887428+0000) 2019-10-13T14:19:59.890 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:19:59.884+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.887866+0000) 2019-10-13T14:19:59.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:19:59.892+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202236 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:19:59.899 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:19:59.894+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:19:59.896548+0000) 2019-10-13T14:20:01.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:01.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:01.755091+0000) 2019-10-13T14:20:02.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:02.018+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:02.023747+0000) 2019-10-13T14:20:04.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:04.348+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.352447+0000) 2019-10-13T14:20:04.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:04.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.637778+0000) 2019-10-13T14:20:04.900 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:04.896+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.898027+0000) 2019-10-13T14:20:04.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:04.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.898678+0000) 2019-10-13T14:20:04.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:04.896+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.899194+0000) 2019-10-13T14:20:04.902 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:04.894+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.899703+0000) 2019-10-13T14:20:04.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:04.896+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.900240+0000) 2019-10-13T14:20:04.904 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:04.896+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.901681+0000) 2019-10-13T14:20:04.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:04.900+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.901958+0000) 2019-10-13T14:20:04.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:04.902+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:04.905657+0000) 2019-10-13T14:20:04.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:04.904+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:05.060 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:20:05.060 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:05.064 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:20:05.065 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:08.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:08.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:08.007299+0000) 2019-10-13T14:20:08.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:08.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:08.233582+0000) 2019-10-13T14:20:08.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:08.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:08.771574+0000) 2019-10-13T14:20:09.794 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:09.790+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.792218+0000) 2019-10-13T14:20:09.914 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:09.908+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.911268+0000) 2019-10-13T14:20:09.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:09.908+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.912951+0000) 2019-10-13T14:20:09.916 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:09.908+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.913541+0000) 2019-10-13T14:20:09.916 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:09.912+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.913975+0000) 2019-10-13T14:20:09.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:09.914+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.916597+0000) 2019-10-13T14:20:09.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:09.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.916892+0000) 2019-10-13T14:20:09.919 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:09.912+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.917256+0000) 2019-10-13T14:20:09.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:09.920+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202298 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:09.927 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:09.922+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:09.924732+0000) 2019-10-13T14:20:11.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:11.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:11.755312+0000) 2019-10-13T14:20:12.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:12.022+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:12.023943+0000) 2019-10-13T14:20:14.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:14.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.352646+0000) 2019-10-13T14:20:14.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:14.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.637979+0000) 2019-10-13T14:20:14.929 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:14.924+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.926660+0000) 2019-10-13T14:20:14.930 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:14.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.927787+0000) 2019-10-13T14:20:14.930 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:14.924+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.928079+0000) 2019-10-13T14:20:14.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:14.924+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.928687+0000) 2019-10-13T14:20:14.932 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:14.924+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.929217+0000) 2019-10-13T14:20:14.932 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:14.924+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.928702+0000) 2019-10-13T14:20:14.932 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:14.928+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.929718+0000) 2019-10-13T14:20:14.936 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:14.930+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:14.933759+0000) 2019-10-13T14:20:14.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:14.932+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:18.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:18.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:18.007511+0000) 2019-10-13T14:20:18.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:18.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:18.233817+0000) 2019-10-13T14:20:18.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:18.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:18.771814+0000) 2019-10-13T14:20:19.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:19.798+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.798720+0000) 2019-10-13T14:20:19.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:19.936+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.940105+0000) 2019-10-13T14:20:19.944 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:19.940+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.941984+0000) 2019-10-13T14:20:19.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:19.938+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.942653+0000) 2019-10-13T14:20:19.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:19.940+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.943445+0000) 2019-10-13T14:20:19.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:19.940+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.943429+0000) 2019-10-13T14:20:19.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:19.940+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.945458+0000) 2019-10-13T14:20:19.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:19.944+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.945907+0000) 2019-10-13T14:20:19.953 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:19.948+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:19.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:19.950+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:19.952499+0000) 2019-10-13T14:20:21.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:21.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:21.755512+0000) 2019-10-13T14:20:22.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:22.022+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:22.024141+0000) 2019-10-13T14:20:24.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:24.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.352852+0000) 2019-10-13T14:20:24.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:24.636+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.638206+0000) 2019-10-13T14:20:24.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:24.952+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.954178+0000) 2019-10-13T14:20:24.957 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:24.952+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.954541+0000) 2019-10-13T14:20:24.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:24.954+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.956573+0000) 2019-10-13T14:20:24.959 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:24.952+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.956889+0000) 2019-10-13T14:20:24.961 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:24.956+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.958576+0000) 2019-10-13T14:20:24.961 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:24.956+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.958686+0000) 2019-10-13T14:20:24.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:24.956+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.959050+0000) 2019-10-13T14:20:24.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:24.960+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202387 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:24.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:24.962+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:24.966552+0000) 2019-10-13T14:20:28.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:28.004+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:28.007724+0000) 2019-10-13T14:20:28.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:28.232+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:28.234021+0000) 2019-10-13T14:20:28.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:28.768+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:28.772022+0000) 2019-10-13T14:20:29.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:29.802+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.805447+0000) 2019-10-13T14:20:29.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:29.964+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.965993+0000) 2019-10-13T14:20:29.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:29.964+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.967343+0000) 2019-10-13T14:20:29.970 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:29.964+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.967947+0000) 2019-10-13T14:20:29.970 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:29.964+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.968408+0000) 2019-10-13T14:20:29.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:29.966+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.970519+0000) 2019-10-13T14:20:29.973 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:29.968+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.970823+0000) 2019-10-13T14:20:29.975 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:29.968+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.972486+0000) 2019-10-13T14:20:29.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:29.974+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:29.976853+0000) 2019-10-13T14:20:29.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:29.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202420 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:31.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:31.752+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:31.755707+0000) 2019-10-13T14:20:32.027 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:32.022+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:32.024368+0000) 2019-10-13T14:20:34.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:34.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.353079+0000) 2019-10-13T14:20:34.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:34.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.638433+0000) 2019-10-13T14:20:34.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:34.975+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.979498+0000) 2019-10-13T14:20:34.982 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:34.975+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.980051+0000) 2019-10-13T14:20:34.984 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:34.979+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.981834+0000) 2019-10-13T14:20:34.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:34.979+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.982258+0000) 2019-10-13T14:20:34.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:34.979+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.982695+0000) 2019-10-13T14:20:34.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:34.978+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.983413+0000) 2019-10-13T14:20:34.986 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:34.979+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.983629+0000) 2019-10-13T14:20:34.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:34.986+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:34.989641+0000) 2019-10-13T14:20:34.993 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:34.987+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:35.174 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:20:35.174 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:35.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:20:35.177 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:20:38.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:38.003+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:38.007919+0000) 2019-10-13T14:20:38.236 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:38.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:38.234217+0000) 2019-10-13T14:20:38.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:38.767+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:38.772235+0000) 2019-10-13T14:20:39.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:39.810+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.812074+0000) 2019-10-13T14:20:39.996 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:39.991+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.993936+0000) 2019-10-13T14:20:39.997 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:39.991+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.994844+0000) 2019-10-13T14:20:39.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:39.991+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.994328+0000) 2019-10-13T14:20:39.998 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:39.991+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.995616+0000) 2019-10-13T14:20:39.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:39.994+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.996576+0000) 2019-10-13T14:20:39.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:39.991+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.996911+0000) 2019-10-13T14:20:40.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:39.991+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:39.997457+0000) 2019-10-13T14:20:40.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:39.998+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:40.001553+0000) 2019-10-13T14:20:40.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:39.999+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:41.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:41.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:41.755923+0000) 2019-10-13T14:20:42.027 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:42.022+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:42.024579+0000) 2019-10-13T14:20:44.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:44.347+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:44.353270+0000) 2019-10-13T14:20:44.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:44.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:44.638634+0000) 2019-10-13T14:20:45.010 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:45.003+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.007487+0000) 2019-10-13T14:20:45.010 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:45.003+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.008059+0000) 2019-10-13T14:20:45.011 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:45.003+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.009184+0000) 2019-10-13T14:20:45.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:45.007+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.009732+0000) 2019-10-13T14:20:45.012 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:45.007+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.009945+0000) 2019-10-13T14:20:45.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:45.005+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.010497+0000) 2019-10-13T14:20:45.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:45.007+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.010598+0000) 2019-10-13T14:20:45.017 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:45.009+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:45.015044+0000) 2019-10-13T14:20:45.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:45.015+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:48.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:48.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.008144+0000) 2019-10-13T14:20:48.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:48.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.234421+0000) 2019-10-13T14:20:48.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:48.771+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:48.772492+0000) 2019-10-13T14:20:49.821 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:49.813+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:49.818799+0000) 2019-10-13T14:20:50.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:50.021+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.024166+0000) 2019-10-13T14:20:50.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:50.019+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.024396+0000) 2019-10-13T14:20:50.027 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:50.019+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.024896+0000) 2019-10-13T14:20:50.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:50.019+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.025504+0000) 2019-10-13T14:20:50.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:50.023+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.026019+0000) 2019-10-13T14:20:50.029 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:50.023+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.026210+0000) 2019-10-13T14:20:50.029 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:50.023+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.026422+0000) 2019-10-13T14:20:50.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:50.025+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:50.029265+0000) 2019-10-13T14:20:50.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:50.027+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:51.758 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:51.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:51.756120+0000) 2019-10-13T14:20:52.027 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:52.021+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:52.024808+0000) 2019-10-13T14:20:54.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:54.351+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:54.353471+0000) 2019-10-13T14:20:54.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:54.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:54.638862+0000) 2019-10-13T14:20:55.037 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:20:55.031+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.034795+0000) 2019-10-13T14:20:55.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:55.031+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.035159+0000) 2019-10-13T14:20:55.038 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:55.031+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.036242+0000) 2019-10-13T14:20:55.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:20:55.033+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.036742+0000) 2019-10-13T14:20:55.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:20:55.035+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.038267+0000) 2019-10-13T14:20:55.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:20:55.035+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.039237+0000) 2019-10-13T14:20:55.042 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:55.035+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.039732+0000) 2019-10-13T14:20:55.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:55.037+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:55.040630+0000) 2019-10-13T14:20:55.047 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:20:55.039+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202574 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:20:58.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:20:58.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.008348+0000) 2019-10-13T14:20:58.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:20:58.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.234644+0000) 2019-10-13T14:20:58.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:20:58.771+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:58.772717+0000) 2019-10-13T14:20:59.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:20:59.821+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:20:59.825517+0000) 2019-10-13T14:21:00.051 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:00.043+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.049080+0000) 2019-10-13T14:21:00.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:00.047+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.049791+0000) 2019-10-13T14:21:00.053 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:00.047+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.050535+0000) 2019-10-13T14:21:00.054 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:00.045+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.051523+0000) 2019-10-13T14:21:00.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:00.047+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.051799+0000) 2019-10-13T14:21:00.055 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:00.047+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.052206+0000) 2019-10-13T14:21:00.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:00.047+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.052510+0000) 2019-10-13T14:21:00.059 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:00.053+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:00.057216+0000) 2019-10-13T14:21:00.061 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:00.055+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202605 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:01.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:01.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:01.756356+0000) 2019-10-13T14:21:02.027 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:02.021+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:02.025034+0000) 2019-10-13T14:21:04.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:04.351+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:04.353684+0000) 2019-10-13T14:21:04.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:04.635+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:04.639069+0000) 2019-10-13T14:21:05.064 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:05.059+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.061898+0000) 2019-10-13T14:21:05.065 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:05.059+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.062535+0000) 2019-10-13T14:21:05.066 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:05.059+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.063016+0000) 2019-10-13T14:21:05.066 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:05.059+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.063872+0000) 2019-10-13T14:21:05.066 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:05.061+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.064239+0000) 2019-10-13T14:21:05.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:05.059+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.064357+0000) 2019-10-13T14:21:05.067 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:05.059+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.064807+0000) 2019-10-13T14:21:05.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:05.065+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:05.068764+0000) 2019-10-13T14:21:05.074 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:05.067+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202635 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:05.280 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:21:05.280 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:05.284 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:21:05.284 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:08.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:08.003+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.008550+0000) 2019-10-13T14:21:08.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:08.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.234839+0000) 2019-10-13T14:21:08.775 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:08.767+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:08.772934+0000) 2019-10-13T14:21:09.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:09.829+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:09.832017+0000) 2019-10-13T14:21:10.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:10.071+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.074850+0000) 2019-10-13T14:21:10.078 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:10.071+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.075384+0000) 2019-10-13T14:21:10.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:10.071+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.075410+0000) 2019-10-13T14:21:10.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:10.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.076061+0000) 2019-10-13T14:21:10.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:10.071+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.076385+0000) 2019-10-13T14:21:10.080 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:10.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.077378+0000) 2019-10-13T14:21:10.080 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:10.071+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.077483+0000) 2019-10-13T14:21:10.087 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:10.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202668 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:10.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:10.081+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:10.086519+0000) 2019-10-13T14:21:11.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:11.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:11.756556+0000) 2019-10-13T14:21:12.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:12.021+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:12.025244+0000) 2019-10-13T14:21:14.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:14.351+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:14.353933+0000) 2019-10-13T14:21:14.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:14.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:14.639303+0000) 2019-10-13T14:21:15.090 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:15.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.087783+0000) 2019-10-13T14:21:15.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:15.083+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.088075+0000) 2019-10-13T14:21:15.092 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:15.087+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.089863+0000) 2019-10-13T14:21:15.092 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:15.087+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.090250+0000) 2019-10-13T14:21:15.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:15.087+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.090638+0000) 2019-10-13T14:21:15.093 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:15.087+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.090992+0000) 2019-10-13T14:21:15.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:15.087+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.091656+0000) 2019-10-13T14:21:15.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:15.093+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:15.097809+0000) 2019-10-13T14:21:15.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:15.095+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202698 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:18.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:18.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.008812+0000) 2019-10-13T14:21:18.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:18.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.235039+0000) 2019-10-13T14:21:18.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:18.771+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:18.773192+0000) 2019-10-13T14:21:19.840 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:19.833+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:19.838105+0000) 2019-10-13T14:21:20.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:20.099+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.102604+0000) 2019-10-13T14:21:20.105 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:20.099+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.102672+0000) 2019-10-13T14:21:20.106 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:20.097+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.103027+0000) 2019-10-13T14:21:20.106 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:20.099+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.103588+0000) 2019-10-13T14:21:20.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:20.099+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.104546+0000) 2019-10-13T14:21:20.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:20.099+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.104302+0000) 2019-10-13T14:21:20.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:20.099+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.105511+0000) 2019-10-13T14:21:20.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:20.105+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:20.109647+0000) 2019-10-13T14:21:20.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:20.107+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:21.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:21.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:21.756756+0000) 2019-10-13T14:21:22.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:22.021+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:22.025453+0000) 2019-10-13T14:21:24.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:24.351+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:24.354155+0000) 2019-10-13T14:21:24.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:24.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:24.639527+0000) 2019-10-13T14:21:25.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:25.111+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.115925+0000) 2019-10-13T14:21:25.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:25.111+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.116871+0000) 2019-10-13T14:21:25.120 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:25.115+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.117883+0000) 2019-10-13T14:21:25.121 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:25.115+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.118472+0000) 2019-10-13T14:21:25.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:25.115+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.118506+0000) 2019-10-13T14:21:25.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:25.113+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.118964+0000) 2019-10-13T14:21:25.122 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:25.115+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.119193+0000) 2019-10-13T14:21:25.125 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:25.117+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:25.123125+0000) 2019-10-13T14:21:25.127 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:25.119+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202760 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:28.011 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:28.003+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.009012+0000) 2019-10-13T14:21:28.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:28.231+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.235228+0000) 2019-10-13T14:21:28.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:28.770+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:28.773426+0000) 2019-10-13T14:21:29.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:29.841+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:29.844604+0000) 2019-10-13T14:21:30.132 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:30.122+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.129152+0000) 2019-10-13T14:21:30.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:30.122+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.129711+0000) 2019-10-13T14:21:30.133 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:30.125+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.130586+0000) 2019-10-13T14:21:30.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:30.126+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.130817+0000) 2019-10-13T14:21:30.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:30.126+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.131088+0000) 2019-10-13T14:21:30.134 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:30.126+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.131676+0000) 2019-10-13T14:21:30.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:30.126+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.132220+0000) 2019-10-13T14:21:30.140 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:30.133+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:30.137979+0000) 2019-10-13T14:21:30.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:30.134+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202794 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:31.759 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:31.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:31.756957+0000) 2019-10-13T14:21:32.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:32.025+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:32.025647+0000) 2019-10-13T14:21:34.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:34.350+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:34.354353+0000) 2019-10-13T14:21:34.642 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:34.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:34.639721+0000) 2019-10-13T14:21:35.145 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:35.138+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.143023+0000) 2019-10-13T14:21:35.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:35.138+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.144806+0000) 2019-10-13T14:21:35.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:35.138+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.145113+0000) 2019-10-13T14:21:35.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:35.138+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.145475+0000) 2019-10-13T14:21:35.149 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:35.141+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.147236+0000) 2019-10-13T14:21:35.151 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:35.142+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.149263+0000) 2019-10-13T14:21:35.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:35.146+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.149958+0000) 2019-10-13T14:21:35.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:35.146+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:35.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:35.149+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:35.156045+0000) 2019-10-13T14:21:35.404 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:21:35.404 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:35.408 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:21:35.408 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:21:38.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:38.006+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.009216+0000) 2019-10-13T14:21:38.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:38.230+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.235440+0000) 2019-10-13T14:21:38.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:38.766+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:38.773642+0000) 2019-10-13T14:21:39.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:39.849+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:39.851120+0000) 2019-10-13T14:21:40.160 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:40.150+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.157111+0000) 2019-10-13T14:21:40.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:40.157+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.161391+0000) 2019-10-13T14:21:40.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:40.165+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.168939+0000) 2019-10-13T14:21:40.177 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:40.154+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.159379+0000) 2019-10-13T14:21:40.178 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:40.154+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.159676+0000) 2019-10-13T14:21:40.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:40.154+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.160593+0000) 2019-10-13T14:21:40.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:40.154+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.161032+0000) 2019-10-13T14:21:40.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:40.158+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:40.161895+0000) 2019-10-13T14:21:40.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:40.162+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:41.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:41.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:41.757207+0000) 2019-10-13T14:21:42.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:42.025+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:42.025841+0000) 2019-10-13T14:21:44.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:44.350+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:44.354550+0000) 2019-10-13T14:21:44.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:44.634+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:44.639980+0000) 2019-10-13T14:21:45.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:45.166+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.170386+0000) 2019-10-13T14:21:45.173 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:45.166+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.171016+0000) 2019-10-13T14:21:45.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:45.166+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.171691+0000) 2019-10-13T14:21:45.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:45.166+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.171945+0000) 2019-10-13T14:21:45.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:45.169+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.172481+0000) 2019-10-13T14:21:45.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:45.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.172948+0000) 2019-10-13T14:21:45.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:45.166+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.173664+0000) 2019-10-13T14:21:45.182 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:45.174+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202887 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:45.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:45.177+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:45.180664+0000) 2019-10-13T14:21:48.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:48.006+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.009419+0000) 2019-10-13T14:21:48.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:48.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.235660+0000) 2019-10-13T14:21:48.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:48.770+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:48.773867+0000) 2019-10-13T14:21:49.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:49.856+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:49.857757+0000) 2019-10-13T14:21:50.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:50.176+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.183585+0000) 2019-10-13T14:21:50.188 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:50.182+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.185960+0000) 2019-10-13T14:21:50.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:50.182+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.186396+0000) 2019-10-13T14:21:50.189 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:50.182+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.187010+0000) 2019-10-13T14:21:50.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:50.182+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.187615+0000) 2019-10-13T14:21:50.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:50.182+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.187826+0000) 2019-10-13T14:21:50.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:50.182+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.188252+0000) 2019-10-13T14:21:50.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:50.184+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:50.191885+0000) 2019-10-13T14:21:50.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:50.186+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202920 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:51.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:51.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:51.757407+0000) 2019-10-13T14:21:52.028 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:52.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:52.026070+0000) 2019-10-13T14:21:54.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:54.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:54.354726+0000) 2019-10-13T14:21:54.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:54.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:54.640181+0000) 2019-10-13T14:21:55.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:55.190+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.195475+0000) 2019-10-13T14:21:55.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:21:55.192+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.196051+0000) 2019-10-13T14:21:55.199 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:55.190+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.196665+0000) 2019-10-13T14:21:55.200 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:21:55.190+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.197460+0000) 2019-10-13T14:21:55.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:21:55.194+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.198388+0000) 2019-10-13T14:21:55.201 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:55.194+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.198973+0000) 2019-10-13T14:21:55.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:21:55.194+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.199817+0000) 2019-10-13T14:21:55.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:55.200+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:55.205565+0000) 2019-10-13T14:21:55.208 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:21:55.198+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:21:58.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:21:58.006+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.009617+0000) 2019-10-13T14:21:58.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:21:58.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.235857+0000) 2019-10-13T14:21:58.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:21:58.770+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:58.774095+0000) 2019-10-13T14:21:59.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:21:59.860+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:21:59.864520+0000) 2019-10-13T14:22:00.212 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:00.202+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.208982+0000) 2019-10-13T14:22:00.212 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:00.202+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.209419+0000) 2019-10-13T14:22:00.213 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:00.206+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.210178+0000) 2019-10-13T14:22:00.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:00.206+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.210572+0000) 2019-10-13T14:22:00.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:00.206+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.211493+0000) 2019-10-13T14:22:00.214 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:00.208+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.212316+0000) 2019-10-13T14:22:00.215 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:00.206+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.211899+0000) 2019-10-13T14:22:00.222 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:00.214+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202982 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:00.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:00.212+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:00.219825+0000) 2019-10-13T14:22:01.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:01.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:01.757642+0000) 2019-10-13T14:22:02.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:02.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:02.026267+0000) 2019-10-13T14:22:04.357 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:04.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:04.354923+0000) 2019-10-13T14:22:04.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:04.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:04.640411+0000) 2019-10-13T14:22:05.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:05.218+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.222903+0000) 2019-10-13T14:22:05.226 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:05.216+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.223424+0000) 2019-10-13T14:22:05.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:05.218+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.223630+0000) 2019-10-13T14:22:05.227 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:05.218+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.224116+0000) 2019-10-13T14:22:05.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:05.218+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.224463+0000) 2019-10-13T14:22:05.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:05.218+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.225019+0000) 2019-10-13T14:22:05.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:05.218+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.225564+0000) 2019-10-13T14:22:05.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:05.228+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:05.232382+0000) 2019-10-13T14:22:05.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:05.226+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:05.493 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:22:05.493 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:05.497 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:22:05.497 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:08.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:08.006+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.009824+0000) 2019-10-13T14:22:08.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:08.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.236094+0000) 2019-10-13T14:22:08.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:08.770+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:08.774285+0000) 2019-10-13T14:22:09.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:09.868+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:09.871364+0000) 2019-10-13T14:22:10.239 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:10.230+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.235924+0000) 2019-10-13T14:22:10.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:10.230+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.236363+0000) 2019-10-13T14:22:10.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:10.230+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.236970+0000) 2019-10-13T14:22:10.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:10.230+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.237557+0000) 2019-10-13T14:22:10.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:10.234+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.238324+0000) 2019-10-13T14:22:10.241 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:10.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.238945+0000) 2019-10-13T14:22:10.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:10.234+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.239418+0000) 2019-10-13T14:22:10.245 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:10.236+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:10.243130+0000) 2019-10-13T14:22:10.248 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:10.238+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:11.760 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:11.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:11.757837+0000) 2019-10-13T14:22:12.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:12.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:12.026458+0000) 2019-10-13T14:22:14.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:14.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:14.355127+0000) 2019-10-13T14:22:14.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:14.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:14.640612+0000) 2019-10-13T14:22:15.251 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:15.242+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.248493+0000) 2019-10-13T14:22:15.252 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:15.242+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.248956+0000) 2019-10-13T14:22:15.252 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:15.242+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.249517+0000) 2019-10-13T14:22:15.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:15.246+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.252186+0000) 2019-10-13T14:22:15.255 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:15.248+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.252855+0000) 2019-10-13T14:22:15.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:15.246+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.252339+0000) 2019-10-13T14:22:15.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:15.254+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.258461+0000) 2019-10-13T14:22:15.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:15.254+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203075 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:15.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:15.256+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:15.262256+0000) 2019-10-13T14:22:18.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:18.006+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.010073+0000) 2019-10-13T14:22:18.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:18.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.236322+0000) 2019-10-13T14:22:18.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:18.770+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:18.774480+0000) 2019-10-13T14:22:19.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:19.876+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:19.878041+0000) 2019-10-13T14:22:20.265 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:20.258+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.262949+0000) 2019-10-13T14:22:20.266 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:20.256+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.263288+0000) 2019-10-13T14:22:20.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:20.258+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.264936+0000) 2019-10-13T14:22:20.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:20.262+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.266085+0000) 2019-10-13T14:22:20.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:20.262+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.266417+0000) 2019-10-13T14:22:20.269 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:20.262+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.266577+0000) 2019-10-13T14:22:20.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:20.262+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.267538+0000) 2019-10-13T14:22:20.289 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:20.266+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203108 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:20.290 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:20.268+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:20.272312+0000) 2019-10-13T14:22:21.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:21.754+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:21.758044+0000) 2019-10-13T14:22:22.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:22.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:22.026653+0000) 2019-10-13T14:22:24.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:24.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:24.355337+0000) 2019-10-13T14:22:24.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:24.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:24.640839+0000) 2019-10-13T14:22:25.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:25.269+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.276378+0000) 2019-10-13T14:22:25.279 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:25.269+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.276743+0000) 2019-10-13T14:22:25.280 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:25.272+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.277510+0000) 2019-10-13T14:22:25.280 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:25.273+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.277824+0000) 2019-10-13T14:22:25.281 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:25.273+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.278400+0000) 2019-10-13T14:22:25.281 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:25.273+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.278566+0000) 2019-10-13T14:22:25.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:25.273+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.279560+0000) 2019-10-13T14:22:25.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:25.276+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:25.282398+0000) 2019-10-13T14:22:25.288 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:25.281+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:28.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:28.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.010291+0000) 2019-10-13T14:22:28.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:28.229+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.236554+0000) 2019-10-13T14:22:28.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:28.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:28.774666+0000) 2019-10-13T14:22:29.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:29.884+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:29.884609+0000) 2019-10-13T14:22:30.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:30.285+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.290065+0000) 2019-10-13T14:22:30.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:30.284+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.290559+0000) 2019-10-13T14:22:30.294 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:30.285+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.290901+0000) 2019-10-13T14:22:30.296 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:30.285+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.293249+0000) 2019-10-13T14:22:30.296 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:30.285+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.293575+0000) 2019-10-13T14:22:30.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:30.289+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.295363+0000) 2019-10-13T14:22:30.298 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:30.289+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.295530+0000) 2019-10-13T14:22:30.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:30.293+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:30.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:30.296+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:30.300486+0000) 2019-10-13T14:22:31.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:31.753+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:31.758260+0000) 2019-10-13T14:22:32.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:32.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:32.026843+0000) 2019-10-13T14:22:34.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:34.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:34.355564+0000) 2019-10-13T14:22:34.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:34.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:34.640993+0000) 2019-10-13T14:22:35.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:35.297+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.303677+0000) 2019-10-13T14:22:35.307 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:35.297+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.304194+0000) 2019-10-13T14:22:35.307 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:35.300+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.304879+0000) 2019-10-13T14:22:35.307 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:35.297+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.305313+0000) 2019-10-13T14:22:35.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:35.297+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.305458+0000) 2019-10-13T14:22:35.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:35.297+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.305698+0000) 2019-10-13T14:22:35.310 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:35.301+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.307525+0000) 2019-10-13T14:22:35.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:35.308+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:35.313537+0000) 2019-10-13T14:22:35.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:35.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203201 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:35.650 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:22:35.650 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:35.654 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:22:35.654 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:22:38.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:38.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.010520+0000) 2019-10-13T14:22:38.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:38.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.236749+0000) 2019-10-13T14:22:38.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:38.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:38.774902+0000) 2019-10-13T14:22:39.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:39.888+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:39.891152+0000) 2019-10-13T14:22:40.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:40.313+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.317172+0000) 2019-10-13T14:22:40.320 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:40.313+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.317664+0000) 2019-10-13T14:22:40.321 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:40.317+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.319340+0000) 2019-10-13T14:22:40.322 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:40.312+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.319887+0000) 2019-10-13T14:22:40.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:40.321+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.325090+0000) 2019-10-13T14:22:40.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:40.325+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.326520+0000) 2019-10-13T14:22:40.329 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:40.320+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.327108+0000) 2019-10-13T14:22:40.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:40.325+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:40.327306+0000) 2019-10-13T14:22:40.330 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:40.325+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203233 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:41.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:41.757+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:41.758463+0000) 2019-10-13T14:22:42.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:42.020+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:42.027041+0000) 2019-10-13T14:22:44.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:44.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:44.355763+0000) 2019-10-13T14:22:44.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:44.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:44.641196+0000) 2019-10-13T14:22:45.333 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:45.329+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.330829+0000) 2019-10-13T14:22:45.334 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:45.328+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.331759+0000) 2019-10-13T14:22:45.335 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:45.329+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.332545+0000) 2019-10-13T14:22:45.335 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:45.329+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.333043+0000) 2019-10-13T14:22:45.336 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:45.329+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.333451+0000) 2019-10-13T14:22:45.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:45.333+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.334627+0000) 2019-10-13T14:22:45.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:45.333+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.334991+0000) 2019-10-13T14:22:45.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:45.332+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:45.337506+0000) 2019-10-13T14:22:45.344 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:45.341+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:48.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:48.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.010739+0000) 2019-10-13T14:22:48.239 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:48.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.236952+0000) 2019-10-13T14:22:48.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:48.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:48.775141+0000) 2019-10-13T14:22:49.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:49.896+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:49.897576+0000) 2019-10-13T14:22:50.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:50.341+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.344997+0000) 2019-10-13T14:22:50.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:50.341+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.345317+0000) 2019-10-13T14:22:50.349 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:50.345+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.346484+0000) 2019-10-13T14:22:50.351 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:50.345+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.348653+0000) 2019-10-13T14:22:50.353 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:50.349+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.350756+0000) 2019-10-13T14:22:50.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:50.348+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.351076+0000) 2019-10-13T14:22:50.354 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:50.349+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.351706+0000) 2019-10-13T14:22:50.357 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:50.352+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:50.354877+0000) 2019-10-13T14:22:50.359 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:50.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:51.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:51.757+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:51.758707+0000) 2019-10-13T14:22:52.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:52.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:52.027263+0000) 2019-10-13T14:22:54.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:54.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:54.355955+0000) 2019-10-13T14:22:54.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:54.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:54.641410+0000) 2019-10-13T14:22:55.362 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:55.357+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.359712+0000) 2019-10-13T14:22:55.363 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:55.357+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.360142+0000) 2019-10-13T14:22:55.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:22:55.359+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.361770+0000) 2019-10-13T14:22:55.365 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:22:55.361+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.362494+0000) 2019-10-13T14:22:55.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:22:55.361+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.362895+0000) 2019-10-13T14:22:55.365 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:55.361+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.362966+0000) 2019-10-13T14:22:55.366 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:22:55.361+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.363487+0000) 2019-10-13T14:22:55.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:55.363+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:55.367644+0000) 2019-10-13T14:22:55.371 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:22:55.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203327 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:22:58.013 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:22:58.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.010987+0000) 2019-10-13T14:22:58.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:22:58.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.237148+0000) 2019-10-13T14:22:58.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:22:58.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:58.775338+0000) 2019-10-13T14:22:59.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:22:59.899+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:22:59.904026+0000) 2019-10-13T14:23:00.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:00.369+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.372604+0000) 2019-10-13T14:23:00.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:00.373+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.374011+0000) 2019-10-13T14:23:00.377 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:00.373+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.374417+0000) 2019-10-13T14:23:00.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:00.373+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.374909+0000) 2019-10-13T14:23:00.378 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:00.373+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.375331+0000) 2019-10-13T14:23:00.378 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:00.371+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.376030+0000) 2019-10-13T14:23:00.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:00.373+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.376387+0000) 2019-10-13T14:23:00.384 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:00.379+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:00.381633+0000) 2019-10-13T14:23:00.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:00.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:01.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:01.757+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:01.758954+0000) 2019-10-13T14:23:02.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:02.023+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:02.027456+0000) 2019-10-13T14:23:04.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:04.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:04.356114+0000) 2019-10-13T14:23:04.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:04.637+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:04.641612+0000) 2019-10-13T14:23:05.390 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:05.385+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.388128+0000) 2019-10-13T14:23:05.391 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:05.385+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.389048+0000) 2019-10-13T14:23:05.392 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:05.389+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.389909+0000) 2019-10-13T14:23:05.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:05.389+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.390401+0000) 2019-10-13T14:23:05.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:05.389+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.390910+0000) 2019-10-13T14:23:05.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:05.389+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.392388+0000) 2019-10-13T14:23:05.395 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:05.391+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.392699+0000) 2019-10-13T14:23:05.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:05.391+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:05.395155+0000) 2019-10-13T14:23:05.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:05.393+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:05.714 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:23:05.715 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:05.718 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:23:05.718 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:08.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:08.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.011148+0000) 2019-10-13T14:23:08.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:08.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.237377+0000) 2019-10-13T14:23:08.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:08.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:08.775536+0000) 2019-10-13T14:23:09.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:09.907+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:09.910594+0000) 2019-10-13T14:23:10.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:10.397+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.399209+0000) 2019-10-13T14:23:10.402 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:10.397+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.399300+0000) 2019-10-13T14:23:10.403 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:10.397+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.399792+0000) 2019-10-13T14:23:10.403 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:10.397+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.400175+0000) 2019-10-13T14:23:10.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:10.397+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.400776+0000) 2019-10-13T14:23:10.404 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:10.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.402151+0000) 2019-10-13T14:23:10.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:10.401+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.402726+0000) 2019-10-13T14:23:10.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:10.407+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:10.408842+0000) 2019-10-13T14:23:10.411 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:10.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:11.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:11.757+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:11.759174+0000) 2019-10-13T14:23:12.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:12.023+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:12.027681+0000) 2019-10-13T14:23:14.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:14.353+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:14.356282+0000) 2019-10-13T14:23:14.644 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:14.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:14.641817+0000) 2019-10-13T14:23:15.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:15.409+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.413227+0000) 2019-10-13T14:23:15.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:15.409+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.413310+0000) 2019-10-13T14:23:15.416 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:15.409+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.413593+0000) 2019-10-13T14:23:15.417 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:15.413+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.414166+0000) 2019-10-13T14:23:15.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:15.413+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.415035+0000) 2019-10-13T14:23:15.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:15.413+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.415512+0000) 2019-10-13T14:23:15.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:15.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.416197+0000) 2019-10-13T14:23:15.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:15.421+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203453 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:15.425 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:15.419+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:15.422918+0000) 2019-10-13T14:23:18.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:18.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.011347+0000) 2019-10-13T14:23:18.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:18.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.237578+0000) 2019-10-13T14:23:18.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:18.773+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:18.775739+0000) 2019-10-13T14:23:19.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:19.915+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:19.916595+0000) 2019-10-13T14:23:20.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:20.424+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.426161+0000) 2019-10-13T14:23:20.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:20.424+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.427237+0000) 2019-10-13T14:23:20.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:20.424+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.427556+0000) 2019-10-13T14:23:20.430 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:20.424+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.428220+0000) 2019-10-13T14:23:20.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:20.424+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.428542+0000) 2019-10-13T14:23:20.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:20.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.429054+0000) 2019-10-13T14:23:20.432 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:20.424+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.429640+0000) 2019-10-13T14:23:20.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:20.431+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:20.434657+0000) 2019-10-13T14:23:20.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:20.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203485 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:21.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:21.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:21.759379+0000) 2019-10-13T14:23:22.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:22.023+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:22.027869+0000) 2019-10-13T14:23:24.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:24.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:24.356498+0000) 2019-10-13T14:23:24.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:24.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:24.642019+0000) 2019-10-13T14:23:25.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:25.436+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.439483+0000) 2019-10-13T14:23:25.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:25.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.441478+0000) 2019-10-13T14:23:25.444 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:25.436+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.441665+0000) 2019-10-13T14:23:25.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:25.440+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.443743+0000) 2019-10-13T14:23:25.446 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:25.440+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.444230+0000) 2019-10-13T14:23:25.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:25.440+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.444924+0000) 2019-10-13T14:23:25.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:25.440+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.445420+0000) 2019-10-13T14:23:25.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:25.443+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:25.445502+0000) 2019-10-13T14:23:25.452 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:25.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:28.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:28.008+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.011583+0000) 2019-10-13T14:23:28.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:28.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.237785+0000) 2019-10-13T14:23:28.778 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:28.772+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:28.775958+0000) 2019-10-13T14:23:29.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:29.919+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:29.923006+0000) 2019-10-13T14:23:30.455 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:30.451+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.452688+0000) 2019-10-13T14:23:30.456 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:30.452+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.454002+0000) 2019-10-13T14:23:30.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:30.452+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.454553+0000) 2019-10-13T14:23:30.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:30.452+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.455064+0000) 2019-10-13T14:23:30.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:30.452+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.455586+0000) 2019-10-13T14:23:30.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:30.452+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.456238+0000) 2019-10-13T14:23:30.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:30.452+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.456617+0000) 2019-10-13T14:23:30.465 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:30.459+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:30.462674+0000) 2019-10-13T14:23:30.465 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:30.460+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:31.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:31.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:31.759579+0000) 2019-10-13T14:23:32.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:32.027+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:32.028081+0000) 2019-10-13T14:23:34.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:34.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:34.356697+0000) 2019-10-13T14:23:34.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:34.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:34.642177+0000) 2019-10-13T14:23:35.469 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:35.464+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.466749+0000) 2019-10-13T14:23:35.470 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:35.464+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.467439+0000) 2019-10-13T14:23:35.470 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:35.463+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.467592+0000) 2019-10-13T14:23:35.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:35.464+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.468094+0000) 2019-10-13T14:23:35.471 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:35.464+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.469067+0000) 2019-10-13T14:23:35.472 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:35.464+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.469496+0000) 2019-10-13T14:23:35.472 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:35.468+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.470179+0000) 2019-10-13T14:23:35.478 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:35.472+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:35.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:35.475+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:35.478902+0000) 2019-10-13T14:23:35.763 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:23:35.763 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:35.767 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:23:35.767 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:23:38.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:38.008+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.011822+0000) 2019-10-13T14:23:38.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:38.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.237946+0000) 2019-10-13T14:23:38.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:38.772+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:38.776169+0000) 2019-10-13T14:23:39.932 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:39.927+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:39.929601+0000) 2019-10-13T14:23:40.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:40.476+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.480143+0000) 2019-10-13T14:23:40.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:40.476+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.480810+0000) 2019-10-13T14:23:40.484 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:40.476+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.481157+0000) 2019-10-13T14:23:40.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:40.479+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.481638+0000) 2019-10-13T14:23:40.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:40.480+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.482189+0000) 2019-10-13T14:23:40.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:40.480+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.482616+0000) 2019-10-13T14:23:40.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:40.480+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.483970+0000) 2019-10-13T14:23:40.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:40.488+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203611 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:40.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:40.487+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:40.491059+0000) 2019-10-13T14:23:41.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:41.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:41.759799+0000) 2019-10-13T14:23:42.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:42.027+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:42.028273+0000) 2019-10-13T14:23:44.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:44.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:44.356930+0000) 2019-10-13T14:23:44.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:44.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:44.642370+0000) 2019-10-13T14:23:45.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:45.492+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.495992+0000) 2019-10-13T14:23:45.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:45.496+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.498000+0000) 2019-10-13T14:23:45.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:45.496+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.500758+0000) 2019-10-13T14:23:45.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:45.499+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.502735+0000) 2019-10-13T14:23:45.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:45.500+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.503234+0000) 2019-10-13T14:23:45.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:45.500+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203643 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:45.507 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:45.500+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.504452+0000) 2019-10-13T14:23:45.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:45.500+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.504675+0000) 2019-10-13T14:23:45.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:45.507+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:45.511244+0000) 2019-10-13T14:23:48.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:48.008+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.012043+0000) 2019-10-13T14:23:48.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:48.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.238173+0000) 2019-10-13T14:23:48.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:48.772+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:48.776360+0000) 2019-10-13T14:23:49.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:49.931+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:49.936032+0000) 2019-10-13T14:23:50.510 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:50.504+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.507590+0000) 2019-10-13T14:23:50.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:50.504+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.508450+0000) 2019-10-13T14:23:50.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:50.504+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.508897+0000) 2019-10-13T14:23:50.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:50.504+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.509584+0000) 2019-10-13T14:23:50.512 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:50.508+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.510116+0000) 2019-10-13T14:23:50.513 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:50.507+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.510711+0000) 2019-10-13T14:23:50.513 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:50.508+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.510791+0000) 2019-10-13T14:23:50.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:50.515+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:50.517223+0000) 2019-10-13T14:23:50.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:50.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:51.762 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:51.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:51.760043+0000) 2019-10-13T14:23:52.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:52.027+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:52.028470+0000) 2019-10-13T14:23:54.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:54.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:54.357130+0000) 2019-10-13T14:23:54.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:54.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:54.642563+0000) 2019-10-13T14:23:55.526 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:23:55.520+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.523379+0000) 2019-10-13T14:23:55.526 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:23:55.520+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.523377+0000) 2019-10-13T14:23:55.527 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:55.520+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.524152+0000) 2019-10-13T14:23:55.527 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:55.520+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.524511+0000) 2019-10-13T14:23:55.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:23:55.523+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.524770+0000) 2019-10-13T14:23:55.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:23:55.520+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.525061+0000) 2019-10-13T14:23:55.528 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:55.524+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.525890+0000) 2019-10-13T14:23:55.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:55.527+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:55.530035+0000) 2019-10-13T14:23:55.533 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:23:55.528+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203705 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:23:58.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:23:58.008+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.012240+0000) 2019-10-13T14:23:58.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:23:58.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.238409+0000) 2019-10-13T14:23:58.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:23:58.776+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:58.776590+0000) 2019-10-13T14:23:59.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:23:59.938+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:23:59.942734+0000) 2019-10-13T14:24:00.537 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:00.532+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.534701+0000) 2019-10-13T14:24:00.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:00.532+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.536324+0000) 2019-10-13T14:24:00.539 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:00.532+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.536811+0000) 2019-10-13T14:24:00.539 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:00.532+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.537067+0000) 2019-10-13T14:24:00.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:00.536+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.539488+0000) 2019-10-13T14:24:00.542 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:00.534+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.539857+0000) 2019-10-13T14:24:00.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:00.536+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.540440+0000) 2019-10-13T14:24:00.547 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:00.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:00.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:00.542+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:00.547104+0000) 2019-10-13T14:24:01.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:01.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:01.760267+0000) 2019-10-13T14:24:02.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:02.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:02.028695+0000) 2019-10-13T14:24:04.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:04.352+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:04.357329+0000) 2019-10-13T14:24:04.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:04.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:04.642763+0000) 2019-10-13T14:24:05.551 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:05.544+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.548628+0000) 2019-10-13T14:24:05.552 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:05.544+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.549660+0000) 2019-10-13T14:24:05.553 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:05.548+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.551183+0000) 2019-10-13T14:24:05.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:05.548+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.552158+0000) 2019-10-13T14:24:05.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:05.548+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.552440+0000) 2019-10-13T14:24:05.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:05.550+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.553161+0000) 2019-10-13T14:24:05.556 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:05.548+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.553328+0000) 2019-10-13T14:24:05.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:05.554+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:05.557000+0000) 2019-10-13T14:24:05.561 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:05.556+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:05.877 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:24:05.877 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:05.882 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:24:05.882 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:08.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:08.008+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.012443+0000) 2019-10-13T14:24:08.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:08.236+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.238617+0000) 2019-10-13T14:24:08.779 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:08.772+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:08.776784+0000) 2019-10-13T14:24:09.951 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:09.946+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:09.949102+0000) 2019-10-13T14:24:10.564 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:10.560+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.562039+0000) 2019-10-13T14:24:10.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:10.558+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.562667+0000) 2019-10-13T14:24:10.565 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:10.560+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.563239+0000) 2019-10-13T14:24:10.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:10.564+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.567539+0000) 2019-10-13T14:24:10.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:10.566+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.569207+0000) 2019-10-13T14:24:10.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:10.564+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.569500+0000) 2019-10-13T14:24:10.573 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:10.568+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.570369+0000) 2019-10-13T14:24:10.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:10.568+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:10.570832+0000) 2019-10-13T14:24:10.575 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:10.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:11.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:11.756+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:11.760505+0000) 2019-10-13T14:24:12.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:12.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:12.028888+0000) 2019-10-13T14:24:14.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:14.356+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:14.357533+0000) 2019-10-13T14:24:14.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:14.640+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:14.642962+0000) 2019-10-13T14:24:15.579 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:15.572+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.576574+0000) 2019-10-13T14:24:15.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:15.572+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.577450+0000) 2019-10-13T14:24:15.580 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:15.576+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.577871+0000) 2019-10-13T14:24:15.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:15.574+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.578399+0000) 2019-10-13T14:24:15.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:15.576+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.578961+0000) 2019-10-13T14:24:15.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:15.576+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.579241+0000) 2019-10-13T14:24:15.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:15.576+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.579920+0000) 2019-10-13T14:24:15.587 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:15.582+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:15.585415+0000) 2019-10-13T14:24:15.588 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:15.584+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203831 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:18.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:18.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.012651+0000) 2019-10-13T14:24:18.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:18.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.238813+0000) 2019-10-13T14:24:18.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:18.775+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:18.776997+0000) 2019-10-13T14:24:19.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:19.950+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:19.955578+0000) 2019-10-13T14:24:20.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:20.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.589187+0000) 2019-10-13T14:24:20.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:20.587+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.589886+0000) 2019-10-13T14:24:20.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:20.587+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.590743+0000) 2019-10-13T14:24:20.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:20.587+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.591607+0000) 2019-10-13T14:24:20.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:20.590+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.592412+0000) 2019-10-13T14:24:20.595 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:20.587+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.592791+0000) 2019-10-13T14:24:20.595 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:20.587+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.593186+0000) 2019-10-13T14:24:20.599 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:20.594+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:20.596408+0000) 2019-10-13T14:24:20.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:20.595+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:21.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:21.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:21.760704+0000) 2019-10-13T14:24:22.031 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:22.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:22.029075+0000) 2019-10-13T14:24:24.360 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:24.355+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:24.357799+0000) 2019-10-13T14:24:24.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:24.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:24.643184+0000) 2019-10-13T14:24:25.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:25.599+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.603160+0000) 2019-10-13T14:24:25.606 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:25.599+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.603610+0000) 2019-10-13T14:24:25.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:25.599+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.603864+0000) 2019-10-13T14:24:25.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:25.599+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.604170+0000) 2019-10-13T14:24:25.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:25.599+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.604745+0000) 2019-10-13T14:24:25.608 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:25.602+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.605304+0000) 2019-10-13T14:24:25.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:25.599+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.605617+0000) 2019-10-13T14:24:25.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:25.610+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:25.612820+0000) 2019-10-13T14:24:25.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:25.607+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:28.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:28.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.012851+0000) 2019-10-13T14:24:28.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:28.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.239024+0000) 2019-10-13T14:24:28.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:28.775+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:28.777219+0000) 2019-10-13T14:24:29.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:29.958+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:29.962151+0000) 2019-10-13T14:24:30.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:30.611+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.615863+0000) 2019-10-13T14:24:30.619 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:30.611+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.616449+0000) 2019-10-13T14:24:30.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:30.611+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.616980+0000) 2019-10-13T14:24:30.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:30.611+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.617285+0000) 2019-10-13T14:24:30.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:30.614+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.617860+0000) 2019-10-13T14:24:30.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:30.615+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.618830+0000) 2019-10-13T14:24:30.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:30.615+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.619212+0000) 2019-10-13T14:24:30.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:30.622+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:30.624746+0000) 2019-10-13T14:24:30.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:30.623+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:31.763 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:31.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:31.760903+0000) 2019-10-13T14:24:32.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:32.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:32.029268+0000) 2019-10-13T14:24:34.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:34.355+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:34.357991+0000) 2019-10-13T14:24:34.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:34.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:34.643385+0000) 2019-10-13T14:24:35.632 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:35.627+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.629713+0000) 2019-10-13T14:24:35.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:35.627+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.630819+0000) 2019-10-13T14:24:35.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:35.626+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.631516+0000) 2019-10-13T14:24:35.635 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:35.627+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.632035+0000) 2019-10-13T14:24:35.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:35.627+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.632605+0000) 2019-10-13T14:24:35.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:35.627+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.633237+0000) 2019-10-13T14:24:35.636 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:35.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.633822+0000) 2019-10-13T14:24:35.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:35.634+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:35.636376+0000) 2019-10-13T14:24:35.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:35.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203957 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:35.999 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:24:36.000 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:36.003 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:24:36.003 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:24:38.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:38.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.013047+0000) 2019-10-13T14:24:38.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:38.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.239235+0000) 2019-10-13T14:24:38.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:38.771+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:38.777420+0000) 2019-10-13T14:24:39.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:39.966+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:39.968703+0000) 2019-10-13T14:24:40.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:40.639+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.643405+0000) 2019-10-13T14:24:40.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:40.639+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.644479+0000) 2019-10-13T14:24:40.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:40.643+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.646227+0000) 2019-10-13T14:24:40.649 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:40.642+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.646415+0000) 2019-10-13T14:24:40.649 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:40.643+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.647160+0000) 2019-10-13T14:24:40.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:40.643+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.647537+0000) 2019-10-13T14:24:40.650 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:40.643+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.647936+0000) 2019-10-13T14:24:40.657 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:40.651+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:40.658 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:40.650+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:40.655302+0000) 2019-10-13T14:24:41.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:41.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:41.761107+0000) 2019-10-13T14:24:42.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:42.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:42.029494+0000) 2019-10-13T14:24:44.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:44.355+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:44.358181+0000) 2019-10-13T14:24:44.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:44.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:44.643595+0000) 2019-10-13T14:24:45.660 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:45.651+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.657698+0000) 2019-10-13T14:24:45.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:45.655+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.659018+0000) 2019-10-13T14:24:45.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:45.654+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.659793+0000) 2019-10-13T14:24:45.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:45.655+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.660268+0000) 2019-10-13T14:24:45.663 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:45.655+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.660628+0000) 2019-10-13T14:24:45.663 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:45.655+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.661248+0000) 2019-10-13T14:24:45.664 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:45.659+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.661776+0000) 2019-10-13T14:24:45.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:45.662+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:45.664742+0000) 2019-10-13T14:24:45.670 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:45.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204020 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:48.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:48.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.013248+0000) 2019-10-13T14:24:48.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:48.235+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.239433+0000) 2019-10-13T14:24:48.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:48.771+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:48.777630+0000) 2019-10-13T14:24:49.983 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:49.978+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:49.980290+0000) 2019-10-13T14:24:50.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:50.667+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.671732+0000) 2019-10-13T14:24:50.675 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:50.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.672603+0000) 2019-10-13T14:24:50.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:50.667+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.672830+0000) 2019-10-13T14:24:50.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:50.667+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.673218+0000) 2019-10-13T14:24:50.676 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:50.667+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.673621+0000) 2019-10-13T14:24:50.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:50.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.674167+0000) 2019-10-13T14:24:50.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:50.671+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.674612+0000) 2019-10-13T14:24:50.683 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:50.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204052 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:50.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:50.678+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:50.680924+0000) 2019-10-13T14:24:51.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:51.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:51.761304+0000) 2019-10-13T14:24:52.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:52.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:52.029696+0000) 2019-10-13T14:24:54.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:54.355+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:54.358385+0000) 2019-10-13T14:24:54.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:54.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:54.643793+0000) 2019-10-13T14:24:55.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:24:55.679+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.683922+0000) 2019-10-13T14:24:55.690 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:24:55.682+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.687448+0000) 2019-10-13T14:24:55.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:55.683+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.689512+0000) 2019-10-13T14:24:55.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:55.687+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.690699+0000) 2019-10-13T14:24:55.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:24:55.687+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.690932+0000) 2019-10-13T14:24:55.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:24:55.687+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.691654+0000) 2019-10-13T14:24:55.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:55.687+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.691961+0000) 2019-10-13T14:24:55.695 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:55.690+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:55.692596+0000) 2019-10-13T14:24:55.696 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:24:55.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204082 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:24:58.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:24:58.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.013470+0000) 2019-10-13T14:24:58.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:24:58.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.239644+0000) 2019-10-13T14:24:58.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:24:58.775+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:58.777826+0000) 2019-10-13T14:24:59.989 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:24:59.982+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:24:59.986806+0000) 2019-10-13T14:25:00.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:00.695+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.698556+0000) 2019-10-13T14:25:00.702 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:00.695+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.698688+0000) 2019-10-13T14:25:00.702 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:00.695+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.699001+0000) 2019-10-13T14:25:00.703 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:00.694+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.699899+0000) 2019-10-13T14:25:00.703 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:00.695+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.700128+0000) 2019-10-13T14:25:00.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:00.695+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.700415+0000) 2019-10-13T14:25:00.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:00.695+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.700751+0000) 2019-10-13T14:25:00.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:00.702+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:00.704537+0000) 2019-10-13T14:25:00.710 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:00.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204114 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:01.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:01.755+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:01.761503+0000) 2019-10-13T14:25:02.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:02.026+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:02.029904+0000) 2019-10-13T14:25:04.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:04.355+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:04.358598+0000) 2019-10-13T14:25:04.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:04.639+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:04.643985+0000) 2019-10-13T14:25:05.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:05.707+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.710317+0000) 2019-10-13T14:25:05.714 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:05.707+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.710900+0000) 2019-10-13T14:25:05.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:05.707+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.712926+0000) 2019-10-13T14:25:05.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:05.709+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.713312+0000) 2019-10-13T14:25:05.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:05.711+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.713794+0000) 2019-10-13T14:25:05.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:05.711+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.714007+0000) 2019-10-13T14:25:05.717 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:05.711+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.714539+0000) 2019-10-13T14:25:05.723 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:05.715+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:05.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:05.717+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:05.721452+0000) 2019-10-13T14:25:06.096 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:25:06.097 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:06.100 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:25:06.100 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:08.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:08.007+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.013682+0000) 2019-10-13T14:25:08.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:08.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.239866+0000) 2019-10-13T14:25:08.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:08.775+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:08.778076+0000) 2019-10-13T14:25:09.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:09.989+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:09.995825+0000) 2019-10-13T14:25:10.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:10.719+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.724484+0000) 2019-10-13T14:25:10.728 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:10.721+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.725919+0000) 2019-10-13T14:25:10.729 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:10.723+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.726485+0000) 2019-10-13T14:25:10.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:10.723+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.726114+0000) 2019-10-13T14:25:10.729 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:10.723+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.726928+0000) 2019-10-13T14:25:10.730 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:10.723+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.727815+0000) 2019-10-13T14:25:10.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:10.723+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.728553+0000) 2019-10-13T14:25:10.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:10.729+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:10.732293+0000) 2019-10-13T14:25:10.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:10.731+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204178 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:11.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:11.759+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:11.761735+0000) 2019-10-13T14:25:12.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:12.025+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:12.030131+0000) 2019-10-13T14:25:14.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:14.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:14.358803+0000) 2019-10-13T14:25:14.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:14.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:14.644140+0000) 2019-10-13T14:25:15.743 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:15.730+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.737315+0000) 2019-10-13T14:25:15.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:15.733+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.738002+0000) 2019-10-13T14:25:15.745 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:15.734+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.738547+0000) 2019-10-13T14:25:15.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:15.734+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.739248+0000) 2019-10-13T14:25:15.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:15.734+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.741486+0000) 2019-10-13T14:25:15.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:15.738+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.742982+0000) 2019-10-13T14:25:15.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:15.738+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.744197+0000) 2019-10-13T14:25:15.752 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:15.742+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:15.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:15.745+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:15.750384+0000) 2019-10-13T14:25:18.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:18.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.013914+0000) 2019-10-13T14:25:18.242 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:18.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.240063+0000) 2019-10-13T14:25:18.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:18.774+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:18.778306+0000) 2019-10-13T14:25:20.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:19.997+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.001552+0000) 2019-10-13T14:25:20.753 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:20.746+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.751195+0000) 2019-10-13T14:25:20.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:20.745+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.751948+0000) 2019-10-13T14:25:20.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:20.746+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.753104+0000) 2019-10-13T14:25:20.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:20.746+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.753412+0000) 2019-10-13T14:25:20.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:20.750+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.754124+0000) 2019-10-13T14:25:20.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:20.750+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.754692+0000) 2019-10-13T14:25:20.758 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:20.750+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.755496+0000) 2019-10-13T14:25:20.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:20.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204240 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:20.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:20.757+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:20.763020+0000) 2019-10-13T14:25:21.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:21.758+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:21.762009+0000) 2019-10-13T14:25:22.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:22.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:22.030324+0000) 2019-10-13T14:25:24.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:24.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:24.358953+0000) 2019-10-13T14:25:24.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:24.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:24.644329+0000) 2019-10-13T14:25:25.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:25.761+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.767879+0000) 2019-10-13T14:25:25.773 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:25.766+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.770577+0000) 2019-10-13T14:25:25.773 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:25.766+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.770711+0000) 2019-10-13T14:25:25.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:25.766+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.771120+0000) 2019-10-13T14:25:25.774 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:25.766+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.771368+0000) 2019-10-13T14:25:25.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:25.766+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.771811+0000) 2019-10-13T14:25:25.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:25.766+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.772859+0000) 2019-10-13T14:25:25.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:25.769+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:25.773497+0000) 2019-10-13T14:25:25.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:25.770+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204272 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:28.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:28.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.014159+0000) 2019-10-13T14:25:28.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:28.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.240292+0000) 2019-10-13T14:25:28.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:28.774+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:28.778504+0000) 2019-10-13T14:25:30.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:30.005+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.009101+0000) 2019-10-13T14:25:30.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:30.774+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.777995+0000) 2019-10-13T14:25:30.781 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:30.774+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.778545+0000) 2019-10-13T14:25:30.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:30.774+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.779907+0000) 2019-10-13T14:25:30.783 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:30.777+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.780800+0000) 2019-10-13T14:25:30.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:30.774+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.780965+0000) 2019-10-13T14:25:30.784 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:30.774+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.781462+0000) 2019-10-13T14:25:30.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:30.778+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.781854+0000) 2019-10-13T14:25:30.786 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:30.777+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:30.783576+0000) 2019-10-13T14:25:30.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:30.782+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204304 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:31.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:31.758+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:31.762208+0000) 2019-10-13T14:25:32.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:32.025+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:32.030523+0000) 2019-10-13T14:25:34.361 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:34.358+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:34.359150+0000) 2019-10-13T14:25:34.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:34.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:34.644517+0000) 2019-10-13T14:25:35.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:35.786+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.791106+0000) 2019-10-13T14:25:35.794 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:35.786+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.792343+0000) 2019-10-13T14:25:35.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:35.786+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.792750+0000) 2019-10-13T14:25:35.796 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:35.786+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.793408+0000) 2019-10-13T14:25:35.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:35.786+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.793535+0000) 2019-10-13T14:25:35.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:35.789+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.794166+0000) 2019-10-13T14:25:35.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:35.790+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.794458+0000) 2019-10-13T14:25:35.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:35.797+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:35.800730+0000) 2019-10-13T14:25:35.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:35.794+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204334 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:36.223 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:25:36.224 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:36.227 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:25:36.227 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:25:38.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:38.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.014333+0000) 2019-10-13T14:25:38.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:38.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.240517+0000) 2019-10-13T14:25:38.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:38.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:38.778723+0000) 2019-10-13T14:25:40.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:40.009+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.015507+0000) 2019-10-13T14:25:40.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:40.798+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.804625+0000) 2019-10-13T14:25:40.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:40.798+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.805622+0000) 2019-10-13T14:25:40.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:40.802+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.806102+0000) 2019-10-13T14:25:40.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:40.801+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.806534+0000) 2019-10-13T14:25:40.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:40.802+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.806678+0000) 2019-10-13T14:25:40.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:40.802+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.807164+0000) 2019-10-13T14:25:40.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:40.802+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.807810+0000) 2019-10-13T14:25:40.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:40.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:40.814882+0000) 2019-10-13T14:25:40.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:40.810+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204366 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:41.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:41.758+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:41.762420+0000) 2019-10-13T14:25:42.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:42.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:42.030712+0000) 2019-10-13T14:25:44.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:44.358+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:44.359366+0000) 2019-10-13T14:25:44.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:44.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:44.644718+0000) 2019-10-13T14:25:45.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:45.814+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.818117+0000) 2019-10-13T14:25:45.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:45.814+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.818988+0000) 2019-10-13T14:25:45.821 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:45.814+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.819411+0000) 2019-10-13T14:25:45.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:45.814+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.819763+0000) 2019-10-13T14:25:45.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:45.817+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.820390+0000) 2019-10-13T14:25:45.823 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:45.814+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.820977+0000) 2019-10-13T14:25:45.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:45.814+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.821519+0000) 2019-10-13T14:25:45.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:45.822+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204398 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:45.831 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:45.825+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:45.828760+0000) 2019-10-13T14:25:48.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:48.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.014524+0000) 2019-10-13T14:25:48.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:48.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.240738+0000) 2019-10-13T14:25:48.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:48.774+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:48.778919+0000) 2019-10-13T14:25:50.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:50.017+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.022054+0000) 2019-10-13T14:25:50.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:50.829+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.832274+0000) 2019-10-13T14:25:50.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:50.826+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.832603+0000) 2019-10-13T14:25:50.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:50.826+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.832860+0000) 2019-10-13T14:25:50.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:50.830+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.834925+0000) 2019-10-13T14:25:50.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:50.830+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.837356+0000) 2019-10-13T14:25:50.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:50.830+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.837594+0000) 2019-10-13T14:25:50.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:50.834+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.837996+0000) 2019-10-13T14:25:50.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:50.838+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:50.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:50.837+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:50.843410+0000) 2019-10-13T14:25:51.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:51.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:51.762661+0000) 2019-10-13T14:25:52.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:52.025+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:52.030876+0000) 2019-10-13T14:25:54.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:54.354+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:54.359569+0000) 2019-10-13T14:25:54.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:54.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:54.644917+0000) 2019-10-13T14:25:55.849 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:25:55.842+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.846474+0000) 2019-10-13T14:25:55.849 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:25:55.841+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.846976+0000) 2019-10-13T14:25:55.850 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:25:55.842+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.847400+0000) 2019-10-13T14:25:55.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:55.842+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.847872+0000) 2019-10-13T14:25:55.852 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:55.842+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.848810+0000) 2019-10-13T14:25:55.852 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:55.842+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.849187+0000) 2019-10-13T14:25:55.852 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:25:55.846+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.850147+0000) 2019-10-13T14:25:55.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:25:55.849+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:55.853391+0000) 2019-10-13T14:25:55.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:25:55.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:25:58.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:25:58.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.014680+0000) 2019-10-13T14:25:58.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:25:58.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.240990+0000) 2019-10-13T14:25:58.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:25:58.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:25:58.779111+0000) 2019-10-13T14:26:00.031 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:00.025+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.028389+0000) 2019-10-13T14:26:00.862 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:00.854+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.859832+0000) 2019-10-13T14:26:00.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:00.854+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.860504+0000) 2019-10-13T14:26:00.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:00.858+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.862433+0000) 2019-10-13T14:26:00.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:00.858+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.863048+0000) 2019-10-13T14:26:00.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:00.857+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.863955+0000) 2019-10-13T14:26:00.867 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:00.858+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.864227+0000) 2019-10-13T14:26:00.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:00.858+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.864835+0000) 2019-10-13T14:26:00.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:00.865+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:00.869849+0000) 2019-10-13T14:26:00.872 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:00.866+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204492 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:01.765 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:01.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:01.762845+0000) 2019-10-13T14:26:02.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:02.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:02.031077+0000) 2019-10-13T14:26:04.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:04.358+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:04.359771+0000) 2019-10-13T14:26:04.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:04.638+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:04.645121+0000) 2019-10-13T14:26:05.877 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:05.870+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.874394+0000) 2019-10-13T14:26:05.878 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:05.870+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.876238+0000) 2019-10-13T14:26:05.879 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:05.870+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.876865+0000) 2019-10-13T14:26:05.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:05.874+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.879037+0000) 2019-10-13T14:26:05.881 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:05.874+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.879189+0000) 2019-10-13T14:26:05.883 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:05.874+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.880894+0000) 2019-10-13T14:26:05.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:05.877+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.882808+0000) 2019-10-13T14:26:05.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:05.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:05.888 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:05.881+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:05.885755+0000) 2019-10-13T14:26:06.349 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:26:06.349 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:06.355 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:26:06.356 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:08.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:08.010+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.014889+0000) 2019-10-13T14:26:08.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:08.234+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.241188+0000) 2019-10-13T14:26:08.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:08.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:08.779314+0000) 2019-10-13T14:26:10.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:10.028+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.034618+0000) 2019-10-13T14:26:10.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:10.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.888858+0000) 2019-10-13T14:26:10.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:10.885+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.890615+0000) 2019-10-13T14:26:10.893 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:10.885+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.891064+0000) 2019-10-13T14:26:10.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:10.885+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.891616+0000) 2019-10-13T14:26:10.894 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:10.888+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.892292+0000) 2019-10-13T14:26:10.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:10.885+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.892090+0000) 2019-10-13T14:26:10.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:10.885+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.892908+0000) 2019-10-13T14:26:10.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:10.888+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:10.894019+0000) 2019-10-13T14:26:10.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:10.893+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204556 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:11.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:11.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:11.763043+0000) 2019-10-13T14:26:12.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:12.028+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:12.031267+0000) 2019-10-13T14:26:14.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:14.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:14.359967+0000) 2019-10-13T14:26:14.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:14.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:14.645344+0000) 2019-10-13T14:26:15.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:15.897+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.902547+0000) 2019-10-13T14:26:15.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:15.897+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.903025+0000) 2019-10-13T14:26:15.907 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:15.897+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.905172+0000) 2019-10-13T14:26:15.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:15.897+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.905541+0000) 2019-10-13T14:26:15.910 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:15.904+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.908450+0000) 2019-10-13T14:26:15.911 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:15.901+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.908522+0000) 2019-10-13T14:26:15.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:15.901+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.909342+0000) 2019-10-13T14:26:15.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:15.905+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204586 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:15.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:15.908+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:15.912976+0000) 2019-10-13T14:26:18.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:18.009+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.015037+0000) 2019-10-13T14:26:18.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:18.233+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.241387+0000) 2019-10-13T14:26:18.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:18.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:18.779535+0000) 2019-10-13T14:26:20.044 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:20.036+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.041170+0000) 2019-10-13T14:26:20.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:20.909+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.915151+0000) 2019-10-13T14:26:20.918 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:20.909+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.915444+0000) 2019-10-13T14:26:20.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:20.909+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.915818+0000) 2019-10-13T14:26:20.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:20.912+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.916567+0000) 2019-10-13T14:26:20.920 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:20.909+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.916638+0000) 2019-10-13T14:26:20.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:20.909+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.917065+0000) 2019-10-13T14:26:20.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:20.913+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.917891+0000) 2019-10-13T14:26:20.927 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:20.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204618 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:20.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:20.920+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:20.926958+0000) 2019-10-13T14:26:21.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:21.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:21.763217+0000) 2019-10-13T14:26:22.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:22.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:22.031494+0000) 2019-10-13T14:26:24.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:24.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:24.360176+0000) 2019-10-13T14:26:24.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:24.641+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:24.645550+0000) 2019-10-13T14:26:25.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:25.925+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.928859+0000) 2019-10-13T14:26:25.933 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:25.929+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.930803+0000) 2019-10-13T14:26:25.933 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:25.929+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.930951+0000) 2019-10-13T14:26:25.934 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:25.929+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.931485+0000) 2019-10-13T14:26:25.936 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:25.933+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.934241+0000) 2019-10-13T14:26:25.939 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:25.932+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.936545+0000) 2019-10-13T14:26:25.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:25.937+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:25.941 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:25.937+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.939002+0000) 2019-10-13T14:26:25.947 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:25.940+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:25.944995+0000) 2019-10-13T14:26:28.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:28.013+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.015237+0000) 2019-10-13T14:26:28.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:28.237+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.241607+0000) 2019-10-13T14:26:28.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:28.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:28.779753+0000) 2019-10-13T14:26:30.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:30.044+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.047236+0000) 2019-10-13T14:26:30.945 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:30.941+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.942934+0000) 2019-10-13T14:26:30.947 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:30.941+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.945040+0000) 2019-10-13T14:26:30.948 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:30.941+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.945628+0000) 2019-10-13T14:26:30.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:30.945+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.946426+0000) 2019-10-13T14:26:30.950 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:30.945+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.947815+0000) 2019-10-13T14:26:30.950 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:30.944+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.948179+0000) 2019-10-13T14:26:30.953 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:30.949+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.950165+0000) 2019-10-13T14:26:30.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:30.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:30.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:30.952+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:30.956447+0000) 2019-10-13T14:26:31.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:31.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:31.763394+0000) 2019-10-13T14:26:32.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:32.028+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:32.031690+0000) 2019-10-13T14:26:34.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:34.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:34.360413+0000) 2019-10-13T14:26:34.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:34.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:34.645732+0000) 2019-10-13T14:26:35.958 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:35.953+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.955762+0000) 2019-10-13T14:26:35.959 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:35.953+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.956318+0000) 2019-10-13T14:26:35.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:35.957+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.959085+0000) 2019-10-13T14:26:35.964 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:35.957+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.961273+0000) 2019-10-13T14:26:35.964 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:35.957+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.961512+0000) 2019-10-13T14:26:35.965 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:35.961+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.962317+0000) 2019-10-13T14:26:35.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:35.956+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.963454+0000) 2019-10-13T14:26:35.966 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:35.960+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:35.964273+0000) 2019-10-13T14:26:35.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:35.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:36.453 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:26:36.453 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:36.457 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:26:36.457 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:26:38.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:38.013+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.015433+0000) 2019-10-13T14:26:38.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:38.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.241803+0000) 2019-10-13T14:26:38.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:38.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:38.779941+0000) 2019-10-13T14:26:40.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:40.052+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.053748+0000) 2019-10-13T14:26:40.972 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:40.965+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.969383+0000) 2019-10-13T14:26:40.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:40.969+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.969982+0000) 2019-10-13T14:26:40.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:40.964+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.970997+0000) 2019-10-13T14:26:40.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:40.969+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.971480+0000) 2019-10-13T14:26:40.974 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:40.969+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.972029+0000) 2019-10-13T14:26:40.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:40.969+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.972208+0000) 2019-10-13T14:26:40.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:40.969+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.972873+0000) 2019-10-13T14:26:40.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:40.972+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:40.977310+0000) 2019-10-13T14:26:40.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:40.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:41.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:41.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:41.763578+0000) 2019-10-13T14:26:42.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:42.024+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:42.031906+0000) 2019-10-13T14:26:44.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:44.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:44.360615+0000) 2019-10-13T14:26:44.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:44.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:44.645977+0000) 2019-10-13T14:26:45.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:45.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.981980+0000) 2019-10-13T14:26:45.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:45.981+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.982540+0000) 2019-10-13T14:26:45.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:45.981+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.983400+0000) 2019-10-13T14:26:45.986 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:45.976+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.983826+0000) 2019-10-13T14:26:45.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:45.981+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.985605+0000) 2019-10-13T14:26:45.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:45.985+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.986425+0000) 2019-10-13T14:26:45.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:45.985+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.986859+0000) 2019-10-13T14:26:45.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:45.984+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:45.989428+0000) 2019-10-13T14:26:45.995 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:45.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:48.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:48.013+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.015624+0000) 2019-10-13T14:26:48.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:48.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.242000+0000) 2019-10-13T14:26:48.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:48.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:48.780136+0000) 2019-10-13T14:26:50.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:50.056+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.060267+0000) 2019-10-13T14:26:50.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:50.993+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.995921+0000) 2019-10-13T14:26:50.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:50.993+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.997085+0000) 2019-10-13T14:26:51.000 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:50.993+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.997540+0000) 2019-10-13T14:26:51.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:50.997+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.998176+0000) 2019-10-13T14:26:51.001 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:50.992+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.998594+0000) 2019-10-13T14:26:51.001 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:50.997+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.998024+0000) 2019-10-13T14:26:51.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:50.997+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:50.998866+0000) 2019-10-13T14:26:51.008 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:51.001+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204808 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:51.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:51.000+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:51.007471+0000) 2019-10-13T14:26:51.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:51.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:51.763753+0000) 2019-10-13T14:26:52.034 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:52.028+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:52.032131+0000) 2019-10-13T14:26:54.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:54.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:54.360812+0000) 2019-10-13T14:26:54.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:54.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:54.646176+0000) 2019-10-13T14:26:56.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:26:56.009+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.011413+0000) 2019-10-13T14:26:56.014 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:26:56.009+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.011472+0000) 2019-10-13T14:26:56.015 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:56.009+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.012078+0000) 2019-10-13T14:26:56.015 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:56.009+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.012367+0000) 2019-10-13T14:26:56.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:26:56.008+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.013203+0000) 2019-10-13T14:26:56.017 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:26:56.009+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.013285+0000) 2019-10-13T14:26:56.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:56.009+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.013460+0000) 2019-10-13T14:26:56.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:26:56.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:26:56.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:26:56.012+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:56.019249+0000) 2019-10-13T14:26:58.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:26:58.013+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.015838+0000) 2019-10-13T14:26:58.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:26:58.241+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.242221+0000) 2019-10-13T14:26:58.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:26:58.777+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:26:58.780337+0000) 2019-10-13T14:27:00.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:00.064+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:00.067103+0000) 2019-10-13T14:27:01.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:01.016+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.022489+0000) 2019-10-13T14:27:01.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:01.021+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.022647+0000) 2019-10-13T14:27:01.026 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:01.021+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.023583+0000) 2019-10-13T14:27:01.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:01.021+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.023965+0000) 2019-10-13T14:27:01.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:01.021+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.024718+0000) 2019-10-13T14:27:01.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:01.021+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.024649+0000) 2019-10-13T14:27:01.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:01.021+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.025324+0000) 2019-10-13T14:27:01.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:01.029+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:01.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:01.028+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.032413+0000) 2019-10-13T14:27:01.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:01.761+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:01.763927+0000) 2019-10-13T14:27:02.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:02.028+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:02.032322+0000) 2019-10-13T14:27:04.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:04.357+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.361026+0000) 2019-10-13T14:27:04.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:04.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:04.646391+0000) 2019-10-13T14:27:06.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:06.032+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.035210+0000) 2019-10-13T14:27:06.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:06.032+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.036104+0000) 2019-10-13T14:27:06.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:06.032+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.036505+0000) 2019-10-13T14:27:06.040 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:06.032+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.037673+0000) 2019-10-13T14:27:06.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:06.036+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.038116+0000) 2019-10-13T14:27:06.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:06.032+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.039017+0000) 2019-10-13T14:27:06.043 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:06.036+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.040445+0000) 2019-10-13T14:27:06.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:06.036+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:06.043278+0000) 2019-10-13T14:27:06.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:06.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:06.610 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:27:06.611 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:06.614 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:27:06.614 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:08.018 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:08.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:08.016101+0000) 2019-10-13T14:27:08.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:08.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:08.242468+0000) 2019-10-13T14:27:08.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:08.776+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:08.780559+0000) 2019-10-13T14:27:10.077 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:10.072+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:10.074193+0000) 2019-10-13T14:27:11.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:11.044+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.049343+0000) 2019-10-13T14:27:11.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:11.044+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.049548+0000) 2019-10-13T14:27:11.053 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:11.048+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.050094+0000) 2019-10-13T14:27:11.054 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:11.048+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.051818+0000) 2019-10-13T14:27:11.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:11.048+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.053071+0000) 2019-10-13T14:27:11.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:11.052+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.055439+0000) 2019-10-13T14:27:11.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:11.056+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.058779+0000) 2019-10-13T14:27:11.061 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:11.056+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.059275+0000) 2019-10-13T14:27:11.062 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:11.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204933 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:11.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:11.760+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:11.764106+0000) 2019-10-13T14:27:12.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:12.032+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:12.032507+0000) 2019-10-13T14:27:14.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:14.356+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.361242+0000) 2019-10-13T14:27:14.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:14.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:14.646583+0000) 2019-10-13T14:27:16.067 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:16.060+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.064339+0000) 2019-10-13T14:27:16.068 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:16.060+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.064602+0000) 2019-10-13T14:27:16.068 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:16.060+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.064943+0000) 2019-10-13T14:27:16.068 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:16.060+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.064937+0000) 2019-10-13T14:27:16.069 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:16.063+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.066299+0000) 2019-10-13T14:27:16.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:16.060+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.065475+0000) 2019-10-13T14:27:16.069 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:16.064+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.065873+0000) 2019-10-13T14:27:16.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:16.071+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:16.073139+0000) 2019-10-13T14:27:16.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:16.068+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204964 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:18.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:18.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:18.016305+0000) 2019-10-13T14:27:18.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:18.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:18.242623+0000) 2019-10-13T14:27:18.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:18.776+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:18.780758+0000) 2019-10-13T14:27:20.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:20.079+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:20.081446+0000) 2019-10-13T14:27:21.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:21.075+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.076890+0000) 2019-10-13T14:27:21.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:21.072+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.077166+0000) 2019-10-13T14:27:21.080 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:21.076+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.077899+0000) 2019-10-13T14:27:21.084 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:21.080+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.082156+0000) 2019-10-13T14:27:21.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:21.080+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.083965+0000) 2019-10-13T14:27:21.087 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:21.083+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.085245+0000) 2019-10-13T14:27:21.088 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:21.080+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.085689+0000) 2019-10-13T14:27:21.089 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:21.084+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.086718+0000) 2019-10-13T14:27:21.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:21.084+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204995 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:21.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:21.760+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:21.764251+0000) 2019-10-13T14:27:22.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:22.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:22.032694+0000) 2019-10-13T14:27:24.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:24.356+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.361467+0000) 2019-10-13T14:27:24.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:24.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:24.646780+0000) 2019-10-13T14:27:26.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:26.088+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.091172+0000) 2019-10-13T14:27:26.094 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:26.088+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.091853+0000) 2019-10-13T14:27:26.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:26.088+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.093502+0000) 2019-10-13T14:27:26.096 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:26.092+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.093949+0000) 2019-10-13T14:27:26.097 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:26.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.094838+0000) 2019-10-13T14:27:26.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:26.091+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.095391+0000) 2019-10-13T14:27:26.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:26.092+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.096188+0000) 2019-10-13T14:27:26.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:26.096+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205028 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:26.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:26.099+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:26.101055+0000) 2019-10-13T14:27:28.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:28.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:28.016458+0000) 2019-10-13T14:27:28.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:28.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:28.242817+0000) 2019-10-13T14:27:28.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:28.780+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:28.780951+0000) 2019-10-13T14:27:30.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:30.087+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:30.088549+0000) 2019-10-13T14:27:31.107 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:31.100+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.104382+0000) 2019-10-13T14:27:31.107 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:31.100+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.104528+0000) 2019-10-13T14:27:31.108 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:31.100+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.105153+0000) 2019-10-13T14:27:31.108 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:31.104+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.105946+0000) 2019-10-13T14:27:31.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:31.104+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.106016+0000) 2019-10-13T14:27:31.109 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:31.104+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.106445+0000) 2019-10-13T14:27:31.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:31.103+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.107556+0000) 2019-10-13T14:27:31.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:31.107+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.109877+0000) 2019-10-13T14:27:31.115 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:31.108+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205059 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:31.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:31.760+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:31.764357+0000) 2019-10-13T14:27:32.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:32.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:32.032865+0000) 2019-10-13T14:27:34.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:34.356+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.361672+0000) 2019-10-13T14:27:34.649 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:34.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:34.646969+0000) 2019-10-13T14:27:36.119 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:36.112+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.116361+0000) 2019-10-13T14:27:36.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:36.112+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.116703+0000) 2019-10-13T14:27:36.120 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:36.116+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.118122+0000) 2019-10-13T14:27:36.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:36.116+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.118364+0000) 2019-10-13T14:27:36.122 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:36.115+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.119695+0000) 2019-10-13T14:27:36.122 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:36.116+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.119423+0000) 2019-10-13T14:27:36.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:36.116+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.120126+0000) 2019-10-13T14:27:36.126 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:36.119+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:36.123717+0000) 2019-10-13T14:27:36.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:36.124+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205089 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:36.724 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:27:36.724 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:36.727 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:27:36.728 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:27:38.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:38.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:38.016615+0000) 2019-10-13T14:27:38.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:38.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:38.243017+0000) 2019-10-13T14:27:38.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:38.776+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:38.781147+0000) 2019-10-13T14:27:40.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:40.091+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:40.095589+0000) 2019-10-13T14:27:41.132 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:41.124+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.129512+0000) 2019-10-13T14:27:41.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:41.127+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.130055+0000) 2019-10-13T14:27:41.133 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:41.128+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.130693+0000) 2019-10-13T14:27:41.134 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:41.128+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.131226+0000) 2019-10-13T14:27:41.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:41.128+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.131655+0000) 2019-10-13T14:27:41.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:41.128+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.132337+0000) 2019-10-13T14:27:41.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:41.128+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.132892+0000) 2019-10-13T14:27:41.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:41.135+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.138995+0000) 2019-10-13T14:27:41.142 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:41.136+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205120 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:41.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:41.760+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:41.764529+0000) 2019-10-13T14:27:42.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:42.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:42.033051+0000) 2019-10-13T14:27:44.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:44.360+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.361901+0000) 2019-10-13T14:27:44.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:44.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:44.647164+0000) 2019-10-13T14:27:46.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:46.140+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.142619+0000) 2019-10-13T14:27:46.147 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:46.143+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.144521+0000) 2019-10-13T14:27:46.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:46.140+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.144887+0000) 2019-10-13T14:27:46.149 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:46.144+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.146927+0000) 2019-10-13T14:27:46.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:46.144+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.147324+0000) 2019-10-13T14:27:46.150 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:46.144+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.147629+0000) 2019-10-13T14:27:46.152 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:46.148+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.150415+0000) 2019-10-13T14:27:46.156 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:46.152+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205152 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:46.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:46.151+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:46.154491+0000) 2019-10-13T14:27:48.019 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:48.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:48.016810+0000) 2019-10-13T14:27:48.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:48.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:48.243216+0000) 2019-10-13T14:27:48.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:48.780+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:48.781360+0000) 2019-10-13T14:27:50.105 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:50.099+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:50.102783+0000) 2019-10-13T14:27:51.160 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:51.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.157336+0000) 2019-10-13T14:27:51.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:51.156+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.157909+0000) 2019-10-13T14:27:51.161 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:51.156+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.158224+0000) 2019-10-13T14:27:51.161 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:51.155+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.159067+0000) 2019-10-13T14:27:51.162 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:51.156+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.159384+0000) 2019-10-13T14:27:51.162 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:51.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.160111+0000) 2019-10-13T14:27:51.163 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:51.156+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.161049+0000) 2019-10-13T14:27:51.167 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:51.163+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.165210+0000) 2019-10-13T14:27:51.170 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:51.164+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:51.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:51.760+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:51.764701+0000) 2019-10-13T14:27:52.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:52.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:52.033216+0000) 2019-10-13T14:27:54.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:54.360+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.362107+0000) 2019-10-13T14:27:54.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:54.644+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:54.647358+0000) 2019-10-13T14:27:56.174 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:27:56.168+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.171750+0000) 2019-10-13T14:27:56.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:27:56.171+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.172580+0000) 2019-10-13T14:27:56.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:56.168+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.172997+0000) 2019-10-13T14:27:56.176 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:56.168+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.173506+0000) 2019-10-13T14:27:56.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:56.172+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.174174+0000) 2019-10-13T14:27:56.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:27:56.172+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.174465+0000) 2019-10-13T14:27:56.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:27:56.172+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.174805+0000) 2019-10-13T14:27:56.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:27:56.175+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:56.178463+0000) 2019-10-13T14:27:56.185 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:27:56.180+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205213 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:27:58.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:27:58.012+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:58.016972+0000) 2019-10-13T14:27:58.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:27:58.240+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:58.243423+0000) 2019-10-13T14:27:58.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:27:58.776+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:27:58.781548+0000) 2019-10-13T14:28:00.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:00.107+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:00.110238+0000) 2019-10-13T14:28:01.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:01.183+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.186358+0000) 2019-10-13T14:28:01.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:01.183+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.186264+0000) 2019-10-13T14:28:01.190 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:01.183+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.186797+0000) 2019-10-13T14:28:01.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:01.183+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.187518+0000) 2019-10-13T14:28:01.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:01.187+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.188602+0000) 2019-10-13T14:28:01.191 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:01.183+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.188935+0000) 2019-10-13T14:28:01.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:01.183+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.189369+0000) 2019-10-13T14:28:01.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:01.191+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.195455+0000) 2019-10-13T14:28:01.198 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:01.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:01.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:01.759+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:01.764890+0000) 2019-10-13T14:28:02.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:02.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:02.033390+0000) 2019-10-13T14:28:04.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:04.359+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.362305+0000) 2019-10-13T14:28:04.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:04.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:04.647595+0000) 2019-10-13T14:28:06.202 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:06.195+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.199299+0000) 2019-10-13T14:28:06.202 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:06.195+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.199462+0000) 2019-10-13T14:28:06.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:06.195+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.200157+0000) 2019-10-13T14:28:06.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:06.195+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.199910+0000) 2019-10-13T14:28:06.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:06.199+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.200484+0000) 2019-10-13T14:28:06.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:06.195+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.200836+0000) 2019-10-13T14:28:06.204 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:06.199+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.202427+0000) 2019-10-13T14:28:06.210 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:06.203+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:06.207735+0000) 2019-10-13T14:28:06.213 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:06.207+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:06.795 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:28:06.796 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:06.800 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:28:06.800 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:08.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:08.011+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:08.017175+0000) 2019-10-13T14:28:08.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:08.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:08.243640+0000) 2019-10-13T14:28:08.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:08.779+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:08.781782+0000) 2019-10-13T14:28:10.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:10.115+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:10.117444+0000) 2019-10-13T14:28:11.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:11.211+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.213826+0000) 2019-10-13T14:28:11.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:11.211+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.215023+0000) 2019-10-13T14:28:11.220 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:11.211+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.217616+0000) 2019-10-13T14:28:11.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:11.215+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.218078+0000) 2019-10-13T14:28:11.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:11.215+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.218722+0000) 2019-10-13T14:28:11.221 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:11.215+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.219127+0000) 2019-10-13T14:28:11.222 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:11.215+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.219673+0000) 2019-10-13T14:28:11.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:11.219+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.220435+0000) 2019-10-13T14:28:11.226 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:11.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:11.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:11.759+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:11.765020+0000) 2019-10-13T14:28:12.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:12.031+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:12.033561+0000) 2019-10-13T14:28:14.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:14.359+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.362512+0000) 2019-10-13T14:28:14.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:14.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:14.647779+0000) 2019-10-13T14:28:16.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:16.223+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.227145+0000) 2019-10-13T14:28:16.231 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:16.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.228433+0000) 2019-10-13T14:28:16.231 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:16.223+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.229175+0000) 2019-10-13T14:28:16.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:16.227+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.229679+0000) 2019-10-13T14:28:16.232 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:16.227+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.230032+0000) 2019-10-13T14:28:16.233 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:16.227+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.230583+0000) 2019-10-13T14:28:16.233 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:16.227+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.231084+0000) 2019-10-13T14:28:16.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:16.231+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:16.234554+0000) 2019-10-13T14:28:16.240 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:16.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205337 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:18.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:18.015+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:18.017360+0000) 2019-10-13T14:28:18.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:18.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:18.243838+0000) 2019-10-13T14:28:18.784 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:18.779+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:18.781994+0000) 2019-10-13T14:28:20.131 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:20.126+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:20.128648+0000) 2019-10-13T14:28:21.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:21.235+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.240867+0000) 2019-10-13T14:28:21.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:21.235+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.241456+0000) 2019-10-13T14:28:21.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:21.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.244061+0000) 2019-10-13T14:28:21.247 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:21.239+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.244304+0000) 2019-10-13T14:28:21.247 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:21.239+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.244900+0000) 2019-10-13T14:28:21.248 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:21.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.245862+0000) 2019-10-13T14:28:21.249 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:21.243+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.246150+0000) 2019-10-13T14:28:21.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:21.246+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.249308+0000) 2019-10-13T14:28:21.254 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:21.247+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:21.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:21.759+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:21.765190+0000) 2019-10-13T14:28:22.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:22.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:22.033750+0000) 2019-10-13T14:28:24.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:24.359+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.362718+0000) 2019-10-13T14:28:24.650 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:24.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:24.647937+0000) 2019-10-13T14:28:26.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:26.251+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.254788+0000) 2019-10-13T14:28:26.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:26.251+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.255108+0000) 2019-10-13T14:28:26.258 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:26.250+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.255855+0000) 2019-10-13T14:28:26.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:26.251+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.257521+0000) 2019-10-13T14:28:26.260 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:26.255+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.257795+0000) 2019-10-13T14:28:26.261 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:26.255+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.259220+0000) 2019-10-13T14:28:26.262 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:26.255+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.260458+0000) 2019-10-13T14:28:26.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:26.262+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:26.264991+0000) 2019-10-13T14:28:26.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:26.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:28.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:28.015+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:28.017591+0000) 2019-10-13T14:28:28.246 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:28.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:28.244042+0000) 2019-10-13T14:28:28.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:28.779+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:28.782182+0000) 2019-10-13T14:28:30.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:30.130+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:30.135242+0000) 2019-10-13T14:28:31.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:31.263+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.268881+0000) 2019-10-13T14:28:31.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:31.267+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.269932+0000) 2019-10-13T14:28:31.273 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:31.266+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.270795+0000) 2019-10-13T14:28:31.273 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:31.267+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.271163+0000) 2019-10-13T14:28:31.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:31.267+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.271523+0000) 2019-10-13T14:28:31.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:31.267+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.272085+0000) 2019-10-13T14:28:31.275 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:31.267+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.272585+0000) 2019-10-13T14:28:31.279 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:31.274+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.276618+0000) 2019-10-13T14:28:31.281 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:31.275+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205431 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:31.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:31.763+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:31.765362+0000) 2019-10-13T14:28:32.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:32.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:32.033975+0000) 2019-10-13T14:28:34.365 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:34.359+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.362947+0000) 2019-10-13T14:28:34.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:34.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:34.648129+0000) 2019-10-13T14:28:36.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:36.279+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.282115+0000) 2019-10-13T14:28:36.285 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:36.279+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.283135+0000) 2019-10-13T14:28:36.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:36.279+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.283372+0000) 2019-10-13T14:28:36.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:36.282+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.284170+0000) 2019-10-13T14:28:36.287 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:36.279+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.284450+0000) 2019-10-13T14:28:36.287 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:36.279+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.284977+0000) 2019-10-13T14:28:36.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:36.279+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.285329+0000) 2019-10-13T14:28:36.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:36.286+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:36.291303+0000) 2019-10-13T14:28:36.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:36.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:36.922 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:28:36.922 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:36.926 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:28:36.926 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:28:38.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:38.015+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:38.017802+0000) 2019-10-13T14:28:38.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:38.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:38.244304+0000) 2019-10-13T14:28:38.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:38.779+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:38.782413+0000) 2019-10-13T14:28:40.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:40.138+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:40.141381+0000) 2019-10-13T14:28:41.301 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:41.295+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.298478+0000) 2019-10-13T14:28:41.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:41.295+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.299275+0000) 2019-10-13T14:28:41.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:41.295+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.300912+0000) 2019-10-13T14:28:41.304 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:41.298+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.301266+0000) 2019-10-13T14:28:41.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:41.299+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.302185+0000) 2019-10-13T14:28:41.305 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:41.299+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.302613+0000) 2019-10-13T14:28:41.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:41.299+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.302826+0000) 2019-10-13T14:28:41.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:41.303+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205494 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:41.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:41.306+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.311986+0000) 2019-10-13T14:28:41.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:41.763+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:41.765561+0000) 2019-10-13T14:28:42.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:42.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:42.034160+0000) 2019-10-13T14:28:44.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:44.359+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.363197+0000) 2019-10-13T14:28:44.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:44.643+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:44.648320+0000) 2019-10-13T14:28:46.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:46.307+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.311515+0000) 2019-10-13T14:28:46.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:46.307+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.311849+0000) 2019-10-13T14:28:46.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:46.307+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.312437+0000) 2019-10-13T14:28:46.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:46.307+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.312979+0000) 2019-10-13T14:28:46.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:46.307+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.313538+0000) 2019-10-13T14:28:46.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:46.310+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.314032+0000) 2019-10-13T14:28:46.317 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:46.311+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.314298+0000) 2019-10-13T14:28:46.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:46.314+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:46.318015+0000) 2019-10-13T14:28:46.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:46.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205524 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:48.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:48.015+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:48.018014+0000) 2019-10-13T14:28:48.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:48.239+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:48.244532+0000) 2019-10-13T14:28:48.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:48.779+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:48.782634+0000) 2019-10-13T14:28:50.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:50.146+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:50.147822+0000) 2019-10-13T14:28:51.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:51.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.323867+0000) 2019-10-13T14:28:51.327 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:51.319+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.324189+0000) 2019-10-13T14:28:51.328 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:51.323+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.326323+0000) 2019-10-13T14:28:51.329 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:51.323+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.326659+0000) 2019-10-13T14:28:51.329 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:51.322+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.327182+0000) 2019-10-13T14:28:51.330 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:51.323+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.327291+0000) 2019-10-13T14:28:51.330 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:51.323+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.327709+0000) 2019-10-13T14:28:51.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:51.330+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.332436+0000) 2019-10-13T14:28:51.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:51.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205554 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:51.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:51.763+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:51.765781+0000) 2019-10-13T14:28:52.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:52.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:52.034325+0000) 2019-10-13T14:28:54.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:54.363+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.363429+0000) 2019-10-13T14:28:54.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:54.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:54.648477+0000) 2019-10-13T14:28:56.340 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:56.331+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.337397+0000) 2019-10-13T14:28:56.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:28:56.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.337975+0000) 2019-10-13T14:28:56.341 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:28:56.335+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.338633+0000) 2019-10-13T14:28:56.341 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:56.335+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.339158+0000) 2019-10-13T14:28:56.342 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:28:56.335+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.339891+0000) 2019-10-13T14:28:56.343 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:56.335+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.340835+0000) 2019-10-13T14:28:56.343 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:28:56.338+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.341215+0000) 2019-10-13T14:28:56.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:28:56.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205585 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:28:56.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:28:56.342+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:56.347854+0000) 2019-10-13T14:28:58.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:28:58.014+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:58.018214+0000) 2019-10-13T14:28:58.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:28:58.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:58.244757+0000) 2019-10-13T14:28:58.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:28:58.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:28:58.782855+0000) 2019-10-13T14:29:00.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:00.150+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:00.153807+0000) 2019-10-13T14:29:01.354 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:01.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.351323+0000) 2019-10-13T14:29:01.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:01.346+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.352717+0000) 2019-10-13T14:29:01.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:01.350+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.355024+0000) 2019-10-13T14:29:01.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:01.350+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.357240+0000) 2019-10-13T14:29:01.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:01.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.359460+0000) 2019-10-13T14:29:01.363 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:01.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205617 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:01.364 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:01.354+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.361483+0000) 2019-10-13T14:29:01.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:01.358+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.362273+0000) 2019-10-13T14:29:01.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:01.366+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.369124+0000) 2019-10-13T14:29:01.768 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:01.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:01.766014+0000) 2019-10-13T14:29:02.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:02.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:02.034520+0000) 2019-10-13T14:29:04.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:04.358+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.363630+0000) 2019-10-13T14:29:04.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:04.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:04.648655+0000) 2019-10-13T14:29:06.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:06.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.364302+0000) 2019-10-13T14:29:06.368 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:06.358+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.365694+0000) 2019-10-13T14:29:06.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:06.362+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.366201+0000) 2019-10-13T14:29:06.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:06.362+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.366431+0000) 2019-10-13T14:29:06.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:06.362+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.368898+0000) 2019-10-13T14:29:06.371 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:06.362+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.369073+0000) 2019-10-13T14:29:06.372 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:06.366+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.370084+0000) 2019-10-13T14:29:06.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:06.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205648 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:06.380 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:06.374+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:06.377021+0000) 2019-10-13T14:29:07.019 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:29:07.020 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:07.023 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:29:07.023 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:08.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:08.014+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:08.018470+0000) 2019-10-13T14:29:08.247 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:08.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:08.244945+0000) 2019-10-13T14:29:08.785 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:08.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:08.783062+0000) 2019-10-13T14:29:10.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:10.158+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:10.160411+0000) 2019-10-13T14:29:11.380 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:11.374+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.378080+0000) 2019-10-13T14:29:11.382 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:11.374+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.380023+0000) 2019-10-13T14:29:11.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:11.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.380492+0000) 2019-10-13T14:29:11.383 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:11.374+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.380747+0000) 2019-10-13T14:29:11.384 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:11.378+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.381210+0000) 2019-10-13T14:29:11.384 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:11.374+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.381376+0000) 2019-10-13T14:29:11.384 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:11.378+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.382211+0000) 2019-10-13T14:29:11.388 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:11.382+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.385677+0000) 2019-10-13T14:29:11.390 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:11.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205678 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:11.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:11.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:11.766208+0000) 2019-10-13T14:29:12.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:12.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:12.034689+0000) 2019-10-13T14:29:14.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:14.358+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.363864+0000) 2019-10-13T14:29:14.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:14.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:14.648814+0000) 2019-10-13T14:29:16.394 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:16.386+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.391428+0000) 2019-10-13T14:29:16.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:16.386+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.392385+0000) 2019-10-13T14:29:16.395 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:16.386+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.392538+0000) 2019-10-13T14:29:16.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:16.386+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.392938+0000) 2019-10-13T14:29:16.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:16.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.394493+0000) 2019-10-13T14:29:16.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:16.390+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.394690+0000) 2019-10-13T14:29:16.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:16.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.395161+0000) 2019-10-13T14:29:16.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:16.398+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:16.400593+0000) 2019-10-13T14:29:16.404 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:16.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:18.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:18.018+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:18.018672+0000) 2019-10-13T14:29:18.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:18.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:18.245154+0000) 2019-10-13T14:29:18.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:18.782+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:18.783252+0000) 2019-10-13T14:29:20.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:20.162+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:20.167060+0000) 2019-10-13T14:29:21.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:21.398+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.405109+0000) 2019-10-13T14:29:21.408 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:21.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.406108+0000) 2019-10-13T14:29:21.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:21.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.405900+0000) 2019-10-13T14:29:21.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:21.402+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.406465+0000) 2019-10-13T14:29:21.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:21.402+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.406879+0000) 2019-10-13T14:29:21.410 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:21.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.407198+0000) 2019-10-13T14:29:21.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:21.402+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.409090+0000) 2019-10-13T14:29:21.417 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:21.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205740 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:21.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:21.410+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.414812+0000) 2019-10-13T14:29:21.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:21.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:21.766389+0000) 2019-10-13T14:29:22.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:22.030+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:22.034856+0000) 2019-10-13T14:29:24.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:24.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.364100+0000) 2019-10-13T14:29:24.651 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:24.642+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:24.648968+0000) 2019-10-13T14:29:26.420 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:26.413+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.417509+0000) 2019-10-13T14:29:26.420 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:26.414+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.418153+0000) 2019-10-13T14:29:26.423 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:26.414+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.421285+0000) 2019-10-13T14:29:26.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:26.418+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.423281+0000) 2019-10-13T14:29:26.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:26.418+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.425050+0000) 2019-10-13T14:29:26.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:26.421+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.425799+0000) 2019-10-13T14:29:26.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:26.422+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.426595+0000) 2019-10-13T14:29:26.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:26.422+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:26.427252+0000) 2019-10-13T14:29:26.430 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:26.422+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205769 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:28.021 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:28.018+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:28.018870+0000) 2019-10-13T14:29:28.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:28.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:28.245407+0000) 2019-10-13T14:29:28.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:28.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:28.783461+0000) 2019-10-13T14:29:30.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:30.169+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:30.173676+0000) 2019-10-13T14:29:31.434 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:31.426+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.431279+0000) 2019-10-13T14:29:31.437 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:31.430+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.433944+0000) 2019-10-13T14:29:31.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:31.430+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.434237+0000) 2019-10-13T14:29:31.438 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:31.430+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.436286+0000) 2019-10-13T14:29:31.441 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:31.433+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.438419+0000) 2019-10-13T14:29:31.444 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:31.438+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205800 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:31.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:31.438+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.442180+0000) 2019-10-13T14:29:31.445 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:31.438+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.442626+0000) 2019-10-13T14:29:31.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:31.441+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.446478+0000) 2019-10-13T14:29:31.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:31.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:31.766581+0000) 2019-10-13T14:29:32.037 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:32.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:32.035025+0000) 2019-10-13T14:29:34.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:34.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.364336+0000) 2019-10-13T14:29:34.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:34.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:34.649203+0000) 2019-10-13T14:29:36.447 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:36.441+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.445149+0000) 2019-10-13T14:29:36.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:36.438+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.445570+0000) 2019-10-13T14:29:36.449 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:36.442+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.447200+0000) 2019-10-13T14:29:36.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:36.442+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.447628+0000) 2019-10-13T14:29:36.450 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:36.442+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.447962+0000) 2019-10-13T14:29:36.451 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:36.442+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.448534+0000) 2019-10-13T14:29:36.451 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:36.442+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.449082+0000) 2019-10-13T14:29:36.455 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:36.449+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:36.452665+0000) 2019-10-13T14:29:36.458 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:36.450+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205829 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:37.143 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:29:37.143 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:37.146 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:29:37.147 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:29:38.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:38.018+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:38.019070+0000) 2019-10-13T14:29:38.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:38.238+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:38.245609+0000) 2019-10-13T14:29:38.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:38.782+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:38.783656+0000) 2019-10-13T14:29:40.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:40.177+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:40.180448+0000) 2019-10-13T14:29:41.462 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:41.454+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.459584+0000) 2019-10-13T14:29:41.463 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:41.454+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.460255+0000) 2019-10-13T14:29:41.463 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:41.454+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.461020+0000) 2019-10-13T14:29:41.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:41.454+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.461497+0000) 2019-10-13T14:29:41.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:41.458+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.462216+0000) 2019-10-13T14:29:41.465 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:41.457+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.462674+0000) 2019-10-13T14:29:41.465 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:41.458+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.463337+0000) 2019-10-13T14:29:41.470 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:41.465+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.468263+0000) 2019-10-13T14:29:41.472 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:41.462+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205862 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:41.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:41.762+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:41.766724+0000) 2019-10-13T14:29:42.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:42.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:42.035212+0000) 2019-10-13T14:29:44.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:44.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.364552+0000) 2019-10-13T14:29:44.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:44.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:44.649401+0000) 2019-10-13T14:29:46.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:46.470+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.473727+0000) 2019-10-13T14:29:46.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:46.470+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.474155+0000) 2019-10-13T14:29:46.477 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:46.470+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.474336+0000) 2019-10-13T14:29:46.477 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:46.470+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.474543+0000) 2019-10-13T14:29:46.478 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:46.473+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.476142+0000) 2019-10-13T14:29:46.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:46.470+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.476420+0000) 2019-10-13T14:29:46.479 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:46.470+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.476740+0000) 2019-10-13T14:29:46.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:46.473+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:46.479441+0000) 2019-10-13T14:29:46.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:46.478+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205891 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:48.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:48.014+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:48.019303+0000) 2019-10-13T14:29:48.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:48.242+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:48.245827+0000) 2019-10-13T14:29:48.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:48.778+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:48.783850+0000) 2019-10-13T14:29:50.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:50.181+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:50.186972+0000) 2019-10-13T14:29:51.489 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:51.482+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.485903+0000) 2019-10-13T14:29:51.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:51.485+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.488638+0000) 2019-10-13T14:29:51.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:51.482+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.489012+0000) 2019-10-13T14:29:51.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:51.482+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.489290+0000) 2019-10-13T14:29:51.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:51.486+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.490126+0000) 2019-10-13T14:29:51.493 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:51.486+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.489809+0000) 2019-10-13T14:29:51.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:51.486+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.490708+0000) 2019-10-13T14:29:51.497 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:51.489+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.494554+0000) 2019-10-13T14:29:51.499 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:51.490+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:51.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:51.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:51.766945+0000) 2019-10-13T14:29:52.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:52.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:52.035379+0000) 2019-10-13T14:29:54.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:54.361+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.364748+0000) 2019-10-13T14:29:54.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:54.645+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:54.649618+0000) 2019-10-13T14:29:56.502 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:29:56.493+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.500139+0000) 2019-10-13T14:29:56.503 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:29:56.493+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.500465+0000) 2019-10-13T14:29:56.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:56.493+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.500889+0000) 2019-10-13T14:29:56.504 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:56.493+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.501193+0000) 2019-10-13T14:29:56.504 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:29:56.497+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.502218+0000) 2019-10-13T14:29:56.505 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:56.497+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.502447+0000) 2019-10-13T14:29:56.505 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:29:56.497+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.503109+0000) 2019-10-13T14:29:56.510 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:29:56.505+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:56.508364+0000) 2019-10-13T14:29:56.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:29:56.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205951 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:29:58.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:29:58.013+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:58.019506+0000) 2019-10-13T14:29:58.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:29:58.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:58.246031+0000) 2019-10-13T14:29:58.786 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:29:58.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:29:58.784044+0000) 2019-10-13T14:30:00.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:00.189+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:00.193388+0000) 2019-10-13T14:30:01.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:01.509+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.513240+0000) 2019-10-13T14:30:01.517 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:01.509+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.515301+0000) 2019-10-13T14:30:01.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:01.509+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.516135+0000) 2019-10-13T14:30:01.519 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:01.509+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.516479+0000) 2019-10-13T14:30:01.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:01.513+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.518193+0000) 2019-10-13T14:30:01.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:01.513+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.520368+0000) 2019-10-13T14:30:01.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:01.517+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.521194+0000) 2019-10-13T14:30:01.524 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:01.517+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.522001+0000) 2019-10-13T14:30:01.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:01.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205984 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:01.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:01.765+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:01.767120+0000) 2019-10-13T14:30:02.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:02.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:02.035584+0000) 2019-10-13T14:30:04.367 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:04.361+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.364941+0000) 2019-10-13T14:30:04.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:04.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:04.649840+0000) 2019-10-13T14:30:06.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:06.521+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.527245+0000) 2019-10-13T14:30:06.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:06.521+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.527641+0000) 2019-10-13T14:30:06.530 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:06.521+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.528158+0000) 2019-10-13T14:30:06.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:06.521+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.528666+0000) 2019-10-13T14:30:06.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:06.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.530191+0000) 2019-10-13T14:30:06.533 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:06.525+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.530429+0000) 2019-10-13T14:30:06.533 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:06.525+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.530978+0000) 2019-10-13T14:30:06.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:06.533+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:06.536705+0000) 2019-10-13T14:30:06.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:06.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:07.282 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:30:07.283 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:07.286 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:30:07.286 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:08.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:08.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:08.019729+0000) 2019-10-13T14:30:08.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:08.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:08.246232+0000) 2019-10-13T14:30:08.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:08.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:08.784273+0000) 2019-10-13T14:30:10.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:10.201+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:10.204514+0000) 2019-10-13T14:30:11.543 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:11.537+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.541076+0000) 2019-10-13T14:30:11.544 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:11.537+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.541702+0000) 2019-10-13T14:30:11.545 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:11.541+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.542599+0000) 2019-10-13T14:30:11.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:11.537+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.543104+0000) 2019-10-13T14:30:11.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:11.541+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.543835+0000) 2019-10-13T14:30:11.546 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:11.541+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.544270+0000) 2019-10-13T14:30:11.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:11.541+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.544824+0000) 2019-10-13T14:30:11.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:11.545+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.549390+0000) 2019-10-13T14:30:11.553 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:11.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206044 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:11.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:11.765+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:11.767300+0000) 2019-10-13T14:30:12.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:12.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:12.035761+0000) 2019-10-13T14:30:14.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:14.361+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.365136+0000) 2019-10-13T14:30:14.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:14.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:14.650075+0000) 2019-10-13T14:30:16.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:16.553+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.554693+0000) 2019-10-13T14:30:16.558 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:16.553+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.555223+0000) 2019-10-13T14:30:16.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:16.553+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.556282+0000) 2019-10-13T14:30:16.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:16.553+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.557228+0000) 2019-10-13T14:30:16.560 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:16.557+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.557792+0000) 2019-10-13T14:30:16.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:16.557+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.557949+0000) 2019-10-13T14:30:16.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:16.553+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.558448+0000) 2019-10-13T14:30:16.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:16.561+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:16.564841+0000) 2019-10-13T14:30:16.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:16.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:18.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:18.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:18.019927+0000) 2019-10-13T14:30:18.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:18.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:18.246435+0000) 2019-10-13T14:30:18.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:18.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:18.784462+0000) 2019-10-13T14:30:20.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:20.209+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:20.211070+0000) 2019-10-13T14:30:21.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:21.565+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.569449+0000) 2019-10-13T14:30:21.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:21.569+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.570646+0000) 2019-10-13T14:30:21.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:21.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.572264+0000) 2019-10-13T14:30:21.575 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:21.569+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.572800+0000) 2019-10-13T14:30:21.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:21.569+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.573179+0000) 2019-10-13T14:30:21.576 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:21.573+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.573730+0000) 2019-10-13T14:30:21.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:21.569+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.574087+0000) 2019-10-13T14:30:21.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:21.573+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.576434+0000) 2019-10-13T14:30:21.581 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:21.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:21.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:21.765+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:21.767513+0000) 2019-10-13T14:30:22.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:22.029+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:22.035944+0000) 2019-10-13T14:30:24.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:24.361+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.365337+0000) 2019-10-13T14:30:24.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:24.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:24.650268+0000) 2019-10-13T14:30:26.585 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:26.581+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.582347+0000) 2019-10-13T14:30:26.585 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:26.581+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.582952+0000) 2019-10-13T14:30:26.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:26.577+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.583401+0000) 2019-10-13T14:30:26.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:26.581+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.583736+0000) 2019-10-13T14:30:26.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:26.581+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.584079+0000) 2019-10-13T14:30:26.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:26.581+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.585142+0000) 2019-10-13T14:30:26.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:26.585+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.585794+0000) 2019-10-13T14:30:26.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:26.585+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:26.591050+0000) 2019-10-13T14:30:26.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:26.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206135 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:28.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:28.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:28.020124+0000) 2019-10-13T14:30:28.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:28.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:28.246608+0000) 2019-10-13T14:30:28.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:28.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:28.784652+0000) 2019-10-13T14:30:30.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:30.216+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:30.217648+0000) 2019-10-13T14:30:31.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:31.593+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.595885+0000) 2019-10-13T14:30:31.599 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:31.593+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.597061+0000) 2019-10-13T14:30:31.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:31.597+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.598263+0000) 2019-10-13T14:30:31.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:31.597+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.598566+0000) 2019-10-13T14:30:31.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:31.592+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.599084+0000) 2019-10-13T14:30:31.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:31.597+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.599624+0000) 2019-10-13T14:30:31.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:31.597+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.600430+0000) 2019-10-13T14:30:31.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:31.596+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.603761+0000) 2019-10-13T14:30:31.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:31.605+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206166 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:31.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:31.765+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:31.767689+0000) 2019-10-13T14:30:32.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:32.032+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:32.036080+0000) 2019-10-13T14:30:34.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:34.365+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.365573+0000) 2019-10-13T14:30:34.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:34.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:34.650464+0000) 2019-10-13T14:30:36.612 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:36.605+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.609648+0000) 2019-10-13T14:30:36.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:36.609+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.611674+0000) 2019-10-13T14:30:36.614 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:36.609+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.611933+0000) 2019-10-13T14:30:36.615 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:36.609+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.612536+0000) 2019-10-13T14:30:36.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:36.609+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.613122+0000) 2019-10-13T14:30:36.616 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:36.608+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.613568+0000) 2019-10-13T14:30:36.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:36.613+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.613845+0000) 2019-10-13T14:30:36.620 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:36.612+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:36.617633+0000) 2019-10-13T14:30:36.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:36.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:37.406 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:30:37.406 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:37.410 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:30:37.410 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:30:38.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:38.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:38.020281+0000) 2019-10-13T14:30:38.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:38.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:38.246809+0000) 2019-10-13T14:30:38.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:38.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:38.784864+0000) 2019-10-13T14:30:40.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:40.220+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:40.224155+0000) 2019-10-13T14:30:41.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:41.621+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.625198+0000) 2019-10-13T14:30:41.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:41.621+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.625609+0000) 2019-10-13T14:30:41.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:41.625+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.627134+0000) 2019-10-13T14:30:41.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:41.625+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.627672+0000) 2019-10-13T14:30:41.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:41.629+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.630007+0000) 2019-10-13T14:30:41.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:41.624+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.630400+0000) 2019-10-13T14:30:41.633 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:41.629+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.630453+0000) 2019-10-13T14:30:41.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:41.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206228 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:41.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:41.632+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.636780+0000) 2019-10-13T14:30:41.770 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:41.765+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:41.767910+0000) 2019-10-13T14:30:42.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:42.032+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:42.036343+0000) 2019-10-13T14:30:44.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:44.365+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.365778+0000) 2019-10-13T14:30:44.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:44.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:44.650663+0000) 2019-10-13T14:30:46.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:46.633+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.637397+0000) 2019-10-13T14:30:46.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:46.632+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.638075+0000) 2019-10-13T14:30:46.641 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:46.637+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.637798+0000) 2019-10-13T14:30:46.641 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:46.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.638806+0000) 2019-10-13T14:30:46.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:46.637+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.639296+0000) 2019-10-13T14:30:46.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:46.637+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.639805+0000) 2019-10-13T14:30:46.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:46.637+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.640282+0000) 2019-10-13T14:30:46.649 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:46.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206257 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:46.651 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:46.644+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:46.648898+0000) 2019-10-13T14:30:48.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:48.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:48.020507+0000) 2019-10-13T14:30:48.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:48.245+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:48.246963+0000) 2019-10-13T14:30:48.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:48.781+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:48.785086+0000) 2019-10-13T14:30:50.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:50.228+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:50.230875+0000) 2019-10-13T14:30:51.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:51.648+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.652103+0000) 2019-10-13T14:30:51.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:51.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.652918+0000) 2019-10-13T14:30:51.656 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:51.648+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.653333+0000) 2019-10-13T14:30:51.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:51.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.655594+0000) 2019-10-13T14:30:51.658 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:51.648+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.655734+0000) 2019-10-13T14:30:51.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:51.652+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.657326+0000) 2019-10-13T14:30:51.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:51.656+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.658521+0000) 2019-10-13T14:30:51.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:51.656+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206288 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:51.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:51.656+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.661587+0000) 2019-10-13T14:30:51.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:51.764+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:51.768088+0000) 2019-10-13T14:30:52.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:52.032+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:52.036533+0000) 2019-10-13T14:30:54.368 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:54.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.365978+0000) 2019-10-13T14:30:54.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:54.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:54.650894+0000) 2019-10-13T14:30:56.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:30:56.660+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.663843+0000) 2019-10-13T14:30:56.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:56.660+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.663974+0000) 2019-10-13T14:30:56.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:56.660+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.665318+0000) 2019-10-13T14:30:56.668 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:30:56.664+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.665974+0000) 2019-10-13T14:30:56.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:30:56.664+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.666337+0000) 2019-10-13T14:30:56.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:56.664+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.666677+0000) 2019-10-13T14:30:56.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:30:56.664+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.666929+0000) 2019-10-13T14:30:56.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:30:56.668+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:56.672964+0000) 2019-10-13T14:30:56.676 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:30:56.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:30:58.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:30:58.016+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:58.020727+0000) 2019-10-13T14:30:58.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:30:58.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:58.247150+0000) 2019-10-13T14:30:58.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:30:58.780+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:30:58.785276+0000) 2019-10-13T14:31:00.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:00.236+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:00.237428+0000) 2019-10-13T14:31:01.679 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:01.672+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.676318+0000) 2019-10-13T14:31:01.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:01.672+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.676775+0000) 2019-10-13T14:31:01.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:01.672+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.678239+0000) 2019-10-13T14:31:01.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:01.676+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.680953+0000) 2019-10-13T14:31:01.684 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:01.676+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.681341+0000) 2019-10-13T14:31:01.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:01.680+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.681950+0000) 2019-10-13T14:31:01.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:01.680+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.683141+0000) 2019-10-13T14:31:01.687 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:01.680+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.684589+0000) 2019-10-13T14:31:01.690 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:01.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206350 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:01.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:01.764+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:01.768297+0000) 2019-10-13T14:31:02.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:02.032+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:02.036727+0000) 2019-10-13T14:31:04.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:04.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.366179+0000) 2019-10-13T14:31:04.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:04.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:04.651120+0000) 2019-10-13T14:31:06.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:06.688+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.691167+0000) 2019-10-13T14:31:06.696 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:06.688+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.693347+0000) 2019-10-13T14:31:06.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:06.688+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.691590+0000) 2019-10-13T14:31:06.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:06.688+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.692372+0000) 2019-10-13T14:31:06.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:06.688+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.692917+0000) 2019-10-13T14:31:06.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:06.692+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.694526+0000) 2019-10-13T14:31:06.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:06.692+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.695600+0000) 2019-10-13T14:31:06.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:06.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:06.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:06.696+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:06.701431+0000) 2019-10-13T14:31:07.488 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:31:07.488 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:07.491 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:31:07.492 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:08.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:08.016+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:08.020924+0000) 2019-10-13T14:31:08.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:08.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:08.247363+0000) 2019-10-13T14:31:08.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:08.780+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:08.785474+0000) 2019-10-13T14:31:10.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:10.240+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:10.243866+0000) 2019-10-13T14:31:11.707 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:11.700+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.704316+0000) 2019-10-13T14:31:11.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:11.700+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.704802+0000) 2019-10-13T14:31:11.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:11.704+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.705921+0000) 2019-10-13T14:31:11.709 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:11.704+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.706607+0000) 2019-10-13T14:31:11.709 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:11.704+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.706847+0000) 2019-10-13T14:31:11.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:11.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.708233+0000) 2019-10-13T14:31:11.711 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:11.704+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.708821+0000) 2019-10-13T14:31:11.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:11.708+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.713609+0000) 2019-10-13T14:31:11.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:11.712+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206410 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:11.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:11.764+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:11.768479+0000) 2019-10-13T14:31:12.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:12.036+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:12.036897+0000) 2019-10-13T14:31:14.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:14.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.366387+0000) 2019-10-13T14:31:14.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:14.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:14.651321+0000) 2019-10-13T14:31:16.720 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:16.716+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.717971+0000) 2019-10-13T14:31:16.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:16.716+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.718949+0000) 2019-10-13T14:31:16.721 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:16.716+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.719181+0000) 2019-10-13T14:31:16.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:16.716+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.719592+0000) 2019-10-13T14:31:16.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:16.716+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.720150+0000) 2019-10-13T14:31:16.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:16.716+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.720895+0000) 2019-10-13T14:31:16.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:16.716+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.723372+0000) 2019-10-13T14:31:16.729 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:16.720+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:16.726821+0000) 2019-10-13T14:31:16.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:16.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:18.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:18.020+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:18.021156+0000) 2019-10-13T14:31:18.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:18.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:18.247577+0000) 2019-10-13T14:31:18.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:18.784+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:18.785685+0000) 2019-10-13T14:31:20.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:20.248+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:20.249956+0000) 2019-10-13T14:31:21.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:21.728+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.731696+0000) 2019-10-13T14:31:21.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:21.728+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.732107+0000) 2019-10-13T14:31:21.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:21.728+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.732601+0000) 2019-10-13T14:31:21.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:21.728+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.733363+0000) 2019-10-13T14:31:21.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:21.732+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.735413+0000) 2019-10-13T14:31:21.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:21.732+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.737397+0000) 2019-10-13T14:31:21.741 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:21.732+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.739116+0000) 2019-10-13T14:31:21.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:21.736+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.739054+0000) 2019-10-13T14:31:21.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:21.736+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206472 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:21.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:21.764+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:21.768669+0000) 2019-10-13T14:31:22.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:22.036+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:22.037106+0000) 2019-10-13T14:31:24.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:24.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.366588+0000) 2019-10-13T14:31:24.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:24.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:24.651519+0000) 2019-10-13T14:31:26.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:26.740+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.744468+0000) 2019-10-13T14:31:26.747 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:26.740+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.745385+0000) 2019-10-13T14:31:26.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:26.740+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.745620+0000) 2019-10-13T14:31:26.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:26.744+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.746203+0000) 2019-10-13T14:31:26.749 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:26.744+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.746953+0000) 2019-10-13T14:31:26.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:26.744+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.747293+0000) 2019-10-13T14:31:26.750 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:26.744+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.747773+0000) 2019-10-13T14:31:26.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:26.748+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:26.753087+0000) 2019-10-13T14:31:26.758 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:26.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:28.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:28.016+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:28.021360+0000) 2019-10-13T14:31:28.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:28.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:28.247766+0000) 2019-10-13T14:31:28.788 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:28.784+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:28.785903+0000) 2019-10-13T14:31:30.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:30.256+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:30.256363+0000) 2019-10-13T14:31:31.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:31.756+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.759357+0000) 2019-10-13T14:31:31.762 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:31.756+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.759663+0000) 2019-10-13T14:31:31.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:31.756+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.760206+0000) 2019-10-13T14:31:31.765 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:31.760+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.762793+0000) 2019-10-13T14:31:31.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:31.760+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.764306+0000) 2019-10-13T14:31:31.767 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:31.764+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.764826+0000) 2019-10-13T14:31:31.769 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:31.764+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.766812+0000) 2019-10-13T14:31:31.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:31.764+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.768821+0000) 2019-10-13T14:31:31.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:31.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:31.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:31.768+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:31.771796+0000) 2019-10-13T14:31:32.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:32.036+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:32.037277+0000) 2019-10-13T14:31:34.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:34.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.366821+0000) 2019-10-13T14:31:34.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:34.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:34.651720+0000) 2019-10-13T14:31:36.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:36.772+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.773823+0000) 2019-10-13T14:31:36.777 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:36.772+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.774982+0000) 2019-10-13T14:31:36.778 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:36.772+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.775501+0000) 2019-10-13T14:31:36.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:36.771+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.776033+0000) 2019-10-13T14:31:36.779 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:36.772+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.775522+0000) 2019-10-13T14:31:36.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:36.772+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.776691+0000) 2019-10-13T14:31:36.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:36.776+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.778830+0000) 2019-10-13T14:31:36.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:36.780+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:36.790 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:36.783+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:36.788071+0000) 2019-10-13T14:31:37.626 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:31:37.626 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:37.630 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:31:37.630 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:31:38.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:38.020+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:38.021592+0000) 2019-10-13T14:31:38.250 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:38.244+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:38.247922+0000) 2019-10-13T14:31:38.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:38.784+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:38.786132+0000) 2019-10-13T14:31:40.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:40.259+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:40.262519+0000) 2019-10-13T14:31:41.771 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:41.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.769009+0000) 2019-10-13T14:31:41.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:41.784+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.787676+0000) 2019-10-13T14:31:41.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:41.784+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.788293+0000) 2019-10-13T14:31:41.792 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:41.788+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.789974+0000) 2019-10-13T14:31:41.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:41.791+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.792403+0000) 2019-10-13T14:31:41.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:41.788+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.792540+0000) 2019-10-13T14:31:41.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:41.788+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.793221+0000) 2019-10-13T14:31:41.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:41.792+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.794739+0000) 2019-10-13T14:31:41.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:41.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:41.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:41.799+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:41.800562+0000) 2019-10-13T14:31:42.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:42.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:42.037505+0000) 2019-10-13T14:31:44.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:44.364+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.367068+0000) 2019-10-13T14:31:44.654 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:44.648+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:44.651974+0000) 2019-10-13T14:31:46.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:46.795+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.800967+0000) 2019-10-13T14:31:46.804 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:46.795+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.801502+0000) 2019-10-13T14:31:46.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:46.799+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.802156+0000) 2019-10-13T14:31:46.805 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:46.799+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.802642+0000) 2019-10-13T14:31:46.805 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:46.799+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.803032+0000) 2019-10-13T14:31:46.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:46.799+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.803428+0000) 2019-10-13T14:31:46.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:46.799+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.803940+0000) 2019-10-13T14:31:46.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:46.807+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:46.810905+0000) 2019-10-13T14:31:46.813 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:46.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:48.024 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:48.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:48.021823+0000) 2019-10-13T14:31:48.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:48.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:48.248108+0000) 2019-10-13T14:31:48.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:48.783+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:48.786330+0000) 2019-10-13T14:31:50.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:50.267+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:50.269190+0000) 2019-10-13T14:31:51.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:51.763+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.769209+0000) 2019-10-13T14:31:51.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:51.811+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.814434+0000) 2019-10-13T14:31:51.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:51.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.814843+0000) 2019-10-13T14:31:51.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:51.811+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.815668+0000) 2019-10-13T14:31:51.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:51.815+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.816242+0000) 2019-10-13T14:31:51.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:51.811+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.816771+0000) 2019-10-13T14:31:51.820 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:51.811+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.817187+0000) 2019-10-13T14:31:51.820 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:51.811+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.817526+0000) 2019-10-13T14:31:51.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:51.819+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:51.822849+0000) 2019-10-13T14:31:51.827 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:51.819+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206653 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:52.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:52.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:52.037707+0000) 2019-10-13T14:31:54.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:54.363+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.367272+0000) 2019-10-13T14:31:54.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:54.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:54.652171+0000) 2019-10-13T14:31:56.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:31:56.823+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.828380+0000) 2019-10-13T14:31:56.833 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:56.827+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.831050+0000) 2019-10-13T14:31:56.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:31:56.827+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.831276+0000) 2019-10-13T14:31:56.834 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:31:56.831+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.832403+0000) 2019-10-13T14:31:56.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:56.827+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.832755+0000) 2019-10-13T14:31:56.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:31:56.827+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.833451+0000) 2019-10-13T14:31:56.836 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:56.831+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.833794+0000) 2019-10-13T14:31:56.839 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:31:56.835+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:56.837026+0000) 2019-10-13T14:31:56.841 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:31:56.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:31:58.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:31:58.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:58.022025+0000) 2019-10-13T14:31:58.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:31:58.243+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:58.248312+0000) 2019-10-13T14:31:58.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:31:58.783+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:31:58.786520+0000) 2019-10-13T14:32:00.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:00.275+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:00.275896+0000) 2019-10-13T14:32:01.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:01.763+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.769435+0000) 2019-10-13T14:32:01.845 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:01.839+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.842251+0000) 2019-10-13T14:32:01.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:01.839+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.842468+0000) 2019-10-13T14:32:01.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:01.839+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.843210+0000) 2019-10-13T14:32:01.846 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:01.839+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.843948+0000) 2019-10-13T14:32:01.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:01.839+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.844525+0000) 2019-10-13T14:32:01.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:01.839+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.845359+0000) 2019-10-13T14:32:01.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:01.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.844947+0000) 2019-10-13T14:32:01.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:01.851+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:01.852556+0000) 2019-10-13T14:32:01.855 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:01.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206715 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:02.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:02.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:02.037875+0000) 2019-10-13T14:32:04.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:04.363+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.367491+0000) 2019-10-13T14:32:04.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:04.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:04.652370+0000) 2019-10-13T14:32:06.859 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:06.851+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.856498+0000) 2019-10-13T14:32:06.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:06.851+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.857224+0000) 2019-10-13T14:32:06.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:06.851+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.857462+0000) 2019-10-13T14:32:06.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:06.855+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.858467+0000) 2019-10-13T14:32:06.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:06.855+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.859654+0000) 2019-10-13T14:32:06.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:06.855+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.859737+0000) 2019-10-13T14:32:06.863 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:06.855+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.860134+0000) 2019-10-13T14:32:06.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:06.863+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:06.864765+0000) 2019-10-13T14:32:06.869 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:06.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:07.756 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:32:07.757 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:07.760 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:32:07.761 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:08.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:08.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:08.022228+0000) 2019-10-13T14:32:08.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:08.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:08.248515+0000) 2019-10-13T14:32:08.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:08.783+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:08.786734+0000) 2019-10-13T14:32:10.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:10.279+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:10.282434+0000) 2019-10-13T14:32:11.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:11.767+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.769642+0000) 2019-10-13T14:32:11.872 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:11.867+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.869990+0000) 2019-10-13T14:32:11.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:11.867+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.871146+0000) 2019-10-13T14:32:11.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:11.867+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.871814+0000) 2019-10-13T14:32:11.875 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:11.867+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.872300+0000) 2019-10-13T14:32:11.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:11.871+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.872694+0000) 2019-10-13T14:32:11.876 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:11.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.873337+0000) 2019-10-13T14:32:11.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:11.867+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.873510+0000) 2019-10-13T14:32:11.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:11.875+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:11.876925+0000) 2019-10-13T14:32:11.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:11.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206775 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:12.040 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:12.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:12.038046+0000) 2019-10-13T14:32:14.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:14.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.367719+0000) 2019-10-13T14:32:14.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:14.647+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:14.652601+0000) 2019-10-13T14:32:16.885 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:16.879+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.882932+0000) 2019-10-13T14:32:16.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:16.879+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.883498+0000) 2019-10-13T14:32:16.886 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:16.879+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.884016+0000) 2019-10-13T14:32:16.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:16.883+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.884490+0000) 2019-10-13T14:32:16.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:16.879+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.885258+0000) 2019-10-13T14:32:16.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:16.879+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.885609+0000) 2019-10-13T14:32:16.889 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:16.883+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.886565+0000) 2019-10-13T14:32:16.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:16.891+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:16.892912+0000) 2019-10-13T14:32:16.895 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:16.887+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:18.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:18.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:18.022437+0000) 2019-10-13T14:32:18.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:18.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:18.248730+0000) 2019-10-13T14:32:18.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:18.783+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:18.786954+0000) 2019-10-13T14:32:20.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:20.291+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:20.293910+0000) 2019-10-13T14:32:21.772 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:21.767+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.769848+0000) 2019-10-13T14:32:21.898 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:21.891+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.895905+0000) 2019-10-13T14:32:21.899 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:21.891+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.896527+0000) 2019-10-13T14:32:21.899 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:21.891+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.896809+0000) 2019-10-13T14:32:21.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:21.895+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.898338+0000) 2019-10-13T14:32:21.901 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:21.895+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.898976+0000) 2019-10-13T14:32:21.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:21.895+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.899246+0000) 2019-10-13T14:32:21.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:21.895+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.898681+0000) 2019-10-13T14:32:21.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:21.903+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:21.905864+0000) 2019-10-13T14:32:21.909 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:21.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206837 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:22.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:22.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:22.038219+0000) 2019-10-13T14:32:24.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:24.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.367879+0000) 2019-10-13T14:32:24.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:24.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:24.652834+0000) 2019-10-13T14:32:26.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:26.903+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.909465+0000) 2019-10-13T14:32:26.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:26.907+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.910379+0000) 2019-10-13T14:32:26.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:26.907+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.910920+0000) 2019-10-13T14:32:26.914 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:26.907+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.911796+0000) 2019-10-13T14:32:26.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:26.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.912574+0000) 2019-10-13T14:32:26.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:26.907+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.912661+0000) 2019-10-13T14:32:26.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:26.907+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.912874+0000) 2019-10-13T14:32:26.920 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:26.915+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:26.918017+0000) 2019-10-13T14:32:26.923 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:26.915+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:28.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:28.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:28.022642+0000) 2019-10-13T14:32:28.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:28.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:28.248924+0000) 2019-10-13T14:32:28.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:28.783+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:28.787148+0000) 2019-10-13T14:32:30.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:30.299+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:30.300365+0000) 2019-10-13T14:32:31.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:31.767+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.770082+0000) 2019-10-13T14:32:31.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:31.919+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.923987+0000) 2019-10-13T14:32:31.927 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:31.919+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.924944+0000) 2019-10-13T14:32:31.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:31.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.926415+0000) 2019-10-13T14:32:31.929 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:31.923+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.926689+0000) 2019-10-13T14:32:31.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:31.923+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.928542+0000) 2019-10-13T14:32:31.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:31.923+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.929338+0000) 2019-10-13T14:32:31.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:31.923+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.929662+0000) 2019-10-13T14:32:31.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:31.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206897 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:31.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:31.931+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:31.935068+0000) 2019-10-13T14:32:32.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:32.035+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:32.038432+0000) 2019-10-13T14:32:34.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:34.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.368113+0000) 2019-10-13T14:32:34.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:34.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:34.653031+0000) 2019-10-13T14:32:36.940 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:36.935+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.938056+0000) 2019-10-13T14:32:36.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:36.935+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.938381+0000) 2019-10-13T14:32:36.942 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:36.935+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.938492+0000) 2019-10-13T14:32:36.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:36.935+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.939009+0000) 2019-10-13T14:32:36.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:36.935+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.940055+0000) 2019-10-13T14:32:36.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:36.935+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.940526+0000) 2019-10-13T14:32:36.943 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:36.935+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.940598+0000) 2019-10-13T14:32:36.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:36.943+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:36.945951+0000) 2019-10-13T14:32:36.951 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:36.943+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206926 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:37.878 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:32:37.878 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:37.882 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:32:37.882 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:32:38.025 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:38.019+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:38.022844+0000) 2019-10-13T14:32:38.251 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:38.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:38.249118+0000) 2019-10-13T14:32:38.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:38.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:38.787345+0000) 2019-10-13T14:32:40.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:40.306+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:40.308434+0000) 2019-10-13T14:32:41.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:41.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.770319+0000) 2019-10-13T14:32:41.956 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:41.946+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.953138+0000) 2019-10-13T14:32:41.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:41.950+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.954001+0000) 2019-10-13T14:32:41.957 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:41.950+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.954501+0000) 2019-10-13T14:32:41.959 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:41.950+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.956467+0000) 2019-10-13T14:32:41.961 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:41.954+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.958526+0000) 2019-10-13T14:32:41.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:41.954+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.960199+0000) 2019-10-13T14:32:41.964 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:41.958+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.961718+0000) 2019-10-13T14:32:41.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:41.958+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:41.961946+0000) 2019-10-13T14:32:41.965 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:41.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206959 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:42.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:42.034+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:42.038560+0000) 2019-10-13T14:32:44.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:44.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.368331+0000) 2019-10-13T14:32:44.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:44.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:44.653253+0000) 2019-10-13T14:32:46.968 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:46.962+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.965878+0000) 2019-10-13T14:32:46.969 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:46.962+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.966499+0000) 2019-10-13T14:32:46.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:46.962+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.967115+0000) 2019-10-13T14:32:46.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:46.962+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.967427+0000) 2019-10-13T14:32:46.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:46.966+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.968410+0000) 2019-10-13T14:32:46.972 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:46.962+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.969362+0000) 2019-10-13T14:32:46.972 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:46.962+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.969527+0000) 2019-10-13T14:32:46.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:46.970+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206988 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:46.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:46.974+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:46.977075+0000) 2019-10-13T14:32:48.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:48.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:48.023072+0000) 2019-10-13T14:32:48.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:48.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:48.249340+0000) 2019-10-13T14:32:48.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:48.786+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:48.787563+0000) 2019-10-13T14:32:50.317 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:50.310+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:50.315087+0000) 2019-10-13T14:32:51.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:51.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.770520+0000) 2019-10-13T14:32:51.983 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:51.974+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.980320+0000) 2019-10-13T14:32:51.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:51.974+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.980567+0000) 2019-10-13T14:32:51.984 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:51.974+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.981592+0000) 2019-10-13T14:32:51.984 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:51.978+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.981917+0000) 2019-10-13T14:32:51.985 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:51.978+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.982325+0000) 2019-10-13T14:32:51.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:51.978+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.982878+0000) 2019-10-13T14:32:51.985 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:51.978+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.983307+0000) 2019-10-13T14:32:51.990 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:51.982+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:51.987958+0000) 2019-10-13T14:32:51.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:51.982+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207019 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:52.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:52.034+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:52.038769+0000) 2019-10-13T14:32:54.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:54.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.368532+0000) 2019-10-13T14:32:54.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:54.646+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:54.653454+0000) 2019-10-13T14:32:56.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:32:56.986+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.992605+0000) 2019-10-13T14:32:56.996 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:56.986+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.993231+0000) 2019-10-13T14:32:56.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:32:56.990+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.994365+0000) 2019-10-13T14:32:56.997 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:56.990+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.994514+0000) 2019-10-13T14:32:56.997 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:56.990+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.995045+0000) 2019-10-13T14:32:56.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:32:56.990+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.995918+0000) 2019-10-13T14:32:56.999 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:32:56.990+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:56.994768+0000) 2019-10-13T14:32:57.002 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:32:56.994+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:57.000078+0000) 2019-10-13T14:32:57.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:32:56.998+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207048 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:32:58.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:32:58.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:58.023299+0000) 2019-10-13T14:32:58.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:32:58.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:58.249560+0000) 2019-10-13T14:32:58.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:32:58.786+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:32:58.787775+0000) 2019-10-13T14:33:00.324 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:00.318+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:00.321589+0000) 2019-10-13T14:33:01.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:01.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:01.770735+0000) 2019-10-13T14:33:02.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:02.002+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.006681+0000) 2019-10-13T14:33:02.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:02.002+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.008833+0000) 2019-10-13T14:33:02.011 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:02.002+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.009231+0000) 2019-10-13T14:33:02.012 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:02.006+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.010076+0000) 2019-10-13T14:33:02.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:02.006+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.010276+0000) 2019-10-13T14:33:02.013 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:02.006+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.010917+0000) 2019-10-13T14:33:02.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:02.006+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.011314+0000) 2019-10-13T14:33:02.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:02.010+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.013113+0000) 2019-10-13T14:33:02.019 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:02.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207081 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:02.041 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:02.034+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:02.038961+0000) 2019-10-13T14:33:04.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:04.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.368689+0000) 2019-10-13T14:33:04.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:04.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:04.653655+0000) 2019-10-13T14:33:07.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:07.014+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.020252+0000) 2019-10-13T14:33:07.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:07.014+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.020895+0000) 2019-10-13T14:33:07.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:07.014+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.021485+0000) 2019-10-13T14:33:07.025 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:07.018+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.023126+0000) 2019-10-13T14:33:07.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:07.018+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.023156+0000) 2019-10-13T14:33:07.028 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:07.018+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.023499+0000) 2019-10-13T14:33:07.028 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:07.018+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.024003+0000) 2019-10-13T14:33:07.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:07.026+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:07.028256+0000) 2019-10-13T14:33:07.034 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:07.026+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:07.950 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:33:07.950 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:07.954 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:33:07.954 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:08.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:08.018+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:08.023499+0000) 2019-10-13T14:33:08.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:08.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:08.249783+0000) 2019-10-13T14:33:08.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:08.782+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:08.787998+0000) 2019-10-13T14:33:10.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:10.326+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:10.328066+0000) 2019-10-13T14:33:11.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:11.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:11.770893+0000) 2019-10-13T14:33:12.038 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:12.030+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.035161+0000) 2019-10-13T14:33:12.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:12.030+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.036873+0000) 2019-10-13T14:33:12.039 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:12.030+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.037343+0000) 2019-10-13T14:33:12.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:12.034+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.037845+0000) 2019-10-13T14:33:12.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:12.034+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.038505+0000) 2019-10-13T14:33:12.042 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:12.034+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.039983+0000) 2019-10-13T14:33:12.045 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:12.038+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.042965+0000) 2019-10-13T14:33:12.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:12.038+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207141 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:12.049 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:12.042+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:12.046820+0000) 2019-10-13T14:33:14.371 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:14.362+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:14.368896+0000) 2019-10-13T14:33:14.656 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:14.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:14.653861+0000) 2019-10-13T14:33:17.051 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:17.042+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.048699+0000) 2019-10-13T14:33:17.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:17.042+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.049174+0000) 2019-10-13T14:33:17.052 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:17.046+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.049864+0000) 2019-10-13T14:33:17.056 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:17.050+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.052605+0000) 2019-10-13T14:33:17.056 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:17.046+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.051229+0000) 2019-10-13T14:33:17.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:17.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.052046+0000) 2019-10-13T14:33:17.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:17.046+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.052314+0000) 2019-10-13T14:33:17.061 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:17.054+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:17.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:17.054+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:17.059666+0000) 2019-10-13T14:33:18.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:18.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:18.023713+0000) 2019-10-13T14:33:18.252 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:18.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:18.249985+0000) 2019-10-13T14:33:18.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:18.782+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:18.788194+0000) 2019-10-13T14:33:18.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:18.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:18.789773+0000) 2019-10-13T14:33:20.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:20.334+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:20.334675+0000) 2019-10-13T14:33:21.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:21.766+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:21.771141+0000) 2019-10-13T14:33:22.065 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:22.058+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.062299+0000) 2019-10-13T14:33:22.065 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:22.058+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.063044+0000) 2019-10-13T14:33:22.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:22.058+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.064744+0000) 2019-10-13T14:33:22.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:22.058+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.064259+0000) 2019-10-13T14:33:22.068 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:22.058+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.064878+0000) 2019-10-13T14:33:22.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:22.062+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.066448+0000) 2019-10-13T14:33:22.071 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:22.066+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.069134+0000) 2019-10-13T14:33:22.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:22.066+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:22.070190+0000) 2019-10-13T14:33:22.076 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:22.066+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:24.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:24.366+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:24.369094+0000) 2019-10-13T14:33:24.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:24.650+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:24.654084+0000) 2019-10-13T14:33:27.079 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:27.070+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.076969+0000) 2019-10-13T14:33:27.080 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:27.074+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.077977+0000) 2019-10-13T14:33:27.081 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:27.074+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.078570+0000) 2019-10-13T14:33:27.081 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:27.074+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.079018+0000) 2019-10-13T14:33:27.082 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:27.074+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.079439+0000) 2019-10-13T14:33:27.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:27.074+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.079092+0000) 2019-10-13T14:33:27.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:27.074+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.080058+0000) 2019-10-13T14:33:27.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:27.078+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:27.084021+0000) 2019-10-13T14:33:27.090 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:27.082+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207232 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:28.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:28.018+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:28.023892+0000) 2019-10-13T14:33:28.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:28.246+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:28.250184+0000) 2019-10-13T14:33:28.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:28.786+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:28.788391+0000) 2019-10-13T14:33:28.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:28.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:28.789974+0000) 2019-10-13T14:33:30.349 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:30.342+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:30.346130+0000) 2019-10-13T14:33:31.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:31.770+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:31.771347+0000) 2019-10-13T14:33:32.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:32.086+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.090422+0000) 2019-10-13T14:33:32.094 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:32.086+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.091205+0000) 2019-10-13T14:33:32.094 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:32.086+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.091623+0000) 2019-10-13T14:33:32.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:32.086+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.092708+0000) 2019-10-13T14:33:32.095 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:32.086+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.093314+0000) 2019-10-13T14:33:32.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:32.086+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.093700+0000) 2019-10-13T14:33:32.096 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:32.090+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.094082+0000) 2019-10-13T14:33:32.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:32.098+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:32.100196+0000) 2019-10-13T14:33:32.103 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:32.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:34.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:34.366+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:34.369328+0000) 2019-10-13T14:33:34.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:34.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:34.654312+0000) 2019-10-13T14:33:37.107 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:37.098+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.104116+0000) 2019-10-13T14:33:37.107 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:37.102+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.104993+0000) 2019-10-13T14:33:37.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:37.098+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.105310+0000) 2019-10-13T14:33:37.109 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:37.102+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.106508+0000) 2019-10-13T14:33:37.109 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:37.102+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.107118+0000) 2019-10-13T14:33:37.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:37.102+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.108071+0000) 2019-10-13T14:33:37.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:37.102+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.108515+0000) 2019-10-13T14:33:37.116 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:37.110+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:37.113706+0000) 2019-10-13T14:33:37.117 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:37.110+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:38.007 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:33:38.007 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:38.011 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:33:38.011 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:33:38.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:38.017+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:38.024043+0000) 2019-10-13T14:33:38.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:38.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:38.250384+0000) 2019-10-13T14:33:38.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:38.785+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:38.788587+0000) 2019-10-13T14:33:38.792 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:38.786+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:38.790193+0000) 2019-10-13T14:33:40.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:40.350+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:40.352682+0000) 2019-10-13T14:33:41.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:41.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:41.771576+0000) 2019-10-13T14:33:42.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:42.113+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.116974+0000) 2019-10-13T14:33:42.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:42.114+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.118514+0000) 2019-10-13T14:33:42.121 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:42.113+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.118452+0000) 2019-10-13T14:33:42.121 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:42.113+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.118925+0000) 2019-10-13T14:33:42.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:42.113+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.119916+0000) 2019-10-13T14:33:42.123 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:42.113+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.120766+0000) 2019-10-13T14:33:42.125 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:42.117+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.122661+0000) 2019-10-13T14:33:42.129 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:42.122+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:42.126869+0000) 2019-10-13T14:33:42.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:42.125+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207325 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:44.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:44.361+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:44.369528+0000) 2019-10-13T14:33:44.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:44.649+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:44.654547+0000) 2019-10-13T14:33:47.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:47.129+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.132857+0000) 2019-10-13T14:33:47.136 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:47.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.133284+0000) 2019-10-13T14:33:47.138 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:47.129+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.135330+0000) 2019-10-13T14:33:47.138 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:47.129+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.136145+0000) 2019-10-13T14:33:47.139 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:47.129+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.136821+0000) 2019-10-13T14:33:47.139 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:47.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.136898+0000) 2019-10-13T14:33:47.140 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:47.129+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.137365+0000) 2019-10-13T14:33:47.145 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:47.137+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207355 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:47.147 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:47.141+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:47.144783+0000) 2019-10-13T14:33:48.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:48.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:48.024246+0000) 2019-10-13T14:33:48.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:48.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:48.250597+0000) 2019-10-13T14:33:48.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:48.785+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:48.788781+0000) 2019-10-13T14:33:48.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:48.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:48.790400+0000) 2019-10-13T14:33:50.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:50.353+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:50.359347+0000) 2019-10-13T14:33:51.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:51.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:51.771788+0000) 2019-10-13T14:33:52.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:52.141+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.147092+0000) 2019-10-13T14:33:52.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:52.141+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.147677+0000) 2019-10-13T14:33:52.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:52.141+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.148335+0000) 2019-10-13T14:33:52.151 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:52.145+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.148773+0000) 2019-10-13T14:33:52.152 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:52.141+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.149653+0000) 2019-10-13T14:33:52.153 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:52.145+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.150565+0000) 2019-10-13T14:33:52.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:52.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.151988+0000) 2019-10-13T14:33:52.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:52.149+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:52.152355+0000) 2019-10-13T14:33:52.159 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:52.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207386 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:54.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:54.369+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:54.369780+0000) 2019-10-13T14:33:54.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:54.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:54.654745+0000) 2019-10-13T14:33:57.163 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:33:57.157+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.160290+0000) 2019-10-13T14:33:57.163 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:57.157+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.160642+0000) 2019-10-13T14:33:57.165 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:57.161+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.162906+0000) 2019-10-13T14:33:57.166 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:57.161+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.163359+0000) 2019-10-13T14:33:57.166 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:57.157+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.163982+0000) 2019-10-13T14:33:57.167 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:33:57.161+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.164430+0000) 2019-10-13T14:33:57.167 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:33:57.161+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.165096+0000) 2019-10-13T14:33:57.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:33:57.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:33:57.173 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:33:57.165+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:57.170717+0000) 2019-10-13T14:33:58.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:33:58.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:58.024440+0000) 2019-10-13T14:33:58.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:33:58.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:58.250798+0000) 2019-10-13T14:33:58.791 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:33:58.785+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:58.788994+0000) 2019-10-13T14:33:58.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:33:58.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:33:58.790591+0000) 2019-10-13T14:34:00.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:00.361+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:00.365208+0000) 2019-10-13T14:34:01.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:01.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:01.771988+0000) 2019-10-13T14:34:02.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:02.169+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.172876+0000) 2019-10-13T14:34:02.177 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:02.173+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.174521+0000) 2019-10-13T14:34:02.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:02.173+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.175054+0000) 2019-10-13T14:34:02.178 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:02.173+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.175424+0000) 2019-10-13T14:34:02.178 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:02.173+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.175751+0000) 2019-10-13T14:34:02.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:02.177+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.177996+0000) 2019-10-13T14:34:02.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:02.173+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.178577+0000) 2019-10-13T14:34:02.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:02.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207449 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:02.186 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:02.181+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:02.184411+0000) 2019-10-13T14:34:04.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:04.369+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:04.369977+0000) 2019-10-13T14:34:04.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:04.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:04.654993+0000) 2019-10-13T14:34:07.189 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:07.185+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.187024+0000) 2019-10-13T14:34:07.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:07.185+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.187479+0000) 2019-10-13T14:34:07.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:07.181+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.187896+0000) 2019-10-13T14:34:07.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:07.185+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.188275+0000) 2019-10-13T14:34:07.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:07.185+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.188388+0000) 2019-10-13T14:34:07.192 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:07.185+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.189708+0000) 2019-10-13T14:34:07.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:07.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.190442+0000) 2019-10-13T14:34:07.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:07.189+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:07.195573+0000) 2019-10-13T14:34:07.200 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:07.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:08.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:08.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:08.024650+0000) 2019-10-13T14:34:08.105 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:34:08.105 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:08.109 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:34:08.109 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:08.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:08.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:08.250999+0000) 2019-10-13T14:34:08.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:08.785+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:08.789198+0000) 2019-10-13T14:34:08.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:08.785+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:08.790819+0000) 2019-10-13T14:34:10.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:10.369+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:10.371673+0000) 2019-10-13T14:34:11.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:11.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:11.772193+0000) 2019-10-13T14:34:12.203 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:12.197+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.200327+0000) 2019-10-13T14:34:12.203 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:12.197+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.200988+0000) 2019-10-13T14:34:12.204 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:12.197+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.201718+0000) 2019-10-13T14:34:12.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:12.197+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.201458+0000) 2019-10-13T14:34:12.206 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:12.201+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.203381+0000) 2019-10-13T14:34:12.206 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:12.201+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.203862+0000) 2019-10-13T14:34:12.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:12.205+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.207617+0000) 2019-10-13T14:34:12.214 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:12.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:12.218 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:12.209+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:12.216008+0000) 2019-10-13T14:34:14.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:14.369+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:14.370198+0000) 2019-10-13T14:34:14.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:14.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:14.655194+0000) 2019-10-13T14:34:17.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:17.213+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.216518+0000) 2019-10-13T14:34:17.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:17.213+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.216091+0000) 2019-10-13T14:34:17.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:17.213+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.217181+0000) 2019-10-13T14:34:17.220 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:17.213+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.216981+0000) 2019-10-13T14:34:17.221 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:17.217+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.217876+0000) 2019-10-13T14:34:17.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:17.217+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.218207+0000) 2019-10-13T14:34:17.221 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:17.217+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.218548+0000) 2019-10-13T14:34:17.227 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:17.221+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:17.224918+0000) 2019-10-13T14:34:17.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:17.221+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:18.027 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:18.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:18.024835+0000) 2019-10-13T14:34:18.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:18.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:18.251221+0000) 2019-10-13T14:34:18.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:18.785+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:18.789415+0000) 2019-10-13T14:34:18.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:18.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:18.791027+0000) 2019-10-13T14:34:20.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:20.373+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:20.378277+0000) 2019-10-13T14:34:21.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:21.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:21.772449+0000) 2019-10-13T14:34:22.232 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:22.229+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.230008+0000) 2019-10-13T14:34:22.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:22.229+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.231917+0000) 2019-10-13T14:34:22.235 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:22.229+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.232630+0000) 2019-10-13T14:34:22.235 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:22.229+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.232874+0000) 2019-10-13T14:34:22.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:22.229+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.233134+0000) 2019-10-13T14:34:22.236 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:22.229+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.233686+0000) 2019-10-13T14:34:22.237 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:22.233+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.233893+0000) 2019-10-13T14:34:22.241 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:22.237+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:22.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:22.233+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:22.239610+0000) 2019-10-13T14:34:24.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:24.369+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:24.370409+0000) 2019-10-13T14:34:24.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:24.653+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:24.655427+0000) 2019-10-13T14:34:27.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:27.241+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.242564+0000) 2019-10-13T14:34:27.245 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:27.241+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.243004+0000) 2019-10-13T14:34:27.246 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:27.241+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.243694+0000) 2019-10-13T14:34:27.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:27.241+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.243956+0000) 2019-10-13T14:34:27.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:27.241+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.244686+0000) 2019-10-13T14:34:27.248 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:27.241+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.245406+0000) 2019-10-13T14:34:27.248 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:27.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.246198+0000) 2019-10-13T14:34:27.253 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:27.245+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:27.251141+0000) 2019-10-13T14:34:27.255 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:27.249+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:28.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:28.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:28.025011+0000) 2019-10-13T14:34:28.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:28.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:28.251421+0000) 2019-10-13T14:34:28.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:28.789+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:28.789610+0000) 2019-10-13T14:34:28.793 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:28.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:28.791217+0000) 2019-10-13T14:34:30.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:30.381+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:30.387224+0000) 2019-10-13T14:34:31.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:31.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:31.772648+0000) 2019-10-13T14:34:32.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:32.253+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.256864+0000) 2019-10-13T14:34:32.260 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:32.253+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.257325+0000) 2019-10-13T14:34:32.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:32.257+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.257995+0000) 2019-10-13T14:34:32.261 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:32.253+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.258814+0000) 2019-10-13T14:34:32.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:32.257+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.260717+0000) 2019-10-13T14:34:32.263 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:32.257+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.261400+0000) 2019-10-13T14:34:32.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:32.261+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.263247+0000) 2019-10-13T14:34:32.267 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:32.261+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:32.265083+0000) 2019-10-13T14:34:32.270 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:32.265+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207634 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:34.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:34.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:34.370614+0000) 2019-10-13T14:34:34.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:34.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:34.655623+0000) 2019-10-13T14:34:37.274 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:37.268+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.271245+0000) 2019-10-13T14:34:37.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:37.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.272325+0000) 2019-10-13T14:34:37.275 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:37.268+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.272763+0000) 2019-10-13T14:34:37.275 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:37.268+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.272807+0000) 2019-10-13T14:34:37.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:37.269+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.273609+0000) 2019-10-13T14:34:37.276 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:37.272+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.273990+0000) 2019-10-13T14:34:37.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:37.272+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.274304+0000) 2019-10-13T14:34:37.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:37.277+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:37.280149+0000) 2019-10-13T14:34:37.283 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:37.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207665 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:38.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:38.020+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:38.025190+0000) 2019-10-13T14:34:38.175 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:34:38.175 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:38.179 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:34:38.179 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:34:38.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:38.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:38.251627+0000) 2019-10-13T14:34:38.792 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:38.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:38.789840+0000) 2019-10-13T14:34:38.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:38.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:38.791444+0000) 2019-10-13T14:34:40.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:40.393+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:40.393624+0000) 2019-10-13T14:34:41.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:41.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:41.772867+0000) 2019-10-13T14:34:42.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:42.280+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.284207+0000) 2019-10-13T14:34:42.287 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:42.280+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.284404+0000) 2019-10-13T14:34:42.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:42.281+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.284808+0000) 2019-10-13T14:34:42.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:42.280+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.285023+0000) 2019-10-13T14:34:42.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:42.284+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.285892+0000) 2019-10-13T14:34:42.289 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:42.284+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.286191+0000) 2019-10-13T14:34:42.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:42.284+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.286462+0000) 2019-10-13T14:34:42.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:42.289+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:42.292819+0000) 2019-10-13T14:34:42.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:42.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207697 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:44.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:44.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:44.370853+0000) 2019-10-13T14:34:44.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:44.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:44.655825+0000) 2019-10-13T14:34:47.298 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:47.293+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.295621+0000) 2019-10-13T14:34:47.298 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:47.292+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.295857+0000) 2019-10-13T14:34:47.299 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:47.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.296806+0000) 2019-10-13T14:34:47.300 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:47.292+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.296916+0000) 2019-10-13T14:34:47.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:47.292+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.297668+0000) 2019-10-13T14:34:47.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:47.296+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.300005+0000) 2019-10-13T14:34:47.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:47.300+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.303238+0000) 2019-10-13T14:34:47.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:47.297+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:47.303690+0000) 2019-10-13T14:34:47.309 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:47.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207726 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:48.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:48.020+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:48.025401+0000) 2019-10-13T14:34:48.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:48.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:48.251833+0000) 2019-10-13T14:34:48.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:48.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:48.790040+0000) 2019-10-13T14:34:48.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:48.788+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:48.791678+0000) 2019-10-13T14:34:50.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:50.396+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:50.400323+0000) 2019-10-13T14:34:51.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:51.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:51.773080+0000) 2019-10-13T14:34:52.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:52.304+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.309616+0000) 2019-10-13T14:34:52.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:52.308+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.311525+0000) 2019-10-13T14:34:52.314 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:52.308+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.312249+0000) 2019-10-13T14:34:52.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:52.308+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.312408+0000) 2019-10-13T14:34:52.316 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:52.308+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.313242+0000) 2019-10-13T14:34:52.316 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:52.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.312989+0000) 2019-10-13T14:34:52.316 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:52.312+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.314216+0000) 2019-10-13T14:34:52.322 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:52.312+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:52.319972+0000) 2019-10-13T14:34:52.323 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:52.316+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207758 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:54.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:54.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:54.371061+0000) 2019-10-13T14:34:54.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:54.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:54.656029+0000) 2019-10-13T14:34:57.325 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:57.320+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.322718+0000) 2019-10-13T14:34:57.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:57.316+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.323533+0000) 2019-10-13T14:34:57.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:34:57.320+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.325526+0000) 2019-10-13T14:34:57.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:57.324+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.325921+0000) 2019-10-13T14:34:57.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:34:57.324+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.328347+0000) 2019-10-13T14:34:57.332 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:57.328+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.330345+0000) 2019-10-13T14:34:57.334 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:34:57.328+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.332023+0000) 2019-10-13T14:34:57.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:34:57.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207789 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:34:57.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:34:57.328+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:57.334565+0000) 2019-10-13T14:34:58.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:34:58.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:58.025582+0000) 2019-10-13T14:34:58.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:34:58.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:58.252006+0000) 2019-10-13T14:34:58.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:34:58.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:58.790239+0000) 2019-10-13T14:34:58.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:34:58.784+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:34:58.791869+0000) 2019-10-13T14:35:00.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:00.400+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:00.406868+0000) 2019-10-13T14:35:01.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:01.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:01.773315+0000) 2019-10-13T14:35:02.340 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:02.332+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.337355+0000) 2019-10-13T14:35:02.340 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:02.332+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.337047+0000) 2019-10-13T14:35:02.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:02.332+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.338266+0000) 2019-10-13T14:35:02.341 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:02.336+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.338752+0000) 2019-10-13T14:35:02.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:02.336+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.339588+0000) 2019-10-13T14:35:02.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:02.336+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.340254+0000) 2019-10-13T14:35:02.343 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:02.336+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.340798+0000) 2019-10-13T14:35:02.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:02.340+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:02.345787+0000) 2019-10-13T14:35:02.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:02.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207822 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:04.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:04.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:04.371263+0000) 2019-10-13T14:35:04.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:04.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:04.656243+0000) 2019-10-13T14:35:07.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:07.344+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.351437+0000) 2019-10-13T14:35:07.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:07.348+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.351749+0000) 2019-10-13T14:35:07.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:07.348+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.352564+0000) 2019-10-13T14:35:07.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:07.348+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.352997+0000) 2019-10-13T14:35:07.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:07.348+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.353293+0000) 2019-10-13T14:35:07.356 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:07.348+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.353530+0000) 2019-10-13T14:35:07.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:07.352+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.354101+0000) 2019-10-13T14:35:07.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:07.352+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:07.358981+0000) 2019-10-13T14:35:07.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:07.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207850 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:08.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:08.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:08.025791+0000) 2019-10-13T14:35:08.227 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:35:08.227 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:08.231 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:35:08.231 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:08.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:08.248+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:08.252185+0000) 2019-10-13T14:35:08.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:08.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:08.790456+0000) 2019-10-13T14:35:08.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:08.788+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:08.792083+0000) 2019-10-13T14:35:10.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:10.412+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:10.413455+0000) 2019-10-13T14:35:11.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:11.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:11.773511+0000) 2019-10-13T14:35:12.368 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:12.364+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.365902+0000) 2019-10-13T14:35:12.369 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:12.364+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.367067+0000) 2019-10-13T14:35:12.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:12.364+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.367733+0000) 2019-10-13T14:35:12.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:12.364+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.369229+0000) 2019-10-13T14:35:12.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:12.368+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.370688+0000) 2019-10-13T14:35:12.374 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:12.368+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.372216+0000) 2019-10-13T14:35:12.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:12.368+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.373298+0000) 2019-10-13T14:35:12.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:12.372+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:12.373895+0000) 2019-10-13T14:35:12.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:12.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:14.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:14.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:14.371512+0000) 2019-10-13T14:35:14.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:14.652+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:14.656484+0000) 2019-10-13T14:35:17.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:17.376+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.378417+0000) 2019-10-13T14:35:17.382 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:17.376+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.379187+0000) 2019-10-13T14:35:17.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:17.372+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.379772+0000) 2019-10-13T14:35:17.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:17.376+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.380456+0000) 2019-10-13T14:35:17.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:17.376+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.381484+0000) 2019-10-13T14:35:17.385 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:17.380+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.383024+0000) 2019-10-13T14:35:17.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:17.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.383338+0000) 2019-10-13T14:35:17.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:17.384+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:17.389251+0000) 2019-10-13T14:35:17.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:17.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:18.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:18.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:18.025975+0000) 2019-10-13T14:35:18.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:18.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:18.252384+0000) 2019-10-13T14:35:18.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:18.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:18.790637+0000) 2019-10-13T14:35:18.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:18.788+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:18.792319+0000) 2019-10-13T14:35:20.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:20.412+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:20.419956+0000) 2019-10-13T14:35:21.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:21.768+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:21.773715+0000) 2019-10-13T14:35:22.395 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:22.388+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.392425+0000) 2019-10-13T14:35:22.395 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:22.388+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.392916+0000) 2019-10-13T14:35:22.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:22.388+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.393545+0000) 2019-10-13T14:35:22.396 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:22.392+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.393932+0000) 2019-10-13T14:35:22.397 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:22.392+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.395025+0000) 2019-10-13T14:35:22.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:22.392+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.395614+0000) 2019-10-13T14:35:22.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:22.392+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.395877+0000) 2019-10-13T14:35:22.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:22.396+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:22.401702+0000) 2019-10-13T14:35:22.405 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:22.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:24.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:24.368+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:24.371726+0000) 2019-10-13T14:35:24.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:24.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:24.656717+0000) 2019-10-13T14:35:27.409 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:27.404+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.406253+0000) 2019-10-13T14:35:27.409 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:27.404+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.406568+0000) 2019-10-13T14:35:27.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:27.404+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.407532+0000) 2019-10-13T14:35:27.410 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:27.404+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.407777+0000) 2019-10-13T14:35:27.411 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:27.404+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.408286+0000) 2019-10-13T14:35:27.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:27.404+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.408460+0000) 2019-10-13T14:35:27.412 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:27.404+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.409412+0000) 2019-10-13T14:35:27.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:27.408+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:27.415581+0000) 2019-10-13T14:35:27.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:27.412+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207974 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:28.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:28.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:28.026161+0000) 2019-10-13T14:35:28.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:28.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:28.252557+0000) 2019-10-13T14:35:28.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:28.788+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:28.790864+0000) 2019-10-13T14:35:28.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:28.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:28.792521+0000) 2019-10-13T14:35:30.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:30.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:30.426496+0000) 2019-10-13T14:35:31.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:31.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:31.773945+0000) 2019-10-13T14:35:32.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:32.415+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.419893+0000) 2019-10-13T14:35:32.423 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:32.415+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.420200+0000) 2019-10-13T14:35:32.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:32.419+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.421940+0000) 2019-10-13T14:35:32.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:32.416+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.422488+0000) 2019-10-13T14:35:32.426 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:32.419+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.423254+0000) 2019-10-13T14:35:32.427 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:32.419+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.424755+0000) 2019-10-13T14:35:32.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:32.420+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.425827+0000) 2019-10-13T14:35:32.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:32.423+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:32.426929+0000) 2019-10-13T14:35:32.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:32.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208007 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:34.374 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:34.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:34.371950+0000) 2019-10-13T14:35:34.659 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:34.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:34.656943+0000) 2019-10-13T14:35:37.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:37.431+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.433725+0000) 2019-10-13T14:35:37.437 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:37.431+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.434495+0000) 2019-10-13T14:35:37.437 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:37.431+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.435066+0000) 2019-10-13T14:35:37.438 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:37.432+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.436154+0000) 2019-10-13T14:35:37.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:37.431+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.436235+0000) 2019-10-13T14:35:37.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:37.431+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.436572+0000) 2019-10-13T14:35:37.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:37.431+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.436998+0000) 2019-10-13T14:35:37.445 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:37.436+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:37.442262+0000) 2019-10-13T14:35:37.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:37.439+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:38.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:38.023+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:38.026336+0000) 2019-10-13T14:35:38.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:38.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:38.252726+0000) 2019-10-13T14:35:38.511 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:35:38.511 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:38.515 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:35:38.515 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:35:38.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:38.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:38.791077+0000) 2019-10-13T14:35:38.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:38.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:38.792727+0000) 2019-10-13T14:35:40.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:40.432+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:40.433009+0000) 2019-10-13T14:35:41.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:41.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:41.774152+0000) 2019-10-13T14:35:42.450 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:42.443+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.447735+0000) 2019-10-13T14:35:42.452 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:42.444+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.449873+0000) 2019-10-13T14:35:42.453 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:42.447+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.450240+0000) 2019-10-13T14:35:42.454 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:42.447+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.451360+0000) 2019-10-13T14:35:42.454 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:42.447+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.451402+0000) 2019-10-13T14:35:42.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:42.447+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.452990+0000) 2019-10-13T14:35:42.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:42.451+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.453944+0000) 2019-10-13T14:35:42.460 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:42.455+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208070 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:42.462 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:42.456+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:42.460276+0000) 2019-10-13T14:35:44.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:44.371+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:44.372215+0000) 2019-10-13T14:35:44.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:44.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:44.657180+0000) 2019-10-13T14:35:47.464 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:47.455+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.461597+0000) 2019-10-13T14:35:47.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:47.459+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.462025+0000) 2019-10-13T14:35:47.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:47.459+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.462769+0000) 2019-10-13T14:35:47.466 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:47.459+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.463191+0000) 2019-10-13T14:35:47.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:47.459+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.463931+0000) 2019-10-13T14:35:47.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:47.460+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.464856+0000) 2019-10-13T14:35:47.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:47.463+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.466159+0000) 2019-10-13T14:35:47.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:47.464+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:47.469951+0000) 2019-10-13T14:35:47.473 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:47.467+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:48.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:48.023+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:48.026517+0000) 2019-10-13T14:35:48.255 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:48.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:48.252901+0000) 2019-10-13T14:35:48.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:48.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:48.791276+0000) 2019-10-13T14:35:48.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:48.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:48.792934+0000) 2019-10-13T14:35:50.443 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:50.440+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:50.440302+0000) 2019-10-13T14:35:51.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:51.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:51.774320+0000) 2019-10-13T14:35:52.477 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:52.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.474855+0000) 2019-10-13T14:35:52.478 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:52.471+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.475696+0000) 2019-10-13T14:35:52.478 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:52.471+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.475608+0000) 2019-10-13T14:35:52.479 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:52.471+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.476079+0000) 2019-10-13T14:35:52.479 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:52.471+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.476449+0000) 2019-10-13T14:35:52.480 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:52.476+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.477029+0000) 2019-10-13T14:35:52.480 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:52.475+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.478133+0000) 2019-10-13T14:35:52.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:52.480+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:52.483572+0000) 2019-10-13T14:35:52.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:52.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208131 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:54.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:54.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:54.372456+0000) 2019-10-13T14:35:54.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:54.651+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:54.657387+0000) 2019-10-13T14:35:57.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:57.483+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.487847+0000) 2019-10-13T14:35:57.491 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:57.483+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.488096+0000) 2019-10-13T14:35:57.491 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:35:57.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.488688+0000) 2019-10-13T14:35:57.492 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:57.483+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.488957+0000) 2019-10-13T14:35:57.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:57.483+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.489305+0000) 2019-10-13T14:35:57.492 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:35:57.483+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.489717+0000) 2019-10-13T14:35:57.493 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:35:57.487+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.490345+0000) 2019-10-13T14:35:57.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:35:57.495+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:57.497377+0000) 2019-10-13T14:35:57.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:35:57.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:35:58.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:35:58.023+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:58.026700+0000) 2019-10-13T14:35:58.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:35:58.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:58.253092+0000) 2019-10-13T14:35:58.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:35:58.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:58.791477+0000) 2019-10-13T14:35:58.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:35:58.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:35:58.793080+0000) 2019-10-13T14:36:00.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:00.443+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:00.446672+0000) 2019-10-13T14:36:01.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:01.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:01.774515+0000) 2019-10-13T14:36:02.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:02.499+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.501897+0000) 2019-10-13T14:36:02.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:02.499+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.502423+0000) 2019-10-13T14:36:02.506 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:02.499+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.503277+0000) 2019-10-13T14:36:02.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:02.499+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.503584+0000) 2019-10-13T14:36:02.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:02.499+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.504307+0000) 2019-10-13T14:36:02.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:02.499+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.504599+0000) 2019-10-13T14:36:02.507 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:02.499+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.504875+0000) 2019-10-13T14:36:02.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:02.507+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:02.510338+0000) 2019-10-13T14:36:02.514 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:02.507+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:04.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:04.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:04.372679+0000) 2019-10-13T14:36:04.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:04.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:04.657650+0000) 2019-10-13T14:36:07.518 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:07.511+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.515475+0000) 2019-10-13T14:36:07.519 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:07.511+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.515786+0000) 2019-10-13T14:36:07.519 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:07.511+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.516094+0000) 2019-10-13T14:36:07.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:07.511+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.517528+0000) 2019-10-13T14:36:07.520 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:07.515+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.518019+0000) 2019-10-13T14:36:07.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:07.515+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.518880+0000) 2019-10-13T14:36:07.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:07.515+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.519589+0000) 2019-10-13T14:36:07.525 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:07.519+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:07.523380+0000) 2019-10-13T14:36:07.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:07.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208221 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:08.029 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:08.023+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:08.026878+0000) 2019-10-13T14:36:08.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:08.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:08.253268+0000) 2019-10-13T14:36:08.558 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:36:08.558 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:08.562 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:36:08.562 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:08.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:08.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:08.791692+0000) 2019-10-13T14:36:08.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:08.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:08.793321+0000) 2019-10-13T14:36:10.456 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:10.451+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:10.453874+0000) 2019-10-13T14:36:11.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:11.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:11.774755+0000) 2019-10-13T14:36:12.530 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:12.523+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.528049+0000) 2019-10-13T14:36:12.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:12.523+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.528894+0000) 2019-10-13T14:36:12.532 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:12.523+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.529317+0000) 2019-10-13T14:36:12.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:12.527+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.529685+0000) 2019-10-13T14:36:12.533 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:12.527+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.530107+0000) 2019-10-13T14:36:12.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:12.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.531126+0000) 2019-10-13T14:36:12.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:12.527+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.531200+0000) 2019-10-13T14:36:12.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:12.531+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:12.534685+0000) 2019-10-13T14:36:12.541 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:12.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:14.375 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:14.367+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:14.372875+0000) 2019-10-13T14:36:14.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:14.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:14.657849+0000) 2019-10-13T14:36:17.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:17.539+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.542243+0000) 2019-10-13T14:36:17.545 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:17.539+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.542806+0000) 2019-10-13T14:36:17.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:17.539+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.543392+0000) 2019-10-13T14:36:17.546 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:17.539+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.543461+0000) 2019-10-13T14:36:17.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:17.543+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.544424+0000) 2019-10-13T14:36:17.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:17.539+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.545263+0000) 2019-10-13T14:36:17.548 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:17.543+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.546098+0000) 2019-10-13T14:36:17.554 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:17.547+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208284 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:17.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:17.551+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:17.553196+0000) 2019-10-13T14:36:18.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:18.023+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:18.027080+0000) 2019-10-13T14:36:18.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:18.247+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:18.253441+0000) 2019-10-13T14:36:18.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:18.787+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:18.791892+0000) 2019-10-13T14:36:18.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:18.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:18.793524+0000) 2019-10-13T14:36:20.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:20.463+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:20.464746+0000) 2019-10-13T14:36:21.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:21.771+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:21.774953+0000) 2019-10-13T14:36:22.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:22.551+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.555592+0000) 2019-10-13T14:36:22.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:22.551+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.555988+0000) 2019-10-13T14:36:22.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:22.555+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.557976+0000) 2019-10-13T14:36:22.561 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:22.555+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.559239+0000) 2019-10-13T14:36:22.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:22.555+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.559729+0000) 2019-10-13T14:36:22.562 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:22.555+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.559924+0000) 2019-10-13T14:36:22.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:22.555+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.560331+0000) 2019-10-13T14:36:22.568 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:22.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:22.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:22.563+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:22.566583+0000) 2019-10-13T14:36:24.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:24.371+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:24.373101+0000) 2019-10-13T14:36:24.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:24.655+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:24.658075+0000) 2019-10-13T14:36:27.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:27.567+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.569346+0000) 2019-10-13T14:36:27.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:27.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.569731+0000) 2019-10-13T14:36:27.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:27.566+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.570802+0000) 2019-10-13T14:36:27.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:27.566+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.571108+0000) 2019-10-13T14:36:27.574 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:27.566+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.571545+0000) 2019-10-13T14:36:27.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:27.566+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.572116+0000) 2019-10-13T14:36:27.575 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:27.566+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.572399+0000) 2019-10-13T14:36:27.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:27.575+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:27.577534+0000) 2019-10-13T14:36:27.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:27.574+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208345 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:28.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:28.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:28.027282+0000) 2019-10-13T14:36:28.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:28.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:28.253621+0000) 2019-10-13T14:36:28.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:28.790+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:28.792149+0000) 2019-10-13T14:36:28.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:28.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:28.793719+0000) 2019-10-13T14:36:30.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:30.467+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:30.471087+0000) 2019-10-13T14:36:31.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:31.770+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:31.775110+0000) 2019-10-13T14:36:32.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:32.579+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.583389+0000) 2019-10-13T14:36:32.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:32.578+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.584213+0000) 2019-10-13T14:36:32.587 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:32.578+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.584638+0000) 2019-10-13T14:36:32.587 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:32.578+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.584855+0000) 2019-10-13T14:36:32.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:32.578+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.584956+0000) 2019-10-13T14:36:32.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:32.578+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.585635+0000) 2019-10-13T14:36:32.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:32.582+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.586478+0000) 2019-10-13T14:36:32.594 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:32.587+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:32.591642+0000) 2019-10-13T14:36:32.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:32.590+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208378 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:34.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:34.370+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:34.373302+0000) 2019-10-13T14:36:34.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:34.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:34.658283+0000) 2019-10-13T14:36:37.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:37.595+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.598044+0000) 2019-10-13T14:36:37.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:37.594+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.598463+0000) 2019-10-13T14:36:37.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:37.594+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.600109+0000) 2019-10-13T14:36:37.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:37.594+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.601539+0000) 2019-10-13T14:36:37.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:37.598+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.603370+0000) 2019-10-13T14:36:37.607 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:37.598+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.605057+0000) 2019-10-13T14:36:37.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:37.603+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.605192+0000) 2019-10-13T14:36:37.609 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:37.602+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:37.606662+0000) 2019-10-13T14:36:37.610 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:37.602+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:38.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:38.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:38.027418+0000) 2019-10-13T14:36:38.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:38.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:38.253811+0000) 2019-10-13T14:36:38.763 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:36:38.763 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:38.767 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:36:38.767 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:36:38.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:38.786+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:38.792343+0000) 2019-10-13T14:36:38.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:38.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:38.793948+0000) 2019-10-13T14:36:40.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:40.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:40.477516+0000) 2019-10-13T14:36:41.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:41.770+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:41.775307+0000) 2019-10-13T14:36:42.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:42.607+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.610948+0000) 2019-10-13T14:36:42.615 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:42.606+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.612147+0000) 2019-10-13T14:36:42.615 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:42.606+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.612746+0000) 2019-10-13T14:36:42.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:42.606+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.613128+0000) 2019-10-13T14:36:42.616 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:42.610+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.613728+0000) 2019-10-13T14:36:42.617 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:42.610+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.614033+0000) 2019-10-13T14:36:42.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:42.610+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.614494+0000) 2019-10-13T14:36:42.620 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:42.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:42.617897+0000) 2019-10-13T14:36:42.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:42.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208440 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:44.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:44.366+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:44.373507+0000) 2019-10-13T14:36:44.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:44.654+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:44.658512+0000) 2019-10-13T14:36:47.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:47.618+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.624492+0000) 2019-10-13T14:36:47.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:47.623+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.625035+0000) 2019-10-13T14:36:47.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:47.618+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.625685+0000) 2019-10-13T14:36:47.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:47.622+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.626648+0000) 2019-10-13T14:36:47.629 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:47.622+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.627253+0000) 2019-10-13T14:36:47.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:47.622+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.627881+0000) 2019-10-13T14:36:47.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:47.622+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.628205+0000) 2019-10-13T14:36:47.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:47.631+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:47.634069+0000) 2019-10-13T14:36:47.639 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:47.630+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208468 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:48.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:48.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:48.027590+0000) 2019-10-13T14:36:48.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:48.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:48.253986+0000) 2019-10-13T14:36:48.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:48.786+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:48.792534+0000) 2019-10-13T14:36:48.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:48.791+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:48.794147+0000) 2019-10-13T14:36:50.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:50.483+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:50.483986+0000) 2019-10-13T14:36:51.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:51.774+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:51.775523+0000) 2019-10-13T14:36:52.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:52.634+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.640060+0000) 2019-10-13T14:36:52.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:52.634+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.640395+0000) 2019-10-13T14:36:52.644 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:52.634+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.641021+0000) 2019-10-13T14:36:52.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:52.634+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.641103+0000) 2019-10-13T14:36:52.644 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:52.639+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.641613+0000) 2019-10-13T14:36:52.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:52.638+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.643328+0000) 2019-10-13T14:36:52.646 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:52.638+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.643562+0000) 2019-10-13T14:36:52.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:52.647+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:52.650420+0000) 2019-10-13T14:36:52.653 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:52.646+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:54.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:54.370+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:54.373799+0000) 2019-10-13T14:36:54.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:54.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:54.658677+0000) 2019-10-13T14:36:57.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:57.651+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.653947+0000) 2019-10-13T14:36:57.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:36:57.650+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.654622+0000) 2019-10-13T14:36:57.657 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:36:57.650+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.654838+0000) 2019-10-13T14:36:57.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:36:57.650+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.655286+0000) 2019-10-13T14:36:57.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:57.650+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.655545+0000) 2019-10-13T14:36:57.659 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:57.650+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.656046+0000) 2019-10-13T14:36:57.659 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:57.650+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.657182+0000) 2019-10-13T14:36:57.666 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:36:57.658+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:36:57.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:36:57.663+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:57.664327+0000) 2019-10-13T14:36:58.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:36:58.022+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:58.027793+0000) 2019-10-13T14:36:58.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:36:58.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:58.254164+0000) 2019-10-13T14:36:58.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:36:58.790+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:58.792731+0000) 2019-10-13T14:36:58.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:36:58.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:36:58.794332+0000) 2019-10-13T14:37:00.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:00.486+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:00.490489+0000) 2019-10-13T14:37:01.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:01.770+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:01.775729+0000) 2019-10-13T14:37:02.669 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:02.662+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.666903+0000) 2019-10-13T14:37:02.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:02.662+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.667647+0000) 2019-10-13T14:37:02.672 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:02.666+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.669863+0000) 2019-10-13T14:37:02.673 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:02.666+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.670283+0000) 2019-10-13T14:37:02.673 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:02.666+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.670907+0000) 2019-10-13T14:37:02.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:02.666+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.671035+0000) 2019-10-13T14:37:02.674 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:02.666+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.672022+0000) 2019-10-13T14:37:02.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:02.674+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:02.676419+0000) 2019-10-13T14:37:02.680 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:02.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208562 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:04.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:04.370+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:04.374023+0000) 2019-10-13T14:37:04.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:04.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:04.658902+0000) 2019-10-13T14:37:07.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:07.674+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.680488+0000) 2019-10-13T14:37:07.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:07.674+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.681271+0000) 2019-10-13T14:37:07.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:07.678+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.681914+0000) 2019-10-13T14:37:07.685 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:07.678+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.682149+0000) 2019-10-13T14:37:07.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:07.678+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.682898+0000) 2019-10-13T14:37:07.686 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:07.678+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.683626+0000) 2019-10-13T14:37:07.686 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:07.678+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.684178+0000) 2019-10-13T14:37:07.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:07.686+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:07.688826+0000) 2019-10-13T14:37:07.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:07.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208590 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:08.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:08.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:08.027990+0000) 2019-10-13T14:37:08.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:08.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:08.254360+0000) 2019-10-13T14:37:08.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:08.790+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:08.792924+0000) 2019-10-13T14:37:08.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:08.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:08.794548+0000) 2019-10-13T14:37:08.876 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:37:08.877 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:08.880 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:37:08.880 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:10.502 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:10.494+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:10.499976+0000) 2019-10-13T14:37:11.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:11.774+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:11.775974+0000) 2019-10-13T14:37:12.696 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:12.690+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.693701+0000) 2019-10-13T14:37:12.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:12.690+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.695484+0000) 2019-10-13T14:37:12.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:12.690+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.695824+0000) 2019-10-13T14:37:12.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:12.690+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.696624+0000) 2019-10-13T14:37:12.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:12.694+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.696862+0000) 2019-10-13T14:37:12.700 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:12.690+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.697179+0000) 2019-10-13T14:37:12.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:12.694+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.699243+0000) 2019-10-13T14:37:12.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:12.702+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:12.705238+0000) 2019-10-13T14:37:12.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:12.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208622 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:14.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:14.374+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:14.374225+0000) 2019-10-13T14:37:14.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:14.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:14.659128+0000) 2019-10-13T14:37:17.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:17.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.708676+0000) 2019-10-13T14:37:17.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:17.702+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.709553+0000) 2019-10-13T14:37:17.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:17.706+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.710199+0000) 2019-10-13T14:37:17.714 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:17.710+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.712172+0000) 2019-10-13T14:37:17.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:17.706+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.712199+0000) 2019-10-13T14:37:17.717 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:17.710+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.714112+0000) 2019-10-13T14:37:17.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:17.710+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.714157+0000) 2019-10-13T14:37:17.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:17.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208652 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:17.723 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:17.718+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:17.721027+0000) 2019-10-13T14:37:18.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:18.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:18.028132+0000) 2019-10-13T14:37:18.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:18.250+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:18.254559+0000) 2019-10-13T14:37:18.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:18.790+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:18.793132+0000) 2019-10-13T14:37:18.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:18.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:18.794746+0000) 2019-10-13T14:37:20.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:20.502+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:20.506394+0000) 2019-10-13T14:37:21.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:21.770+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:21.776180+0000) 2019-10-13T14:37:22.724 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:22.717+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.721579+0000) 2019-10-13T14:37:22.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:22.717+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.722300+0000) 2019-10-13T14:37:22.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:22.717+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.722562+0000) 2019-10-13T14:37:22.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:22.717+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.722981+0000) 2019-10-13T14:37:22.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:22.717+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.723488+0000) 2019-10-13T14:37:22.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:22.718+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.723937+0000) 2019-10-13T14:37:22.727 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:22.717+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.725159+0000) 2019-10-13T14:37:22.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:22.726+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:22.730644+0000) 2019-10-13T14:37:22.734 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:22.725+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:24.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:24.369+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:24.374450+0000) 2019-10-13T14:37:24.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:24.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:24.659334+0000) 2019-10-13T14:37:27.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:27.729+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.735720+0000) 2019-10-13T14:37:27.739 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:27.729+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.736293+0000) 2019-10-13T14:37:27.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:27.729+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.737357+0000) 2019-10-13T14:37:27.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:27.733+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.739290+0000) 2019-10-13T14:37:27.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:27.733+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.739844+0000) 2019-10-13T14:37:27.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:27.738+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.740383+0000) 2019-10-13T14:37:27.743 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:27.733+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.740846+0000) 2019-10-13T14:37:27.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:27.738+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:27.743903+0000) 2019-10-13T14:37:27.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:27.741+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208712 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:28.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:28.021+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:28.028317+0000) 2019-10-13T14:37:28.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:28.249+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:28.254747+0000) 2019-10-13T14:37:28.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:28.789+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:28.793330+0000) 2019-10-13T14:37:28.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:28.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:28.794969+0000) 2019-10-13T14:37:30.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:30.510+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:30.512993+0000) 2019-10-13T14:37:31.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:31.769+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:31.776378+0000) 2019-10-13T14:37:32.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:32.745+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.749563+0000) 2019-10-13T14:37:32.753 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:32.745+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.749825+0000) 2019-10-13T14:37:32.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:32.745+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.751276+0000) 2019-10-13T14:37:32.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:32.745+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.751981+0000) 2019-10-13T14:37:32.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:32.745+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.752307+0000) 2019-10-13T14:37:32.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:32.750+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.753376+0000) 2019-10-13T14:37:32.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:32.745+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.753596+0000) 2019-10-13T14:37:32.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:32.754+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:32.757433+0000) 2019-10-13T14:37:32.762 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:32.753+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:34.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:34.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:34.374690+0000) 2019-10-13T14:37:34.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:34.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:34.659538+0000) 2019-10-13T14:37:37.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:37.761+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.763012+0000) 2019-10-13T14:37:37.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:37.761+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.763366+0000) 2019-10-13T14:37:37.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:37.761+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.764454+0000) 2019-10-13T14:37:37.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:37.761+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.764888+0000) 2019-10-13T14:37:37.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:37.761+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.765704+0000) 2019-10-13T14:37:37.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:37.762+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.766149+0000) 2019-10-13T14:37:37.769 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:37.765+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.766437+0000) 2019-10-13T14:37:37.776 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:37.769+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:37.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:37.770+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:37.773405+0000) 2019-10-13T14:37:38.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:38.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:38.028511+0000) 2019-10-13T14:37:38.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:38.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:38.254919+0000) 2019-10-13T14:37:38.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:38.789+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:38.793510+0000) 2019-10-13T14:37:38.797 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:38.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:38.795162+0000) 2019-10-13T14:37:38.992 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:37:38.993 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:38.996 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:37:38.997 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:37:40.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:40.518+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:40.519547+0000) 2019-10-13T14:37:41.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:41.773+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:41.776575+0000) 2019-10-13T14:37:42.779 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:42.773+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.777095+0000) 2019-10-13T14:37:42.781 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:42.774+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.779107+0000) 2019-10-13T14:37:42.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:42.777+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.779453+0000) 2019-10-13T14:37:42.783 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:42.777+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.781126+0000) 2019-10-13T14:37:42.784 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:42.777+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.781451+0000) 2019-10-13T14:37:42.785 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:42.781+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.781868+0000) 2019-10-13T14:37:42.789 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:42.785+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.786426+0000) 2019-10-13T14:37:42.790 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:42.785+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208806 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:42.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:42.790+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:42.793728+0000) 2019-10-13T14:37:44.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:44.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:44.374979+0000) 2019-10-13T14:37:44.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:44.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:44.659761+0000) 2019-10-13T14:37:47.795 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:47.789+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.792604+0000) 2019-10-13T14:37:47.795 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:47.789+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.793171+0000) 2019-10-13T14:37:47.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:47.790+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.793853+0000) 2019-10-13T14:37:47.797 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:47.793+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.794572+0000) 2019-10-13T14:37:47.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:47.793+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.794446+0000) 2019-10-13T14:37:47.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:47.797+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.798643+0000) 2019-10-13T14:37:47.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:47.797+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.799333+0000) 2019-10-13T14:37:47.803 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:47.797+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:47.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:47.798+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:47.801770+0000) 2019-10-13T14:37:48.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:48.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:48.028692+0000) 2019-10-13T14:37:48.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:48.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:48.255096+0000) 2019-10-13T14:37:48.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:48.793+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:48.793720+0000) 2019-10-13T14:37:48.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:48.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:48.795389+0000) 2019-10-13T14:37:50.533 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:50.530+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:50.530786+0000) 2019-10-13T14:37:51.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:51.773+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:51.776774+0000) 2019-10-13T14:37:52.807 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:52.801+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.804447+0000) 2019-10-13T14:37:52.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:52.801+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.805080+0000) 2019-10-13T14:37:52.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:52.805+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.805802+0000) 2019-10-13T14:37:52.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:52.805+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.806020+0000) 2019-10-13T14:37:52.809 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:52.805+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.806754+0000) 2019-10-13T14:37:52.809 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:52.805+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.806894+0000) 2019-10-13T14:37:52.810 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:52.802+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.807322+0000) 2019-10-13T14:37:52.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:52.810+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:52.814024+0000) 2019-10-13T14:37:52.817 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:52.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208866 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:54.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:54.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:54.375227+0000) 2019-10-13T14:37:54.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:54.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:54.659959+0000) 2019-10-13T14:37:57.820 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:57.813+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.817161+0000) 2019-10-13T14:37:57.820 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:57.817+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.817807+0000) 2019-10-13T14:37:57.821 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:37:57.817+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.818865+0000) 2019-10-13T14:37:57.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:37:57.817+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.819695+0000) 2019-10-13T14:37:57.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:57.814+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.820011+0000) 2019-10-13T14:37:57.823 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:37:57.817+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.820466+0000) 2019-10-13T14:37:57.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:57.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.820516+0000) 2019-10-13T14:37:57.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:37:57.822+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:57.825971+0000) 2019-10-13T14:37:57.830 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:37:57.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208896 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:37:58.031 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:37:58.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:58.028871+0000) 2019-10-13T14:37:58.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:37:58.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:58.255272+0000) 2019-10-13T14:37:58.796 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:37:58.793+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:58.793957+0000) 2019-10-13T14:37:58.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:37:58.790+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:37:58.795606+0000) 2019-10-13T14:38:00.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:00.534+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:00.536613+0000) 2019-10-13T14:38:01.779 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:01.773+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:01.776983+0000) 2019-10-13T14:38:02.834 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:02.829+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.831201+0000) 2019-10-13T14:38:02.834 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:02.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.831749+0000) 2019-10-13T14:38:02.835 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:02.829+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.832646+0000) 2019-10-13T14:38:02.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:02.829+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.833473+0000) 2019-10-13T14:38:02.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:02.833+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.835170+0000) 2019-10-13T14:38:02.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:02.830+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.835504+0000) 2019-10-13T14:38:02.840 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:02.833+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.837653+0000) 2019-10-13T14:38:02.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:02.834+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:02.839483+0000) 2019-10-13T14:38:02.845 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:02.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208927 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:04.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:04.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:04.375432+0000) 2019-10-13T14:38:04.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:04.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:04.660168+0000) 2019-10-13T14:38:07.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:07.841+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.845073+0000) 2019-10-13T14:38:07.848 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:07.845+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.845812+0000) 2019-10-13T14:38:07.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:07.845+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.846968+0000) 2019-10-13T14:38:07.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:07.845+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.847958+0000) 2019-10-13T14:38:07.851 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:07.845+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.848341+0000) 2019-10-13T14:38:07.851 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:07.845+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.848773+0000) 2019-10-13T14:38:07.853 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:07.849+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.850540+0000) 2019-10-13T14:38:07.855 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:07.849+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:07.852720+0000) 2019-10-13T14:38:07.859 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:07.853+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208956 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:08.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:08.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:08.029111+0000) 2019-10-13T14:38:08.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:08.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:08.255485+0000) 2019-10-13T14:38:08.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:08.793+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:08.794095+0000) 2019-10-13T14:38:08.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:08.789+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:08.795795+0000) 2019-10-13T14:38:09.346 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:38:09.347 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:09.350 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:38:09.350 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:10.545 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:10.541+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:10.542762+0000) 2019-10-13T14:38:11.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:11.777+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:11.777193+0000) 2019-10-13T14:38:12.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:12.857+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.859831+0000) 2019-10-13T14:38:12.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:12.857+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.860858+0000) 2019-10-13T14:38:12.864 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:12.857+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.861603+0000) 2019-10-13T14:38:12.864 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:12.857+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.862074+0000) 2019-10-13T14:38:12.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:12.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.862560+0000) 2019-10-13T14:38:12.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:12.861+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.862457+0000) 2019-10-13T14:38:12.866 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:12.861+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.863355+0000) 2019-10-13T14:38:12.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:12.865+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:12.870091+0000) 2019-10-13T14:38:12.874 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:12.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208987 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:14.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:14.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:14.375641+0000) 2019-10-13T14:38:14.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:14.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:14.660353+0000) 2019-10-13T14:38:17.877 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:17.873+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.874402+0000) 2019-10-13T14:38:17.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:17.873+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.875020+0000) 2019-10-13T14:38:17.878 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:17.873+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.875571+0000) 2019-10-13T14:38:17.878 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:17.869+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.875821+0000) 2019-10-13T14:38:17.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:17.873+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.876100+0000) 2019-10-13T14:38:17.879 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:17.873+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.876592+0000) 2019-10-13T14:38:17.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:17.873+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.877697+0000) 2019-10-13T14:38:17.885 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:17.877+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:17.883271+0000) 2019-10-13T14:38:17.887 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:17.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:18.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:18.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:18.029303+0000) 2019-10-13T14:38:18.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:18.253+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:18.255662+0000) 2019-10-13T14:38:18.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:18.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:18.794273+0000) 2019-10-13T14:38:18.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:18.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:18.796002+0000) 2019-10-13T14:38:20.552 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:20.545+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:20.549294+0000) 2019-10-13T14:38:21.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:21.772+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:21.777407+0000) 2019-10-13T14:38:22.890 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:22.884+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.887404+0000) 2019-10-13T14:38:22.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:22.884+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.887958+0000) 2019-10-13T14:38:22.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:22.885+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.888413+0000) 2019-10-13T14:38:22.893 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:22.888+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.890861+0000) 2019-10-13T14:38:22.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:22.888+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.891567+0000) 2019-10-13T14:38:22.894 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:22.888+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.891768+0000) 2019-10-13T14:38:22.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:22.888+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.892124+0000) 2019-10-13T14:38:22.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:22.893+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:22.898399+0000) 2019-10-13T14:38:22.901 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:22.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:24.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:24.372+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:24.375876+0000) 2019-10-13T14:38:24.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:24.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:24.660554+0000) 2019-10-13T14:38:27.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:27.896+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.901259+0000) 2019-10-13T14:38:27.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:27.897+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.903541+0000) 2019-10-13T14:38:27.906 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:27.900+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.904109+0000) 2019-10-13T14:38:27.907 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:27.900+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.904932+0000) 2019-10-13T14:38:27.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:27.900+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.905247+0000) 2019-10-13T14:38:27.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:27.904+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.906208+0000) 2019-10-13T14:38:27.909 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:27.904+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.906980+0000) 2019-10-13T14:38:27.915 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:27.909+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:27.912622+0000) 2019-10-13T14:38:27.915 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:27.908+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209078 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:28.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:28.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:28.029487+0000) 2019-10-13T14:38:28.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:28.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:28.255835+0000) 2019-10-13T14:38:28.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:28.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:28.794452+0000) 2019-10-13T14:38:28.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:28.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:28.796199+0000) 2019-10-13T14:38:30.558 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:30.549+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:30.555850+0000) 2019-10-13T14:38:31.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:31.772+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:31.777608+0000) 2019-10-13T14:38:32.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:32.912+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.916071+0000) 2019-10-13T14:38:32.920 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:32.912+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.917426+0000) 2019-10-13T14:38:32.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:32.916+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.918600+0000) 2019-10-13T14:38:32.921 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:32.916+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.919028+0000) 2019-10-13T14:38:32.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:32.913+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.919159+0000) 2019-10-13T14:38:32.922 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:32.916+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.919624+0000) 2019-10-13T14:38:32.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:32.916+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.919864+0000) 2019-10-13T14:38:32.929 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:32.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:32.929 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:32.921+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:32.927017+0000) 2019-10-13T14:38:34.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:34.372+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:34.376075+0000) 2019-10-13T14:38:34.663 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:34.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:34.660748+0000) 2019-10-13T14:38:37.933 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:37.928+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.930328+0000) 2019-10-13T14:38:37.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:37.925+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.930748+0000) 2019-10-13T14:38:37.935 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:37.928+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.933101+0000) 2019-10-13T14:38:37.936 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:37.928+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.933394+0000) 2019-10-13T14:38:37.937 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:37.932+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.934712+0000) 2019-10-13T14:38:37.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:37.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.937018+0000) 2019-10-13T14:38:37.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:37.932+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.936638+0000) 2019-10-13T14:38:37.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:37.933+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:37.937894+0000) 2019-10-13T14:38:37.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:37.936+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209140 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:38.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:38.024+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:38.029622+0000) 2019-10-13T14:38:38.258 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:38.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:38.255998+0000) 2019-10-13T14:38:38.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:38.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:38.794662+0000) 2019-10-13T14:38:38.798 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:38.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:38.796372+0000) 2019-10-13T14:38:39.476 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:38:39.476 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:39.480 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:38:39.480 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:38:40.565 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:40.557+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:40.562425+0000) 2019-10-13T14:38:41.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:41.776+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:41.777796+0000) 2019-10-13T14:38:42.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:42.940+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.944091+0000) 2019-10-13T14:38:42.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:42.941+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.945449+0000) 2019-10-13T14:38:42.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:42.944+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.945871+0000) 2019-10-13T14:38:42.949 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:42.944+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.946810+0000) 2019-10-13T14:38:42.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:42.944+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.948260+0000) 2019-10-13T14:38:42.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:42.944+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.948723+0000) 2019-10-13T14:38:42.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:42.944+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.948938+0000) 2019-10-13T14:38:42.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:42.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209171 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:42.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:42.949+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:42.955791+0000) 2019-10-13T14:38:44.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:44.372+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:44.376278+0000) 2019-10-13T14:38:44.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:44.656+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:44.660942+0000) 2019-10-13T14:38:47.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:47.956+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.958541+0000) 2019-10-13T14:38:47.962 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:47.956+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.959151+0000) 2019-10-13T14:38:47.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:47.953+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.959438+0000) 2019-10-13T14:38:47.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:47.956+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.960047+0000) 2019-10-13T14:38:47.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:47.956+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.959739+0000) 2019-10-13T14:38:47.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:47.956+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.960611+0000) 2019-10-13T14:38:47.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:47.956+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.961130+0000) 2019-10-13T14:38:47.971 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:47.965+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:47.968583+0000) 2019-10-13T14:38:47.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:47.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:48.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:48.028+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:48.029840+0000) 2019-10-13T14:38:48.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:48.252+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:48.256175+0000) 2019-10-13T14:38:48.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:48.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:48.794869+0000) 2019-10-13T14:38:48.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:48.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:48.796557+0000) 2019-10-13T14:38:50.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:50.565+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:50.568937+0000) 2019-10-13T14:38:51.780 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:51.776+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:51.778000+0000) 2019-10-13T14:38:52.976 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:52.968+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.973148+0000) 2019-10-13T14:38:52.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:52.972+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.974243+0000) 2019-10-13T14:38:52.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:52.972+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.975083+0000) 2019-10-13T14:38:52.978 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:52.972+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.975339+0000) 2019-10-13T14:38:52.978 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:52.972+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.976224+0000) 2019-10-13T14:38:52.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:52.972+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.976410+0000) 2019-10-13T14:38:52.979 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:52.973+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.977172+0000) 2019-10-13T14:38:52.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:52.977+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:52.980245+0000) 2019-10-13T14:38:52.986 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:52.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209231 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:54.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:54.372+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:54.376480+0000) 2019-10-13T14:38:54.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:54.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:54.661178+0000) 2019-10-13T14:38:57.989 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:57.984+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.986350+0000) 2019-10-13T14:38:57.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:57.984+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.987808+0000) 2019-10-13T14:38:57.991 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:57.984+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.988662+0000) 2019-10-13T14:38:57.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:38:57.984+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.988212+0000) 2019-10-13T14:38:57.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:57.985+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.989747+0000) 2019-10-13T14:38:57.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:38:57.988+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.990029+0000) 2019-10-13T14:38:57.993 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:38:57.988+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.990433+0000) 2019-10-13T14:38:57.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:38:57.989+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:57.993826+0000) 2019-10-13T14:38:57.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:38:57.992+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:38:58.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:38:58.028+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:58.029998+0000) 2019-10-13T14:38:58.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:38:58.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:58.256348+0000) 2019-10-13T14:38:58.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:38:58.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:58.795041+0000) 2019-10-13T14:38:58.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:38:58.793+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:38:58.796748+0000) 2019-10-13T14:39:00.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:00.573+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:00.575445+0000) 2019-10-13T14:39:01.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:01.776+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:01.778256+0000) 2019-10-13T14:39:03.002 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:02.996+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:02.999508+0000) 2019-10-13T14:39:03.003 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:02.996+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.000249+0000) 2019-10-13T14:39:03.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:02.996+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.001341+0000) 2019-10-13T14:39:03.004 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:02.997+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.001972+0000) 2019-10-13T14:39:03.005 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:02.996+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.001707+0000) 2019-10-13T14:39:03.005 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:03.000+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.002806+0000) 2019-10-13T14:39:03.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:03.000+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.002986+0000) 2019-10-13T14:39:03.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:03.005+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:03.008453+0000) 2019-10-13T14:39:03.013 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:03.008+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209293 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:04.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:04.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:04.376720+0000) 2019-10-13T14:39:04.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:04.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:04.661413+0000) 2019-10-13T14:39:08.017 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:08.012+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.014524+0000) 2019-10-13T14:39:08.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:08.012+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.014885+0000) 2019-10-13T14:39:08.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:08.012+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.015381+0000) 2019-10-13T14:39:08.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:08.012+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.016439+0000) 2019-10-13T14:39:08.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:08.013+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.016870+0000) 2019-10-13T14:39:08.020 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:08.012+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.017496+0000) 2019-10-13T14:39:08.020 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:08.016+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.017798+0000) 2019-10-13T14:39:08.026 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:08.017+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.023670+0000) 2019-10-13T14:39:08.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:08.020+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:08.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:08.028+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.030116+0000) 2019-10-13T14:39:08.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:08.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.256544+0000) 2019-10-13T14:39:08.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:08.792+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.795213+0000) 2019-10-13T14:39:08.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:08.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:08.796920+0000) 2019-10-13T14:39:09.605 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:39:09.605 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:09.609 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:39:09.609 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:10.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:10.576+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:10.582034+0000) 2019-10-13T14:39:11.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:11.776+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:11.778500+0000) 2019-10-13T14:39:13.031 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:13.024+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.028353+0000) 2019-10-13T14:39:13.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:13.024+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.028667+0000) 2019-10-13T14:39:13.032 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:13.024+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.028895+0000) 2019-10-13T14:39:13.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:13.024+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.029528+0000) 2019-10-13T14:39:13.034 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:13.028+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.031213+0000) 2019-10-13T14:39:13.035 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:13.028+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.033168+0000) 2019-10-13T14:39:13.037 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:13.028+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.034493+0000) 2019-10-13T14:39:13.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:13.032+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:13.034894+0000) 2019-10-13T14:39:13.040 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:13.036+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209353 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:14.379 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:14.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:14.376919+0000) 2019-10-13T14:39:14.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:14.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:14.661615+0000) 2019-10-13T14:39:18.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:18.027+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.030300+0000) 2019-10-13T14:39:18.045 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:18.039+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.042661+0000) 2019-10-13T14:39:18.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:18.039+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.044173+0000) 2019-10-13T14:39:18.047 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:18.040+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.044522+0000) 2019-10-13T14:39:18.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:18.039+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.045182+0000) 2019-10-13T14:39:18.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:18.039+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.045615+0000) 2019-10-13T14:39:18.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:18.043+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.047037+0000) 2019-10-13T14:39:18.050 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:18.043+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.048153+0000) 2019-10-13T14:39:18.053 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:18.047+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:18.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:18.048+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.053459+0000) 2019-10-13T14:39:18.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:18.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.256720+0000) 2019-10-13T14:39:18.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:18.791+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.795380+0000) 2019-10-13T14:39:18.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:18.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:18.797109+0000) 2019-10-13T14:39:20.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:20.584+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:20.588551+0000) 2019-10-13T14:39:21.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:21.775+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:21.778719+0000) 2019-10-13T14:39:23.057 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:23.051+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.054637+0000) 2019-10-13T14:39:23.058 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:23.051+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.054843+0000) 2019-10-13T14:39:23.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:23.051+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.055960+0000) 2019-10-13T14:39:23.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:23.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.057190+0000) 2019-10-13T14:39:23.060 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:23.051+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.057642+0000) 2019-10-13T14:39:23.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:23.051+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.057204+0000) 2019-10-13T14:39:23.061 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:23.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.058722+0000) 2019-10-13T14:39:23.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:23.059+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209415 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:23.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:23.060+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:23.065733+0000) 2019-10-13T14:39:24.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:24.371+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:24.377145+0000) 2019-10-13T14:39:24.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:24.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:24.661849+0000) 2019-10-13T14:39:28.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:28.027+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.030471+0000) 2019-10-13T14:39:28.071 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:28.063+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.068469+0000) 2019-10-13T14:39:28.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:28.067+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.070433+0000) 2019-10-13T14:39:28.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:28.067+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.072152+0000) 2019-10-13T14:39:28.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:28.067+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.072580+0000) 2019-10-13T14:39:28.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:28.067+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.073571+0000) 2019-10-13T14:39:28.076 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:28.071+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.074092+0000) 2019-10-13T14:39:28.077 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:28.068+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.074516+0000) 2019-10-13T14:39:28.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:28.072+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.077226+0000) 2019-10-13T14:39:28.080 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:28.071+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209443 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:28.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:28.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.256903+0000) 2019-10-13T14:39:28.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:28.791+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.795511+0000) 2019-10-13T14:39:28.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:28.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:28.797282+0000) 2019-10-13T14:39:30.601 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:30.596+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:30.598643+0000) 2019-10-13T14:39:31.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:31.775+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:31.778927+0000) 2019-10-13T14:39:33.086 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:33.079+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.083668+0000) 2019-10-13T14:39:33.087 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:33.080+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.084579+0000) 2019-10-13T14:39:33.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:33.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.084201+0000) 2019-10-13T14:39:33.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:33.083+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.086875+0000) 2019-10-13T14:39:33.090 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:33.083+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.087203+0000) 2019-10-13T14:39:33.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:33.083+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.087781+0000) 2019-10-13T14:39:33.091 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:33.083+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.088581+0000) 2019-10-13T14:39:33.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:33.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209475 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:33.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:33.088+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:33.094284+0000) 2019-10-13T14:39:34.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:34.375+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:34.377364+0000) 2019-10-13T14:39:34.664 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:34.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:34.662048+0000) 2019-10-13T14:39:38.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:38.027+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.030704+0000) 2019-10-13T14:39:38.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:38.091+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.095725+0000) 2019-10-13T14:39:38.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:38.091+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.096239+0000) 2019-10-13T14:39:38.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:38.092+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.097122+0000) 2019-10-13T14:39:38.100 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:38.091+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.097602+0000) 2019-10-13T14:39:38.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:38.095+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.099462+0000) 2019-10-13T14:39:38.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:38.095+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.100066+0000) 2019-10-13T14:39:38.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:38.099+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.101986+0000) 2019-10-13T14:39:38.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:38.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:38.112 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:38.104+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.109668+0000) 2019-10-13T14:39:38.259 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:38.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.257119+0000) 2019-10-13T14:39:38.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:38.791+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.795724+0000) 2019-10-13T14:39:38.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:38.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:38.797406+0000) 2019-10-13T14:39:39.712 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:39:39.712 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:39.715 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:39:39.716 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:39:40.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:40.604+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:40.605204+0000) 2019-10-13T14:39:41.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:41.775+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:41.779166+0000) 2019-10-13T14:39:43.113 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:43.107+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.110472+0000) 2019-10-13T14:39:43.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:43.107+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.111050+0000) 2019-10-13T14:39:43.116 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:43.107+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.113544+0000) 2019-10-13T14:39:43.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:43.107+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.113683+0000) 2019-10-13T14:39:43.117 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:43.111+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.114951+0000) 2019-10-13T14:39:43.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:43.108+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.115343+0000) 2019-10-13T14:39:43.118 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:43.112+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.116378+0000) 2019-10-13T14:39:43.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:43.115+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:43.117811+0000) 2019-10-13T14:39:43.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:43.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209537 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:44.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:44.371+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:44.377579+0000) 2019-10-13T14:39:44.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:44.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:44.662266+0000) 2019-10-13T14:39:48.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:48.027+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.030859+0000) 2019-10-13T14:39:48.127 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:48.120+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.124356+0000) 2019-10-13T14:39:48.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:48.119+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.124731+0000) 2019-10-13T14:39:48.128 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:48.119+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.125081+0000) 2019-10-13T14:39:48.128 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:48.123+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.125866+0000) 2019-10-13T14:39:48.129 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:48.123+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.126269+0000) 2019-10-13T14:39:48.130 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:48.123+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.127476+0000) 2019-10-13T14:39:48.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:48.123+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.127962+0000) 2019-10-13T14:39:48.132 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:48.124+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.130141+0000) 2019-10-13T14:39:48.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:48.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209565 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:48.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:48.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.257329+0000) 2019-10-13T14:39:48.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:48.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.795897+0000) 2019-10-13T14:39:48.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:48.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:48.797570+0000) 2019-10-13T14:39:50.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:50.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:50.611674+0000) 2019-10-13T14:39:51.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:51.775+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:51.779366+0000) 2019-10-13T14:39:53.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:53.135+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.137964+0000) 2019-10-13T14:39:53.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:53.135+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.139152+0000) 2019-10-13T14:39:53.142 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:53.135+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.139162+0000) 2019-10-13T14:39:53.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:53.135+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.139586+0000) 2019-10-13T14:39:53.143 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:53.136+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.140729+0000) 2019-10-13T14:39:53.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:53.135+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.141169+0000) 2019-10-13T14:39:53.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:53.135+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.141509+0000) 2019-10-13T14:39:53.151 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:53.144+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:53.148909+0000) 2019-10-13T14:39:53.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:53.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:54.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:54.375+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:54.377842+0000) 2019-10-13T14:39:54.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:54.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:54.662482+0000) 2019-10-13T14:39:58.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:58.027+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.031059+0000) 2019-10-13T14:39:58.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:39:58.147+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.152568+0000) 2019-10-13T14:39:58.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:58.151+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.154238+0000) 2019-10-13T14:39:58.157 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:39:58.151+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.154396+0000) 2019-10-13T14:39:58.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:58.148+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.154674+0000) 2019-10-13T14:39:58.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:58.151+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.155537+0000) 2019-10-13T14:39:58.158 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:39:58.151+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.155782+0000) 2019-10-13T14:39:58.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:39:58.151+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.156761+0000) 2019-10-13T14:39:58.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:39:58.159+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209626 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:39:58.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:39:58.156+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.163668+0000) 2019-10-13T14:39:58.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:39:58.251+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.257505+0000) 2019-10-13T14:39:58.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:39:58.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.796100+0000) 2019-10-13T14:39:58.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:39:58.792+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:39:58.797744+0000) 2019-10-13T14:40:00.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:00.616+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:00.618223+0000) 2019-10-13T14:40:01.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:01.779+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:01.779570+0000) 2019-10-13T14:40:03.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:03.160+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.167490+0000) 2019-10-13T14:40:03.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:03.163+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.167683+0000) 2019-10-13T14:40:03.171 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:03.163+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.167924+0000) 2019-10-13T14:40:03.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:03.163+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.168622+0000) 2019-10-13T14:40:03.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:03.163+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.168408+0000) 2019-10-13T14:40:03.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:03.163+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.169582+0000) 2019-10-13T14:40:03.172 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:03.167+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.170050+0000) 2019-10-13T14:40:03.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:03.168+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:03.173582+0000) 2019-10-13T14:40:03.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:03.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:04.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:04.375+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:04.378061+0000) 2019-10-13T14:40:04.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:04.659+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:04.662707+0000) 2019-10-13T14:40:08.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:08.031+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.031262+0000) 2019-10-13T14:40:08.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:08.175+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.181544+0000) 2019-10-13T14:40:08.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:08.179+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.182619+0000) 2019-10-13T14:40:08.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:08.179+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.182961+0000) 2019-10-13T14:40:08.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:08.179+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.183437+0000) 2019-10-13T14:40:08.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:08.180+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.183797+0000) 2019-10-13T14:40:08.187 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:08.179+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.183938+0000) 2019-10-13T14:40:08.187 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:08.179+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.184533+0000) 2019-10-13T14:40:08.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:08.188+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.191678+0000) 2019-10-13T14:40:08.195 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:08.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209684 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:08.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:08.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.257683+0000) 2019-10-13T14:40:08.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:08.791+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.796242+0000) 2019-10-13T14:40:08.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:08.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:08.797915+0000) 2019-10-13T14:40:09.843 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:40:09.843 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:09.847 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:40:09.847 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:10.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:10.624+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:10.624638+0000) 2019-10-13T14:40:11.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:11.774+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:11.779799+0000) 2019-10-13T14:40:13.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:13.190+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.196713+0000) 2019-10-13T14:40:13.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:13.190+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.197557+0000) 2019-10-13T14:40:13.202 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:13.196+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.199881+0000) 2019-10-13T14:40:13.204 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:13.198+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.201935+0000) 2019-10-13T14:40:13.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:13.200+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.203463+0000) 2019-10-13T14:40:13.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:13.198+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.203903+0000) 2019-10-13T14:40:13.208 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:13.198+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.205453+0000) 2019-10-13T14:40:13.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:13.202+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:13.207229+0000) 2019-10-13T14:40:13.210 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:13.202+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209714 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:14.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:14.374+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:14.378265+0000) 2019-10-13T14:40:14.665 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:14.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:14.662908+0000) 2019-10-13T14:40:18.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:18.030+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.031477+0000) 2019-10-13T14:40:18.214 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:18.206+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.211589+0000) 2019-10-13T14:40:18.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:18.207+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.211948+0000) 2019-10-13T14:40:18.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:18.206+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.213328+0000) 2019-10-13T14:40:18.216 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:18.206+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.213721+0000) 2019-10-13T14:40:18.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:18.210+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.214025+0000) 2019-10-13T14:40:18.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:18.210+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.215092+0000) 2019-10-13T14:40:18.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:18.210+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.215498+0000) 2019-10-13T14:40:18.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:18.219+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.220563+0000) 2019-10-13T14:40:18.224 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:18.218+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209744 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:18.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:18.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.257846+0000) 2019-10-13T14:40:18.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:18.794+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.796446+0000) 2019-10-13T14:40:18.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:18.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:18.798107+0000) 2019-10-13T14:40:20.633 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:20.627+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:20.631179+0000) 2019-10-13T14:40:21.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:21.778+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:21.780019+0000) 2019-10-13T14:40:23.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:23.223+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.225203+0000) 2019-10-13T14:40:23.228 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:23.218+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.225551+0000) 2019-10-13T14:40:23.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:23.222+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.226704+0000) 2019-10-13T14:40:23.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:23.222+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.227198+0000) 2019-10-13T14:40:23.230 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:23.222+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.227776+0000) 2019-10-13T14:40:23.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:23.222+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.228102+0000) 2019-10-13T14:40:23.231 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:23.222+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.228489+0000) 2019-10-13T14:40:23.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:23.231+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:23.233215+0000) 2019-10-13T14:40:23.238 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:23.230+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209774 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:24.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:24.374+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:24.378496+0000) 2019-10-13T14:40:24.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:24.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:24.663153+0000) 2019-10-13T14:40:28.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:28.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.031693+0000) 2019-10-13T14:40:28.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:28.234+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.240520+0000) 2019-10-13T14:40:28.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:28.234+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.241102+0000) 2019-10-13T14:40:28.244 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:28.234+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.241526+0000) 2019-10-13T14:40:28.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:28.239+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.241929+0000) 2019-10-13T14:40:28.245 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:28.238+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.242739+0000) 2019-10-13T14:40:28.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:28.238+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.243151+0000) 2019-10-13T14:40:28.246 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:28.238+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.243553+0000) 2019-10-13T14:40:28.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:28.247+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.248635+0000) 2019-10-13T14:40:28.252 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:28.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209801 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:28.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:28.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.257969+0000) 2019-10-13T14:40:28.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:28.790+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.796655+0000) 2019-10-13T14:40:28.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:28.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:28.798285+0000) 2019-10-13T14:40:30.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:30.635+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:30.637636+0000) 2019-10-13T14:40:31.783 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:31.774+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:31.780243+0000) 2019-10-13T14:40:33.256 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:33.246+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.253494+0000) 2019-10-13T14:40:33.257 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:33.250+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.255193+0000) 2019-10-13T14:40:33.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:33.250+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.255783+0000) 2019-10-13T14:40:33.260 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:33.255+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.257607+0000) 2019-10-13T14:40:33.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:33.254+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.257829+0000) 2019-10-13T14:40:33.261 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:33.254+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.259190+0000) 2019-10-13T14:40:33.264 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:33.254+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.261614+0000) 2019-10-13T14:40:33.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:33.258+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209834 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:33.269 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:33.263+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:33.266440+0000) 2019-10-13T14:40:34.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:34.378+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:34.378701+0000) 2019-10-13T14:40:34.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:34.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:34.663347+0000) 2019-10-13T14:40:38.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:38.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.031896+0000) 2019-10-13T14:40:38.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:38.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.258122+0000) 2019-10-13T14:40:38.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:38.262+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.268651+0000) 2019-10-13T14:40:38.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:38.262+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.268975+0000) 2019-10-13T14:40:38.273 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:38.266+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.270682+0000) 2019-10-13T14:40:38.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:38.266+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.271124+0000) 2019-10-13T14:40:38.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:38.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.271445+0000) 2019-10-13T14:40:38.274 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:38.266+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.272280+0000) 2019-10-13T14:40:38.276 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:38.271+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.273452+0000) 2019-10-13T14:40:38.279 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:38.270+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209861 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:38.283 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:38.279+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.280984+0000) 2019-10-13T14:40:38.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:38.794+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.796837+0000) 2019-10-13T14:40:38.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:38.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:38.798492+0000) 2019-10-13T14:40:39.915 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:40:39.916 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:39.919 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:40:39.919 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:40:40.646 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:40.643+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:40.644115+0000) 2019-10-13T14:40:41.783 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:41.774+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:41.780452+0000) 2019-10-13T14:40:43.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:43.274+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.280276+0000) 2019-10-13T14:40:43.283 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:43.274+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.280887+0000) 2019-10-13T14:40:43.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:43.274+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.281313+0000) 2019-10-13T14:40:43.286 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:43.278+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.284048+0000) 2019-10-13T14:40:43.288 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:43.283+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.285816+0000) 2019-10-13T14:40:43.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:43.282+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.286302+0000) 2019-10-13T14:40:43.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:43.282+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.286714+0000) 2019-10-13T14:40:43.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:43.287+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:43.289982+0000) 2019-10-13T14:40:43.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:43.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209892 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:44.381 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:44.374+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:44.378902+0000) 2019-10-13T14:40:44.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:44.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:44.663601+0000) 2019-10-13T14:40:48.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:48.030+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.032094+0000) 2019-10-13T14:40:48.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:48.254+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.258331+0000) 2019-10-13T14:40:48.299 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:48.290+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.296717+0000) 2019-10-13T14:40:48.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:48.290+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.297385+0000) 2019-10-13T14:40:48.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:48.294+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.297794+0000) 2019-10-13T14:40:48.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:48.294+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.298001+0000) 2019-10-13T14:40:48.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:48.294+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.300278+0000) 2019-10-13T14:40:48.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:48.298+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.302470+0000) 2019-10-13T14:40:48.306 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:48.303+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.304334+0000) 2019-10-13T14:40:48.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:48.302+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:48.311 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:48.307+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.309115+0000) 2019-10-13T14:40:48.799 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:48.794+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.797030+0000) 2019-10-13T14:40:48.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:48.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:48.798666+0000) 2019-10-13T14:40:50.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:50.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:50.650516+0000) 2019-10-13T14:40:51.783 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:51.778+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:51.780672+0000) 2019-10-13T14:40:53.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:53.302+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.309278+0000) 2019-10-13T14:40:53.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:53.306+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.309949+0000) 2019-10-13T14:40:53.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:53.306+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.310595+0000) 2019-10-13T14:40:53.313 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:53.307+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.310946+0000) 2019-10-13T14:40:53.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:53.306+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.311095+0000) 2019-10-13T14:40:53.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:53.306+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.311788+0000) 2019-10-13T14:40:53.315 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:53.306+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.313273+0000) 2019-10-13T14:40:53.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:53.315+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:53.318139+0000) 2019-10-13T14:40:53.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:53.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:54.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:54.378+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:54.379104+0000) 2019-10-13T14:40:54.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:54.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:54.663817+0000) 2019-10-13T14:40:58.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:58.026+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.032289+0000) 2019-10-13T14:40:58.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:58.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.258525+0000) 2019-10-13T14:40:58.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:40:58.318+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.323998+0000) 2019-10-13T14:40:58.327 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:40:58.318+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.324450+0000) 2019-10-13T14:40:58.328 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:58.323+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.325933+0000) 2019-10-13T14:40:58.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:58.322+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.326391+0000) 2019-10-13T14:40:58.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:40:58.322+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.326486+0000) 2019-10-13T14:40:58.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:40:58.322+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.327358+0000) 2019-10-13T14:40:58.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:40:58.322+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.327510+0000) 2019-10-13T14:40:58.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:40:58.331+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.332893+0000) 2019-10-13T14:40:58.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:40:58.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:40:58.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:40:58.794+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.797225+0000) 2019-10-13T14:40:58.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:40:58.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:40:58.798836+0000) 2019-10-13T14:41:00.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:00.655+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:00.657133+0000) 2019-10-13T14:41:01.783 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:01.778+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:01.780870+0000) 2019-10-13T14:41:03.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:03.330+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.337483+0000) 2019-10-13T14:41:03.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:03.330+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.337572+0000) 2019-10-13T14:41:03.341 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:03.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.338069+0000) 2019-10-13T14:41:03.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:03.334+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.340434+0000) 2019-10-13T14:41:03.343 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:03.339+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.340895+0000) 2019-10-13T14:41:03.345 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:03.338+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.342873+0000) 2019-10-13T14:41:03.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:03.338+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.344724+0000) 2019-10-13T14:41:03.350 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:03.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210009 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:03.355 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:03.351+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:03.353018+0000) 2019-10-13T14:41:04.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:04.378+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:04.379335+0000) 2019-10-13T14:41:04.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:04.658+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:04.664050+0000) 2019-10-13T14:41:08.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:08.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.032488+0000) 2019-10-13T14:41:08.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:08.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.258752+0000) 2019-10-13T14:41:08.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:08.345+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.350853+0000) 2019-10-13T14:41:08.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:08.345+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.351544+0000) 2019-10-13T14:41:08.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:08.345+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.352729+0000) 2019-10-13T14:41:08.355 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:08.345+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.352988+0000) 2019-10-13T14:41:08.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:08.351+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.353633+0000) 2019-10-13T14:41:08.356 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:08.349+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.353991+0000) 2019-10-13T14:41:08.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:08.349+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.354542+0000) 2019-10-13T14:41:08.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:08.353+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:08.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:08.359+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.362069+0000) 2019-10-13T14:41:08.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:08.793+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.797403+0000) 2019-10-13T14:41:08.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:08.795+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:08.799029+0000) 2019-10-13T14:41:10.039 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:41:10.039 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:10.043 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:41:10.043 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:10.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:10.659+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:10.663544+0000) 2019-10-13T14:41:11.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:11.777+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:11.781070+0000) 2019-10-13T14:41:13.369 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:13.361+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.365804+0000) 2019-10-13T14:41:13.369 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:13.361+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.366264+0000) 2019-10-13T14:41:13.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:13.363+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.366728+0000) 2019-10-13T14:41:13.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:13.361+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.367958+0000) 2019-10-13T14:41:13.371 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:13.361+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.368301+0000) 2019-10-13T14:41:13.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:13.365+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.371988+0000) 2019-10-13T14:41:13.376 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:13.369+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.373988+0000) 2019-10-13T14:41:13.378 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:13.369+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210069 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:13.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:13.375+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:13.376536+0000) 2019-10-13T14:41:14.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:14.373+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:14.379545+0000) 2019-10-13T14:41:14.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:14.657+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:14.664249+0000) 2019-10-13T14:41:18.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:18.025+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.032716+0000) 2019-10-13T14:41:18.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:18.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.258950+0000) 2019-10-13T14:41:18.382 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:18.375+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.379623+0000) 2019-10-13T14:41:18.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:18.373+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.379895+0000) 2019-10-13T14:41:18.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:18.373+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.380303+0000) 2019-10-13T14:41:18.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:18.377+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.382721+0000) 2019-10-13T14:41:18.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:18.377+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.382985+0000) 2019-10-13T14:41:18.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:18.377+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.383639+0000) 2019-10-13T14:41:18.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:18.377+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.383819+0000) 2019-10-13T14:41:18.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:18.387+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.388582+0000) 2019-10-13T14:41:18.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:18.385+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210098 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:18.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:18.793+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.797584+0000) 2019-10-13T14:41:18.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:18.799+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:18.799222+0000) 2019-10-13T14:41:20.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:20.666+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:20.670076+0000) 2019-10-13T14:41:21.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:21.777+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:21.781306+0000) 2019-10-13T14:41:23.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:23.390+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.394699+0000) 2019-10-13T14:41:23.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:23.393+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.396484+0000) 2019-10-13T14:41:23.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:23.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.397186+0000) 2019-10-13T14:41:23.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:23.397+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.397780+0000) 2019-10-13T14:41:23.401 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:23.397+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.398619+0000) 2019-10-13T14:41:23.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:23.397+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.399150+0000) 2019-10-13T14:41:23.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:23.397+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.399558+0000) 2019-10-13T14:41:23.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:23.398+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:23.403384+0000) 2019-10-13T14:41:23.408 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:23.405+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:24.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:24.377+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:24.379783+0000) 2019-10-13T14:41:24.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:24.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:24.664454+0000) 2019-10-13T14:41:28.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:28.029+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.032915+0000) 2019-10-13T14:41:28.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:28.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.259176+0000) 2019-10-13T14:41:28.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:28.409+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.409712+0000) 2019-10-13T14:41:28.413 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:28.409+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.410480+0000) 2019-10-13T14:41:28.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:28.409+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.410162+0000) 2019-10-13T14:41:28.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:28.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.411382+0000) 2019-10-13T14:41:28.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:28.409+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.410979+0000) 2019-10-13T14:41:28.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:28.409+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.411625+0000) 2019-10-13T14:41:28.415 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:28.409+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.411841+0000) 2019-10-13T14:41:28.422 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:28.414+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.419790+0000) 2019-10-13T14:41:28.422 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:28.417+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210157 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:28.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:28.797+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.797790+0000) 2019-10-13T14:41:28.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:28.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:28.799394+0000) 2019-10-13T14:41:30.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:30.674+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:30.676662+0000) 2019-10-13T14:41:31.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:31.777+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:31.781507+0000) 2019-10-13T14:41:33.426 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:33.421+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.424020+0000) 2019-10-13T14:41:33.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:33.421+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.424914+0000) 2019-10-13T14:41:33.428 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:33.421+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.425597+0000) 2019-10-13T14:41:33.428 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:33.422+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.426210+0000) 2019-10-13T14:41:33.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:33.425+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.426350+0000) 2019-10-13T14:41:33.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:33.425+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.427538+0000) 2019-10-13T14:41:33.430 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:33.425+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.427752+0000) 2019-10-13T14:41:33.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:33.430+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:33.433906+0000) 2019-10-13T14:41:33.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:33.433+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:34.382 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:34.377+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:34.379969+0000) 2019-10-13T14:41:34.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:34.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:34.664642+0000) 2019-10-13T14:41:38.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:38.029+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.033127+0000) 2019-10-13T14:41:38.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:38.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.259372+0000) 2019-10-13T14:41:38.439 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:38.433+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.437008+0000) 2019-10-13T14:41:38.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:38.437+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.437903+0000) 2019-10-13T14:41:38.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:38.437+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.439534+0000) 2019-10-13T14:41:38.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:38.437+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.439474+0000) 2019-10-13T14:41:38.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:38.437+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.440157+0000) 2019-10-13T14:41:38.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:38.437+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.440571+0000) 2019-10-13T14:41:38.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:38.438+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.440880+0000) 2019-10-13T14:41:38.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:38.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.443758+0000) 2019-10-13T14:41:38.450 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:38.445+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:38.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:38.797+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.798025+0000) 2019-10-13T14:41:38.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:38.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:38.799567+0000) 2019-10-13T14:41:40.137 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:41:40.137 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:40.141 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:41:40.141 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:41:40.686 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:40.682+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:40.683148+0000) 2019-10-13T14:41:41.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:41.781+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:41.781711+0000) 2019-10-13T14:41:43.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:43.446+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.451077+0000) 2019-10-13T14:41:43.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:43.449+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.452197+0000) 2019-10-13T14:41:43.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:43.449+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.452475+0000) 2019-10-13T14:41:43.455 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:43.449+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.452988+0000) 2019-10-13T14:41:43.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:43.449+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.453439+0000) 2019-10-13T14:41:43.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:43.453+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.453787+0000) 2019-10-13T14:41:43.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:43.453+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.454366+0000) 2019-10-13T14:41:43.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:43.454+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:43.457659+0000) 2019-10-13T14:41:43.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:43.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:44.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:44.377+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:44.380180+0000) 2019-10-13T14:41:44.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:44.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:44.664871+0000) 2019-10-13T14:41:48.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:48.033+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.033324+0000) 2019-10-13T14:41:48.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:48.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.259569+0000) 2019-10-13T14:41:48.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:48.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.464629+0000) 2019-10-13T14:41:48.468 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:48.461+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.464820+0000) 2019-10-13T14:41:48.468 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:48.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.465316+0000) 2019-10-13T14:41:48.468 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:48.462+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.466096+0000) 2019-10-13T14:41:48.469 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:48.465+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.466363+0000) 2019-10-13T14:41:48.469 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:48.465+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.466838+0000) 2019-10-13T14:41:48.470 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:48.465+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.467218+0000) 2019-10-13T14:41:48.473 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:48.466+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.471389+0000) 2019-10-13T14:41:48.476 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:48.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210277 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:48.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:48.797+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.798205+0000) 2019-10-13T14:41:48.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:48.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:48.799741+0000) 2019-10-13T14:41:50.697 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:50.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:50.694846+0000) 2019-10-13T14:41:51.784 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:51.781+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:51.781961+0000) 2019-10-13T14:41:53.480 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:53.473+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.477344+0000) 2019-10-13T14:41:53.480 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:53.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.478127+0000) 2019-10-13T14:41:53.481 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:53.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.479002+0000) 2019-10-13T14:41:53.483 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:53.477+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.480672+0000) 2019-10-13T14:41:53.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:53.478+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.482806+0000) 2019-10-13T14:41:53.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:53.481+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.484655+0000) 2019-10-13T14:41:53.487 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:53.482+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.485459+0000) 2019-10-13T14:41:53.489 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:53.485+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:53.486416+0000) 2019-10-13T14:41:53.489 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:53.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210309 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:54.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:54.377+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:54.380359+0000) 2019-10-13T14:41:54.667 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:54.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:54.665071+0000) 2019-10-13T14:41:58.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:58.033+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.033527+0000) 2019-10-13T14:41:58.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:58.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.259778+0000) 2019-10-13T14:41:58.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:41:58.489+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.489884+0000) 2019-10-13T14:41:58.493 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:58.486+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.491049+0000) 2019-10-13T14:41:58.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:41:58.489+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.491499+0000) 2019-10-13T14:41:58.494 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:41:58.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.492143+0000) 2019-10-13T14:41:58.495 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:41:58.489+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.493018+0000) 2019-10-13T14:41:58.496 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:58.489+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.493303+0000) 2019-10-13T14:41:58.496 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:41:58.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.494391+0000) 2019-10-13T14:41:58.501 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:41:58.494+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.498779+0000) 2019-10-13T14:41:58.504 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:41:58.497+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:41:58.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:41:58.797+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.798408+0000) 2019-10-13T14:41:58.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:41:58.794+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:41:58.799911+0000) 2019-10-13T14:42:00.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:00.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:00.701612+0000) 2019-10-13T14:42:01.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:01.781+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:01.782207+0000) 2019-10-13T14:42:03.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:03.500+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.504091+0000) 2019-10-13T14:42:03.507 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:03.500+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.504641+0000) 2019-10-13T14:42:03.508 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:03.500+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.505318+0000) 2019-10-13T14:42:03.510 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:03.502+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.507783+0000) 2019-10-13T14:42:03.511 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:03.504+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.508627+0000) 2019-10-13T14:42:03.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:03.508+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.510623+0000) 2019-10-13T14:42:03.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:03.508+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.511618+0000) 2019-10-13T14:42:03.515 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:03.510+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:03.512662+0000) 2019-10-13T14:42:03.517 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:03.512+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:04.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:04.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:04.380551+0000) 2019-10-13T14:42:04.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:04.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:04.665222+0000) 2019-10-13T14:42:08.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:08.028+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.033720+0000) 2019-10-13T14:42:08.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:08.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.259964+0000) 2019-10-13T14:42:08.520 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:08.516+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.518041+0000) 2019-10-13T14:42:08.522 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:08.516+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.519349+0000) 2019-10-13T14:42:08.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:08.516+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.519785+0000) 2019-10-13T14:42:08.523 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:08.516+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.520089+0000) 2019-10-13T14:42:08.523 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:08.516+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.521092+0000) 2019-10-13T14:42:08.525 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:08.520+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.522621+0000) 2019-10-13T14:42:08.528 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:08.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.526051+0000) 2019-10-13T14:42:08.531 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:08.524+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210397 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:08.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:08.526+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.529902+0000) 2019-10-13T14:42:08.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:08.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.798584+0000) 2019-10-13T14:42:08.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:08.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:08.800077+0000) 2019-10-13T14:42:10.235 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:42:10.236 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:10.239 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:42:10.240 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:10.711 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:10.706+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:10.708209+0000) 2019-10-13T14:42:11.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:11.780+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:11.782403+0000) 2019-10-13T14:42:13.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:13.528+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.531852+0000) 2019-10-13T14:42:13.535 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:13.528+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.533222+0000) 2019-10-13T14:42:13.536 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:13.528+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.533660+0000) 2019-10-13T14:42:13.536 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:13.530+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.534087+0000) 2019-10-13T14:42:13.537 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:13.532+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.534541+0000) 2019-10-13T14:42:13.537 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:13.532+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.535110+0000) 2019-10-13T14:42:13.538 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:13.532+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.535830+0000) 2019-10-13T14:42:13.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:13.538+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:13.541763+0000) 2019-10-13T14:42:13.545 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:13.540+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:14.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:14.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:14.380767+0000) 2019-10-13T14:42:14.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:14.660+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:14.665420+0000) 2019-10-13T14:42:18.036 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:18.032+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.033951+0000) 2019-10-13T14:42:18.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:18.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.260160+0000) 2019-10-13T14:42:18.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:18.544+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.546191+0000) 2019-10-13T14:42:18.549 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:18.544+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.547191+0000) 2019-10-13T14:42:18.551 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:18.544+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.549245+0000) 2019-10-13T14:42:18.552 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:18.544+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.549501+0000) 2019-10-13T14:42:18.555 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:18.548+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.552519+0000) 2019-10-13T14:42:18.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:18.550+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.555078+0000) 2019-10-13T14:42:18.558 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:18.552+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.556049+0000) 2019-10-13T14:42:18.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:18.554+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.556924+0000) 2019-10-13T14:42:18.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:18.552+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210458 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:18.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:18.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.798759+0000) 2019-10-13T14:42:18.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:18.798+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:18.800283+0000) 2019-10-13T14:42:20.717 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:20.710+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:20.714828+0000) 2019-10-13T14:42:21.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:21.780+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:21.782635+0000) 2019-10-13T14:42:23.564 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:23.556+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.561474+0000) 2019-10-13T14:42:23.564 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:23.560+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.561952+0000) 2019-10-13T14:42:23.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:23.560+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.562930+0000) 2019-10-13T14:42:23.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:23.560+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.564563+0000) 2019-10-13T14:42:23.567 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:23.560+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.564879+0000) 2019-10-13T14:42:23.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:23.564+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.566668+0000) 2019-10-13T14:42:23.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:23.562+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.567121+0000) 2019-10-13T14:42:23.574 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:23.568+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:23.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:23.570+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:23.573589+0000) 2019-10-13T14:42:24.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:24.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:24.380949+0000) 2019-10-13T14:42:24.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:24.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:24.665620+0000) 2019-10-13T14:42:28.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:28.032+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.034204+0000) 2019-10-13T14:42:28.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:28.256+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.260396+0000) 2019-10-13T14:42:28.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:28.569+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.574985+0000) 2019-10-13T14:42:28.578 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:28.572+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.575397+0000) 2019-10-13T14:42:28.578 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:28.572+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.575802+0000) 2019-10-13T14:42:28.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:28.576+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.578564+0000) 2019-10-13T14:42:28.581 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:28.576+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.578795+0000) 2019-10-13T14:42:28.582 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:28.576+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.579547+0000) 2019-10-13T14:42:28.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:28.576+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.581217+0000) 2019-10-13T14:42:28.585 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:28.577+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.583153+0000) 2019-10-13T14:42:28.587 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:28.580+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210517 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:28.801 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:28.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.798935+0000) 2019-10-13T14:42:28.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:28.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:28.800451+0000) 2019-10-13T14:42:30.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:30.717+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:30.721498+0000) 2019-10-13T14:42:31.785 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:31.780+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:31.782855+0000) 2019-10-13T14:42:33.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:33.585+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.588314+0000) 2019-10-13T14:42:33.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:33.584+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.588570+0000) 2019-10-13T14:42:33.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:33.584+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.589112+0000) 2019-10-13T14:42:33.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:33.584+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.589584+0000) 2019-10-13T14:42:33.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:33.588+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.590304+0000) 2019-10-13T14:42:33.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:33.588+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.591028+0000) 2019-10-13T14:42:33.594 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:33.588+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.591632+0000) 2019-10-13T14:42:33.600 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:33.593+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:33.597662+0000) 2019-10-13T14:42:33.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:33.596+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:34.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:34.380+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:34.381163+0000) 2019-10-13T14:42:34.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:34.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:34.665818+0000) 2019-10-13T14:42:38.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:38.032+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.034400+0000) 2019-10-13T14:42:38.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:38.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.260616+0000) 2019-10-13T14:42:38.604 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:38.597+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.602023+0000) 2019-10-13T14:42:38.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:38.600+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.602797+0000) 2019-10-13T14:42:38.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:38.600+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.603056+0000) 2019-10-13T14:42:38.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:38.600+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.603443+0000) 2019-10-13T14:42:38.606 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:38.600+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.604393+0000) 2019-10-13T14:42:38.607 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:38.600+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.604749+0000) 2019-10-13T14:42:38.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:38.600+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.605377+0000) 2019-10-13T14:42:38.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:38.609+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.612228+0000) 2019-10-13T14:42:38.615 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:38.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:38.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:38.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.799135+0000) 2019-10-13T14:42:38.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:38.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:38.800612+0000) 2019-10-13T14:42:40.359 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:42:40.359 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:40.363 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:42:40.363 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:42:40.730 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:40.725+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:40.728145+0000) 2019-10-13T14:42:41.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:41.780+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:41.783088+0000) 2019-10-13T14:42:43.619 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:43.612+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.616171+0000) 2019-10-13T14:42:43.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:43.612+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.616466+0000) 2019-10-13T14:42:43.620 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:43.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.617072+0000) 2019-10-13T14:42:43.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:43.612+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.617296+0000) 2019-10-13T14:42:43.621 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:43.616+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.617896+0000) 2019-10-13T14:42:43.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:43.616+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.618123+0000) 2019-10-13T14:42:43.621 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:43.616+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.618974+0000) 2019-10-13T14:42:43.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:43.621+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:43.625566+0000) 2019-10-13T14:42:43.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:43.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:44.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:44.380+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:44.381362+0000) 2019-10-13T14:42:44.668 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:44.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:44.666023+0000) 2019-10-13T14:42:48.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:48.032+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.034600+0000) 2019-10-13T14:42:48.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:48.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.260853+0000) 2019-10-13T14:42:48.633 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:48.628+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.630298+0000) 2019-10-13T14:42:48.634 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:48.628+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.631662+0000) 2019-10-13T14:42:48.636 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:48.628+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.633667+0000) 2019-10-13T14:42:48.636 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:48.629+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.633963+0000) 2019-10-13T14:42:48.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:48.632+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.635305+0000) 2019-10-13T14:42:48.639 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:48.632+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.637315+0000) 2019-10-13T14:42:48.640 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:48.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.637559+0000) 2019-10-13T14:42:48.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:48.636+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.638341+0000) 2019-10-13T14:42:48.644 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:48.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210639 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:48.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:48.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.799325+0000) 2019-10-13T14:42:48.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:48.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:48.800800+0000) 2019-10-13T14:42:50.737 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:50.733+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:50.734652+0000) 2019-10-13T14:42:51.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:51.780+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:51.783291+0000) 2019-10-13T14:42:53.647 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:53.640+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.644448+0000) 2019-10-13T14:42:53.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:53.644+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.646864+0000) 2019-10-13T14:42:53.649 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:53.644+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.647222+0000) 2019-10-13T14:42:53.651 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:53.645+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.648518+0000) 2019-10-13T14:42:53.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:53.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.650580+0000) 2019-10-13T14:42:53.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:53.648+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.651243+0000) 2019-10-13T14:42:53.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:53.648+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.651770+0000) 2019-10-13T14:42:53.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:53.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:53.654316+0000) 2019-10-13T14:42:53.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:53.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:54.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:54.376+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:54.381531+0000) 2019-10-13T14:42:54.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:54.664+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:54.666219+0000) 2019-10-13T14:42:58.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:58.032+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.034797+0000) 2019-10-13T14:42:58.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:58.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.261053+0000) 2019-10-13T14:42:58.664 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:42:58.656+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.661318+0000) 2019-10-13T14:42:58.664 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:42:58.656+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.661298+0000) 2019-10-13T14:42:58.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:42:58.656+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.661586+0000) 2019-10-13T14:42:58.665 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:42:58.660+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.661988+0000) 2019-10-13T14:42:58.665 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:58.660+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.662320+0000) 2019-10-13T14:42:58.666 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:42:58.660+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.664021+0000) 2019-10-13T14:42:58.667 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:58.661+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.664881+0000) 2019-10-13T14:42:58.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:42:58.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:42:58.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:42:58.665+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.671626+0000) 2019-10-13T14:42:58.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:42:58.796+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.799508+0000) 2019-10-13T14:42:58.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:42:58.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:42:58.800969+0000) 2019-10-13T14:43:00.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:00.737+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:00.741306+0000) 2019-10-13T14:43:01.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:01.779+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:01.783527+0000) 2019-10-13T14:43:03.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:03.669+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.674596+0000) 2019-10-13T14:43:03.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:03.671+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.674824+0000) 2019-10-13T14:43:03.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:03.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.676152+0000) 2019-10-13T14:43:03.679 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:03.671+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.676180+0000) 2019-10-13T14:43:03.680 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:03.675+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.678200+0000) 2019-10-13T14:43:03.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:03.675+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.679244+0000) 2019-10-13T14:43:03.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:03.675+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.679838+0000) 2019-10-13T14:43:03.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:03.679+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:03.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:03.681+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:03.686508+0000) 2019-10-13T14:43:04.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:04.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:04.381784+0000) 2019-10-13T14:43:04.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:04.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:04.666415+0000) 2019-10-13T14:43:08.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:08.031+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.034994+0000) 2019-10-13T14:43:08.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:08.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.261254+0000) 2019-10-13T14:43:08.690 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:08.683+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.687898+0000) 2019-10-13T14:43:08.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:08.683+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.689460+0000) 2019-10-13T14:43:08.692 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:08.687+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.690098+0000) 2019-10-13T14:43:08.693 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:08.685+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.690618+0000) 2019-10-13T14:43:08.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:08.687+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.691493+0000) 2019-10-13T14:43:08.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:08.687+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.691847+0000) 2019-10-13T14:43:08.695 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:08.687+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.692596+0000) 2019-10-13T14:43:08.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:08.693+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.697701+0000) 2019-10-13T14:43:08.701 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:08.695+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:08.808 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:08.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.801122+0000) 2019-10-13T14:43:08.808 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:08.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:08.799701+0000) 2019-10-13T14:43:10.549 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:43:10.549 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:10.553 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:43:10.554 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:10.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:10.749+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:10.752544+0000) 2019-10-13T14:43:11.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:11.779+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:11.783739+0000) 2019-10-13T14:43:13.705 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:13.699+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.703080+0000) 2019-10-13T14:43:13.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:13.697+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.703947+0000) 2019-10-13T14:43:13.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:13.699+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.704624+0000) 2019-10-13T14:43:13.708 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:13.699+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.704627+0000) 2019-10-13T14:43:13.708 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:13.699+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.705154+0000) 2019-10-13T14:43:13.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:13.699+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.704225+0000) 2019-10-13T14:43:13.708 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:13.699+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.705597+0000) 2019-10-13T14:43:13.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:13.709+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:13.712417+0000) 2019-10-13T14:43:13.715 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:13.707+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:14.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:14.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:14.382001+0000) 2019-10-13T14:43:14.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:14.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:14.666628+0000) 2019-10-13T14:43:18.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:18.031+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.035227+0000) 2019-10-13T14:43:18.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:18.255+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.261452+0000) 2019-10-13T14:43:18.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:18.711+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.715842+0000) 2019-10-13T14:43:18.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:18.711+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.717171+0000) 2019-10-13T14:43:18.720 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:18.711+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.717591+0000) 2019-10-13T14:43:18.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:18.713+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.718660+0000) 2019-10-13T14:43:18.721 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:18.715+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.719147+0000) 2019-10-13T14:43:18.722 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:18.715+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.719522+0000) 2019-10-13T14:43:18.722 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:18.715+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.720059+0000) 2019-10-13T14:43:18.729 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:18.723+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:18.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:18.725+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.729156+0000) 2019-10-13T14:43:18.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:18.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.799887+0000) 2019-10-13T14:43:18.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:18.801+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:18.801320+0000) 2019-10-13T14:43:20.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:20.753+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:20.759133+0000) 2019-10-13T14:43:21.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:21.779+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:21.783969+0000) 2019-10-13T14:43:23.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:23.725+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.730012+0000) 2019-10-13T14:43:23.733 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:23.727+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.730369+0000) 2019-10-13T14:43:23.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:23.727+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.731207+0000) 2019-10-13T14:43:23.734 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:23.727+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.731476+0000) 2019-10-13T14:43:23.735 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:23.727+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.732654+0000) 2019-10-13T14:43:23.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:23.727+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.732901+0000) 2019-10-13T14:43:23.736 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:23.731+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.733774+0000) 2019-10-13T14:43:23.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:23.733+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:23.737259+0000) 2019-10-13T14:43:23.743 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:23.735+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:24.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:24.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:24.382179+0000) 2019-10-13T14:43:24.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:24.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:24.666809+0000) 2019-10-13T14:43:28.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:28.031+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.035461+0000) 2019-10-13T14:43:28.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:28.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.261678+0000) 2019-10-13T14:43:28.747 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:28.739+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.744422+0000) 2019-10-13T14:43:28.747 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:28.739+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.744830+0000) 2019-10-13T14:43:28.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:28.739+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.745222+0000) 2019-10-13T14:43:28.748 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:28.741+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.745874+0000) 2019-10-13T14:43:28.748 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:28.739+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.745415+0000) 2019-10-13T14:43:28.749 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:28.743+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.746873+0000) 2019-10-13T14:43:28.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:28.743+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.746983+0000) 2019-10-13T14:43:28.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:28.745+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.749909+0000) 2019-10-13T14:43:28.757 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:28.751+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:28.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:28.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.800139+0000) 2019-10-13T14:43:28.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:28.797+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:28.801504+0000) 2019-10-13T14:43:30.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:30.764+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:30.766188+0000) 2019-10-13T14:43:31.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:31.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:31.784172+0000) 2019-10-13T14:43:33.761 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:33.755+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.758838+0000) 2019-10-13T14:43:33.762 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:33.755+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.759504+0000) 2019-10-13T14:43:33.763 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:33.755+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.761339+0000) 2019-10-13T14:43:33.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:33.756+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.762276+0000) 2019-10-13T14:43:33.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:33.759+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.763676+0000) 2019-10-13T14:43:33.767 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:33.759+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.765272+0000) 2019-10-13T14:43:33.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:33.760+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.765856+0000) 2019-10-13T14:43:33.769 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:33.763+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:33.767220+0000) 2019-10-13T14:43:33.772 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:33.767+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210913 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:34.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:34.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:34.382382+0000) 2019-10-13T14:43:34.669 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:34.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:34.667015+0000) 2019-10-13T14:43:38.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:38.035+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.035660+0000) 2019-10-13T14:43:38.264 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:38.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.261897+0000) 2019-10-13T14:43:38.776 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:38.768+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.773379+0000) 2019-10-13T14:43:38.776 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:38.767+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.773695+0000) 2019-10-13T14:43:38.777 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:38.771+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.774399+0000) 2019-10-13T14:43:38.778 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:38.771+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.775821+0000) 2019-10-13T14:43:38.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:38.775+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.777973+0000) 2019-10-13T14:43:38.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:38.775+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.778283+0000) 2019-10-13T14:43:38.785 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:38.779+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.782560+0000) 2019-10-13T14:43:38.786 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:38.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210943 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:38.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:38.780+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.785850+0000) 2019-10-13T14:43:38.802 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:38.795+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.800255+0000) 2019-10-13T14:43:38.804 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:38.796+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:38.801667+0000) 2019-10-13T14:43:40.672 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:43:40.672 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:40.676 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:43:40.676 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:43:40.776 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:40.772+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:40.773320+0000) 2019-10-13T14:43:41.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:41.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:41.784378+0000) 2019-10-13T14:43:43.789 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:43.783+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.786400+0000) 2019-10-13T14:43:43.790 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:43.780+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.787840+0000) 2019-10-13T14:43:43.790 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:43.783+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.788139+0000) 2019-10-13T14:43:43.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:43.783+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.788508+0000) 2019-10-13T14:43:43.791 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:43.783+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.788663+0000) 2019-10-13T14:43:43.793 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:43.787+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.790641+0000) 2019-10-13T14:43:43.794 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:43.787+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.791882+0000) 2019-10-13T14:43:43.799 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:43.792+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:43.796863+0000) 2019-10-13T14:43:43.800 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:43.791+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210972 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:44.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:44.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:44.382622+0000) 2019-10-13T14:43:44.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:44.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:44.667191+0000) 2019-10-13T14:43:48.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:48.035+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.035904+0000) 2019-10-13T14:43:48.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:48.259+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.262110+0000) 2019-10-13T14:43:48.804 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:48.795+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.801468+0000) 2019-10-13T14:43:48.805 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:48.799+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.802278+0000) 2019-10-13T14:43:48.805 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:48.799+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.802568+0000) 2019-10-13T14:43:48.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:48.799+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.805019+0000) 2019-10-13T14:43:48.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:48.803+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.807182+0000) 2019-10-13T14:43:48.812 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:48.807+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.810145+0000) 2019-10-13T14:43:48.813 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:48.804+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.811327+0000) 2019-10-13T14:43:48.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:48.804+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:48.811584+0000) 2019-10-13T14:43:48.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:48.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:50.782 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:50.776+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:50.779935+0000) 2019-10-13T14:43:51.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:51.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:51.784584+0000) 2019-10-13T14:43:53.823 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:53.812+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.818659+0000) 2019-10-13T14:43:53.824 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:53.811+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.816467+0000) 2019-10-13T14:43:53.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:53.811+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.816951+0000) 2019-10-13T14:43:53.825 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:53.811+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.817515+0000) 2019-10-13T14:43:53.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:53.815+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.817802+0000) 2019-10-13T14:43:53.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:53.815+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.819043+0000) 2019-10-13T14:43:53.825 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:53.815+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.819341+0000) 2019-10-13T14:43:53.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:53.820+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:53.824639+0000) 2019-10-13T14:43:53.829 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:53.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211033 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:43:54.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:54.379+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:54.382806+0000) 2019-10-13T14:43:54.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:54.663+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:54.667372+0000) 2019-10-13T14:43:55.553 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:55.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:55.551018+0000) 2019-10-13T14:43:55.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:55.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:55.552385+0000) 2019-10-13T14:43:58.038 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:58.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.036105+0000) 2019-10-13T14:43:58.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:58.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.262322+0000) 2019-10-13T14:43:58.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:43:58.822+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.829650+0000) 2019-10-13T14:43:58.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:43:58.826+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.830428+0000) 2019-10-13T14:43:58.835 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:43:58.826+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.832447+0000) 2019-10-13T14:43:58.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:43:58.826+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.833262+0000) 2019-10-13T14:43:58.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:43:58.826+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.833720+0000) 2019-10-13T14:43:58.837 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:43:58.830+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.834765+0000) 2019-10-13T14:43:58.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:43:58.828+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.835550+0000) 2019-10-13T14:43:58.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:43:58.832+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:43:58.839218+0000) 2019-10-13T14:43:58.844 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:43:58.834+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211063 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:00.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:00.784+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:00.786916+0000) 2019-10-13T14:44:01.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:01.782+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:01.784812+0000) 2019-10-13T14:44:03.847 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:03.838+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.844795+0000) 2019-10-13T14:44:03.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:03.842+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.846465+0000) 2019-10-13T14:44:03.850 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:03.842+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.846960+0000) 2019-10-13T14:44:03.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:03.842+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.847387+0000) 2019-10-13T14:44:03.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:03.842+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.847759+0000) 2019-10-13T14:44:03.851 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:03.842+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.848505+0000) 2019-10-13T14:44:03.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:03.844+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.849121+0000) 2019-10-13T14:44:03.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:03.848+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:03.854942+0000) 2019-10-13T14:44:03.858 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:03.850+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211092 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:04.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:04.378+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:04.382989+0000) 2019-10-13T14:44:04.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:04.662+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:04.667561+0000) 2019-10-13T14:44:05.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:05.546+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:05.551251+0000) 2019-10-13T14:44:05.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:05.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:05.552581+0000) 2019-10-13T14:44:08.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:08.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.036306+0000) 2019-10-13T14:44:08.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:08.258+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.262471+0000) 2019-10-13T14:44:08.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:08.854+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.857774+0000) 2019-10-13T14:44:08.861 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:08.854+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.858738+0000) 2019-10-13T14:44:08.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:08.854+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.859189+0000) 2019-10-13T14:44:08.862 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:08.854+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.859098+0000) 2019-10-13T14:44:08.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:08.852+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.859956+0000) 2019-10-13T14:44:08.863 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:08.854+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.860325+0000) 2019-10-13T14:44:08.863 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:08.854+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.860600+0000) 2019-10-13T14:44:08.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:08.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211122 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:08.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:08.864+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:08.868920+0000) 2019-10-13T14:44:10.795 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:44:10.795 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:10.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:10.792+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:10.794080+0000) 2019-10-13T14:44:10.799 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:44:10.799 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:11.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:11.782+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:11.785066+0000) 2019-10-13T14:44:13.873 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:13.866+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.870083+0000) 2019-10-13T14:44:13.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:13.866+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.871500+0000) 2019-10-13T14:44:13.874 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:13.866+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.871410+0000) 2019-10-13T14:44:13.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:13.868+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.872265+0000) 2019-10-13T14:44:13.876 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:13.870+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.874028+0000) 2019-10-13T14:44:13.877 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:13.870+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.873922+0000) 2019-10-13T14:44:13.877 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:13.870+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.874171+0000) 2019-10-13T14:44:13.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:13.876+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:13.882052+0000) 2019-10-13T14:44:13.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:13.878+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211153 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:14.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:14.382+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:14.383219+0000) 2019-10-13T14:44:14.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:14.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:14.667765+0000) 2019-10-13T14:44:15.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:15.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:15.551444+0000) 2019-10-13T14:44:15.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:15.552+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:15.552783+0000) 2019-10-13T14:44:18.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:18.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.036508+0000) 2019-10-13T14:44:18.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:18.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.262708+0000) 2019-10-13T14:44:18.888 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:18.878+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.885477+0000) 2019-10-13T14:44:18.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:18.882+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.885909+0000) 2019-10-13T14:44:18.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:18.882+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.886540+0000) 2019-10-13T14:44:18.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:18.882+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.887017+0000) 2019-10-13T14:44:18.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:18.882+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.887505+0000) 2019-10-13T14:44:18.891 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:18.882+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.888896+0000) 2019-10-13T14:44:18.892 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:18.884+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.889670+0000) 2019-10-13T14:44:18.900 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:18.890+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:18.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:18.892+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:18.897989+0000) 2019-10-13T14:44:20.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:20.796+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:20.801258+0000) 2019-10-13T14:44:21.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:21.782+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:21.785275+0000) 2019-10-13T14:44:23.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:23.894+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.900730+0000) 2019-10-13T14:44:23.904 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:23.894+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.901147+0000) 2019-10-13T14:44:23.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:23.898+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.902592+0000) 2019-10-13T14:44:23.905 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:23.898+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.902795+0000) 2019-10-13T14:44:23.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:23.896+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.903241+0000) 2019-10-13T14:44:23.906 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:23.898+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.903555+0000) 2019-10-13T14:44:23.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:23.898+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.904107+0000) 2019-10-13T14:44:23.912 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:23.904+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:23.909491+0000) 2019-10-13T14:44:23.913 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:23.906+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211212 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:24.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:24.382+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:24.383406+0000) 2019-10-13T14:44:24.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:24.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:24.667936+0000) 2019-10-13T14:44:25.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:25.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:25.551630+0000) 2019-10-13T14:44:25.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:25.548+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:25.552992+0000) 2019-10-13T14:44:28.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:28.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.036736+0000) 2019-10-13T14:44:28.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:28.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.262934+0000) 2019-10-13T14:44:28.917 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:28.910+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.914245+0000) 2019-10-13T14:44:28.917 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:28.912+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.914709+0000) 2019-10-13T14:44:28.918 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:28.910+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.914949+0000) 2019-10-13T14:44:28.918 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:28.910+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.915291+0000) 2019-10-13T14:44:28.919 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:28.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.917230+0000) 2019-10-13T14:44:28.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:28.914+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.919009+0000) 2019-10-13T14:44:28.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:28.914+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.920403+0000) 2019-10-13T14:44:28.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:28.920+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:28.923768+0000) 2019-10-13T14:44:28.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:28.918+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211242 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:30.810 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:30.804+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:30.807785+0000) 2019-10-13T14:44:31.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:31.778+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:31.785478+0000) 2019-10-13T14:44:33.932 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:33.922+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.928754+0000) 2019-10-13T14:44:33.932 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:33.926+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.929789+0000) 2019-10-13T14:44:33.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:33.926+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.930211+0000) 2019-10-13T14:44:33.933 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:33.926+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.930619+0000) 2019-10-13T14:44:33.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:33.928+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.930924+0000) 2019-10-13T14:44:33.935 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:33.926+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.932669+0000) 2019-10-13T14:44:33.936 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:33.930+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.934088+0000) 2019-10-13T14:44:33.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:33.932+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:33.935575+0000) 2019-10-13T14:44:33.942 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:33.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211273 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:34.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:34.378+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:34.383593+0000) 2019-10-13T14:44:34.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:34.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:34.668110+0000) 2019-10-13T14:44:35.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:35.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:35.551820+0000) 2019-10-13T14:44:35.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:35.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:35.553219+0000) 2019-10-13T14:44:38.039 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:38.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.036932+0000) 2019-10-13T14:44:38.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:38.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.263132+0000) 2019-10-13T14:44:38.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:38.939+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.943209+0000) 2019-10-13T14:44:38.946 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:38.938+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.943771+0000) 2019-10-13T14:44:38.947 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:38.938+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.944051+0000) 2019-10-13T14:44:38.947 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:38.938+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.944761+0000) 2019-10-13T14:44:38.948 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:38.942+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.945820+0000) 2019-10-13T14:44:38.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:38.942+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.946478+0000) 2019-10-13T14:44:38.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:38.942+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.946722+0000) 2019-10-13T14:44:38.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:38.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211303 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:38.956 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:38.951+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:38.953704+0000) 2019-10-13T14:44:40.817 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:40.811+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:40.814262+0000) 2019-10-13T14:44:40.852 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:44:40.852 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:40.856 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:44:40.856 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:44:41.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:41.782+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:41.785762+0000) 2019-10-13T14:44:43.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:43.950+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.956857+0000) 2019-10-13T14:44:43.960 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:43.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.957501+0000) 2019-10-13T14:44:43.960 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:43.950+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.957432+0000) 2019-10-13T14:44:43.961 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:43.954+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.958435+0000) 2019-10-13T14:44:43.961 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:43.954+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.958801+0000) 2019-10-13T14:44:43.962 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:43.954+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.959456+0000) 2019-10-13T14:44:43.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:43.955+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.959957+0000) 2019-10-13T14:44:43.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:43.963+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:43.966596+0000) 2019-10-13T14:44:43.970 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:43.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211332 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:44.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:44.382+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:44.383782+0000) 2019-10-13T14:44:44.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:44.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:44.668296+0000) 2019-10-13T14:44:45.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:45.550+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:45.552056+0000) 2019-10-13T14:44:45.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:45.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:45.553411+0000) 2019-10-13T14:44:48.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:48.030+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.037130+0000) 2019-10-13T14:44:48.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:48.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.263334+0000) 2019-10-13T14:44:48.973 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:48.966+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.970662+0000) 2019-10-13T14:44:48.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:48.966+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.971287+0000) 2019-10-13T14:44:48.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:48.966+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.971713+0000) 2019-10-13T14:44:48.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:48.966+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.972622+0000) 2019-10-13T14:44:48.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:48.971+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.973015+0000) 2019-10-13T14:44:48.976 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:48.966+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.973288+0000) 2019-10-13T14:44:48.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:48.966+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.973534+0000) 2019-10-13T14:44:48.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:48.975+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:48.978416+0000) 2019-10-13T14:44:48.984 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:48.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211362 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:50.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:50.819+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:50.820652+0000) 2019-10-13T14:44:51.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:51.781+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:51.785980+0000) 2019-10-13T14:44:53.987 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:53.977+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.984267+0000) 2019-10-13T14:44:53.987 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:53.977+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.984883+0000) 2019-10-13T14:44:53.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:53.981+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.985926+0000) 2019-10-13T14:44:53.989 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:53.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.986422+0000) 2019-10-13T14:44:53.989 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:53.983+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.986935+0000) 2019-10-13T14:44:53.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:53.981+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.987899+0000) 2019-10-13T14:44:53.991 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:53.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.987146+0000) 2019-10-13T14:44:53.996 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:53.991+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:53.993671+0000) 2019-10-13T14:44:53.999 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:53.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211393 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:44:54.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:54.377+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:54.383942+0000) 2019-10-13T14:44:54.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:54.661+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:54.668468+0000) 2019-10-13T14:44:55.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:55.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:55.552248+0000) 2019-10-13T14:44:55.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:55.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:55.553622+0000) 2019-10-13T14:44:58.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:58.033+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:58.037332+0000) 2019-10-13T14:44:58.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:58.257+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:58.263543+0000) 2019-10-13T14:44:59.003 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:44:58.993+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.000362+0000) 2019-10-13T14:44:59.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:44:58.993+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.000662+0000) 2019-10-13T14:44:59.004 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:44:58.993+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.000971+0000) 2019-10-13T14:44:59.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:44:58.993+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.001428+0000) 2019-10-13T14:44:59.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:44:58.997+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.001808+0000) 2019-10-13T14:44:59.005 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:44:58.997+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.002956+0000) 2019-10-13T14:44:59.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:44:58.999+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.003823+0000) 2019-10-13T14:44:59.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:44:59.007+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:44:59.008588+0000) 2019-10-13T14:44:59.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:44:59.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211423 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:00.830 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:00.823+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:00.827235+0000) 2019-10-13T14:45:01.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:01.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:01.786184+0000) 2019-10-13T14:45:04.015 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:04.009+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.012923+0000) 2019-10-13T14:45:04.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:04.009+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.013420+0000) 2019-10-13T14:45:04.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:04.009+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.015244+0000) 2019-10-13T14:45:04.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:04.009+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.015853+0000) 2019-10-13T14:45:04.019 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:04.015+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.016748+0000) 2019-10-13T14:45:04.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:04.009+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.016997+0000) 2019-10-13T14:45:04.020 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:04.013+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.018398+0000) 2019-10-13T14:45:04.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:04.017+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:04.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:04.023+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.025055+0000) 2019-10-13T14:45:04.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:04.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.384108+0000) 2019-10-13T14:45:04.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:04.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:04.668638+0000) 2019-10-13T14:45:05.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:05.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:05.552435+0000) 2019-10-13T14:45:05.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:05.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:05.553822+0000) 2019-10-13T14:45:08.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:08.033+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:08.037585+0000) 2019-10-13T14:45:08.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:08.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:08.263778+0000) 2019-10-13T14:45:09.030 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:09.027+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.028142+0000) 2019-10-13T14:45:09.031 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:09.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.028635+0000) 2019-10-13T14:45:09.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:09.029+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.030350+0000) 2019-10-13T14:45:09.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:09.029+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.030860+0000) 2019-10-13T14:45:09.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:09.029+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.031318+0000) 2019-10-13T14:45:09.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:09.029+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.032238+0000) 2019-10-13T14:45:09.035 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:09.029+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.032921+0000) 2019-10-13T14:45:09.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:09.037+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211480 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:09.042 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:09.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:09.039451+0000) 2019-10-13T14:45:10.836 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:10.831+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:10.833838+0000) 2019-10-13T14:45:10.992 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:45:10.992 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:10.997 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:45:10.997 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:11.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:11.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:11.786412+0000) 2019-10-13T14:45:14.044 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:14.039+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.041357+0000) 2019-10-13T14:45:14.044 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:14.037+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.041612+0000) 2019-10-13T14:45:14.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:14.041+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.042867+0000) 2019-10-13T14:45:14.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:14.041+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.043460+0000) 2019-10-13T14:45:14.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:14.041+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.044537+0000) 2019-10-13T14:45:14.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:14.045+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.046383+0000) 2019-10-13T14:45:14.049 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:14.045+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.046360+0000) 2019-10-13T14:45:14.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:14.051+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.053085+0000) 2019-10-13T14:45:14.056 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:14.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:14.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:14.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.384289+0000) 2019-10-13T14:45:14.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:14.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:14.668809+0000) 2019-10-13T14:45:15.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:15.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:15.552629+0000) 2019-10-13T14:45:15.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:15.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:15.554017+0000) 2019-10-13T14:45:18.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:18.037+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:18.037784+0000) 2019-10-13T14:45:18.266 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:18.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:18.263978+0000) 2019-10-13T14:45:19.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:19.055+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.056167+0000) 2019-10-13T14:45:19.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:19.057+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.058097+0000) 2019-10-13T14:45:19.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:19.057+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.058151+0000) 2019-10-13T14:45:19.062 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:19.057+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.059556+0000) 2019-10-13T14:45:19.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:19.057+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.059803+0000) 2019-10-13T14:45:19.065 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:19.061+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.062960+0000) 2019-10-13T14:45:19.066 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:19.061+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.063370+0000) 2019-10-13T14:45:19.069 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:19.065+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:19.072 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:19.067+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:19.069590+0000) 2019-10-13T14:45:20.843 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:20.839+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:20.840450+0000) 2019-10-13T14:45:21.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:21.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:21.786630+0000) 2019-10-13T14:45:24.074 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:24.067+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.071340+0000) 2019-10-13T14:45:24.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:24.069+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.071646+0000) 2019-10-13T14:45:24.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:24.069+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.072248+0000) 2019-10-13T14:45:24.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:24.069+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.073271+0000) 2019-10-13T14:45:24.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:24.069+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.073470+0000) 2019-10-13T14:45:24.077 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:24.073+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.074570+0000) 2019-10-13T14:45:24.077 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:24.073+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.074797+0000) 2019-10-13T14:45:24.082 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:24.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211570 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:24.083 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:24.079+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.080952+0000) 2019-10-13T14:45:24.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:24.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.384471+0000) 2019-10-13T14:45:24.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:24.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:24.668939+0000) 2019-10-13T14:45:25.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:25.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:25.552821+0000) 2019-10-13T14:45:25.556 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:25.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:25.554213+0000) 2019-10-13T14:45:28.040 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:28.037+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:28.037981+0000) 2019-10-13T14:45:28.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:28.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:28.264176+0000) 2019-10-13T14:45:29.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:29.079+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.083888+0000) 2019-10-13T14:45:29.087 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:29.081+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.084264+0000) 2019-10-13T14:45:29.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:29.081+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.084722+0000) 2019-10-13T14:45:29.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:29.081+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.085496+0000) 2019-10-13T14:45:29.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:29.085+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.086759+0000) 2019-10-13T14:45:29.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:29.085+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.088393+0000) 2019-10-13T14:45:29.091 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:29.085+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.088898+0000) 2019-10-13T14:45:29.097 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:29.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211598 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:29.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:29.091+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:29.095895+0000) 2019-10-13T14:45:30.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:30.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:30.851950+0000) 2019-10-13T14:45:31.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:31.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:31.786848+0000) 2019-10-13T14:45:34.102 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:34.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.098797+0000) 2019-10-13T14:45:34.102 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:34.097+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.099014+0000) 2019-10-13T14:45:34.102 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:34.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.099500+0000) 2019-10-13T14:45:34.103 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:34.097+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.099899+0000) 2019-10-13T14:45:34.103 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:34.099+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.100788+0000) 2019-10-13T14:45:34.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:34.097+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.100999+0000) 2019-10-13T14:45:34.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:34.097+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.101191+0000) 2019-10-13T14:45:34.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:34.103+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.106910+0000) 2019-10-13T14:45:34.111 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:34.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211630 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:34.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:34.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.384659+0000) 2019-10-13T14:45:34.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:34.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:34.669140+0000) 2019-10-13T14:45:35.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:35.549+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:35.553011+0000) 2019-10-13T14:45:35.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:35.551+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:35.554439+0000) 2019-10-13T14:45:38.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:38.037+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:38.038213+0000) 2019-10-13T14:45:38.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:38.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:38.264403+0000) 2019-10-13T14:45:39.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:39.109+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.112410+0000) 2019-10-13T14:45:39.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:39.113+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.114174+0000) 2019-10-13T14:45:39.117 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:39.113+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.114561+0000) 2019-10-13T14:45:39.117 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:39.113+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.114717+0000) 2019-10-13T14:45:39.118 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:39.113+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.115080+0000) 2019-10-13T14:45:39.118 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:39.111+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.115726+0000) 2019-10-13T14:45:39.119 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:39.113+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.116059+0000) 2019-10-13T14:45:39.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:39.119+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:39.122135+0000) 2019-10-13T14:45:39.125 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:39.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:40.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:40.854+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:40.858535+0000) 2019-10-13T14:45:41.058 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:45:41.058 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:41.062 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:45:41.062 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:45:41.789 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:41.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:41.787089+0000) 2019-10-13T14:45:44.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:44.125+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.126158+0000) 2019-10-13T14:45:44.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:44.125+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.126921+0000) 2019-10-13T14:45:44.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:44.125+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.127140+0000) 2019-10-13T14:45:44.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:44.125+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.128117+0000) 2019-10-13T14:45:44.131 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:44.125+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.128490+0000) 2019-10-13T14:45:44.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:44.125+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.128836+0000) 2019-10-13T14:45:44.132 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:44.126+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.130025+0000) 2019-10-13T14:45:44.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:44.130+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.135526+0000) 2019-10-13T14:45:44.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:44.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211687 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:44.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:44.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.384861+0000) 2019-10-13T14:45:44.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:44.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:44.669357+0000) 2019-10-13T14:45:45.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:45.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:45.553218+0000) 2019-10-13T14:45:45.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:45.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:45.554637+0000) 2019-10-13T14:45:48.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:48.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:48.038412+0000) 2019-10-13T14:45:48.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:48.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:48.264633+0000) 2019-10-13T14:45:49.143 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:49.136+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.140022+0000) 2019-10-13T14:45:49.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:49.136+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.140244+0000) 2019-10-13T14:45:49.144 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:49.136+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.140582+0000) 2019-10-13T14:45:49.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:49.136+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.140867+0000) 2019-10-13T14:45:49.144 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:49.136+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.141242+0000) 2019-10-13T14:45:49.144 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:49.140+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.141913+0000) 2019-10-13T14:45:49.146 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:49.138+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.143861+0000) 2019-10-13T14:45:49.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:49.146+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:49.150220+0000) 2019-10-13T14:45:49.154 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:49.148+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211716 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:50.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:50.862+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:50.865235+0000) 2019-10-13T14:45:51.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:51.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:51.787322+0000) 2019-10-13T14:45:54.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:54.152+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.155551+0000) 2019-10-13T14:45:54.159 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:54.150+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.156029+0000) 2019-10-13T14:45:54.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:54.152+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.156327+0000) 2019-10-13T14:45:54.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:54.152+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.156927+0000) 2019-10-13T14:45:54.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:54.152+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.157446+0000) 2019-10-13T14:45:54.161 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:54.156+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.157986+0000) 2019-10-13T14:45:54.161 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:54.156+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.158484+0000) 2019-10-13T14:45:54.168 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:54.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211747 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:54.168 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:54.162+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.166087+0000) 2019-10-13T14:45:54.387 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:54.380+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.385044+0000) 2019-10-13T14:45:54.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:54.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:54.669547+0000) 2019-10-13T14:45:55.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:55.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:55.553451+0000) 2019-10-13T14:45:55.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:55.554+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:55.554843+0000) 2019-10-13T14:45:58.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:58.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:58.038608+0000) 2019-10-13T14:45:58.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:58.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:58.264864+0000) 2019-10-13T14:45:59.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:45:59.164+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.167934+0000) 2019-10-13T14:45:59.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:45:59.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.168494+0000) 2019-10-13T14:45:59.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:45:59.166+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.169508+0000) 2019-10-13T14:45:59.174 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:45:59.168+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.171052+0000) 2019-10-13T14:45:59.174 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:45:59.168+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.171534+0000) 2019-10-13T14:45:59.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:45:59.168+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.172895+0000) 2019-10-13T14:45:59.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:45:59.168+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.173331+0000) 2019-10-13T14:45:59.183 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:45:59.176+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211778 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:45:59.184 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:45:59.178+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:45:59.181827+0000) 2019-10-13T14:46:00.874 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:00.870+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:00.871670+0000) 2019-10-13T14:46:01.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:01.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:01.787515+0000) 2019-10-13T14:46:04.186 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:04.180+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.183930+0000) 2019-10-13T14:46:04.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:04.184+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.186001+0000) 2019-10-13T14:46:04.189 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:04.184+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.186030+0000) 2019-10-13T14:46:04.189 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:04.184+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.186658+0000) 2019-10-13T14:46:04.190 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:04.184+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.187094+0000) 2019-10-13T14:46:04.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:04.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.187568+0000) 2019-10-13T14:46:04.191 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:04.184+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.187852+0000) 2019-10-13T14:46:04.196 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:04.192+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211807 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:04.197 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:04.190+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.194391+0000) 2019-10-13T14:46:04.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:04.384+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.385236+0000) 2019-10-13T14:46:04.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:04.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:04.669729+0000) 2019-10-13T14:46:05.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:05.548+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:05.553660+0000) 2019-10-13T14:46:05.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:05.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:05.555067+0000) 2019-10-13T14:46:08.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:08.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:08.038818+0000) 2019-10-13T14:46:08.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:08.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:08.265062+0000) 2019-10-13T14:46:09.200 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:09.192+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.197020+0000) 2019-10-13T14:46:09.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:09.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.197179+0000) 2019-10-13T14:46:09.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:09.196+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.198259+0000) 2019-10-13T14:46:09.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:09.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.198704+0000) 2019-10-13T14:46:09.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:09.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.198965+0000) 2019-10-13T14:46:09.202 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:09.196+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.199356+0000) 2019-10-13T14:46:09.202 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:09.196+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.199659+0000) 2019-10-13T14:46:09.211 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:09.204+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211838 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:09.211 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:09.206+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:09.208524+0000) 2019-10-13T14:46:10.881 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:10.874+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:10.878131+0000) 2019-10-13T14:46:11.173 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:46:11.173 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:11.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:46:11.177 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:11.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:11.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:11.787717+0000) 2019-10-13T14:46:14.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:14.208+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.211933+0000) 2019-10-13T14:46:14.215 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:14.208+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.211372+0000) 2019-10-13T14:46:14.216 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:14.208+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.213191+0000) 2019-10-13T14:46:14.217 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:14.208+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.213697+0000) 2019-10-13T14:46:14.218 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:14.210+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.215172+0000) 2019-10-13T14:46:14.218 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:14.212+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.214521+0000) 2019-10-13T14:46:14.219 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:14.212+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.214645+0000) 2019-10-13T14:46:14.225 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:14.220+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211869 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:14.229 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:14.222+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.226422+0000) 2019-10-13T14:46:14.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:14.384+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.385468+0000) 2019-10-13T14:46:14.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:14.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:14.669906+0000) 2019-10-13T14:46:15.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:15.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:15.553881+0000) 2019-10-13T14:46:15.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:15.554+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:15.555264+0000) 2019-10-13T14:46:18.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:18.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:18.039019+0000) 2019-10-13T14:46:18.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:18.260+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:18.265210+0000) 2019-10-13T14:46:19.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:19.224+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.226832+0000) 2019-10-13T14:46:19.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:19.224+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.227041+0000) 2019-10-13T14:46:19.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:19.224+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.227701+0000) 2019-10-13T14:46:19.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:19.224+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.229234+0000) 2019-10-13T14:46:19.232 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:19.226+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.229549+0000) 2019-10-13T14:46:19.232 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:19.228+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.229903+0000) 2019-10-13T14:46:19.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:19.228+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.230248+0000) 2019-10-13T14:46:19.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:19.232+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211900 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:19.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:19.234+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:19.237115+0000) 2019-10-13T14:46:20.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:20.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:20.884571+0000) 2019-10-13T14:46:21.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:21.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:21.787919+0000) 2019-10-13T14:46:24.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:24.236+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.240709+0000) 2019-10-13T14:46:24.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:24.236+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.241197+0000) 2019-10-13T14:46:24.244 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:24.236+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.241657+0000) 2019-10-13T14:46:24.245 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:24.240+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.242660+0000) 2019-10-13T14:46:24.246 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:24.240+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.243091+0000) 2019-10-13T14:46:24.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:24.238+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.243329+0000) 2019-10-13T14:46:24.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:24.240+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.243535+0000) 2019-10-13T14:46:24.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:24.242+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.245700+0000) 2019-10-13T14:46:24.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:24.248+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211929 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:24.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:24.384+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.385670+0000) 2019-10-13T14:46:24.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:24.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:24.670092+0000) 2019-10-13T14:46:25.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:25.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:25.554067+0000) 2019-10-13T14:46:25.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:25.550+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:25.555452+0000) 2019-10-13T14:46:28.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:28.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:28.039233+0000) 2019-10-13T14:46:28.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:28.264+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:28.265425+0000) 2019-10-13T14:46:29.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:29.252+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.254705+0000) 2019-10-13T14:46:29.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:29.252+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.255279+0000) 2019-10-13T14:46:29.260 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:29.252+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.257014+0000) 2019-10-13T14:46:29.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:29.252+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.257319+0000) 2019-10-13T14:46:29.261 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:29.256+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.258937+0000) 2019-10-13T14:46:29.262 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:29.254+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.259191+0000) 2019-10-13T14:46:29.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:29.256+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.260774+0000) 2019-10-13T14:46:29.267 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:29.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211961 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:29.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:29.266+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:29.269995+0000) 2019-10-13T14:46:30.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:30.890+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:30.891048+0000) 2019-10-13T14:46:31.790 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:31.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:31.788137+0000) 2019-10-13T14:46:34.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:34.264+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.268338+0000) 2019-10-13T14:46:34.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:34.264+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.268720+0000) 2019-10-13T14:46:34.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:34.264+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.269635+0000) 2019-10-13T14:46:34.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:34.268+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.270226+0000) 2019-10-13T14:46:34.273 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:34.268+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.271132+0000) 2019-10-13T14:46:34.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:34.268+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.271493+0000) 2019-10-13T14:46:34.274 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:34.266+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.272067+0000) 2019-10-13T14:46:34.281 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:34.276+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211991 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:34.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:34.274+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.279714+0000) 2019-10-13T14:46:34.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:34.384+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.385859+0000) 2019-10-13T14:46:34.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:34.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:34.670302+0000) 2019-10-13T14:46:35.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:35.552+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:35.554253+0000) 2019-10-13T14:46:35.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:35.554+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:35.555644+0000) 2019-10-13T14:46:38.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:38.036+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:38.039462+0000) 2019-10-13T14:46:38.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:38.264+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:38.265648+0000) 2019-10-13T14:46:39.285 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:39.280+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.282016+0000) 2019-10-13T14:46:39.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:39.280+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.282449+0000) 2019-10-13T14:46:39.285 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:39.280+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.283027+0000) 2019-10-13T14:46:39.287 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:39.280+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.284767+0000) 2019-10-13T14:46:39.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:39.282+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.285008+0000) 2019-10-13T14:46:39.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:39.280+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.285483+0000) 2019-10-13T14:46:39.289 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:39.284+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.286602+0000) 2019-10-13T14:46:39.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:39.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212022 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:39.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:39.290+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:39.295453+0000) 2019-10-13T14:46:40.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:40.898+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:40.900838+0000) 2019-10-13T14:46:41.279 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:46:41.280 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:41.283 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:46:41.283 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:46:41.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:41.784+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:41.788370+0000) 2019-10-13T14:46:44.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:44.291+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.294381+0000) 2019-10-13T14:46:44.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:44.291+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.295394+0000) 2019-10-13T14:46:44.300 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:44.295+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.298320+0000) 2019-10-13T14:46:44.303 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:44.295+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.300352+0000) 2019-10-13T14:46:44.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:44.297+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.302591+0000) 2019-10-13T14:46:44.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:44.299+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.302664+0000) 2019-10-13T14:46:44.305 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:44.297+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.303215+0000) 2019-10-13T14:46:44.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:44.299+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.303812+0000) 2019-10-13T14:46:44.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:44.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212051 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:44.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:44.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.386045+0000) 2019-10-13T14:46:44.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:44.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:44.670511+0000) 2019-10-13T14:46:45.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:45.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:45.554479+0000) 2019-10-13T14:46:45.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:45.549+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:45.555837+0000) 2019-10-13T14:46:48.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:48.035+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:48.039664+0000) 2019-10-13T14:46:48.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:48.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:48.265846+0000) 2019-10-13T14:46:49.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:49.303+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.308491+0000) 2019-10-13T14:46:49.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:49.303+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.309041+0000) 2019-10-13T14:46:49.312 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:49.303+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.308759+0000) 2019-10-13T14:46:49.312 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:49.303+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.309562+0000) 2019-10-13T14:46:49.313 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:49.307+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.310457+0000) 2019-10-13T14:46:49.313 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:49.307+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.310897+0000) 2019-10-13T14:46:49.314 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:49.305+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.311972+0000) 2019-10-13T14:46:49.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:49.313+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:49.317828+0000) 2019-10-13T14:46:49.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:49.315+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:50.910 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:50.901+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:50.907394+0000) 2019-10-13T14:46:51.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:51.787+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:51.788574+0000) 2019-10-13T14:46:54.325 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:54.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.322640+0000) 2019-10-13T14:46:54.326 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:54.319+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.323399+0000) 2019-10-13T14:46:54.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:54.319+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.323805+0000) 2019-10-13T14:46:54.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:54.319+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.325422+0000) 2019-10-13T14:46:54.330 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:54.323+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.327643+0000) 2019-10-13T14:46:54.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:54.321+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.327980+0000) 2019-10-13T14:46:54.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:54.323+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.329385+0000) 2019-10-13T14:46:54.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:54.329+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.332229+0000) 2019-10-13T14:46:54.337 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:54.331+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212113 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:46:54.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:54.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.386226+0000) 2019-10-13T14:46:54.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:54.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:54.670691+0000) 2019-10-13T14:46:55.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:55.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:55.554677+0000) 2019-10-13T14:46:55.558 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:55.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:55.556030+0000) 2019-10-13T14:46:58.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:58.035+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:58.039890+0000) 2019-10-13T14:46:58.268 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:58.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:58.266078+0000) 2019-10-13T14:46:59.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:46:59.335+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.339183+0000) 2019-10-13T14:46:59.342 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:46:59.335+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.339397+0000) 2019-10-13T14:46:59.344 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:46:59.337+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.342080+0000) 2019-10-13T14:46:59.345 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:46:59.339+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.342360+0000) 2019-10-13T14:46:59.346 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:46:59.339+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.343437+0000) 2019-10-13T14:46:59.346 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:46:59.339+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.343700+0000) 2019-10-13T14:46:59.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:46:59.339+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.344487+0000) 2019-10-13T14:46:59.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:46:59.341+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:46:59.346344+0000) 2019-10-13T14:46:59.351 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:46:59.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212144 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:00.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:00.913+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:00.918856+0000) 2019-10-13T14:47:01.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:01.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:01.788792+0000) 2019-10-13T14:47:04.355 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:04.347+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.353006+0000) 2019-10-13T14:47:04.356 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:04.347+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.353380+0000) 2019-10-13T14:47:04.356 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:04.351+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.353910+0000) 2019-10-13T14:47:04.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:04.351+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.354123+0000) 2019-10-13T14:47:04.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:04.349+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.355056+0000) 2019-10-13T14:47:04.358 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:04.351+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.355753+0000) 2019-10-13T14:47:04.358 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:04.351+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.356176+0000) 2019-10-13T14:47:04.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:04.357+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.361068+0000) 2019-10-13T14:47:04.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:04.359+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212173 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:04.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:04.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.386426+0000) 2019-10-13T14:47:04.673 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:04.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:04.670866+0000) 2019-10-13T14:47:05.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:05.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:05.554873+0000) 2019-10-13T14:47:05.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:05.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:05.556285+0000) 2019-10-13T14:47:08.042 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:08.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:08.040095+0000) 2019-10-13T14:47:08.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:08.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:08.266269+0000) 2019-10-13T14:47:09.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:09.363+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.368003+0000) 2019-10-13T14:47:09.371 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:09.365+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.368357+0000) 2019-10-13T14:47:09.371 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:09.363+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.368604+0000) 2019-10-13T14:47:09.373 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:09.367+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.370834+0000) 2019-10-13T14:47:09.375 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:09.367+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.372855+0000) 2019-10-13T14:47:09.378 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:09.369+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.375779+0000) 2019-10-13T14:47:09.378 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:09.371+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.375644+0000) 2019-10-13T14:47:09.379 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:09.371+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:09.376948+0000) 2019-10-13T14:47:09.382 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:09.375+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:10.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:10.929+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:10.930217+0000) 2019-10-13T14:47:11.395 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:47:11.395 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:11.399 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:47:11.399 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:11.791 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:11.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:11.789051+0000) 2019-10-13T14:47:14.386 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:14.379+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.383450+0000) 2019-10-13T14:47:14.387 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:14.379+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.384467+0000) 2019-10-13T14:47:14.387 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:14.379+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.385173+0000) 2019-10-13T14:47:14.388 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:14.379+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.384598+0000) 2019-10-13T14:47:14.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:14.379+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.385631+0000) 2019-10-13T14:47:14.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:14.381+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.386266+0000) 2019-10-13T14:47:14.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:14.383+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.386489+0000) 2019-10-13T14:47:14.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:14.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.386569+0000) 2019-10-13T14:47:14.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:14.385+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.390705+0000) 2019-10-13T14:47:14.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:14.387+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212235 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:14.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:14.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:14.671047+0000) 2019-10-13T14:47:15.557 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:15.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:15.555061+0000) 2019-10-13T14:47:15.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:15.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:15.556486+0000) 2019-10-13T14:47:18.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:18.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:18.040340+0000) 2019-10-13T14:47:18.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:18.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:18.266483+0000) 2019-10-13T14:47:19.399 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:19.391+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.396148+0000) 2019-10-13T14:47:19.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:19.391+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.396405+0000) 2019-10-13T14:47:19.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:19.393+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.396992+0000) 2019-10-13T14:47:19.400 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:19.391+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.397692+0000) 2019-10-13T14:47:19.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:19.395+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.398556+0000) 2019-10-13T14:47:19.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:19.395+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.399033+0000) 2019-10-13T14:47:19.402 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:19.395+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.399293+0000) 2019-10-13T14:47:19.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:19.401+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:19.405256+0000) 2019-10-13T14:47:19.410 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:19.403+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212266 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:20.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:20.933+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:20.936706+0000) 2019-10-13T14:47:21.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:21.783+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:21.789284+0000) 2019-10-13T14:47:24.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:24.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.386763+0000) 2019-10-13T14:47:24.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:24.407+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.410340+0000) 2019-10-13T14:47:24.414 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:24.407+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.411102+0000) 2019-10-13T14:47:24.414 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:24.407+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.412043+0000) 2019-10-13T14:47:24.415 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:24.407+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.412298+0000) 2019-10-13T14:47:24.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:24.407+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.412706+0000) 2019-10-13T14:47:24.416 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:24.409+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.413686+0000) 2019-10-13T14:47:24.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:24.411+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.414862+0000) 2019-10-13T14:47:24.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:24.417+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.421106+0000) 2019-10-13T14:47:24.424 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:24.415+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212294 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:24.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:24.667+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:24.671232+0000) 2019-10-13T14:47:25.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:25.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:25.555276+0000) 2019-10-13T14:47:25.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:25.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:25.556682+0000) 2019-10-13T14:47:28.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:28.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:28.040578+0000) 2019-10-13T14:47:28.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:28.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:28.266699+0000) 2019-10-13T14:47:29.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:29.419+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.424910+0000) 2019-10-13T14:47:29.429 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:29.423+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.426535+0000) 2019-10-13T14:47:29.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:29.423+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.427235+0000) 2019-10-13T14:47:29.430 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:29.423+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.428235+0000) 2019-10-13T14:47:29.431 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:29.423+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.428714+0000) 2019-10-13T14:47:29.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:29.423+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.429181+0000) 2019-10-13T14:47:29.432 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:29.425+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.429779+0000) 2019-10-13T14:47:29.438 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:29.431+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:29.439 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:29.433+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:29.437211+0000) 2019-10-13T14:47:30.946 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:30.941+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:30.943171+0000) 2019-10-13T14:47:31.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:31.787+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:31.789523+0000) 2019-10-13T14:47:34.389 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:34.383+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.386962+0000) 2019-10-13T14:47:34.443 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:34.435+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.440128+0000) 2019-10-13T14:47:34.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:34.435+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.441112+0000) 2019-10-13T14:47:34.444 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:34.437+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.442283+0000) 2019-10-13T14:47:34.445 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:34.439+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.442561+0000) 2019-10-13T14:47:34.445 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:34.439+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.442957+0000) 2019-10-13T14:47:34.446 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:34.439+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.443501+0000) 2019-10-13T14:47:34.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:34.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.443734+0000) 2019-10-13T14:47:34.450 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:34.441+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.447988+0000) 2019-10-13T14:47:34.453 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:34.447+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212356 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:34.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:34.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:34.671375+0000) 2019-10-13T14:47:35.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:35.551+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:35.555495+0000) 2019-10-13T14:47:35.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:35.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:35.556871+0000) 2019-10-13T14:47:38.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:38.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:38.040778+0000) 2019-10-13T14:47:38.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:38.263+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:38.266898+0000) 2019-10-13T14:47:39.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:39.447+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.453318+0000) 2019-10-13T14:47:39.456 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:39.449+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.453867+0000) 2019-10-13T14:47:39.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:39.451+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.454559+0000) 2019-10-13T14:47:39.457 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:39.451+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.455275+0000) 2019-10-13T14:47:39.458 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:39.451+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.455803+0000) 2019-10-13T14:47:39.458 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:39.451+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.455908+0000) 2019-10-13T14:47:39.459 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:39.451+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.456390+0000) 2019-10-13T14:47:39.463 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:39.457+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:39.461278+0000) 2019-10-13T14:47:39.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:39.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212388 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:40.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:40.949+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:40.949734+0000) 2019-10-13T14:47:41.536 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:47:41.536 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:41.540 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:47:41.540 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:47:41.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:41.786+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:41.789801+0000) 2019-10-13T14:47:44.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:44.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.387199+0000) 2019-10-13T14:47:44.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:44.461+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.466883+0000) 2019-10-13T14:47:44.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:44.462+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.467384+0000) 2019-10-13T14:47:44.471 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:44.462+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.468094+0000) 2019-10-13T14:47:44.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:44.462+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.468703+0000) 2019-10-13T14:47:44.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:44.462+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.469238+0000) 2019-10-13T14:47:44.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:44.462+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.469555+0000) 2019-10-13T14:47:44.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:44.466+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.471241+0000) 2019-10-13T14:47:44.476 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:44.469+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.474114+0000) 2019-10-13T14:47:44.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:44.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212416 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:44.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:44.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:44.671561+0000) 2019-10-13T14:47:45.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:45.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:45.555660+0000) 2019-10-13T14:47:45.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:45.553+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:45.557094+0000) 2019-10-13T14:47:48.043 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:48.034+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:48.040977+0000) 2019-10-13T14:47:48.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:48.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:48.267097+0000) 2019-10-13T14:47:49.483 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:49.476+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.480696+0000) 2019-10-13T14:47:49.483 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:49.474+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.481253+0000) 2019-10-13T14:47:49.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:49.478+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.481734+0000) 2019-10-13T14:47:49.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:49.478+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.482891+0000) 2019-10-13T14:47:49.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:49.478+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.483305+0000) 2019-10-13T14:47:49.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:49.478+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.483668+0000) 2019-10-13T14:47:49.487 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:49.478+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.484521+0000) 2019-10-13T14:47:49.492 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:49.484+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:49.490323+0000) 2019-10-13T14:47:49.494 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:49.486+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:50.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:50.956+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:50.960703+0000) 2019-10-13T14:47:51.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:51.786+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:51.790045+0000) 2019-10-13T14:47:54.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:54.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.387400+0000) 2019-10-13T14:47:54.498 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:54.490+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.495379+0000) 2019-10-13T14:47:54.498 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:54.490+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.496206+0000) 2019-10-13T14:47:54.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:54.492+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.496500+0000) 2019-10-13T14:47:54.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:54.490+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.497231+0000) 2019-10-13T14:47:54.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:54.490+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.497643+0000) 2019-10-13T14:47:54.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:54.494+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.498596+0000) 2019-10-13T14:47:54.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:54.494+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.501085+0000) 2019-10-13T14:47:54.508 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:54.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212478 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:47:54.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:54.500+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.505963+0000) 2019-10-13T14:47:54.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:54.666+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:54.671770+0000) 2019-10-13T14:47:55.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:55.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:55.555829+0000) 2019-10-13T14:47:55.559 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:55.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:55.557288+0000) 2019-10-13T14:47:58.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:58.038+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:58.041210+0000) 2019-10-13T14:47:58.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:58.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:58.267328+0000) 2019-10-13T14:47:59.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:47:59.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.509617+0000) 2019-10-13T14:47:59.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:47:59.506+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.510112+0000) 2019-10-13T14:47:59.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:47:59.506+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.511849+0000) 2019-10-13T14:47:59.514 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:47:59.506+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.512133+0000) 2019-10-13T14:47:59.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:47:59.506+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.513692+0000) 2019-10-13T14:47:59.516 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:47:59.506+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.513714+0000) 2019-10-13T14:47:59.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:47:59.510+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.514088+0000) 2019-10-13T14:47:59.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:47:59.512+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:47:59.517763+0000) 2019-10-13T14:47:59.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:47:59.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212510 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:00.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:00.968+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:00.971882+0000) 2019-10-13T14:48:01.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:01.786+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:01.790200+0000) 2019-10-13T14:48:04.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:04.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.387610+0000) 2019-10-13T14:48:04.526 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:04.518+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.523325+0000) 2019-10-13T14:48:04.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:04.520+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.524484+0000) 2019-10-13T14:48:04.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:04.518+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.525007+0000) 2019-10-13T14:48:04.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:04.518+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.524599+0000) 2019-10-13T14:48:04.528 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:04.518+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.525171+0000) 2019-10-13T14:48:04.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:04.518+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.525477+0000) 2019-10-13T14:48:04.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:04.522+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.525923+0000) 2019-10-13T14:48:04.532 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:04.524+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.530133+0000) 2019-10-13T14:48:04.537 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:04.530+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212538 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:04.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:04.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:04.671962+0000) 2019-10-13T14:48:05.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:05.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:05.555984+0000) 2019-10-13T14:48:05.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:05.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:05.557479+0000) 2019-10-13T14:48:08.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:08.038+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:08.041368+0000) 2019-10-13T14:48:08.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:08.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:08.267545+0000) 2019-10-13T14:48:09.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:09.530+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.537576+0000) 2019-10-13T14:48:09.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:09.534+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.540492+0000) 2019-10-13T14:48:09.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:09.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.540562+0000) 2019-10-13T14:48:09.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:09.534+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.540990+0000) 2019-10-13T14:48:09.544 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:09.534+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.541595+0000) 2019-10-13T14:48:09.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:09.538+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.542937+0000) 2019-10-13T14:48:09.547 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:09.538+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.544686+0000) 2019-10-13T14:48:09.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:09.542+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212571 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:09.555 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:09.548+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:09.553325+0000) 2019-10-13T14:48:10.981 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:10.976+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:10.978290+0000) 2019-10-13T14:48:11.597 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:48:11.597 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:11.601 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:48:11.601 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:11.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:11.786+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:11.790389+0000) 2019-10-13T14:48:14.390 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:14.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.387853+0000) 2019-10-13T14:48:14.555 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:14.546+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.552195+0000) 2019-10-13T14:48:14.555 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:14.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.552740+0000) 2019-10-13T14:48:14.556 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:14.546+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.553107+0000) 2019-10-13T14:48:14.556 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:14.546+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.553514+0000) 2019-10-13T14:48:14.557 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:14.550+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.555085+0000) 2019-10-13T14:48:14.558 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:14.550+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.556109+0000) 2019-10-13T14:48:14.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:14.554+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.560460+0000) 2019-10-13T14:48:14.565 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:14.558+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212600 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:14.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:14.560+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.565563+0000) 2019-10-13T14:48:14.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:14.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:14.672135+0000) 2019-10-13T14:48:15.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:15.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:15.556128+0000) 2019-10-13T14:48:15.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:15.552+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:15.557672+0000) 2019-10-13T14:48:18.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:18.038+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:18.041598+0000) 2019-10-13T14:48:18.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:18.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:18.267740+0000) 2019-10-13T14:48:19.569 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:19.562+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.566640+0000) 2019-10-13T14:48:19.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:19.562+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.568953+0000) 2019-10-13T14:48:19.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:19.562+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.569264+0000) 2019-10-13T14:48:19.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:19.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.569776+0000) 2019-10-13T14:48:19.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:19.566+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.570505+0000) 2019-10-13T14:48:19.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:19.566+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.570420+0000) 2019-10-13T14:48:19.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:19.566+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.571085+0000) 2019-10-13T14:48:19.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:19.570+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:19.579 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:19.572+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:19.576942+0000) 2019-10-13T14:48:20.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:20.988+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:20.989973+0000) 2019-10-13T14:48:21.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:21.790+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:21.790602+0000) 2019-10-13T14:48:24.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:24.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.388123+0000) 2019-10-13T14:48:24.583 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:24.574+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.580208+0000) 2019-10-13T14:48:24.584 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:24.574+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.581495+0000) 2019-10-13T14:48:24.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:24.574+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.581546+0000) 2019-10-13T14:48:24.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:24.576+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.581975+0000) 2019-10-13T14:48:24.585 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:24.578+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.582140+0000) 2019-10-13T14:48:24.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:24.578+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.582900+0000) 2019-10-13T14:48:24.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:24.578+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.583036+0000) 2019-10-13T14:48:24.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:24.584+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.590602+0000) 2019-10-13T14:48:24.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:24.586+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212660 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:24.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:24.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:24.672309+0000) 2019-10-13T14:48:25.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:25.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:25.556289+0000) 2019-10-13T14:48:25.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:25.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:25.557892+0000) 2019-10-13T14:48:28.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:28.038+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:28.041801+0000) 2019-10-13T14:48:28.270 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:28.262+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:28.267949+0000) 2019-10-13T14:48:29.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:29.590+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.596535+0000) 2019-10-13T14:48:29.600 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:29.590+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.597628+0000) 2019-10-13T14:48:29.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:29.592+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.597969+0000) 2019-10-13T14:48:29.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:29.594+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.598293+0000) 2019-10-13T14:48:29.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:29.594+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.598752+0000) 2019-10-13T14:48:29.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:29.594+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.599063+0000) 2019-10-13T14:48:29.602 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:29.594+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.599383+0000) 2019-10-13T14:48:29.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:29.596+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:29.603595+0000) 2019-10-13T14:48:29.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:29.598+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212693 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:30.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:30.996+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:30.996453+0000) 2019-10-13T14:48:31.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:31.790+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:31.790805+0000) 2019-10-13T14:48:34.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:34.386+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.388324+0000) 2019-10-13T14:48:34.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:34.602+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.608932+0000) 2019-10-13T14:48:34.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:34.606+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.610606+0000) 2019-10-13T14:48:34.614 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:34.606+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.611516+0000) 2019-10-13T14:48:34.615 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:34.608+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.613407+0000) 2019-10-13T14:48:34.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:34.610+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.614137+0000) 2019-10-13T14:48:34.617 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:34.610+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.614657+0000) 2019-10-13T14:48:34.618 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:34.610+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.615021+0000) 2019-10-13T14:48:34.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:34.616+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.620339+0000) 2019-10-13T14:48:34.624 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:34.614+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212722 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:34.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:34.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:34.672475+0000) 2019-10-13T14:48:35.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:35.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:35.556412+0000) 2019-10-13T14:48:35.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:35.556+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:35.558066+0000) 2019-10-13T14:48:38.044 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:38.037+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:38.041996+0000) 2019-10-13T14:48:38.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:38.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:38.268144+0000) 2019-10-13T14:48:39.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:39.617+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.622412+0000) 2019-10-13T14:48:39.626 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:39.617+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.624135+0000) 2019-10-13T14:48:39.627 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:39.617+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.624576+0000) 2019-10-13T14:48:39.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:39.620+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.625190+0000) 2019-10-13T14:48:39.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:39.617+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.625375+0000) 2019-10-13T14:48:39.629 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:39.621+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.627211+0000) 2019-10-13T14:48:39.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:39.621+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.628297+0000) 2019-10-13T14:48:39.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:39.629+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:39.639 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:39.632+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:39.636661+0000) 2019-10-13T14:48:41.005 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:41.000+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:41.002951+0000) 2019-10-13T14:48:41.710 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:48:41.710 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:41.714 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:48:41.714 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:48:41.793 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:41.785+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:41.791008+0000) 2019-10-13T14:48:44.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:44.381+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.388527+0000) 2019-10-13T14:48:44.640 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:44.633+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.637513+0000) 2019-10-13T14:48:44.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:44.633+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.638663+0000) 2019-10-13T14:48:44.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:44.633+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.639192+0000) 2019-10-13T14:48:44.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:44.632+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.639898+0000) 2019-10-13T14:48:44.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:44.633+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.639826+0000) 2019-10-13T14:48:44.643 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:44.633+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.639407+0000) 2019-10-13T14:48:44.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:44.633+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.640470+0000) 2019-10-13T14:48:44.649 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:44.640+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.646650+0000) 2019-10-13T14:48:44.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:44.641+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:44.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:44.665+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:44.672683+0000) 2019-10-13T14:48:45.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:45.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:45.556634+0000) 2019-10-13T14:48:45.560 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:45.552+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:45.558255+0000) 2019-10-13T14:48:48.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:48.041+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:48.042197+0000) 2019-10-13T14:48:48.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:48.261+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:48.268337+0000) 2019-10-13T14:48:49.655 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:49.645+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.652735+0000) 2019-10-13T14:48:49.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:49.649+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.654431+0000) 2019-10-13T14:48:49.657 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:49.652+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.654994+0000) 2019-10-13T14:48:49.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:49.649+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.654639+0000) 2019-10-13T14:48:49.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:49.649+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.655603+0000) 2019-10-13T14:48:49.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:49.653+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.657905+0000) 2019-10-13T14:48:49.661 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:49.653+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.658167+0000) 2019-10-13T14:48:49.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:49.657+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212815 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:49.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:49.660+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:49.662333+0000) 2019-10-13T14:48:51.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:51.008+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:51.009477+0000) 2019-10-13T14:48:51.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:51.789+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:51.791219+0000) 2019-10-13T14:48:54.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:54.385+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.388764+0000) 2019-10-13T14:48:54.667 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:54.661+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.664979+0000) 2019-10-13T14:48:54.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:54.661+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.665554+0000) 2019-10-13T14:48:54.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:54.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.666407+0000) 2019-10-13T14:48:54.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:54.665+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.666805+0000) 2019-10-13T14:48:54.670 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:54.665+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.667442+0000) 2019-10-13T14:48:54.670 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:54.665+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.668054+0000) 2019-10-13T14:48:54.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:54.665+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.668736+0000) 2019-10-13T14:48:54.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:54.671+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.672744+0000) 2019-10-13T14:48:54.675 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:54.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:54.672856+0000) 2019-10-13T14:48:54.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:54.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212844 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:48:55.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:55.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:55.556827+0000) 2019-10-13T14:48:55.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:55.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:55.558430+0000) 2019-10-13T14:48:58.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:58.041+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:58.042399+0000) 2019-10-13T14:48:58.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:58.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:58.268532+0000) 2019-10-13T14:48:59.682 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:48:59.677+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.679205+0000) 2019-10-13T14:48:59.682 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:48:59.675+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.679568+0000) 2019-10-13T14:48:59.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:48:59.677+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.679811+0000) 2019-10-13T14:48:59.683 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:48:59.677+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.680358+0000) 2019-10-13T14:48:59.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:48:59.677+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.680583+0000) 2019-10-13T14:48:59.684 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:48:59.677+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.680937+0000) 2019-10-13T14:48:59.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:48:59.677+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.681664+0000) 2019-10-13T14:48:59.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:48:59.687+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:48:59.688694+0000) 2019-10-13T14:48:59.691 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:48:59.685+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212875 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:01.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:01.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:01.016019+0000) 2019-10-13T14:49:01.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:01.789+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:01.791427+0000) 2019-10-13T14:49:04.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:04.385+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.388965+0000) 2019-10-13T14:49:04.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:04.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.673047+0000) 2019-10-13T14:49:04.695 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:04.689+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.693097+0000) 2019-10-13T14:49:04.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:04.689+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.693618+0000) 2019-10-13T14:49:04.696 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:04.693+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.693983+0000) 2019-10-13T14:49:04.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:04.693+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.694437+0000) 2019-10-13T14:49:04.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:04.693+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.694694+0000) 2019-10-13T14:49:04.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:04.691+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.695749+0000) 2019-10-13T14:49:04.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:04.693+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.696267+0000) 2019-10-13T14:49:04.704 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:04.699+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:04.701648+0000) 2019-10-13T14:49:04.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:04.701+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212904 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:05.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:05.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:05.557044+0000) 2019-10-13T14:49:05.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:05.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:05.558596+0000) 2019-10-13T14:49:08.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:08.041+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:08.042625+0000) 2019-10-13T14:49:08.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:08.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:08.268739+0000) 2019-10-13T14:49:09.709 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:09.705+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.706975+0000) 2019-10-13T14:49:09.710 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:09.703+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.707938+0000) 2019-10-13T14:49:09.712 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:09.705+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.709332+0000) 2019-10-13T14:49:09.712 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:09.705+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.709719+0000) 2019-10-13T14:49:09.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:09.709+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.709970+0000) 2019-10-13T14:49:09.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:09.709+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.711289+0000) 2019-10-13T14:49:09.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:09.709+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.711910+0000) 2019-10-13T14:49:09.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:09.715+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:09.717268+0000) 2019-10-13T14:49:09.720 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:09.713+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212937 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:11.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:11.019+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:11.022521+0000) 2019-10-13T14:49:11.756 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:49:11.756 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:11.760 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:49:11.760 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:11.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:11.789+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:11.791625+0000) 2019-10-13T14:49:14.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:14.385+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.389169+0000) 2019-10-13T14:49:14.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:14.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.673246+0000) 2019-10-13T14:49:14.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:14.717+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.720441+0000) 2019-10-13T14:49:14.724 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:14.721+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.722193+0000) 2019-10-13T14:49:14.725 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:14.721+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.723070+0000) 2019-10-13T14:49:14.726 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:14.719+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.723345+0000) 2019-10-13T14:49:14.726 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:14.721+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.723658+0000) 2019-10-13T14:49:14.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:14.721+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.724132+0000) 2019-10-13T14:49:14.727 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:14.721+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.724801+0000) 2019-10-13T14:49:14.734 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:14.729+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212966 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:14.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:14.727+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:14.731889+0000) 2019-10-13T14:49:15.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:15.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:15.557224+0000) 2019-10-13T14:49:15.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:15.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:15.558773+0000) 2019-10-13T14:49:18.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:18.041+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:18.042826+0000) 2019-10-13T14:49:18.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:18.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:18.268934+0000) 2019-10-13T14:49:19.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:19.733+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.735285+0000) 2019-10-13T14:49:19.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:19.733+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.735512+0000) 2019-10-13T14:49:19.739 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:19.733+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.736758+0000) 2019-10-13T14:49:19.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:19.733+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.737198+0000) 2019-10-13T14:49:19.740 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:19.737+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.737943+0000) 2019-10-13T14:49:19.740 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:19.737+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.738004+0000) 2019-10-13T14:49:19.741 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:19.735+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.738490+0000) 2019-10-13T14:49:19.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:19.743+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:19.745320+0000) 2019-10-13T14:49:19.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:19.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212997 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:21.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:21.027+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:21.029144+0000) 2019-10-13T14:49:21.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:21.789+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:21.791833+0000) 2019-10-13T14:49:24.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:24.385+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.389368+0000) 2019-10-13T14:49:24.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:24.669+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.673464+0000) 2019-10-13T14:49:24.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:24.745+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.749648+0000) 2019-10-13T14:49:24.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:24.749+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.750686+0000) 2019-10-13T14:49:24.753 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:24.749+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.751058+0000) 2019-10-13T14:49:24.754 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:24.749+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.751574+0000) 2019-10-13T14:49:24.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:24.751+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.752346+0000) 2019-10-13T14:49:24.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:24.749+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.752979+0000) 2019-10-13T14:49:24.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:24.749+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.753395+0000) 2019-10-13T14:49:24.761 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:24.755+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:24.758973+0000) 2019-10-13T14:49:24.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:24.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213027 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:25.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:25.553+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:25.557401+0000) 2019-10-13T14:49:25.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:25.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:25.558902+0000) 2019-10-13T14:49:28.045 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:28.041+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:28.043064+0000) 2019-10-13T14:49:28.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:28.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:28.269130+0000) 2019-10-13T14:49:29.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:29.761+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.763075+0000) 2019-10-13T14:49:29.766 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:29.761+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.764127+0000) 2019-10-13T14:49:29.767 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:29.761+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.765289+0000) 2019-10-13T14:49:29.768 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:29.763+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.765558+0000) 2019-10-13T14:49:29.768 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:29.765+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.765735+0000) 2019-10-13T14:49:29.769 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:29.765+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.766191+0000) 2019-10-13T14:49:29.770 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:29.765+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.767551+0000) 2019-10-13T14:49:29.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:29.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213060 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:29.778 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:29.775+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:29.776248+0000) 2019-10-13T14:49:31.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:31.031+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:31.035825+0000) 2019-10-13T14:49:31.794 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:31.789+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:31.792036+0000) 2019-10-13T14:49:34.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:34.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.389587+0000) 2019-10-13T14:49:34.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:34.668+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.673664+0000) 2019-10-13T14:49:34.780 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:34.772+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.777581+0000) 2019-10-13T14:49:34.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:34.776+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.777944+0000) 2019-10-13T14:49:34.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:34.776+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.779082+0000) 2019-10-13T14:49:34.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:34.775+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.779956+0000) 2019-10-13T14:49:34.782 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:34.776+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.780344+0000) 2019-10-13T14:49:34.783 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:34.776+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.780298+0000) 2019-10-13T14:49:34.783 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:34.776+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.780828+0000) 2019-10-13T14:49:34.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:34.783+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:34.784951+0000) 2019-10-13T14:49:34.791 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:34.784+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213090 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:35.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:35.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:35.557574+0000) 2019-10-13T14:49:35.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:35.555+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:35.559071+0000) 2019-10-13T14:49:38.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:38.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:38.043269+0000) 2019-10-13T14:49:38.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:38.264+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:38.269369+0000) 2019-10-13T14:49:39.795 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:39.788+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.792647+0000) 2019-10-13T14:49:39.797 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:39.792+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.794600+0000) 2019-10-13T14:49:39.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:39.792+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.795682+0000) 2019-10-13T14:49:39.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:39.792+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.796076+0000) 2019-10-13T14:49:39.799 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:39.792+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.796745+0000) 2019-10-13T14:49:39.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:39.792+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.797201+0000) 2019-10-13T14:49:39.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:39.795+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.797732+0000) 2019-10-13T14:49:39.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:39.799+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:39.801049+0000) 2019-10-13T14:49:39.805 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:39.800+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:41.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:41.039+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:41.042379+0000) 2019-10-13T14:49:41.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:41.788+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:41.792275+0000) 2019-10-13T14:49:41.880 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:49:41.880 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:41.883 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:49:41.884 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:49:44.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:44.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.389806+0000) 2019-10-13T14:49:44.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:44.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.673898+0000) 2019-10-13T14:49:44.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:44.804+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.806926+0000) 2019-10-13T14:49:44.810 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:44.804+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.807207+0000) 2019-10-13T14:49:44.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:44.804+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.807575+0000) 2019-10-13T14:49:44.811 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:44.804+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.808266+0000) 2019-10-13T14:49:44.811 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:44.804+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.807810+0000) 2019-10-13T14:49:44.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:44.807+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.808861+0000) 2019-10-13T14:49:44.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:44.804+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.809390+0000) 2019-10-13T14:49:44.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:44.815+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:44.816113+0000) 2019-10-13T14:49:44.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:44.812+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213151 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:45.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:45.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:45.557792+0000) 2019-10-13T14:49:45.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:45.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:45.559268+0000) 2019-10-13T14:49:48.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:48.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:48.043475+0000) 2019-10-13T14:49:48.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:48.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:48.269568+0000) 2019-10-13T14:49:49.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:49.816+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.819576+0000) 2019-10-13T14:49:49.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:49.816+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.820046+0000) 2019-10-13T14:49:49.824 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:49.819+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.821514+0000) 2019-10-13T14:49:49.824 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:49.820+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.821792+0000) 2019-10-13T14:49:49.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:49.820+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.822227+0000) 2019-10-13T14:49:49.828 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:49.820+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.822917+0000) 2019-10-13T14:49:49.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:49.820+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.823655+0000) 2019-10-13T14:49:49.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:49.827+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:49.831239+0000) 2019-10-13T14:49:49.835 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:49.828+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213183 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:51.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:51.047+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:51.048911+0000) 2019-10-13T14:49:51.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:51.788+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:51.792492+0000) 2019-10-13T14:49:54.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:54.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.390047+0000) 2019-10-13T14:49:54.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:54.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.674121+0000) 2019-10-13T14:49:54.838 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:54.831+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.836040+0000) 2019-10-13T14:49:54.839 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:54.832+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.836401+0000) 2019-10-13T14:49:54.840 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:54.832+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.837625+0000) 2019-10-13T14:49:54.840 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:54.836+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.838004+0000) 2019-10-13T14:49:54.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:54.836+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.838593+0000) 2019-10-13T14:49:54.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:54.836+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.838211+0000) 2019-10-13T14:49:54.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:54.836+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.839577+0000) 2019-10-13T14:49:54.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:54.840+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213214 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:49:54.848 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:54.843+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:54.845872+0000) 2019-10-13T14:49:55.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:55.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:55.557960+0000) 2019-10-13T14:49:55.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:55.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:55.559463+0000) 2019-10-13T14:49:58.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:58.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:58.043698+0000) 2019-10-13T14:49:58.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:58.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:58.269790+0000) 2019-10-13T14:49:59.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:49:59.844+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.848641+0000) 2019-10-13T14:49:59.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:49:59.844+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.848787+0000) 2019-10-13T14:49:59.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:49:59.846+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.849696+0000) 2019-10-13T14:49:59.854 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:49:59.848+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.851435+0000) 2019-10-13T14:49:59.856 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:49:59.848+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.853513+0000) 2019-10-13T14:49:59.856 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:49:59.852+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.853966+0000) 2019-10-13T14:49:59.858 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:49:59.852+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.855396+0000) 2019-10-13T14:49:59.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:49:59.854+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:49:59.858109+0000) 2019-10-13T14:49:59.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:49:59.856+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213244 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:01.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:01.054+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:01.055359+0000) 2019-10-13T14:50:01.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:01.792+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:01.792785+0000) 2019-10-13T14:50:04.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:04.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.390218+0000) 2019-10-13T14:50:04.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:04.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.674340+0000) 2019-10-13T14:50:04.865 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:04.860+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.862129+0000) 2019-10-13T14:50:04.865 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:04.860+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.862646+0000) 2019-10-13T14:50:04.865 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:04.860+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.862977+0000) 2019-10-13T14:50:04.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:04.860+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.864245+0000) 2019-10-13T14:50:04.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:04.862+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.866244+0000) 2019-10-13T14:50:04.870 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:04.864+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.868168+0000) 2019-10-13T14:50:04.872 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:04.866+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.869577+0000) 2019-10-13T14:50:04.872 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:04.868+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:04.870006+0000) 2019-10-13T14:50:04.875 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:04.868+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213275 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:05.560 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:05.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:05.558145+0000) 2019-10-13T14:50:05.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:05.554+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:05.559671+0000) 2019-10-13T14:50:08.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:08.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:08.043902+0000) 2019-10-13T14:50:08.272 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:08.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:08.270007+0000) 2019-10-13T14:50:09.880 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:09.872+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.877140+0000) 2019-10-13T14:50:09.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:09.874+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.877501+0000) 2019-10-13T14:50:09.881 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:09.876+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.879127+0000) 2019-10-13T14:50:09.882 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:09.876+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.879522+0000) 2019-10-13T14:50:09.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:09.876+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.881446+0000) 2019-10-13T14:50:09.885 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:09.880+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.882781+0000) 2019-10-13T14:50:09.885 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:09.880+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.883137+0000) 2019-10-13T14:50:09.889 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:09.884+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213307 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:09.890 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:09.882+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:09.887780+0000) 2019-10-13T14:50:11.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:11.058+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:11.061763+0000) 2019-10-13T14:50:11.795 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:11.788+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:11.792985+0000) 2019-10-13T14:50:12.003 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:50:12.004 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:12.007 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:50:12.007 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:14.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:14.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.390420+0000) 2019-10-13T14:50:14.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:14.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.674555+0000) 2019-10-13T14:50:14.893 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:14.886+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.890375+0000) 2019-10-13T14:50:14.895 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:14.888+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.892287+0000) 2019-10-13T14:50:14.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:14.888+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.892680+0000) 2019-10-13T14:50:14.896 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:14.888+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.893045+0000) 2019-10-13T14:50:14.896 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:14.888+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.893681+0000) 2019-10-13T14:50:14.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:14.892+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.894304+0000) 2019-10-13T14:50:14.898 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:14.892+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.895422+0000) 2019-10-13T14:50:14.903 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:14.898+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:14.900784+0000) 2019-10-13T14:50:14.903 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:14.896+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213338 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:15.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:15.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:15.558299+0000) 2019-10-13T14:50:15.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:15.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:15.559838+0000) 2019-10-13T14:50:18.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:18.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:18.044099+0000) 2019-10-13T14:50:18.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:18.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:18.270203+0000) 2019-10-13T14:50:19.907 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:19.900+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.904934+0000) 2019-10-13T14:50:19.908 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:19.900+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.905638+0000) 2019-10-13T14:50:19.908 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:19.900+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.905708+0000) 2019-10-13T14:50:19.909 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:19.902+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.906167+0000) 2019-10-13T14:50:19.910 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:19.904+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.907493+0000) 2019-10-13T14:50:19.910 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:19.904+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.908175+0000) 2019-10-13T14:50:19.911 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:19.904+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.908191+0000) 2019-10-13T14:50:19.916 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:19.910+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:19.913847+0000) 2019-10-13T14:50:19.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:19.912+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213368 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:21.070 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:21.066+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:21.068168+0000) 2019-10-13T14:50:21.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:21.788+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:21.793227+0000) 2019-10-13T14:50:24.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:24.388+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.390639+0000) 2019-10-13T14:50:24.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:24.672+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.674755+0000) 2019-10-13T14:50:24.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:24.912+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.917555+0000) 2019-10-13T14:50:24.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:24.916+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.918013+0000) 2019-10-13T14:50:24.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:24.916+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.918617+0000) 2019-10-13T14:50:24.922 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:24.914+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.919304+0000) 2019-10-13T14:50:24.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:24.916+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.918888+0000) 2019-10-13T14:50:24.922 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:24.916+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.919605+0000) 2019-10-13T14:50:24.923 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:24.916+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.919864+0000) 2019-10-13T14:50:24.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:24.922+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:24.927460+0000) 2019-10-13T14:50:24.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:24.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213399 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:25.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:25.556+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:25.558473+0000) 2019-10-13T14:50:25.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:25.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:25.560008+0000) 2019-10-13T14:50:28.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:28.040+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:28.044294+0000) 2019-10-13T14:50:28.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:28.268+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:28.270420+0000) 2019-10-13T14:50:29.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:29.927+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.930909+0000) 2019-10-13T14:50:29.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:29.927+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.931744+0000) 2019-10-13T14:50:29.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:29.927+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.932451+0000) 2019-10-13T14:50:29.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:29.930+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.932925+0000) 2019-10-13T14:50:29.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:29.927+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.932966+0000) 2019-10-13T14:50:29.936 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:29.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.933100+0000) 2019-10-13T14:50:29.937 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:29.931+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.934494+0000) 2019-10-13T14:50:29.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:29.938+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:29.940376+0000) 2019-10-13T14:50:29.944 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:29.939+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213429 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:31.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:31.074+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:31.077750+0000) 2019-10-13T14:50:31.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:31.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:31.793429+0000) 2019-10-13T14:50:34.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:34.387+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.390837+0000) 2019-10-13T14:50:34.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:34.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.674972+0000) 2019-10-13T14:50:34.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:34.942+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.945468+0000) 2019-10-13T14:50:34.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:34.943+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.945941+0000) 2019-10-13T14:50:34.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:34.943+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.948206+0000) 2019-10-13T14:50:34.951 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:34.943+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.948238+0000) 2019-10-13T14:50:34.951 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:34.943+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.948820+0000) 2019-10-13T14:50:34.952 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:34.943+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.949048+0000) 2019-10-13T14:50:34.953 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:34.947+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.950891+0000) 2019-10-13T14:50:34.958 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:34.951+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:34.962 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:34.958+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:34.960166+0000) 2019-10-13T14:50:35.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:35.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:35.558652+0000) 2019-10-13T14:50:35.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:35.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:35.560178+0000) 2019-10-13T14:50:38.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:38.043+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:38.044510+0000) 2019-10-13T14:50:38.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:38.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:38.270610+0000) 2019-10-13T14:50:39.962 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:39.954+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.959603+0000) 2019-10-13T14:50:39.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:39.955+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.959973+0000) 2019-10-13T14:50:39.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:39.955+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.960785+0000) 2019-10-13T14:50:39.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:39.955+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.961143+0000) 2019-10-13T14:50:39.964 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:39.959+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.961882+0000) 2019-10-13T14:50:39.965 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:39.959+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.962460+0000) 2019-10-13T14:50:39.965 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:39.959+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.962835+0000) 2019-10-13T14:50:39.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:39.962+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:39.967761+0000) 2019-10-13T14:50:39.972 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:39.967+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213488 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:41.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:41.082+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:41.084141+0000) 2019-10-13T14:50:41.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:41.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:41.793646+0000) 2019-10-13T14:50:42.123 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:50:42.123 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:42.127 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:50:42.127 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:50:44.393 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:44.387+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.391069+0000) 2019-10-13T14:50:44.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:44.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.675201+0000) 2019-10-13T14:50:44.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:44.967+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.973103+0000) 2019-10-13T14:50:44.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:44.971+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.974216+0000) 2019-10-13T14:50:44.977 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:44.971+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.974590+0000) 2019-10-13T14:50:44.978 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:44.974+0000 7ff33ca6f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.976194+0000) 2019-10-13T14:50:44.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:44.971+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.976292+0000) 2019-10-13T14:50:44.980 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:44.975+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.978174+0000) 2019-10-13T14:50:44.982 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:44.975+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.979664+0000) 2019-10-13T14:50:44.987 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:44.979+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213518 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:44.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:44.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:44.986395+0000) 2019-10-13T14:50:45.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:45.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:45.558833+0000) 2019-10-13T14:50:45.562 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:45.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:45.560384+0000) 2019-10-13T14:50:48.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:48.043+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:48.044748+0000) 2019-10-13T14:50:48.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:48.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:48.270814+0000) 2019-10-13T14:50:49.990 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:49.983+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.987976+0000) 2019-10-13T14:50:49.991 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:49.983+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.988669+0000) 2019-10-13T14:50:49.992 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:49.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.989466+0000) 2019-10-13T14:50:49.992 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:49.986+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.989939+0000) 2019-10-13T14:50:49.993 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:49.987+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.990156+0000) 2019-10-13T14:50:49.993 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:49.987+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.991068+0000) 2019-10-13T14:50:49.994 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:49.987+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.991912+0000) 2019-10-13T14:50:49.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:49.994+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:49.996986+0000) 2019-10-13T14:50:50.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:49.995+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213549 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:51.093 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:51.086+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:51.090569+0000) 2019-10-13T14:50:51.796 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:51.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:51.793880+0000) 2019-10-13T14:50:54.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:54.387+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:54.391265+0000) 2019-10-13T14:50:54.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:54.671+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:54.675405+0000) 2019-10-13T14:50:55.005 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:54.999+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.002981+0000) 2019-10-13T14:50:55.006 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:54.999+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.003763+0000) 2019-10-13T14:50:55.007 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:50:54.999+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.004233+0000) 2019-10-13T14:50:55.007 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:55.002+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.004679+0000) 2019-10-13T14:50:55.007 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:54.999+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.004814+0000) 2019-10-13T14:50:55.008 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:50:54.999+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.005520+0000) 2019-10-13T14:50:55.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:50:55.003+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.007153+0000) 2019-10-13T14:50:55.015 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:50:55.007+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:50:55.016 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:50:55.010+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.013661+0000) 2019-10-13T14:50:55.561 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:50:55.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.559039+0000) 2019-10-13T14:50:55.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:50:55.558+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:55.560557+0000) 2019-10-13T14:50:58.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:50:58.043+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:58.044969+0000) 2019-10-13T14:50:58.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:50:58.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:50:58.271014+0000) 2019-10-13T14:51:00.019 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:00.011+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.016542+0000) 2019-10-13T14:51:00.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:00.011+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.017133+0000) 2019-10-13T14:51:00.020 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:00.011+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.017607+0000) 2019-10-13T14:51:00.021 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:00.015+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.017821+0000) 2019-10-13T14:51:00.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:00.014+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.017945+0000) 2019-10-13T14:51:00.021 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:00.015+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.018836+0000) 2019-10-13T14:51:00.022 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:00.015+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.019649+0000) 2019-10-13T14:51:00.028 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:00.022+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:00.025800+0000) 2019-10-13T14:51:00.029 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:00.023+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:01.099 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:01.094+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:01.097101+0000) 2019-10-13T14:51:01.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:01.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:01.794089+0000) 2019-10-13T14:51:04.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:04.391+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:04.391477+0000) 2019-10-13T14:51:04.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:04.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:04.675606+0000) 2019-10-13T14:51:05.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:05.027+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.030098+0000) 2019-10-13T14:51:05.033 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:05.027+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.030573+0000) 2019-10-13T14:51:05.034 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:05.027+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.031691+0000) 2019-10-13T14:51:05.034 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:05.027+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.032257+0000) 2019-10-13T14:51:05.035 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:05.027+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.033352+0000) 2019-10-13T14:51:05.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:05.029+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.033780+0000) 2019-10-13T14:51:05.036 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:05.031+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.034198+0000) 2019-10-13T14:51:05.043 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:05.037+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.040822+0000) 2019-10-13T14:51:05.043 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:05.035+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:05.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:05.555+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.559236+0000) 2019-10-13T14:51:05.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:05.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:05.560763+0000) 2019-10-13T14:51:08.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:08.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:08.045202+0000) 2019-10-13T14:51:08.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:08.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:08.271239+0000) 2019-10-13T14:51:10.048 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:10.039+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.045585+0000) 2019-10-13T14:51:10.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:10.043+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.046152+0000) 2019-10-13T14:51:10.049 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:10.043+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.046438+0000) 2019-10-13T14:51:10.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:10.041+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.046924+0000) 2019-10-13T14:51:10.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:10.047+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.052060+0000) 2019-10-13T14:51:10.056 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:10.049+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.053731+0000) 2019-10-13T14:51:10.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:10.051+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.054458+0000) 2019-10-13T14:51:10.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:10.051+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:10.058 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:10.051+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:10.055635+0000) 2019-10-13T14:51:11.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:11.105+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:11.108487+0000) 2019-10-13T14:51:11.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:11.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:11.794294+0000) 2019-10-13T14:51:12.421 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:51:12.421 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:12.425 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:51:12.425 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:14.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:14.391+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:14.391720+0000) 2019-10-13T14:51:14.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:14.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:14.675758+0000) 2019-10-13T14:51:15.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:15.055+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.058745+0000) 2019-10-13T14:51:15.062 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:15.055+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.059741+0000) 2019-10-13T14:51:15.062 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:15.055+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.060209+0000) 2019-10-13T14:51:15.063 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:15.057+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.060485+0000) 2019-10-13T14:51:15.063 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:15.055+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.060632+0000) 2019-10-13T14:51:15.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:15.055+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.061586+0000) 2019-10-13T14:51:15.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:15.059+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.062214+0000) 2019-10-13T14:51:15.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:15.061+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.066354+0000) 2019-10-13T14:51:15.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:15.063+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:15.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:15.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.559410+0000) 2019-10-13T14:51:15.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:15.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:15.561005+0000) 2019-10-13T14:51:18.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:18.039+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:18.045426+0000) 2019-10-13T14:51:18.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:18.267+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:18.271439+0000) 2019-10-13T14:51:20.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:20.067+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.072700+0000) 2019-10-13T14:51:20.077 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:20.069+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.074614+0000) 2019-10-13T14:51:20.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:20.071+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.074945+0000) 2019-10-13T14:51:20.078 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:20.071+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.075278+0000) 2019-10-13T14:51:20.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:20.071+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.075933+0000) 2019-10-13T14:51:20.079 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:20.071+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.076591+0000) 2019-10-13T14:51:20.079 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:20.071+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.077033+0000) 2019-10-13T14:51:20.086 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:20.077+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:20.083501+0000) 2019-10-13T14:51:20.086 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:20.079+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:21.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:21.113+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:21.114856+0000) 2019-10-13T14:51:21.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:21.791+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:21.794490+0000) 2019-10-13T14:51:24.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:24.387+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:24.391928+0000) 2019-10-13T14:51:24.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:24.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:24.675974+0000) 2019-10-13T14:51:25.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:25.082+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.087626+0000) 2019-10-13T14:51:25.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:25.082+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.087991+0000) 2019-10-13T14:51:25.091 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:25.082+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.088466+0000) 2019-10-13T14:51:25.091 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:25.085+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.089122+0000) 2019-10-13T14:51:25.092 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:25.082+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.089342+0000) 2019-10-13T14:51:25.094 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:25.086+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.091492+0000) 2019-10-13T14:51:25.094 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:25.086+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.092087+0000) 2019-10-13T14:51:25.098 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:25.093+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.096253+0000) 2019-10-13T14:51:25.100 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:25.094+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:25.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:25.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.559575+0000) 2019-10-13T14:51:25.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:25.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:25.561195+0000) 2019-10-13T14:51:28.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:28.042+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:28.045628+0000) 2019-10-13T14:51:28.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:28.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:28.271636+0000) 2019-10-13T14:51:30.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:30.094+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.101338+0000) 2019-10-13T14:51:30.104 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:30.098+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.101980+0000) 2019-10-13T14:51:30.105 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:30.097+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.102815+0000) 2019-10-13T14:51:30.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:30.098+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.102664+0000) 2019-10-13T14:51:30.107 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:30.098+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.104347+0000) 2019-10-13T14:51:30.107 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:30.098+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.104906+0000) 2019-10-13T14:51:30.109 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:30.102+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.107054+0000) 2019-10-13T14:51:30.113 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:30.105+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:30.110577+0000) 2019-10-13T14:51:30.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:30.106+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:31.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:31.117+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:31.121389+0000) 2019-10-13T14:51:31.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:31.790+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:31.794717+0000) 2019-10-13T14:51:34.394 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:34.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:34.392126+0000) 2019-10-13T14:51:34.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:34.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:34.676204+0000) 2019-10-13T14:51:35.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:35.110+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.116117+0000) 2019-10-13T14:51:35.119 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:35.110+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.116168+0000) 2019-10-13T14:51:35.119 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:35.110+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.116630+0000) 2019-10-13T14:51:35.120 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:35.110+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.116924+0000) 2019-10-13T14:51:35.120 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:35.114+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.118055+0000) 2019-10-13T14:51:35.121 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:35.113+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.118581+0000) 2019-10-13T14:51:35.121 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:35.114+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.118739+0000) 2019-10-13T14:51:35.127 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:35.121+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.125385+0000) 2019-10-13T14:51:35.129 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:35.122+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:35.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:35.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.559781+0000) 2019-10-13T14:51:35.563 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:35.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:35.561372+0000) 2019-10-13T14:51:38.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:38.042+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.045830+0000) 2019-10-13T14:51:38.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:38.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:38.271876+0000) 2019-10-13T14:51:40.134 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:40.126+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.131276+0000) 2019-10-13T14:51:40.134 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:40.126+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.132013+0000) 2019-10-13T14:51:40.135 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:40.126+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.132219+0000) 2019-10-13T14:51:40.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:40.126+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.132813+0000) 2019-10-13T14:51:40.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:40.126+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.133189+0000) 2019-10-13T14:51:40.137 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:40.130+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.134788+0000) 2019-10-13T14:51:40.138 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:40.129+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.135522+0000) 2019-10-13T14:51:40.143 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:40.134+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:40.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:40.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:40.141526+0000) 2019-10-13T14:51:41.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:41.125+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:41.127829+0000) 2019-10-13T14:51:41.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:41.790+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:41.794949+0000) 2019-10-13T14:51:42.464 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:51:42.465 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:42.469 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:51:42.469 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:51:44.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:44.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:44.392334+0000) 2019-10-13T14:51:44.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:44.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:44.676405+0000) 2019-10-13T14:51:45.146 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:45.138+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.144139+0000) 2019-10-13T14:51:45.148 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:45.142+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.146033+0000) 2019-10-13T14:51:45.149 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:45.142+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.146640+0000) 2019-10-13T14:51:45.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:45.142+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.147069+0000) 2019-10-13T14:51:45.150 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:45.141+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.147503+0000) 2019-10-13T14:51:45.150 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:45.142+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.147806+0000) 2019-10-13T14:51:45.150 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:45.142+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.147966+0000) 2019-10-13T14:51:45.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:45.150+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:45.160 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:45.153+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.157532+0000) 2019-10-13T14:51:45.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:45.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.559960+0000) 2019-10-13T14:51:45.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:45.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:45.561566+0000) 2019-10-13T14:51:48.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:48.042+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.046031+0000) 2019-10-13T14:51:48.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:48.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:48.272096+0000) 2019-10-13T14:51:50.160 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:50.154+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.157860+0000) 2019-10-13T14:51:50.161 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:50.154+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.158261+0000) 2019-10-13T14:51:50.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:50.154+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.158602+0000) 2019-10-13T14:51:50.162 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:50.153+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.159785+0000) 2019-10-13T14:51:50.162 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:50.154+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.159348+0000) 2019-10-13T14:51:50.163 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:50.154+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.160376+0000) 2019-10-13T14:51:50.163 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:50.154+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.160796+0000) 2019-10-13T14:51:50.171 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:50.165+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:50.169058+0000) 2019-10-13T14:51:50.172 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:50.166+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213914 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:51.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:51.133+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:51.134475+0000) 2019-10-13T14:51:51.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:51.794+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:51.795149+0000) 2019-10-13T14:51:54.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:54.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:54.392573+0000) 2019-10-13T14:51:54.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:54.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:54.676638+0000) 2019-10-13T14:51:55.176 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:51:55.166+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.173676+0000) 2019-10-13T14:51:55.177 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:55.170+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.173852+0000) 2019-10-13T14:51:55.177 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:55.169+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.174412+0000) 2019-10-13T14:51:55.178 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:55.170+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.174712+0000) 2019-10-13T14:51:55.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:51:55.170+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.176106+0000) 2019-10-13T14:51:55.179 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:51:55.170+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.176341+0000) 2019-10-13T14:51:55.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:55.174+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.178984+0000) 2019-10-13T14:51:55.187 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:51:55.178+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:51:55.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:51:55.181+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.186543+0000) 2019-10-13T14:51:55.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:51:55.554+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.560135+0000) 2019-10-13T14:51:55.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:51:55.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:55.561771+0000) 2019-10-13T14:51:58.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:51:58.042+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.046236+0000) 2019-10-13T14:51:58.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:51:58.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:51:58.272294+0000) 2019-10-13T14:52:00.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:00.182+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.187793+0000) 2019-10-13T14:52:00.191 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:00.182+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.188233+0000) 2019-10-13T14:52:00.191 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:00.185+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.189162+0000) 2019-10-13T14:52:00.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:00.186+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.189899+0000) 2019-10-13T14:52:00.192 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:00.186+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.189988+0000) 2019-10-13T14:52:00.193 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:00.186+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.190252+0000) 2019-10-13T14:52:00.193 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:00.186+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.191137+0000) 2019-10-13T14:52:00.201 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:00.194+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213974 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:00.203 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:00.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:00.200490+0000) 2019-10-13T14:52:01.144 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:01.137+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:01.141055+0000) 2019-10-13T14:52:01.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:01.794+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:01.795356+0000) 2019-10-13T14:52:04.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:04.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:04.392777+0000) 2019-10-13T14:52:04.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:04.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:04.676865+0000) 2019-10-13T14:52:05.205 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:05.198+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.202885+0000) 2019-10-13T14:52:05.207 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:05.198+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.205205+0000) 2019-10-13T14:52:05.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:05.198+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.205469+0000) 2019-10-13T14:52:05.208 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:05.201+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.205925+0000) 2019-10-13T14:52:05.209 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:05.202+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.206220+0000) 2019-10-13T14:52:05.209 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:05.202+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.206573+0000) 2019-10-13T14:52:05.209 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:05.202+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.207006+0000) 2019-10-13T14:52:05.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:05.206+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214005 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:05.216 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:05.209+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.214239+0000) 2019-10-13T14:52:05.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:05.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.560305+0000) 2019-10-13T14:52:05.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:05.557+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:05.561947+0000) 2019-10-13T14:52:08.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:08.042+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.046482+0000) 2019-10-13T14:52:08.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:08.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:08.272489+0000) 2019-10-13T14:52:10.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:10.212+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.216453+0000) 2019-10-13T14:52:10.219 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:10.210+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.216756+0000) 2019-10-13T14:52:10.220 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:10.210+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.217118+0000) 2019-10-13T14:52:10.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:10.210+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.217384+0000) 2019-10-13T14:52:10.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:10.214+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.218211+0000) 2019-10-13T14:52:10.221 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:10.214+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.218881+0000) 2019-10-13T14:52:10.223 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:10.214+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.220544+0000) 2019-10-13T14:52:10.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:10.222+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214036 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:10.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:10.224+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:10.231641+0000) 2019-10-13T14:52:11.150 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:11.144+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:11.147549+0000) 2019-10-13T14:52:11.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:11.794+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:11.795577+0000) 2019-10-13T14:52:12.590 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:52:12.590 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:12.594 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:52:12.594 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:14.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:14.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:14.392974+0000) 2019-10-13T14:52:14.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:14.670+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:14.677097+0000) 2019-10-13T14:52:15.233 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:15.224+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.230526+0000) 2019-10-13T14:52:15.234 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:15.226+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.231818+0000) 2019-10-13T14:52:15.236 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:15.230+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.233841+0000) 2019-10-13T14:52:15.236 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:15.230+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.234392+0000) 2019-10-13T14:52:15.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:15.230+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.234659+0000) 2019-10-13T14:52:15.237 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:15.230+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.234952+0000) 2019-10-13T14:52:15.238 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:15.230+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.235341+0000) 2019-10-13T14:52:15.242 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:15.236+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.240309+0000) 2019-10-13T14:52:15.243 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:15.234+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214067 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:15.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:15.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.560448+0000) 2019-10-13T14:52:15.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:15.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:15.562123+0000) 2019-10-13T14:52:18.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:18.046+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.046685+0000) 2019-10-13T14:52:18.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:18.266+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:18.272713+0000) 2019-10-13T14:52:20.247 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:20.240+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.244480+0000) 2019-10-13T14:52:20.247 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:20.238+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.245024+0000) 2019-10-13T14:52:20.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:20.238+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.245320+0000) 2019-10-13T14:52:20.248 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:20.238+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.245680+0000) 2019-10-13T14:52:20.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:20.242+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.246823+0000) 2019-10-13T14:52:20.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:20.242+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.247481+0000) 2019-10-13T14:52:20.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:20.242+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.247920+0000) 2019-10-13T14:52:20.255 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:20.248+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:20.252721+0000) 2019-10-13T14:52:20.258 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:20.250+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:21.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:21.152+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:21.153948+0000) 2019-10-13T14:52:21.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:21.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:21.795780+0000) 2019-10-13T14:52:24.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:24.389+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:24.393208+0000) 2019-10-13T14:52:24.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:24.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:24.677293+0000) 2019-10-13T14:52:25.262 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:25.253+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.259998+0000) 2019-10-13T14:52:25.263 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:25.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.260584+0000) 2019-10-13T14:52:25.263 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:25.253+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.260956+0000) 2019-10-13T14:52:25.266 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:25.257+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.263678+0000) 2019-10-13T14:52:25.266 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:25.260+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.264076+0000) 2019-10-13T14:52:25.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:25.257+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.265396+0000) 2019-10-13T14:52:25.268 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:25.261+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.266023+0000) 2019-10-13T14:52:25.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:25.260+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.267664+0000) 2019-10-13T14:52:25.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:25.261+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214128 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:25.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:25.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.560618+0000) 2019-10-13T14:52:25.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:25.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:25.562327+0000) 2019-10-13T14:52:28.049 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:28.045+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.046904+0000) 2019-10-13T14:52:28.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:28.265+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:28.272922+0000) 2019-10-13T14:52:30.275 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:30.265+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.272110+0000) 2019-10-13T14:52:30.275 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:30.265+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.273007+0000) 2019-10-13T14:52:30.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:30.265+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.273494+0000) 2019-10-13T14:52:30.277 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:30.269+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.274352+0000) 2019-10-13T14:52:30.277 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:30.269+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.274457+0000) 2019-10-13T14:52:30.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:30.269+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.275302+0000) 2019-10-13T14:52:30.278 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:30.268+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.275924+0000) 2019-10-13T14:52:30.284 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:30.276+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:30.282115+0000) 2019-10-13T14:52:30.287 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:30.277+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214158 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:31.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:31.156+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:31.160398+0000) 2019-10-13T14:52:31.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:31.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:31.795981+0000) 2019-10-13T14:52:34.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:34.389+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:34.393410+0000) 2019-10-13T14:52:34.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:34.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:34.677499+0000) 2019-10-13T14:52:35.290 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:35.285+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.287713+0000) 2019-10-13T14:52:35.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:35.285+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.288256+0000) 2019-10-13T14:52:35.291 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:35.285+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.289147+0000) 2019-10-13T14:52:35.292 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:35.285+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.289472+0000) 2019-10-13T14:52:35.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:35.289+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.290129+0000) 2019-10-13T14:52:35.293 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:35.284+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.290284+0000) 2019-10-13T14:52:35.293 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:35.289+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.291127+0000) 2019-10-13T14:52:35.300 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:35.297+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214189 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:35.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:35.296+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.300325+0000) 2019-10-13T14:52:35.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:35.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.560811+0000) 2019-10-13T14:52:35.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:35.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:35.562498+0000) 2019-10-13T14:52:38.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:38.045+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.047102+0000) 2019-10-13T14:52:38.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:38.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:38.273142+0000) 2019-10-13T14:52:40.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:40.297+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.301551+0000) 2019-10-13T14:52:40.304 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:40.301+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.301937+0000) 2019-10-13T14:52:40.306 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:40.301+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.303772+0000) 2019-10-13T14:52:40.306 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:40.300+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.304131+0000) 2019-10-13T14:52:40.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:40.301+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.304713+0000) 2019-10-13T14:52:40.308 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:40.301+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.305024+0000) 2019-10-13T14:52:40.308 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:40.301+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.304650+0000) 2019-10-13T14:52:40.314 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:40.309+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214218 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:40.314 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:40.304+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:40.311933+0000) 2019-10-13T14:52:41.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:41.164+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:41.166868+0000) 2019-10-13T14:52:41.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:41.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:41.796186+0000) 2019-10-13T14:52:42.703 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:52:42.703 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:42.707 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:52:42.707 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:52:44.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:44.389+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:44.393610+0000) 2019-10-13T14:52:44.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:44.673+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:44.677697+0000) 2019-10-13T14:52:45.318 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:45.313+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.314997+0000) 2019-10-13T14:52:45.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:45.313+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.315365+0000) 2019-10-13T14:52:45.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:45.312+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.316433+0000) 2019-10-13T14:52:45.319 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:45.313+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.315768+0000) 2019-10-13T14:52:45.319 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:45.313+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.316645+0000) 2019-10-13T14:52:45.320 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:45.317+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.317891+0000) 2019-10-13T14:52:45.320 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:45.317+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.318199+0000) 2019-10-13T14:52:45.326 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:45.316+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.323749+0000) 2019-10-13T14:52:45.327 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:45.321+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214251 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:45.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:45.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.561025+0000) 2019-10-13T14:52:45.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:45.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:45.562671+0000) 2019-10-13T14:52:48.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:48.045+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.047348+0000) 2019-10-13T14:52:48.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:48.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:48.273367+0000) 2019-10-13T14:52:50.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:50.325+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.328558+0000) 2019-10-13T14:52:50.331 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:50.324+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.329160+0000) 2019-10-13T14:52:50.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:50.325+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.329415+0000) 2019-10-13T14:52:50.332 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:50.329+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.330289+0000) 2019-10-13T14:52:50.334 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:50.329+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.331927+0000) 2019-10-13T14:52:50.337 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:50.333+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.334250+0000) 2019-10-13T14:52:50.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:50.333+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.334365+0000) 2019-10-13T14:52:50.338 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:50.328+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:50.335983+0000) 2019-10-13T14:52:50.342 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:50.337+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214279 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:51.181 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:51.176+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:51.178486+0000) 2019-10-13T14:52:51.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:51.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:51.796382+0000) 2019-10-13T14:52:54.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:54.393+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:54.393811+0000) 2019-10-13T14:52:54.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:54.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:54.677960+0000) 2019-10-13T14:52:55.345 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:52:55.341+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.342604+0000) 2019-10-13T14:52:55.348 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:55.340+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.345161+0000) 2019-10-13T14:52:55.348 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:55.341+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.343324+0000) 2019-10-13T14:52:55.348 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:55.341+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.343688+0000) 2019-10-13T14:52:55.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:52:55.341+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.344365+0000) 2019-10-13T14:52:55.349 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:55.341+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.345307+0000) 2019-10-13T14:52:55.349 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:52:55.345+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.345736+0000) 2019-10-13T14:52:55.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:52:55.348+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.352390+0000) 2019-10-13T14:52:55.356 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:52:55.349+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:52:55.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:52:55.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.561216+0000) 2019-10-13T14:52:55.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:52:55.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:55.562844+0000) 2019-10-13T14:52:58.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:52:58.045+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.047586+0000) 2019-10-13T14:52:58.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:52:58.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:52:58.273560+0000) 2019-10-13T14:53:00.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:00.352+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.356473+0000) 2019-10-13T14:53:00.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:00.353+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.356907+0000) 2019-10-13T14:53:00.360 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:00.353+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.357296+0000) 2019-10-13T14:53:00.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:00.357+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.358734+0000) 2019-10-13T14:53:00.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:00.357+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.359461+0000) 2019-10-13T14:53:00.363 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:00.357+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.360639+0000) 2019-10-13T14:53:00.364 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:00.357+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.361435+0000) 2019-10-13T14:53:00.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:00.360+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:00.365576+0000) 2019-10-13T14:53:00.370 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:00.365+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:01.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:01.184+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:01.184909+0000) 2019-10-13T14:53:01.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:01.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:01.796593+0000) 2019-10-13T14:53:04.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:04.393+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:04.394019+0000) 2019-10-13T14:53:04.680 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:04.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:04.678159+0000) 2019-10-13T14:53:05.374 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:05.369+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.371621+0000) 2019-10-13T14:53:05.374 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:05.369+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.372080+0000) 2019-10-13T14:53:05.375 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:05.369+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.372504+0000) 2019-10-13T14:53:05.376 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:05.373+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.374219+0000) 2019-10-13T14:53:05.377 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:05.373+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.374388+0000) 2019-10-13T14:53:05.377 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:05.368+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.374897+0000) 2019-10-13T14:53:05.378 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:05.373+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.375922+0000) 2019-10-13T14:53:05.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:05.376+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.381318+0000) 2019-10-13T14:53:05.385 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:05.381+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:05.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:05.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.561402+0000) 2019-10-13T14:53:05.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:05.560+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:05.563014+0000) 2019-10-13T14:53:08.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:08.045+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.047790+0000) 2019-10-13T14:53:08.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:08.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:08.273780+0000) 2019-10-13T14:53:10.388 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:10.381+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.385702+0000) 2019-10-13T14:53:10.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:10.385+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.386594+0000) 2019-10-13T14:53:10.391 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:10.385+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.388969+0000) 2019-10-13T14:53:10.392 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:10.385+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.389446+0000) 2019-10-13T14:53:10.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:10.389+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.391153+0000) 2019-10-13T14:53:10.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:10.392+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.392743+0000) 2019-10-13T14:53:10.396 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:10.393+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.394335+0000) 2019-10-13T14:53:10.397 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:10.392+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:10.394509+0000) 2019-10-13T14:53:10.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:10.393+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214401 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:11.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:11.188+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:11.191294+0000) 2019-10-13T14:53:11.799 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:11.793+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:11.796820+0000) 2019-10-13T14:53:12.767 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:53:12.767 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:12.770 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:53:12.771 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:14.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:14.393+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:14.394238+0000) 2019-10-13T14:53:14.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:14.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:14.678393+0000) 2019-10-13T14:53:15.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:15.397+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.400849+0000) 2019-10-13T14:53:15.404 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:15.397+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.401256+0000) 2019-10-13T14:53:15.405 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:15.399+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.401871+0000) 2019-10-13T14:53:15.405 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:15.401+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.402113+0000) 2019-10-13T14:53:15.406 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:15.401+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.403076+0000) 2019-10-13T14:53:15.406 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:15.401+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.403414+0000) 2019-10-13T14:53:15.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:15.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.404316+0000) 2019-10-13T14:53:15.411 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:15.407+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.408983+0000) 2019-10-13T14:53:15.414 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:15.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214433 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:15.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:15.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.561573+0000) 2019-10-13T14:53:15.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:15.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:15.563184+0000) 2019-10-13T14:53:18.050 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:18.044+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.047991+0000) 2019-10-13T14:53:18.276 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:18.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:18.273974+0000) 2019-10-13T14:53:20.418 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:20.412+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.415781+0000) 2019-10-13T14:53:20.419 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:20.412+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.415773+0000) 2019-10-13T14:53:20.419 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:20.412+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.416256+0000) 2019-10-13T14:53:20.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:20.412+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.417687+0000) 2019-10-13T14:53:20.421 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:20.416+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.418494+0000) 2019-10-13T14:53:20.421 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:20.415+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.419116+0000) 2019-10-13T14:53:20.422 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:20.416+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.419525+0000) 2019-10-13T14:53:20.429 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:20.424+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214461 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:20.430 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:20.423+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:20.427932+0000) 2019-10-13T14:53:21.200 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:21.195+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:21.198017+0000) 2019-10-13T14:53:21.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:21.792+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:21.797020+0000) 2019-10-13T14:53:24.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:24.392+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:24.394486+0000) 2019-10-13T14:53:24.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:24.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:24.678584+0000) 2019-10-13T14:53:25.432 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:25.424+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.429504+0000) 2019-10-13T14:53:25.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:25.428+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.430626+0000) 2019-10-13T14:53:25.433 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:25.427+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.431206+0000) 2019-10-13T14:53:25.434 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:25.428+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.431736+0000) 2019-10-13T14:53:25.435 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:25.428+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.432716+0000) 2019-10-13T14:53:25.435 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:25.428+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.432876+0000) 2019-10-13T14:53:25.436 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:25.428+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.433150+0000) 2019-10-13T14:53:25.442 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:25.435+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.439929+0000) 2019-10-13T14:53:25.443 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:25.436+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214495 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:25.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:25.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.561787+0000) 2019-10-13T14:53:25.565 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:25.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:25.563349+0000) 2019-10-13T14:53:28.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:28.044+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.048188+0000) 2019-10-13T14:53:28.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:28.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:28.274187+0000) 2019-10-13T14:53:30.446 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:30.440+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.444031+0000) 2019-10-13T14:53:30.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:30.440+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.444706+0000) 2019-10-13T14:53:30.448 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:30.444+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.445864+0000) 2019-10-13T14:53:30.448 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:30.444+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.446124+0000) 2019-10-13T14:53:30.449 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:30.443+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.446793+0000) 2019-10-13T14:53:30.449 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:30.444+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.447269+0000) 2019-10-13T14:53:30.450 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:30.444+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.447671+0000) 2019-10-13T14:53:30.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:30.447+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:30.450805+0000) 2019-10-13T14:53:30.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:30.452+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:31.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:31.203+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:31.204446+0000) 2019-10-13T14:53:31.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:31.796+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:31.797257+0000) 2019-10-13T14:53:34.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:34.392+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:34.394693+0000) 2019-10-13T14:53:34.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:34.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:34.678817+0000) 2019-10-13T14:53:35.460 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:35.456+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.457912+0000) 2019-10-13T14:53:35.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:35.456+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.459672+0000) 2019-10-13T14:53:35.462 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:35.456+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.459940+0000) 2019-10-13T14:53:35.463 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:35.456+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.460703+0000) 2019-10-13T14:53:35.464 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:35.456+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.461478+0000) 2019-10-13T14:53:35.464 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:35.456+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.461183+0000) 2019-10-13T14:53:35.464 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:35.459+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.461968+0000) 2019-10-13T14:53:35.466 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:35.459+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.464043+0000) 2019-10-13T14:53:35.470 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:35.464+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214555 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:35.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:35.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.561983+0000) 2019-10-13T14:53:35.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:35.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:35.563554+0000) 2019-10-13T14:53:38.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:38.044+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.048425+0000) 2019-10-13T14:53:38.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:38.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:38.274399+0000) 2019-10-13T14:53:40.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:40.468+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.471585+0000) 2019-10-13T14:53:40.475 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:40.471+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.472081+0000) 2019-10-13T14:53:40.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:40.468+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.472501+0000) 2019-10-13T14:53:40.476 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:40.468+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.472805+0000) 2019-10-13T14:53:40.476 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:40.468+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.473518+0000) 2019-10-13T14:53:40.476 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:40.472+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.474146+0000) 2019-10-13T14:53:40.477 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:40.472+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.474829+0000) 2019-10-13T14:53:40.482 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:40.479+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:40.480082+0000) 2019-10-13T14:53:40.485 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:40.480+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214583 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:41.213 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:41.207+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:41.210976+0000) 2019-10-13T14:53:41.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:41.796+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:41.797468+0000) 2019-10-13T14:53:42.875 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:53:42.876 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:42.879 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:53:42.880 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:53:44.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:44.392+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:44.394851+0000) 2019-10-13T14:53:44.681 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:44.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:44.679014+0000) 2019-10-13T14:53:45.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:45.484+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.486999+0000) 2019-10-13T14:53:45.490 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:45.484+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.487217+0000) 2019-10-13T14:53:45.490 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:45.484+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.487622+0000) 2019-10-13T14:53:45.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:45.484+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.487932+0000) 2019-10-13T14:53:45.491 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:45.487+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.488403+0000) 2019-10-13T14:53:45.492 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:45.484+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.488788+0000) 2019-10-13T14:53:45.492 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:45.484+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.489368+0000) 2019-10-13T14:53:45.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:45.491+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.493146+0000) 2019-10-13T14:53:45.500 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:45.492+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214616 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:45.564 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:45.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.562180+0000) 2019-10-13T14:53:45.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:45.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:45.563726+0000) 2019-10-13T14:53:48.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:48.044+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.048625+0000) 2019-10-13T14:53:48.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:48.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:48.274552+0000) 2019-10-13T14:53:50.503 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:50.496+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.500872+0000) 2019-10-13T14:53:50.505 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:50.500+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.502829+0000) 2019-10-13T14:53:50.505 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:50.499+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.503232+0000) 2019-10-13T14:53:50.506 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:50.500+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.503994+0000) 2019-10-13T14:53:50.508 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:50.500+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.505526+0000) 2019-10-13T14:53:50.511 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:50.504+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.508687+0000) 2019-10-13T14:53:50.513 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:50.508+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.510729+0000) 2019-10-13T14:53:50.514 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:50.508+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:50.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:50.515+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:50.518162+0000) 2019-10-13T14:53:51.225 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:51.219+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:51.222237+0000) 2019-10-13T14:53:51.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:51.792+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:51.797701+0000) 2019-10-13T14:53:54.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:54.392+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:54.395052+0000) 2019-10-13T14:53:54.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:54.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:54.679249+0000) 2019-10-13T14:53:55.518 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:53:55.512+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.515221+0000) 2019-10-13T14:53:55.519 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:55.512+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.517149+0000) 2019-10-13T14:53:55.520 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:55.516+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.518205+0000) 2019-10-13T14:53:55.521 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:55.515+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.518768+0000) 2019-10-13T14:53:55.521 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:55.516+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.519089+0000) 2019-10-13T14:53:55.522 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:53:55.516+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.519762+0000) 2019-10-13T14:53:55.522 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:53:55.516+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.520036+0000) 2019-10-13T14:53:55.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:53:55.523+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.524288+0000) 2019-10-13T14:53:55.527 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:53:55.520+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214675 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:53:55.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:53:55.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.562355+0000) 2019-10-13T14:53:55.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:53:55.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:55.563902+0000) 2019-10-13T14:53:58.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:53:58.048+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.048826+0000) 2019-10-13T14:53:58.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:53:58.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:53:58.274774+0000) 2019-10-13T14:54:00.532 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:00.524+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.529165+0000) 2019-10-13T14:54:00.533 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:00.527+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.529614+0000) 2019-10-13T14:54:00.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:00.528+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.529865+0000) 2019-10-13T14:54:00.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:00.528+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.530568+0000) 2019-10-13T14:54:00.534 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:00.528+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.531564+0000) 2019-10-13T14:54:00.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:00.528+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.532109+0000) 2019-10-13T14:54:00.536 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:00.528+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.533506+0000) 2019-10-13T14:54:00.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:00.531+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:00.535178+0000) 2019-10-13T14:54:00.542 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:00.536+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:01.231 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:01.227+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:01.228697+0000) 2019-10-13T14:54:01.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:01.796+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:01.797969+0000) 2019-10-13T14:54:04.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:04.392+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:04.395253+0000) 2019-10-13T14:54:04.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:04.676+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:04.679478+0000) 2019-10-13T14:54:05.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:05.540+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.542698+0000) 2019-10-13T14:54:05.546 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:05.540+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.543326+0000) 2019-10-13T14:54:05.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:05.540+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.545530+0000) 2019-10-13T14:54:05.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:05.544+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.546244+0000) 2019-10-13T14:54:05.549 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:05.543+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.546734+0000) 2019-10-13T14:54:05.549 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:05.544+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.546798+0000) 2019-10-13T14:54:05.550 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:05.544+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.547335+0000) 2019-10-13T14:54:05.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:05.551+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.553590+0000) 2019-10-13T14:54:05.556 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:05.548+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214736 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:05.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:05.560+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.562493+0000) 2019-10-13T14:54:05.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:05.559+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:05.564051+0000) 2019-10-13T14:54:08.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:08.044+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.048981+0000) 2019-10-13T14:54:08.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:08.272+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:08.274967+0000) 2019-10-13T14:54:10.560 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:10.552+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.557560+0000) 2019-10-13T14:54:10.561 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:10.556+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.558474+0000) 2019-10-13T14:54:10.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:10.556+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.558762+0000) 2019-10-13T14:54:10.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:10.555+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.559213+0000) 2019-10-13T14:54:10.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:10.556+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.559457+0000) 2019-10-13T14:54:10.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:10.556+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.560119+0000) 2019-10-13T14:54:10.563 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:10.556+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.560395+0000) 2019-10-13T14:54:10.568 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:10.563+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:10.565945+0000) 2019-10-13T14:54:10.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:10.564+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214764 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:11.237 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:11.231+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:11.235123+0000) 2019-10-13T14:54:11.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:11.796+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:11.798173+0000) 2019-10-13T14:54:13.008 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:54:13.008 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:13.012 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:54:13.012 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:14.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:14.391+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:14.395492+0000) 2019-10-13T14:54:14.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:14.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:14.679686+0000) 2019-10-13T14:54:15.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:15.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.562658+0000) 2019-10-13T14:54:15.566 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:15.563+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.564278+0000) 2019-10-13T14:54:15.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:15.567+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.571066+0000) 2019-10-13T14:54:15.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:15.567+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.571634+0000) 2019-10-13T14:54:15.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:15.567+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.571862+0000) 2019-10-13T14:54:15.575 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:15.567+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.572565+0000) 2019-10-13T14:54:15.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:15.571+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.573376+0000) 2019-10-13T14:54:15.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:15.567+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.573677+0000) 2019-10-13T14:54:15.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:15.571+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.575197+0000) 2019-10-13T14:54:15.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:15.579+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:15.580715+0000) 2019-10-13T14:54:15.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:15.575+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:18.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:18.047+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.049197+0000) 2019-10-13T14:54:18.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:18.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:18.275167+0000) 2019-10-13T14:54:20.588 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:20.579+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.585168+0000) 2019-10-13T14:54:20.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:20.583+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.586065+0000) 2019-10-13T14:54:20.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:20.583+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.587283+0000) 2019-10-13T14:54:20.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:20.583+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.586642+0000) 2019-10-13T14:54:20.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:20.583+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.587733+0000) 2019-10-13T14:54:20.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:20.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.588379+0000) 2019-10-13T14:54:20.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:20.583+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.588534+0000) 2019-10-13T14:54:20.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:20.586+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:20.591309+0000) 2019-10-13T14:54:20.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:20.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214823 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:21.244 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:21.238+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:21.241512+0000) 2019-10-13T14:54:21.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:21.795+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:21.798424+0000) 2019-10-13T14:54:24.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:24.391+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:24.395717+0000) 2019-10-13T14:54:24.682 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:24.679+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:24.679885+0000) 2019-10-13T14:54:25.565 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:25.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.562854+0000) 2019-10-13T14:54:25.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:25.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.564471+0000) 2019-10-13T14:54:25.601 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:25.595+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.598776+0000) 2019-10-13T14:54:25.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:25.595+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.598720+0000) 2019-10-13T14:54:25.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:25.595+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.601422+0000) 2019-10-13T14:54:25.606 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:25.599+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.604247+0000) 2019-10-13T14:54:25.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:25.603+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.605886+0000) 2019-10-13T14:54:25.608 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:25.603+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.606146+0000) 2019-10-13T14:54:25.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:25.602+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.606802+0000) 2019-10-13T14:54:25.612 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:25.606+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:25.609433+0000) 2019-10-13T14:54:25.612 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:25.603+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214856 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:28.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:28.043+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.049411+0000) 2019-10-13T14:54:28.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:28.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:28.275405+0000) 2019-10-13T14:54:30.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:30.607+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.613141+0000) 2019-10-13T14:54:30.616 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:30.607+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.613296+0000) 2019-10-13T14:54:30.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:30.610+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.615546+0000) 2019-10-13T14:54:30.618 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:30.611+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.615834+0000) 2019-10-13T14:54:30.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:30.611+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.616916+0000) 2019-10-13T14:54:30.620 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:30.611+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.617372+0000) 2019-10-13T14:54:30.620 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:30.615+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.617780+0000) 2019-10-13T14:54:30.625 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:30.618+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:30.622935+0000) 2019-10-13T14:54:30.626 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:30.619+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214883 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:31.250 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:31.242+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:31.247972+0000) 2019-10-13T14:54:31.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:31.795+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:31.798629+0000) 2019-10-13T14:54:34.398 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:34.395+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:34.395939+0000) 2019-10-13T14:54:34.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:34.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:34.680080+0000) 2019-10-13T14:54:35.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:35.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.563051+0000) 2019-10-13T14:54:35.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:35.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.564664+0000) 2019-10-13T14:54:35.630 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:35.622+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.627765+0000) 2019-10-13T14:54:35.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:35.623+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.627920+0000) 2019-10-13T14:54:35.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:35.623+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.628392+0000) 2019-10-13T14:54:35.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:35.623+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.628836+0000) 2019-10-13T14:54:35.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:35.623+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.629231+0000) 2019-10-13T14:54:35.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:35.623+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.629706+0000) 2019-10-13T14:54:35.633 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:35.627+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.630111+0000) 2019-10-13T14:54:35.636 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:35.630+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:35.633527+0000) 2019-10-13T14:54:35.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:35.635+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:38.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:38.047+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.049613+0000) 2019-10-13T14:54:38.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:38.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:38.275614+0000) 2019-10-13T14:54:40.645 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:40.639+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.642208+0000) 2019-10-13T14:54:40.645 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:40.639+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.642354+0000) 2019-10-13T14:54:40.646 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:40.638+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.643603+0000) 2019-10-13T14:54:40.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:40.639+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.643799+0000) 2019-10-13T14:54:40.646 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:40.639+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.644257+0000) 2019-10-13T14:54:40.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:40.639+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.644941+0000) 2019-10-13T14:54:40.647 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:40.639+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.645205+0000) 2019-10-13T14:54:40.653 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:40.646+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:40.651090+0000) 2019-10-13T14:54:40.659 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:40.647+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214942 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:41.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:41.250+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:41.254587+0000) 2019-10-13T14:54:41.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:41.795+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:41.798844+0000) 2019-10-13T14:54:43.130 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:54:43.130 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:43.134 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:54:43.134 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:54:44.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:44.395+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:44.396163+0000) 2019-10-13T14:54:44.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:44.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:44.680335+0000) 2019-10-13T14:54:45.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:45.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.563243+0000) 2019-10-13T14:54:45.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:45.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.564883+0000) 2019-10-13T14:54:45.658 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:45.651+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.655460+0000) 2019-10-13T14:54:45.661 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:45.655+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.658413+0000) 2019-10-13T14:54:45.661 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:45.655+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.659002+0000) 2019-10-13T14:54:45.666 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:45.658+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.661705+0000) 2019-10-13T14:54:45.667 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:45.658+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.662633+0000) 2019-10-13T14:54:45.667 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:45.655+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.660890+0000) 2019-10-13T14:54:45.667 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:45.655+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.661382+0000) 2019-10-13T14:54:45.667 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:45.659+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:45.662789+0000) 2019-10-13T14:54:45.669 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:45.663+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214976 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:48.052 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:48.047+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.049841+0000) 2019-10-13T14:54:48.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:48.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:48.275811+0000) 2019-10-13T14:54:50.673 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:50.667+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.670188+0000) 2019-10-13T14:54:50.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:50.667+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.670749+0000) 2019-10-13T14:54:50.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:50.667+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.671469+0000) 2019-10-13T14:54:50.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:50.667+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.671808+0000) 2019-10-13T14:54:50.677 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:50.671+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.674401+0000) 2019-10-13T14:54:50.677 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:50.670+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.673731+0000) 2019-10-13T14:54:50.679 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:50.671+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.677076+0000) 2019-10-13T14:54:50.682 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:50.675+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215003 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:50.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:50.678+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:50.680469+0000) 2019-10-13T14:54:51.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:51.258+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:51.261120+0000) 2019-10-13T14:54:51.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:51.795+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:51.799079+0000) 2019-10-13T14:54:54.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:54.391+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:54.396375+0000) 2019-10-13T14:54:54.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:54.675+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:54.680539+0000) 2019-10-13T14:54:55.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:55.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.563454+0000) 2019-10-13T14:54:55.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:55.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.565105+0000) 2019-10-13T14:54:55.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:54:55.679+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.682269+0000) 2019-10-13T14:54:55.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:55.679+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.683122+0000) 2019-10-13T14:54:55.686 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:55.679+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.684069+0000) 2019-10-13T14:54:55.687 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:54:55.682+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.684277+0000) 2019-10-13T14:54:55.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:54:55.679+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.684649+0000) 2019-10-13T14:54:55.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:54:55.679+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.684857+0000) 2019-10-13T14:54:55.688 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:54:55.679+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.685189+0000) 2019-10-13T14:54:55.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:54:55.686+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:55.689751+0000) 2019-10-13T14:54:55.697 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:54:55.691+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215035 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:54:58.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:54:58.047+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.050088+0000) 2019-10-13T14:54:58.278 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:54:58.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:54:58.276008+0000) 2019-10-13T14:55:00.701 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:00.695+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.698553+0000) 2019-10-13T14:55:00.702 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:00.695+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.699695+0000) 2019-10-13T14:55:00.703 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:00.695+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.700493+0000) 2019-10-13T14:55:00.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:00.695+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.700935+0000) 2019-10-13T14:55:00.704 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:00.695+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.701121+0000) 2019-10-13T14:55:00.704 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:00.698+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.701649+0000) 2019-10-13T14:55:00.705 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:00.699+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.702635+0000) 2019-10-13T14:55:00.711 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:00.703+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:00.714 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:00.706+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:00.711953+0000) 2019-10-13T14:55:01.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:01.266+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:01.268346+0000) 2019-10-13T14:55:01.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:01.799+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:01.799296+0000) 2019-10-13T14:55:04.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:04.395+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:04.396601+0000) 2019-10-13T14:55:04.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:04.679+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:04.680782+0000) 2019-10-13T14:55:05.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:05.559+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.563604+0000) 2019-10-13T14:55:05.567 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:05.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.565307+0000) 2019-10-13T14:55:05.715 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:05.707+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.712317+0000) 2019-10-13T14:55:05.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:05.710+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.713017+0000) 2019-10-13T14:55:05.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:05.707+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.713642+0000) 2019-10-13T14:55:05.716 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:05.711+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.714159+0000) 2019-10-13T14:55:05.719 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:05.711+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.716470+0000) 2019-10-13T14:55:05.721 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:05.715+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.718689+0000) 2019-10-13T14:55:05.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:05.715+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.720489+0000) 2019-10-13T14:55:05.724 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:05.718+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:05.722049+0000) 2019-10-13T14:55:05.725 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:05.719+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215096 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:08.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:08.047+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.050287+0000) 2019-10-13T14:55:08.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:08.271+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:08.276207+0000) 2019-10-13T14:55:10.729 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:10.722+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.726527+0000) 2019-10-13T14:55:10.730 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:10.722+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.728146+0000) 2019-10-13T14:55:10.731 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:10.722+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.728480+0000) 2019-10-13T14:55:10.732 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:10.726+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.729800+0000) 2019-10-13T14:55:10.733 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:10.726+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.731022+0000) 2019-10-13T14:55:10.734 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:10.726+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.731652+0000) 2019-10-13T14:55:10.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:10.726+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.732064+0000) 2019-10-13T14:55:10.740 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:10.734+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215123 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:10.742 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:10.734+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:10.739369+0000) 2019-10-13T14:55:11.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:11.274+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:11.275724+0000) 2019-10-13T14:55:11.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:11.794+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:11.799519+0000) 2019-10-13T14:55:13.214 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:55:13.214 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:13.218 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:55:13.218 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:14.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:14.390+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:14.396803+0000) 2019-10-13T14:55:14.683 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:14.678+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:14.680987+0000) 2019-10-13T14:55:15.566 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:15.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.563828+0000) 2019-10-13T14:55:15.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:15.562+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.565507+0000) 2019-10-13T14:55:15.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:15.734+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.741433+0000) 2019-10-13T14:55:15.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:15.738+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.742029+0000) 2019-10-13T14:55:15.745 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:15.738+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.742614+0000) 2019-10-13T14:55:15.746 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:15.738+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.743433+0000) 2019-10-13T14:55:15.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:15.738+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.744112+0000) 2019-10-13T14:55:15.747 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:15.738+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.744007+0000) 2019-10-13T14:55:15.747 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:15.738+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.744645+0000) 2019-10-13T14:55:15.754 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:15.746+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:15.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:15.750+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:15.752473+0000) 2019-10-13T14:55:18.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:18.046+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.050489+0000) 2019-10-13T14:55:18.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:18.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:18.276441+0000) 2019-10-13T14:55:20.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:20.750+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.755869+0000) 2019-10-13T14:55:20.759 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:20.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.756403+0000) 2019-10-13T14:55:20.759 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:20.750+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.756562+0000) 2019-10-13T14:55:20.760 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:20.754+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.758333+0000) 2019-10-13T14:55:20.761 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:20.754+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.758798+0000) 2019-10-13T14:55:20.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:20.754+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.760363+0000) 2019-10-13T14:55:20.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:20.754+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.760704+0000) 2019-10-13T14:55:20.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:20.762+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:20.767375+0000) 2019-10-13T14:55:20.770 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:20.762+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215182 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:21.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:21.282+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:21.282786+0000) 2019-10-13T14:55:21.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:21.794+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:21.799721+0000) 2019-10-13T14:55:24.399 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:24.394+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:24.397011+0000) 2019-10-13T14:55:24.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:24.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:24.681176+0000) 2019-10-13T14:55:25.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:25.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.564078+0000) 2019-10-13T14:55:25.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:25.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.565703+0000) 2019-10-13T14:55:25.776 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:25.766+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.773283+0000) 2019-10-13T14:55:25.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:25.766+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.773208+0000) 2019-10-13T14:55:25.777 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:25.770+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.773905+0000) 2019-10-13T14:55:25.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:25.770+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.774112+0000) 2019-10-13T14:55:25.777 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:25.770+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.775176+0000) 2019-10-13T14:55:25.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:25.769+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.775833+0000) 2019-10-13T14:55:25.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:25.770+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.776126+0000) 2019-10-13T14:55:25.783 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:25.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215215 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:25.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:25.777+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:25.782227+0000) 2019-10-13T14:55:28.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:28.046+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.050689+0000) 2019-10-13T14:55:28.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:28.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:28.276645+0000) 2019-10-13T14:55:30.787 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:30.781+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.784576+0000) 2019-10-13T14:55:30.787 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:30.778+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.785085+0000) 2019-10-13T14:55:30.788 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:30.778+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.785583+0000) 2019-10-13T14:55:30.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:30.782+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.786972+0000) 2019-10-13T14:55:30.790 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:30.782+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.787685+0000) 2019-10-13T14:55:30.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:30.786+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.789805+0000) 2019-10-13T14:55:30.794 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:30.786+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.791869+0000) 2019-10-13T14:55:30.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:30.790+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215243 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:30.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:30.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:30.795984+0000) 2019-10-13T14:55:31.292 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:31.285+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:31.289793+0000) 2019-10-13T14:55:31.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:31.798+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:31.799977+0000) 2019-10-13T14:55:34.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:34.394+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:34.397231+0000) 2019-10-13T14:55:34.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:34.678+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:34.681413+0000) 2019-10-13T14:55:35.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:35.558+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.564266+0000) 2019-10-13T14:55:35.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:35.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.565923+0000) 2019-10-13T14:55:35.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:35.794+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.798739+0000) 2019-10-13T14:55:35.802 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:35.794+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.799026+0000) 2019-10-13T14:55:35.803 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:35.794+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.800772+0000) 2019-10-13T14:55:35.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:35.794+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.801092+0000) 2019-10-13T14:55:35.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:35.797+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.803763+0000) 2019-10-13T14:55:35.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:35.798+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.805510+0000) 2019-10-13T14:55:35.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:35.802+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.807615+0000) 2019-10-13T14:55:35.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:35.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:35.815 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:35.809+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:35.812945+0000) 2019-10-13T14:55:38.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:38.050+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.050917+0000) 2019-10-13T14:55:38.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:38.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:38.276901+0000) 2019-10-13T14:55:40.815 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:40.806+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.812597+0000) 2019-10-13T14:55:40.816 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:40.806+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.813615+0000) 2019-10-13T14:55:40.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:40.810+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.813873+0000) 2019-10-13T14:55:40.817 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:40.810+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.814440+0000) 2019-10-13T14:55:40.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:40.810+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.815333+0000) 2019-10-13T14:55:40.818 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:40.810+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.815812+0000) 2019-10-13T14:55:40.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:40.813+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.816787+0000) 2019-10-13T14:55:40.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:40.817+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:40.823153+0000) 2019-10-13T14:55:40.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:40.818+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:41.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:41.293+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:41.296831+0000) 2019-10-13T14:55:41.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:41.798+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:41.800193+0000) 2019-10-13T14:55:43.337 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:55:43.337 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:43.341 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:55:43.341 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:55:44.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:44.394+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:44.397420+0000) 2019-10-13T14:55:44.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:44.674+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:44.681676+0000) 2019-10-13T14:55:45.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:45.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.564459+0000) 2019-10-13T14:55:45.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:45.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.566137+0000) 2019-10-13T14:55:45.830 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:45.822+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.826950+0000) 2019-10-13T14:55:45.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:45.822+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.827435+0000) 2019-10-13T14:55:45.830 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:45.822+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.828110+0000) 2019-10-13T14:55:45.831 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:45.825+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.828610+0000) 2019-10-13T14:55:45.831 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:45.822+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.829206+0000) 2019-10-13T14:55:45.832 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:45.822+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.829719+0000) 2019-10-13T14:55:45.832 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:45.826+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.830162+0000) 2019-10-13T14:55:45.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:45.829+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:45.833016+0000) 2019-10-13T14:55:45.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:45.830+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215335 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:48.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:48.050+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.051121+0000) 2019-10-13T14:55:48.279 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:48.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:48.277097+0000) 2019-10-13T14:55:50.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:50.834+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.839394+0000) 2019-10-13T14:55:50.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:50.833+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.840060+0000) 2019-10-13T14:55:50.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:50.834+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.840936+0000) 2019-10-13T14:55:50.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:50.834+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.841279+0000) 2019-10-13T14:55:50.844 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:50.838+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.841944+0000) 2019-10-13T14:55:50.845 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:50.838+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.842611+0000) 2019-10-13T14:55:50.846 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:50.838+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.842817+0000) 2019-10-13T14:55:50.849 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:50.841+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:50.846710+0000) 2019-10-13T14:55:50.854 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:50.846+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215363 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:51.306 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:51.297+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:51.303990+0000) 2019-10-13T14:55:51.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:51.798+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:51.800374+0000) 2019-10-13T14:55:54.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:54.394+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:54.397624+0000) 2019-10-13T14:55:54.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:54.678+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:54.681910+0000) 2019-10-13T14:55:55.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:55.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.564660+0000) 2019-10-13T14:55:55.568 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:55.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.566330+0000) 2019-10-13T14:55:55.858 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:55:55.850+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.855282+0000) 2019-10-13T14:55:55.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:55:55.850+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.856563+0000) 2019-10-13T14:55:55.859 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:55.850+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.857035+0000) 2019-10-13T14:55:55.860 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:55:55.850+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.857410+0000) 2019-10-13T14:55:55.862 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:55:55.857+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.860208+0000) 2019-10-13T14:55:55.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:55.858+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.862119+0000) 2019-10-13T14:55:55.866 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:55:55.858+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.864256+0000) 2019-10-13T14:55:55.868 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:55:55.862+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215394 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:55:55.875 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:55:55.869+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:55.872762+0000) 2019-10-13T14:55:58.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:55:58.046+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.051329+0000) 2019-10-13T14:55:58.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:55:58.270+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:55:58.277284+0000) 2019-10-13T14:56:00.873 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:00.866+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.870257+0000) 2019-10-13T14:56:00.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:00.866+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.870474+0000) 2019-10-13T14:56:00.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:00.866+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.872238+0000) 2019-10-13T14:56:00.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:00.866+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.872813+0000) 2019-10-13T14:56:00.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:00.869+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.873280+0000) 2019-10-13T14:56:00.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:00.870+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.873782+0000) 2019-10-13T14:56:00.878 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:00.870+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.875218+0000) 2019-10-13T14:56:00.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:00.874+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:00.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:00.877+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:00.880907+0000) 2019-10-13T14:56:01.313 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:01.309+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:01.311046+0000) 2019-10-13T14:56:01.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:01.798+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:01.800569+0000) 2019-10-13T14:56:04.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:04.394+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:04.397820+0000) 2019-10-13T14:56:04.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:04.682+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:04.682136+0000) 2019-10-13T14:56:05.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:05.557+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.564863+0000) 2019-10-13T14:56:05.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:05.561+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.566526+0000) 2019-10-13T14:56:05.887 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:05.877+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.884232+0000) 2019-10-13T14:56:05.887 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:05.877+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.884587+0000) 2019-10-13T14:56:05.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:05.881+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.885840+0000) 2019-10-13T14:56:05.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:05.881+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.886576+0000) 2019-10-13T14:56:05.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:05.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.886955+0000) 2019-10-13T14:56:05.890 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:05.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.887733+0000) 2019-10-13T14:56:05.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:05.881+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.888130+0000) 2019-10-13T14:56:05.894 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:05.889+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:05.892296+0000) 2019-10-13T14:56:05.897 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:05.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215455 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:08.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:08.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.051531+0000) 2019-10-13T14:56:08.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:08.269+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:08.277475+0000) 2019-10-13T14:56:10.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:10.893+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.898351+0000) 2019-10-13T14:56:10.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:10.893+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.899382+0000) 2019-10-13T14:56:10.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:10.893+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.900934+0000) 2019-10-13T14:56:10.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:10.893+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.901243+0000) 2019-10-13T14:56:10.905 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:10.897+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.902495+0000) 2019-10-13T14:56:10.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:10.897+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.902692+0000) 2019-10-13T14:56:10.906 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:10.897+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.903529+0000) 2019-10-13T14:56:10.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:10.897+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:10.903956+0000) 2019-10-13T14:56:10.910 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:10.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215483 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:11.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:11.313+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:11.318203+0000) 2019-10-13T14:56:11.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:11.797+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:11.800745+0000) 2019-10-13T14:56:13.382 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:56:13.382 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:13.386 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:56:13.386 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:14.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:14.393+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:14.398010+0000) 2019-10-13T14:56:14.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:14.677+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:14.682330+0000) 2019-10-13T14:56:15.567 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:15.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.565058+0000) 2019-10-13T14:56:15.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:15.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.566733+0000) 2019-10-13T14:56:15.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:15.905+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.913152+0000) 2019-10-13T14:56:15.916 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:15.905+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.913499+0000) 2019-10-13T14:56:15.917 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:15.909+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.915322+0000) 2019-10-13T14:56:15.918 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:15.909+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.915842+0000) 2019-10-13T14:56:15.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:15.913+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.917355+0000) 2019-10-13T14:56:15.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:15.909+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.917444+0000) 2019-10-13T14:56:15.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:15.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.917721+0000) 2019-10-13T14:56:15.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:15.913+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:15.918476+0000) 2019-10-13T14:56:15.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:15.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215516 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:18.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:18.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.051742+0000) 2019-10-13T14:56:18.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:18.273+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:18.277667+0000) 2019-10-13T14:56:20.929 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:20.925+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.925983+0000) 2019-10-13T14:56:20.929 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:20.925+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.927028+0000) 2019-10-13T14:56:20.930 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:20.925+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.927327+0000) 2019-10-13T14:56:20.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:20.925+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.927867+0000) 2019-10-13T14:56:20.930 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:20.925+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.928125+0000) 2019-10-13T14:56:20.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:20.925+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.928875+0000) 2019-10-13T14:56:20.932 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:20.925+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.929886+0000) 2019-10-13T14:56:20.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:20.929+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:20.933147+0000) 2019-10-13T14:56:20.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:20.933+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215544 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:21.328 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:21.325+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:21.325265+0000) 2019-10-13T14:56:21.803 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:21.797+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:21.800921+0000) 2019-10-13T14:56:24.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:24.397+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:24.398245+0000) 2019-10-13T14:56:24.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:24.681+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:24.682548+0000) 2019-10-13T14:56:25.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:25.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.565284+0000) 2019-10-13T14:56:25.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:25.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.566933+0000) 2019-10-13T14:56:25.942 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:25.937+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.939561+0000) 2019-10-13T14:56:25.943 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:25.937+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.940468+0000) 2019-10-13T14:56:25.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:25.937+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.940973+0000) 2019-10-13T14:56:25.944 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:25.937+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.941305+0000) 2019-10-13T14:56:25.944 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:25.937+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.941499+0000) 2019-10-13T14:56:25.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:25.941+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.942626+0000) 2019-10-13T14:56:25.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:25.941+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.943110+0000) 2019-10-13T14:56:25.950 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:25.945+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:25.948138+0000) 2019-10-13T14:56:25.955 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:25.949+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215579 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:28.054 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:28.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.051941+0000) 2019-10-13T14:56:28.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:28.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:28.277894+0000) 2019-10-13T14:56:30.957 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:30.953+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.954780+0000) 2019-10-13T14:56:30.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:30.953+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.956458+0000) 2019-10-13T14:56:30.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:30.953+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.957380+0000) 2019-10-13T14:56:30.962 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:30.957+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.959885+0000) 2019-10-13T14:56:30.963 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:30.956+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.960180+0000) 2019-10-13T14:56:30.963 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:30.957+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.960387+0000) 2019-10-13T14:56:30.963 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:30.957+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.960904+0000) 2019-10-13T14:56:30.965 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:30.956+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:30.963129+0000) 2019-10-13T14:56:30.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:30.965+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215607 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:31.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:31.328+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:31.332440+0000) 2019-10-13T14:56:31.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:31.797+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:31.801124+0000) 2019-10-13T14:56:34.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:34.397+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:34.398479+0000) 2019-10-13T14:56:34.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:34.681+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:34.682765+0000) 2019-10-13T14:56:35.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:35.561+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.565517+0000) 2019-10-13T14:56:35.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:35.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.567134+0000) 2019-10-13T14:56:35.972 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:35.965+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.969168+0000) 2019-10-13T14:56:35.972 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:35.969+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.969915+0000) 2019-10-13T14:56:35.973 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:35.964+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.971374+0000) 2019-10-13T14:56:35.975 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:35.969+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.972409+0000) 2019-10-13T14:56:35.975 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:35.969+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.972989+0000) 2019-10-13T14:56:35.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:35.969+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.973681+0000) 2019-10-13T14:56:35.976 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:35.973+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.973736+0000) 2019-10-13T14:56:35.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:35.972+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:35.978119+0000) 2019-10-13T14:56:35.981 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:35.977+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215640 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:38.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:38.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.052120+0000) 2019-10-13T14:56:38.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:38.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:38.278083+0000) 2019-10-13T14:56:40.985 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:40.981+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.982794+0000) 2019-10-13T14:56:40.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:40.981+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.983344+0000) 2019-10-13T14:56:40.987 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:40.981+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.984101+0000) 2019-10-13T14:56:40.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:40.980+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.984651+0000) 2019-10-13T14:56:40.988 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:40.981+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.984916+0000) 2019-10-13T14:56:40.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:40.981+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.985185+0000) 2019-10-13T14:56:40.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:40.985+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.986057+0000) 2019-10-13T14:56:40.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:40.988+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:40.992181+0000) 2019-10-13T14:56:40.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:40.989+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215667 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:41.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:41.340+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:41.343913+0000) 2019-10-13T14:56:41.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:41.797+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:41.801319+0000) 2019-10-13T14:56:43.505 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:56:43.505 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:43.509 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:56:43.509 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:56:44.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:44.397+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:44.398686+0000) 2019-10-13T14:56:44.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:44.681+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:44.682970+0000) 2019-10-13T14:56:45.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:45.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.565711+0000) 2019-10-13T14:56:45.569 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:45.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.567333+0000) 2019-10-13T14:56:45.999 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:45.993+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.996884+0000) 2019-10-13T14:56:46.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:45.993+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.997158+0000) 2019-10-13T14:56:46.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:45.997+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.999732+0000) 2019-10-13T14:56:46.002 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:45.997+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:45.999901+0000) 2019-10-13T14:56:46.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:45.996+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:46.000516+0000) 2019-10-13T14:56:46.004 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:46.001+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:46.002369+0000) 2019-10-13T14:56:46.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:46.001+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:46.004427+0000) 2019-10-13T14:56:46.011 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:46.004+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:46.008613+0000) 2019-10-13T14:56:46.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:46.005+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:48.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:48.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.052391+0000) 2019-10-13T14:56:48.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:48.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:48.278278+0000) 2019-10-13T14:56:51.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:51.009+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.011773+0000) 2019-10-13T14:56:51.016 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:51.009+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.013468+0000) 2019-10-13T14:56:51.016 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:51.013+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.013856+0000) 2019-10-13T14:56:51.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:51.013+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.016311+0000) 2019-10-13T14:56:51.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:51.013+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.016940+0000) 2019-10-13T14:56:51.020 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:51.012+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.017589+0000) 2019-10-13T14:56:51.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:51.017+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.018149+0000) 2019-10-13T14:56:51.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:51.021+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:51.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:51.020+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.024929+0000) 2019-10-13T14:56:51.353 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:51.348+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.350590+0000) 2019-10-13T14:56:51.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:51.797+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:51.801494+0000) 2019-10-13T14:56:54.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:54.397+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:54.398886+0000) 2019-10-13T14:56:54.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:54.681+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:54.683169+0000) 2019-10-13T14:56:55.568 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:55.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:55.565965+0000) 2019-10-13T14:56:55.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:55.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:55.567527+0000) 2019-10-13T14:56:56.027 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:56.021+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.024879+0000) 2019-10-13T14:56:56.028 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:56:56.025+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.026123+0000) 2019-10-13T14:56:56.030 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:56:56.025+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.027419+0000) 2019-10-13T14:56:56.030 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:56:56.025+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.027936+0000) 2019-10-13T14:56:56.033 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:56:56.029+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.030439+0000) 2019-10-13T14:56:56.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:56:56.024+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.031142+0000) 2019-10-13T14:56:56.034 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:56.029+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.031422+0000) 2019-10-13T14:56:56.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:56:56.032+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:56.036564+0000) 2019-10-13T14:56:56.039 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:56:56.033+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:56:58.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:56:58.049+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.052596+0000) 2019-10-13T14:56:58.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:56:58.277+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:56:58.278473+0000) 2019-10-13T14:57:01.043 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:01.037+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.040350+0000) 2019-10-13T14:57:01.043 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:01.037+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.041111+0000) 2019-10-13T14:57:01.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:01.036+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.043358+0000) 2019-10-13T14:57:01.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:01.041+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.043730+0000) 2019-10-13T14:57:01.047 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:01.041+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.044501+0000) 2019-10-13T14:57:01.047 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:01.041+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.045037+0000) 2019-10-13T14:57:01.048 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:01.041+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.045531+0000) 2019-10-13T14:57:01.053 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:01.044+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.051026+0000) 2019-10-13T14:57:01.054 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:01.049+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215791 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:01.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:01.356+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.357240+0000) 2019-10-13T14:57:01.804 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:01.796+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:01.801678+0000) 2019-10-13T14:57:04.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:04.396+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:04.399090+0000) 2019-10-13T14:57:04.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:04.680+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:04.683429+0000) 2019-10-13T14:57:05.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:05.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:05.566152+0000) 2019-10-13T14:57:05.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:05.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:05.567738+0000) 2019-10-13T14:57:06.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:06.052+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.054930+0000) 2019-10-13T14:57:06.059 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:06.052+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.056746+0000) 2019-10-13T14:57:06.060 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:06.052+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.057188+0000) 2019-10-13T14:57:06.060 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:06.052+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.057598+0000) 2019-10-13T14:57:06.061 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:06.056+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.058426+0000) 2019-10-13T14:57:06.061 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:06.056+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.058849+0000) 2019-10-13T14:57:06.062 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:06.056+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.059279+0000) 2019-10-13T14:57:06.067 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:06.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215825 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:06.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:06.064+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:06.068762+0000) 2019-10-13T14:57:08.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:08.048+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.052849+0000) 2019-10-13T14:57:08.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:08.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:08.278685+0000) 2019-10-13T14:57:11.071 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:11.064+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.068731+0000) 2019-10-13T14:57:11.072 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:11.068+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.070108+0000) 2019-10-13T14:57:11.073 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:11.064+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.070704+0000) 2019-10-13T14:57:11.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:11.068+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.070754+0000) 2019-10-13T14:57:11.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:11.068+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.071599+0000) 2019-10-13T14:57:11.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:11.068+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.072232+0000) 2019-10-13T14:57:11.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:11.068+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.072813+0000) 2019-10-13T14:57:11.078 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:11.068+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.075734+0000) 2019-10-13T14:57:11.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:11.076+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215854 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:11.366 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:11.360+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.363093+0000) 2019-10-13T14:57:11.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:11.800+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:11.801873+0000) 2019-10-13T14:57:13.619 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:57:13.619 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:13.623 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:57:13.623 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:14.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:14.396+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:14.399322+0000) 2019-10-13T14:57:14.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:14.680+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:14.683626+0000) 2019-10-13T14:57:15.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:15.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:15.566365+0000) 2019-10-13T14:57:15.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:15.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:15.567939+0000) 2019-10-13T14:57:16.085 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:16.080+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.082488+0000) 2019-10-13T14:57:16.086 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:16.080+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.084264+0000) 2019-10-13T14:57:16.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:16.080+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.085054+0000) 2019-10-13T14:57:16.088 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:16.080+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.085487+0000) 2019-10-13T14:57:16.088 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:16.080+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.085968+0000) 2019-10-13T14:57:16.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:16.084+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.086297+0000) 2019-10-13T14:57:16.089 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:16.084+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.087033+0000) 2019-10-13T14:57:16.095 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:16.088+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:16.092905+0000) 2019-10-13T14:57:16.095 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:16.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215886 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:18.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:18.052+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.053054+0000) 2019-10-13T14:57:18.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:18.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:18.278889+0000) 2019-10-13T14:57:21.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:21.092+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.096724+0000) 2019-10-13T14:57:21.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:21.092+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.097852+0000) 2019-10-13T14:57:21.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:21.096+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.099262+0000) 2019-10-13T14:57:21.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:21.096+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.099591+0000) 2019-10-13T14:57:21.102 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:21.096+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.099930+0000) 2019-10-13T14:57:21.103 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:21.096+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.100290+0000) 2019-10-13T14:57:21.103 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:21.096+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.100579+0000) 2019-10-13T14:57:21.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:21.100+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.104515+0000) 2019-10-13T14:57:21.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:21.104+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:21.377 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:21.372+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.374733+0000) 2019-10-13T14:57:21.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:21.800+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:21.802094+0000) 2019-10-13T14:57:24.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:24.396+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:24.399521+0000) 2019-10-13T14:57:24.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:24.680+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:24.683829+0000) 2019-10-13T14:57:25.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:25.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:25.566583+0000) 2019-10-13T14:57:25.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:25.564+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:25.568141+0000) 2019-10-13T14:57:26.113 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:26.108+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.111134+0000) 2019-10-13T14:57:26.116 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:26.108+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.113318+0000) 2019-10-13T14:57:26.116 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:26.112+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.113831+0000) 2019-10-13T14:57:26.116 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:26.112+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.114160+0000) 2019-10-13T14:57:26.119 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:26.112+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.117067+0000) 2019-10-13T14:57:26.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:26.112+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.118273+0000) 2019-10-13T14:57:26.121 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:26.116+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.118432+0000) 2019-10-13T14:57:26.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:26.116+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:26.120312+0000) 2019-10-13T14:57:26.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:26.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215949 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:28.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:28.052+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.053283+0000) 2019-10-13T14:57:28.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:28.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:28.279089+0000) 2019-10-13T14:57:31.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:31.120+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.125210+0000) 2019-10-13T14:57:31.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:31.124+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.125785+0000) 2019-10-13T14:57:31.129 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:31.124+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.126516+0000) 2019-10-13T14:57:31.130 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:31.124+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.126748+0000) 2019-10-13T14:57:31.130 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:31.124+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.127262+0000) 2019-10-13T14:57:31.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:31.124+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.127296+0000) 2019-10-13T14:57:31.130 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:31.124+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.128048+0000) 2019-10-13T14:57:31.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:31.132+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215977 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:31.139 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:31.136+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.137371+0000) 2019-10-13T14:57:31.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:31.380+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.381091+0000) 2019-10-13T14:57:31.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:31.800+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:31.802300+0000) 2019-10-13T14:57:34.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:34.396+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.399721+0000) 2019-10-13T14:57:34.686 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:34.680+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:34.684027+0000) 2019-10-13T14:57:35.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:35.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:35.566808+0000) 2019-10-13T14:57:35.570 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:35.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:35.568338+0000) 2019-10-13T14:57:36.140 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:36.136+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.138168+0000) 2019-10-13T14:57:36.142 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:36.136+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.139824+0000) 2019-10-13T14:57:36.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:36.136+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.140222+0000) 2019-10-13T14:57:36.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:36.136+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.140584+0000) 2019-10-13T14:57:36.144 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:36.140+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.142044+0000) 2019-10-13T14:57:36.145 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:36.139+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.142670+0000) 2019-10-13T14:57:36.145 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:36.140+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.142972+0000) 2019-10-13T14:57:36.149 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:36.143+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:36.147245+0000) 2019-10-13T14:57:36.151 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:36.144+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:38.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:38.052+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:38.053495+0000) 2019-10-13T14:57:38.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:38.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:38.279287+0000) 2019-10-13T14:57:41.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:41.148+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.152158+0000) 2019-10-13T14:57:41.155 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:41.148+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.152494+0000) 2019-10-13T14:57:41.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:41.148+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.153483+0000) 2019-10-13T14:57:41.156 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:41.151+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.154266+0000) 2019-10-13T14:57:41.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:41.152+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.154660+0000) 2019-10-13T14:57:41.158 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:41.152+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.155890+0000) 2019-10-13T14:57:41.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:41.152+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.156415+0000) 2019-10-13T14:57:41.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:41.155+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.159968+0000) 2019-10-13T14:57:41.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:41.160+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216038 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:41.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:41.387+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.388169+0000) 2019-10-13T14:57:41.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:41.800+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:41.802484+0000) 2019-10-13T14:57:43.732 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:57:43.732 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:43.736 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:57:43.736 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:57:44.402 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:44.396+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.399932+0000) 2019-10-13T14:57:44.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:44.680+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:44.684221+0000) 2019-10-13T14:57:45.569 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:45.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:45.567031+0000) 2019-10-13T14:57:45.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:45.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:45.568536+0000) 2019-10-13T14:57:46.169 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:46.164+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.166234+0000) 2019-10-13T14:57:46.169 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:46.164+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.166560+0000) 2019-10-13T14:57:46.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:46.164+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.168205+0000) 2019-10-13T14:57:46.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:46.164+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.167854+0000) 2019-10-13T14:57:46.171 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:46.164+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.168416+0000) 2019-10-13T14:57:46.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:46.167+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.169078+0000) 2019-10-13T14:57:46.172 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:46.164+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.168766+0000) 2019-10-13T14:57:46.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:46.171+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:46.173641+0000) 2019-10-13T14:57:46.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:46.172+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:48.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:48.048+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:48.053687+0000) 2019-10-13T14:57:48.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:48.276+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:48.279480+0000) 2019-10-13T14:57:51.183 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:51.176+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.180287+0000) 2019-10-13T14:57:51.183 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:51.176+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.180831+0000) 2019-10-13T14:57:51.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:51.176+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.181264+0000) 2019-10-13T14:57:51.185 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:51.180+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.182283+0000) 2019-10-13T14:57:51.185 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:51.180+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.182961+0000) 2019-10-13T14:57:51.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:51.180+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.182891+0000) 2019-10-13T14:57:51.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:51.179+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.183477+0000) 2019-10-13T14:57:51.192 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:51.187+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.189415+0000) 2019-10-13T14:57:51.193 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:51.188+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:51.397 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:51.391+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.394733+0000) 2019-10-13T14:57:51.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:51.800+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:51.802689+0000) 2019-10-13T14:57:54.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:54.400+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.400188+0000) 2019-10-13T14:57:54.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:54.684+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:54.684426+0000) 2019-10-13T14:57:55.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:55.564+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:55.567228+0000) 2019-10-13T14:57:55.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:55.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:55.568756+0000) 2019-10-13T14:57:56.197 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:56.192+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.194280+0000) 2019-10-13T14:57:56.197 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:57:56.192+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.194890+0000) 2019-10-13T14:57:56.198 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:57:56.192+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.195649+0000) 2019-10-13T14:57:56.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:57:56.191+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.195986+0000) 2019-10-13T14:57:56.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:56.192+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.196692+0000) 2019-10-13T14:57:56.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:57:56.192+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.197147+0000) 2019-10-13T14:57:56.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:57:56.196+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.197864+0000) 2019-10-13T14:57:56.206 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:57:56.200+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:57:56.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:57:56.203+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:56.204590+0000) 2019-10-13T14:57:58.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:57:58.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:58.053929+0000) 2019-10-13T14:57:58.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:57:58.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:57:58.279676+0000) 2019-10-13T14:58:01.210 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:01.203+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.207006+0000) 2019-10-13T14:58:01.210 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:01.203+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.207612+0000) 2019-10-13T14:58:01.211 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:01.203+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.208028+0000) 2019-10-13T14:58:01.213 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:01.207+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.210866+0000) 2019-10-13T14:58:01.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:01.211+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.212737+0000) 2019-10-13T14:58:01.216 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:01.207+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.213439+0000) 2019-10-13T14:58:01.217 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:01.211+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.214500+0000) 2019-10-13T14:58:01.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:01.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:01.224 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:01.219+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.221952+0000) 2019-10-13T14:58:01.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:01.399+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.400942+0000) 2019-10-13T14:58:01.805 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:01.799+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:01.802880+0000) 2019-10-13T14:58:04.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:04.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.400419+0000) 2019-10-13T14:58:04.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:04.679+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:04.684627+0000) 2019-10-13T14:58:05.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:05.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:05.567424+0000) 2019-10-13T14:58:05.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:05.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:05.568950+0000) 2019-10-13T14:58:06.224 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:06.215+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.221208+0000) 2019-10-13T14:58:06.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:06.219+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.222615+0000) 2019-10-13T14:58:06.225 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:06.219+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.223084+0000) 2019-10-13T14:58:06.226 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:06.219+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.224334+0000) 2019-10-13T14:58:06.228 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:06.223+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.225691+0000) 2019-10-13T14:58:06.228 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:06.223+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.225948+0000) 2019-10-13T14:58:06.229 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:06.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.227023+0000) 2019-10-13T14:58:06.233 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:06.227+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:06.231018+0000) 2019-10-13T14:58:06.234 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:06.227+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:08.056 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:08.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:08.054140+0000) 2019-10-13T14:58:08.282 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:08.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:08.279883+0000) 2019-10-13T14:58:11.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:11.231+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.235543+0000) 2019-10-13T14:58:11.239 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:11.231+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.235649+0000) 2019-10-13T14:58:11.240 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:11.235+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.238102+0000) 2019-10-13T14:58:11.242 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:11.235+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.239431+0000) 2019-10-13T14:58:11.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:11.235+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.240884+0000) 2019-10-13T14:58:11.244 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:11.239+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.242170+0000) 2019-10-13T14:58:11.245 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:11.239+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.243313+0000) 2019-10-13T14:58:11.248 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:11.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:11.253 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:11.247+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.250583+0000) 2019-10-13T14:58:11.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:11.407+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.407537+0000) 2019-10-13T14:58:11.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:11.799+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:11.803060+0000) 2019-10-13T14:58:13.804 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:58:13.804 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:13.808 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:58:13.808 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:14.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:14.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.400647+0000) 2019-10-13T14:58:14.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:14.683+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:14.684831+0000) 2019-10-13T14:58:15.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:15.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:15.567616+0000) 2019-10-13T14:58:15.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:15.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:15.569171+0000) 2019-10-13T14:58:16.252 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:16.247+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.250046+0000) 2019-10-13T14:58:16.253 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:16.247+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.251083+0000) 2019-10-13T14:58:16.254 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:16.247+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.251150+0000) 2019-10-13T14:58:16.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:16.247+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.251946+0000) 2019-10-13T14:58:16.254 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:16.247+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.251766+0000) 2019-10-13T14:58:16.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:16.247+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.252257+0000) 2019-10-13T14:58:16.255 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:16.247+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.252956+0000) 2019-10-13T14:58:16.261 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:16.255+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:16.259091+0000) 2019-10-13T14:58:16.262 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:16.255+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:18.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:18.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:18.054337+0000) 2019-10-13T14:58:18.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:18.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:18.280107+0000) 2019-10-13T14:58:21.266 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:21.259+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.263315+0000) 2019-10-13T14:58:21.267 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:21.259+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.264382+0000) 2019-10-13T14:58:21.267 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:21.259+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.264787+0000) 2019-10-13T14:58:21.269 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:21.263+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.266859+0000) 2019-10-13T14:58:21.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:21.263+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.267633+0000) 2019-10-13T14:58:21.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:21.267+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.268138+0000) 2019-10-13T14:58:21.271 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:21.263+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.268428+0000) 2019-10-13T14:58:21.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:21.271+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.273584+0000) 2019-10-13T14:58:21.277 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:21.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216283 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:21.416 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:21.411+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.413963+0000) 2019-10-13T14:58:21.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:21.799+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:21.803244+0000) 2019-10-13T14:58:24.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:24.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.400863+0000) 2019-10-13T14:58:24.687 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:24.683+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:24.685032+0000) 2019-10-13T14:58:25.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:25.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:25.567804+0000) 2019-10-13T14:58:25.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:25.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:25.569384+0000) 2019-10-13T14:58:26.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:26.275+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.277840+0000) 2019-10-13T14:58:26.281 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:26.275+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.278040+0000) 2019-10-13T14:58:26.281 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:26.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.278767+0000) 2019-10-13T14:58:26.282 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:26.275+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.279915+0000) 2019-10-13T14:58:26.283 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:26.279+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.280531+0000) 2019-10-13T14:58:26.283 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:26.275+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.280800+0000) 2019-10-13T14:58:26.284 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:26.275+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.281113+0000) 2019-10-13T14:58:26.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:26.287+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:26.289117+0000) 2019-10-13T14:58:26.292 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:26.283+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216316 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:28.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:28.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:28.054566+0000) 2019-10-13T14:58:28.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:28.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:28.280318+0000) 2019-10-13T14:58:31.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:31.287+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.292312+0000) 2019-10-13T14:58:31.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:31.287+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.292815+0000) 2019-10-13T14:58:31.296 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:31.291+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.293741+0000) 2019-10-13T14:58:31.297 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:31.291+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.294456+0000) 2019-10-13T14:58:31.297 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:31.291+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.294046+0000) 2019-10-13T14:58:31.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:31.291+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.295671+0000) 2019-10-13T14:58:31.298 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:31.291+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.296147+0000) 2019-10-13T14:58:31.303 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:31.299+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.301229+0000) 2019-10-13T14:58:31.307 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:31.299+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:31.423 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:31.419+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.420462+0000) 2019-10-13T14:58:31.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:31.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:31.803426+0000) 2019-10-13T14:58:34.403 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:34.395+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.401094+0000) 2019-10-13T14:58:34.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:34.683+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:34.685278+0000) 2019-10-13T14:58:35.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:35.563+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:35.567994+0000) 2019-10-13T14:58:35.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:35.567+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:35.569564+0000) 2019-10-13T14:58:36.310 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:36.303+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.307648+0000) 2019-10-13T14:58:36.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:36.303+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.308143+0000) 2019-10-13T14:58:36.312 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:36.307+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.309856+0000) 2019-10-13T14:58:36.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:36.307+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.310373+0000) 2019-10-13T14:58:36.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:36.307+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.310740+0000) 2019-10-13T14:58:36.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:36.307+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.311309+0000) 2019-10-13T14:58:36.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:36.307+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.312308+0000) 2019-10-13T14:58:36.320 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:36.311+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216376 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:36.320 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:36.315+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:36.318293+0000) 2019-10-13T14:58:38.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:38.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:38.054765+0000) 2019-10-13T14:58:38.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:38.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:38.280502+0000) 2019-10-13T14:58:41.324 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:41.319+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.321814+0000) 2019-10-13T14:58:41.325 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:41.318+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.321892+0000) 2019-10-13T14:58:41.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:41.319+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.323267+0000) 2019-10-13T14:58:41.326 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:41.319+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.323887+0000) 2019-10-13T14:58:41.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:41.319+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.324132+0000) 2019-10-13T14:58:41.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:41.319+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.324629+0000) 2019-10-13T14:58:41.329 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:41.323+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.327075+0000) 2019-10-13T14:58:41.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:41.330+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.332749+0000) 2019-10-13T14:58:41.335 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:41.327+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:41.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:41.422+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.427090+0000) 2019-10-13T14:58:41.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:41.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:41.803610+0000) 2019-10-13T14:58:43.912 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:58:43.912 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:43.916 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:58:43.916 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:58:44.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:44.395+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.401293+0000) 2019-10-13T14:58:44.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:44.683+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:44.685483+0000) 2019-10-13T14:58:45.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:45.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:45.568218+0000) 2019-10-13T14:58:45.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:45.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:45.569782+0000) 2019-10-13T14:58:46.340 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:46.331+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.337157+0000) 2019-10-13T14:58:46.340 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:46.331+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.337409+0000) 2019-10-13T14:58:46.341 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:46.334+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.338451+0000) 2019-10-13T14:58:46.342 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:46.335+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.339869+0000) 2019-10-13T14:58:46.342 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:46.335+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.339820+0000) 2019-10-13T14:58:46.343 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:46.335+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.340389+0000) 2019-10-13T14:58:46.344 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:46.339+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.342076+0000) 2019-10-13T14:58:46.348 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:46.342+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:46.345990+0000) 2019-10-13T14:58:46.349 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:46.343+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216438 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:48.057 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:48.051+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:48.054978+0000) 2019-10-13T14:58:48.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:48.275+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:48.280679+0000) 2019-10-13T14:58:51.353 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:51.347+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.350031+0000) 2019-10-13T14:58:51.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:51.347+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.350452+0000) 2019-10-13T14:58:51.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:51.346+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.350973+0000) 2019-10-13T14:58:51.354 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:51.347+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.351850+0000) 2019-10-13T14:58:51.354 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:51.347+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.351396+0000) 2019-10-13T14:58:51.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:51.347+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.352500+0000) 2019-10-13T14:58:51.355 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:51.347+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.352763+0000) 2019-10-13T14:58:51.362 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:51.354+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.360020+0000) 2019-10-13T14:58:51.364 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:51.359+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:51.441 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:51.434+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.438623+0000) 2019-10-13T14:58:51.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:51.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:51.803796+0000) 2019-10-13T14:58:54.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:54.398+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.401499+0000) 2019-10-13T14:58:54.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:54.682+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:54.685691+0000) 2019-10-13T14:58:55.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:55.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:55.568419+0000) 2019-10-13T14:58:55.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:55.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:55.570007+0000) 2019-10-13T14:58:56.368 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:58:56.358+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.365233+0000) 2019-10-13T14:58:56.368 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:58:56.358+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.364875+0000) 2019-10-13T14:58:56.369 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:56.362+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.365720+0000) 2019-10-13T14:58:56.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:56.362+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.366038+0000) 2019-10-13T14:58:56.370 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:58:56.362+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.366608+0000) 2019-10-13T14:58:56.370 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:58:56.362+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.366748+0000) 2019-10-13T14:58:56.372 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:58:56.362+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.369520+0000) 2019-10-13T14:58:56.379 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:58:56.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216498 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:58:56.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:58:56.374+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:56.377109+0000) 2019-10-13T14:58:58.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:58:58.050+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:58.055163+0000) 2019-10-13T14:58:58.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:58:58.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:58:58.280872+0000) 2019-10-13T14:59:01.382 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:01.374+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.379354+0000) 2019-10-13T14:59:01.383 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:01.378+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.380665+0000) 2019-10-13T14:59:01.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:01.374+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.380947+0000) 2019-10-13T14:59:01.384 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:01.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.381412+0000) 2019-10-13T14:59:01.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:01.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.381853+0000) 2019-10-13T14:59:01.385 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:01.378+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.382475+0000) 2019-10-13T14:59:01.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:01.378+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.382811+0000) 2019-10-13T14:59:01.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:01.382+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.387884+0000) 2019-10-13T14:59:01.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:01.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216530 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:01.447 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:01.442+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.445166+0000) 2019-10-13T14:59:01.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:01.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:01.803975+0000) 2019-10-13T14:59:04.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:04.398+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.401735+0000) 2019-10-13T14:59:04.688 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:04.682+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:04.685958+0000) 2019-10-13T14:59:05.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:05.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:05.568671+0000) 2019-10-13T14:59:05.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:05.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:05.570234+0000) 2019-10-13T14:59:06.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:06.390+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.395077+0000) 2019-10-13T14:59:06.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:06.390+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.395650+0000) 2019-10-13T14:59:06.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:06.394+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.396493+0000) 2019-10-13T14:59:06.399 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:06.390+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.396464+0000) 2019-10-13T14:59:06.400 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:06.390+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.397065+0000) 2019-10-13T14:59:06.400 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:06.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.397853+0000) 2019-10-13T14:59:06.400 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:06.394+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.398077+0000) 2019-10-13T14:59:06.405 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:06.398+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:06.402388+0000) 2019-10-13T14:59:06.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:06.398+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216560 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:08.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:08.054+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:08.055360+0000) 2019-10-13T14:59:08.283 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:08.278+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:08.281065+0000) 2019-10-13T14:59:11.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:11.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.408360+0000) 2019-10-13T14:59:11.411 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:11.402+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.408914+0000) 2019-10-13T14:59:11.412 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:11.402+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.409368+0000) 2019-10-13T14:59:11.412 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:11.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.409792+0000) 2019-10-13T14:59:11.413 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:11.406+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.410044+0000) 2019-10-13T14:59:11.413 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:11.406+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.410344+0000) 2019-10-13T14:59:11.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:11.406+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.410613+0000) 2019-10-13T14:59:11.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:11.410+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.414966+0000) 2019-10-13T14:59:11.420 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:11.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216589 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:11.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:11.450+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.452072+0000) 2019-10-13T14:59:11.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:11.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:11.804153+0000) 2019-10-13T14:59:14.043 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:59:14.043 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:14.047 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:59:14.047 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:14.404 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:14.398+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.402002+0000) 2019-10-13T14:59:14.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:14.682+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:14.686189+0000) 2019-10-13T14:59:15.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:15.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:15.568859+0000) 2019-10-13T14:59:15.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:15.566+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:15.570409+0000) 2019-10-13T14:59:16.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:16.414+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.421475+0000) 2019-10-13T14:59:16.425 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:16.418+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.422345+0000) 2019-10-13T14:59:16.426 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:16.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.423370+0000) 2019-10-13T14:59:16.426 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:16.418+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.423592+0000) 2019-10-13T14:59:16.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:16.418+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.423715+0000) 2019-10-13T14:59:16.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:16.418+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.424385+0000) 2019-10-13T14:59:16.427 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:16.418+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.424042+0000) 2019-10-13T14:59:16.433 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:16.426+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:16.431139+0000) 2019-10-13T14:59:16.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:16.426+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216620 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:18.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:18.054+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:18.055559+0000) 2019-10-13T14:59:18.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:18.278+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:18.281233+0000) 2019-10-13T14:59:21.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:21.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.436073+0000) 2019-10-13T14:59:21.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:21.430+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.436656+0000) 2019-10-13T14:59:21.441 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:21.434+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.438562+0000) 2019-10-13T14:59:21.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:21.434+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.439010+0000) 2019-10-13T14:59:21.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:21.434+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.439723+0000) 2019-10-13T14:59:21.445 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:21.438+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.442992+0000) 2019-10-13T14:59:21.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:21.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.443328+0000) 2019-10-13T14:59:21.447 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:21.438+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.444623+0000) 2019-10-13T14:59:21.449 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:21.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216651 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:21.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:21.458+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.461903+0000) 2019-10-13T14:59:21.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:21.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:21.804348+0000) 2019-10-13T14:59:24.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:24.398+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.402210+0000) 2019-10-13T14:59:24.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:24.682+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:24.686394+0000) 2019-10-13T14:59:25.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:25.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:25.569055+0000) 2019-10-13T14:59:25.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:25.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:25.570606+0000) 2019-10-13T14:59:26.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:26.446+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.450481+0000) 2019-10-13T14:59:26.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:26.446+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.451266+0000) 2019-10-13T14:59:26.454 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:26.446+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.451751+0000) 2019-10-13T14:59:26.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:26.446+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.452282+0000) 2019-10-13T14:59:26.455 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:26.446+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.452664+0000) 2019-10-13T14:59:26.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:26.446+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.453243+0000) 2019-10-13T14:59:26.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:26.450+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.453813+0000) 2019-10-13T14:59:26.459 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:26.454+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:26.457223+0000) 2019-10-13T14:59:26.463 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:26.454+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216682 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:28.058 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:28.054+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:28.055764+0000) 2019-10-13T14:59:28.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:28.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:28.281445+0000) 2019-10-13T14:59:31.466 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:31.458+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.463798+0000) 2019-10-13T14:59:31.467 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:31.458+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.464415+0000) 2019-10-13T14:59:31.467 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:31.458+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.465304+0000) 2019-10-13T14:59:31.468 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:31.462+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.466219+0000) 2019-10-13T14:59:31.469 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:31.462+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.466578+0000) 2019-10-13T14:59:31.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:31.462+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.466676+0000) 2019-10-13T14:59:31.470 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:31.462+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.467402+0000) 2019-10-13T14:59:31.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:31.466+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.471383+0000) 2019-10-13T14:59:31.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:31.466+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.471480+0000) 2019-10-13T14:59:31.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:31.470+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216711 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:31.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:31.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:31.804552+0000) 2019-10-13T14:59:34.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:34.402+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.402408+0000) 2019-10-13T14:59:34.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:34.686+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:34.686624+0000) 2019-10-13T14:59:35.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:35.562+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:35.569255+0000) 2019-10-13T14:59:35.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:35.570+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:35.570797+0000) 2019-10-13T14:59:36.480 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:36.474+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.477802+0000) 2019-10-13T14:59:36.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:36.474+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.478851+0000) 2019-10-13T14:59:36.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:36.474+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.479784+0000) 2019-10-13T14:59:36.482 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:36.474+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.480195+0000) 2019-10-13T14:59:36.483 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:36.474+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.480807+0000) 2019-10-13T14:59:36.484 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:36.478+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.481508+0000) 2019-10-13T14:59:36.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:36.478+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.481900+0000) 2019-10-13T14:59:36.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:36.482+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:36.487953+0000) 2019-10-13T14:59:36.491 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:36.482+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216742 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:38.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:38.050+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:38.055959+0000) 2019-10-13T14:59:38.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:38.274+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:38.281639+0000) 2019-10-13T14:59:41.474 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:41.470+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.471670+0000) 2019-10-13T14:59:41.494 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:41.486+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.491508+0000) 2019-10-13T14:59:41.494 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:41.486+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.491924+0000) 2019-10-13T14:59:41.497 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:41.490+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.494986+0000) 2019-10-13T14:59:41.499 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:41.490+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.496552+0000) 2019-10-13T14:59:41.501 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:41.494+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.498498+0000) 2019-10-13T14:59:41.501 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:41.494+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.499167+0000) 2019-10-13T14:59:41.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:41.494+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.499657+0000) 2019-10-13T14:59:41.505 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:41.498+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216773 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:41.513 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:41.506+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.511386+0000) 2019-10-13T14:59:41.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:41.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:41.804757+0000) 2019-10-13T14:59:44.158 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T14:59:44.158 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:44.162 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T14:59:44.162 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T14:59:44.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:44.402+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.402611+0000) 2019-10-13T14:59:44.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:44.686+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:44.686849+0000) 2019-10-13T14:59:45.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:45.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:45.569447+0000) 2019-10-13T14:59:45.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:45.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:45.571000+0000) 2019-10-13T14:59:46.510 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:46.502+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.507616+0000) 2019-10-13T14:59:46.511 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:46.502+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.508076+0000) 2019-10-13T14:59:46.511 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:46.502+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.508354+0000) 2019-10-13T14:59:46.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:46.502+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.508639+0000) 2019-10-13T14:59:46.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:46.505+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.509349+0000) 2019-10-13T14:59:46.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:46.502+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.508866+0000) 2019-10-13T14:59:46.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:46.506+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.509800+0000) 2019-10-13T14:59:46.520 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:46.514+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216804 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:46.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:46.513+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:46.519490+0000) 2019-10-13T14:59:48.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:48.050+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:48.056163+0000) 2019-10-13T14:59:48.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:48.278+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:48.281845+0000) 2019-10-13T14:59:51.480 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:51.473+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.477409+0000) 2019-10-13T14:59:51.523 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:51.513+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.520814+0000) 2019-10-13T14:59:51.524 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:51.517+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.521730+0000) 2019-10-13T14:59:51.524 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:51.517+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.522040+0000) 2019-10-13T14:59:51.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:51.517+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.522437+0000) 2019-10-13T14:59:51.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:51.517+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.523108+0000) 2019-10-13T14:59:51.526 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:51.517+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.523789+0000) 2019-10-13T14:59:51.527 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:51.517+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.524507+0000) 2019-10-13T14:59:51.534 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:51.525+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.531622+0000) 2019-10-13T14:59:51.534 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:51.525+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216833 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:51.807 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:51.801+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:51.804987+0000) 2019-10-13T14:59:54.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:54.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.402807+0000) 2019-10-13T14:59:54.689 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:54.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:54.687055+0000) 2019-10-13T14:59:55.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:55.565+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:55.569638+0000) 2019-10-13T14:59:55.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:55.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:55.571193+0000) 2019-10-13T14:59:56.538 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:56.529+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.535531+0000) 2019-10-13T14:59:56.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T14:59:56.529+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.537208+0000) 2019-10-13T14:59:56.540 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T14:59:56.533+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.537932+0000) 2019-10-13T14:59:56.541 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T14:59:56.533+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.538318+0000) 2019-10-13T14:59:56.541 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T14:59:56.529+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.537532+0000) 2019-10-13T14:59:56.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:56.533+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.538871+0000) 2019-10-13T14:59:56.542 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T14:59:56.533+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.539312+0000) 2019-10-13T14:59:56.547 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T14:59:56.541+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:56.545091+0000) 2019-10-13T14:59:56.549 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T14:59:56.541+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T14:59:58.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T14:59:58.053+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:58.056391+0000) 2019-10-13T14:59:58.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T14:59:58.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T13:59:58.282053+0000) 2019-10-13T15:00:01.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:01.481+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.484050+0000) 2019-10-13T15:00:01.552 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:01.545+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.550166+0000) 2019-10-13T15:00:01.553 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:01.545+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.550530+0000) 2019-10-13T15:00:01.554 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:01.545+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.551868+0000) 2019-10-13T15:00:01.555 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:01.545+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.552361+0000) 2019-10-13T15:00:01.555 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:01.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.552521+0000) 2019-10-13T15:00:01.555 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:01.545+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.552920+0000) 2019-10-13T15:00:01.556 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:01.545+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.553249+0000) 2019-10-13T15:00:01.560 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:01.553+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.558269+0000) 2019-10-13T15:00:01.564 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:01.553+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216895 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:01.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:01.801+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:01.805167+0000) 2019-10-13T15:00:04.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:04.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.403007+0000) 2019-10-13T15:00:04.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:04.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:04.687245+0000) 2019-10-13T15:00:05.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:05.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:05.569837+0000) 2019-10-13T15:00:05.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:05.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:05.571382+0000) 2019-10-13T15:00:06.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:06.561+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.564921+0000) 2019-10-13T15:00:06.568 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:06.561+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.565154+0000) 2019-10-13T15:00:06.568 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:06.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.566021+0000) 2019-10-13T15:00:06.569 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:06.565+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.566526+0000) 2019-10-13T15:00:06.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:06.569+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.571086+0000) 2019-10-13T15:00:06.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:06.569+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.571369+0000) 2019-10-13T15:00:06.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:06.565+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.571869+0000) 2019-10-13T15:00:06.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:06.569+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:06.575876+0000) 2019-10-13T15:00:06.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:06.573+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:08.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:08.053+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:08.056588+0000) 2019-10-13T15:00:08.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:08.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:08.282252+0000) 2019-10-13T15:00:11.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:11.489+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.490543+0000) 2019-10-13T15:00:11.583 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:11.577+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.580292+0000) 2019-10-13T15:00:11.583 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:11.577+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.580682+0000) 2019-10-13T15:00:11.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:11.577+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.581195+0000) 2019-10-13T15:00:11.584 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:11.577+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.581334+0000) 2019-10-13T15:00:11.584 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:11.577+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.581701+0000) 2019-10-13T15:00:11.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:11.581+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.582499+0000) 2019-10-13T15:00:11.585 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:11.581+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.582868+0000) 2019-10-13T15:00:11.591 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:11.585+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.588712+0000) 2019-10-13T15:00:11.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:11.589+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:11.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:11.801+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:11.805343+0000) 2019-10-13T15:00:14.272 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:00:14.272 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:14.276 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:00:14.276 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:14.405 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:14.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.403167+0000) 2019-10-13T15:00:14.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:14.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:14.687472+0000) 2019-10-13T15:00:15.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:15.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:15.570061+0000) 2019-10-13T15:00:15.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:15.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:15.571601+0000) 2019-10-13T15:00:16.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:16.593+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.594608+0000) 2019-10-13T15:00:16.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:16.593+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.594958+0000) 2019-10-13T15:00:16.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:16.593+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.596971+0000) 2019-10-13T15:00:16.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:16.593+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.597340+0000) 2019-10-13T15:00:16.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:16.597+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.597894+0000) 2019-10-13T15:00:16.601 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:16.597+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.598086+0000) 2019-10-13T15:00:16.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:16.593+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.598823+0000) 2019-10-13T15:00:16.607 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:16.601+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:16.605144+0000) 2019-10-13T15:00:16.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:16.601+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216985 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:18.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:18.053+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:18.056805+0000) 2019-10-13T15:00:18.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:18.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:18.282450+0000) 2019-10-13T15:00:21.504 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:21.497+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.502126+0000) 2019-10-13T15:00:21.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:21.605+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.608520+0000) 2019-10-13T15:00:21.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:21.605+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.609560+0000) 2019-10-13T15:00:21.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:21.609+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.610446+0000) 2019-10-13T15:00:21.615 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:21.609+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.612773+0000) 2019-10-13T15:00:21.616 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:21.609+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.612995+0000) 2019-10-13T15:00:21.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:21.609+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.613646+0000) 2019-10-13T15:00:21.617 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:21.609+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.614652+0000) 2019-10-13T15:00:21.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:21.617+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.620797+0000) 2019-10-13T15:00:21.623 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:21.617+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217017 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:21.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:21.805+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:21.805527+0000) 2019-10-13T15:00:24.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:24.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.403421+0000) 2019-10-13T15:00:24.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:24.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:24.687702+0000) 2019-10-13T15:00:25.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:25.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:25.570257+0000) 2019-10-13T15:00:25.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:25.565+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:25.571809+0000) 2019-10-13T15:00:26.627 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:26.621+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.624672+0000) 2019-10-13T15:00:26.628 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:26.625+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.625917+0000) 2019-10-13T15:00:26.629 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:26.625+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.626139+0000) 2019-10-13T15:00:26.629 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:26.621+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.626601+0000) 2019-10-13T15:00:26.629 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:26.625+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.626947+0000) 2019-10-13T15:00:26.630 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:26.625+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.628119+0000) 2019-10-13T15:00:26.631 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:26.625+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.628969+0000) 2019-10-13T15:00:26.637 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:26.633+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217047 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:26.638 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:26.633+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:26.636261+0000) 2019-10-13T15:00:28.059 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:28.053+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:28.057005+0000) 2019-10-13T15:00:28.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:28.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:28.282664+0000) 2019-10-13T15:00:31.511 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:31.505+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.508712+0000) 2019-10-13T15:00:31.640 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:31.633+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.637856+0000) 2019-10-13T15:00:31.641 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:31.637+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.638841+0000) 2019-10-13T15:00:31.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:31.637+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.639431+0000) 2019-10-13T15:00:31.642 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:31.637+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.640139+0000) 2019-10-13T15:00:31.643 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:31.637+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.640150+0000) 2019-10-13T15:00:31.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:31.637+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.640686+0000) 2019-10-13T15:00:31.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:31.641+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.641787+0000) 2019-10-13T15:00:31.650 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:31.641+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.647502+0000) 2019-10-13T15:00:31.651 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:31.645+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217077 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:31.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:31.805+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:31.805767+0000) 2019-10-13T15:00:34.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:34.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.403620+0000) 2019-10-13T15:00:34.690 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:34.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:34.687908+0000) 2019-10-13T15:00:35.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:35.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:35.570455+0000) 2019-10-13T15:00:35.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:35.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:35.572004+0000) 2019-10-13T15:00:36.654 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:36.649+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.651865+0000) 2019-10-13T15:00:36.656 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:36.649+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.653257+0000) 2019-10-13T15:00:36.656 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:36.649+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.653457+0000) 2019-10-13T15:00:36.656 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:36.653+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.653924+0000) 2019-10-13T15:00:36.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:36.653+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.654448+0000) 2019-10-13T15:00:36.657 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:36.653+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.655152+0000) 2019-10-13T15:00:36.658 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:36.653+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.655405+0000) 2019-10-13T15:00:36.663 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:36.657+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:36.660841+0000) 2019-10-13T15:00:36.665 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:36.661+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217107 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:38.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:38.057+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:38.057212+0000) 2019-10-13T15:00:38.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:38.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:38.282883+0000) 2019-10-13T15:00:41.518 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:41.509+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.515270+0000) 2019-10-13T15:00:41.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:41.661+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.666032+0000) 2019-10-13T15:00:41.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:41.665+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.666791+0000) 2019-10-13T15:00:41.670 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:41.665+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.667569+0000) 2019-10-13T15:00:41.671 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:41.665+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.668305+0000) 2019-10-13T15:00:41.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:41.665+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.668715+0000) 2019-10-13T15:00:41.672 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:41.665+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.668905+0000) 2019-10-13T15:00:41.672 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:41.665+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.669268+0000) 2019-10-13T15:00:41.676 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:41.669+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.674129+0000) 2019-10-13T15:00:41.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:41.673+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217139 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:41.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:41.805+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:41.805957+0000) 2019-10-13T15:00:44.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:44.401+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.403819+0000) 2019-10-13T15:00:44.453 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:00:44.453 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:44.457 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:00:44.457 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:00:44.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:44.685+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:44.688106+0000) 2019-10-13T15:00:45.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:45.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:45.570681+0000) 2019-10-13T15:00:45.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:45.569+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:45.572185+0000) 2019-10-13T15:00:46.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:46.676+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.680054+0000) 2019-10-13T15:00:46.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:46.677+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.680491+0000) 2019-10-13T15:00:46.684 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:46.676+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.681225+0000) 2019-10-13T15:00:46.684 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:46.680+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.682056+0000) 2019-10-13T15:00:46.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:46.680+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.682715+0000) 2019-10-13T15:00:46.685 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:46.680+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.683086+0000) 2019-10-13T15:00:46.686 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:46.680+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.683915+0000) 2019-10-13T15:00:46.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:46.681+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:46.687196+0000) 2019-10-13T15:00:46.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:46.688+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217169 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:48.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:48.052+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:48.057407+0000) 2019-10-13T15:00:48.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:48.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:48.283075+0000) 2019-10-13T15:00:51.524 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:51.520+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.521822+0000) 2019-10-13T15:00:51.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:51.692+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.694704+0000) 2019-10-13T15:00:51.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:51.692+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.695047+0000) 2019-10-13T15:00:51.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:51.692+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.695920+0000) 2019-10-13T15:00:51.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:51.692+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.696441+0000) 2019-10-13T15:00:51.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:51.692+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.696940+0000) 2019-10-13T15:00:51.700 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:51.692+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.697326+0000) 2019-10-13T15:00:51.700 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:51.696+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.698002+0000) 2019-10-13T15:00:51.707 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:51.700+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.704399+0000) 2019-10-13T15:00:51.708 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:51.704+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:51.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:51.804+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:51.806141+0000) 2019-10-13T15:00:54.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:54.400+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.404058+0000) 2019-10-13T15:00:54.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:54.684+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:54.688297+0000) 2019-10-13T15:00:55.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:55.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:55.570869+0000) 2019-10-13T15:00:55.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:55.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:55.572346+0000) 2019-10-13T15:00:56.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:00:56.704+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.709545+0000) 2019-10-13T15:00:56.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:00:56.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.710419+0000) 2019-10-13T15:00:56.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:00:56.708+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.710230+0000) 2019-10-13T15:00:56.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:00:56.708+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.712455+0000) 2019-10-13T15:00:56.715 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:00:56.708+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.713027+0000) 2019-10-13T15:00:56.716 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:56.708+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.713528+0000) 2019-10-13T15:00:56.717 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:56.712+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.714320+0000) 2019-10-13T15:00:56.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:00:56.712+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:56.717896+0000) 2019-10-13T15:00:56.722 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:00:56.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:00:58.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:00:58.056+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:58.057603+0000) 2019-10-13T15:00:58.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:00:58.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:00:58.283311+0000) 2019-10-13T15:01:01.531 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:01.524+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.528406+0000) 2019-10-13T15:01:01.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:01.716+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.723014+0000) 2019-10-13T15:01:01.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:01.720+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.723703+0000) 2019-10-13T15:01:01.727 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:01.720+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.724434+0000) 2019-10-13T15:01:01.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:01.720+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.724764+0000) 2019-10-13T15:01:01.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:01.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.725615+0000) 2019-10-13T15:01:01.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:01.724+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.725893+0000) 2019-10-13T15:01:01.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:01.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.726262+0000) 2019-10-13T15:01:01.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:01.724+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.731864+0000) 2019-10-13T15:01:01.737 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:01.732+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217261 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:01.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:01.804+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:01.806334+0000) 2019-10-13T15:01:04.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:04.400+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.404299+0000) 2019-10-13T15:01:04.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:04.684+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:04.688519+0000) 2019-10-13T15:01:05.573 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:05.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:05.571061+0000) 2019-10-13T15:01:05.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:05.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:05.572532+0000) 2019-10-13T15:01:06.740 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:06.736+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.738010+0000) 2019-10-13T15:01:06.742 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:06.732+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.739927+0000) 2019-10-13T15:01:06.744 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:06.740+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.742039+0000) 2019-10-13T15:01:06.745 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:06.740+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.742572+0000) 2019-10-13T15:01:06.746 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:06.740+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.743945+0000) 2019-10-13T15:01:06.747 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:06.740+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.744386+0000) 2019-10-13T15:01:06.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:06.740+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.745623+0000) 2019-10-13T15:01:06.751 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:06.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217291 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:06.754 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:06.744+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:06.751759+0000) 2019-10-13T15:01:08.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:08.056+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:08.057834+0000) 2019-10-13T15:01:08.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:08.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:08.283547+0000) 2019-10-13T15:01:11.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:11.532+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.534820+0000) 2019-10-13T15:01:11.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:11.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.752392+0000) 2019-10-13T15:01:11.756 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:11.752+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.753838+0000) 2019-10-13T15:01:11.757 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:11.752+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.754403+0000) 2019-10-13T15:01:11.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:11.752+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.754858+0000) 2019-10-13T15:01:11.758 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:11.748+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.755668+0000) 2019-10-13T15:01:11.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:11.756+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.758272+0000) 2019-10-13T15:01:11.761 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:11.756+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.758785+0000) 2019-10-13T15:01:11.766 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:11.760+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:11.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:11.760+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.766114+0000) 2019-10-13T15:01:11.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:11.804+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:11.806529+0000) 2019-10-13T15:01:14.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:14.400+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.404500+0000) 2019-10-13T15:01:14.576 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:01:14.576 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:14.580 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:01:14.580 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:14.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:14.684+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:14.688724+0000) 2019-10-13T15:01:15.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:15.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:15.571211+0000) 2019-10-13T15:01:15.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:15.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:15.572757+0000) 2019-10-13T15:01:16.771 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:16.764+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.768388+0000) 2019-10-13T15:01:16.771 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:16.764+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.769157+0000) 2019-10-13T15:01:16.774 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:16.768+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.771660+0000) 2019-10-13T15:01:16.776 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:16.768+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.773562+0000) 2019-10-13T15:01:16.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:16.768+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.774835+0000) 2019-10-13T15:01:16.777 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:16.772+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.775149+0000) 2019-10-13T15:01:16.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:16.772+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.776892+0000) 2019-10-13T15:01:16.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:16.776+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:16.779007+0000) 2019-10-13T15:01:16.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:16.776+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217351 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:18.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:18.056+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:18.058035+0000) 2019-10-13T15:01:18.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:18.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:18.283794+0000) 2019-10-13T15:01:21.544 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:21.540+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.541463+0000) 2019-10-13T15:01:21.786 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:21.780+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.783883+0000) 2019-10-13T15:01:21.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:21.780+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.784492+0000) 2019-10-13T15:01:21.787 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:21.780+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.784628+0000) 2019-10-13T15:01:21.788 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:21.780+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.785072+0000) 2019-10-13T15:01:21.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:21.780+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.785800+0000) 2019-10-13T15:01:21.789 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:21.784+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.785993+0000) 2019-10-13T15:01:21.790 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:21.784+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.787970+0000) 2019-10-13T15:01:21.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:21.784+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.789715+0000) 2019-10-13T15:01:21.797 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:21.792+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217383 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:21.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:21.804+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:21.806678+0000) 2019-10-13T15:01:24.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:24.400+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.404707+0000) 2019-10-13T15:01:24.691 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:24.688+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:24.688961+0000) 2019-10-13T15:01:25.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:25.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:25.571436+0000) 2019-10-13T15:01:25.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:25.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:25.572954+0000) 2019-10-13T15:01:26.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:26.792+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.797547+0000) 2019-10-13T15:01:26.801 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:26.792+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.798437+0000) 2019-10-13T15:01:26.801 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:26.796+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.798572+0000) 2019-10-13T15:01:26.802 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:26.796+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.799755+0000) 2019-10-13T15:01:26.802 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:26.796+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.799941+0000) 2019-10-13T15:01:26.803 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:26.796+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.800358+0000) 2019-10-13T15:01:26.808 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:26.804+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.805732+0000) 2019-10-13T15:01:26.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:26.804+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217413 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:26.812 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:26.804+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:26.809390+0000) 2019-10-13T15:01:28.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:28.056+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:28.058250+0000) 2019-10-13T15:01:28.286 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:28.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:28.284016+0000) 2019-10-13T15:01:31.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:31.540+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.548027+0000) 2019-10-13T15:01:31.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:31.804+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.806816+0000) 2019-10-13T15:01:31.815 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:31.808+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.812610+0000) 2019-10-13T15:01:31.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:31.812+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.813945+0000) 2019-10-13T15:01:31.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:31.812+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.814408+0000) 2019-10-13T15:01:31.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:31.812+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.814675+0000) 2019-10-13T15:01:31.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:31.812+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.815324+0000) 2019-10-13T15:01:31.818 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:31.812+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.815006+0000) 2019-10-13T15:01:31.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:31.812+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.816269+0000) 2019-10-13T15:01:31.824 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:31.816+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:31.821655+0000) 2019-10-13T15:01:31.825 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:31.820+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217442 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:34.407 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:34.404+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.404943+0000) 2019-10-13T15:01:34.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:34.684+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:34.689157+0000) 2019-10-13T15:01:35.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:35.568+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:35.571652+0000) 2019-10-13T15:01:35.575 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:35.568+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:35.573157+0000) 2019-10-13T15:01:36.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:36.824+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.826496+0000) 2019-10-13T15:01:36.830 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:36.824+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.826961+0000) 2019-10-13T15:01:36.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:36.824+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.827345+0000) 2019-10-13T15:01:36.830 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:36.824+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.827965+0000) 2019-10-13T15:01:36.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:36.824+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.828307+0000) 2019-10-13T15:01:36.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:36.824+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.828981+0000) 2019-10-13T15:01:36.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:36.824+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.829225+0000) 2019-10-13T15:01:36.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:36.828+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:36.832555+0000) 2019-10-13T15:01:36.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:36.832+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217471 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:38.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:38.056+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:38.058451+0000) 2019-10-13T15:01:38.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:38.280+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:38.284214+0000) 2019-10-13T15:01:41.557 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:41.548+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.554514+0000) 2019-10-13T15:01:41.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:41.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.807016+0000) 2019-10-13T15:01:41.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:41.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.840000+0000) 2019-10-13T15:01:41.843 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:41.835+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.840674+0000) 2019-10-13T15:01:41.843 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:41.836+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.841343+0000) 2019-10-13T15:01:41.846 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:41.839+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.843035+0000) 2019-10-13T15:01:41.846 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:41.839+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.843563+0000) 2019-10-13T15:01:41.847 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:41.839+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.843825+0000) 2019-10-13T15:01:41.847 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:41.839+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.844233+0000) 2019-10-13T15:01:41.853 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:41.844+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:41.850859+0000) 2019-10-13T15:01:41.853 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:41.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217505 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:44.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:44.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.405140+0000) 2019-10-13T15:01:44.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:44.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:44.689385+0000) 2019-10-13T15:01:44.707 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:01:44.708 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:44.711 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:01:44.711 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:01:45.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:45.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:45.571844+0000) 2019-10-13T15:01:45.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:45.572+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:45.573369+0000) 2019-10-13T15:01:46.856 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:46.848+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.853750+0000) 2019-10-13T15:01:46.857 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:46.851+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.854639+0000) 2019-10-13T15:01:46.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:46.851+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.857027+0000) 2019-10-13T15:01:46.860 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:46.851+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.857316+0000) 2019-10-13T15:01:46.861 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:46.851+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.857641+0000) 2019-10-13T15:01:46.861 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:46.855+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.858251+0000) 2019-10-13T15:01:46.861 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:46.855+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.858643+0000) 2019-10-13T15:01:46.867 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:46.859+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217534 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:46.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:46.860+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:46.865503+0000) 2019-10-13T15:01:48.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:48.055+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:48.058656+0000) 2019-10-13T15:01:48.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:48.279+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:48.284408+0000) 2019-10-13T15:01:51.563 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:51.560+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.560812+0000) 2019-10-13T15:01:51.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:51.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.807170+0000) 2019-10-13T15:01:51.871 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:51.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.868451+0000) 2019-10-13T15:01:51.871 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:51.863+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.869114+0000) 2019-10-13T15:01:51.872 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:51.863+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.869545+0000) 2019-10-13T15:01:51.873 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:51.867+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.870090+0000) 2019-10-13T15:01:51.873 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:51.867+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.870401+0000) 2019-10-13T15:01:51.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:51.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.871765+0000) 2019-10-13T15:01:51.874 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:51.867+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.872038+0000) 2019-10-13T15:01:51.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:51.872+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:51.874960+0000) 2019-10-13T15:01:51.882 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:51.875+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:54.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:54.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.405376+0000) 2019-10-13T15:01:54.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:54.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:54.689578+0000) 2019-10-13T15:01:55.574 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:55.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:55.572043+0000) 2019-10-13T15:01:55.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:55.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:55.573562+0000) 2019-10-13T15:01:56.886 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:01:56.879+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.884038+0000) 2019-10-13T15:01:56.887 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:56.879+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.884470+0000) 2019-10-13T15:01:56.887 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:01:56.883+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.885173+0000) 2019-10-13T15:01:56.888 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:01:56.879+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.885454+0000) 2019-10-13T15:01:56.888 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:01:56.883+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.885866+0000) 2019-10-13T15:01:56.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:01:56.883+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.886436+0000) 2019-10-13T15:01:56.889 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:56.883+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.886684+0000) 2019-10-13T15:01:56.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:01:56.891+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:56.893717+0000) 2019-10-13T15:01:56.896 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:01:56.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:01:58.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:01:58.055+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:58.058888+0000) 2019-10-13T15:01:58.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:01:58.279+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:01:58.284610+0000) 2019-10-13T15:02:01.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:01.563+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.567203+0000) 2019-10-13T15:02:01.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:01.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.807410+0000) 2019-10-13T15:02:01.900 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:01.891+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.897333+0000) 2019-10-13T15:02:01.901 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:01.895+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.898073+0000) 2019-10-13T15:02:01.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:01.891+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.897342+0000) 2019-10-13T15:02:01.902 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:01.895+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.899144+0000) 2019-10-13T15:02:01.902 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:01.895+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.899388+0000) 2019-10-13T15:02:01.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:01.899+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.902028+0000) 2019-10-13T15:02:01.907 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:01.899+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.904300+0000) 2019-10-13T15:02:01.908 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:01.903+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:01.906341+0000) 2019-10-13T15:02:01.911 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:01.903+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217629 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:04.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:04.399+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.405573+0000) 2019-10-13T15:02:04.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:04.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:04.689803+0000) 2019-10-13T15:02:05.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:05.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:05.572233+0000) 2019-10-13T15:02:05.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:05.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:05.573795+0000) 2019-10-13T15:02:06.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:06.907+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.911992+0000) 2019-10-13T15:02:06.915 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:06.907+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.912689+0000) 2019-10-13T15:02:06.915 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:06.911+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.913034+0000) 2019-10-13T15:02:06.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:06.911+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.913938+0000) 2019-10-13T15:02:06.918 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:06.911+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.915595+0000) 2019-10-13T15:02:06.918 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:06.911+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.915845+0000) 2019-10-13T15:02:06.919 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:06.911+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.916626+0000) 2019-10-13T15:02:06.922 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:06.915+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:06.919608+0000) 2019-10-13T15:02:06.925 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:06.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:08.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:08.055+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:08.059116+0000) 2019-10-13T15:02:08.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:08.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:08.284825+0000) 2019-10-13T15:02:11.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:11.571+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.573572+0000) 2019-10-13T15:02:11.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:11.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.807686+0000) 2019-10-13T15:02:11.929 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:11.923+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.925950+0000) 2019-10-13T15:02:11.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:11.923+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.926520+0000) 2019-10-13T15:02:11.930 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:11.923+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.927372+0000) 2019-10-13T15:02:11.931 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:11.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.927582+0000) 2019-10-13T15:02:11.931 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:11.923+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.927842+0000) 2019-10-13T15:02:11.931 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:11.923+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.928225+0000) 2019-10-13T15:02:11.931 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:11.923+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.929143+0000) 2019-10-13T15:02:11.937 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:11.931+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:11.935256+0000) 2019-10-13T15:02:11.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:11.931+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:14.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:14.403+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.405807+0000) 2019-10-13T15:02:14.692 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:14.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:14.690001+0000) 2019-10-13T15:02:14.832 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:02:14.832 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:14.835 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:02:14.836 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:15.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:15.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:15.572481+0000) 2019-10-13T15:02:15.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:15.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:15.574028+0000) 2019-10-13T15:02:16.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:16.935+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.939803+0000) 2019-10-13T15:02:16.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:16.935+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.940314+0000) 2019-10-13T15:02:16.943 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:16.935+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.940686+0000) 2019-10-13T15:02:16.944 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:16.935+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.941535+0000) 2019-10-13T15:02:16.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:16.939+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.942227+0000) 2019-10-13T15:02:16.945 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:16.939+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.942687+0000) 2019-10-13T15:02:16.945 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:16.939+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.943034+0000) 2019-10-13T15:02:16.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:16.947+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:16.949656+0000) 2019-10-13T15:02:16.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:16.947+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217719 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:18.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:18.055+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:18.059321+0000) 2019-10-13T15:02:18.287 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:18.279+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:18.285049+0000) 2019-10-13T15:02:21.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:21.579+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.580138+0000) 2019-10-13T15:02:21.810 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:21.803+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.807909+0000) 2019-10-13T15:02:21.958 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:21.951+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.955276+0000) 2019-10-13T15:02:21.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:21.951+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.955683+0000) 2019-10-13T15:02:21.960 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:21.951+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.957299+0000) 2019-10-13T15:02:21.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:21.955+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.958609+0000) 2019-10-13T15:02:21.963 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:21.955+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.960625+0000) 2019-10-13T15:02:21.965 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:21.959+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.963308+0000) 2019-10-13T15:02:21.967 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:21.963+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.964861+0000) 2019-10-13T15:02:21.968 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:21.959+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:21.965168+0000) 2019-10-13T15:02:21.968 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:21.963+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217752 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:24.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:24.403+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.406046+0000) 2019-10-13T15:02:24.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:24.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:24.690206+0000) 2019-10-13T15:02:25.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:25.567+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:25.572678+0000) 2019-10-13T15:02:25.576 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:25.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:25.574220+0000) 2019-10-13T15:02:26.972 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:26.967+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.969758+0000) 2019-10-13T15:02:26.973 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:26.967+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.970096+0000) 2019-10-13T15:02:26.974 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:26.967+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.970430+0000) 2019-10-13T15:02:26.974 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:26.967+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.970989+0000) 2019-10-13T15:02:26.976 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:26.971+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.974264+0000) 2019-10-13T15:02:26.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:26.971+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.976008+0000) 2019-10-13T15:02:26.980 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:26.975+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.977832+0000) 2019-10-13T15:02:26.980 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:26.975+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:26.978248+0000) 2019-10-13T15:02:26.982 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:26.975+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:28.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:28.059+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:28.059519+0000) 2019-10-13T15:02:28.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:28.279+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:28.285245+0000) 2019-10-13T15:02:31.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:31.583+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.586665+0000) 2019-10-13T15:02:31.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:31.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.808140+0000) 2019-10-13T15:02:31.986 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:31.979+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.983819+0000) 2019-10-13T15:02:31.989 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:31.983+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.986512+0000) 2019-10-13T15:02:31.990 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:31.983+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.987083+0000) 2019-10-13T15:02:31.990 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:31.983+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.987210+0000) 2019-10-13T15:02:31.990 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:31.983+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.987771+0000) 2019-10-13T15:02:31.991 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:31.983+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.988034+0000) 2019-10-13T15:02:31.991 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:31.983+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.988653+0000) 2019-10-13T15:02:31.998 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:31.991+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217813 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:31.998 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:31.991+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:31.995846+0000) 2019-10-13T15:02:34.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:34.403+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.406250+0000) 2019-10-13T15:02:34.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:34.687+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:34.690424+0000) 2019-10-13T15:02:35.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:35.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:35.572875+0000) 2019-10-13T15:02:35.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:35.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:35.574405+0000) 2019-10-13T15:02:37.001 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:36.995+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:36.998531+0000) 2019-10-13T15:02:37.001 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:36.995+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:36.998882+0000) 2019-10-13T15:02:37.002 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:36.995+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:36.999832+0000) 2019-10-13T15:02:37.003 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:36.995+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:37.000214+0000) 2019-10-13T15:02:37.003 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:36.995+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:37.000771+0000) 2019-10-13T15:02:37.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:36.999+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:37.002605+0000) 2019-10-13T15:02:37.006 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:36.999+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:37.003428+0000) 2019-10-13T15:02:37.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:37.003+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217843 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:37.012 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:37.007+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:37.009566+0000) 2019-10-13T15:02:38.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:38.055+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:38.059733+0000) 2019-10-13T15:02:38.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:38.282+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:38.285446+0000) 2019-10-13T15:02:41.596 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:41.591+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:41.593297+0000) 2019-10-13T15:02:41.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:41.806+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:41.808350+0000) 2019-10-13T15:02:42.018 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:42.010+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.014882+0000) 2019-10-13T15:02:42.018 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:42.010+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.015299+0000) 2019-10-13T15:02:42.018 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:42.015+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.016236+0000) 2019-10-13T15:02:42.019 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:42.010+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.016313+0000) 2019-10-13T15:02:42.019 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:42.010+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.016412+0000) 2019-10-13T15:02:42.020 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:42.010+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.016944+0000) 2019-10-13T15:02:42.020 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:42.010+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.017589+0000) 2019-10-13T15:02:42.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:42.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217876 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:42.027 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:42.019+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:42.023820+0000) 2019-10-13T15:02:44.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:44.402+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.406466+0000) 2019-10-13T15:02:44.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:44.686+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:44.690618+0000) 2019-10-13T15:02:44.978 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:02:44.978 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:44.982 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:02:44.982 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:02:45.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:45.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:45.573087+0000) 2019-10-13T15:02:45.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:45.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:45.574614+0000) 2019-10-13T15:02:47.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:47.022+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.027968+0000) 2019-10-13T15:02:47.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:47.022+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.028397+0000) 2019-10-13T15:02:47.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:47.022+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.028957+0000) 2019-10-13T15:02:47.032 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:47.022+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.029619+0000) 2019-10-13T15:02:47.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:47.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.030258+0000) 2019-10-13T15:02:47.033 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:47.027+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.030677+0000) 2019-10-13T15:02:47.041 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:47.035+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.037175+0000) 2019-10-13T15:02:47.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:47.026+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:47.031316+0000) 2019-10-13T15:02:47.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:47.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217906 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:48.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:48.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:48.059947+0000) 2019-10-13T15:02:48.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:48.278+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:48.285646+0000) 2019-10-13T15:02:51.602 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:51.599+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:51.599888+0000) 2019-10-13T15:02:51.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:51.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:51.808553+0000) 2019-10-13T15:02:52.044 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:52.034+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.041147+0000) 2019-10-13T15:02:52.045 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:52.038+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.042753+0000) 2019-10-13T15:02:52.046 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:52.039+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.043071+0000) 2019-10-13T15:02:52.046 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:52.038+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.043263+0000) 2019-10-13T15:02:52.046 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:52.038+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.044298+0000) 2019-10-13T15:02:52.047 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:52.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.044839+0000) 2019-10-13T15:02:52.048 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:52.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.045486+0000) 2019-10-13T15:02:52.054 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:52.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217937 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:52.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:52.051+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:52.052638+0000) 2019-10-13T15:02:54.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:54.402+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.406670+0000) 2019-10-13T15:02:54.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:54.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:54.690854+0000) 2019-10-13T15:02:55.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:55.566+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:55.573289+0000) 2019-10-13T15:02:55.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:55.571+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:55.574833+0000) 2019-10-13T15:02:57.058 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:57.050+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.055260+0000) 2019-10-13T15:02:57.058 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:02:57.050+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.055830+0000) 2019-10-13T15:02:57.059 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:02:57.050+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.056602+0000) 2019-10-13T15:02:57.059 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:02:57.055+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.057387+0000) 2019-10-13T15:02:57.060 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:57.054+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.058056+0000) 2019-10-13T15:02:57.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:02:57.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.058116+0000) 2019-10-13T15:02:57.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:02:57.054+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.058539+0000) 2019-10-13T15:02:57.067 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:02:57.063+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:57.064735+0000) 2019-10-13T15:02:57.068 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:02:57.058+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217967 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:02:58.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:02:58.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:58.060175+0000) 2019-10-13T15:02:58.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:02:58.282+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:02:58.285867+0000) 2019-10-13T15:03:01.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:01.602+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:01.606461+0000) 2019-10-13T15:03:01.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:01.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:01.808760+0000) 2019-10-13T15:03:02.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:02.066+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.069442+0000) 2019-10-13T15:03:02.072 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:02.066+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.070086+0000) 2019-10-13T15:03:02.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:02.066+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.070440+0000) 2019-10-13T15:03:02.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:02.066+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.071165+0000) 2019-10-13T15:03:02.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:02.066+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.071377+0000) 2019-10-13T15:03:02.074 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:02.066+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.071494+0000) 2019-10-13T15:03:02.075 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:02.066+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.071891+0000) 2019-10-13T15:03:02.079 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:02.074+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:02.076380+0000) 2019-10-13T15:03:02.081 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:02.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218000 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:04.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:04.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.406869+0000) 2019-10-13T15:03:04.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:04.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:04.691055+0000) 2019-10-13T15:03:05.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:05.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:05.573485+0000) 2019-10-13T15:03:05.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:05.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:05.575030+0000) 2019-10-13T15:03:07.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:07.078+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.082460+0000) 2019-10-13T15:03:07.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:07.078+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.083474+0000) 2019-10-13T15:03:07.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:07.078+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.085511+0000) 2019-10-13T15:03:07.088 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:07.082+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.085933+0000) 2019-10-13T15:03:07.089 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:07.082+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.086543+0000) 2019-10-13T15:03:07.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:07.082+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.086929+0000) 2019-10-13T15:03:07.090 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:07.082+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.087201+0000) 2019-10-13T15:03:07.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:07.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:07.088411+0000) 2019-10-13T15:03:07.095 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:07.086+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218030 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:08.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:08.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:08.060389+0000) 2019-10-13T15:03:08.288 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:08.282+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:08.286070+0000) 2019-10-13T15:03:11.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:11.610+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:11.612349+0000) 2019-10-13T15:03:11.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:11.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:11.808958+0000) 2019-10-13T15:03:12.098 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:12.090+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.095565+0000) 2019-10-13T15:03:12.098 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:12.090+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.095857+0000) 2019-10-13T15:03:12.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:12.090+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.096495+0000) 2019-10-13T15:03:12.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:12.090+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.096819+0000) 2019-10-13T15:03:12.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:12.094+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.097758+0000) 2019-10-13T15:03:12.101 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:12.094+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.098864+0000) 2019-10-13T15:03:12.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:12.094+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.099171+0000) 2019-10-13T15:03:12.107 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:12.102+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:12.104399+0000) 2019-10-13T15:03:12.110 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:12.102+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:14.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:14.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.407105+0000) 2019-10-13T15:03:14.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:14.686+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:14.691273+0000) 2019-10-13T15:03:15.318 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:03:15.318 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:15.322 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:03:15.322 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:15.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:15.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:15.573714+0000) 2019-10-13T15:03:15.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:15.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:15.575253+0000) 2019-10-13T15:03:17.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:17.106+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.111058+0000) 2019-10-13T15:03:17.114 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:17.106+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.111431+0000) 2019-10-13T15:03:17.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:17.106+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.111653+0000) 2019-10-13T15:03:17.115 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:17.106+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.112317+0000) 2019-10-13T15:03:17.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:17.106+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.112582+0000) 2019-10-13T15:03:17.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:17.106+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.113020+0000) 2019-10-13T15:03:17.117 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:17.110+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.115281+0000) 2019-10-13T15:03:17.124 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:17.118+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218091 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:17.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:17.118+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:17.122340+0000) 2019-10-13T15:03:18.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:18.054+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:18.060566+0000) 2019-10-13T15:03:18.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:18.282+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:18.286257+0000) 2019-10-13T15:03:21.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:21.618+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:21.618850+0000) 2019-10-13T15:03:21.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:21.802+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:21.809184+0000) 2019-10-13T15:03:22.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:22.118+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.125106+0000) 2019-10-13T15:03:22.128 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:22.118+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.125556+0000) 2019-10-13T15:03:22.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:22.122+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.126171+0000) 2019-10-13T15:03:22.129 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:22.122+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.127016+0000) 2019-10-13T15:03:22.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:22.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.127566+0000) 2019-10-13T15:03:22.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:22.122+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.127838+0000) 2019-10-13T15:03:22.131 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:22.122+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.128242+0000) 2019-10-13T15:03:22.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:22.130+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:22.135360+0000) 2019-10-13T15:03:22.138 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:22.130+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218124 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:24.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:24.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.407312+0000) 2019-10-13T15:03:24.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:24.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:24.691486+0000) 2019-10-13T15:03:25.576 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:25.570+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:25.573955+0000) 2019-10-13T15:03:25.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:25.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:25.575456+0000) 2019-10-13T15:03:27.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:27.134+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.138358+0000) 2019-10-13T15:03:27.154 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:27.134+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.139912+0000) 2019-10-13T15:03:27.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:27.134+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.140657+0000) 2019-10-13T15:03:27.155 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:27.134+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.140800+0000) 2019-10-13T15:03:27.155 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:27.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.141256+0000) 2019-10-13T15:03:27.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:27.138+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.142683+0000) 2019-10-13T15:03:27.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:27.138+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.144734+0000) 2019-10-13T15:03:27.164 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:27.142+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218154 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:27.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:27.146+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:27.150371+0000) 2019-10-13T15:03:28.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:28.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:28.060742+0000) 2019-10-13T15:03:28.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:28.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:28.286462+0000) 2019-10-13T15:03:31.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:31.622+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:31.625442+0000) 2019-10-13T15:03:31.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:31.806+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:31.809393+0000) 2019-10-13T15:03:32.154 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:32.146+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.152027+0000) 2019-10-13T15:03:32.155 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:32.146+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.152551+0000) 2019-10-13T15:03:32.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:32.146+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.153600+0000) 2019-10-13T15:03:32.157 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:32.150+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.154093+0000) 2019-10-13T15:03:32.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:32.150+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.154346+0000) 2019-10-13T15:03:32.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:32.150+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.155363+0000) 2019-10-13T15:03:32.158 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:32.150+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.155500+0000) 2019-10-13T15:03:32.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:32.158+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:32.161342+0000) 2019-10-13T15:03:32.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:32.158+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:34.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:34.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:34.407537+0000) 2019-10-13T15:03:34.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:34.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:34.691680+0000) 2019-10-13T15:03:35.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:35.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.574180+0000) 2019-10-13T15:03:35.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:35.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:35.575641+0000) 2019-10-13T15:03:37.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:37.161+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.166254+0000) 2019-10-13T15:03:37.169 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:37.161+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.166719+0000) 2019-10-13T15:03:37.170 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:37.161+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.167137+0000) 2019-10-13T15:03:37.171 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:37.161+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.168937+0000) 2019-10-13T15:03:37.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:37.166+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.169390+0000) 2019-10-13T15:03:37.172 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:37.161+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.169617+0000) 2019-10-13T15:03:37.173 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:37.165+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.170843+0000) 2019-10-13T15:03:37.179 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:37.169+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:37.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:37.174+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:37.176967+0000) 2019-10-13T15:03:38.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:38.057+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:38.060915+0000) 2019-10-13T15:03:38.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:38.281+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:38.286674+0000) 2019-10-13T15:03:41.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:41.630+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:41.631942+0000) 2019-10-13T15:03:41.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:41.805+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:41.809630+0000) 2019-10-13T15:03:42.182 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:42.173+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.179760+0000) 2019-10-13T15:03:42.183 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:42.173+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.180374+0000) 2019-10-13T15:03:42.183 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:42.173+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.180714+0000) 2019-10-13T15:03:42.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:42.177+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.184282+0000) 2019-10-13T15:03:42.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:42.181+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.186708+0000) 2019-10-13T15:03:42.189 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:42.181+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.186983+0000) 2019-10-13T15:03:42.190 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:42.182+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.187913+0000) 2019-10-13T15:03:42.191 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:42.186+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:42.188972+0000) 2019-10-13T15:03:42.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:42.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218250 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:44.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:44.405+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:44.407733+0000) 2019-10-13T15:03:44.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:44.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:44.691837+0000) 2019-10-13T15:03:45.441 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:03:45.441 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:45.445 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:03:45.445 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:03:45.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:45.569+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.574401+0000) 2019-10-13T15:03:45.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:45.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:45.575908+0000) 2019-10-13T15:03:47.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:47.189+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.195285+0000) 2019-10-13T15:03:47.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:47.189+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.195875+0000) 2019-10-13T15:03:47.199 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:47.189+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.195970+0000) 2019-10-13T15:03:47.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:47.189+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.196574+0000) 2019-10-13T15:03:47.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:47.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.197027+0000) 2019-10-13T15:03:47.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:47.189+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.197470+0000) 2019-10-13T15:03:47.200 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:47.194+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.198044+0000) 2019-10-13T15:03:47.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:47.202+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:47.204672+0000) 2019-10-13T15:03:47.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:47.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218280 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:48.063 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:48.057+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:48.061092+0000) 2019-10-13T15:03:48.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:48.285+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:48.286867+0000) 2019-10-13T15:03:51.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:51.638+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:51.638574+0000) 2019-10-13T15:03:51.812 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:51.809+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:51.809833+0000) 2019-10-13T15:03:52.213 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:52.209+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.209803+0000) 2019-10-13T15:03:52.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:52.209+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.210536+0000) 2019-10-13T15:03:52.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:52.209+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.211194+0000) 2019-10-13T15:03:52.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:52.209+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.212882+0000) 2019-10-13T15:03:52.216 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:52.210+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.213243+0000) 2019-10-13T15:03:52.216 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:52.209+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.213427+0000) 2019-10-13T15:03:52.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:52.209+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.213042+0000) 2019-10-13T15:03:52.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:52.214+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:52.218331+0000) 2019-10-13T15:03:52.223 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:52.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:54.410 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:54.405+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:54.407965+0000) 2019-10-13T15:03:54.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:54.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:54.692045+0000) 2019-10-13T15:03:55.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:55.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.574630+0000) 2019-10-13T15:03:55.578 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:55.574+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:55.576133+0000) 2019-10-13T15:03:57.227 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:03:57.221+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.224192+0000) 2019-10-13T15:03:57.227 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:57.221+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.224474+0000) 2019-10-13T15:03:57.228 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:03:57.221+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.224824+0000) 2019-10-13T15:03:57.228 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:57.225+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.226050+0000) 2019-10-13T15:03:57.229 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:03:57.222+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.227009+0000) 2019-10-13T15:03:57.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:03:57.225+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.227548+0000) 2019-10-13T15:03:57.230 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:03:57.225+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.227572+0000) 2019-10-13T15:03:57.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:03:57.230+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:57.232258+0000) 2019-10-13T15:03:57.237 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:03:57.233+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218342 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:03:58.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:03:58.057+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:58.061273+0000) 2019-10-13T15:03:58.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:03:58.285+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:03:58.287076+0000) 2019-10-13T15:04:01.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:01.642+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:01.645048+0000) 2019-10-13T15:04:01.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:01.809+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:01.810039+0000) 2019-10-13T15:04:02.240 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:02.233+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.237377+0000) 2019-10-13T15:04:02.241 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:02.237+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.237920+0000) 2019-10-13T15:04:02.241 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:02.237+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.238503+0000) 2019-10-13T15:04:02.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:02.237+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.238975+0000) 2019-10-13T15:04:02.242 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:02.234+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.239959+0000) 2019-10-13T15:04:02.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:02.237+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.239977+0000) 2019-10-13T15:04:02.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:02.237+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.240831+0000) 2019-10-13T15:04:02.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:02.245+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:02.252 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:02.246+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:02.249868+0000) 2019-10-13T15:04:04.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:04.405+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:04.408197+0000) 2019-10-13T15:04:04.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:04.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:04.692245+0000) 2019-10-13T15:04:05.577 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:05.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.574871+0000) 2019-10-13T15:04:05.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:05.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:05.576325+0000) 2019-10-13T15:04:07.254 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:07.249+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.251011+0000) 2019-10-13T15:04:07.256 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:07.253+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.254001+0000) 2019-10-13T15:04:07.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:07.253+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.254170+0000) 2019-10-13T15:04:07.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:07.253+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.254622+0000) 2019-10-13T15:04:07.259 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:07.253+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.256466+0000) 2019-10-13T15:04:07.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:07.253+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.256793+0000) 2019-10-13T15:04:07.259 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:07.253+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.257107+0000) 2019-10-13T15:04:07.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:07.261+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218406 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:07.268 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:07.261+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:07.266150+0000) 2019-10-13T15:04:08.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:08.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:08.061487+0000) 2019-10-13T15:04:08.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:08.285+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:08.287230+0000) 2019-10-13T15:04:11.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:11.649+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:11.651568+0000) 2019-10-13T15:04:11.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:11.809+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:11.810243+0000) 2019-10-13T15:04:12.269 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:12.265+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.266877+0000) 2019-10-13T15:04:12.270 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:12.265+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.267576+0000) 2019-10-13T15:04:12.271 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:12.265+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.267584+0000) 2019-10-13T15:04:12.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:12.265+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.267800+0000) 2019-10-13T15:04:12.271 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:12.265+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.268236+0000) 2019-10-13T15:04:12.272 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:12.265+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.268835+0000) 2019-10-13T15:04:12.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:12.265+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.269079+0000) 2019-10-13T15:04:12.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:12.269+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:12.273749+0000) 2019-10-13T15:04:12.278 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:12.273+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:14.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:14.405+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:14.408389+0000) 2019-10-13T15:04:14.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:14.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:14.692463+0000) 2019-10-13T15:04:15.555 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:04:15.555 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:15.559 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:04:15.559 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:15.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:15.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.575070+0000) 2019-10-13T15:04:15.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:15.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:15.576486+0000) 2019-10-13T15:04:17.282 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:17.277+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.279850+0000) 2019-10-13T15:04:17.284 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:17.277+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.281506+0000) 2019-10-13T15:04:17.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:17.281+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.281862+0000) 2019-10-13T15:04:17.285 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:17.281+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.282544+0000) 2019-10-13T15:04:17.285 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:17.281+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.282961+0000) 2019-10-13T15:04:17.286 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:17.277+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.283990+0000) 2019-10-13T15:04:17.287 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:17.281+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.284576+0000) 2019-10-13T15:04:17.293 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:17.285+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:17.290552+0000) 2019-10-13T15:04:17.293 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:17.289+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:18.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:18.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:18.061670+0000) 2019-10-13T15:04:18.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:18.285+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:18.287454+0000) 2019-10-13T15:04:21.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:21.653+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:21.657947+0000) 2019-10-13T15:04:21.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:21.809+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:21.810445+0000) 2019-10-13T15:04:22.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:22.293+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.294464+0000) 2019-10-13T15:04:22.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:22.293+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.294690+0000) 2019-10-13T15:04:22.300 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:22.293+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.296150+0000) 2019-10-13T15:04:22.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:22.297+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.298021+0000) 2019-10-13T15:04:22.301 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:22.293+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.298604+0000) 2019-10-13T15:04:22.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:22.297+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.298746+0000) 2019-10-13T15:04:22.302 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:22.297+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.299352+0000) 2019-10-13T15:04:22.304 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:22.297+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:22.302090+0000) 2019-10-13T15:04:22.308 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:22.301+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218501 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:24.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:24.405+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:24.408584+0000) 2019-10-13T15:04:24.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:24.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:24.692661+0000) 2019-10-13T15:04:25.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:25.573+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.575284+0000) 2019-10-13T15:04:25.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:25.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:25.576726+0000) 2019-10-13T15:04:27.311 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:27.305+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.308609+0000) 2019-10-13T15:04:27.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:27.305+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.309052+0000) 2019-10-13T15:04:27.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:27.309+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.310645+0000) 2019-10-13T15:04:27.314 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:27.309+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.310935+0000) 2019-10-13T15:04:27.314 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:27.309+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.311287+0000) 2019-10-13T15:04:27.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:27.305+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.311432+0000) 2019-10-13T15:04:27.315 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:27.309+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.312090+0000) 2019-10-13T15:04:27.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:27.313+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:27.318419+0000) 2019-10-13T15:04:27.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:27.317+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:28.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:28.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:28.061864+0000) 2019-10-13T15:04:28.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:28.285+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:28.287667+0000) 2019-10-13T15:04:31.671 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:31.665+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:31.668728+0000) 2019-10-13T15:04:31.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:31.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:31.810670+0000) 2019-10-13T15:04:32.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:32.320+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.323035+0000) 2019-10-13T15:04:32.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:32.320+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.323397+0000) 2019-10-13T15:04:32.326 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:32.320+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.324170+0000) 2019-10-13T15:04:32.327 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:32.321+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.324821+0000) 2019-10-13T15:04:32.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:32.320+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.324797+0000) 2019-10-13T15:04:32.328 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:32.320+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.325609+0000) 2019-10-13T15:04:32.328 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:32.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.325910+0000) 2019-10-13T15:04:32.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:32.329+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:32.332549+0000) 2019-10-13T15:04:32.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:32.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:34.411 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:34.404+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:34.408744+0000) 2019-10-13T15:04:34.695 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:34.688+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:34.692877+0000) 2019-10-13T15:04:35.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:35.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.575498+0000) 2019-10-13T15:04:35.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:35.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:35.576921+0000) 2019-10-13T15:04:37.340 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:37.332+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.337671+0000) 2019-10-13T15:04:37.342 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:37.336+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.339739+0000) 2019-10-13T15:04:37.342 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:37.333+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.339995+0000) 2019-10-13T15:04:37.343 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:37.336+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.340481+0000) 2019-10-13T15:04:37.343 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:37.336+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.340968+0000) 2019-10-13T15:04:37.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:37.340+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.343305+0000) 2019-10-13T15:04:37.347 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:37.340+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.345082+0000) 2019-10-13T15:04:37.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:37.341+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:37.347713+0000) 2019-10-13T15:04:37.352 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:37.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218595 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:38.064 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:38.060+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:38.062034+0000) 2019-10-13T15:04:38.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:38.284+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:38.287839+0000) 2019-10-13T15:04:41.677 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:41.673+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:41.675145+0000) 2019-10-13T15:04:41.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:41.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:41.810899+0000) 2019-10-13T15:04:42.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:42.348+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.353608+0000) 2019-10-13T15:04:42.357 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:42.352+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.355102+0000) 2019-10-13T15:04:42.358 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:42.352+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.355633+0000) 2019-10-13T15:04:42.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:42.349+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.355897+0000) 2019-10-13T15:04:42.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:42.352+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.356396+0000) 2019-10-13T15:04:42.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:42.356+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.358150+0000) 2019-10-13T15:04:42.362 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:42.356+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.359934+0000) 2019-10-13T15:04:42.365 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:42.357+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:42.362723+0000) 2019-10-13T15:04:42.367 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:42.360+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218627 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:44.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:44.404+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:44.408946+0000) 2019-10-13T15:04:44.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:44.688+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:44.693077+0000) 2019-10-13T15:04:45.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:45.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.575726+0000) 2019-10-13T15:04:45.579 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:45.573+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:45.577108+0000) 2019-10-13T15:04:45.679 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:04:45.679 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:45.683 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:04:45.684 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:04:47.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:47.364+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.367501+0000) 2019-10-13T15:04:47.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:47.368+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.370390+0000) 2019-10-13T15:04:47.373 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:47.368+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.370741+0000) 2019-10-13T15:04:47.374 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:47.368+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.371134+0000) 2019-10-13T15:04:47.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:47.368+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.371616+0000) 2019-10-13T15:04:47.376 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:47.369+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.373656+0000) 2019-10-13T15:04:47.378 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:47.372+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.375574+0000) 2019-10-13T15:04:47.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:47.373+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:47.377571+0000) 2019-10-13T15:04:47.382 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:47.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218657 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:48.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:48.060+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:48.062223+0000) 2019-10-13T15:04:48.290 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:48.284+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:48.288027+0000) 2019-10-13T15:04:51.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:51.681+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:51.681469+0000) 2019-10-13T15:04:51.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:51.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:51.811098+0000) 2019-10-13T15:04:52.385 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:52.380+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.382764+0000) 2019-10-13T15:04:52.386 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:52.380+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.383154+0000) 2019-10-13T15:04:52.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:52.380+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.383903+0000) 2019-10-13T15:04:52.387 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:52.380+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.384637+0000) 2019-10-13T15:04:52.389 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:52.384+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.386687+0000) 2019-10-13T15:04:52.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:52.384+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.387532+0000) 2019-10-13T15:04:52.390 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:52.381+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.387953+0000) 2019-10-13T15:04:52.395 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:52.389+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:52.392897+0000) 2019-10-13T15:04:52.396 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:52.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218689 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:54.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:54.408+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:54.409153+0000) 2019-10-13T15:04:54.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:54.688+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:54.693272+0000) 2019-10-13T15:04:55.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:55.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.575916+0000) 2019-10-13T15:04:55.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:55.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:55.577330+0000) 2019-10-13T15:04:57.400 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:04:57.393+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.397139+0000) 2019-10-13T15:04:57.400 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:57.392+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.397689+0000) 2019-10-13T15:04:57.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:04:57.396+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.398328+0000) 2019-10-13T15:04:57.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:04:57.396+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.398399+0000) 2019-10-13T15:04:57.401 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:04:57.396+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.398829+0000) 2019-10-13T15:04:57.402 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:57.396+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.399438+0000) 2019-10-13T15:04:57.403 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:04:57.396+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.400452+0000) 2019-10-13T15:04:57.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:04:57.401+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:57.406936+0000) 2019-10-13T15:04:57.410 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:04:57.404+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218721 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:04:58.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:04:58.060+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:58.062405+0000) 2019-10-13T15:04:58.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:04:58.284+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:04:58.288219+0000) 2019-10-13T15:05:01.690 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:01.685+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:01.687937+0000) 2019-10-13T15:05:01.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:01.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:01.811305+0000) 2019-10-13T15:05:02.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:02.408+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.411047+0000) 2019-10-13T15:05:02.414 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:02.408+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.411813+0000) 2019-10-13T15:05:02.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:02.409+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.412315+0000) 2019-10-13T15:05:02.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:02.412+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.414174+0000) 2019-10-13T15:05:02.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:02.412+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.414595+0000) 2019-10-13T15:05:02.417 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:02.412+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.414956+0000) 2019-10-13T15:05:02.418 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:02.412+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.415196+0000) 2019-10-13T15:05:02.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:02.417+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:02.421835+0000) 2019-10-13T15:05:02.425 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:02.420+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:04.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:04.408+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:04.409358+0000) 2019-10-13T15:05:04.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:04.692+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:04.693462+0000) 2019-10-13T15:05:05.578 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:05.572+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.576111+0000) 2019-10-13T15:05:05.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:05.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:05.577549+0000) 2019-10-13T15:05:07.428 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:07.420+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.425245+0000) 2019-10-13T15:05:07.428 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:07.420+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.425370+0000) 2019-10-13T15:05:07.429 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:07.424+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.426005+0000) 2019-10-13T15:05:07.430 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:07.421+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.427422+0000) 2019-10-13T15:05:07.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:07.424+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.427460+0000) 2019-10-13T15:05:07.430 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:07.424+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.427941+0000) 2019-10-13T15:05:07.431 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:07.424+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.428582+0000) 2019-10-13T15:05:07.436 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:07.429+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:07.433779+0000) 2019-10-13T15:05:07.437 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:07.432+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218783 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:08.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:08.060+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:08.062577+0000) 2019-10-13T15:05:08.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:08.284+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:08.288416+0000) 2019-10-13T15:05:11.700 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:11.692+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:11.694499+0000) 2019-10-13T15:05:11.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:11.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:11.811512+0000) 2019-10-13T15:05:12.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:12.432+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.436791+0000) 2019-10-13T15:05:12.440 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:12.432+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.437604+0000) 2019-10-13T15:05:12.441 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:12.436+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.438310+0000) 2019-10-13T15:05:12.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:12.436+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.439795+0000) 2019-10-13T15:05:12.443 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:12.432+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.439947+0000) 2019-10-13T15:05:12.443 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:12.436+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.440167+0000) 2019-10-13T15:05:12.443 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:12.436+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.441194+0000) 2019-10-13T15:05:12.449 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:12.440+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:12.447228+0000) 2019-10-13T15:05:12.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:12.444+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218814 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:14.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:14.408+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:14.409567+0000) 2019-10-13T15:05:14.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:14.688+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:14.693695+0000) 2019-10-13T15:05:15.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:15.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.576334+0000) 2019-10-13T15:05:15.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:15.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:15.577745+0000) 2019-10-13T15:05:15.777 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:05:15.777 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:15.781 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:05:15.781 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:17.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:17.448+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.452625+0000) 2019-10-13T15:05:17.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:17.448+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.453651+0000) 2019-10-13T15:05:17.458 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:17.448+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.455052+0000) 2019-10-13T15:05:17.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:17.452+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.454710+0000) 2019-10-13T15:05:17.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:17.452+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.454407+0000) 2019-10-13T15:05:17.459 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:17.452+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.454369+0000) 2019-10-13T15:05:17.459 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:17.452+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.455384+0000) 2019-10-13T15:05:17.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:17.460+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:17.464398+0000) 2019-10-13T15:05:17.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:17.460+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218846 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:18.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:18.060+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:18.062782+0000) 2019-10-13T15:05:18.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:18.284+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:18.288595+0000) 2019-10-13T15:05:21.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:21.704+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:21.706165+0000) 2019-10-13T15:05:21.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:21.808+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:21.811734+0000) 2019-10-13T15:05:22.470 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:22.464+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.467798+0000) 2019-10-13T15:05:22.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:22.464+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.468481+0000) 2019-10-13T15:05:22.472 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:22.464+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.469742+0000) 2019-10-13T15:05:22.473 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:22.468+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.471026+0000) 2019-10-13T15:05:22.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:22.468+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.471284+0000) 2019-10-13T15:05:22.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:22.468+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.472849+0000) 2019-10-13T15:05:22.477 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:22.468+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.474345+0000) 2019-10-13T15:05:22.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:22.472+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:22.474617+0000) 2019-10-13T15:05:22.481 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:22.476+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:24.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:24.408+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:24.409782+0000) 2019-10-13T15:05:24.696 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:24.692+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:24.693929+0000) 2019-10-13T15:05:25.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:25.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.576559+0000) 2019-10-13T15:05:25.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:25.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:25.577942+0000) 2019-10-13T15:05:27.485 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:27.479+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.481998+0000) 2019-10-13T15:05:27.485 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:27.479+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.482547+0000) 2019-10-13T15:05:27.485 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:27.479+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.482934+0000) 2019-10-13T15:05:27.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:27.479+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.485176+0000) 2019-10-13T15:05:27.488 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:27.479+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.485679+0000) 2019-10-13T15:05:27.490 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:27.484+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.488156+0000) 2019-10-13T15:05:27.493 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:27.487+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.491218+0000) 2019-10-13T15:05:27.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:27.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218907 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:27.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:27.496+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:27.501739+0000) 2019-10-13T15:05:28.065 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:28.059+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:28.062994+0000) 2019-10-13T15:05:28.291 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:28.287+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:28.288770+0000) 2019-10-13T15:05:31.715 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:31.712+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:31.712764+0000) 2019-10-13T15:05:31.814 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:31.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:31.811936+0000) 2019-10-13T15:05:32.499 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:32.492+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.496333+0000) 2019-10-13T15:05:32.500 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:32.491+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.496656+0000) 2019-10-13T15:05:32.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:32.491+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.497127+0000) 2019-10-13T15:05:32.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:32.499+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.502834+0000) 2019-10-13T15:05:32.507 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:32.499+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.504547+0000) 2019-10-13T15:05:32.508 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:32.503+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.506287+0000) 2019-10-13T15:05:32.509 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:32.503+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.507297+0000) 2019-10-13T15:05:32.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:32.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218938 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:32.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:32.508+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:32.514483+0000) 2019-10-13T15:05:34.412 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:34.407+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:34.409976+0000) 2019-10-13T15:05:34.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:34.691+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:34.694155+0000) 2019-10-13T15:05:35.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:35.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.576788+0000) 2019-10-13T15:05:35.580 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:35.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:35.578154+0000) 2019-10-13T15:05:37.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:37.507+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.511214+0000) 2019-10-13T15:05:37.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:37.507+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.511731+0000) 2019-10-13T15:05:37.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:37.507+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.512229+0000) 2019-10-13T15:05:37.515 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:37.507+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.512954+0000) 2019-10-13T15:05:37.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:37.507+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.513507+0000) 2019-10-13T15:05:37.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:37.508+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.513916+0000) 2019-10-13T15:05:37.517 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:37.511+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.514428+0000) 2019-10-13T15:05:37.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:37.516+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:37.520888+0000) 2019-10-13T15:05:37.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:37.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218970 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:38.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:38.059+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:38.063174+0000) 2019-10-13T15:05:38.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:38.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:38.288964+0000) 2019-10-13T15:05:41.721 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:41.716+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:41.719122+0000) 2019-10-13T15:05:41.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:41.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:41.812173+0000) 2019-10-13T15:05:42.528 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:42.519+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.525022+0000) 2019-10-13T15:05:42.528 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:42.519+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.525669+0000) 2019-10-13T15:05:42.529 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:42.523+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.526298+0000) 2019-10-13T15:05:42.529 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:42.523+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.527034+0000) 2019-10-13T15:05:42.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:42.523+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.527367+0000) 2019-10-13T15:05:42.531 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:42.524+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.528508+0000) 2019-10-13T15:05:42.531 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:42.523+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.528136+0000) 2019-10-13T15:05:42.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:42.528+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:42.532739+0000) 2019-10-13T15:05:42.539 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:42.531+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:44.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:44.407+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:44.410173+0000) 2019-10-13T15:05:44.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:44.691+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:44.694389+0000) 2019-10-13T15:05:45.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:45.571+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.577013+0000) 2019-10-13T15:05:45.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:45.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:45.578393+0000) 2019-10-13T15:05:45.895 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:05:45.895 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:45.899 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:05:45.899 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:05:47.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:47.535+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.539666+0000) 2019-10-13T15:05:47.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:47.535+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.540126+0000) 2019-10-13T15:05:47.543 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:47.535+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.540659+0000) 2019-10-13T15:05:47.544 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:47.535+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.540950+0000) 2019-10-13T15:05:47.545 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:47.536+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.542592+0000) 2019-10-13T15:05:47.545 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:47.539+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.542855+0000) 2019-10-13T15:05:47.546 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:47.539+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.543058+0000) 2019-10-13T15:05:47.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:47.540+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:47.547744+0000) 2019-10-13T15:05:47.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:47.547+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219031 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:48.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:48.059+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:48.063357+0000) 2019-10-13T15:05:48.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:48.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:48.289138+0000) 2019-10-13T15:05:51.728 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:51.724+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:51.725544+0000) 2019-10-13T15:05:51.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:51.811+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:51.812391+0000) 2019-10-13T15:05:52.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:52.547+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.553568+0000) 2019-10-13T15:05:52.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:52.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.555064+0000) 2019-10-13T15:05:52.558 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:52.551+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.555885+0000) 2019-10-13T15:05:52.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:52.551+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.556500+0000) 2019-10-13T15:05:52.559 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:52.551+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.557021+0000) 2019-10-13T15:05:52.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:52.551+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.557426+0000) 2019-10-13T15:05:52.561 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:52.555+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.558550+0000) 2019-10-13T15:05:52.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:52.559+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219062 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:52.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:52.560+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:52.564773+0000) 2019-10-13T15:05:54.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:54.407+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:54.410378+0000) 2019-10-13T15:05:54.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:54.691+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:54.694619+0000) 2019-10-13T15:05:55.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:55.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.577205+0000) 2019-10-13T15:05:55.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:55.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:55.578591+0000) 2019-10-13T15:05:57.569 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:57.563+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.566785+0000) 2019-10-13T15:05:57.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:05:57.563+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.567415+0000) 2019-10-13T15:05:57.570 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:57.563+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.567828+0000) 2019-10-13T15:05:57.571 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:05:57.563+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.568310+0000) 2019-10-13T15:05:57.571 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:05:57.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.569101+0000) 2019-10-13T15:05:57.572 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:05:57.563+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.569280+0000) 2019-10-13T15:05:57.572 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:05:57.563+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.569693+0000) 2019-10-13T15:05:57.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:05:57.568+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:57.575859+0000) 2019-10-13T15:05:57.579 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:05:57.571+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219094 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:05:58.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:05:58.059+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:58.063542+0000) 2019-10-13T15:05:58.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:05:58.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:05:58.289313+0000) 2019-10-13T15:06:01.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:01.724+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:01.731950+0000) 2019-10-13T15:06:01.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:01.811+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:01.812590+0000) 2019-10-13T15:06:02.582 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:02.575+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.579983+0000) 2019-10-13T15:06:02.583 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:02.575+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.580977+0000) 2019-10-13T15:06:02.584 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:02.575+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.581125+0000) 2019-10-13T15:06:02.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:02.576+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.582883+0000) 2019-10-13T15:06:02.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:02.579+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.583124+0000) 2019-10-13T15:06:02.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:02.579+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.583998+0000) 2019-10-13T15:06:02.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:02.579+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.584472+0000) 2019-10-13T15:06:02.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:02.584+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:02.589828+0000) 2019-10-13T15:06:02.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:02.587+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:04.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:04.407+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:04.410585+0000) 2019-10-13T15:06:04.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:04.691+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:04.694827+0000) 2019-10-13T15:06:05.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:05.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.577401+0000) 2019-10-13T15:06:05.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:05.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:05.578784+0000) 2019-10-13T15:06:07.596 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:07.587+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.593144+0000) 2019-10-13T15:06:07.596 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:07.587+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.593714+0000) 2019-10-13T15:06:07.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:07.591+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.594264+0000) 2019-10-13T15:06:07.597 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:07.591+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.595052+0000) 2019-10-13T15:06:07.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:07.588+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.596055+0000) 2019-10-13T15:06:07.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:07.591+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.596509+0000) 2019-10-13T15:06:07.599 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:07.591+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.597155+0000) 2019-10-13T15:06:07.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:07.596+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:07.603998+0000) 2019-10-13T15:06:07.607 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:07.599+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219155 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:08.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:08.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:08.063754+0000) 2019-10-13T15:06:08.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:08.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:08.289487+0000) 2019-10-13T15:06:11.739 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:11.736+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:11.737182+0000) 2019-10-13T15:06:11.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:11.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:11.812788+0000) 2019-10-13T15:06:12.610 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:12.603+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.607654+0000) 2019-10-13T15:06:12.611 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:12.603+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.608241+0000) 2019-10-13T15:06:12.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:12.603+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.609587+0000) 2019-10-13T15:06:12.612 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:12.607+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.609731+0000) 2019-10-13T15:06:12.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:12.607+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.610215+0000) 2019-10-13T15:06:12.613 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:12.607+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.610618+0000) 2019-10-13T15:06:12.613 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:12.608+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.611237+0000) 2019-10-13T15:06:12.621 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:12.615+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:12.622 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:12.616+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:12.619845+0000) 2019-10-13T15:06:14.413 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:14.407+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:14.410788+0000) 2019-10-13T15:06:14.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:14.691+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:14.695032+0000) 2019-10-13T15:06:15.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:15.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.577600+0000) 2019-10-13T15:06:15.581 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:15.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:15.578982+0000) 2019-10-13T15:06:16.000 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:06:16.000 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:16.004 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:06:16.004 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:17.624 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:17.619+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.621354+0000) 2019-10-13T15:06:17.625 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:17.619+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.622944+0000) 2019-10-13T15:06:17.628 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:17.619+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.625161+0000) 2019-10-13T15:06:17.628 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:17.623+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.626046+0000) 2019-10-13T15:06:17.630 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:17.623+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.627415+0000) 2019-10-13T15:06:17.631 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:17.623+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.628571+0000) 2019-10-13T15:06:17.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:17.623+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.629396+0000) 2019-10-13T15:06:17.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:17.627+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:17.629481+0000) 2019-10-13T15:06:17.638 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:17.627+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219216 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:18.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:18.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:18.063954+0000) 2019-10-13T15:06:18.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:18.283+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:18.289664+0000) 2019-10-13T15:06:21.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:21.739+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:21.743586+0000) 2019-10-13T15:06:21.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:21.807+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:21.812994+0000) 2019-10-13T15:06:22.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:22.631+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.635396+0000) 2019-10-13T15:06:22.640 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:22.635+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.637792+0000) 2019-10-13T15:06:22.641 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:22.635+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.638607+0000) 2019-10-13T15:06:22.642 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:22.635+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.639581+0000) 2019-10-13T15:06:22.643 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:22.635+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.639971+0000) 2019-10-13T15:06:22.643 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:22.635+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.640110+0000) 2019-10-13T15:06:22.643 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:22.635+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.640774+0000) 2019-10-13T15:06:22.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:22.639+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:22.642364+0000) 2019-10-13T15:06:22.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:22.639+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219248 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:24.429 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:24.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:24.411006+0000) 2019-10-13T15:06:24.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:24.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:24.695247+0000) 2019-10-13T15:06:25.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:25.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.577826+0000) 2019-10-13T15:06:25.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:25.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:25.579178+0000) 2019-10-13T15:06:27.652 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:27.642+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.649005+0000) 2019-10-13T15:06:27.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:27.647+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.650137+0000) 2019-10-13T15:06:27.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:27.646+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.650372+0000) 2019-10-13T15:06:27.655 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:27.646+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.652242+0000) 2019-10-13T15:06:27.658 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:27.650+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.655982+0000) 2019-10-13T15:06:27.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:27.655+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.656267+0000) 2019-10-13T15:06:27.660 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:27.654+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.658171+0000) 2019-10-13T15:06:27.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:27.654+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:27.659147+0000) 2019-10-13T15:06:27.663 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:27.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219277 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:28.066 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:28.062+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:28.064133+0000) 2019-10-13T15:06:28.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:28.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:28.289840+0000) 2019-10-13T15:06:31.752 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:31.747+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:31.749720+0000) 2019-10-13T15:06:31.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:31.810+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:31.813224+0000) 2019-10-13T15:06:32.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:32.658+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.665181+0000) 2019-10-13T15:06:32.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:32.662+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.666327+0000) 2019-10-13T15:06:32.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:32.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.667449+0000) 2019-10-13T15:06:32.670 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:32.662+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.666807+0000) 2019-10-13T15:06:32.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:32.662+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.668404+0000) 2019-10-13T15:06:32.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:32.662+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.668611+0000) 2019-10-13T15:06:32.671 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:32.662+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.669197+0000) 2019-10-13T15:06:32.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:32.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219308 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:32.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:32.675+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:32.677139+0000) 2019-10-13T15:06:34.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:34.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:34.411253+0000) 2019-10-13T15:06:34.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:34.694+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:34.695425+0000) 2019-10-13T15:06:35.580 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:35.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.578022+0000) 2019-10-13T15:06:35.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:35.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:35.579373+0000) 2019-10-13T15:06:37.681 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:37.674+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.678884+0000) 2019-10-13T15:06:37.682 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:37.674+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.679397+0000) 2019-10-13T15:06:37.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:37.674+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.680310+0000) 2019-10-13T15:06:37.684 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:37.679+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.681538+0000) 2019-10-13T15:06:37.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:37.674+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.681392+0000) 2019-10-13T15:06:37.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:37.674+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.681661+0000) 2019-10-13T15:06:37.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:37.678+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.682124+0000) 2019-10-13T15:06:37.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:37.683+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:37.686613+0000) 2019-10-13T15:06:37.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:37.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219339 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:38.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:38.062+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:38.064316+0000) 2019-10-13T15:06:38.292 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:38.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:38.290020+0000) 2019-10-13T15:06:41.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:41.755+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:41.756151+0000) 2019-10-13T15:06:41.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:41.806+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:41.813447+0000) 2019-10-13T15:06:42.696 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:42.686+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.693491+0000) 2019-10-13T15:06:42.697 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:42.690+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.694195+0000) 2019-10-13T15:06:42.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:42.690+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.694468+0000) 2019-10-13T15:06:42.698 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:42.690+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.695504+0000) 2019-10-13T15:06:42.698 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:42.690+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.695938+0000) 2019-10-13T15:06:42.699 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:42.695+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.696521+0000) 2019-10-13T15:06:42.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:42.690+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.697070+0000) 2019-10-13T15:06:42.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:42.699+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:42.702913+0000) 2019-10-13T15:06:42.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:42.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219370 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:44.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:44.410+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:44.411463+0000) 2019-10-13T15:06:44.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:44.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:44.695684+0000) 2019-10-13T15:06:45.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:45.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.578224+0000) 2019-10-13T15:06:45.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:45.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:45.579529+0000) 2019-10-13T15:06:46.122 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:06:46.123 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:46.127 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:06:46.127 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:06:47.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:47.702+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.708367+0000) 2019-10-13T15:06:47.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:47.702+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.709402+0000) 2019-10-13T15:06:47.713 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:47.707+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.710351+0000) 2019-10-13T15:06:47.714 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:47.706+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.710265+0000) 2019-10-13T15:06:47.714 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:47.706+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.710516+0000) 2019-10-13T15:06:47.714 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:47.706+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.710922+0000) 2019-10-13T15:06:47.715 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:47.706+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.711432+0000) 2019-10-13T15:06:47.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:47.715+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:47.716482+0000) 2019-10-13T15:06:47.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:47.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219400 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:48.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:48.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:48.064497+0000) 2019-10-13T15:06:48.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:48.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:48.290227+0000) 2019-10-13T15:06:51.765 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:51.759+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:51.762616+0000) 2019-10-13T15:06:51.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:51.806+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:51.813643+0000) 2019-10-13T15:06:52.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:52.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.722332+0000) 2019-10-13T15:06:52.725 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:52.718+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.722773+0000) 2019-10-13T15:06:52.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:52.718+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.723620+0000) 2019-10-13T15:06:52.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:52.718+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.724220+0000) 2019-10-13T15:06:52.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:52.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.724817+0000) 2019-10-13T15:06:52.731 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:52.722+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.726865+0000) 2019-10-13T15:06:52.732 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:52.727+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.729111+0000) 2019-10-13T15:06:52.733 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:52.727+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:52.731227+0000) 2019-10-13T15:06:52.736 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:52.726+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219430 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:54.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:54.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:54.411681+0000) 2019-10-13T15:06:54.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:54.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:54.695901+0000) 2019-10-13T15:06:55.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:55.574+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.578434+0000) 2019-10-13T15:06:55.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:55.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:55.579721+0000) 2019-10-13T15:06:57.742 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:57.734+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.738883+0000) 2019-10-13T15:06:57.743 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:06:57.735+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.739259+0000) 2019-10-13T15:06:57.743 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:06:57.734+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.739770+0000) 2019-10-13T15:06:57.743 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:57.734+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.740373+0000) 2019-10-13T15:06:57.744 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:06:57.738+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.742179+0000) 2019-10-13T15:06:57.745 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:06:57.738+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.743020+0000) 2019-10-13T15:06:57.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:06:57.738+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.744973+0000) 2019-10-13T15:06:57.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:06:57.742+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:06:57.753 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:06:57.747+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:57.748620+0000) 2019-10-13T15:06:58.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:06:58.062+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:58.064679+0000) 2019-10-13T15:06:58.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:06:58.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:06:58.290421+0000) 2019-10-13T15:07:01.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:01.767+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:01.771891+0000) 2019-10-13T15:07:01.816 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:01.810+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:01.813868+0000) 2019-10-13T15:07:02.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:02.746+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.751794+0000) 2019-10-13T15:07:02.755 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:02.746+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.752369+0000) 2019-10-13T15:07:02.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:02.751+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.753281+0000) 2019-10-13T15:07:02.756 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:02.746+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.753517+0000) 2019-10-13T15:07:02.756 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:02.750+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.753958+0000) 2019-10-13T15:07:02.757 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:02.750+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.754987+0000) 2019-10-13T15:07:02.759 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:02.750+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.756533+0000) 2019-10-13T15:07:02.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:02.759+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:02.760531+0000) 2019-10-13T15:07:02.765 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:02.758+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219491 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:04.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:04.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:04.411885+0000) 2019-10-13T15:07:04.698 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:04.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:04.696102+0000) 2019-10-13T15:07:05.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:05.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.578645+0000) 2019-10-13T15:07:05.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:05.575+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:05.579919+0000) 2019-10-13T15:07:07.768 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:07.758+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.765554+0000) 2019-10-13T15:07:07.770 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:07.762+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.767321+0000) 2019-10-13T15:07:07.770 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:07.762+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.767539+0000) 2019-10-13T15:07:07.770 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:07.763+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.767923+0000) 2019-10-13T15:07:07.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:07.762+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.768419+0000) 2019-10-13T15:07:07.774 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:07.766+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.771994+0000) 2019-10-13T15:07:07.775 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:07.771+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.772730+0000) 2019-10-13T15:07:07.776 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:07.770+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:07.774073+0000) 2019-10-13T15:07:07.780 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:07.774+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219522 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:08.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:08.058+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:08.064869+0000) 2019-10-13T15:07:08.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:08.286+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:08.290640+0000) 2019-10-13T15:07:11.787 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:11.775+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:11.778183+0000) 2019-10-13T15:07:11.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:11.810+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:11.814071+0000) 2019-10-13T15:07:12.784 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:12.778+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.781931+0000) 2019-10-13T15:07:12.785 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:12.779+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.782279+0000) 2019-10-13T15:07:12.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:12.778+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.782539+0000) 2019-10-13T15:07:12.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:12.778+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.783381+0000) 2019-10-13T15:07:12.786 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:12.778+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.783856+0000) 2019-10-13T15:07:12.787 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:12.778+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.784347+0000) 2019-10-13T15:07:12.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:12.778+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.784787+0000) 2019-10-13T15:07:12.795 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:12.786+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219551 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:12.796 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:12.791+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:12.793686+0000) 2019-10-13T15:07:14.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:14.406+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:14.412084+0000) 2019-10-13T15:07:14.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:14.690+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:14.696268+0000) 2019-10-13T15:07:15.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:15.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.578849+0000) 2019-10-13T15:07:15.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:15.579+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:15.580127+0000) 2019-10-13T15:07:16.236 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:07:16.236 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:16.241 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:07:16.241 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:17.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:17.790+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.795750+0000) 2019-10-13T15:07:17.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:17.790+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.796435+0000) 2019-10-13T15:07:17.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:17.790+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.796252+0000) 2019-10-13T15:07:17.800 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:17.795+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.796938+0000) 2019-10-13T15:07:17.800 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:17.794+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.797819+0000) 2019-10-13T15:07:17.800 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:17.794+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.798200+0000) 2019-10-13T15:07:17.804 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:17.798+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.801773+0000) 2019-10-13T15:07:17.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:17.799+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:17.802613+0000) 2019-10-13T15:07:17.808 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:17.802+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:18.067 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:18.062+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:18.065029+0000) 2019-10-13T15:07:18.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:18.290+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:18.290779+0000) 2019-10-13T15:07:21.788 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:21.782+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:21.784738+0000) 2019-10-13T15:07:21.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:21.809+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:21.814289+0000) 2019-10-13T15:07:22.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:22.805+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.809238+0000) 2019-10-13T15:07:22.812 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:22.805+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.810105+0000) 2019-10-13T15:07:22.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:22.805+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.811511+0000) 2019-10-13T15:07:22.814 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:22.810+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.812212+0000) 2019-10-13T15:07:22.815 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:22.805+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.812435+0000) 2019-10-13T15:07:22.817 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:22.809+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.814435+0000) 2019-10-13T15:07:22.817 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:22.809+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.814970+0000) 2019-10-13T15:07:22.821 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:22.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:22.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:22.814+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:22.819274+0000) 2019-10-13T15:07:24.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:24.409+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:24.412263+0000) 2019-10-13T15:07:24.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:24.689+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:24.696472+0000) 2019-10-13T15:07:25.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:25.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.579046+0000) 2019-10-13T15:07:25.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:25.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:25.580322+0000) 2019-10-13T15:07:27.825 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:27.817+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.821709+0000) 2019-10-13T15:07:27.825 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:27.817+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.821976+0000) 2019-10-13T15:07:27.826 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:27.818+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.822992+0000) 2019-10-13T15:07:27.826 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:27.817+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.823691+0000) 2019-10-13T15:07:27.827 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:27.817+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.823964+0000) 2019-10-13T15:07:27.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:27.817+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.824415+0000) 2019-10-13T15:07:27.828 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:27.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.825431+0000) 2019-10-13T15:07:27.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:27.826+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:27.830983+0000) 2019-10-13T15:07:27.835 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:27.825+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219644 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:28.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:28.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:28.065211+0000) 2019-10-13T15:07:28.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:28.289+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:28.290968+0000) 2019-10-13T15:07:31.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:31.790+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:31.791147+0000) 2019-10-13T15:07:31.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:31.813+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:31.814489+0000) 2019-10-13T15:07:32.839 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:32.829+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.836192+0000) 2019-10-13T15:07:32.839 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:32.829+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.836432+0000) 2019-10-13T15:07:32.839 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:32.829+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.837141+0000) 2019-10-13T15:07:32.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:32.829+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.837505+0000) 2019-10-13T15:07:32.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:32.829+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.837688+0000) 2019-10-13T15:07:32.841 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:32.834+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.838175+0000) 2019-10-13T15:07:32.841 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:32.833+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.839136+0000) 2019-10-13T15:07:32.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:32.842+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:32.844781+0000) 2019-10-13T15:07:32.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:32.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219673 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:34.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:34.409+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:34.412485+0000) 2019-10-13T15:07:34.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:34.693+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:34.696674+0000) 2019-10-13T15:07:35.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:35.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.579239+0000) 2019-10-13T15:07:35.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:35.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:35.580563+0000) 2019-10-13T15:07:37.851 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:37.845+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.848995+0000) 2019-10-13T15:07:37.852 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:37.845+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.849417+0000) 2019-10-13T15:07:37.854 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:37.849+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.851716+0000) 2019-10-13T15:07:37.855 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:37.849+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.852015+0000) 2019-10-13T15:07:37.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:37.849+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.852303+0000) 2019-10-13T15:07:37.857 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:37.850+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.855138+0000) 2019-10-13T15:07:37.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:37.853+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.857002+0000) 2019-10-13T15:07:37.863 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:37.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219706 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:37.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:37.862+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:37.865412+0000) 2019-10-13T15:07:38.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:38.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:38.065345+0000) 2019-10-13T15:07:38.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:38.289+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:38.291175+0000) 2019-10-13T15:07:41.801 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:41.794+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:41.798383+0000) 2019-10-13T15:07:41.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:41.813+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:41.814701+0000) 2019-10-13T15:07:42.867 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:42.861+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.864575+0000) 2019-10-13T15:07:42.867 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:42.861+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.864849+0000) 2019-10-13T15:07:42.868 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:42.861+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.865019+0000) 2019-10-13T15:07:42.868 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:42.862+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.865428+0000) 2019-10-13T15:07:42.869 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:42.865+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.866486+0000) 2019-10-13T15:07:42.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:42.865+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.866730+0000) 2019-10-13T15:07:42.870 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:42.865+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.867936+0000) 2019-10-13T15:07:42.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:42.870+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:42.874699+0000) 2019-10-13T15:07:42.879 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:42.873+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:44.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:44.409+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:44.412712+0000) 2019-10-13T15:07:44.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:44.693+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:44.696871+0000) 2019-10-13T15:07:45.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:45.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.579431+0000) 2019-10-13T15:07:45.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:45.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:45.580756+0000) 2019-10-13T15:07:46.349 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:07:46.349 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:46.353 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:07:46.353 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:07:47.889 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:47.877+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.881063+0000) 2019-10-13T15:07:47.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:47.877+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.881494+0000) 2019-10-13T15:07:47.891 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:47.881+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.882014+0000) 2019-10-13T15:07:47.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:47.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.882686+0000) 2019-10-13T15:07:47.894 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:47.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.884952+0000) 2019-10-13T15:07:47.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:47.881+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.885653+0000) 2019-10-13T15:07:47.896 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:47.885+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.886360+0000) 2019-10-13T15:07:47.901 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:47.886+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:47.889783+0000) 2019-10-13T15:07:47.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:47.889+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:48.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:48.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:48.065485+0000) 2019-10-13T15:07:48.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:48.289+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:48.291350+0000) 2019-10-13T15:07:51.808 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:51.802+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:51.805438+0000) 2019-10-13T15:07:51.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:51.813+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:51.814877+0000) 2019-10-13T15:07:52.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:52.889+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.893052+0000) 2019-10-13T15:07:52.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:52.893+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.893838+0000) 2019-10-13T15:07:52.896 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:52.893+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.893860+0000) 2019-10-13T15:07:52.897 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:52.890+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.895127+0000) 2019-10-13T15:07:52.898 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:52.893+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.895844+0000) 2019-10-13T15:07:52.901 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:52.897+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.898946+0000) 2019-10-13T15:07:52.902 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:52.897+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.899942+0000) 2019-10-13T15:07:52.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:52.898+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:52.901833+0000) 2019-10-13T15:07:52.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:52.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219795 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:54.415 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:54.409+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:54.412911+0000) 2019-10-13T15:07:54.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:54.693+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:54.697106+0000) 2019-10-13T15:07:55.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:55.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.579658+0000) 2019-10-13T15:07:55.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:55.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:55.580962+0000) 2019-10-13T15:07:57.909 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:57.905+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.906867+0000) 2019-10-13T15:07:57.910 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:07:57.905+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.907758+0000) 2019-10-13T15:07:57.913 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:07:57.905+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.909575+0000) 2019-10-13T15:07:57.913 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:07:57.909+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.910270+0000) 2019-10-13T15:07:57.913 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:07:57.909+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.910787+0000) 2019-10-13T15:07:57.914 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:07:57.906+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.911248+0000) 2019-10-13T15:07:57.914 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:57.909+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.911492+0000) 2019-10-13T15:07:57.917 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:07:57.910+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:57.914525+0000) 2019-10-13T15:07:57.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:07:57.917+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219828 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:07:58.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:07:58.065+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:58.065665+0000) 2019-10-13T15:07:58.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:07:58.289+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:07:58.291506+0000) 2019-10-13T15:08:01.815 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:01.810+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:01.812595+0000) 2019-10-13T15:08:01.817 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:01.813+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:01.815099+0000) 2019-10-13T15:08:02.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:02.921+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.922184+0000) 2019-10-13T15:08:02.940 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:02.921+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.922652+0000) 2019-10-13T15:08:02.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:02.921+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.922879+0000) 2019-10-13T15:08:02.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:02.921+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.923340+0000) 2019-10-13T15:08:02.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:02.921+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.923738+0000) 2019-10-13T15:08:02.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:02.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.925143+0000) 2019-10-13T15:08:02.942 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:02.921+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.924626+0000) 2019-10-13T15:08:02.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:02.929+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219857 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:02.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:02.930+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:02.933482+0000) 2019-10-13T15:08:04.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:04.409+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:04.413118+0000) 2019-10-13T15:08:04.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:04.693+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:04.697353+0000) 2019-10-13T15:08:05.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:05.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.579879+0000) 2019-10-13T15:08:05.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:05.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:05.581157+0000) 2019-10-13T15:08:07.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:07.933+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.935936+0000) 2019-10-13T15:08:07.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:07.933+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.936219+0000) 2019-10-13T15:08:07.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:07.933+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.936832+0000) 2019-10-13T15:08:07.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:07.933+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.937052+0000) 2019-10-13T15:08:07.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:07.937+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.938010+0000) 2019-10-13T15:08:07.942 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:07.934+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.939672+0000) 2019-10-13T15:08:07.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:07.937+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.939908+0000) 2019-10-13T15:08:07.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:07.942+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:07.946657+0000) 2019-10-13T15:08:07.949 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:07.945+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219888 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:08.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:08.065+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:08.065847+0000) 2019-10-13T15:08:08.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:08.289+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:08.291699+0000) 2019-10-13T15:08:11.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:11.813+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:11.815302+0000) 2019-10-13T15:08:11.822 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:11.818+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:11.819615+0000) 2019-10-13T15:08:12.953 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:12.949+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.950396+0000) 2019-10-13T15:08:12.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:12.949+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.951237+0000) 2019-10-13T15:08:12.955 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:12.949+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.952368+0000) 2019-10-13T15:08:12.955 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:12.949+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.952847+0000) 2019-10-13T15:08:12.955 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:12.949+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.953131+0000) 2019-10-13T15:08:12.956 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:12.950+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.953506+0000) 2019-10-13T15:08:12.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:12.949+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.953714+0000) 2019-10-13T15:08:12.960 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:12.954+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:12.958351+0000) 2019-10-13T15:08:12.963 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:12.957+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219917 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:14.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:14.413+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:14.413341+0000) 2019-10-13T15:08:14.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:14.693+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:14.697614+0000) 2019-10-13T15:08:15.582 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:15.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.580096+0000) 2019-10-13T15:08:15.583 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:15.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:15.581349+0000) 2019-10-13T15:08:16.415 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:08:16.415 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:16.419 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:08:16.419 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:17.967 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:17.960+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.964360+0000) 2019-10-13T15:08:17.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:17.960+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.964740+0000) 2019-10-13T15:08:17.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:17.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.965406+0000) 2019-10-13T15:08:17.968 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:17.960+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.965634+0000) 2019-10-13T15:08:17.970 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:17.962+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.967143+0000) 2019-10-13T15:08:17.970 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:17.964+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.966402+0000) 2019-10-13T15:08:17.970 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:17.964+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.967620+0000) 2019-10-13T15:08:17.975 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:17.970+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:17.973385+0000) 2019-10-13T15:08:17.978 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:17.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:18.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:18.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:18.066044+0000) 2019-10-13T15:08:18.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:18.288+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:18.291915+0000) 2019-10-13T15:08:21.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:21.812+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:21.815520+0000) 2019-10-13T15:08:21.828 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:21.822+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:21.825893+0000) 2019-10-13T15:08:22.981 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:22.976+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.978647+0000) 2019-10-13T15:08:22.982 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:22.974+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.979467+0000) 2019-10-13T15:08:22.983 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:22.976+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.980450+0000) 2019-10-13T15:08:22.983 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:22.976+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.980734+0000) 2019-10-13T15:08:22.984 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:22.976+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.981182+0000) 2019-10-13T15:08:22.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:22.976+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.981686+0000) 2019-10-13T15:08:22.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:22.980+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.982371+0000) 2019-10-13T15:08:22.992 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:22.982+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:22.985932+0000) 2019-10-13T15:08:22.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:22.988+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219979 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:24.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:24.408+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:24.413582+0000) 2019-10-13T15:08:24.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:24.696+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:24.697829+0000) 2019-10-13T15:08:25.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:25.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.580248+0000) 2019-10-13T15:08:25.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:25.578+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:25.581548+0000) 2019-10-13T15:08:27.996 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:27.992+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.993914+0000) 2019-10-13T15:08:27.998 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:27.992+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.995616+0000) 2019-10-13T15:08:27.999 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:27.989+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.996051+0000) 2019-10-13T15:08:27.999 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:27.992+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.997051+0000) 2019-10-13T15:08:28.000 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:27.996+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.997951+0000) 2019-10-13T15:08:28.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:27.996+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.998244+0000) 2019-10-13T15:08:28.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:27.996+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:27.999722+0000) 2019-10-13T15:08:28.006 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:27.997+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:28.003887+0000) 2019-10-13T15:08:28.007 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:28.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220010 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:28.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:28.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:28.066240+0000) 2019-10-13T15:08:28.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:28.288+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:28.292090+0000) 2019-10-13T15:08:31.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:31.812+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:31.815736+0000) 2019-10-13T15:08:31.835 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:31.829+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:31.833028+0000) 2019-10-13T15:08:33.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:33.004+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.008074+0000) 2019-10-13T15:08:33.012 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:33.008+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.010057+0000) 2019-10-13T15:08:33.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:33.008+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.010655+0000) 2019-10-13T15:08:33.013 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:33.005+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.011092+0000) 2019-10-13T15:08:33.014 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:33.008+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.011329+0000) 2019-10-13T15:08:33.014 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:33.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.011958+0000) 2019-10-13T15:08:33.015 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:33.008+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.012363+0000) 2019-10-13T15:08:33.021 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:33.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220039 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:33.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:33.017+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:33.021152+0000) 2019-10-13T15:08:34.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:34.412+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:34.413824+0000) 2019-10-13T15:08:34.700 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:34.696+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:34.698061+0000) 2019-10-13T15:08:35.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:35.576+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.580439+0000) 2019-10-13T15:08:35.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:35.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:35.581742+0000) 2019-10-13T15:08:38.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:38.020+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.021962+0000) 2019-10-13T15:08:38.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:38.020+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.023687+0000) 2019-10-13T15:08:38.027 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:38.020+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.025008+0000) 2019-10-13T15:08:38.028 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:38.020+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.025412+0000) 2019-10-13T15:08:38.028 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:38.024+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.026160+0000) 2019-10-13T15:08:38.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:38.021+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.026713+0000) 2019-10-13T15:08:38.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:38.024+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.026994+0000) 2019-10-13T15:08:38.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:38.029+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.033082+0000) 2019-10-13T15:08:38.036 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:38.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220072 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:38.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:38.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.066439+0000) 2019-10-13T15:08:38.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:38.288+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:38.292287+0000) 2019-10-13T15:08:41.818 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:41.812+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:41.815940+0000) 2019-10-13T15:08:41.842 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:41.837+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:41.840200+0000) 2019-10-13T15:08:43.039 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:43.032+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.035957+0000) 2019-10-13T15:08:43.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:43.032+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.036740+0000) 2019-10-13T15:08:43.040 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:43.032+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.037251+0000) 2019-10-13T15:08:43.041 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:43.036+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.038127+0000) 2019-10-13T15:08:43.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:43.036+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.038945+0000) 2019-10-13T15:08:43.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:43.036+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.039079+0000) 2019-10-13T15:08:43.042 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:43.033+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.039575+0000) 2019-10-13T15:08:43.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:43.041+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:43.046316+0000) 2019-10-13T15:08:43.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:43.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220100 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:44.416 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:44.412+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:44.414036+0000) 2019-10-13T15:08:44.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:44.696+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:44.698295+0000) 2019-10-13T15:08:45.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:45.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.580641+0000) 2019-10-13T15:08:45.584 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:45.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:45.581932+0000) 2019-10-13T15:08:46.472 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:08:46.472 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:46.476 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:08:46.476 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:08:48.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:48.048+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.050252+0000) 2019-10-13T15:08:48.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:48.048+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.050754+0000) 2019-10-13T15:08:48.054 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:48.048+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.051190+0000) 2019-10-13T15:08:48.054 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:48.048+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.051990+0000) 2019-10-13T15:08:48.055 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:48.049+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.052312+0000) 2019-10-13T15:08:48.056 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:48.052+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.053962+0000) 2019-10-13T15:08:48.057 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:48.052+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.054340+0000) 2019-10-13T15:08:48.063 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:48.057+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.060865+0000) 2019-10-13T15:08:48.064 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:48.060+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220132 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:48.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:48.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.066572+0000) 2019-10-13T15:08:48.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:48.288+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:48.292462+0000) 2019-10-13T15:08:51.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:51.812+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:51.816173+0000) 2019-10-13T15:08:51.850 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:51.845+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:51.847259+0000) 2019-10-13T15:08:53.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:53.060+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.065102+0000) 2019-10-13T15:08:53.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:53.064+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.067319+0000) 2019-10-13T15:08:53.072 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:53.064+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.069618+0000) 2019-10-13T15:08:53.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:53.068+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.069909+0000) 2019-10-13T15:08:53.073 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:53.065+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.070245+0000) 2019-10-13T15:08:53.073 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:53.068+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.070872+0000) 2019-10-13T15:08:53.073 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:53.068+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.071339+0000) 2019-10-13T15:08:53.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:53.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220160 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:53.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:53.073+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:53.078004+0000) 2019-10-13T15:08:54.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:54.412+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:54.414267+0000) 2019-10-13T15:08:54.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:54.696+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:54.698496+0000) 2019-10-13T15:08:55.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:55.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.580856+0000) 2019-10-13T15:08:55.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:55.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:55.582157+0000) 2019-10-13T15:08:58.069 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:58.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.066757+0000) 2019-10-13T15:08:58.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:08:58.076+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.078972+0000) 2019-10-13T15:08:58.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:08:58.076+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.079425+0000) 2019-10-13T15:08:58.082 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:08:58.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.079817+0000) 2019-10-13T15:08:58.083 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:58.076+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.081092+0000) 2019-10-13T15:08:58.084 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:08:58.076+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.081658+0000) 2019-10-13T15:08:58.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:08:58.080+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.083947+0000) 2019-10-13T15:08:58.089 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:08:58.084+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.086538+0000) 2019-10-13T15:08:58.093 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:08:58.088+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220194 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:08:58.096 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:08:58.089+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.094249+0000) 2019-10-13T15:08:58.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:08:58.288+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:08:58.292669+0000) 2019-10-13T15:09:01.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:01.812+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:01.816383+0000) 2019-10-13T15:09:01.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:01.853+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:01.854697+0000) 2019-10-13T15:09:03.097 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:03.092+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.094217+0000) 2019-10-13T15:09:03.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:03.092+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.094649+0000) 2019-10-13T15:09:03.098 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:03.089+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.095208+0000) 2019-10-13T15:09:03.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:03.092+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.095555+0000) 2019-10-13T15:09:03.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:03.092+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.096170+0000) 2019-10-13T15:09:03.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:03.092+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.096903+0000) 2019-10-13T15:09:03.100 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:03.092+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.097123+0000) 2019-10-13T15:09:03.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:03.100+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220222 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:03.108 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:03.101+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:03.106416+0000) 2019-10-13T15:09:04.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:04.412+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:04.414502+0000) 2019-10-13T15:09:04.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:04.696+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:04.698715+0000) 2019-10-13T15:09:05.583 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:05.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.581050+0000) 2019-10-13T15:09:05.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:05.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:05.582384+0000) 2019-10-13T15:09:08.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:08.064+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.066982+0000) 2019-10-13T15:09:08.111 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:08.104+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.108540+0000) 2019-10-13T15:09:08.112 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:08.104+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.108898+0000) 2019-10-13T15:09:08.112 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:08.108+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.109860+0000) 2019-10-13T15:09:08.113 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:08.108+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.110212+0000) 2019-10-13T15:09:08.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:08.105+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.110736+0000) 2019-10-13T15:09:08.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:08.108+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.111057+0000) 2019-10-13T15:09:08.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:08.108+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.111871+0000) 2019-10-13T15:09:08.121 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:08.113+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.118632+0000) 2019-10-13T15:09:08.122 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:08.116+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220254 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:08.295 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:08.292+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:08.292855+0000) 2019-10-13T15:09:11.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:11.816+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:11.816599+0000) 2019-10-13T15:09:11.864 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:11.857+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:11.861984+0000) 2019-10-13T15:09:13.125 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:13.119+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.122950+0000) 2019-10-13T15:09:13.126 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:13.119+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.123749+0000) 2019-10-13T15:09:13.128 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:13.121+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.125686+0000) 2019-10-13T15:09:13.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:13.119+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.125279+0000) 2019-10-13T15:09:13.129 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:13.123+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.126135+0000) 2019-10-13T15:09:13.130 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:13.123+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.126700+0000) 2019-10-13T15:09:13.130 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:13.123+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.127858+0000) 2019-10-13T15:09:13.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:13.129+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:13.133867+0000) 2019-10-13T15:09:13.136 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:13.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220282 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:14.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:14.411+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:14.414742+0000) 2019-10-13T15:09:14.701 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:14.695+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:14.698953+0000) 2019-10-13T15:09:15.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:15.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.581296+0000) 2019-10-13T15:09:15.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:15.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:15.582572+0000) 2019-10-13T15:09:16.528 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:09:16.528 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:16.532 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:09:16.532 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:18.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:18.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.067216+0000) 2019-10-13T15:09:18.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:18.135+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.138351+0000) 2019-10-13T15:09:18.141 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:18.135+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.138828+0000) 2019-10-13T15:09:18.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:18.135+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.138973+0000) 2019-10-13T15:09:18.142 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:18.133+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.139569+0000) 2019-10-13T15:09:18.143 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:18.135+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.140055+0000) 2019-10-13T15:09:18.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:18.135+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.140554+0000) 2019-10-13T15:09:18.143 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:18.135+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.140964+0000) 2019-10-13T15:09:18.150 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:18.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220315 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:18.152 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:18.145+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.149566+0000) 2019-10-13T15:09:18.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:18.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:18.293007+0000) 2019-10-13T15:09:21.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:21.815+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:21.816800+0000) 2019-10-13T15:09:21.871 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:21.865+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:21.869106+0000) 2019-10-13T15:09:23.154 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:23.147+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.151451+0000) 2019-10-13T15:09:23.154 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:23.147+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.151728+0000) 2019-10-13T15:09:23.156 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:23.151+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.153919+0000) 2019-10-13T15:09:23.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:23.149+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.155827+0000) 2019-10-13T15:09:23.158 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:23.151+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.156384+0000) 2019-10-13T15:09:23.159 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:23.151+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.156733+0000) 2019-10-13T15:09:23.159 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:23.151+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.157089+0000) 2019-10-13T15:09:23.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:23.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:23.160557+0000) 2019-10-13T15:09:23.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:23.159+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220344 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:24.417 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:24.411+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:24.414976+0000) 2019-10-13T15:09:24.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:24.695+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:24.699169+0000) 2019-10-13T15:09:25.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:25.575+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.581488+0000) 2019-10-13T15:09:25.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:25.577+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:25.582768+0000) 2019-10-13T15:09:28.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:28.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.067421+0000) 2019-10-13T15:09:28.169 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:28.163+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.166423+0000) 2019-10-13T15:09:28.170 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:28.163+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.166973+0000) 2019-10-13T15:09:28.171 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:28.163+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.167429+0000) 2019-10-13T15:09:28.171 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:28.163+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.167978+0000) 2019-10-13T15:09:28.171 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:28.163+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.168203+0000) 2019-10-13T15:09:28.172 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:28.165+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.168625+0000) 2019-10-13T15:09:28.172 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:28.163+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.169171+0000) 2019-10-13T15:09:28.178 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:28.173+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.176253+0000) 2019-10-13T15:09:28.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:28.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220375 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:28.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:28.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:28.293187+0000) 2019-10-13T15:09:31.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:31.815+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.817006+0000) 2019-10-13T15:09:31.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:31.873+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:31.876260+0000) 2019-10-13T15:09:33.183 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:33.175+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.181014+0000) 2019-10-13T15:09:33.184 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:33.176+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.181368+0000) 2019-10-13T15:09:33.184 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:33.179+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.182039+0000) 2019-10-13T15:09:33.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:33.179+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.182908+0000) 2019-10-13T15:09:33.186 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:33.179+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.183897+0000) 2019-10-13T15:09:33.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:33.179+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.184240+0000) 2019-10-13T15:09:33.187 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:33.179+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.184315+0000) 2019-10-13T15:09:33.193 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:33.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220404 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:33.195 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:33.188+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:33.192933+0000) 2019-10-13T15:09:34.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:34.411+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:34.415204+0000) 2019-10-13T15:09:34.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:34.695+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:34.699389+0000) 2019-10-13T15:09:35.584 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:35.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:35.581684+0000) 2019-10-13T15:09:35.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:35.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:35.582985+0000) 2019-10-13T15:09:38.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:38.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.067621+0000) 2019-10-13T15:09:38.197 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:38.188+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.194720+0000) 2019-10-13T15:09:38.198 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:38.191+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.195830+0000) 2019-10-13T15:09:38.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:38.191+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.196439+0000) 2019-10-13T15:09:38.199 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:38.191+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.196861+0000) 2019-10-13T15:09:38.200 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:38.191+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.197410+0000) 2019-10-13T15:09:38.200 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:38.195+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.197793+0000) 2019-10-13T15:09:38.201 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:38.195+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.198619+0000) 2019-10-13T15:09:38.205 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:38.196+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.202822+0000) 2019-10-13T15:09:38.207 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:38.199+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220437 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:38.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:38.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:38.293372+0000) 2019-10-13T15:09:41.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:41.811+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.817215+0000) 2019-10-13T15:09:41.898 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:41.876+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:41.883326+0000) 2019-10-13T15:09:43.211 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:43.203+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.208464+0000) 2019-10-13T15:09:43.211 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:43.203+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.209222+0000) 2019-10-13T15:09:43.212 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:43.204+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.209722+0000) 2019-10-13T15:09:43.213 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:43.207+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.211092+0000) 2019-10-13T15:09:43.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:43.207+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.211689+0000) 2019-10-13T15:09:43.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:43.207+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.212343+0000) 2019-10-13T15:09:43.215 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:43.207+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.213172+0000) 2019-10-13T15:09:43.217 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:43.208+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:43.215363+0000) 2019-10-13T15:09:43.221 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:43.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:44.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:44.411+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:44.415421+0000) 2019-10-13T15:09:44.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:44.695+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:44.699608+0000) 2019-10-13T15:09:45.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:45.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:45.581972+0000) 2019-10-13T15:09:45.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:45.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:45.583136+0000) 2019-10-13T15:09:46.647 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:09:46.647 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:46.651 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:09:46.651 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:09:48.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:48.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.067835+0000) 2019-10-13T15:09:48.225 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:48.219+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.221830+0000) 2019-10-13T15:09:48.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:48.219+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.222173+0000) 2019-10-13T15:09:48.225 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:48.216+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.223029+0000) 2019-10-13T15:09:48.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:48.219+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.223381+0000) 2019-10-13T15:09:48.226 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:48.219+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.223723+0000) 2019-10-13T15:09:48.227 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:48.219+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.224418+0000) 2019-10-13T15:09:48.227 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:48.219+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.225312+0000) 2019-10-13T15:09:48.234 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:48.228+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.232230+0000) 2019-10-13T15:09:48.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:48.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220497 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:48.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:48.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:48.293567+0000) 2019-10-13T15:09:51.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:51.815+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.817420+0000) 2019-10-13T15:09:51.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:51.888+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:51.890622+0000) 2019-10-13T15:09:53.239 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:53.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.237255+0000) 2019-10-13T15:09:53.240 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:53.231+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.237466+0000) 2019-10-13T15:09:53.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:53.235+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.237856+0000) 2019-10-13T15:09:53.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:53.235+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.239282+0000) 2019-10-13T15:09:53.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:53.235+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.239731+0000) 2019-10-13T15:09:53.243 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:53.235+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.240058+0000) 2019-10-13T15:09:53.243 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:53.235+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.240296+0000) 2019-10-13T15:09:53.248 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:53.240+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:53.245895+0000) 2019-10-13T15:09:53.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:53.243+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220526 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:54.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:54.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:54.415656+0000) 2019-10-13T15:09:54.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:54.699+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:54.699808+0000) 2019-10-13T15:09:55.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:55.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:55.582220+0000) 2019-10-13T15:09:55.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:55.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:55.583364+0000) 2019-10-13T15:09:58.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:58.063+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.068030+0000) 2019-10-13T15:09:58.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:09:58.247+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.251261+0000) 2019-10-13T15:09:58.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:09:58.247+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.252503+0000) 2019-10-13T15:09:58.258 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:09:58.247+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.252949+0000) 2019-10-13T15:09:58.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:09:58.248+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.253547+0000) 2019-10-13T15:09:58.259 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:09:58.251+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.253884+0000) 2019-10-13T15:09:58.259 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:09:58.251+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.254425+0000) 2019-10-13T15:09:58.260 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:58.251+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.254738+0000) 2019-10-13T15:09:58.262 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:09:58.252+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.258729+0000) 2019-10-13T15:09:58.266 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:09:58.259+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220559 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:09:58.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:09:58.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:09:58.293775+0000) 2019-10-13T15:10:01.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:01.815+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.817622+0000) 2019-10-13T15:10:01.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:01.896+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:01.897234+0000) 2019-10-13T15:10:03.270 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:03.263+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.267160+0000) 2019-10-13T15:10:03.270 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:03.263+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.267535+0000) 2019-10-13T15:10:03.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:03.264+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.268319+0000) 2019-10-13T15:10:03.271 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:03.263+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.268908+0000) 2019-10-13T15:10:03.272 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:03.263+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.269683+0000) 2019-10-13T15:10:03.273 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:03.267+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.270393+0000) 2019-10-13T15:10:03.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:03.267+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.270564+0000) 2019-10-13T15:10:03.276 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:03.268+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:03.273480+0000) 2019-10-13T15:10:03.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:03.271+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220588 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:04.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:04.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:04.415860+0000) 2019-10-13T15:10:04.702 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:04.695+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:04.700030+0000) 2019-10-13T15:10:05.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:05.579+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:05.582410+0000) 2019-10-13T15:10:05.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:05.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:05.583590+0000) 2019-10-13T15:10:08.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:08.067+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.068236+0000) 2019-10-13T15:10:08.284 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:08.275+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.281046+0000) 2019-10-13T15:10:08.284 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:08.279+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.282011+0000) 2019-10-13T15:10:08.285 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:08.276+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.282581+0000) 2019-10-13T15:10:08.285 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:08.279+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.282752+0000) 2019-10-13T15:10:08.286 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:08.279+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.283390+0000) 2019-10-13T15:10:08.286 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:08.279+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.283918+0000) 2019-10-13T15:10:08.287 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:08.279+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.284650+0000) 2019-10-13T15:10:08.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:08.284+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.289434+0000) 2019-10-13T15:10:08.294 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:08.287+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220619 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:08.296 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:08.291+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:08.294068+0000) 2019-10-13T15:10:11.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:11.814+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.817823+0000) 2019-10-13T15:10:11.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:11.900+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:11.903803+0000) 2019-10-13T15:10:13.319 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:13.292+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.297707+0000) 2019-10-13T15:10:13.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:13.296+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.302837+0000) 2019-10-13T15:10:13.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:13.290+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.294574+0000) 2019-10-13T15:10:13.337 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:13.290+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.296027+0000) 2019-10-13T15:10:13.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:13.290+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.296840+0000) 2019-10-13T15:10:13.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:13.290+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.297156+0000) 2019-10-13T15:10:13.338 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:13.294+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.297888+0000) 2019-10-13T15:10:13.338 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:13.294+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:13.298111+0000) 2019-10-13T15:10:13.338 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:13.298+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220649 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:14.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:14.410+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:14.416010+0000) 2019-10-13T15:10:14.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:14.694+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:14.700226+0000) 2019-10-13T15:10:15.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:15.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:15.582601+0000) 2019-10-13T15:10:15.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:15.580+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:15.583781+0000) 2019-10-13T15:10:16.685 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:10:16.685 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:16.689 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:10:16.689 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:18.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:18.066+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.068436+0000) 2019-10-13T15:10:18.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:18.290+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.294279+0000) 2019-10-13T15:10:18.311 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:18.302+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.308145+0000) 2019-10-13T15:10:18.312 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:18.302+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.309382+0000) 2019-10-13T15:10:18.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:18.302+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.309716+0000) 2019-10-13T15:10:18.312 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:18.306+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.310056+0000) 2019-10-13T15:10:18.314 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:18.306+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.312155+0000) 2019-10-13T15:10:18.315 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:18.306+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.312494+0000) 2019-10-13T15:10:18.315 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:18.308+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.313016+0000) 2019-10-13T15:10:18.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:18.312+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:18.318428+0000) 2019-10-13T15:10:18.322 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:18.314+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220681 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:21.820 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:21.814+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.818048+0000) 2019-10-13T15:10:21.918 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:21.912+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:21.915408+0000) 2019-10-13T15:10:23.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:23.318+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.322820+0000) 2019-10-13T15:10:23.326 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:23.316+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.323326+0000) 2019-10-13T15:10:23.326 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:23.318+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.323681+0000) 2019-10-13T15:10:23.327 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:23.318+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.324112+0000) 2019-10-13T15:10:23.327 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:23.318+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.325136+0000) 2019-10-13T15:10:23.328 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:23.318+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.325352+0000) 2019-10-13T15:10:23.329 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:23.322+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.326396+0000) 2019-10-13T15:10:23.335 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:23.324+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:23.330838+0000) 2019-10-13T15:10:23.336 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:23.330+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220710 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:24.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:24.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:24.416217+0000) 2019-10-13T15:10:24.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:24.694+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:24.700469+0000) 2019-10-13T15:10:25.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:25.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:25.582800+0000) 2019-10-13T15:10:25.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:25.576+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:25.583969+0000) 2019-10-13T15:10:28.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:28.066+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.068651+0000) 2019-10-13T15:10:28.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:28.290+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.294493+0000) 2019-10-13T15:10:28.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:28.332+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.338512+0000) 2019-10-13T15:10:28.354 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:28.334+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.338943+0000) 2019-10-13T15:10:28.355 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:28.334+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.339438+0000) 2019-10-13T15:10:28.355 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:28.334+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.339872+0000) 2019-10-13T15:10:28.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:28.334+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.340649+0000) 2019-10-13T15:10:28.356 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:28.334+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.341549+0000) 2019-10-13T15:10:28.363 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:28.344+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.349539+0000) 2019-10-13T15:10:28.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:28.338+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:28.343955+0000) 2019-10-13T15:10:28.587 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:28.342+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220741 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:31.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:31.814+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.818295+0000) 2019-10-13T15:10:31.925 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:31.920+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:31.923094+0000) 2019-10-13T15:10:33.355 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:33.346+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.352333+0000) 2019-10-13T15:10:33.357 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:33.350+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.354431+0000) 2019-10-13T15:10:33.357 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:33.350+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.355067+0000) 2019-10-13T15:10:33.358 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:33.350+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.355618+0000) 2019-10-13T15:10:33.358 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:33.350+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.356006+0000) 2019-10-13T15:10:33.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:33.356+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.356657+0000) 2019-10-13T15:10:33.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:33.350+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.356888+0000) 2019-10-13T15:10:33.365 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:33.358+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220772 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:33.367 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:33.364+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:33.364813+0000) 2019-10-13T15:10:34.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:34.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:34.416452+0000) 2019-10-13T15:10:34.703 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:34.698+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:34.700665+0000) 2019-10-13T15:10:35.585 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:35.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:35.583014+0000) 2019-10-13T15:10:35.586 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:35.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:35.584196+0000) 2019-10-13T15:10:38.071 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:38.066+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.068853+0000) 2019-10-13T15:10:38.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:38.290+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.294706+0000) 2019-10-13T15:10:38.369 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:38.362+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.366390+0000) 2019-10-13T15:10:38.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:38.362+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.367366+0000) 2019-10-13T15:10:38.370 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:38.362+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.367627+0000) 2019-10-13T15:10:38.370 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:38.362+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.368004+0000) 2019-10-13T15:10:38.371 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:38.362+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.368610+0000) 2019-10-13T15:10:38.371 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:38.362+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.369055+0000) 2019-10-13T15:10:38.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:38.367+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.370293+0000) 2019-10-13T15:10:38.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:38.375+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:38.376900+0000) 2019-10-13T15:10:38.380 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:38.370+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220803 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:41.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:41.818+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.818496+0000) 2019-10-13T15:10:41.932 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:41.927+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:41.929642+0000) 2019-10-13T15:10:43.385 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:43.378+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.382205+0000) 2019-10-13T15:10:43.385 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:43.378+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.383364+0000) 2019-10-13T15:10:43.386 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:43.378+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.383780+0000) 2019-10-13T15:10:43.388 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:43.382+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.385903+0000) 2019-10-13T15:10:43.389 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:43.382+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.386351+0000) 2019-10-13T15:10:43.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:43.383+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.386810+0000) 2019-10-13T15:10:43.391 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:43.382+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.388304+0000) 2019-10-13T15:10:43.395 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:43.386+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220832 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:43.396 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:43.391+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:43.393886+0000) 2019-10-13T15:10:44.435 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:44.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:44.416649+0000) 2019-10-13T15:10:44.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:44.694+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:44.700891+0000) 2019-10-13T15:10:45.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:45.578+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:45.583184+0000) 2019-10-13T15:10:45.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:45.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:45.584409+0000) 2019-10-13T15:10:46.799 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:10:46.799 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:46.803 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:10:46.803 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:10:48.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:48.066+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.069057+0000) 2019-10-13T15:10:48.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:48.294+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.294909+0000) 2019-10-13T15:10:48.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:48.395+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.396452+0000) 2019-10-13T15:10:48.399 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:48.390+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.396640+0000) 2019-10-13T15:10:48.400 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:48.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.397304+0000) 2019-10-13T15:10:48.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:48.394+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.398396+0000) 2019-10-13T15:10:48.401 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:48.394+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.398592+0000) 2019-10-13T15:10:48.402 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:48.394+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.398828+0000) 2019-10-13T15:10:48.402 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:48.394+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.399500+0000) 2019-10-13T15:10:48.404 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:48.399+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:48.401872+0000) 2019-10-13T15:10:48.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:48.402+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220863 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:51.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:51.814+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.818723+0000) 2019-10-13T15:10:51.938 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:51.935+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:51.936157+0000) 2019-10-13T15:10:53.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:53.406+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.410346+0000) 2019-10-13T15:10:53.414 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:53.407+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.410883+0000) 2019-10-13T15:10:53.414 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:53.406+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.411025+0000) 2019-10-13T15:10:53.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:53.406+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.411473+0000) 2019-10-13T15:10:53.415 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:53.406+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.413277+0000) 2019-10-13T15:10:53.416 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:53.406+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.413672+0000) 2019-10-13T15:10:53.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:53.410+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.414902+0000) 2019-10-13T15:10:53.423 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:53.414+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220894 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:10:53.425 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:53.419+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:53.423106+0000) 2019-10-13T15:10:54.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:54.410+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:54.416866+0000) 2019-10-13T15:10:54.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:54.698+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:54.701106+0000) 2019-10-13T15:10:55.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:55.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:55.583388+0000) 2019-10-13T15:10:55.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:55.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:55.584593+0000) 2019-10-13T15:10:58.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:58.066+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.069258+0000) 2019-10-13T15:10:58.297 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:58.294+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.295151+0000) 2019-10-13T15:10:58.427 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:10:58.418+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.424275+0000) 2019-10-13T15:10:58.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:10:58.418+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.424756+0000) 2019-10-13T15:10:58.427 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:10:58.418+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.424830+0000) 2019-10-13T15:10:58.428 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:10:58.418+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.425478+0000) 2019-10-13T15:10:58.428 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:10:58.423+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.426089+0000) 2019-10-13T15:10:58.429 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:10:58.422+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.426327+0000) 2019-10-13T15:10:58.429 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:10:58.422+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.426699+0000) 2019-10-13T15:10:58.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:10:58.427+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:10:58.431710+0000) 2019-10-13T15:10:58.436 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:10:58.430+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220925 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:01.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:01.818+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.818931+0000) 2019-10-13T15:11:01.945 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:01.939+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:01.942723+0000) 2019-10-13T15:11:03.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:03.434+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.437091+0000) 2019-10-13T15:11:03.442 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:03.434+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.437630+0000) 2019-10-13T15:11:03.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:03.434+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.440251+0000) 2019-10-13T15:11:03.443 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:03.434+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.440365+0000) 2019-10-13T15:11:03.443 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:03.434+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.440556+0000) 2019-10-13T15:11:03.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:03.439+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.443442+0000) 2019-10-13T15:11:03.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:03.438+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.445122+0000) 2019-10-13T15:11:03.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:03.442+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220955 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:03.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:03.447+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:03.449385+0000) 2019-10-13T15:11:04.419 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:04.410+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:04.417086+0000) 2019-10-13T15:11:04.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:04.698+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:04.701327+0000) 2019-10-13T15:11:05.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:05.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:05.583583+0000) 2019-10-13T15:11:05.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:05.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:05.584815+0000) 2019-10-13T15:11:08.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:08.061+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.069491+0000) 2019-10-13T15:11:08.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:08.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.295384+0000) 2019-10-13T15:11:08.454 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:08.445+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.452002+0000) 2019-10-13T15:11:08.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:08.449+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.453882+0000) 2019-10-13T15:11:08.457 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:08.449+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.454792+0000) 2019-10-13T15:11:08.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:08.451+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.455315+0000) 2019-10-13T15:11:08.458 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:08.449+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.455767+0000) 2019-10-13T15:11:08.459 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:08.449+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.457049+0000) 2019-10-13T15:11:08.461 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:08.453+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.459029+0000) 2019-10-13T15:11:08.467 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:08.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220986 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:08.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:08.463+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:08.465165+0000) 2019-10-13T15:11:11.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:11.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.819163+0000) 2019-10-13T15:11:11.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:11.947+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:11.949255+0000) 2019-10-13T15:11:13.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:13.461+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.467109+0000) 2019-10-13T15:11:13.471 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:13.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.468690+0000) 2019-10-13T15:11:13.473 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:13.467+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.470523+0000) 2019-10-13T15:11:13.473 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:13.465+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.470915+0000) 2019-10-13T15:11:13.473 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:13.465+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.471150+0000) 2019-10-13T15:11:13.474 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:13.465+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.471729+0000) 2019-10-13T15:11:13.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:13.465+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.472426+0000) 2019-10-13T15:11:13.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:13.475+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:13.476945+0000) 2019-10-13T15:11:13.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:13.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:14.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:14.413+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:14.417279+0000) 2019-10-13T15:11:14.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:14.697+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:14.701576+0000) 2019-10-13T15:11:15.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:15.577+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:15.583795+0000) 2019-10-13T15:11:15.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:15.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:15.585024+0000) 2019-10-13T15:11:16.923 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:11:16.924 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:16.927 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:11:16.928 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:18.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:18.065+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.069685+0000) 2019-10-13T15:11:18.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:18.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.295616+0000) 2019-10-13T15:11:18.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:18.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.480993+0000) 2019-10-13T15:11:18.485 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:18.479+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.481868+0000) 2019-10-13T15:11:18.485 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:18.477+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.482820+0000) 2019-10-13T15:11:18.486 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:18.477+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.483023+0000) 2019-10-13T15:11:18.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:18.481+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.486815+0000) 2019-10-13T15:11:18.490 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:18.481+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.487749+0000) 2019-10-13T15:11:18.491 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:18.481+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.488574+0000) 2019-10-13T15:11:18.493 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:18.487+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:18.491370+0000) 2019-10-13T15:11:18.495 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:18.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221049 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:21.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:21.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.819365+0000) 2019-10-13T15:11:21.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:21.959+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:21.960560+0000) 2019-10-13T15:11:23.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:23.493+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.495354+0000) 2019-10-13T15:11:23.499 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:23.493+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.496266+0000) 2019-10-13T15:11:23.500 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:23.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.497323+0000) 2019-10-13T15:11:23.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:23.493+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.497488+0000) 2019-10-13T15:11:23.501 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:23.493+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.497631+0000) 2019-10-13T15:11:23.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:23.495+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.498152+0000) 2019-10-13T15:11:23.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:23.497+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.499044+0000) 2019-10-13T15:11:23.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:23.503+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:23.505699+0000) 2019-10-13T15:11:23.518 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:23.505+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:24.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:24.413+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:24.417485+0000) 2019-10-13T15:11:24.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:24.701+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:24.701812+0000) 2019-10-13T15:11:25.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:25.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:25.583991+0000) 2019-10-13T15:11:25.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:25.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:25.585223+0000) 2019-10-13T15:11:28.072 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:28.069+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.069954+0000) 2019-10-13T15:11:28.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:28.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.295820+0000) 2019-10-13T15:11:28.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:28.509+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.510923+0000) 2019-10-13T15:11:28.514 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:28.509+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.511780+0000) 2019-10-13T15:11:28.514 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:28.509+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.512288+0000) 2019-10-13T15:11:28.515 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:28.509+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.512944+0000) 2019-10-13T15:11:28.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:28.509+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.513652+0000) 2019-10-13T15:11:28.516 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:28.511+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.514086+0000) 2019-10-13T15:11:28.517 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:28.513+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.514835+0000) 2019-10-13T15:11:28.523 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:28.519+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:28.520706+0000) 2019-10-13T15:11:28.525 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:28.521+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221110 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:31.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:31.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.819560+0000) 2019-10-13T15:11:31.969 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:31.963+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:31.967069+0000) 2019-10-13T15:11:33.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:33.525+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.525800+0000) 2019-10-13T15:11:33.529 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:33.523+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.526475+0000) 2019-10-13T15:11:33.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:33.525+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.527122+0000) 2019-10-13T15:11:33.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:33.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.527717+0000) 2019-10-13T15:11:33.530 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:33.525+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.528010+0000) 2019-10-13T15:11:33.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:33.525+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.528646+0000) 2019-10-13T15:11:33.531 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:33.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.529180+0000) 2019-10-13T15:11:33.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:33.531+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:33.535180+0000) 2019-10-13T15:11:33.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:33.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221142 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:34.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:34.413+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:34.417659+0000) 2019-10-13T15:11:34.704 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:34.701+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:34.702008+0000) 2019-10-13T15:11:35.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:35.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:35.584200+0000) 2019-10-13T15:11:35.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:35.583+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:35.585433+0000) 2019-10-13T15:11:38.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:38.069+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.070159+0000) 2019-10-13T15:11:38.298 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:38.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.296066+0000) 2019-10-13T15:11:38.543 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:38.537+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.540696+0000) 2019-10-13T15:11:38.544 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:38.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.541511+0000) 2019-10-13T15:11:38.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:38.541+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.541947+0000) 2019-10-13T15:11:38.545 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:38.541+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.542179+0000) 2019-10-13T15:11:38.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:38.543+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.544479+0000) 2019-10-13T15:11:38.549 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:38.545+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.547111+0000) 2019-10-13T15:11:38.552 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:38.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.550101+0000) 2019-10-13T15:11:38.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:38.549+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221173 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:38.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:38.555+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:38.556905+0000) 2019-10-13T15:11:41.822 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:41.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.819720+0000) 2019-10-13T15:11:41.976 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:41.971+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:41.973755+0000) 2019-10-13T15:11:43.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:43.553+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.556716+0000) 2019-10-13T15:11:43.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:43.553+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.557506+0000) 2019-10-13T15:11:43.560 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:43.557+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.558257+0000) 2019-10-13T15:11:43.561 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:43.557+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.558635+0000) 2019-10-13T15:11:43.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:43.554+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.559209+0000) 2019-10-13T15:11:43.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:43.557+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.558827+0000) 2019-10-13T15:11:43.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:43.557+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.559878+0000) 2019-10-13T15:11:43.566 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:43.558+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:43.563860+0000) 2019-10-13T15:11:43.569 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:43.565+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221202 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:44.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:44.417+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:44.417845+0000) 2019-10-13T15:11:44.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:44.701+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:44.702217+0000) 2019-10-13T15:11:45.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:45.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:45.584424+0000) 2019-10-13T15:11:45.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:45.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:45.585628+0000) 2019-10-13T15:11:47.269 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:11:47.270 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:47.286 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:11:47.286 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:11:48.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:48.069+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.070385+0000) 2019-10-13T15:11:48.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:48.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.296284+0000) 2019-10-13T15:11:48.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:48.569+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.570084+0000) 2019-10-13T15:11:48.574 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:48.569+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.571729+0000) 2019-10-13T15:11:48.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:48.569+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.572311+0000) 2019-10-13T15:11:48.575 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:48.569+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.573009+0000) 2019-10-13T15:11:48.575 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:48.569+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.573071+0000) 2019-10-13T15:11:48.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:48.570+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.573901+0000) 2019-10-13T15:11:48.577 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:48.569+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.573479+0000) 2019-10-13T15:11:48.580 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:48.574+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:48.577649+0000) 2019-10-13T15:11:48.584 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:48.581+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221234 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:51.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:51.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.819965+0000) 2019-10-13T15:11:51.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:51.982+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:51.984718+0000) 2019-10-13T15:11:53.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:53.581+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.585468+0000) 2019-10-13T15:11:53.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:53.585+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.585999+0000) 2019-10-13T15:11:53.589 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:53.585+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.586475+0000) 2019-10-13T15:11:53.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:53.585+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.587166+0000) 2019-10-13T15:11:53.590 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:53.585+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.587977+0000) 2019-10-13T15:11:53.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:53.586+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.588206+0000) 2019-10-13T15:11:53.591 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:53.585+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.588670+0000) 2019-10-13T15:11:53.596 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:53.590+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:53.593851+0000) 2019-10-13T15:11:53.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:53.593+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221265 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:11:54.420 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:54.417+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:54.418033+0000) 2019-10-13T15:11:54.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:54.701+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:54.702450+0000) 2019-10-13T15:11:55.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:55.581+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:55.584621+0000) 2019-10-13T15:11:55.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:55.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:55.585854+0000) 2019-10-13T15:11:58.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:58.069+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.070603+0000) 2019-10-13T15:11:58.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:58.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.296503+0000) 2019-10-13T15:11:58.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:11:58.597+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.600662+0000) 2019-10-13T15:11:58.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:11:58.597+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.601242+0000) 2019-10-13T15:11:58.604 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:11:58.601+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.602027+0000) 2019-10-13T15:11:58.605 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:11:58.598+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.602648+0000) 2019-10-13T15:11:58.606 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:11:58.601+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.604097+0000) 2019-10-13T15:11:58.607 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:11:58.601+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.604330+0000) 2019-10-13T15:11:58.608 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:11:58.605+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.606204+0000) 2019-10-13T15:11:58.609 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:11:58.602+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:11:58.606627+0000) 2019-10-13T15:11:58.616 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:11:58.609+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:01.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:01.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.820194+0000) 2019-10-13T15:12:01.994 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:01.986+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:01.991239+0000) 2019-10-13T15:12:03.618 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:03.612+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.615019+0000) 2019-10-13T15:12:03.618 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:03.610+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.615372+0000) 2019-10-13T15:12:03.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:03.612+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.615729+0000) 2019-10-13T15:12:03.619 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:03.612+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.616530+0000) 2019-10-13T15:12:03.621 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:03.616+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.618084+0000) 2019-10-13T15:12:03.621 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:03.616+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.618735+0000) 2019-10-13T15:12:03.622 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:03.616+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.619367+0000) 2019-10-13T15:12:03.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:03.622+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:03.625383+0000) 2019-10-13T15:12:03.629 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:03.624+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221326 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:04.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:04.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:04.418218+0000) 2019-10-13T15:12:04.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:04.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:04.702680+0000) 2019-10-13T15:12:05.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:05.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:05.584814+0000) 2019-10-13T15:12:05.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:05.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:05.586048+0000) 2019-10-13T15:12:08.073 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:08.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.070805+0000) 2019-10-13T15:12:08.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:08.292+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.296724+0000) 2019-10-13T15:12:08.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:08.624+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.629681+0000) 2019-10-13T15:12:08.633 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:08.628+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.630368+0000) 2019-10-13T15:12:08.633 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:08.626+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.630764+0000) 2019-10-13T15:12:08.635 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:08.628+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.632345+0000) 2019-10-13T15:12:08.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:08.628+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.632547+0000) 2019-10-13T15:12:08.638 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:08.632+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.635460+0000) 2019-10-13T15:12:08.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:08.632+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.637554+0000) 2019-10-13T15:12:08.643 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:08.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221358 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:08.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:08.642+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:08.645326+0000) 2019-10-13T15:12:11.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:11.816+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.820407+0000) 2019-10-13T15:12:12.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:11.994+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:11.997777+0000) 2019-10-13T15:12:13.647 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:13.640+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.644738+0000) 2019-10-13T15:12:13.648 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:13.640+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.645118+0000) 2019-10-13T15:12:13.648 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:13.640+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.645613+0000) 2019-10-13T15:12:13.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:13.644+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.647880+0000) 2019-10-13T15:12:13.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:13.646+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.650054+0000) 2019-10-13T15:12:13.654 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:13.648+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.651515+0000) 2019-10-13T15:12:13.655 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:13.650+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.652228+0000) 2019-10-13T15:12:13.657 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:13.652+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:13.654479+0000) 2019-10-13T15:12:13.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:13.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:14.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:14.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:14.418404+0000) 2019-10-13T15:12:14.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:14.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:14.702905+0000) 2019-10-13T15:12:15.587 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:15.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:15.585004+0000) 2019-10-13T15:12:15.588 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:15.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:15.586247+0000) 2019-10-13T15:12:17.444 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:12:17.444 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:17.448 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:12:17.448 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:18.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:18.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.071006+0000) 2019-10-13T15:12:18.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:18.296+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.296934+0000) 2019-10-13T15:12:18.673 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:18.656+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.659479+0000) 2019-10-13T15:12:18.674 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:18.654+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.659980+0000) 2019-10-13T15:12:18.674 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:18.656+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.660176+0000) 2019-10-13T15:12:18.674 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:18.656+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.660441+0000) 2019-10-13T15:12:18.674 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:18.656+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.660928+0000) 2019-10-13T15:12:18.674 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:18.656+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.661721+0000) 2019-10-13T15:12:18.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:18.662+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.665203+0000) 2019-10-13T15:12:18.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:18.660+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:18.663381+0000) 2019-10-13T15:12:18.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:18.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221422 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:21.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:21.816+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:21.820650+0000) 2019-10-13T15:12:22.007 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:22.002+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:22.004369+0000) 2019-10-13T15:12:23.676 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:23.672+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.673895+0000) 2019-10-13T15:12:23.677 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:23.672+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.675291+0000) 2019-10-13T15:12:23.678 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:23.670+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.675664+0000) 2019-10-13T15:12:23.678 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:23.672+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.676102+0000) 2019-10-13T15:12:23.679 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:23.672+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.676344+0000) 2019-10-13T15:12:23.679 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:23.672+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.676670+0000) 2019-10-13T15:12:23.680 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:23.676+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.677888+0000) 2019-10-13T15:12:23.683 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:23.678+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:23.680647+0000) 2019-10-13T15:12:23.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:23.680+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:24.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:24.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:24.418590+0000) 2019-10-13T15:12:24.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:24.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:24.703101+0000) 2019-10-13T15:12:25.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:25.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:25.585226+0000) 2019-10-13T15:12:25.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:25.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:25.586471+0000) 2019-10-13T15:12:28.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:28.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.071261+0000) 2019-10-13T15:12:28.299 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:28.292+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.297144+0000) 2019-10-13T15:12:28.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:28.686+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.689308+0000) 2019-10-13T15:12:28.692 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:28.684+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.689497+0000) 2019-10-13T15:12:28.692 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:28.688+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.689898+0000) 2019-10-13T15:12:28.693 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:28.688+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.690860+0000) 2019-10-13T15:12:28.693 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:28.688+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.690615+0000) 2019-10-13T15:12:28.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:28.688+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.691320+0000) 2019-10-13T15:12:28.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:28.688+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.691991+0000) 2019-10-13T15:12:28.701 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:28.694+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:28.698586+0000) 2019-10-13T15:12:28.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:28.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221484 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:31.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:31.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:31.820909+0000) 2019-10-13T15:12:32.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:32.014+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:32.016063+0000) 2019-10-13T15:12:33.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:33.700+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.703556+0000) 2019-10-13T15:12:33.719 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:33.702+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.704927+0000) 2019-10-13T15:12:33.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:33.706+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.710275+0000) 2019-10-13T15:12:33.735 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:33.700+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.704302+0000) 2019-10-13T15:12:33.735 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:33.700+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.705305+0000) 2019-10-13T15:12:33.736 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:33.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.705803+0000) 2019-10-13T15:12:33.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:33.704+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.706243+0000) 2019-10-13T15:12:33.749 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:33.704+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:33.706984+0000) 2019-10-13T15:12:33.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:33.708+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221515 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:34.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:34.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:34.418789+0000) 2019-10-13T15:12:34.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:34.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:34.703323+0000) 2019-10-13T15:12:35.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:35.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:35.585475+0000) 2019-10-13T15:12:35.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:35.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:35.586660+0000) 2019-10-13T15:12:38.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:38.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.071437+0000) 2019-10-13T15:12:38.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:38.292+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.297305+0000) 2019-10-13T15:12:38.720 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:38.712+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.717694+0000) 2019-10-13T15:12:38.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:38.714+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.718573+0000) 2019-10-13T15:12:38.721 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:38.716+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.718745+0000) 2019-10-13T15:12:38.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:38.716+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.719104+0000) 2019-10-13T15:12:38.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:38.716+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.719638+0000) 2019-10-13T15:12:38.722 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:38.716+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.719964+0000) 2019-10-13T15:12:38.723 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:38.716+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.720935+0000) 2019-10-13T15:12:38.727 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:38.722+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:38.724638+0000) 2019-10-13T15:12:38.730 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:38.724+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:41.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:41.816+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:41.821145+0000) 2019-10-13T15:12:42.025 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:42.022+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:42.022553+0000) 2019-10-13T15:12:43.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:43.728+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.731661+0000) 2019-10-13T15:12:43.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:43.726+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.732065+0000) 2019-10-13T15:12:43.735 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:43.728+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.733170+0000) 2019-10-13T15:12:43.737 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:43.732+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.734974+0000) 2019-10-13T15:12:43.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:43.732+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.735306+0000) 2019-10-13T15:12:43.739 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:43.732+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.736916+0000) 2019-10-13T15:12:43.743 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:43.736+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.740492+0000) 2019-10-13T15:12:43.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:43.738+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:43.741363+0000) 2019-10-13T15:12:43.745 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:43.740+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221577 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:44.421 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:44.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:44.418968+0000) 2019-10-13T15:12:44.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:44.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:44.703527+0000) 2019-10-13T15:12:45.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:45.580+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:45.585711+0000) 2019-10-13T15:12:45.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:45.582+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:45.586856+0000) 2019-10-13T15:12:47.551 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:12:47.551 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:47.555 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:12:47.555 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:12:48.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:48.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.071663+0000) 2019-10-13T15:12:48.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:48.292+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.297518+0000) 2019-10-13T15:12:48.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:48.744+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.745901+0000) 2019-10-13T15:12:48.750 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:48.744+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.747958+0000) 2019-10-13T15:12:48.751 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:48.746+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.748674+0000) 2019-10-13T15:12:48.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:48.744+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.749143+0000) 2019-10-13T15:12:48.752 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:48.748+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.749845+0000) 2019-10-13T15:12:48.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:48.748+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.750793+0000) 2019-10-13T15:12:48.754 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:48.748+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.751565+0000) 2019-10-13T15:12:48.759 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:48.750+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:48.755743+0000) 2019-10-13T15:12:48.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:48.752+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221610 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:51.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:51.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:51.821347+0000) 2019-10-13T15:12:52.032 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:52.025+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:52.029260+0000) 2019-10-13T15:12:53.762 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:53.756+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.759419+0000) 2019-10-13T15:12:53.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:53.756+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.760108+0000) 2019-10-13T15:12:53.763 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:53.757+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.760532+0000) 2019-10-13T15:12:53.763 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:53.756+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.760787+0000) 2019-10-13T15:12:53.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:53.756+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.761490+0000) 2019-10-13T15:12:53.766 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:53.760+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.762059+0000) 2019-10-13T15:12:53.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:53.760+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.762416+0000) 2019-10-13T15:12:53.770 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:53.761+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:53.767514+0000) 2019-10-13T15:12:53.773 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:53.768+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:12:54.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:54.416+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:54.419198+0000) 2019-10-13T15:12:54.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:54.700+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:54.703707+0000) 2019-10-13T15:12:55.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:55.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:55.585931+0000) 2019-10-13T15:12:55.589 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:55.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:55.587059+0000) 2019-10-13T15:12:58.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:58.068+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.071886+0000) 2019-10-13T15:12:58.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:58.296+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.297775+0000) 2019-10-13T15:12:58.775 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:12:58.767+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.772689+0000) 2019-10-13T15:12:58.776 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:12:58.767+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.773601+0000) 2019-10-13T15:12:58.778 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:12:58.771+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.775343+0000) 2019-10-13T15:12:58.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:12:58.771+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.775375+0000) 2019-10-13T15:12:58.778 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:12:58.769+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.775954+0000) 2019-10-13T15:12:58.779 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:12:58.771+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.776614+0000) 2019-10-13T15:12:58.781 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:12:58.775+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.778573+0000) 2019-10-13T15:12:58.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:12:58.777+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:12:58.782081+0000) 2019-10-13T15:12:58.787 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:12:58.779+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:01.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:01.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:01.821568+0000) 2019-10-13T15:13:02.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:02.029+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:02.035848+0000) 2019-10-13T15:13:03.790 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:03.783+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.787733+0000) 2019-10-13T15:13:03.793 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:03.787+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.790701+0000) 2019-10-13T15:13:03.794 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:03.785+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.790914+0000) 2019-10-13T15:13:03.796 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:03.787+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.793637+0000) 2019-10-13T15:13:03.797 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:03.789+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.794117+0000) 2019-10-13T15:13:03.798 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:03.791+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.795511+0000) 2019-10-13T15:13:03.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:03.791+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.797017+0000) 2019-10-13T15:13:03.801 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:03.795+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:03.798652+0000) 2019-10-13T15:13:03.802 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:03.795+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221703 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:04.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:04.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:04.419416+0000) 2019-10-13T15:13:04.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:04.699+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:04.703884+0000) 2019-10-13T15:13:05.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:05.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:05.586086+0000) 2019-10-13T15:13:05.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:05.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:05.587283+0000) 2019-10-13T15:13:08.074 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:08.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.072089+0000) 2019-10-13T15:13:08.300 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:08.295+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.297972+0000) 2019-10-13T15:13:08.806 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:08.799+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.803154+0000) 2019-10-13T15:13:08.806 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:08.799+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.804084+0000) 2019-10-13T15:13:08.808 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:08.803+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.805976+0000) 2019-10-13T15:13:08.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:08.801+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.806357+0000) 2019-10-13T15:13:08.809 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:08.803+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.806568+0000) 2019-10-13T15:13:08.809 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:08.803+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.807322+0000) 2019-10-13T15:13:08.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:08.803+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.807492+0000) 2019-10-13T15:13:08.815 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:08.807+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221735 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:08.815 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:08.809+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:08.812794+0000) 2019-10-13T15:13:11.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:11.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:11.821775+0000) 2019-10-13T15:13:12.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:12.041+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:12.042380+0000) 2019-10-13T15:13:13.820 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:13.811+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.817129+0000) 2019-10-13T15:13:13.821 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:13.815+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.818589+0000) 2019-10-13T15:13:13.821 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:13.815+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.819074+0000) 2019-10-13T15:13:13.822 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:13.813+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.819455+0000) 2019-10-13T15:13:13.822 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:13.815+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.819815+0000) 2019-10-13T15:13:13.823 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:13.815+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.820782+0000) 2019-10-13T15:13:13.824 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:13.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.822219+0000) 2019-10-13T15:13:13.828 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:13.823+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221766 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:13.829 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:13.821+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:13.827186+0000) 2019-10-13T15:13:14.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:14.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:14.419627+0000) 2019-10-13T15:13:14.706 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:14.699+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:14.704093+0000) 2019-10-13T15:13:15.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:15.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:15.586291+0000) 2019-10-13T15:13:15.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:15.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:15.587473+0000) 2019-10-13T15:13:17.898 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:13:17.898 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:17.902 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:13:17.902 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:18.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:18.067+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.072289+0000) 2019-10-13T15:13:18.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:18.295+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.298170+0000) 2019-10-13T15:13:18.831 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:18.823+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.828944+0000) 2019-10-13T15:13:18.832 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:18.825+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.829331+0000) 2019-10-13T15:13:18.832 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:18.823+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.829707+0000) 2019-10-13T15:13:18.833 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:18.827+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.830327+0000) 2019-10-13T15:13:18.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:18.827+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.831575+0000) 2019-10-13T15:13:18.835 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:18.827+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.832340+0000) 2019-10-13T15:13:18.836 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:18.827+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.833438+0000) 2019-10-13T15:13:18.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:18.833+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:18.839315+0000) 2019-10-13T15:13:18.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:18.835+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221798 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:21.824 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:21.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:21.821981+0000) 2019-10-13T15:13:22.051 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:22.045+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:22.049134+0000) 2019-10-13T15:13:23.847 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:23.841+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.844764+0000) 2019-10-13T15:13:23.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:23.839+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.845020+0000) 2019-10-13T15:13:23.848 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:23.839+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.845109+0000) 2019-10-13T15:13:23.849 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:23.839+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.845402+0000) 2019-10-13T15:13:23.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:23.843+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.845949+0000) 2019-10-13T15:13:23.849 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:23.843+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.846387+0000) 2019-10-13T15:13:23.850 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:23.843+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.846574+0000) 2019-10-13T15:13:23.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:23.849+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:23.852122+0000) 2019-10-13T15:13:23.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:23.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221826 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:24.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:24.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:24.419871+0000) 2019-10-13T15:13:24.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:24.699+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:24.704269+0000) 2019-10-13T15:13:25.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:25.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:25.586485+0000) 2019-10-13T15:13:25.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:25.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:25.587689+0000) 2019-10-13T15:13:28.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:28.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.072493+0000) 2019-10-13T15:13:28.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:28.295+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.298394+0000) 2019-10-13T15:13:28.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:28.853+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.856895+0000) 2019-10-13T15:13:28.860 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:28.851+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.857363+0000) 2019-10-13T15:13:28.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:28.855+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.858336+0000) 2019-10-13T15:13:28.861 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:28.855+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.858740+0000) 2019-10-13T15:13:28.862 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:28.855+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.859786+0000) 2019-10-13T15:13:28.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:28.855+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.861484+0000) 2019-10-13T15:13:28.865 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:28.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.863134+0000) 2019-10-13T15:13:28.867 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:28.861+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:28.864461+0000) 2019-10-13T15:13:28.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:28.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:31.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:31.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:31.822226+0000) 2019-10-13T15:13:32.058 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:32.053+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:32.055584+0000) 2019-10-13T15:13:33.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:33.867+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.870740+0000) 2019-10-13T15:13:33.874 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:33.865+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.871437+0000) 2019-10-13T15:13:33.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:33.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.872137+0000) 2019-10-13T15:13:33.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:33.867+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.872306+0000) 2019-10-13T15:13:33.875 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:33.867+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.872709+0000) 2019-10-13T15:13:33.876 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:33.867+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.873144+0000) 2019-10-13T15:13:33.876 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:33.871+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.873815+0000) 2019-10-13T15:13:33.879 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:33.873+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:33.877350+0000) 2019-10-13T15:13:33.884 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:33.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221889 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:34.422 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:34.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:34.420057+0000) 2019-10-13T15:13:34.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:34.699+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:34.704458+0000) 2019-10-13T15:13:35.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:35.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:35.586673+0000) 2019-10-13T15:13:35.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:35.581+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:35.587890+0000) 2019-10-13T15:13:38.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:38.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.072711+0000) 2019-10-13T15:13:38.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:38.295+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.298598+0000) 2019-10-13T15:13:38.887 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:38.879+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.885035+0000) 2019-10-13T15:13:38.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:38.879+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.885578+0000) 2019-10-13T15:13:38.889 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:38.881+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.887223+0000) 2019-10-13T15:13:38.890 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:38.883+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.887950+0000) 2019-10-13T15:13:38.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:38.883+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.888333+0000) 2019-10-13T15:13:38.892 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:38.887+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.889803+0000) 2019-10-13T15:13:38.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:38.887+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.890260+0000) 2019-10-13T15:13:38.898 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:38.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221922 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:38.902 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:38.893+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:38.899704+0000) 2019-10-13T15:13:41.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:41.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:41.822385+0000) 2019-10-13T15:13:42.069 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:42.057+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:42.062190+0000) 2019-10-13T15:13:43.902 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:43.895+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.899648+0000) 2019-10-13T15:13:43.903 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:43.895+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.900285+0000) 2019-10-13T15:13:43.903 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:43.895+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.900755+0000) 2019-10-13T15:13:43.904 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:43.897+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.901572+0000) 2019-10-13T15:13:43.904 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:43.899+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.901887+0000) 2019-10-13T15:13:43.905 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:43.899+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.902486+0000) 2019-10-13T15:13:43.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:43.899+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.902905+0000) 2019-10-13T15:13:43.911 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:43.905+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:43.908857+0000) 2019-10-13T15:13:43.912 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:43.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221950 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:44.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:44.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:44.420257+0000) 2019-10-13T15:13:44.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:44.703+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:44.704637+0000) 2019-10-13T15:13:45.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:45.583+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:45.586871+0000) 2019-10-13T15:13:45.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:45.585+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:45.588129+0000) 2019-10-13T15:13:47.954 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:13:47.954 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:47.958 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:13:47.958 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:13:48.075 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:48.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.072910+0000) 2019-10-13T15:13:48.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:48.295+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.298814+0000) 2019-10-13T15:13:48.917 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:48.911+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.914485+0000) 2019-10-13T15:13:48.917 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:48.911+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.914834+0000) 2019-10-13T15:13:48.918 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:48.911+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.915791+0000) 2019-10-13T15:13:48.919 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:48.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.917252+0000) 2019-10-13T15:13:48.920 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:48.911+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.917600+0000) 2019-10-13T15:13:48.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:48.915+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.918304+0000) 2019-10-13T15:13:48.921 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:48.915+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.918723+0000) 2019-10-13T15:13:48.928 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:48.921+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:48.925572+0000) 2019-10-13T15:13:48.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:48.919+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221983 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:51.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:51.819+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:51.822535+0000) 2019-10-13T15:13:52.071 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:52.065+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:52.068759+0000) 2019-10-13T15:13:53.931 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:53.923+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.928242+0000) 2019-10-13T15:13:53.931 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:53.923+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.928985+0000) 2019-10-13T15:13:53.932 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:53.927+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.930015+0000) 2019-10-13T15:13:53.933 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:53.925+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.930030+0000) 2019-10-13T15:13:53.933 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:53.923+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.929243+0000) 2019-10-13T15:13:53.933 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:53.927+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.931272+0000) 2019-10-13T15:13:53.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:53.927+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.931651+0000) 2019-10-13T15:13:53.939 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:53.933+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:53.937174+0000) 2019-10-13T15:13:53.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:53.935+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222013 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:13:54.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:54.415+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:54.420434+0000) 2019-10-13T15:13:54.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:54.703+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:54.704811+0000) 2019-10-13T15:13:55.589 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:55.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:55.587064+0000) 2019-10-13T15:13:55.590 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:55.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:55.588327+0000) 2019-10-13T15:13:58.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:58.070+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.073147+0000) 2019-10-13T15:13:58.301 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:58.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.299014+0000) 2019-10-13T15:13:58.944 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:13:58.938+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.942041+0000) 2019-10-13T15:13:58.945 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:13:58.936+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.942858+0000) 2019-10-13T15:13:58.945 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:13:58.938+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.942502+0000) 2019-10-13T15:13:58.946 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:13:58.938+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.943051+0000) 2019-10-13T15:13:58.946 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:13:58.938+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.944270+0000) 2019-10-13T15:13:58.947 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:13:58.938+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.945166+0000) 2019-10-13T15:13:58.948 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:13:58.942+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.945954+0000) 2019-10-13T15:13:58.953 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:13:58.944+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:13:58.949814+0000) 2019-10-13T15:13:58.954 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:13:58.946+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222046 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:01.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:01.818+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:01.822704+0000) 2019-10-13T15:14:02.088 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:02.072+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:02.075235+0000) 2019-10-13T15:14:03.958 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:03.950+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.954917+0000) 2019-10-13T15:14:03.958 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:03.948+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.955492+0000) 2019-10-13T15:14:03.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:03.950+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.956513+0000) 2019-10-13T15:14:03.959 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:03.950+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.956797+0000) 2019-10-13T15:14:03.960 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:03.950+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.957366+0000) 2019-10-13T15:14:03.960 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:03.954+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.958250+0000) 2019-10-13T15:14:03.961 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:03.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.958867+0000) 2019-10-13T15:14:03.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:03.960+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:03.966307+0000) 2019-10-13T15:14:03.969 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:03.962+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222074 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:04.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:04.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:04.420608+0000) 2019-10-13T15:14:04.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:04.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:04.704945+0000) 2019-10-13T15:14:05.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:05.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:05.587281+0000) 2019-10-13T15:14:05.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:05.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:05.588539+0000) 2019-10-13T15:14:08.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:08.070+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.073369+0000) 2019-10-13T15:14:08.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:08.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.299179+0000) 2019-10-13T15:14:08.974 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:08.966+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.971858+0000) 2019-10-13T15:14:08.975 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:08.968+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.972418+0000) 2019-10-13T15:14:08.975 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:08.966+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.972787+0000) 2019-10-13T15:14:08.977 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:08.970+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.974456+0000) 2019-10-13T15:14:08.977 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:08.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.975245+0000) 2019-10-13T15:14:08.978 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:08.970+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.975309+0000) 2019-10-13T15:14:08.979 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:08.970+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.976422+0000) 2019-10-13T15:14:08.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:08.972+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:08.976965+0000) 2019-10-13T15:14:08.983 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:08.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222106 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:11.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:11.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:11.822873+0000) 2019-10-13T15:14:12.084 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:12.080+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:12.081692+0000) 2019-10-13T15:14:13.987 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:13.980+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.984247+0000) 2019-10-13T15:14:13.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:13.978+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.984916+0000) 2019-10-13T15:14:13.988 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:13.978+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.985281+0000) 2019-10-13T15:14:13.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:13.982+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.987369+0000) 2019-10-13T15:14:13.991 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:13.982+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.987890+0000) 2019-10-13T15:14:13.991 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:13.982+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.988095+0000) 2019-10-13T15:14:13.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:13.982+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.988986+0000) 2019-10-13T15:14:13.997 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:13.988+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:13.994194+0000) 2019-10-13T15:14:13.997 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:13.990+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222137 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:14.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:14.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:14.420782+0000) 2019-10-13T15:14:14.707 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:14.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:14.705148+0000) 2019-10-13T15:14:15.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:15.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:15.587507+0000) 2019-10-13T15:14:15.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:15.584+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:15.588748+0000) 2019-10-13T15:14:18.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:18.070+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:18.073595+0000) 2019-10-13T15:14:18.284 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:14:18.285 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:18.300 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:14:18.301 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:18.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:18.294+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:18.299369+0000) 2019-10-13T15:14:19.000 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:18.992+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:18.997796+0000) 2019-10-13T15:14:19.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:18.994+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:18.998351+0000) 2019-10-13T15:14:19.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:18.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.000498+0000) 2019-10-13T15:14:19.003 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:18.994+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.001150+0000) 2019-10-13T15:14:19.004 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:18.998+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.002059+0000) 2019-10-13T15:14:19.004 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:18.998+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.002075+0000) 2019-10-13T15:14:19.005 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:18.998+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.002795+0000) 2019-10-13T15:14:19.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:19.000+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:19.006534+0000) 2019-10-13T15:14:19.012 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:19.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222169 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:21.825 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:21.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:21.823030+0000) 2019-10-13T15:14:22.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:22.084+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:22.088368+0000) 2019-10-13T15:14:24.014 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:24.008+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.012256+0000) 2019-10-13T15:14:24.015 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:24.006+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.012681+0000) 2019-10-13T15:14:24.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:24.006+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.013696+0000) 2019-10-13T15:14:24.016 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:24.010+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.014340+0000) 2019-10-13T15:14:24.017 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:24.010+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.014755+0000) 2019-10-13T15:14:24.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:24.010+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.015745+0000) 2019-10-13T15:14:24.019 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:24.010+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.016406+0000) 2019-10-13T15:14:24.024 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:24.016+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.021437+0000) 2019-10-13T15:14:24.025 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:24.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222198 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:24.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:24.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.420920+0000) 2019-10-13T15:14:24.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:24.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:24.705316+0000) 2019-10-13T15:14:25.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:25.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:25.587697+0000) 2019-10-13T15:14:25.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:25.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:25.588984+0000) 2019-10-13T15:14:28.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:28.070+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:28.073814+0000) 2019-10-13T15:14:28.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:28.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:28.299607+0000) 2019-10-13T15:14:29.029 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:29.022+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.026784+0000) 2019-10-13T15:14:29.030 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:29.022+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.027563+0000) 2019-10-13T15:14:29.030 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:29.022+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.027652+0000) 2019-10-13T15:14:29.030 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:29.022+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.027077+0000) 2019-10-13T15:14:29.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:29.022+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.028466+0000) 2019-10-13T15:14:29.031 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:29.022+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.029069+0000) 2019-10-13T15:14:29.032 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:29.024+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.029594+0000) 2019-10-13T15:14:29.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:29.032+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:29.036901+0000) 2019-10-13T15:14:29.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:29.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222230 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:31.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:31.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:31.823228+0000) 2019-10-13T15:14:32.097 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:32.092+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:32.094919+0000) 2019-10-13T15:14:34.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:34.036+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.042715+0000) 2019-10-13T15:14:34.051 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:34.038+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.041578+0000) 2019-10-13T15:14:34.051 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:34.038+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.042255+0000) 2019-10-13T15:14:34.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:34.038+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.043280+0000) 2019-10-13T15:14:34.051 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:34.038+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.044020+0000) 2019-10-13T15:14:34.051 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:34.038+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.044801+0000) 2019-10-13T15:14:34.052 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:34.038+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.044959+0000) 2019-10-13T15:14:34.052 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:34.044+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.049362+0000) 2019-10-13T15:14:34.055 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:34.046+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222262 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:34.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:34.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.421117+0000) 2019-10-13T15:14:34.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:34.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:34.705519+0000) 2019-10-13T15:14:35.590 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:35.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:35.587885+0000) 2019-10-13T15:14:35.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:35.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:35.589176+0000) 2019-10-13T15:14:38.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:38.070+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:38.074013+0000) 2019-10-13T15:14:38.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:38.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:38.299811+0000) 2019-10-13T15:14:39.058 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:39.050+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.056199+0000) 2019-10-13T15:14:39.059 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:39.050+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.057007+0000) 2019-10-13T15:14:39.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:39.050+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.056970+0000) 2019-10-13T15:14:39.060 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:39.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.057971+0000) 2019-10-13T15:14:39.061 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:39.054+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.058910+0000) 2019-10-13T15:14:39.061 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:39.054+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.059277+0000) 2019-10-13T15:14:39.062 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:39.052+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.059815+0000) 2019-10-13T15:14:39.066 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:39.056+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:39.063995+0000) 2019-10-13T15:14:39.070 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:39.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222292 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:41.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:41.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:41.823408+0000) 2019-10-13T15:14:42.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:42.100+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:42.101496+0000) 2019-10-13T15:14:44.073 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:44.066+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.070640+0000) 2019-10-13T15:14:44.074 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:44.066+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.071276+0000) 2019-10-13T15:14:44.074 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:44.066+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.071974+0000) 2019-10-13T15:14:44.075 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:44.068+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.072446+0000) 2019-10-13T15:14:44.075 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:44.066+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.073322+0000) 2019-10-13T15:14:44.076 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:44.070+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.073723+0000) 2019-10-13T15:14:44.077 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:44.070+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.074310+0000) 2019-10-13T15:14:44.080 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:44.072+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.077974+0000) 2019-10-13T15:14:44.084 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:44.074+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222322 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:44.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:44.414+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.421323+0000) 2019-10-13T15:14:44.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:44.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:44.705697+0000) 2019-10-13T15:14:45.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:45.582+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:45.588075+0000) 2019-10-13T15:14:45.591 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:45.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:45.589375+0000) 2019-10-13T15:14:48.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:48.074+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:48.074252+0000) 2019-10-13T15:14:48.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:48.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:48.300035+0000) 2019-10-13T15:14:48.350 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:14:48.351 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:48.393 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:14:48.393 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:14:49.096 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:49.078+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.085535+0000) 2019-10-13T15:14:49.096 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:49.082+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.086093+0000) 2019-10-13T15:14:49.097 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:49.080+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.086862+0000) 2019-10-13T15:14:49.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:49.082+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.087070+0000) 2019-10-13T15:14:49.103 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:49.088+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.092850+0000) 2019-10-13T15:14:49.104 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:49.082+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.087542+0000) 2019-10-13T15:14:49.104 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:49.082+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.088174+0000) 2019-10-13T15:14:49.105 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:49.082+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:49.087941+0000) 2019-10-13T15:14:49.106 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:49.090+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222352 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:51.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:51.817+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:51.823578+0000) 2019-10-13T15:14:52.110 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:52.104+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:52.108020+0000) 2019-10-13T15:14:54.102 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:54.093+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.099636+0000) 2019-10-13T15:14:54.103 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:54.093+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.100789+0000) 2019-10-13T15:14:54.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:54.093+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.101466+0000) 2019-10-13T15:14:54.104 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:54.097+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.102062+0000) 2019-10-13T15:14:54.105 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:54.097+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.102332+0000) 2019-10-13T15:14:54.105 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:54.096+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.103087+0000) 2019-10-13T15:14:54.106 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:54.097+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.103437+0000) 2019-10-13T15:14:54.111 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:54.104+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.108813+0000) 2019-10-13T15:14:54.114 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:54.105+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222384 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:14:54.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:54.413+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.421568+0000) 2019-10-13T15:14:54.708 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:54.701+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:54.705926+0000) 2019-10-13T15:14:55.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:55.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:55.588271+0000) 2019-10-13T15:14:55.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:55.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:55.589563+0000) 2019-10-13T15:14:58.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:58.073+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:58.074476+0000) 2019-10-13T15:14:58.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:58.293+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:58.300233+0000) 2019-10-13T15:14:59.117 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:14:59.109+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.114575+0000) 2019-10-13T15:14:59.118 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:14:59.109+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.116020+0000) 2019-10-13T15:14:59.119 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:14:59.109+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.116842+0000) 2019-10-13T15:14:59.120 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:14:59.109+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.117709+0000) 2019-10-13T15:14:59.122 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:14:59.113+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.119738+0000) 2019-10-13T15:14:59.124 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:14:59.113+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.121509+0000) 2019-10-13T15:14:59.125 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:14:59.120+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.123308+0000) 2019-10-13T15:14:59.126 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:14:59.120+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:14:59.123664+0000) 2019-10-13T15:14:59.130 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:14:59.121+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222414 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:01.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:01.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:01.823739+0000) 2019-10-13T15:15:02.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:02.111+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:02.114601+0000) 2019-10-13T15:15:04.147 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:04.125+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.130645+0000) 2019-10-13T15:15:04.147 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:04.125+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.131333+0000) 2019-10-13T15:15:04.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:04.125+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.131795+0000) 2019-10-13T15:15:04.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:04.131+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.134696+0000) 2019-10-13T15:15:04.159 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:04.139+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.140821+0000) 2019-10-13T15:15:04.159 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:04.125+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.133458+0000) 2019-10-13T15:15:04.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:04.129+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.134165+0000) 2019-10-13T15:15:04.175 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:04.129+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.135498+0000) 2019-10-13T15:15:04.175 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:04.133+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222444 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:04.441 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:04.417+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.421777+0000) 2019-10-13T15:15:04.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:04.705+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:04.706115+0000) 2019-10-13T15:15:05.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:05.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:05.588467+0000) 2019-10-13T15:15:05.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:05.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:05.589758+0000) 2019-10-13T15:15:08.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:08.073+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:08.074713+0000) 2019-10-13T15:15:08.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:08.297+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:08.300429+0000) 2019-10-13T15:15:09.147 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:09.143+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.144520+0000) 2019-10-13T15:15:09.147 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:09.141+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.145261+0000) 2019-10-13T15:15:09.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:09.145+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.145875+0000) 2019-10-13T15:15:09.149 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:09.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.146906+0000) 2019-10-13T15:15:09.149 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:09.145+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.147223+0000) 2019-10-13T15:15:09.150 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:09.145+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.147766+0000) 2019-10-13T15:15:09.151 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:09.145+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.148668+0000) 2019-10-13T15:15:09.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:09.151+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:09.154672+0000) 2019-10-13T15:15:09.157 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:09.153+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222474 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:11.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:11.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:11.823916+0000) 2019-10-13T15:15:12.124 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:12.119+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:12.121087+0000) 2019-10-13T15:15:14.161 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:14.157+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.158827+0000) 2019-10-13T15:15:14.162 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:14.157+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.159669+0000) 2019-10-13T15:15:14.163 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:14.159+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.160644+0000) 2019-10-13T15:15:14.163 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:14.157+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.161072+0000) 2019-10-13T15:15:14.164 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:14.161+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.161818+0000) 2019-10-13T15:15:14.164 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:14.161+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.161919+0000) 2019-10-13T15:15:14.165 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:14.161+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.162247+0000) 2019-10-13T15:15:14.171 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:14.165+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222506 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:14.172 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:14.167+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.169817+0000) 2019-10-13T15:15:14.424 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:14.421+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.421925+0000) 2019-10-13T15:15:14.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:14.705+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:14.706300+0000) 2019-10-13T15:15:15.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:15.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:15.588687+0000) 2019-10-13T15:15:15.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:15.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:15.589946+0000) 2019-10-13T15:15:18.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:18.073+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:18.074916+0000) 2019-10-13T15:15:18.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:18.297+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:18.300636+0000) 2019-10-13T15:15:18.531 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:15:18.532 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:18.535 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:15:18.535 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:19.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:19.171+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.172901+0000) 2019-10-13T15:15:19.176 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:19.173+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.173781+0000) 2019-10-13T15:15:19.176 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:19.173+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.174089+0000) 2019-10-13T15:15:19.178 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:19.173+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.175472+0000) 2019-10-13T15:15:19.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:19.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.179597+0000) 2019-10-13T15:15:19.184 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:19.181+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.181733+0000) 2019-10-13T15:15:19.185 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:19.181+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.182745+0000) 2019-10-13T15:15:19.186 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:19.181+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222536 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:19.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:19.183+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:19.186045+0000) 2019-10-13T15:15:21.826 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:21.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:21.824109+0000) 2019-10-13T15:15:22.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:22.123+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:22.127709+0000) 2019-10-13T15:15:24.190 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:24.185+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.187649+0000) 2019-10-13T15:15:24.192 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:24.185+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.189281+0000) 2019-10-13T15:15:24.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:24.189+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.190383+0000) 2019-10-13T15:15:24.193 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:24.187+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.191254+0000) 2019-10-13T15:15:24.194 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:24.189+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.191578+0000) 2019-10-13T15:15:24.194 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:24.189+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.192091+0000) 2019-10-13T15:15:24.195 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:24.189+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.192354+0000) 2019-10-13T15:15:24.198 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:24.195+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.196265+0000) 2019-10-13T15:15:24.199 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:24.193+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222566 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:24.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:24.421+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.422089+0000) 2019-10-13T15:15:24.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:24.705+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:24.706428+0000) 2019-10-13T15:15:25.591 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:25.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:25.588839+0000) 2019-10-13T15:15:25.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:25.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:25.590156+0000) 2019-10-13T15:15:28.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:28.073+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:28.075147+0000) 2019-10-13T15:15:28.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:28.297+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:28.300828+0000) 2019-10-13T15:15:29.204 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:29.201+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.202035+0000) 2019-10-13T15:15:29.205 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:29.201+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.202724+0000) 2019-10-13T15:15:29.206 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:29.199+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.203445+0000) 2019-10-13T15:15:29.206 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:29.201+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.203089+0000) 2019-10-13T15:15:29.207 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:29.201+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.204043+0000) 2019-10-13T15:15:29.208 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:29.205+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.205919+0000) 2019-10-13T15:15:29.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:29.205+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.207940+0000) 2019-10-13T15:15:29.215 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:29.209+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222596 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:29.220 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:29.215+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:29.217967+0000) 2019-10-13T15:15:31.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:31.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:31.824311+0000) 2019-10-13T15:15:32.137 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:32.131+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:32.134294+0000) 2019-10-13T15:15:34.219 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:34.213+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.215663+0000) 2019-10-13T15:15:34.219 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:34.213+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.216037+0000) 2019-10-13T15:15:34.219 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:34.213+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.216337+0000) 2019-10-13T15:15:34.220 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:34.213+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.216733+0000) 2019-10-13T15:15:34.221 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:34.215+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.218241+0000) 2019-10-13T15:15:34.221 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:34.217+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.218904+0000) 2019-10-13T15:15:34.222 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:34.217+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.219476+0000) 2019-10-13T15:15:34.226 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:34.219+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.224011+0000) 2019-10-13T15:15:34.229 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:34.225+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222628 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:34.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:34.421+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.422241+0000) 2019-10-13T15:15:34.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:34.705+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:34.706600+0000) 2019-10-13T15:15:35.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:35.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:35.589061+0000) 2019-10-13T15:15:35.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:35.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:35.590350+0000) 2019-10-13T15:15:38.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:38.073+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:38.075346+0000) 2019-10-13T15:15:38.303 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:38.297+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:38.301020+0000) 2019-10-13T15:15:39.234 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:39.229+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.231013+0000) 2019-10-13T15:15:39.234 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:39.229+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.231223+0000) 2019-10-13T15:15:39.235 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:39.229+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.232964+0000) 2019-10-13T15:15:39.236 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:39.229+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.233179+0000) 2019-10-13T15:15:39.238 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:39.233+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.234913+0000) 2019-10-13T15:15:39.240 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:39.235+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.237970+0000) 2019-10-13T15:15:39.240 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:39.235+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.238043+0000) 2019-10-13T15:15:39.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:39.237+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:39.238888+0000) 2019-10-13T15:15:39.245 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:39.241+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222658 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:41.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:41.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:41.824514+0000) 2019-10-13T15:15:42.143 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:42.139+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:42.140665+0000) 2019-10-13T15:15:44.248 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:44.241+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.245684+0000) 2019-10-13T15:15:44.249 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:44.245+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.246181+0000) 2019-10-13T15:15:44.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:44.245+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.246566+0000) 2019-10-13T15:15:44.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:44.245+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.247453+0000) 2019-10-13T15:15:44.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:44.245+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.248098+0000) 2019-10-13T15:15:44.251 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:44.247+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.249113+0000) 2019-10-13T15:15:44.252 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:44.245+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.248755+0000) 2019-10-13T15:15:44.256 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:44.251+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.253803+0000) 2019-10-13T15:15:44.261 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:44.257+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:44.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:44.421+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.422429+0000) 2019-10-13T15:15:44.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:44.705+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:44.706807+0000) 2019-10-13T15:15:45.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:45.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:45.589257+0000) 2019-10-13T15:15:45.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:45.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:45.590579+0000) 2019-10-13T15:15:48.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:48.072+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:48.075535+0000) 2019-10-13T15:15:48.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:48.300+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:48.301225+0000) 2019-10-13T15:15:48.638 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:15:48.638 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:48.642 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:15:48.642 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:15:49.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:49.260+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.262052+0000) 2019-10-13T15:15:49.265 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:49.260+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.262406+0000) 2019-10-13T15:15:49.265 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:49.260+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.262786+0000) 2019-10-13T15:15:49.266 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:49.260+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.263285+0000) 2019-10-13T15:15:49.266 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:49.260+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.263720+0000) 2019-10-13T15:15:49.267 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:49.260+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.264821+0000) 2019-10-13T15:15:49.268 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:49.263+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.265469+0000) 2019-10-13T15:15:49.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:49.267+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:49.269727+0000) 2019-10-13T15:15:49.275 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:49.268+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:51.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:51.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:51.824710+0000) 2019-10-13T15:15:52.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:52.143+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:52.147230+0000) 2019-10-13T15:15:54.279 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:54.272+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.276206+0000) 2019-10-13T15:15:54.280 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:54.272+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.277429+0000) 2019-10-13T15:15:54.280 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:54.276+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.277733+0000) 2019-10-13T15:15:54.281 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:54.275+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.278197+0000) 2019-10-13T15:15:54.281 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:54.276+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.278681+0000) 2019-10-13T15:15:54.282 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:54.276+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.279664+0000) 2019-10-13T15:15:54.282 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:54.276+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.280079+0000) 2019-10-13T15:15:54.287 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:54.283+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.284681+0000) 2019-10-13T15:15:54.290 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:54.284+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222750 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:15:54.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:54.420+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.422630+0000) 2019-10-13T15:15:54.709 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:54.704+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:54.706983+0000) 2019-10-13T15:15:55.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:55.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:55.589480+0000) 2019-10-13T15:15:55.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:55.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:55.590774+0000) 2019-10-13T15:15:58.078 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:58.072+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:58.075757+0000) 2019-10-13T15:15:58.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:58.296+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:58.301420+0000) 2019-10-13T15:15:59.294 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:15:59.288+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.291276+0000) 2019-10-13T15:15:59.294 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:15:59.287+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.291999+0000) 2019-10-13T15:15:59.295 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:15:59.288+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.293001+0000) 2019-10-13T15:15:59.296 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:15:59.288+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.293406+0000) 2019-10-13T15:15:59.297 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:15:59.292+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.294277+0000) 2019-10-13T15:15:59.297 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:15:59.292+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.294790+0000) 2019-10-13T15:15:59.298 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:15:59.292+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.295178+0000) 2019-10-13T15:15:59.301 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:15:59.295+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:15:59.298920+0000) 2019-10-13T15:15:59.304 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:15:59.300+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:01.827 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:01.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:01.824920+0000) 2019-10-13T15:16:02.156 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:02.151+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:02.153847+0000) 2019-10-13T15:16:04.308 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:04.300+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.305281+0000) 2019-10-13T15:16:04.308 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:04.304+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.305937+0000) 2019-10-13T15:16:04.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:04.304+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.306309+0000) 2019-10-13T15:16:04.310 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:04.307+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.308390+0000) 2019-10-13T15:16:04.311 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:04.304+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.308833+0000) 2019-10-13T15:16:04.313 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:04.308+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.311358+0000) 2019-10-13T15:16:04.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:04.312+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.315097+0000) 2019-10-13T15:16:04.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:04.311+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.315691+0000) 2019-10-13T15:16:04.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:04.312+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222810 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:04.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:04.420+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.422812+0000) 2019-10-13T15:16:04.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:04.704+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:04.707163+0000) 2019-10-13T15:16:05.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:05.584+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:05.589666+0000) 2019-10-13T15:16:05.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:05.587+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:05.591000+0000) 2019-10-13T15:16:08.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:08.072+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:08.075971+0000) 2019-10-13T15:16:08.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:08.296+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:08.301649+0000) 2019-10-13T15:16:09.322 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:09.316+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.319788+0000) 2019-10-13T15:16:09.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:09.316+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.320694+0000) 2019-10-13T15:16:09.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:09.318+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.321217+0000) 2019-10-13T15:16:09.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:09.316+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.321447+0000) 2019-10-13T15:16:09.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:09.320+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.322211+0000) 2019-10-13T15:16:09.325 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:09.320+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.322539+0000) 2019-10-13T15:16:09.326 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:09.320+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.323913+0000) 2019-10-13T15:16:09.333 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:09.326+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:09.330712+0000) 2019-10-13T15:16:09.334 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:09.328+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222840 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:11.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:11.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:11.825137+0000) 2019-10-13T15:16:12.163 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:12.158+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:12.160452+0000) 2019-10-13T15:16:14.337 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:14.332+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.334757+0000) 2019-10-13T15:16:14.338 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:14.332+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.335758+0000) 2019-10-13T15:16:14.339 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:14.332+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.336081+0000) 2019-10-13T15:16:14.339 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:14.334+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.336823+0000) 2019-10-13T15:16:14.339 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:14.332+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.336687+0000) 2019-10-13T15:16:14.340 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:14.332+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.337230+0000) 2019-10-13T15:16:14.340 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:14.332+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.337690+0000) 2019-10-13T15:16:14.347 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:14.342+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.344391+0000) 2019-10-13T15:16:14.348 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:14.344+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:14.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:14.420+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.422995+0000) 2019-10-13T15:16:14.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:14.704+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:14.707341+0000) 2019-10-13T15:16:15.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:15.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:15.589829+0000) 2019-10-13T15:16:15.593 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:15.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:15.591195+0000) 2019-10-13T15:16:18.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:18.072+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:18.076166+0000) 2019-10-13T15:16:18.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:18.300+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:18.301850+0000) 2019-10-13T15:16:18.861 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:16:18.861 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:18.865 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:16:18.865 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:19.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:19.350+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.352386+0000) 2019-10-13T15:16:19.359 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:19.344+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.349345+0000) 2019-10-13T15:16:19.359 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:19.348+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.349980+0000) 2019-10-13T15:16:19.360 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:19.348+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.350111+0000) 2019-10-13T15:16:19.360 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:19.348+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.350612+0000) 2019-10-13T15:16:19.360 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:19.348+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.352731+0000) 2019-10-13T15:16:19.360 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:19.348+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.353297+0000) 2019-10-13T15:16:19.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:19.354+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:19.357069+0000) 2019-10-13T15:16:19.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:19.356+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222901 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:21.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:21.820+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:21.825343+0000) 2019-10-13T15:16:22.169 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:22.166+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:22.167089+0000) 2019-10-13T15:16:24.366 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:24.360+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.363755+0000) 2019-10-13T15:16:24.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:24.360+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.364525+0000) 2019-10-13T15:16:24.367 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:24.360+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.364967+0000) 2019-10-13T15:16:24.370 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:24.364+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.368020+0000) 2019-10-13T15:16:24.372 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:24.366+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.370008+0000) 2019-10-13T15:16:24.373 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:24.368+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.370189+0000) 2019-10-13T15:16:24.373 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:24.368+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.370749+0000) 2019-10-13T15:16:24.375 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:24.370+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.373179+0000) 2019-10-13T15:16:24.377 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:24.372+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222931 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:24.426 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:24.420+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.423129+0000) 2019-10-13T15:16:24.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:24.704+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:24.707533+0000) 2019-10-13T15:16:25.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:25.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:25.590021+0000) 2019-10-13T15:16:25.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:25.586+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:25.591402+0000) 2019-10-13T15:16:28.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:28.072+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:28.076386+0000) 2019-10-13T15:16:28.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:28.300+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:28.302076+0000) 2019-10-13T15:16:29.380 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:29.374+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.377728+0000) 2019-10-13T15:16:29.381 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:29.376+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.378219+0000) 2019-10-13T15:16:29.381 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:29.376+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.378593+0000) 2019-10-13T15:16:29.381 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:29.376+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.379351+0000) 2019-10-13T15:16:29.382 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:29.376+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.379636+0000) 2019-10-13T15:16:29.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:29.376+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.380505+0000) 2019-10-13T15:16:29.383 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:29.376+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.380706+0000) 2019-10-13T15:16:29.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:29.386+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:29.388624+0000) 2019-10-13T15:16:29.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:29.384+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:31.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:31.824+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:31.825566+0000) 2019-10-13T15:16:32.176 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:32.170+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:32.173701+0000) 2019-10-13T15:16:34.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:34.388+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.393023+0000) 2019-10-13T15:16:34.397 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:34.392+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.393848+0000) 2019-10-13T15:16:34.397 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:34.392+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.394117+0000) 2019-10-13T15:16:34.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:34.392+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.394356+0000) 2019-10-13T15:16:34.398 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:34.392+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.394780+0000) 2019-10-13T15:16:34.398 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:34.392+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.395119+0000) 2019-10-13T15:16:34.399 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:34.394+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.396411+0000) 2019-10-13T15:16:34.401 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:34.394+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.398942+0000) 2019-10-13T15:16:34.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:34.400+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222993 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:34.426 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:34.420+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.423292+0000) 2019-10-13T15:16:34.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:34.704+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:34.707734+0000) 2019-10-13T15:16:35.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:35.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:35.590208+0000) 2019-10-13T15:16:35.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:35.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:35.591590+0000) 2019-10-13T15:16:38.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:38.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:38.076647+0000) 2019-10-13T15:16:38.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:38.300+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:38.302272+0000) 2019-10-13T15:16:39.411 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:39.404+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.407392+0000) 2019-10-13T15:16:39.411 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:39.404+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.407631+0000) 2019-10-13T15:16:39.411 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:39.404+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.408105+0000) 2019-10-13T15:16:39.413 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:39.406+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.409593+0000) 2019-10-13T15:16:39.413 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:39.404+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.408737+0000) 2019-10-13T15:16:39.413 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:39.404+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.409090+0000) 2019-10-13T15:16:39.414 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:39.408+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.410277+0000) 2019-10-13T15:16:39.417 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:39.410+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:39.414341+0000) 2019-10-13T15:16:39.421 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:39.416+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223023 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:41.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:41.824+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:41.825773+0000) 2019-10-13T15:16:42.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:42.178+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:42.180290+0000) 2019-10-13T15:16:44.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:44.415+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.421166+0000) 2019-10-13T15:16:44.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:44.415+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.421628+0000) 2019-10-13T15:16:44.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:44.419+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.422322+0000) 2019-10-13T15:16:44.425 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:44.418+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.422686+0000) 2019-10-13T15:16:44.426 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:44.419+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.423493+0000) 2019-10-13T15:16:44.426 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:44.419+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.423265+0000) 2019-10-13T15:16:44.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:44.423+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.427698+0000) 2019-10-13T15:16:44.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:44.426+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.431404+0000) 2019-10-13T15:16:44.435 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:44.427+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223053 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:44.710 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:44.703+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:44.707941+0000) 2019-10-13T15:16:45.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:45.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:45.590424+0000) 2019-10-13T15:16:45.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:45.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:45.591779+0000) 2019-10-13T15:16:48.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:48.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:48.076882+0000) 2019-10-13T15:16:48.456 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:48.299+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:48.302489+0000) 2019-10-13T15:16:49.068 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:16:49.068 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:49.075 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:16:49.075 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:16:49.439 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:49.431+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.436315+0000) 2019-10-13T15:16:49.440 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:49.434+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.437050+0000) 2019-10-13T15:16:49.440 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:49.431+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.437265+0000) 2019-10-13T15:16:49.441 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:49.435+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.438452+0000) 2019-10-13T15:16:49.441 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:49.435+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.438750+0000) 2019-10-13T15:16:49.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:49.435+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.439196+0000) 2019-10-13T15:16:49.443 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:49.435+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.439596+0000) 2019-10-13T15:16:49.446 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:49.442+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:49.444200+0000) 2019-10-13T15:16:49.451 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:49.443+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:51.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:51.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:51.174015+0000) 2019-10-13T15:16:51.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:51.823+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:51.825939+0000) 2019-10-13T15:16:52.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:52.182+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:52.186810+0000) 2019-10-13T15:16:54.455 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:54.447+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.452417+0000) 2019-10-13T15:16:54.456 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:54.447+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.453128+0000) 2019-10-13T15:16:54.456 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:54.451+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.453735+0000) 2019-10-13T15:16:54.456 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:54.447+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.453512+0000) 2019-10-13T15:16:54.457 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:54.450+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.454663+0000) 2019-10-13T15:16:54.457 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:54.451+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.455124+0000) 2019-10-13T15:16:54.458 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:54.451+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.455609+0000) 2019-10-13T15:16:54.460 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:54.454+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.458284+0000) 2019-10-13T15:16:54.466 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:54.459+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223115 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:16:54.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:54.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:54.708123+0000) 2019-10-13T15:16:55.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:55.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:55.590619+0000) 2019-10-13T15:16:55.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:55.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:55.592022+0000) 2019-10-13T15:16:58.079 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:58.075+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:58.077075+0000) 2019-10-13T15:16:58.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:58.299+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:58.302687+0000) 2019-10-13T15:16:59.470 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:16:59.463+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.467439+0000) 2019-10-13T15:16:59.471 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:16:59.463+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.467753+0000) 2019-10-13T15:16:59.472 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:16:59.463+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.468467+0000) 2019-10-13T15:16:59.472 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:16:59.463+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.468666+0000) 2019-10-13T15:16:59.473 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:16:59.467+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.470769+0000) 2019-10-13T15:16:59.474 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:16:59.467+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.471728+0000) 2019-10-13T15:16:59.476 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:16:59.470+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.473682+0000) 2019-10-13T15:16:59.479 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:16:59.474+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:16:59.477016+0000) 2019-10-13T15:16:59.480 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:16:59.475+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223145 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:01.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:01.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:01.174236+0000) 2019-10-13T15:17:01.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:01.823+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:01.826159+0000) 2019-10-13T15:17:02.196 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:02.190+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:02.193423+0000) 2019-10-13T15:17:04.484 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:04.479+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.481785+0000) 2019-10-13T15:17:04.485 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:04.479+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.482124+0000) 2019-10-13T15:17:04.485 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:04.479+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.482455+0000) 2019-10-13T15:17:04.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:04.479+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.482773+0000) 2019-10-13T15:17:04.486 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:04.479+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.483983+0000) 2019-10-13T15:17:04.487 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:04.479+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.484127+0000) 2019-10-13T15:17:04.487 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:04.482+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.484475+0000) 2019-10-13T15:17:04.494 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:04.486+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.491969+0000) 2019-10-13T15:17:04.496 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:04.487+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223175 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:04.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:04.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:04.708299+0000) 2019-10-13T15:17:05.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:05.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:05.590844+0000) 2019-10-13T15:17:05.594 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:05.590+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:05.592291+0000) 2019-10-13T15:17:08.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:08.075+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:08.077281+0000) 2019-10-13T15:17:08.305 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:08.299+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:08.302886+0000) 2019-10-13T15:17:09.500 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:09.491+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.496890+0000) 2019-10-13T15:17:09.500 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:09.491+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.497280+0000) 2019-10-13T15:17:09.500 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:09.495+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.497870+0000) 2019-10-13T15:17:09.501 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:09.495+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.498377+0000) 2019-10-13T15:17:09.502 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:09.495+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.499130+0000) 2019-10-13T15:17:09.502 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:09.494+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.499511+0000) 2019-10-13T15:17:09.502 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:09.495+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.499934+0000) 2019-10-13T15:17:09.508 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:09.502+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:09.505416+0000) 2019-10-13T15:17:09.510 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:09.503+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223207 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:11.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:11.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:11.174459+0000) 2019-10-13T15:17:11.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:11.823+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:11.826365+0000) 2019-10-13T15:17:12.202 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:12.193+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:12.199875+0000) 2019-10-13T15:17:14.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:14.507+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.511585+0000) 2019-10-13T15:17:14.515 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:14.507+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.511781+0000) 2019-10-13T15:17:14.515 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:14.509+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.512265+0000) 2019-10-13T15:17:14.515 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:14.507+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.512578+0000) 2019-10-13T15:17:14.516 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:14.507+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.513507+0000) 2019-10-13T15:17:14.516 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:14.511+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.514187+0000) 2019-10-13T15:17:14.518 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:14.511+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.515023+0000) 2019-10-13T15:17:14.522 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:14.517+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.520293+0000) 2019-10-13T15:17:14.526 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:14.519+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:14.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:14.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:14.708501+0000) 2019-10-13T15:17:15.593 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:15.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:15.591033+0000) 2019-10-13T15:17:15.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:15.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:15.592509+0000) 2019-10-13T15:17:18.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:18.071+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:18.077478+0000) 2019-10-13T15:17:18.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:18.299+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:18.303122+0000) 2019-10-13T15:17:19.203 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:17:19.203 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:19.216 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:17:19.216 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:19.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:19.523+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.526539+0000) 2019-10-13T15:17:19.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:19.523+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.528615+0000) 2019-10-13T15:17:19.531 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:19.523+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.528828+0000) 2019-10-13T15:17:19.532 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:19.525+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.529318+0000) 2019-10-13T15:17:19.532 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:19.523+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.529659+0000) 2019-10-13T15:17:19.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:19.527+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.530268+0000) 2019-10-13T15:17:19.533 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:19.527+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.530619+0000) 2019-10-13T15:17:19.539 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:19.533+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:19.536792+0000) 2019-10-13T15:17:19.540 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:19.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223267 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:21.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:21.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:21.174701+0000) 2019-10-13T15:17:21.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:21.823+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:21.826583+0000) 2019-10-13T15:17:22.209 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:22.201+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:22.206394+0000) 2019-10-13T15:17:24.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:24.535+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.540805+0000) 2019-10-13T15:17:24.544 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:24.535+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.541325+0000) 2019-10-13T15:17:24.544 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:24.537+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.541903+0000) 2019-10-13T15:17:24.547 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:24.539+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.544542+0000) 2019-10-13T15:17:24.547 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:24.539+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.544948+0000) 2019-10-13T15:17:24.548 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:24.539+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.545394+0000) 2019-10-13T15:17:24.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:24.539+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.545687+0000) 2019-10-13T15:17:24.553 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:24.545+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.550547+0000) 2019-10-13T15:17:24.555 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:24.547+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223297 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:24.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:24.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:24.708696+0000) 2019-10-13T15:17:25.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:25.587+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:25.591254+0000) 2019-10-13T15:17:25.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:25.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:25.592766+0000) 2019-10-13T15:17:28.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:28.075+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:28.077685+0000) 2019-10-13T15:17:28.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:28.303+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:28.303352+0000) 2019-10-13T15:17:29.558 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:29.551+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.555878+0000) 2019-10-13T15:17:29.559 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:29.551+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.556638+0000) 2019-10-13T15:17:29.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:29.553+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.559129+0000) 2019-10-13T15:17:29.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:29.555+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.559353+0000) 2019-10-13T15:17:29.564 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:29.555+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.561717+0000) 2019-10-13T15:17:29.564 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:29.559+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.561795+0000) 2019-10-13T15:17:29.565 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:29.559+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.562887+0000) 2019-10-13T15:17:29.570 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:29.563+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223329 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:29.571 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:29.565+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:29.569011+0000) 2019-10-13T15:17:31.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:31.171+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:31.174940+0000) 2019-10-13T15:17:31.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:31.823+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:31.826788+0000) 2019-10-13T15:17:32.215 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:32.209+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:32.212885+0000) 2019-10-13T15:17:34.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:34.567+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.571149+0000) 2019-10-13T15:17:34.574 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:34.567+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.571536+0000) 2019-10-13T15:17:34.574 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:34.567+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.571892+0000) 2019-10-13T15:17:34.576 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:34.571+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.574282+0000) 2019-10-13T15:17:34.579 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:34.571+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.576607+0000) 2019-10-13T15:17:34.582 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:34.573+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.579428+0000) 2019-10-13T15:17:34.583 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:34.577+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.580864+0000) 2019-10-13T15:17:34.583 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:34.575+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.581313+0000) 2019-10-13T15:17:34.585 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:34.579+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:34.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:34.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:34.708866+0000) 2019-10-13T15:17:35.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:35.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:35.592955+0000) 2019-10-13T15:17:35.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:35.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:35.591451+0000) 2019-10-13T15:17:38.080 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:38.075+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:38.077915+0000) 2019-10-13T15:17:38.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:38.299+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:38.303544+0000) 2019-10-13T15:17:39.590 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:39.583+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.587390+0000) 2019-10-13T15:17:39.591 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:39.583+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.586360+0000) 2019-10-13T15:17:39.592 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:39.585+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.588363+0000) 2019-10-13T15:17:39.592 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:39.583+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.588624+0000) 2019-10-13T15:17:39.592 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:39.583+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.589110+0000) 2019-10-13T15:17:39.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:39.583+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.589389+0000) 2019-10-13T15:17:39.593 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:39.587+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.590241+0000) 2019-10-13T15:17:39.596 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:39.589+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:39.594156+0000) 2019-10-13T15:17:39.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:39.591+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223389 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:41.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:41.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:41.175140+0000) 2019-10-13T15:17:41.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:41.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:41.826995+0000) 2019-10-13T15:17:42.222 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:42.217+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:42.219604+0000) 2019-10-13T15:17:44.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:44.594+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.599776+0000) 2019-10-13T15:17:44.603 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:44.597+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.600619+0000) 2019-10-13T15:17:44.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:44.594+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.601424+0000) 2019-10-13T15:17:44.604 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:44.594+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.601645+0000) 2019-10-13T15:17:44.605 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:44.598+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.602235+0000) 2019-10-13T15:17:44.605 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:44.598+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.602424+0000) 2019-10-13T15:17:44.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:44.598+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.603939+0000) 2019-10-13T15:17:44.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:44.605+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.610903+0000) 2019-10-13T15:17:44.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:44.606+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223419 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:44.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:44.706+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:44.709082+0000) 2019-10-13T15:17:45.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:45.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:45.591648+0000) 2019-10-13T15:17:45.595 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:45.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:45.593151+0000) 2019-10-13T15:17:48.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:48.074+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:48.078118+0000) 2019-10-13T15:17:48.306 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:48.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:48.303762+0000) 2019-10-13T15:17:49.325 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:17:49.325 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:49.329 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:17:49.329 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:17:49.618 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:49.610+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.615228+0000) 2019-10-13T15:17:49.618 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:49.610+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.615580+0000) 2019-10-13T15:17:49.619 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:49.613+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.616850+0000) 2019-10-13T15:17:49.619 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:49.610+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.616245+0000) 2019-10-13T15:17:49.620 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:49.610+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.617208+0000) 2019-10-13T15:17:49.620 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:49.610+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.617724+0000) 2019-10-13T15:17:49.621 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:49.614+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.618200+0000) 2019-10-13T15:17:49.624 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:49.617+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:49.621723+0000) 2019-10-13T15:17:49.627 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:49.618+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223451 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:51.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:51.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:51.175356+0000) 2019-10-13T15:17:51.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:51.826+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:51.827228+0000) 2019-10-13T15:17:52.228 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:52.225+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:52.226094+0000) 2019-10-13T15:17:54.631 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:54.622+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.628345+0000) 2019-10-13T15:17:54.631 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:54.622+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.628636+0000) 2019-10-13T15:17:54.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:54.622+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.629259+0000) 2019-10-13T15:17:54.632 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:54.626+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.629974+0000) 2019-10-13T15:17:54.633 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:54.626+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.630588+0000) 2019-10-13T15:17:54.634 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:54.626+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.630675+0000) 2019-10-13T15:17:54.634 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:54.629+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.632166+0000) 2019-10-13T15:17:54.641 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:54.634+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:17:54.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:54.633+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.638903+0000) 2019-10-13T15:17:54.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:54.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:54.709265+0000) 2019-10-13T15:17:55.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:55.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:55.591882+0000) 2019-10-13T15:17:55.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:55.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:55.593360+0000) 2019-10-13T15:17:58.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:58.074+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:58.078315+0000) 2019-10-13T15:17:58.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:58.302+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:58.303960+0000) 2019-10-13T15:17:59.644 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:17:59.638+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.641576+0000) 2019-10-13T15:17:59.645 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:17:59.638+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.642383+0000) 2019-10-13T15:17:59.645 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:17:59.638+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.643058+0000) 2019-10-13T15:17:59.646 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:17:59.638+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.643691+0000) 2019-10-13T15:17:59.647 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:17:59.638+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.644723+0000) 2019-10-13T15:17:59.648 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:17:59.641+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.646137+0000) 2019-10-13T15:17:59.649 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:17:59.642+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.646928+0000) 2019-10-13T15:17:59.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:17:59.645+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:17:59.651919+0000) 2019-10-13T15:17:59.656 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:17:59.650+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:01.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:01.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:01.175511+0000) 2019-10-13T15:18:01.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:01.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:01.827423+0000) 2019-10-13T15:18:02.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:02.229+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:02.232510+0000) 2019-10-13T15:18:04.660 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:04.650+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.657447+0000) 2019-10-13T15:18:04.660 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:04.654+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.657817+0000) 2019-10-13T15:18:04.661 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:04.654+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.658448+0000) 2019-10-13T15:18:04.661 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:04.654+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.658807+0000) 2019-10-13T15:18:04.662 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:04.653+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.659717+0000) 2019-10-13T15:18:04.662 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:04.654+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.660134+0000) 2019-10-13T15:18:04.663 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:04.654+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.660623+0000) 2019-10-13T15:18:04.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:04.657+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.661775+0000) 2019-10-13T15:18:04.672 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:04.662+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:04.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:04.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:04.709448+0000) 2019-10-13T15:18:05.594 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:05.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:05.592059+0000) 2019-10-13T15:18:05.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:05.589+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:05.593559+0000) 2019-10-13T15:18:08.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:08.074+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:08.078516+0000) 2019-10-13T15:18:08.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:08.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:08.304158+0000) 2019-10-13T15:18:09.675 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:09.666+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.672822+0000) 2019-10-13T15:18:09.676 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:09.669+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.673149+0000) 2019-10-13T15:18:09.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:09.666+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.673550+0000) 2019-10-13T15:18:09.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:09.670+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.674063+0000) 2019-10-13T15:18:09.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:09.670+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.674756+0000) 2019-10-13T15:18:09.678 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:09.670+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.674485+0000) 2019-10-13T15:18:09.678 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:09.670+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.676112+0000) 2019-10-13T15:18:09.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:09.677+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:09.681673+0000) 2019-10-13T15:18:09.687 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:09.678+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223573 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:11.178 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:11.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:11.175745+0000) 2019-10-13T15:18:11.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:11.822+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:11.827624+0000) 2019-10-13T15:18:12.241 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:12.233+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:12.238983+0000) 2019-10-13T15:18:14.692 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:14.682+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.688948+0000) 2019-10-13T15:18:14.692 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:14.685+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.689436+0000) 2019-10-13T15:18:14.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:14.686+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.691215+0000) 2019-10-13T15:18:14.694 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:14.686+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.691946+0000) 2019-10-13T15:18:14.695 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:14.686+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.692362+0000) 2019-10-13T15:18:14.697 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:14.690+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.694401+0000) 2019-10-13T15:18:14.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:14.690+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.694498+0000) 2019-10-13T15:18:14.702 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:14.694+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:14.703 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:14.697+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.700821+0000) 2019-10-13T15:18:14.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:14.702+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:14.709609+0000) 2019-10-13T15:18:15.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:15.586+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:15.592263+0000) 2019-10-13T15:18:15.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:15.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:15.593727+0000) 2019-10-13T15:18:18.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:18.078+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:18.078749+0000) 2019-10-13T15:18:18.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:18.302+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:18.304352+0000) 2019-10-13T15:18:19.671 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:18:19.671 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:19.678 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:18:19.678 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:19.705 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:19.698+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.702184+0000) 2019-10-13T15:18:19.705 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:19.698+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.702271+0000) 2019-10-13T15:18:19.706 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:19.696+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.703190+0000) 2019-10-13T15:18:19.706 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:19.698+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.702707+0000) 2019-10-13T15:18:19.706 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:19.698+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.702768+0000) 2019-10-13T15:18:19.707 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:19.698+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.704058+0000) 2019-10-13T15:18:19.707 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:19.698+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.704124+0000) 2019-10-13T15:18:19.710 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:19.700+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:19.707567+0000) 2019-10-13T15:18:19.717 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:19.710+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:21.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:21.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:21.175947+0000) 2019-10-13T15:18:21.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:21.826+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:21.827827+0000) 2019-10-13T15:18:22.253 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:22.248+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:22.250325+0000) 2019-10-13T15:18:24.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:24.706+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.709778+0000) 2019-10-13T15:18:24.719 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:24.714+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.716525+0000) 2019-10-13T15:18:24.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:24.714+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.719622+0000) 2019-10-13T15:18:24.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:24.714+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.720989+0000) 2019-10-13T15:18:24.724 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:24.716+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.721397+0000) 2019-10-13T15:18:24.724 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:24.714+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.721327+0000) 2019-10-13T15:18:24.725 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:24.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.721933+0000) 2019-10-13T15:18:24.725 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:24.718+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.721979+0000) 2019-10-13T15:18:24.731 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:24.722+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223662 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:24.732 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:24.724+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:24.729254+0000) 2019-10-13T15:18:25.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:25.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:25.592452+0000) 2019-10-13T15:18:25.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:25.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:25.593916+0000) 2019-10-13T15:18:28.081 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:28.078+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:28.078955+0000) 2019-10-13T15:18:28.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:28.298+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:28.304547+0000) 2019-10-13T15:18:29.734 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:29.726+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.731675+0000) 2019-10-13T15:18:29.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:29.726+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.732016+0000) 2019-10-13T15:18:29.735 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:29.728+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.733337+0000) 2019-10-13T15:18:29.736 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:29.726+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.733521+0000) 2019-10-13T15:18:29.738 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:29.730+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.735530+0000) 2019-10-13T15:18:29.738 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:29.730+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.735783+0000) 2019-10-13T15:18:29.740 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:29.730+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.737681+0000) 2019-10-13T15:18:29.746 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:29.738+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223695 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:29.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:29.740+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:29.744965+0000) 2019-10-13T15:18:31.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:31.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:31.176146+0000) 2019-10-13T15:18:31.830 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:31.826+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:31.828057+0000) 2019-10-13T15:18:32.259 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:32.256+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:32.256970+0000) 2019-10-13T15:18:34.712 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:34.706+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.709993+0000) 2019-10-13T15:18:34.748 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:34.738+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.745326+0000) 2019-10-13T15:18:34.750 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:34.742+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.747702+0000) 2019-10-13T15:18:34.751 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:34.742+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.748518+0000) 2019-10-13T15:18:34.751 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:34.742+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.748835+0000) 2019-10-13T15:18:34.752 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:34.744+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.749299+0000) 2019-10-13T15:18:34.752 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:34.742+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.749434+0000) 2019-10-13T15:18:34.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:34.746+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.750093+0000) 2019-10-13T15:18:34.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:34.748+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:34.753009+0000) 2019-10-13T15:18:34.760 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:34.754+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223724 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:35.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:35.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:35.592623+0000) 2019-10-13T15:18:35.596 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:35.588+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:35.594124+0000) 2019-10-13T15:18:38.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:38.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:38.079158+0000) 2019-10-13T15:18:38.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:38.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:38.304747+0000) 2019-10-13T15:18:39.763 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:39.753+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.760888+0000) 2019-10-13T15:18:39.764 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:39.757+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.761730+0000) 2019-10-13T15:18:39.764 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:39.756+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.762334+0000) 2019-10-13T15:18:39.765 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:39.757+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.762400+0000) 2019-10-13T15:18:39.765 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:39.757+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.762736+0000) 2019-10-13T15:18:39.766 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:39.757+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.763217+0000) 2019-10-13T15:18:39.766 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:39.757+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.763715+0000) 2019-10-13T15:18:39.772 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:39.764+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:39.769412+0000) 2019-10-13T15:18:39.775 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:39.765+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223755 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:41.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:41.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:41.176366+0000) 2019-10-13T15:18:41.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:41.821+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:41.828291+0000) 2019-10-13T15:18:42.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:42.256+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:42.262776+0000) 2019-10-13T15:18:44.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:44.709+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.710195+0000) 2019-10-13T15:18:44.778 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:44.769+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.775529+0000) 2019-10-13T15:18:44.779 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:44.769+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.776566+0000) 2019-10-13T15:18:44.781 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:44.773+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.778724+0000) 2019-10-13T15:18:44.781 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:44.773+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.779139+0000) 2019-10-13T15:18:44.782 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:44.773+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.779543+0000) 2019-10-13T15:18:44.782 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:44.773+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.780050+0000) 2019-10-13T15:18:44.789 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:44.781+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223784 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:44.803 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:44.772+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.777120+0000) 2019-10-13T15:18:44.804 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:44.796+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:44.801460+0000) 2019-10-13T15:18:45.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:45.585+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:45.592801+0000) 2019-10-13T15:18:45.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:45.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:45.594297+0000) 2019-10-13T15:18:48.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:48.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:48.079358+0000) 2019-10-13T15:18:48.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:48.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:48.304962+0000) 2019-10-13T15:18:49.792 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:49.785+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.789102+0000) 2019-10-13T15:18:49.792 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:49.785+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.789550+0000) 2019-10-13T15:18:49.795 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:49.788+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.792507+0000) 2019-10-13T15:18:49.797 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:49.785+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.790094+0000) 2019-10-13T15:18:49.797 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:49.785+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.791605+0000) 2019-10-13T15:18:49.798 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:49.785+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.792533+0000) 2019-10-13T15:18:49.798 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:49.785+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.792732+0000) 2019-10-13T15:18:49.804 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:49.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223816 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:49.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:49.796+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:49.801668+0000) 2019-10-13T15:18:49.860 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:18:49.860 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:49.892 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:18:49.893 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:18:51.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:51.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:51.176579+0000) 2019-10-13T15:18:51.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:51.825+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:51.828528+0000) 2019-10-13T15:18:52.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:52.268+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:52.269305+0000) 2019-10-13T15:18:54.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:54.709+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.710425+0000) 2019-10-13T15:18:54.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:54.801+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.805149+0000) 2019-10-13T15:18:54.808 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:54.801+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.805651+0000) 2019-10-13T15:18:54.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:54.805+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.806206+0000) 2019-10-13T15:18:54.809 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:54.800+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.806550+0000) 2019-10-13T15:18:54.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:54.805+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.806650+0000) 2019-10-13T15:18:54.811 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:54.805+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.808936+0000) 2019-10-13T15:18:54.813 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:54.809+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.810684+0000) 2019-10-13T15:18:54.816 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:54.808+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:54.813404+0000) 2019-10-13T15:18:54.819 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:54.813+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223847 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:55.595 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:55.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:55.592932+0000) 2019-10-13T15:18:55.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:55.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:55.594502+0000) 2019-10-13T15:18:58.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:58.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:58.079558+0000) 2019-10-13T15:18:58.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:58.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:58.305169+0000) 2019-10-13T15:18:59.823 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:18:59.817+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.820138+0000) 2019-10-13T15:18:59.823 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:18:59.817+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.820642+0000) 2019-10-13T15:18:59.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:18:59.817+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.821662+0000) 2019-10-13T15:18:59.825 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:18:59.816+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.822325+0000) 2019-10-13T15:18:59.825 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:18:59.821+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.822883+0000) 2019-10-13T15:18:59.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:18:59.821+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.823667+0000) 2019-10-13T15:18:59.826 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:18:59.821+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.823736+0000) 2019-10-13T15:18:59.832 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:18:59.829+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223877 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:18:59.833 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:18:59.824+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:18:59.831313+0000) 2019-10-13T15:19:01.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:01.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:01.176798+0000) 2019-10-13T15:19:01.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:01.825+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:01.828725+0000) 2019-10-13T15:19:02.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:02.272+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:02.275897+0000) 2019-10-13T15:19:04.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:04.709+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.710616+0000) 2019-10-13T15:19:04.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:04.829+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.833449+0000) 2019-10-13T15:19:04.837 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:04.833+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.834087+0000) 2019-10-13T15:19:04.837 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:04.833+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.834943+0000) 2019-10-13T15:19:04.838 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:04.833+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.835290+0000) 2019-10-13T15:19:04.838 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:04.833+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.835464+0000) 2019-10-13T15:19:04.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:04.832+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.836922+0000) 2019-10-13T15:19:04.840 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:04.833+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.837093+0000) 2019-10-13T15:19:04.846 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:04.840+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:04.844320+0000) 2019-10-13T15:19:04.847 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:04.841+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223908 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:05.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:05.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:05.593131+0000) 2019-10-13T15:19:05.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:05.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:05.594706+0000) 2019-10-13T15:19:08.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:08.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:08.079755+0000) 2019-10-13T15:19:08.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:08.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:08.305364+0000) 2019-10-13T15:19:09.851 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:09.845+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.848756+0000) 2019-10-13T15:19:09.852 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:09.845+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.848572+0000) 2019-10-13T15:19:09.852 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:09.844+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.849262+0000) 2019-10-13T15:19:09.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:09.849+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.849898+0000) 2019-10-13T15:19:09.853 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:09.849+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.850319+0000) 2019-10-13T15:19:09.853 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:09.849+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.850926+0000) 2019-10-13T15:19:09.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:09.849+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.852077+0000) 2019-10-13T15:19:09.858 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:09.848+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:09.856040+0000) 2019-10-13T15:19:09.861 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:09.857+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223940 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:11.179 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:11.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:11.176974+0000) 2019-10-13T15:19:11.831 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:11.825+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:11.828955+0000) 2019-10-13T15:19:12.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:12.280+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:12.282527+0000) 2019-10-13T15:19:14.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:14.709+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.710852+0000) 2019-10-13T15:19:14.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:14.861+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.863141+0000) 2019-10-13T15:19:14.866 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:14.856+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.863816+0000) 2019-10-13T15:19:14.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:14.861+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.865057+0000) 2019-10-13T15:19:14.868 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:14.861+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.865423+0000) 2019-10-13T15:19:14.869 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:14.865+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.866072+0000) 2019-10-13T15:19:14.869 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:14.865+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.866375+0000) 2019-10-13T15:19:14.869 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:14.865+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.866643+0000) 2019-10-13T15:19:14.876 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:14.869+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223971 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:14.876 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:14.868+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:14.873439+0000) 2019-10-13T15:19:15.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:15.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:15.593308+0000) 2019-10-13T15:19:15.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:15.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:15.594906+0000) 2019-10-13T15:19:18.082 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:18.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:18.079959+0000) 2019-10-13T15:19:18.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:18.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:18.305567+0000) 2019-10-13T15:19:19.879 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:19.873+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.876401+0000) 2019-10-13T15:19:19.881 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:19.877+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.878985+0000) 2019-10-13T15:19:19.883 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:19.877+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.880826+0000) 2019-10-13T15:19:19.884 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:19.880+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.881246+0000) 2019-10-13T15:19:19.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:19.881+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.881825+0000) 2019-10-13T15:19:19.886 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:19.881+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.883398+0000) 2019-10-13T15:19:19.886 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:19.881+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.883718+0000) 2019-10-13T15:19:19.890 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:19.885+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224001 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:19.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:19.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:19.891149+0000) 2019-10-13T15:19:19.976 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:19:19.976 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:19.980 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:19:19.980 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:21.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:21.173+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:21.177177+0000) 2019-10-13T15:19:21.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:21.825+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:21.829165+0000) 2019-10-13T15:19:22.291 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:22.288+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:22.289181+0000) 2019-10-13T15:19:24.713 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:24.709+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.711055+0000) 2019-10-13T15:19:24.894 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:24.889+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.891338+0000) 2019-10-13T15:19:24.895 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:24.889+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.892463+0000) 2019-10-13T15:19:24.897 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:24.893+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.894502+0000) 2019-10-13T15:19:24.897 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:24.893+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.894987+0000) 2019-10-13T15:19:24.898 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:24.891+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.895272+0000) 2019-10-13T15:19:24.899 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:24.893+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.896432+0000) 2019-10-13T15:19:24.901 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:24.897+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.898582+0000) 2019-10-13T15:19:24.905 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:24.901+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:24.907 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:24.903+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:24.905132+0000) 2019-10-13T15:19:25.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:25.589+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:25.593474+0000) 2019-10-13T15:19:25.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:25.591+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:25.595077+0000) 2019-10-13T15:19:28.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:28.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:28.080165+0000) 2019-10-13T15:19:28.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:28.305+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:28.305788+0000) 2019-10-13T15:19:29.909 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:29.905+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.906170+0000) 2019-10-13T15:19:29.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:29.905+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.906456+0000) 2019-10-13T15:19:29.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:29.907+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.908969+0000) 2019-10-13T15:19:29.913 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:29.909+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.910666+0000) 2019-10-13T15:19:29.913 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:29.909+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.911183+0000) 2019-10-13T15:19:29.915 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:29.909+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.912385+0000) 2019-10-13T15:19:29.915 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:29.909+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.913090+0000) 2019-10-13T15:19:29.920 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:29.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224064 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:29.921 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:29.915+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:29.919081+0000) 2019-10-13T15:19:31.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:31.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:31.177402+0000) 2019-10-13T15:19:31.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:31.829+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:31.829371+0000) 2019-10-13T15:19:32.298 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:32.291+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:32.295855+0000) 2019-10-13T15:19:34.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:34.708+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.711284+0000) 2019-10-13T15:19:34.923 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:34.916+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.919911+0000) 2019-10-13T15:19:34.925 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:34.920+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.923090+0000) 2019-10-13T15:19:34.926 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:34.920+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.923539+0000) 2019-10-13T15:19:34.927 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:34.920+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.924415+0000) 2019-10-13T15:19:34.929 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:34.923+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.926625+0000) 2019-10-13T15:19:34.930 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:34.924+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.927431+0000) 2019-10-13T15:19:34.930 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:34.924+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.928109+0000) 2019-10-13T15:19:34.933 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:34.927+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:34.930774+0000) 2019-10-13T15:19:34.933 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:34.928+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224095 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:35.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:35.588+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:35.593652+0000) 2019-10-13T15:19:35.597 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:35.591+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:35.595247+0000) 2019-10-13T15:19:38.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:38.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:38.080409+0000) 2019-10-13T15:19:38.308 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:38.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:38.305945+0000) 2019-10-13T15:19:39.938 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:39.932+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.935193+0000) 2019-10-13T15:19:39.938 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:39.931+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.935894+0000) 2019-10-13T15:19:39.939 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:39.932+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.936151+0000) 2019-10-13T15:19:39.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:39.932+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.936355+0000) 2019-10-13T15:19:39.939 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:39.932+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.936608+0000) 2019-10-13T15:19:39.940 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:39.932+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.937066+0000) 2019-10-13T15:19:39.941 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:39.936+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.939019+0000) 2019-10-13T15:19:39.948 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:39.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224125 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:39.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:39.943+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:39.945671+0000) 2019-10-13T15:19:41.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:41.172+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:41.177615+0000) 2019-10-13T15:19:41.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:41.824+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:41.829604+0000) 2019-10-13T15:19:42.305 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:42.299+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:42.302575+0000) 2019-10-13T15:19:44.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:44.708+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.711513+0000) 2019-10-13T15:19:44.951 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:44.944+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.948766+0000) 2019-10-13T15:19:44.953 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:44.947+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.951047+0000) 2019-10-13T15:19:44.954 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:44.948+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.951298+0000) 2019-10-13T15:19:44.954 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:44.948+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.951723+0000) 2019-10-13T15:19:44.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:44.948+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.953689+0000) 2019-10-13T15:19:44.956 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:44.952+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.954060+0000) 2019-10-13T15:19:44.957 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:44.952+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.954613+0000) 2019-10-13T15:19:44.962 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:44.956+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224156 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:44.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:44.959+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:44.960837+0000) 2019-10-13T15:19:45.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:45.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:45.593850+0000) 2019-10-13T15:19:45.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:45.591+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:45.595494+0000) 2019-10-13T15:19:48.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:48.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:48.080648+0000) 2019-10-13T15:19:48.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:48.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:48.306176+0000) 2019-10-13T15:19:49.967 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:49.960+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.964023+0000) 2019-10-13T15:19:49.967 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:49.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.964279+0000) 2019-10-13T15:19:49.967 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:49.960+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.964306+0000) 2019-10-13T15:19:49.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:49.963+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.964765+0000) 2019-10-13T15:19:49.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:49.960+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.965353+0000) 2019-10-13T15:19:49.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:49.960+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.965657+0000) 2019-10-13T15:19:49.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:49.964+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.966054+0000) 2019-10-13T15:19:49.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:49.967+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:49.971663+0000) 2019-10-13T15:19:49.979 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:49.972+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224186 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:50.040 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:19:50.040 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:50.045 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:19:50.045 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:19:51.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:51.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:51.177802+0000) 2019-10-13T15:19:51.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:51.828+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:51.829842+0000) 2019-10-13T15:19:52.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:52.307+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:52.309117+0000) 2019-10-13T15:19:54.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:54.708+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.711715+0000) 2019-10-13T15:19:54.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:54.972+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.977697+0000) 2019-10-13T15:19:54.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:54.975+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.978212+0000) 2019-10-13T15:19:54.982 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:54.976+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.979350+0000) 2019-10-13T15:19:54.982 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:54.976+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.979669+0000) 2019-10-13T15:19:54.983 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:54.976+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.980257+0000) 2019-10-13T15:19:54.983 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:54.976+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.980560+0000) 2019-10-13T15:19:54.983 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:54.976+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.981073+0000) 2019-10-13T15:19:54.988 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:54.983+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:54.985415+0000) 2019-10-13T15:19:54.991 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:19:54.984+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224217 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:19:55.596 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:55.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:55.594069+0000) 2019-10-13T15:19:55.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:55.591+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:55.595691+0000) 2019-10-13T15:19:58.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:58.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:58.080848+0000) 2019-10-13T15:19:58.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:58.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:58.306394+0000) 2019-10-13T15:19:59.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:19:59.988+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.992216+0000) 2019-10-13T15:19:59.995 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:19:59.988+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.993011+0000) 2019-10-13T15:19:59.997 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:19:59.992+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.994189+0000) 2019-10-13T15:19:59.997 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:19:59.991+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.993779+0000) 2019-10-13T15:19:59.998 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:19:59.992+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.994899+0000) 2019-10-13T15:19:59.998 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:19:59.992+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.995160+0000) 2019-10-13T15:19:59.998 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:19:59.992+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:19:59.995483+0000) 2019-10-13T15:20:00.004 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:19:59.999+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:00.002125+0000) 2019-10-13T15:20:00.006 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:00.000+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224245 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:01.180 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:01.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:01.178019+0000) 2019-10-13T15:20:01.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:01.828+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:01.830050+0000) 2019-10-13T15:20:02.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:02.311+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:02.315783+0000) 2019-10-13T15:20:04.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:04.708+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:04.711956+0000) 2019-10-13T15:20:05.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:05.003+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.007033+0000) 2019-10-13T15:20:05.011 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:05.004+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.008935+0000) 2019-10-13T15:20:05.012 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:05.004+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.009403+0000) 2019-10-13T15:20:05.012 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:05.008+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.010152+0000) 2019-10-13T15:20:05.013 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:05.008+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.010398+0000) 2019-10-13T15:20:05.013 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:05.008+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.010744+0000) 2019-10-13T15:20:05.014 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:05.008+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.011196+0000) 2019-10-13T15:20:05.020 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:05.015+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.017648+0000) 2019-10-13T15:20:05.020 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:05.016+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224276 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:05.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:05.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.594263+0000) 2019-10-13T15:20:05.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:05.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:05.595860+0000) 2019-10-13T15:20:08.083 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:08.080+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:08.081053+0000) 2019-10-13T15:20:08.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:08.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:08.306635+0000) 2019-10-13T15:20:10.023 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:10.016+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.020208+0000) 2019-10-13T15:20:10.023 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:10.016+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.021081+0000) 2019-10-13T15:20:10.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:10.020+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.021985+0000) 2019-10-13T15:20:10.025 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:10.020+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.022104+0000) 2019-10-13T15:20:10.029 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:10.023+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.026753+0000) 2019-10-13T15:20:10.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:10.027+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.028259+0000) 2019-10-13T15:20:10.031 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:10.024+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.028431+0000) 2019-10-13T15:20:10.033 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:10.028+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:10.030422+0000) 2019-10-13T15:20:10.035 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:10.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224306 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:11.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:11.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:11.178244+0000) 2019-10-13T15:20:11.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:11.828+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:11.830274+0000) 2019-10-13T15:20:12.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:12.319+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:12.322336+0000) 2019-10-13T15:20:14.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:14.708+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:14.712192+0000) 2019-10-13T15:20:15.039 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:15.035+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.036397+0000) 2019-10-13T15:20:15.039 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:15.032+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.036927+0000) 2019-10-13T15:20:15.040 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:15.032+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.037602+0000) 2019-10-13T15:20:15.040 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:15.036+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.037896+0000) 2019-10-13T15:20:15.041 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:15.036+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.038107+0000) 2019-10-13T15:20:15.041 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:15.036+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.038463+0000) 2019-10-13T15:20:15.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:15.036+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.038902+0000) 2019-10-13T15:20:15.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:15.043+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.046076+0000) 2019-10-13T15:20:15.049 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:15.044+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224337 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:15.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:15.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.594429+0000) 2019-10-13T15:20:15.598 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:15.591+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:15.596035+0000) 2019-10-13T15:20:18.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:18.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:18.081282+0000) 2019-10-13T15:20:18.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:18.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:18.306849+0000) 2019-10-13T15:20:20.053 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:20.048+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.050367+0000) 2019-10-13T15:20:20.053 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:20.048+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.050845+0000) 2019-10-13T15:20:20.055 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:20.048+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.052727+0000) 2019-10-13T15:20:20.055 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:20.048+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.053090+0000) 2019-10-13T15:20:20.057 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:20.052+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.054491+0000) 2019-10-13T15:20:20.057 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:20.051+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.054968+0000) 2019-10-13T15:20:20.057 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:20.052+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.055118+0000) 2019-10-13T15:20:20.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:20.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:20.064 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:20.059+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:20.061246+0000) 2019-10-13T15:20:20.152 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:20:20.152 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:20.156 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:20:20.156 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:21.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:21.176+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:21.178456+0000) 2019-10-13T15:20:21.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:21.828+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:21.830473+0000) 2019-10-13T15:20:22.331 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:22.327+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:22.328146+0000) 2019-10-13T15:20:24.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:24.712+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:24.712475+0000) 2019-10-13T15:20:25.067 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:25.060+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.064663+0000) 2019-10-13T15:20:25.068 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:25.060+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.065242+0000) 2019-10-13T15:20:25.068 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:25.063+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.065569+0000) 2019-10-13T15:20:25.068 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:25.064+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.065871+0000) 2019-10-13T15:20:25.069 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:25.064+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.066479+0000) 2019-10-13T15:20:25.069 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:25.064+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.067318+0000) 2019-10-13T15:20:25.070 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:25.064+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.067812+0000) 2019-10-13T15:20:25.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:25.071+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.073016+0000) 2019-10-13T15:20:25.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:25.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224397 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:25.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:25.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.594601+0000) 2019-10-13T15:20:25.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:25.595+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:25.596220+0000) 2019-10-13T15:20:28.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:28.076+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:28.081486+0000) 2019-10-13T15:20:28.309 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:28.304+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:28.307068+0000) 2019-10-13T15:20:30.082 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:30.075+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.079488+0000) 2019-10-13T15:20:30.082 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:30.075+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.079873+0000) 2019-10-13T15:20:30.085 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:30.078+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.082395+0000) 2019-10-13T15:20:30.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:30.079+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.083974+0000) 2019-10-13T15:20:30.088 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:30.083+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.085864+0000) 2019-10-13T15:20:30.090 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:30.083+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.087828+0000) 2019-10-13T15:20:30.091 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:30.086+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.088933+0000) 2019-10-13T15:20:30.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:30.087+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:30.090318+0000) 2019-10-13T15:20:30.094 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:30.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:31.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:31.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:31.178672+0000) 2019-10-13T15:20:31.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:31.827+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:31.830678+0000) 2019-10-13T15:20:32.337 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:32.330+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:32.334639+0000) 2019-10-13T15:20:34.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:34.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:34.712707+0000) 2019-10-13T15:20:35.097 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:35.091+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.094773+0000) 2019-10-13T15:20:35.099 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:35.091+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.096362+0000) 2019-10-13T15:20:35.100 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:35.094+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.097497+0000) 2019-10-13T15:20:35.100 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:35.091+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.097687+0000) 2019-10-13T15:20:35.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:35.095+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.098255+0000) 2019-10-13T15:20:35.101 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:35.095+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.098579+0000) 2019-10-13T15:20:35.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:35.095+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.099265+0000) 2019-10-13T15:20:35.104 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:35.098+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.102371+0000) 2019-10-13T15:20:35.109 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:35.103+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224459 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:35.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:35.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.594794+0000) 2019-10-13T15:20:35.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:35.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:35.596392+0000) 2019-10-13T15:20:38.084 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:38.075+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:38.081686+0000) 2019-10-13T15:20:38.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:38.303+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:38.307269+0000) 2019-10-13T15:20:40.113 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:40.107+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.110464+0000) 2019-10-13T15:20:40.114 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:40.107+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.110573+0000) 2019-10-13T15:20:40.114 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:40.106+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.111860+0000) 2019-10-13T15:20:40.115 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:40.107+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.111535+0000) 2019-10-13T15:20:40.115 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:40.107+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.111972+0000) 2019-10-13T15:20:40.115 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:40.107+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.112283+0000) 2019-10-13T15:20:40.115 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:40.107+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.112704+0000) 2019-10-13T15:20:40.120 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:40.114+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:40.117388+0000) 2019-10-13T15:20:40.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:40.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224486 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:41.181 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:41.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:41.178883+0000) 2019-10-13T15:20:41.833 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:41.827+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:41.830878+0000) 2019-10-13T15:20:42.343 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:42.338+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:42.341158+0000) 2019-10-13T15:20:44.715 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:44.711+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:44.712906+0000) 2019-10-13T15:20:45.128 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:45.119+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.124458+0000) 2019-10-13T15:20:45.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:45.119+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.124930+0000) 2019-10-13T15:20:45.128 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:45.119+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.125198+0000) 2019-10-13T15:20:45.129 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:45.123+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.126007+0000) 2019-10-13T15:20:45.129 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:45.123+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.126781+0000) 2019-10-13T15:20:45.130 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:45.122+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.127614+0000) 2019-10-13T15:20:45.130 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:45.123+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.128314+0000) 2019-10-13T15:20:45.135 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:45.130+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.132676+0000) 2019-10-13T15:20:45.139 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:45.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224519 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:45.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:45.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.594926+0000) 2019-10-13T15:20:45.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:45.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:45.596564+0000) 2019-10-13T15:20:48.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:48.079+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:48.081974+0000) 2019-10-13T15:20:48.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:48.303+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:48.307479+0000) 2019-10-13T15:20:50.142 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:50.135+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.139507+0000) 2019-10-13T15:20:50.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:50.135+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.139950+0000) 2019-10-13T15:20:50.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:50.135+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.140815+0000) 2019-10-13T15:20:50.144 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:50.135+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.141263+0000) 2019-10-13T15:20:50.144 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:50.138+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.141934+0000) 2019-10-13T15:20:50.145 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:50.135+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.141548+0000) 2019-10-13T15:20:50.145 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:50.139+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.142786+0000) 2019-10-13T15:20:50.148 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:50.142+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:50.146323+0000) 2019-10-13T15:20:50.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:50.147+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224548 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:50.341 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:20:50.341 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:50.345 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:20:50.345 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:20:51.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:51.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:51.179098+0000) 2019-10-13T15:20:51.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:51.827+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:51.831119+0000) 2019-10-13T15:20:52.350 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:52.346+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:52.347661+0000) 2019-10-13T15:20:54.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:54.711+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:54.713107+0000) 2019-10-13T15:20:55.156 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:55.147+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.153261+0000) 2019-10-13T15:20:55.157 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:55.151+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.154016+0000) 2019-10-13T15:20:55.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:20:55.151+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.154429+0000) 2019-10-13T15:20:55.157 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:55.150+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.154940+0000) 2019-10-13T15:20:55.158 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:20:55.151+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.156170+0000) 2019-10-13T15:20:55.159 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:55.151+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.156228+0000) 2019-10-13T15:20:55.159 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:20:55.151+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.156829+0000) 2019-10-13T15:20:55.166 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:20:55.158+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.163668+0000) 2019-10-13T15:20:55.167 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:20:55.159+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:20:55.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:20:55.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.595103+0000) 2019-10-13T15:20:55.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:20:55.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:55.596714+0000) 2019-10-13T15:20:58.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:20:58.079+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:58.082166+0000) 2019-10-13T15:20:58.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:20:58.303+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:20:58.307708+0000) 2019-10-13T15:21:00.170 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:00.162+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.167721+0000) 2019-10-13T15:21:00.170 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:00.163+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.167822+0000) 2019-10-13T15:21:00.171 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:00.163+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.168992+0000) 2019-10-13T15:21:00.172 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:00.167+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.169839+0000) 2019-10-13T15:21:00.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:00.163+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.169591+0000) 2019-10-13T15:21:00.173 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:00.167+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.170286+0000) 2019-10-13T15:21:00.173 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:00.167+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.170500+0000) 2019-10-13T15:21:00.180 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:00.174+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:00.177777+0000) 2019-10-13T15:21:00.182 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:00.175+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224608 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:01.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:01.175+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:01.179306+0000) 2019-10-13T15:21:01.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:01.827+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:01.831323+0000) 2019-10-13T15:21:02.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:02.350+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:02.354872+0000) 2019-10-13T15:21:04.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:04.711+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:04.713303+0000) 2019-10-13T15:21:05.185 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:05.179+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.182854+0000) 2019-10-13T15:21:05.186 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:05.179+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.183484+0000) 2019-10-13T15:21:05.188 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:05.182+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.185458+0000) 2019-10-13T15:21:05.189 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:05.183+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.186827+0000) 2019-10-13T15:21:05.190 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:05.183+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.187772+0000) 2019-10-13T15:21:05.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:05.183+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.188229+0000) 2019-10-13T15:21:05.191 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:05.183+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.188521+0000) 2019-10-13T15:21:05.194 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:05.186+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.191552+0000) 2019-10-13T15:21:05.197 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:05.191+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224641 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:05.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:05.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.595280+0000) 2019-10-13T15:21:05.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:05.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:05.596885+0000) 2019-10-13T15:21:08.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:08.079+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:08.082366+0000) 2019-10-13T15:21:08.310 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:08.307+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:08.307909+0000) 2019-10-13T15:21:10.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:10.195+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.198199+0000) 2019-10-13T15:21:10.203 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:10.195+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.200712+0000) 2019-10-13T15:21:10.204 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:10.195+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.201219+0000) 2019-10-13T15:21:10.206 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:10.199+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.203285+0000) 2019-10-13T15:21:10.206 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:10.199+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.203527+0000) 2019-10-13T15:21:10.206 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:10.198+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.203524+0000) 2019-10-13T15:21:10.207 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:10.199+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.204291+0000) 2019-10-13T15:21:10.207 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:10.202+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:10.204555+0000) 2019-10-13T15:21:10.212 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:10.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224670 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:11.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:11.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:11.179502+0000) 2019-10-13T15:21:11.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:11.827+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:11.831561+0000) 2019-10-13T15:21:12.364 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:12.358+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:12.361935+0000) 2019-10-13T15:21:14.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:14.711+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:14.713510+0000) 2019-10-13T15:21:15.215 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:15.207+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.212960+0000) 2019-10-13T15:21:15.217 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:15.211+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.214484+0000) 2019-10-13T15:21:15.218 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:15.211+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.215409+0000) 2019-10-13T15:21:15.218 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:15.211+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.214906+0000) 2019-10-13T15:21:15.219 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:15.214+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.216504+0000) 2019-10-13T15:21:15.219 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:15.211+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.216387+0000) 2019-10-13T15:21:15.220 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:15.211+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.217678+0000) 2019-10-13T15:21:15.223 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:15.218+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.220578+0000) 2019-10-13T15:21:15.227 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:15.219+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224702 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:15.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:15.591+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.595469+0000) 2019-10-13T15:21:15.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:15.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:15.597070+0000) 2019-10-13T15:21:18.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:18.079+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.082570+0000) 2019-10-13T15:21:18.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:18.307+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:18.308110+0000) 2019-10-13T15:21:20.232 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:20.223+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.229002+0000) 2019-10-13T15:21:20.232 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:20.223+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.229406+0000) 2019-10-13T15:21:20.233 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:20.227+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.230574+0000) 2019-10-13T15:21:20.233 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:20.227+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.230247+0000) 2019-10-13T15:21:20.234 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:20.226+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.231193+0000) 2019-10-13T15:21:20.234 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:20.227+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.230790+0000) 2019-10-13T15:21:20.234 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:20.227+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.231573+0000) 2019-10-13T15:21:20.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:20.234+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:20.237261+0000) 2019-10-13T15:21:20.242 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:20.235+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224730 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:20.388 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:21:20.389 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:20.392 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:21:20.392 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:21.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:21.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:21.179716+0000) 2019-10-13T15:21:21.834 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:21.831+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:21.831764+0000) 2019-10-13T15:21:22.371 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:22.366+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:22.368973+0000) 2019-10-13T15:21:24.716 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:24.707+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:24.713702+0000) 2019-10-13T15:21:25.246 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:25.242+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.244200+0000) 2019-10-13T15:21:25.248 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:25.243+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.246261+0000) 2019-10-13T15:21:25.249 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:25.243+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.246699+0000) 2019-10-13T15:21:25.249 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:25.243+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.246984+0000) 2019-10-13T15:21:25.250 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:25.243+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.247609+0000) 2019-10-13T15:21:25.250 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:25.243+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.247729+0000) 2019-10-13T15:21:25.251 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:25.243+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.248382+0000) 2019-10-13T15:21:25.257 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:25.251+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:25.258 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:25.250+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.255191+0000) 2019-10-13T15:21:25.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:25.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.595680+0000) 2019-10-13T15:21:25.599 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:25.594+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:25.597265+0000) 2019-10-13T15:21:28.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:28.078+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.082807+0000) 2019-10-13T15:21:28.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:28.302+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:28.308299+0000) 2019-10-13T15:21:30.261 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:30.254+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.258626+0000) 2019-10-13T15:21:30.262 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:30.254+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.259233+0000) 2019-10-13T15:21:30.262 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:30.254+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.259553+0000) 2019-10-13T15:21:30.263 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:30.254+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.259835+0000) 2019-10-13T15:21:30.263 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:30.254+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.260426+0000) 2019-10-13T15:21:30.263 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:30.254+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.260702+0000) 2019-10-13T15:21:30.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:30.254+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.261178+0000) 2019-10-13T15:21:30.270 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:30.262+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:30.267972+0000) 2019-10-13T15:21:30.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:30.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:31.182 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:31.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:31.179910+0000) 2019-10-13T15:21:31.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:31.826+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:31.831971+0000) 2019-10-13T15:21:32.379 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:32.374+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:32.375994+0000) 2019-10-13T15:21:34.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:34.710+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:34.713897+0000) 2019-10-13T15:21:35.275 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:35.266+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.272075+0000) 2019-10-13T15:21:35.275 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:35.269+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.272820+0000) 2019-10-13T15:21:35.276 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:35.270+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.273974+0000) 2019-10-13T15:21:35.277 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:35.270+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.274251+0000) 2019-10-13T15:21:35.277 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:35.270+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.274585+0000) 2019-10-13T15:21:35.278 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:35.270+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.275179+0000) 2019-10-13T15:21:35.278 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:35.270+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.275909+0000) 2019-10-13T15:21:35.282 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:35.273+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.279943+0000) 2019-10-13T15:21:35.286 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:35.278+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224824 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:35.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:35.590+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.595861+0000) 2019-10-13T15:21:35.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:35.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:35.597439+0000) 2019-10-13T15:21:38.085 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:38.082+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.083044+0000) 2019-10-13T15:21:38.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:38.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:38.308494+0000) 2019-10-13T15:21:40.289 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:40.282+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.286512+0000) 2019-10-13T15:21:40.290 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:40.281+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.287287+0000) 2019-10-13T15:21:40.290 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:40.282+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.288226+0000) 2019-10-13T15:21:40.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:40.286+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.290231+0000) 2019-10-13T15:21:40.293 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:40.286+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.290808+0000) 2019-10-13T15:21:40.293 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:40.286+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.291296+0000) 2019-10-13T15:21:40.295 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:40.286+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.292454+0000) 2019-10-13T15:21:40.299 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:40.293+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:40.296962+0000) 2019-10-13T15:21:40.302 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:40.294+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224852 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:41.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:41.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:41.180093+0000) 2019-10-13T15:21:41.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:41.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:41.832186+0000) 2019-10-13T15:21:42.385 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:42.377+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:42.383042+0000) 2019-10-13T15:21:44.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:44.710+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:44.714156+0000) 2019-10-13T15:21:45.305 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:45.298+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.302369+0000) 2019-10-13T15:21:45.307 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:45.298+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.304392+0000) 2019-10-13T15:21:45.307 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:45.298+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.304938+0000) 2019-10-13T15:21:45.308 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:45.301+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.305723+0000) 2019-10-13T15:21:45.308 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:45.298+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.305343+0000) 2019-10-13T15:21:45.309 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:45.302+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.306393+0000) 2019-10-13T15:21:45.309 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:45.302+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.306744+0000) 2019-10-13T15:21:45.315 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:45.309+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.313338+0000) 2019-10-13T15:21:45.316 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:45.310+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224884 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:45.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:45.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.596040+0000) 2019-10-13T15:21:45.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:45.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:45.597611+0000) 2019-10-13T15:21:48.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:48.078+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.083242+0000) 2019-10-13T15:21:48.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:48.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:48.308700+0000) 2019-10-13T15:21:50.322 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:50.314+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.319152+0000) 2019-10-13T15:21:50.322 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:50.314+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.319218+0000) 2019-10-13T15:21:50.323 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:50.317+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.320484+0000) 2019-10-13T15:21:50.323 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:50.314+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.319717+0000) 2019-10-13T15:21:50.323 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:50.314+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.320888+0000) 2019-10-13T15:21:50.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:50.318+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.322764+0000) 2019-10-13T15:21:50.325 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:50.318+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.323313+0000) 2019-10-13T15:21:50.332 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:50.326+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224915 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:50.334 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:50.325+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:50.331916+0000) 2019-10-13T15:21:50.627 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:21:50.628 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:50.631 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:21:50.631 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:21:51.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:51.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:51.180230+0000) 2019-10-13T15:21:51.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:51.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:51.832395+0000) 2019-10-13T15:21:52.393 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:52.385+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:52.390114+0000) 2019-10-13T15:21:54.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:54.710+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:54.714400+0000) 2019-10-13T15:21:55.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:21:55.326+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.333564+0000) 2019-10-13T15:21:55.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:55.329+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.333948+0000) 2019-10-13T15:21:55.337 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:21:55.330+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.334589+0000) 2019-10-13T15:21:55.337 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:55.330+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.335113+0000) 2019-10-13T15:21:55.338 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:55.330+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.335486+0000) 2019-10-13T15:21:55.341 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:55.334+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.338990+0000) 2019-10-13T15:21:55.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:21:55.334+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.340560+0000) 2019-10-13T15:21:55.346 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:21:55.337+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.343765+0000) 2019-10-13T15:21:55.347 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:21:55.338+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224946 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:21:55.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:21:55.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.596221+0000) 2019-10-13T15:21:55.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:21:55.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:55.597790+0000) 2019-10-13T15:21:58.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:21:58.078+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.083475+0000) 2019-10-13T15:21:58.311 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:21:58.302+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:21:58.308950+0000) 2019-10-13T15:22:00.350 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:00.341+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.347607+0000) 2019-10-13T15:22:00.350 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:00.342+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.348136+0000) 2019-10-13T15:22:00.351 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:00.342+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.348689+0000) 2019-10-13T15:22:00.351 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:00.342+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.348807+0000) 2019-10-13T15:22:00.352 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:00.342+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.349154+0000) 2019-10-13T15:22:00.353 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:00.346+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.350977+0000) 2019-10-13T15:22:00.354 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:00.346+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.351946+0000) 2019-10-13T15:22:00.361 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:00.354+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224975 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:00.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:00.353+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:00.359008+0000) 2019-10-13T15:22:01.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:01.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:01.180407+0000) 2019-10-13T15:22:01.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:01.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:01.832653+0000) 2019-10-13T15:22:02.400 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:02.393+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:02.397384+0000) 2019-10-13T15:22:04.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:04.714+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:04.714595+0000) 2019-10-13T15:22:05.365 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:05.358+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.362262+0000) 2019-10-13T15:22:05.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:05.357+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.362687+0000) 2019-10-13T15:22:05.366 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:05.358+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.362937+0000) 2019-10-13T15:22:05.366 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:05.358+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.363556+0000) 2019-10-13T15:22:05.366 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:05.358+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.363861+0000) 2019-10-13T15:22:05.367 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:05.358+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.364497+0000) 2019-10-13T15:22:05.367 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:05.358+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.365005+0000) 2019-10-13T15:22:05.370 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:05.365+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.368229+0000) 2019-10-13T15:22:05.376 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:05.366+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225008 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:05.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:05.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.596392+0000) 2019-10-13T15:22:05.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:05.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:05.597965+0000) 2019-10-13T15:22:08.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:08.082+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.083673+0000) 2019-10-13T15:22:08.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:08.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:08.309109+0000) 2019-10-13T15:22:10.380 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:10.370+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.377580+0000) 2019-10-13T15:22:10.381 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:10.374+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.378768+0000) 2019-10-13T15:22:10.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:10.374+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.381267+0000) 2019-10-13T15:22:10.385 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:10.377+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.382684+0000) 2019-10-13T15:22:10.385 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:10.378+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.382830+0000) 2019-10-13T15:22:10.386 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:10.378+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.384095+0000) 2019-10-13T15:22:10.387 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:10.378+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.385332+0000) 2019-10-13T15:22:10.390 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:10.382+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225037 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:10.390 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:10.381+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:10.387952+0000) 2019-10-13T15:22:11.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:11.174+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:11.180602+0000) 2019-10-13T15:22:11.835 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:11.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:11.832897+0000) 2019-10-13T15:22:12.407 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:12.401+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:12.404470+0000) 2019-10-13T15:22:14.717 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:14.714+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:14.714800+0000) 2019-10-13T15:22:15.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:15.386+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.391422+0000) 2019-10-13T15:22:15.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:15.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.391997+0000) 2019-10-13T15:22:15.395 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:15.386+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.392061+0000) 2019-10-13T15:22:15.396 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:15.389+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.392515+0000) 2019-10-13T15:22:15.396 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:15.386+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.392971+0000) 2019-10-13T15:22:15.396 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:15.386+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.393378+0000) 2019-10-13T15:22:15.397 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:15.390+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.394553+0000) 2019-10-13T15:22:15.403 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:15.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225068 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:15.406 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:15.397+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.403805+0000) 2019-10-13T15:22:15.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:15.594+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.596572+0000) 2019-10-13T15:22:15.600 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:15.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:15.598138+0000) 2019-10-13T15:22:18.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:18.082+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.083878+0000) 2019-10-13T15:22:18.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:18.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:18.309307+0000) 2019-10-13T15:22:20.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:20.398+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.404620+0000) 2019-10-13T15:22:20.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:20.398+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.404992+0000) 2019-10-13T15:22:20.408 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:20.402+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.406080+0000) 2019-10-13T15:22:20.409 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:20.402+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.406600+0000) 2019-10-13T15:22:20.409 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:20.401+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.407135+0000) 2019-10-13T15:22:20.410 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:20.402+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.407272+0000) 2019-10-13T15:22:20.410 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:20.402+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.407648+0000) 2019-10-13T15:22:20.413 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:20.405+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:20.410532+0000) 2019-10-13T15:22:20.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:20.410+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225097 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:20.724 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:22:20.724 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:20.727 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:22:20.727 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:21.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:21.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:21.180814+0000) 2019-10-13T15:22:21.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:21.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:21.833098+0000) 2019-10-13T15:22:22.414 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:22.405+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:22.411983+0000) 2019-10-13T15:22:24.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:24.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:24.715009+0000) 2019-10-13T15:22:25.422 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:25.413+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.419207+0000) 2019-10-13T15:22:25.422 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:25.413+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.419860+0000) 2019-10-13T15:22:25.423 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:25.413+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.420631+0000) 2019-10-13T15:22:25.423 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:25.413+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.421095+0000) 2019-10-13T15:22:25.424 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:25.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.421462+0000) 2019-10-13T15:22:25.424 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:25.417+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.421882+0000) 2019-10-13T15:22:25.425 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:25.417+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.422868+0000) 2019-10-13T15:22:25.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:25.421+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.426906+0000) 2019-10-13T15:22:25.433 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:25.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225130 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:25.599 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:25.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.596749+0000) 2019-10-13T15:22:25.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:25.597+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:25.598309+0000) 2019-10-13T15:22:28.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:28.077+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.084111+0000) 2019-10-13T15:22:28.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:28.301+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:28.309541+0000) 2019-10-13T15:22:30.437 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:30.429+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.434413+0000) 2019-10-13T15:22:30.438 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:30.429+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.435205+0000) 2019-10-13T15:22:30.440 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:30.433+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.437501+0000) 2019-10-13T15:22:30.440 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:30.429+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.437496+0000) 2019-10-13T15:22:30.442 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:30.433+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.439766+0000) 2019-10-13T15:22:30.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:30.433+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.441618+0000) 2019-10-13T15:22:30.444 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:30.437+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.442074+0000) 2019-10-13T15:22:30.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:30.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225159 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:30.452 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:30.445+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:30.450081+0000) 2019-10-13T15:22:31.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:31.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:31.180991+0000) 2019-10-13T15:22:31.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:31.829+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:31.833303+0000) 2019-10-13T15:22:32.421 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:32.417+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:32.418640+0000) 2019-10-13T15:22:34.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:34.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:34.715211+0000) 2019-10-13T15:22:35.451 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:35.441+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.448699+0000) 2019-10-13T15:22:35.453 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:35.445+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.450652+0000) 2019-10-13T15:22:35.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:35.445+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.450961+0000) 2019-10-13T15:22:35.454 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:35.445+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.451392+0000) 2019-10-13T15:22:35.455 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:35.445+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.452274+0000) 2019-10-13T15:22:35.455 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:35.445+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.452495+0000) 2019-10-13T15:22:35.456 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:35.449+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.454017+0000) 2019-10-13T15:22:35.461 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:35.453+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.458646+0000) 2019-10-13T15:22:35.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:35.453+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225190 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:35.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:35.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.596927+0000) 2019-10-13T15:22:35.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:35.593+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:35.598501+0000) 2019-10-13T15:22:38.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:38.081+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.084333+0000) 2019-10-13T15:22:38.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:38.309+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:38.309774+0000) 2019-10-13T15:22:40.465 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:40.461+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.462671+0000) 2019-10-13T15:22:40.466 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:40.461+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.463817+0000) 2019-10-13T15:22:40.467 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:40.460+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.464530+0000) 2019-10-13T15:22:40.467 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:40.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.464511+0000) 2019-10-13T15:22:40.468 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:40.461+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.464977+0000) 2019-10-13T15:22:40.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:40.465+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.465976+0000) 2019-10-13T15:22:40.469 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:40.465+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.466104+0000) 2019-10-13T15:22:40.472 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:40.464+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:40.470049+0000) 2019-10-13T15:22:40.477 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:40.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225219 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:41.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:41.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:41.181188+0000) 2019-10-13T15:22:41.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:41.829+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:41.833500+0000) 2019-10-13T15:22:42.428 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:42.424+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:42.425841+0000) 2019-10-13T15:22:44.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:44.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:44.715438+0000) 2019-10-13T15:22:45.481 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:45.477+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.478077+0000) 2019-10-13T15:22:45.481 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:45.477+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.479151+0000) 2019-10-13T15:22:45.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:45.477+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.479609+0000) 2019-10-13T15:22:45.482 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:45.477+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.479997+0000) 2019-10-13T15:22:45.483 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:45.477+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.480682+0000) 2019-10-13T15:22:45.483 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:45.476+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.481298+0000) 2019-10-13T15:22:45.484 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:45.477+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.481423+0000) 2019-10-13T15:22:45.489 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:45.480+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.486655+0000) 2019-10-13T15:22:45.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:45.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225252 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:45.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:45.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.597117+0000) 2019-10-13T15:22:45.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:45.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:45.598705+0000) 2019-10-13T15:22:48.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:48.081+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.084539+0000) 2019-10-13T15:22:48.312 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:48.309+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:48.309973+0000) 2019-10-13T15:22:50.495 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:50.489+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.493050+0000) 2019-10-13T15:22:50.496 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:50.488+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.493874+0000) 2019-10-13T15:22:50.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:50.489+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.493458+0000) 2019-10-13T15:22:50.497 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:50.493+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.494437+0000) 2019-10-13T15:22:50.497 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:50.493+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.495062+0000) 2019-10-13T15:22:50.498 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:50.493+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.495552+0000) 2019-10-13T15:22:50.499 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:50.493+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.497288+0000) 2019-10-13T15:22:50.505 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:50.496+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:50.502710+0000) 2019-10-13T15:22:50.506 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:50.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225281 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:50.838 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:22:50.838 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:50.842 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:22:50.842 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:22:51.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:51.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:51.181397+0000) 2019-10-13T15:22:51.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:51.833+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:51.833709+0000) 2019-10-13T15:22:52.435 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:52.428+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:52.432699+0000) 2019-10-13T15:22:54.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:54.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:54.715638+0000) 2019-10-13T15:22:55.509 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:22:55.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.506734+0000) 2019-10-13T15:22:55.511 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:55.505+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.508465+0000) 2019-10-13T15:22:55.511 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:55.505+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.509093+0000) 2019-10-13T15:22:55.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:22:55.509+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.509812+0000) 2019-10-13T15:22:55.514 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:55.504+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.511499+0000) 2019-10-13T15:22:55.514 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:22:55.509+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.511938+0000) 2019-10-13T15:22:55.516 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:55.513+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.513858+0000) 2019-10-13T15:22:55.520 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:22:55.512+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.518197+0000) 2019-10-13T15:22:55.521 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:22:55.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225311 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:22:55.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:22:55.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.597243+0000) 2019-10-13T15:22:55.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:22:55.596+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:55.598891+0000) 2019-10-13T15:22:58.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:22:58.081+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.084761+0000) 2019-10-13T15:22:58.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:22:58.309+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:22:58.310181+0000) 2019-10-13T15:23:00.524 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:00.517+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.521575+0000) 2019-10-13T15:23:00.525 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:00.521+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.522115+0000) 2019-10-13T15:23:00.526 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:00.521+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.523846+0000) 2019-10-13T15:23:00.527 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:00.520+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.524600+0000) 2019-10-13T15:23:00.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:00.525+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.526624+0000) 2019-10-13T15:23:00.529 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:00.525+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.527217+0000) 2019-10-13T15:23:00.531 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:00.525+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.529273+0000) 2019-10-13T15:23:00.536 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:00.528+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:00.533413+0000) 2019-10-13T15:23:00.536 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:00.529+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225341 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:01.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:01.177+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:01.181582+0000) 2019-10-13T15:23:01.836 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:01.833+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:01.833956+0000) 2019-10-13T15:23:02.442 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:02.436+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:02.439188+0000) 2019-10-13T15:23:04.718 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:04.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:04.715868+0000) 2019-10-13T15:23:05.539 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:05.533+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.536456+0000) 2019-10-13T15:23:05.539 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:05.533+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.536578+0000) 2019-10-13T15:23:05.540 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:05.533+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.537256+0000) 2019-10-13T15:23:05.542 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:05.537+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.539564+0000) 2019-10-13T15:23:05.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:05.532+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.539941+0000) 2019-10-13T15:23:05.543 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:05.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.540153+0000) 2019-10-13T15:23:05.543 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:05.537+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.541007+0000) 2019-10-13T15:23:05.550 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:05.544+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.548271+0000) 2019-10-13T15:23:05.551 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:05.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225373 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:05.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:05.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.597401+0000) 2019-10-13T15:23:05.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:05.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:05.599016+0000) 2019-10-13T15:23:08.087 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:08.081+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.084976+0000) 2019-10-13T15:23:08.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:08.309+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:08.310415+0000) 2019-10-13T15:23:10.554 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:10.549+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.551645+0000) 2019-10-13T15:23:10.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:10.553+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.553998+0000) 2019-10-13T15:23:10.557 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:10.553+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.554488+0000) 2019-10-13T15:23:10.557 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:10.548+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.555147+0000) 2019-10-13T15:23:10.559 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:10.553+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.556786+0000) 2019-10-13T15:23:10.560 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:10.553+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.557018+0000) 2019-10-13T15:23:10.560 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:10.553+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.557397+0000) 2019-10-13T15:23:10.567 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:10.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225403 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:10.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:10.560+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:10.564716+0000) 2019-10-13T15:23:11.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:11.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:11.181793+0000) 2019-10-13T15:23:11.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:11.833+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:11.834165+0000) 2019-10-13T15:23:12.448 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:12.444+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:12.445669+0000) 2019-10-13T15:23:14.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:14.713+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:14.716067+0000) 2019-10-13T15:23:15.570 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:15.565+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.567610+0000) 2019-10-13T15:23:15.571 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:15.565+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.568989+0000) 2019-10-13T15:23:15.572 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:15.565+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.569260+0000) 2019-10-13T15:23:15.572 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:15.564+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.569769+0000) 2019-10-13T15:23:15.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:15.569+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.570081+0000) 2019-10-13T15:23:15.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:15.569+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.570404+0000) 2019-10-13T15:23:15.573 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:15.569+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.570951+0000) 2019-10-13T15:23:15.581 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:15.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225432 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:15.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:15.576+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.580310+0000) 2019-10-13T15:23:15.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:15.593+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.597578+0000) 2019-10-13T15:23:15.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:15.592+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:15.599140+0000) 2019-10-13T15:23:18.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:18.081+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.085208+0000) 2019-10-13T15:23:18.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:18.309+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:18.310608+0000) 2019-10-13T15:23:20.586 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:20.580+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.583233+0000) 2019-10-13T15:23:20.586 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:20.580+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.583507+0000) 2019-10-13T15:23:20.587 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:20.580+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.585214+0000) 2019-10-13T15:23:20.588 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:20.580+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.585377+0000) 2019-10-13T15:23:20.588 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:20.584+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.586207+0000) 2019-10-13T15:23:20.589 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:20.584+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.586600+0000) 2019-10-13T15:23:20.589 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:20.584+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.586769+0000) 2019-10-13T15:23:20.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:20.584+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:20.590370+0000) 2019-10-13T15:23:20.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:20.588+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225462 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:20.910 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:23:20.911 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:20.914 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:23:20.914 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:21.184 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:21.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:21.181977+0000) 2019-10-13T15:23:21.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:21.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:21.834381+0000) 2019-10-13T15:23:22.454 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:22.448+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:22.452141+0000) 2019-10-13T15:23:24.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:24.712+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:24.716276+0000) 2019-10-13T15:23:25.598 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:25.592+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.595523+0000) 2019-10-13T15:23:25.598 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:25.592+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.595905+0000) 2019-10-13T15:23:25.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:25.592+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.597709+0000) 2019-10-13T15:23:25.601 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:25.596+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.598361+0000) 2019-10-13T15:23:25.603 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:25.596+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.600964+0000) 2019-10-13T15:23:25.605 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:25.600+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.602369+0000) 2019-10-13T15:23:25.608 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:25.600+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.605350+0000) 2019-10-13T15:23:25.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:25.604+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225493 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:25.609 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:25.600+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.606658+0000) 2019-10-13T15:23:25.613 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:25.604+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:25.611321+0000) 2019-10-13T15:23:28.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:28.084+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.085479+0000) 2019-10-13T15:23:28.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:28.308+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:28.310810+0000) 2019-10-13T15:23:30.612 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:30.604+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.609579+0000) 2019-10-13T15:23:30.614 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:30.604+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.612029+0000) 2019-10-13T15:23:30.616 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:30.608+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.612228+0000) 2019-10-13T15:23:30.616 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:30.608+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.612539+0000) 2019-10-13T15:23:30.616 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:30.608+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.613165+0000) 2019-10-13T15:23:30.617 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:30.608+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.613312+0000) 2019-10-13T15:23:30.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:30.612+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.614323+0000) 2019-10-13T15:23:30.622 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:30.616+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225523 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:30.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:30.620+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:30.624324+0000) 2019-10-13T15:23:31.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:31.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:31.182168+0000) 2019-10-13T15:23:31.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:31.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:31.834606+0000) 2019-10-13T15:23:32.352 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:32.348+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:32.349701+0000) 2019-10-13T15:23:32.464 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:32.456+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:32.461892+0000) 2019-10-13T15:23:34.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:34.712+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:34.716492+0000) 2019-10-13T15:23:35.600 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:35.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.597927+0000) 2019-10-13T15:23:35.627 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:35.620+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.624335+0000) 2019-10-13T15:23:35.627 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:35.620+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.625161+0000) 2019-10-13T15:23:35.628 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:35.620+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.625382+0000) 2019-10-13T15:23:35.628 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:35.624+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.625801+0000) 2019-10-13T15:23:35.629 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:35.624+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.626250+0000) 2019-10-13T15:23:35.629 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:35.624+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.626836+0000) 2019-10-13T15:23:35.630 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:35.624+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.627784+0000) 2019-10-13T15:23:35.632 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:35.624+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:35.629980+0000) 2019-10-13T15:23:35.636 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:35.632+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225552 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:38.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:38.080+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.085713+0000) 2019-10-13T15:23:38.313 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:38.308+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:38.311009+0000) 2019-10-13T15:23:40.640 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:40.632+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.637699+0000) 2019-10-13T15:23:40.641 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:40.636+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.638869+0000) 2019-10-13T15:23:40.642 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:40.636+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.639248+0000) 2019-10-13T15:23:40.642 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:40.636+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.639066+0000) 2019-10-13T15:23:40.642 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:40.636+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.639866+0000) 2019-10-13T15:23:40.643 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:40.636+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.640528+0000) 2019-10-13T15:23:40.644 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:40.636+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.641370+0000) 2019-10-13T15:23:40.647 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:40.644+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:40.645081+0000) 2019-10-13T15:23:40.650 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:40.644+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225581 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:41.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:41.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:41.182362+0000) 2019-10-13T15:23:41.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:41.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:41.834815+0000) 2019-10-13T15:23:42.352 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:42.348+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:42.349916+0000) 2019-10-13T15:23:42.471 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:42.468+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:42.468579+0000) 2019-10-13T15:23:44.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:44.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:44.716695+0000) 2019-10-13T15:23:45.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:45.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.598145+0000) 2019-10-13T15:23:45.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:45.648+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.651117+0000) 2019-10-13T15:23:45.655 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:45.648+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.652337+0000) 2019-10-13T15:23:45.655 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:45.648+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.652928+0000) 2019-10-13T15:23:45.656 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:45.648+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.653087+0000) 2019-10-13T15:23:45.656 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:45.651+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.654097+0000) 2019-10-13T15:23:45.657 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:45.648+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.653569+0000) 2019-10-13T15:23:45.657 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:45.652+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.654762+0000) 2019-10-13T15:23:45.662 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:45.655+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:45.659879+0000) 2019-10-13T15:23:45.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:45.660+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:48.088 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:48.084+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.085989+0000) 2019-10-13T15:23:48.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:48.308+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:48.311229+0000) 2019-10-13T15:23:50.668 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:50.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.665286+0000) 2019-10-13T15:23:50.668 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:50.664+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.665730+0000) 2019-10-13T15:23:50.669 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:50.664+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.666125+0000) 2019-10-13T15:23:50.670 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:50.663+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.667409+0000) 2019-10-13T15:23:50.670 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:50.664+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.668014+0000) 2019-10-13T15:23:50.671 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:50.664+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.668432+0000) 2019-10-13T15:23:50.671 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:50.664+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.668510+0000) 2019-10-13T15:23:50.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:50.671+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:50.672760+0000) 2019-10-13T15:23:50.678 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:50.672+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225642 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:50.966 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:23:50.966 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:50.970 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:23:50.970 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:23:51.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:51.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:51.182551+0000) 2019-10-13T15:23:51.837 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:51.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:51.835021+0000) 2019-10-13T15:23:52.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:52.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:52.350119+0000) 2019-10-13T15:23:52.481 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:52.475+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:52.478063+0000) 2019-10-13T15:23:54.719 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:54.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:54.716894+0000) 2019-10-13T15:23:55.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:55.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.598356+0000) 2019-10-13T15:23:55.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:23:55.676+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.679544+0000) 2019-10-13T15:23:55.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:23:55.676+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.679976+0000) 2019-10-13T15:23:55.683 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:23:55.679+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.680512+0000) 2019-10-13T15:23:55.683 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:55.676+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.680690+0000) 2019-10-13T15:23:55.685 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:23:55.680+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.682425+0000) 2019-10-13T15:23:55.685 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:55.680+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.682044+0000) 2019-10-13T15:23:55.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:23:55.680+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.682670+0000) 2019-10-13T15:23:55.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:23:55.683+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:55.687282+0000) 2019-10-13T15:23:55.692 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:23:55.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225672 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:23:58.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:23:58.084+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.086220+0000) 2019-10-13T15:23:58.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:23:58.308+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:23:58.311418+0000) 2019-10-13T15:24:00.697 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:00.692+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.694115+0000) 2019-10-13T15:24:00.697 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:00.692+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.694422+0000) 2019-10-13T15:24:00.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:00.691+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.695359+0000) 2019-10-13T15:24:00.698 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:00.692+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.695736+0000) 2019-10-13T15:24:00.699 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:00.692+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.696370+0000) 2019-10-13T15:24:00.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:00.692+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.696810+0000) 2019-10-13T15:24:00.700 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:00.696+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.697877+0000) 2019-10-13T15:24:00.706 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:00.699+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:00.703531+0000) 2019-10-13T15:24:00.706 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:00.700+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225701 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:01.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:01.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:01.182744+0000) 2019-10-13T15:24:01.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:01.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:01.835233+0000) 2019-10-13T15:24:02.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:02.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:02.350283+0000) 2019-10-13T15:24:02.490 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:02.483+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:02.487195+0000) 2019-10-13T15:24:04.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:04.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:04.717089+0000) 2019-10-13T15:24:05.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:05.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.598585+0000) 2019-10-13T15:24:05.709 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:05.703+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.706878+0000) 2019-10-13T15:24:05.711 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:05.704+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.708922+0000) 2019-10-13T15:24:05.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:05.704+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.709712+0000) 2019-10-13T15:24:05.713 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:05.708+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.710727+0000) 2019-10-13T15:24:05.713 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:05.708+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.711254+0000) 2019-10-13T15:24:05.714 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:05.708+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.711857+0000) 2019-10-13T15:24:05.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:05.708+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.711993+0000) 2019-10-13T15:24:05.719 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:05.715+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:05.716592+0000) 2019-10-13T15:24:05.721 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:05.716+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225733 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:08.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:08.084+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.086427+0000) 2019-10-13T15:24:08.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:08.308+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:08.311612+0000) 2019-10-13T15:24:10.725 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:10.719+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.722490+0000) 2019-10-13T15:24:10.726 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:10.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.723096+0000) 2019-10-13T15:24:10.726 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:10.720+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.723424+0000) 2019-10-13T15:24:10.726 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:10.720+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.724297+0000) 2019-10-13T15:24:10.727 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:10.720+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.724429+0000) 2019-10-13T15:24:10.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:10.720+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.725441+0000) 2019-10-13T15:24:10.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:10.724+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.725821+0000) 2019-10-13T15:24:10.734 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:10.727+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:10.731321+0000) 2019-10-13T15:24:10.735 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:10.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225762 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:11.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:11.180+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:11.182914+0000) 2019-10-13T15:24:11.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:11.832+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:11.835436+0000) 2019-10-13T15:24:12.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:12.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:12.350478+0000) 2019-10-13T15:24:12.496 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:12.491+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:12.493818+0000) 2019-10-13T15:24:14.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:14.712+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:14.717323+0000) 2019-10-13T15:24:15.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:15.596+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.598800+0000) 2019-10-13T15:24:15.738 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:15.732+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.735960+0000) 2019-10-13T15:24:15.739 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:15.732+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.736341+0000) 2019-10-13T15:24:15.740 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:15.735+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.737636+0000) 2019-10-13T15:24:15.741 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:15.736+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.738363+0000) 2019-10-13T15:24:15.742 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:15.736+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.739411+0000) 2019-10-13T15:24:15.742 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:15.736+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.740210+0000) 2019-10-13T15:24:15.744 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:15.736+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.740727+0000) 2019-10-13T15:24:15.746 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:15.739+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:15.743541+0000) 2019-10-13T15:24:15.749 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:15.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225792 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:18.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:18.083+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.086659+0000) 2019-10-13T15:24:18.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:18.307+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:18.311817+0000) 2019-10-13T15:24:20.753 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:20.747+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.750038+0000) 2019-10-13T15:24:20.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:20.747+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.751340+0000) 2019-10-13T15:24:20.754 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:20.751+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.752186+0000) 2019-10-13T15:24:20.755 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:20.747+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.752235+0000) 2019-10-13T15:24:20.755 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:20.747+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.752660+0000) 2019-10-13T15:24:20.756 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:20.747+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.753153+0000) 2019-10-13T15:24:20.756 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:20.751+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.753820+0000) 2019-10-13T15:24:20.760 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:20.755+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:20.757785+0000) 2019-10-13T15:24:20.763 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:20.755+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225821 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:21.023 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:24:21.023 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:21.026 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:24:21.027 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:21.185 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:21.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:21.183137+0000) 2019-10-13T15:24:21.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:21.831+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:21.835653+0000) 2019-10-13T15:24:22.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:22.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:22.350706+0000) 2019-10-13T15:24:22.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:22.499+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:22.500332+0000) 2019-10-13T15:24:24.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:24.715+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:24.717581+0000) 2019-10-13T15:24:25.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:25.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.599021+0000) 2019-10-13T15:24:25.766 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:25.759+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.763518+0000) 2019-10-13T15:24:25.766 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:25.759+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.763857+0000) 2019-10-13T15:24:25.767 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:25.759+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.765155+0000) 2019-10-13T15:24:25.768 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:25.759+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.764921+0000) 2019-10-13T15:24:25.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:25.763+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.766240+0000) 2019-10-13T15:24:25.769 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:25.763+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.766863+0000) 2019-10-13T15:24:25.769 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:25.763+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.767260+0000) 2019-10-13T15:24:25.774 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:25.767+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:25.771551+0000) 2019-10-13T15:24:25.777 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:25.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225853 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:28.089 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:28.083+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.086872+0000) 2019-10-13T15:24:28.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:28.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:28.312010+0000) 2019-10-13T15:24:30.781 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:30.775+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.778226+0000) 2019-10-13T15:24:30.781 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:30.775+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.778640+0000) 2019-10-13T15:24:30.782 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:30.775+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.779447+0000) 2019-10-13T15:24:30.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:30.775+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.780414+0000) 2019-10-13T15:24:30.783 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:30.775+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.781315+0000) 2019-10-13T15:24:30.784 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:30.779+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.781685+0000) 2019-10-13T15:24:30.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:30.779+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.781780+0000) 2019-10-13T15:24:30.792 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:30.783+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225882 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:30.793 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:30.787+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:30.791032+0000) 2019-10-13T15:24:31.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:31.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:31.183263+0000) 2019-10-13T15:24:31.838 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:31.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:31.835859+0000) 2019-10-13T15:24:32.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:32.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:32.350940+0000) 2019-10-13T15:24:32.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:32.503+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:32.506218+0000) 2019-10-13T15:24:34.720 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:34.715+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:34.717778+0000) 2019-10-13T15:24:35.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:35.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.599212+0000) 2019-10-13T15:24:35.796 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:35.787+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.793352+0000) 2019-10-13T15:24:35.796 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:35.791+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.793749+0000) 2019-10-13T15:24:35.797 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:35.791+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.793955+0000) 2019-10-13T15:24:35.797 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:35.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.794570+0000) 2019-10-13T15:24:35.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:35.791+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.795498+0000) 2019-10-13T15:24:35.798 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:35.791+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.795656+0000) 2019-10-13T15:24:35.798 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:35.791+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.796006+0000) 2019-10-13T15:24:35.803 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:35.799+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:35.800978+0000) 2019-10-13T15:24:35.806 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:35.799+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225912 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:38.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:38.083+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.087105+0000) 2019-10-13T15:24:38.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:38.307+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:38.312225+0000) 2019-10-13T15:24:40.809 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:40.803+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.806901+0000) 2019-10-13T15:24:40.810 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:40.803+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.807289+0000) 2019-10-13T15:24:40.811 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:40.807+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.808516+0000) 2019-10-13T15:24:40.811 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:40.803+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.809263+0000) 2019-10-13T15:24:40.812 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:40.803+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.809364+0000) 2019-10-13T15:24:40.812 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:40.807+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.809789+0000) 2019-10-13T15:24:40.814 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:40.807+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.811446+0000) 2019-10-13T15:24:40.820 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:40.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225941 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:40.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:40.819+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:40.821088+0000) 2019-10-13T15:24:41.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:41.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:41.183452+0000) 2019-10-13T15:24:41.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:41.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:41.836064+0000) 2019-10-13T15:24:42.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:42.347+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:42.351141+0000) 2019-10-13T15:24:42.514 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:42.507+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:42.511698+0000) 2019-10-13T15:24:44.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:44.715+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:44.717973+0000) 2019-10-13T15:24:45.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:45.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.599406+0000) 2019-10-13T15:24:45.824 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:45.815+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.821194+0000) 2019-10-13T15:24:45.826 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:45.819+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.824001+0000) 2019-10-13T15:24:45.827 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:45.819+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.824529+0000) 2019-10-13T15:24:45.828 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:45.819+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.825612+0000) 2019-10-13T15:24:45.828 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:45.823+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.826348+0000) 2019-10-13T15:24:45.829 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:45.823+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.826371+0000) 2019-10-13T15:24:45.829 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:45.823+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.826710+0000) 2019-10-13T15:24:45.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:45.827+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:45.831694+0000) 2019-10-13T15:24:45.834 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:45.827+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225973 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:48.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:48.083+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.087311+0000) 2019-10-13T15:24:48.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:48.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:48.312425+0000) 2019-10-13T15:24:50.839 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:50.831+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.836549+0000) 2019-10-13T15:24:50.839 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:50.835+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.836769+0000) 2019-10-13T15:24:50.840 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:50.831+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.837340+0000) 2019-10-13T15:24:50.840 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:50.835+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.837797+0000) 2019-10-13T15:24:50.841 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:50.835+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.838065+0000) 2019-10-13T15:24:50.842 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:50.835+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.839601+0000) 2019-10-13T15:24:50.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:50.835+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.840400+0000) 2019-10-13T15:24:50.844 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:50.839+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:50.842339+0000) 2019-10-13T15:24:50.848 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:50.843+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226002 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:51.135 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:24:51.135 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:51.139 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:24:51.139 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:24:51.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:51.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:51.183638+0000) 2019-10-13T15:24:51.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:51.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:51.836271+0000) 2019-10-13T15:24:52.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:52.346+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:52.351365+0000) 2019-10-13T15:24:52.521 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:52.514+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:52.518494+0000) 2019-10-13T15:24:54.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:54.715+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:54.718173+0000) 2019-10-13T15:24:55.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:55.595+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.599632+0000) 2019-10-13T15:24:55.852 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:24:55.847+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.849890+0000) 2019-10-13T15:24:55.853 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:24:55.847+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.850082+0000) 2019-10-13T15:24:55.853 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:24:55.846+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.851241+0000) 2019-10-13T15:24:55.854 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:55.847+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.850803+0000) 2019-10-13T15:24:55.854 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:24:55.847+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.851706+0000) 2019-10-13T15:24:55.855 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:24:55.847+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.851762+0000) 2019-10-13T15:24:55.855 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:55.847+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.852111+0000) 2019-10-13T15:24:55.861 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:24:55.854+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:55.858825+0000) 2019-10-13T15:24:55.862 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:24:55.855+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226032 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:24:58.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:24:58.087+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.087516+0000) 2019-10-13T15:24:58.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:24:58.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:24:58.312620+0000) 2019-10-13T15:25:00.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:00.859+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.863285+0000) 2019-10-13T15:25:00.867 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:00.862+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.865218+0000) 2019-10-13T15:25:00.868 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:00.859+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.865450+0000) 2019-10-13T15:25:00.868 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:00.863+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.866022+0000) 2019-10-13T15:25:00.870 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:00.863+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.868195+0000) 2019-10-13T15:25:00.871 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:00.863+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.868872+0000) 2019-10-13T15:25:00.872 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:00.863+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.869486+0000) 2019-10-13T15:25:00.877 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:00.870+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:00.874462+0000) 2019-10-13T15:25:00.877 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:00.871+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226061 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:01.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:01.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:01.183836+0000) 2019-10-13T15:25:01.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:01.831+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:01.836482+0000) 2019-10-13T15:25:02.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:02.346+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:02.351559+0000) 2019-10-13T15:25:02.528 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:02.522+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:02.525071+0000) 2019-10-13T15:25:04.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:04.715+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:04.718399+0000) 2019-10-13T15:25:05.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:05.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.599823+0000) 2019-10-13T15:25:05.883 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:05.875+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.880003+0000) 2019-10-13T15:25:05.883 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:05.875+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.880413+0000) 2019-10-13T15:25:05.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:05.875+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.880780+0000) 2019-10-13T15:25:05.884 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:05.875+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.881006+0000) 2019-10-13T15:25:05.884 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:05.879+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.881852+0000) 2019-10-13T15:25:05.885 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:05.878+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.882284+0000) 2019-10-13T15:25:05.885 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:05.879+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.882567+0000) 2019-10-13T15:25:05.887 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:05.882+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:05.884859+0000) 2019-10-13T15:25:05.892 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:05.883+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226093 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:08.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:08.087+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.087751+0000) 2019-10-13T15:25:08.315 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:08.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:08.312822+0000) 2019-10-13T15:25:10.895 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:10.887+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.892363+0000) 2019-10-13T15:25:10.896 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:10.890+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.893013+0000) 2019-10-13T15:25:10.896 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:10.887+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.893242+0000) 2019-10-13T15:25:10.897 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:10.887+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.893682+0000) 2019-10-13T15:25:10.897 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:10.891+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.894032+0000) 2019-10-13T15:25:10.897 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:10.891+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.894511+0000) 2019-10-13T15:25:10.898 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:10.891+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.896084+0000) 2019-10-13T15:25:10.904 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:10.898+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:10.901397+0000) 2019-10-13T15:25:10.906 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:10.899+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226121 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:11.186 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:11.179+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:11.183983+0000) 2019-10-13T15:25:11.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:11.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:11.836679+0000) 2019-10-13T15:25:12.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:12.350+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:12.351771+0000) 2019-10-13T15:25:12.537 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:12.530+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:12.534228+0000) 2019-10-13T15:25:14.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:14.714+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:14.718618+0000) 2019-10-13T15:25:15.602 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:15.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.600017+0000) 2019-10-13T15:25:15.911 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:15.906+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.908800+0000) 2019-10-13T15:25:15.912 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:15.902+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.908956+0000) 2019-10-13T15:25:15.912 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:15.902+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.909669+0000) 2019-10-13T15:25:15.914 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:15.906+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.911979+0000) 2019-10-13T15:25:15.915 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:15.906+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.912603+0000) 2019-10-13T15:25:15.915 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:15.906+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.913068+0000) 2019-10-13T15:25:15.916 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:15.910+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.913970+0000) 2019-10-13T15:25:15.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:15.914+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:15.916764+0000) 2019-10-13T15:25:15.921 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:15.914+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226152 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:18.090 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:18.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.087958+0000) 2019-10-13T15:25:18.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:18.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:18.313018+0000) 2019-10-13T15:25:20.925 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:20.918+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.922301+0000) 2019-10-13T15:25:20.925 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:20.918+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.922970+0000) 2019-10-13T15:25:20.926 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:20.918+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.923684+0000) 2019-10-13T15:25:20.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:20.918+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.923873+0000) 2019-10-13T15:25:20.927 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:20.922+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.925311+0000) 2019-10-13T15:25:20.928 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:20.922+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.925818+0000) 2019-10-13T15:25:20.929 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:20.922+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.926445+0000) 2019-10-13T15:25:20.934 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:20.926+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:20.931659+0000) 2019-10-13T15:25:20.936 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:20.930+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226180 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:21.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:21.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:21.184164+0000) 2019-10-13T15:25:21.198 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:25:21.199 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:21.202 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:25:21.202 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:21.839 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:21.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:21.836858+0000) 2019-10-13T15:25:22.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:22.350+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:22.351991+0000) 2019-10-13T15:25:22.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:22.538+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:22.540641+0000) 2019-10-13T15:25:24.721 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:24.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:24.718836+0000) 2019-10-13T15:25:25.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:25.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.600208+0000) 2019-10-13T15:25:25.939 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:25.930+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.937004+0000) 2019-10-13T15:25:25.940 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:25.930+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.937719+0000) 2019-10-13T15:25:25.940 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:25.934+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.938160+0000) 2019-10-13T15:25:25.941 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:25.934+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.938450+0000) 2019-10-13T15:25:25.941 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:25.934+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.939117+0000) 2019-10-13T15:25:25.942 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:25.934+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.939497+0000) 2019-10-13T15:25:25.944 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:25.938+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.941855+0000) 2019-10-13T15:25:25.949 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:25.942+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:25.946514+0000) 2019-10-13T15:25:25.950 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:25.942+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226213 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:28.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:28.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.088187+0000) 2019-10-13T15:25:28.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:28.310+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:28.313210+0000) 2019-10-13T15:25:30.954 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:30.946+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.951121+0000) 2019-10-13T15:25:30.956 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:30.946+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.953570+0000) 2019-10-13T15:25:30.956 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:30.950+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.954234+0000) 2019-10-13T15:25:30.957 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:30.950+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.954335+0000) 2019-10-13T15:25:30.959 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:30.950+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.956232+0000) 2019-10-13T15:25:30.959 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:30.954+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.956849+0000) 2019-10-13T15:25:30.959 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:30.950+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.957247+0000) 2019-10-13T15:25:30.963 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:30.958+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:30.961192+0000) 2019-10-13T15:25:30.966 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:30.958+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226241 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:31.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:31.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:31.184375+0000) 2019-10-13T15:25:31.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:31.830+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:31.837034+0000) 2019-10-13T15:25:32.354 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:32.350+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:32.352198+0000) 2019-10-13T15:25:32.549 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:32.542+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:32.547065+0000) 2019-10-13T15:25:34.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:34.714+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:34.719065+0000) 2019-10-13T15:25:35.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:35.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.600419+0000) 2019-10-13T15:25:35.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:35.962+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.966340+0000) 2019-10-13T15:25:35.969 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:35.962+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.966821+0000) 2019-10-13T15:25:35.972 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:35.966+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.969897+0000) 2019-10-13T15:25:35.974 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:35.966+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.971340+0000) 2019-10-13T15:25:35.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:35.966+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.972354+0000) 2019-10-13T15:25:35.975 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:35.966+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.973173+0000) 2019-10-13T15:25:35.977 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:35.970+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.974179+0000) 2019-10-13T15:25:35.979 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:35.974+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:35.976826+0000) 2019-10-13T15:25:35.980 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:35.974+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226274 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:38.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:38.082+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.088390+0000) 2019-10-13T15:25:38.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:38.310+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:38.313415+0000) 2019-10-13T15:25:40.983 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:40.978+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.980854+0000) 2019-10-13T15:25:40.984 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:40.974+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.981544+0000) 2019-10-13T15:25:40.985 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:40.978+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.982425+0000) 2019-10-13T15:25:40.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:40.978+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.982117+0000) 2019-10-13T15:25:40.986 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:40.978+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.983595+0000) 2019-10-13T15:25:40.988 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:40.978+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.985561+0000) 2019-10-13T15:25:40.990 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:40.982+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.987812+0000) 2019-10-13T15:25:40.996 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:40.986+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226302 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:40.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:40.994+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:40.997200+0000) 2019-10-13T15:25:41.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:41.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:41.184549+0000) 2019-10-13T15:25:41.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:41.834+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:41.837215+0000) 2019-10-13T15:25:42.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:42.350+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:42.352388+0000) 2019-10-13T15:25:42.556 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:42.550+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:42.553643+0000) 2019-10-13T15:25:44.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:44.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:44.719271+0000) 2019-10-13T15:25:45.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:45.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.600615+0000) 2019-10-13T15:25:46.000 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:45.990+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.997073+0000) 2019-10-13T15:25:46.000 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:45.994+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.998000+0000) 2019-10-13T15:25:46.001 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:45.994+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.998563+0000) 2019-10-13T15:25:46.001 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:45.994+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.998934+0000) 2019-10-13T15:25:46.002 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:45.994+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:45.999962+0000) 2019-10-13T15:25:46.003 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:45.998+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:46.000291+0000) 2019-10-13T15:25:46.003 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:45.994+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:46.000648+0000) 2019-10-13T15:25:46.010 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:46.002+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:46.007711+0000) 2019-10-13T15:25:46.011 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:46.002+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226337 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:48.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:48.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.088587+0000) 2019-10-13T15:25:48.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:48.306+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:48.313613+0000) 2019-10-13T15:25:51.014 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:51.006+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.011863+0000) 2019-10-13T15:25:51.015 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:51.006+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.012892+0000) 2019-10-13T15:25:51.016 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:51.006+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.013495+0000) 2019-10-13T15:25:51.016 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:51.010+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.014187+0000) 2019-10-13T15:25:51.017 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:51.010+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.014917+0000) 2019-10-13T15:25:51.018 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:51.010+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.015321+0000) 2019-10-13T15:25:51.018 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:51.010+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.015904+0000) 2019-10-13T15:25:51.023 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:51.018+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.021034+0000) 2019-10-13T15:25:51.026 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:51.018+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226365 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:51.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:51.178+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.184729+0000) 2019-10-13T15:25:51.245 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:25:51.245 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:51.249 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:25:51.249 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:25:51.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:51.834+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:51.837432+0000) 2019-10-13T15:25:52.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:52.350+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:52.352564+0000) 2019-10-13T15:25:52.562 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:52.558+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:52.560068+0000) 2019-10-13T15:25:54.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:54.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:54.719469+0000) 2019-10-13T15:25:55.603 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:55.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:55.600803+0000) 2019-10-13T15:25:56.030 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:25:56.022+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.027972+0000) 2019-10-13T15:25:56.032 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:56.022+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.029710+0000) 2019-10-13T15:25:56.032 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:25:56.026+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.030094+0000) 2019-10-13T15:25:56.033 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:56.026+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.030578+0000) 2019-10-13T15:25:56.033 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:25:56.026+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.030812+0000) 2019-10-13T15:25:56.034 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:25:56.026+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.031271+0000) 2019-10-13T15:25:56.036 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:25:56.030+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.033757+0000) 2019-10-13T15:25:56.039 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:25:56.034+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:56.037353+0000) 2019-10-13T15:25:56.041 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:25:56.034+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226397 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:25:58.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:25:58.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.088790+0000) 2019-10-13T15:25:58.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:25:58.310+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:25:58.313807+0000) 2019-10-13T15:26:01.045 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:01.038+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.042367+0000) 2019-10-13T15:26:01.046 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:01.038+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.044212+0000) 2019-10-13T15:26:01.047 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:01.038+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.044857+0000) 2019-10-13T15:26:01.048 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:01.038+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.045322+0000) 2019-10-13T15:26:01.049 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:01.042+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.046792+0000) 2019-10-13T15:26:01.050 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:01.041+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.047921+0000) 2019-10-13T15:26:01.052 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:01.046+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.049781+0000) 2019-10-13T15:26:01.055 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:01.049+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.052841+0000) 2019-10-13T15:26:01.057 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:01.050+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226426 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:01.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:01.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.184915+0000) 2019-10-13T15:26:01.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:01.834+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:01.837617+0000) 2019-10-13T15:26:02.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:02.349+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:02.352719+0000) 2019-10-13T15:26:02.569 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:02.561+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:02.566825+0000) 2019-10-13T15:26:04.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:04.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:04.719694+0000) 2019-10-13T15:26:05.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:05.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:05.601001+0000) 2019-10-13T15:26:06.060 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:06.054+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.057615+0000) 2019-10-13T15:26:06.061 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:06.054+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.058219+0000) 2019-10-13T15:26:06.061 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:06.054+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.058828+0000) 2019-10-13T15:26:06.062 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:06.054+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.059061+0000) 2019-10-13T15:26:06.064 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:06.054+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.061034+0000) 2019-10-13T15:26:06.064 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:06.054+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.061381+0000) 2019-10-13T15:26:06.064 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:06.057+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.061872+0000) 2019-10-13T15:26:06.068 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:06.061+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:06.065694+0000) 2019-10-13T15:26:06.072 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:06.062+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226460 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:08.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:08.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.089033+0000) 2019-10-13T15:26:08.316 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:08.310+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:08.314031+0000) 2019-10-13T15:26:11.075 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:11.065+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.072673+0000) 2019-10-13T15:26:11.076 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:11.065+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.073007+0000) 2019-10-13T15:26:11.077 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:11.069+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.075132+0000) 2019-10-13T15:26:11.078 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:11.069+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.075522+0000) 2019-10-13T15:26:11.079 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:11.073+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.076510+0000) 2019-10-13T15:26:11.081 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:11.073+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.078156+0000) 2019-10-13T15:26:11.081 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:11.073+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.078353+0000) 2019-10-13T15:26:11.087 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:11.077+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226489 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:11.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:11.081+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.087309+0000) 2019-10-13T15:26:11.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:11.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.185106+0000) 2019-10-13T15:26:11.840 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:11.833+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:11.837826+0000) 2019-10-13T15:26:12.355 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:12.349+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:12.352972+0000) 2019-10-13T15:26:12.576 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:12.569+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:12.573334+0000) 2019-10-13T15:26:14.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:14.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:14.719895+0000) 2019-10-13T15:26:15.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:15.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:15.601195+0000) 2019-10-13T15:26:16.091 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:16.081+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.088186+0000) 2019-10-13T15:26:16.091 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:16.081+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.088935+0000) 2019-10-13T15:26:16.092 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:16.081+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.089411+0000) 2019-10-13T15:26:16.092 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:16.085+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.090191+0000) 2019-10-13T15:26:16.095 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:16.085+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.092384+0000) 2019-10-13T15:26:16.097 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:16.089+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.094506+0000) 2019-10-13T15:26:16.098 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:16.089+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.096345+0000) 2019-10-13T15:26:16.102 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:16.093+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226521 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:16.102 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:16.097+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:16.100388+0000) 2019-10-13T15:26:18.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:18.085+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.089246+0000) 2019-10-13T15:26:18.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:18.313+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:18.314248+0000) 2019-10-13T15:26:21.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:21.097+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.102746+0000) 2019-10-13T15:26:21.106 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:21.097+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.104226+0000) 2019-10-13T15:26:21.107 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:21.097+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.104959+0000) 2019-10-13T15:26:21.108 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:21.097+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.104661+0000) 2019-10-13T15:26:21.109 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:21.101+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.107165+0000) 2019-10-13T15:26:21.110 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:21.101+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.107479+0000) 2019-10-13T15:26:21.110 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:21.101+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.107634+0000) 2019-10-13T15:26:21.117 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:21.109+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.114558+0000) 2019-10-13T15:26:21.118 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:21.109+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226550 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:21.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:21.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.185295+0000) 2019-10-13T15:26:21.367 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:26:21.367 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:21.371 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:26:21.371 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:21.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:21.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:21.838003+0000) 2019-10-13T15:26:22.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:22.349+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:22.353175+0000) 2019-10-13T15:26:22.582 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:22.577+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:22.579840+0000) 2019-10-13T15:26:24.722 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:24.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:24.720101+0000) 2019-10-13T15:26:25.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:25.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:25.601403+0000) 2019-10-13T15:26:26.122 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:26.117+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.118831+0000) 2019-10-13T15:26:26.122 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:26.117+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.119114+0000) 2019-10-13T15:26:26.122 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:26.117+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.119772+0000) 2019-10-13T15:26:26.123 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:26.117+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.120180+0000) 2019-10-13T15:26:26.123 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:26.117+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.120704+0000) 2019-10-13T15:26:26.123 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:26.117+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.120657+0000) 2019-10-13T15:26:26.124 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:26.117+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.121785+0000) 2019-10-13T15:26:26.130 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:26.121+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:26.127847+0000) 2019-10-13T15:26:26.132 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:26.129+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226584 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:28.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:28.085+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.089477+0000) 2019-10-13T15:26:28.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:28.313+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:28.314450+0000) 2019-10-13T15:26:31.135 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:31.129+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.132794+0000) 2019-10-13T15:26:31.136 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:31.133+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.133862+0000) 2019-10-13T15:26:31.137 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:31.133+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.133815+0000) 2019-10-13T15:26:31.137 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:31.133+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.134646+0000) 2019-10-13T15:26:31.137 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:31.133+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.134941+0000) 2019-10-13T15:26:31.138 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:31.133+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.136246+0000) 2019-10-13T15:26:31.139 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:31.133+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.137084+0000) 2019-10-13T15:26:31.146 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:31.137+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.143710+0000) 2019-10-13T15:26:31.148 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:31.141+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226613 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:31.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:31.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.185501+0000) 2019-10-13T15:26:31.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:31.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:31.838218+0000) 2019-10-13T15:26:32.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:32.349+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:32.353400+0000) 2019-10-13T15:26:32.589 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:32.585+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:32.586380+0000) 2019-10-13T15:26:34.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:34.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:34.720304+0000) 2019-10-13T15:26:35.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:35.597+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:35.601622+0000) 2019-10-13T15:26:36.152 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:36.145+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.149308+0000) 2019-10-13T15:26:36.153 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:36.145+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.149693+0000) 2019-10-13T15:26:36.153 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:36.149+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.150294+0000) 2019-10-13T15:26:36.153 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:36.149+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.150021+0000) 2019-10-13T15:26:36.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:36.149+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.150638+0000) 2019-10-13T15:26:36.154 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:36.145+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.151232+0000) 2019-10-13T15:26:36.154 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:36.149+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.151887+0000) 2019-10-13T15:26:36.161 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:36.157+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226645 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:36.162 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:36.157+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:36.160319+0000) 2019-10-13T15:26:38.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:38.089+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.089709+0000) 2019-10-13T15:26:38.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:38.313+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:38.314677+0000) 2019-10-13T15:26:41.165 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:41.157+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.162320+0000) 2019-10-13T15:26:41.165 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:41.161+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.162738+0000) 2019-10-13T15:26:41.166 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:41.161+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.163067+0000) 2019-10-13T15:26:41.166 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:41.161+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.163676+0000) 2019-10-13T15:26:41.168 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:41.165+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.166066+0000) 2019-10-13T15:26:41.169 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:41.165+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.166800+0000) 2019-10-13T15:26:41.170 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:41.165+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.167513+0000) 2019-10-13T15:26:41.176 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:41.173+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226674 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:41.177 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:41.169+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.174447+0000) 2019-10-13T15:26:41.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:41.181+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.185625+0000) 2019-10-13T15:26:41.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:41.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:41.838426+0000) 2019-10-13T15:26:42.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:42.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:42.353591+0000) 2019-10-13T15:26:42.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:42.589+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:42.593026+0000) 2019-10-13T15:26:44.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:44.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:44.720555+0000) 2019-10-13T15:26:45.604 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:45.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:45.601862+0000) 2019-10-13T15:26:46.180 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:46.173+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.177206+0000) 2019-10-13T15:26:46.180 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:46.173+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.177596+0000) 2019-10-13T15:26:46.181 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:46.173+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.177977+0000) 2019-10-13T15:26:46.181 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:46.177+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.178306+0000) 2019-10-13T15:26:46.181 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:46.177+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.178742+0000) 2019-10-13T15:26:46.182 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:46.177+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.179255+0000) 2019-10-13T15:26:46.183 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:46.177+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.180480+0000) 2019-10-13T15:26:46.188 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:46.181+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:46.186049+0000) 2019-10-13T15:26:46.191 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:46.185+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226708 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:48.092 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:48.089+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.089950+0000) 2019-10-13T15:26:48.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:48.313+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:48.314881+0000) 2019-10-13T15:26:51.188 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:51.185+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.185806+0000) 2019-10-13T15:26:51.194 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:51.189+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.191372+0000) 2019-10-13T15:26:51.195 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:51.189+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.193040+0000) 2019-10-13T15:26:51.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:51.189+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.193575+0000) 2019-10-13T15:26:51.197 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:51.193+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.194297+0000) 2019-10-13T15:26:51.197 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:51.193+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.193772+0000) 2019-10-13T15:26:51.197 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:51.193+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.194911+0000) 2019-10-13T15:26:51.198 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:51.189+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.195626+0000) 2019-10-13T15:26:51.204 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:51.197+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.201398+0000) 2019-10-13T15:26:51.205 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:51.201+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226737 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:51.523 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:26:51.523 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:51.527 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:26:51.527 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:26:51.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:51.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:51.838658+0000) 2019-10-13T15:26:52.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:52.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:52.353788+0000) 2019-10-13T15:26:52.602 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:52.597+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:52.599475+0000) 2019-10-13T15:26:54.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:54.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:54.720719+0000) 2019-10-13T15:26:55.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:55.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:55.602072+0000) 2019-10-13T15:26:56.209 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:26:56.205+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.206612+0000) 2019-10-13T15:26:56.210 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:56.205+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.207169+0000) 2019-10-13T15:26:56.210 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:56.205+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.207752+0000) 2019-10-13T15:26:56.211 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:26:56.205+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.208103+0000) 2019-10-13T15:26:56.212 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:26:56.205+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.207514+0000) 2019-10-13T15:26:56.212 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:26:56.209+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.209947+0000) 2019-10-13T15:26:56.215 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:26:56.209+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.212828+0000) 2019-10-13T15:26:56.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:26:56.209+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:56.216074+0000) 2019-10-13T15:26:56.220 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:26:56.217+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226768 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:26:58.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:26:58.089+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:58.090224+0000) 2019-10-13T15:26:58.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:26:58.313+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:26:58.315096+0000) 2019-10-13T15:27:01.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:01.185+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.186032+0000) 2019-10-13T15:27:01.224 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:01.217+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.221354+0000) 2019-10-13T15:27:01.225 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:01.221+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.222262+0000) 2019-10-13T15:27:01.225 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:01.221+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.222774+0000) 2019-10-13T15:27:01.225 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:01.221+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.223186+0000) 2019-10-13T15:27:01.226 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:01.221+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.223406+0000) 2019-10-13T15:27:01.226 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:01.221+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.223762+0000) 2019-10-13T15:27:01.227 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:01.221+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.224554+0000) 2019-10-13T15:27:01.232 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:01.225+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.229683+0000) 2019-10-13T15:27:01.235 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:01.229+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226797 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:01.841 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:01.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:01.838881+0000) 2019-10-13T15:27:02.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:02.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:02.353986+0000) 2019-10-13T15:27:02.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:02.605+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:02.606129+0000) 2019-10-13T15:27:04.723 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:04.717+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:04.720904+0000) 2019-10-13T15:27:05.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:05.601+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:05.602269+0000) 2019-10-13T15:27:06.238 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:06.232+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.235894+0000) 2019-10-13T15:27:06.239 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:06.232+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.236436+0000) 2019-10-13T15:27:06.240 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:06.232+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.236995+0000) 2019-10-13T15:27:06.241 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:06.236+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.238046+0000) 2019-10-13T15:27:06.241 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:06.236+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.238338+0000) 2019-10-13T15:27:06.241 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:06.236+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.238689+0000) 2019-10-13T15:27:06.242 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:06.236+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.238729+0000) 2019-10-13T15:27:06.250 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:06.244+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226830 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:06.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:06.244+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:06.248566+0000) 2019-10-13T15:27:08.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:08.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:08.090422+0000) 2019-10-13T15:27:08.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:08.312+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:08.315292+0000) 2019-10-13T15:27:11.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:11.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.186272+0000) 2019-10-13T15:27:11.253 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:11.248+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.250513+0000) 2019-10-13T15:27:11.254 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:11.248+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.250930+0000) 2019-10-13T15:27:11.254 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:11.248+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.252167+0000) 2019-10-13T15:27:11.255 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:11.248+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.252371+0000) 2019-10-13T15:27:11.257 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:11.252+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.254311+0000) 2019-10-13T15:27:11.257 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:11.252+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.255021+0000) 2019-10-13T15:27:11.258 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:11.252+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.255490+0000) 2019-10-13T15:27:11.264 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:11.256+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.262242+0000) 2019-10-13T15:27:11.265 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:11.260+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226859 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:11.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:11.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:11.839059+0000) 2019-10-13T15:27:12.356 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:12.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:12.354176+0000) 2019-10-13T15:27:12.615 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:12.608+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:12.612633+0000) 2019-10-13T15:27:14.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:14.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:14.721085+0000) 2019-10-13T15:27:15.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:15.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:15.602493+0000) 2019-10-13T15:27:16.268 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:16.264+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.265938+0000) 2019-10-13T15:27:16.269 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:16.264+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.266269+0000) 2019-10-13T15:27:16.270 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:16.260+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.267994+0000) 2019-10-13T15:27:16.271 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:16.264+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.268175+0000) 2019-10-13T15:27:16.271 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:16.264+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.268431+0000) 2019-10-13T15:27:16.272 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:16.264+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.269237+0000) 2019-10-13T15:27:16.272 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:16.268+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.270045+0000) 2019-10-13T15:27:16.278 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:16.272+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:16.276436+0000) 2019-10-13T15:27:16.280 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:16.272+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226890 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:18.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:18.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:18.090653+0000) 2019-10-13T15:27:18.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:18.312+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:18.315485+0000) 2019-10-13T15:27:21.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:21.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.186498+0000) 2019-10-13T15:27:21.284 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:21.276+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.281117+0000) 2019-10-13T15:27:21.284 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:21.280+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.281962+0000) 2019-10-13T15:27:21.286 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:21.280+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.283515+0000) 2019-10-13T15:27:21.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:21.280+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.283989+0000) 2019-10-13T15:27:21.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:21.280+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.284546+0000) 2019-10-13T15:27:21.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:21.284+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.286091+0000) 2019-10-13T15:27:21.292 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:21.284+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.289630+0000) 2019-10-13T15:27:21.295 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:21.288+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.292681+0000) 2019-10-13T15:27:21.295 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:21.288+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226919 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:21.637 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:27:21.638 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:21.641 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:27:21.641 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:21.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:21.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:21.839267+0000) 2019-10-13T15:27:22.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:22.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:22.354371+0000) 2019-10-13T15:27:22.621 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:22.616+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:22.618967+0000) 2019-10-13T15:27:24.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:24.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:24.721306+0000) 2019-10-13T15:27:25.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:25.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:25.602685+0000) 2019-10-13T15:27:26.299 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:26.292+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.296475+0000) 2019-10-13T15:27:26.299 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:26.292+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.297089+0000) 2019-10-13T15:27:26.300 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:26.292+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.297634+0000) 2019-10-13T15:27:26.300 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:26.296+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.297733+0000) 2019-10-13T15:27:26.301 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:26.296+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.298253+0000) 2019-10-13T15:27:26.302 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:26.296+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.299542+0000) 2019-10-13T15:27:26.302 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:26.296+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.299909+0000) 2019-10-13T15:27:26.307 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:26.300+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:26.305079+0000) 2019-10-13T15:27:26.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:26.304+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226952 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:28.093 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:28.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:28.090898+0000) 2019-10-13T15:27:28.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:28.312+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:28.315691+0000) 2019-10-13T15:27:31.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:31.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.186716+0000) 2019-10-13T15:27:31.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:31.308+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.311425+0000) 2019-10-13T15:27:31.315 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:31.308+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.312426+0000) 2019-10-13T15:27:31.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:31.308+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.312692+0000) 2019-10-13T15:27:31.316 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:31.308+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.312846+0000) 2019-10-13T15:27:31.316 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:31.308+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.313180+0000) 2019-10-13T15:27:31.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:31.308+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.313721+0000) 2019-10-13T15:27:31.317 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:31.312+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.314284+0000) 2019-10-13T15:27:31.325 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:31.316+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.320383+0000) 2019-10-13T15:27:31.325 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:31.320+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226981 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:31.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:31.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:31.839481+0000) 2019-10-13T15:27:32.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:32.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:32.354596+0000) 2019-10-13T15:27:32.628 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:32.624+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:32.625545+0000) 2019-10-13T15:27:34.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:34.716+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:34.721500+0000) 2019-10-13T15:27:35.605 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:35.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:35.602941+0000) 2019-10-13T15:27:36.329 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:36.324+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.326437+0000) 2019-10-13T15:27:36.329 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:36.324+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.326627+0000) 2019-10-13T15:27:36.330 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:36.324+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.327266+0000) 2019-10-13T15:27:36.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:36.320+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.327805+0000) 2019-10-13T15:27:36.331 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:36.324+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.327956+0000) 2019-10-13T15:27:36.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:36.324+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.328231+0000) 2019-10-13T15:27:36.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:36.324+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.328655+0000) 2019-10-13T15:27:36.340 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:36.332+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227011 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:36.340 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:36.332+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:36.337683+0000) 2019-10-13T15:27:38.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:38.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:38.091103+0000) 2019-10-13T15:27:38.318 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:38.312+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:38.315842+0000) 2019-10-13T15:27:41.189 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:41.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.186934+0000) 2019-10-13T15:27:41.343 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:41.336+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.341032+0000) 2019-10-13T15:27:41.344 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:41.336+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.341421+0000) 2019-10-13T15:27:41.346 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:41.340+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.343606+0000) 2019-10-13T15:27:41.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:41.340+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.344601+0000) 2019-10-13T15:27:41.349 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:41.344+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.347093+0000) 2019-10-13T15:27:41.352 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:41.344+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.349428+0000) 2019-10-13T15:27:41.353 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:41.344+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.350729+0000) 2019-10-13T15:27:41.354 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:41.348+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227041 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:41.360 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:41.352+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.358018+0000) 2019-10-13T15:27:41.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:41.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:41.839662+0000) 2019-10-13T15:27:42.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:42.348+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:42.354783+0000) 2019-10-13T15:27:42.634 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:42.628+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:42.632056+0000) 2019-10-13T15:27:44.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:44.720+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:44.721723+0000) 2019-10-13T15:27:45.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:45.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:45.603133+0000) 2019-10-13T15:27:46.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:46.352+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.355895+0000) 2019-10-13T15:27:46.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:46.352+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.356328+0000) 2019-10-13T15:27:46.362 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:46.356+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.359365+0000) 2019-10-13T15:27:46.362 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:46.356+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.359806+0000) 2019-10-13T15:27:46.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:46.356+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.360426+0000) 2019-10-13T15:27:46.363 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:46.356+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.360710+0000) 2019-10-13T15:27:46.364 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:46.356+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.360757+0000) 2019-10-13T15:27:46.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:46.360+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:46.365801+0000) 2019-10-13T15:27:46.369 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:46.364+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227073 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:48.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:48.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:48.091300+0000) 2019-10-13T15:27:48.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:48.312+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:48.316073+0000) 2019-10-13T15:27:51.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:51.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.187150+0000) 2019-10-13T15:27:51.372 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:51.364+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.369445+0000) 2019-10-13T15:27:51.372 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:51.368+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.370257+0000) 2019-10-13T15:27:51.373 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:51.364+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.370900+0000) 2019-10-13T15:27:51.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:51.368+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.372471+0000) 2019-10-13T15:27:51.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:51.372+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.374429+0000) 2019-10-13T15:27:51.377 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:51.372+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.375077+0000) 2019-10-13T15:27:51.378 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:51.372+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.376061+0000) 2019-10-13T15:27:51.381 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:51.372+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.378880+0000) 2019-10-13T15:27:51.383 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:51.376+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227103 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:51.762 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:27:51.762 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:51.766 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:27:51.766 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:27:51.842 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:51.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:51.839845+0000) 2019-10-13T15:27:52.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:52.348+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:52.354978+0000) 2019-10-13T15:27:52.641 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:52.636+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:52.638748+0000) 2019-10-13T15:27:54.724 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:54.720+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:54.721983+0000) 2019-10-13T15:27:55.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:55.600+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:55.603327+0000) 2019-10-13T15:27:56.388 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:56.380+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.385430+0000) 2019-10-13T15:27:56.388 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:56.384+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.385885+0000) 2019-10-13T15:27:56.389 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:27:56.380+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.386812+0000) 2019-10-13T15:27:56.390 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:27:56.384+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.387108+0000) 2019-10-13T15:27:56.392 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:27:56.384+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.389441+0000) 2019-10-13T15:27:56.394 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:27:56.388+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.391286+0000) 2019-10-13T15:27:56.394 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:27:56.388+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.391574+0000) 2019-10-13T15:27:56.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:27:56.392+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227133 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:27:56.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:27:56.388+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:56.395990+0000) 2019-10-13T15:27:58.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:27:58.088+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:58.091494+0000) 2019-10-13T15:27:58.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:27:58.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:27:58.316272+0000) 2019-10-13T15:28:01.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:01.184+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.187385+0000) 2019-10-13T15:28:01.401 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:01.396+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.398810+0000) 2019-10-13T15:28:01.402 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:01.392+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.399276+0000) 2019-10-13T15:28:01.403 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:01.396+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.400449+0000) 2019-10-13T15:28:01.403 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:01.396+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.400466+0000) 2019-10-13T15:28:01.404 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:01.396+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.400780+0000) 2019-10-13T15:28:01.404 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:01.396+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.401233+0000) 2019-10-13T15:28:01.406 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:01.400+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.403455+0000) 2019-10-13T15:28:01.410 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:01.404+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.408192+0000) 2019-10-13T15:28:01.412 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:01.408+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227165 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:01.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:01.836+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:01.840048+0000) 2019-10-13T15:28:02.357 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:02.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:02.355182+0000) 2019-10-13T15:28:02.648 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:02.644+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:02.645253+0000) 2019-10-13T15:28:04.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:04.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:04.722185+0000) 2019-10-13T15:28:05.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:05.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:05.603517+0000) 2019-10-13T15:28:06.416 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:06.407+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.413331+0000) 2019-10-13T15:28:06.416 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:06.411+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.414163+0000) 2019-10-13T15:28:06.417 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:06.411+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.414783+0000) 2019-10-13T15:28:06.418 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:06.412+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.415381+0000) 2019-10-13T15:28:06.418 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:06.411+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.415953+0000) 2019-10-13T15:28:06.419 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:06.411+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.416147+0000) 2019-10-13T15:28:06.419 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:06.411+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.416384+0000) 2019-10-13T15:28:06.426 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:06.420+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:06.423584+0000) 2019-10-13T15:28:06.427 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:06.419+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227195 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:08.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:08.087+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:08.091701+0000) 2019-10-13T15:28:08.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:08.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:08.316469+0000) 2019-10-13T15:28:11.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:11.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.187594+0000) 2019-10-13T15:28:11.430 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:11.423+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.427047+0000) 2019-10-13T15:28:11.431 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:11.427+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.429004+0000) 2019-10-13T15:28:11.432 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:11.423+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.429495+0000) 2019-10-13T15:28:11.433 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:11.427+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.430330+0000) 2019-10-13T15:28:11.433 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:11.427+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.430472+0000) 2019-10-13T15:28:11.433 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:11.427+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.430389+0000) 2019-10-13T15:28:11.434 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:11.427+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.430758+0000) 2019-10-13T15:28:11.437 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:11.431+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.434872+0000) 2019-10-13T15:28:11.441 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:11.435+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227225 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:11.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:11.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:11.840232+0000) 2019-10-13T15:28:12.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:12.351+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:12.355349+0000) 2019-10-13T15:28:12.654 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:12.647+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:12.651765+0000) 2019-10-13T15:28:14.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:14.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:14.722384+0000) 2019-10-13T15:28:15.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:15.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:15.603715+0000) 2019-10-13T15:28:16.444 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:16.439+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.441784+0000) 2019-10-13T15:28:16.446 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:16.439+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.443426+0000) 2019-10-13T15:28:16.446 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:16.439+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.443752+0000) 2019-10-13T15:28:16.447 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:16.439+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.444271+0000) 2019-10-13T15:28:16.447 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:16.439+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.444322+0000) 2019-10-13T15:28:16.447 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:16.439+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.444783+0000) 2019-10-13T15:28:16.448 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:16.439+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.445538+0000) 2019-10-13T15:28:16.453 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:16.447+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:16.450715+0000) 2019-10-13T15:28:16.456 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:16.451+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227255 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:18.094 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:18.087+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:18.091909+0000) 2019-10-13T15:28:18.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:18.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:18.316663+0000) 2019-10-13T15:28:21.190 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:21.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.187787+0000) 2019-10-13T15:28:21.460 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:21.451+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.457358+0000) 2019-10-13T15:28:21.460 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:21.455+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.458179+0000) 2019-10-13T15:28:21.461 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:21.455+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.458549+0000) 2019-10-13T15:28:21.461 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:21.455+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.459258+0000) 2019-10-13T15:28:21.462 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:21.455+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.459595+0000) 2019-10-13T15:28:21.462 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:21.455+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.460025+0000) 2019-10-13T15:28:21.463 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:21.455+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.460453+0000) 2019-10-13T15:28:21.467 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:21.463+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.464793+0000) 2019-10-13T15:28:21.471 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:21.463+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227286 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:21.823 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:28:21.823 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:21.826 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:28:21.827 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:21.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:21.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:21.840407+0000) 2019-10-13T15:28:22.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:22.351+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:22.355524+0000) 2019-10-13T15:28:22.660 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:22.655+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:22.657767+0000) 2019-10-13T15:28:24.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:24.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:24.722602+0000) 2019-10-13T15:28:25.606 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:25.599+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:25.603952+0000) 2019-10-13T15:28:26.475 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:26.467+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.472143+0000) 2019-10-13T15:28:26.475 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:26.467+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.472545+0000) 2019-10-13T15:28:26.475 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:26.467+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.473169+0000) 2019-10-13T15:28:26.476 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:26.471+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.473926+0000) 2019-10-13T15:28:26.477 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:26.471+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.474243+0000) 2019-10-13T15:28:26.477 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:26.471+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.474633+0000) 2019-10-13T15:28:26.478 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:26.471+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.474871+0000) 2019-10-13T15:28:26.483 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:26.479+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:26.480624+0000) 2019-10-13T15:28:26.486 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:26.479+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227317 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:28.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:28.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:28.092152+0000) 2019-10-13T15:28:28.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:28.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:28.316874+0000) 2019-10-13T15:28:31.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:31.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.187999+0000) 2019-10-13T15:28:31.489 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:31.483+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.486836+0000) 2019-10-13T15:28:31.491 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:31.483+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.488333+0000) 2019-10-13T15:28:31.491 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:31.483+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.489041+0000) 2019-10-13T15:28:31.492 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:31.483+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.489551+0000) 2019-10-13T15:28:31.492 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:31.487+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.490071+0000) 2019-10-13T15:28:31.493 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:31.487+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.490642+0000) 2019-10-13T15:28:31.494 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:31.487+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.492022+0000) 2019-10-13T15:28:31.499 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:31.495+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.496839+0000) 2019-10-13T15:28:31.501 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:31.495+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227346 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:31.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:31.839+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:31.840601+0000) 2019-10-13T15:28:32.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:32.351+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:32.355698+0000) 2019-10-13T15:28:32.666 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:32.659+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:32.663602+0000) 2019-10-13T15:28:34.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:34.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:34.722836+0000) 2019-10-13T15:28:35.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:35.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:35.604185+0000) 2019-10-13T15:28:36.504 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:36.495+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.501407+0000) 2019-10-13T15:28:36.505 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:36.499+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.502950+0000) 2019-10-13T15:28:36.506 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:36.499+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.503648+0000) 2019-10-13T15:28:36.506 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:36.499+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.503743+0000) 2019-10-13T15:28:36.507 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:36.503+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.504563+0000) 2019-10-13T15:28:36.507 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:36.499+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.504830+0000) 2019-10-13T15:28:36.508 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:36.499+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.505603+0000) 2019-10-13T15:28:36.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:36.507+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:36.509557+0000) 2019-10-13T15:28:36.515 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:36.507+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227377 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:38.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:38.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:38.092351+0000) 2019-10-13T15:28:38.319 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:38.311+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:38.317065+0000) 2019-10-13T15:28:41.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:41.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.188191+0000) 2019-10-13T15:28:41.518 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:41.511+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.515424+0000) 2019-10-13T15:28:41.519 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:41.511+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.516985+0000) 2019-10-13T15:28:41.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:41.511+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.517530+0000) 2019-10-13T15:28:41.522 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:41.515+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.519537+0000) 2019-10-13T15:28:41.522 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:41.515+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.519666+0000) 2019-10-13T15:28:41.524 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:41.519+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.521829+0000) 2019-10-13T15:28:41.525 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:41.519+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.522543+0000) 2019-10-13T15:28:41.529 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:41.523+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227408 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:41.530 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:41.527+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.528252+0000) 2019-10-13T15:28:41.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:41.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:41.840802+0000) 2019-10-13T15:28:42.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:42.351+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:42.355884+0000) 2019-10-13T15:28:42.672 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:42.667+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:42.669987+0000) 2019-10-13T15:28:44.725 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:44.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:44.723059+0000) 2019-10-13T15:28:45.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:45.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:45.604409+0000) 2019-10-13T15:28:46.533 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:46.527+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.530593+0000) 2019-10-13T15:28:46.534 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:46.527+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.531394+0000) 2019-10-13T15:28:46.534 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:46.527+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.531640+0000) 2019-10-13T15:28:46.535 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:46.531+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.532379+0000) 2019-10-13T15:28:46.535 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:46.527+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.531887+0000) 2019-10-13T15:28:46.535 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:46.527+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.532540+0000) 2019-10-13T15:28:46.536 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:46.527+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.533189+0000) 2019-10-13T15:28:46.541 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:46.535+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:46.538577+0000) 2019-10-13T15:28:46.543 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:46.535+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:48.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:48.087+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:48.092612+0000) 2019-10-13T15:28:48.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:48.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:48.317262+0000) 2019-10-13T15:28:51.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:51.183+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.188374+0000) 2019-10-13T15:28:51.547 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:51.543+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.544395+0000) 2019-10-13T15:28:51.547 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:51.539+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.544757+0000) 2019-10-13T15:28:51.548 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:51.539+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.544966+0000) 2019-10-13T15:28:51.548 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:51.539+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.545630+0000) 2019-10-13T15:28:51.549 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:51.543+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.547176+0000) 2019-10-13T15:28:51.550 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:51.543+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.547429+0000) 2019-10-13T15:28:51.550 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:51.543+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.547886+0000) 2019-10-13T15:28:51.554 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:51.551+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.552138+0000) 2019-10-13T15:28:51.559 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:51.551+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227469 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:51.843 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:51.835+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:51.840992+0000) 2019-10-13T15:28:51.946 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:28:51.946 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:51.949 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:28:51.949 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:28:52.358 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:52.351+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:52.356057+0000) 2019-10-13T15:28:52.679 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:52.675+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:52.676572+0000) 2019-10-13T15:28:54.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:54.719+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:54.723251+0000) 2019-10-13T15:28:55.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:55.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:55.604660+0000) 2019-10-13T15:28:56.562 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:28:56.555+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.559515+0000) 2019-10-13T15:28:56.563 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:28:56.555+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.559893+0000) 2019-10-13T15:28:56.564 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:28:56.559+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.562053+0000) 2019-10-13T15:28:56.567 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:56.559+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.564295+0000) 2019-10-13T15:28:56.567 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:28:56.559+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.564834+0000) 2019-10-13T15:28:56.569 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:28:56.563+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.566995+0000) 2019-10-13T15:28:56.571 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:56.563+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.568482+0000) 2019-10-13T15:28:56.573 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:28:56.567+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227500 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:28:56.575 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:28:56.571+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:56.573108+0000) 2019-10-13T15:28:58.095 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:28:58.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:58.092872+0000) 2019-10-13T15:28:58.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:28:58.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:28:58.317460+0000) 2019-10-13T15:29:01.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:01.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.188610+0000) 2019-10-13T15:29:01.576 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:01.570+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.573673+0000) 2019-10-13T15:29:01.577 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:01.571+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.574778+0000) 2019-10-13T15:29:01.581 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:01.574+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.579026+0000) 2019-10-13T15:29:01.583 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:01.574+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.580734+0000) 2019-10-13T15:29:01.585 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:01.578+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.582934+0000) 2019-10-13T15:29:01.586 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:01.578+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.584209+0000) 2019-10-13T15:29:01.587 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:01.578+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.585175+0000) 2019-10-13T15:29:01.589 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:01.582+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227532 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:01.592 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:01.587+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.589954+0000) 2019-10-13T15:29:01.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:01.834+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:01.841171+0000) 2019-10-13T15:29:02.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:02.355+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:02.356292+0000) 2019-10-13T15:29:02.685 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:02.679+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:02.683083+0000) 2019-10-13T15:29:04.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:04.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:04.723482+0000) 2019-10-13T15:29:05.607 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:05.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:05.604883+0000) 2019-10-13T15:29:06.592 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:06.582+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.589487+0000) 2019-10-13T15:29:06.594 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:06.586+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.591245+0000) 2019-10-13T15:29:06.594 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:06.586+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.591854+0000) 2019-10-13T15:29:06.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:06.590+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.594428+0000) 2019-10-13T15:29:06.601 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:06.595+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.598514+0000) 2019-10-13T15:29:06.602 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:06.594+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.599463+0000) 2019-10-13T15:29:06.602 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:06.594+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.599814+0000) 2019-10-13T15:29:06.605 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:06.598+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227563 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:06.606 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:06.599+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:06.603637+0000) 2019-10-13T15:29:08.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:08.090+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:08.093071+0000) 2019-10-13T15:29:08.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:08.314+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:08.317660+0000) 2019-10-13T15:29:11.191 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:11.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.188855+0000) 2019-10-13T15:29:11.608 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:11.598+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.605598+0000) 2019-10-13T15:29:11.610 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:11.602+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.607140+0000) 2019-10-13T15:29:11.610 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:11.602+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.607698+0000) 2019-10-13T15:29:11.610 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:11.607+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.608316+0000) 2019-10-13T15:29:11.611 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:11.602+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.608668+0000) 2019-10-13T15:29:11.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:11.602+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.609348+0000) 2019-10-13T15:29:11.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:11.606+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.611763+0000) 2019-10-13T15:29:11.619 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:11.615+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.616712+0000) 2019-10-13T15:29:11.619 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:11.610+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227593 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:11.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:11.838+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:11.841349+0000) 2019-10-13T15:29:12.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:12.355+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:12.356457+0000) 2019-10-13T15:29:12.692 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:12.687+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:12.689747+0000) 2019-10-13T15:29:14.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:14.722+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:14.723711+0000) 2019-10-13T15:29:15.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:15.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:15.605032+0000) 2019-10-13T15:29:16.623 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:16.614+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.620618+0000) 2019-10-13T15:29:16.624 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:16.614+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.621382+0000) 2019-10-13T15:29:16.624 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:16.618+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.622074+0000) 2019-10-13T15:29:16.625 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:16.618+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.622667+0000) 2019-10-13T15:29:16.626 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:16.618+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.622863+0000) 2019-10-13T15:29:16.626 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:16.618+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.623050+0000) 2019-10-13T15:29:16.626 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:16.618+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.623954+0000) 2019-10-13T15:29:16.634 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:16.626+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227624 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:16.635 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:16.630+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:16.633133+0000) 2019-10-13T15:29:18.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:18.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:18.093341+0000) 2019-10-13T15:29:18.320 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:18.314+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:18.317865+0000) 2019-10-13T15:29:21.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:21.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.189060+0000) 2019-10-13T15:29:21.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:21.630+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.634324+0000) 2019-10-13T15:29:21.637 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:21.630+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.635148+0000) 2019-10-13T15:29:21.638 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:21.630+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.635376+0000) 2019-10-13T15:29:21.640 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:21.630+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.636321+0000) 2019-10-13T15:29:21.640 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:21.630+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.637196+0000) 2019-10-13T15:29:21.641 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:21.634+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.638210+0000) 2019-10-13T15:29:21.641 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:21.634+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.638762+0000) 2019-10-13T15:29:21.645 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:21.638+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.642795+0000) 2019-10-13T15:29:21.648 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:21.638+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227656 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:21.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:21.838+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:21.841525+0000) 2019-10-13T15:29:22.058 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:29:22.059 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:22.063 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:29:22.064 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:22.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:22.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:22.356636+0000) 2019-10-13T15:29:22.699 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:22.694+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:22.696297+0000) 2019-10-13T15:29:24.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:24.722+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:24.723930+0000) 2019-10-13T15:29:25.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:25.598+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:25.605224+0000) 2019-10-13T15:29:26.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:26.642+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.647899+0000) 2019-10-13T15:29:26.651 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:26.642+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.649178+0000) 2019-10-13T15:29:26.652 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:26.642+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.649614+0000) 2019-10-13T15:29:26.652 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:26.646+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.650134+0000) 2019-10-13T15:29:26.653 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:26.646+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.650329+0000) 2019-10-13T15:29:26.653 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:26.646+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.650789+0000) 2019-10-13T15:29:26.654 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:26.646+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.652076+0000) 2019-10-13T15:29:26.659 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:26.654+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:26.656869+0000) 2019-10-13T15:29:26.664 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:26.654+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227688 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:28.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:28.086+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:28.093548+0000) 2019-10-13T15:29:28.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:28.314+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:28.318024+0000) 2019-10-13T15:29:31.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:31.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.189220+0000) 2019-10-13T15:29:31.668 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:31.658+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.664900+0000) 2019-10-13T15:29:31.668 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:31.662+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.666155+0000) 2019-10-13T15:29:31.669 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:31.662+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.666516+0000) 2019-10-13T15:29:31.671 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:31.662+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.668573+0000) 2019-10-13T15:29:31.673 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:31.666+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.670719+0000) 2019-10-13T15:29:31.673 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:31.666+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.671151+0000) 2019-10-13T15:29:31.675 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:31.666+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.672638+0000) 2019-10-13T15:29:31.676 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:31.670+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.674292+0000) 2019-10-13T15:29:31.679 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:31.670+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227717 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:31.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:31.838+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:31.841703+0000) 2019-10-13T15:29:32.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:32.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:32.356813+0000) 2019-10-13T15:29:32.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:32.698+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:32.702974+0000) 2019-10-13T15:29:34.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:34.722+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:34.724129+0000) 2019-10-13T15:29:35.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:35.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:35.605414+0000) 2019-10-13T15:29:36.682 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:36.674+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.679515+0000) 2019-10-13T15:29:36.683 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:36.674+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.680509+0000) 2019-10-13T15:29:36.684 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:36.674+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.681355+0000) 2019-10-13T15:29:36.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:36.678+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.682260+0000) 2019-10-13T15:29:36.685 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:36.678+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.682561+0000) 2019-10-13T15:29:36.688 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:36.678+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.684056+0000) 2019-10-13T15:29:36.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:36.678+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.683425+0000) 2019-10-13T15:29:36.689 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:36.682+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:36.686414+0000) 2019-10-13T15:29:36.693 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:36.686+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227749 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:38.096 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:38.090+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:38.093772+0000) 2019-10-13T15:29:38.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:38.314+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:38.318241+0000) 2019-10-13T15:29:41.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:41.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.189423+0000) 2019-10-13T15:29:41.697 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:41.690+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.694196+0000) 2019-10-13T15:29:41.698 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:41.690+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.694856+0000) 2019-10-13T15:29:41.698 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:41.690+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.695540+0000) 2019-10-13T15:29:41.699 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:41.690+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.695160+0000) 2019-10-13T15:29:41.699 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:41.690+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.695809+0000) 2019-10-13T15:29:41.699 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:41.690+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.696164+0000) 2019-10-13T15:29:41.700 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:41.690+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.697249+0000) 2019-10-13T15:29:41.705 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:41.698+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.703138+0000) 2019-10-13T15:29:41.707 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:41.698+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227780 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:41.844 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:41.838+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:41.841911+0000) 2019-10-13T15:29:42.359 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:42.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:42.356981+0000) 2019-10-13T15:29:42.712 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:42.706+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:42.709565+0000) 2019-10-13T15:29:44.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:44.718+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:44.724331+0000) 2019-10-13T15:29:45.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:45.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:45.605647+0000) 2019-10-13T15:29:46.711 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:46.702+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.707838+0000) 2019-10-13T15:29:46.712 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:46.706+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.709367+0000) 2019-10-13T15:29:46.712 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:46.702+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.708880+0000) 2019-10-13T15:29:46.712 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:46.702+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.709712+0000) 2019-10-13T15:29:46.713 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:46.706+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.710097+0000) 2019-10-13T15:29:46.713 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:46.706+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.710720+0000) 2019-10-13T15:29:46.714 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:46.706+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.711287+0000) 2019-10-13T15:29:46.720 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:46.714+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:46.717696+0000) 2019-10-13T15:29:46.724 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:46.714+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:48.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:48.090+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:48.093978+0000) 2019-10-13T15:29:48.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:48.314+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:48.318456+0000) 2019-10-13T15:29:51.192 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:51.182+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.189587+0000) 2019-10-13T15:29:51.727 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:51.722+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.725008+0000) 2019-10-13T15:29:51.728 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:51.718+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.725416+0000) 2019-10-13T15:29:51.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:51.722+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.726152+0000) 2019-10-13T15:29:51.729 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:51.722+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.726517+0000) 2019-10-13T15:29:51.729 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:51.722+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.727029+0000) 2019-10-13T15:29:51.730 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:51.722+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.727256+0000) 2019-10-13T15:29:51.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:51.722+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.728738+0000) 2019-10-13T15:29:51.736 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:51.730+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.734023+0000) 2019-10-13T15:29:51.739 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:51.730+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:51.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:51.838+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:51.842095+0000) 2019-10-13T15:29:52.188 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:29:52.188 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:52.192 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:29:52.192 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:29:52.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:52.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:52.357149+0000) 2019-10-13T15:29:52.718 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:52.714+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:52.716133+0000) 2019-10-13T15:29:54.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:54.722+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:54.724554+0000) 2019-10-13T15:29:55.608 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:55.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:55.605824+0000) 2019-10-13T15:29:56.744 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:29:56.738+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.742124+0000) 2019-10-13T15:29:56.745 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:29:56.737+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.742580+0000) 2019-10-13T15:29:56.745 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:29:56.737+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.743115+0000) 2019-10-13T15:29:56.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:29:56.737+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.743559+0000) 2019-10-13T15:29:56.746 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:56.737+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.743930+0000) 2019-10-13T15:29:56.748 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:56.737+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.745602+0000) 2019-10-13T15:29:56.748 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:29:56.742+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.746090+0000) 2019-10-13T15:29:56.749 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:29:56.741+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:56.746208+0000) 2019-10-13T15:29:56.753 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:29:56.745+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227872 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:29:58.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:29:58.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:58.094220+0000) 2019-10-13T15:29:58.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:29:58.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:29:58.318652+0000) 2019-10-13T15:30:01.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:01.185+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.189811+0000) 2019-10-13T15:30:01.757 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:01.749+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.753787+0000) 2019-10-13T15:30:01.757 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:01.749+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.754396+0000) 2019-10-13T15:30:01.758 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:01.749+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.756219+0000) 2019-10-13T15:30:01.759 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:01.754+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.756678+0000) 2019-10-13T15:30:01.760 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:01.749+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.757481+0000) 2019-10-13T15:30:01.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:01.753+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.758577+0000) 2019-10-13T15:30:01.762 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:01.753+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.759457+0000) 2019-10-13T15:30:01.767 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:01.762+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.764581+0000) 2019-10-13T15:30:01.767 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:01.757+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227902 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:01.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:01.837+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:01.842299+0000) 2019-10-13T15:30:02.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:02.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:02.357326+0000) 2019-10-13T15:30:02.725 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:02.722+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:02.722531+0000) 2019-10-13T15:30:04.727 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:04.721+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:04.724752+0000) 2019-10-13T15:30:05.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:05.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:05.606084+0000) 2019-10-13T15:30:06.771 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:06.761+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.768388+0000) 2019-10-13T15:30:06.772 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:06.761+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.769105+0000) 2019-10-13T15:30:06.772 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:06.761+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.769634+0000) 2019-10-13T15:30:06.773 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:06.766+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.770541+0000) 2019-10-13T15:30:06.773 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:06.765+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.770021+0000) 2019-10-13T15:30:06.774 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:06.765+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.770880+0000) 2019-10-13T15:30:06.774 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:06.765+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.771398+0000) 2019-10-13T15:30:06.781 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:06.774+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:06.778301+0000) 2019-10-13T15:30:06.782 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:06.773+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:08.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:08.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:08.094425+0000) 2019-10-13T15:30:08.321 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:08.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:08.318851+0000) 2019-10-13T15:30:11.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:11.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.190035+0000) 2019-10-13T15:30:11.785 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:11.781+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.782741+0000) 2019-10-13T15:30:11.787 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:11.781+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.784602+0000) 2019-10-13T15:30:11.787 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:11.781+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.784181+0000) 2019-10-13T15:30:11.788 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:11.782+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.785674+0000) 2019-10-13T15:30:11.788 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:11.785+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.785775+0000) 2019-10-13T15:30:11.789 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:11.785+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.786335+0000) 2019-10-13T15:30:11.789 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:11.785+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.786780+0000) 2019-10-13T15:30:11.792 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:11.786+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.790138+0000) 2019-10-13T15:30:11.796 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:11.793+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227962 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:11.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:11.841+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:11.842497+0000) 2019-10-13T15:30:12.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:12.354+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:12.357514+0000) 2019-10-13T15:30:12.731 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:12.726+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:12.729110+0000) 2019-10-13T15:30:14.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:14.721+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:14.725009+0000) 2019-10-13T15:30:15.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:15.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:15.606286+0000) 2019-10-13T15:30:16.801 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:16.797+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.798577+0000) 2019-10-13T15:30:16.803 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:16.797+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.800870+0000) 2019-10-13T15:30:16.806 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:16.802+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.804184+0000) 2019-10-13T15:30:16.808 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:16.805+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.805818+0000) 2019-10-13T15:30:16.809 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:16.805+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.806950+0000) 2019-10-13T15:30:16.810 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:16.805+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.807489+0000) 2019-10-13T15:30:16.811 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:16.805+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.808228+0000) 2019-10-13T15:30:16.811 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:16.805+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227994 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:16.818 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:16.810+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:16.815755+0000) 2019-10-13T15:30:18.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:18.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:18.094643+0000) 2019-10-13T15:30:18.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:18.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:18.319048+0000) 2019-10-13T15:30:21.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:21.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.190239+0000) 2019-10-13T15:30:21.816 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:21.809+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.813386+0000) 2019-10-13T15:30:21.817 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:21.809+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.813350+0000) 2019-10-13T15:30:21.817 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:21.813+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.813698+0000) 2019-10-13T15:30:21.817 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:21.813+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.814894+0000) 2019-10-13T15:30:21.818 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:21.810+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.815725+0000) 2019-10-13T15:30:21.819 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:21.813+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.815911+0000) 2019-10-13T15:30:21.819 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:21.813+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.816296+0000) 2019-10-13T15:30:21.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:21.818+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.820542+0000) 2019-10-13T15:30:21.826 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:21.821+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228024 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:21.845 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:21.841+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:21.842689+0000) 2019-10-13T15:30:22.336 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:30:22.336 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:22.340 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:30:22.340 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:22.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:22.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:22.357697+0000) 2019-10-13T15:30:22.738 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:22.729+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:22.735525+0000) 2019-10-13T15:30:24.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:24.721+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:24.725205+0000) 2019-10-13T15:30:25.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:25.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:25.606476+0000) 2019-10-13T15:30:26.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:26.825+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.827822+0000) 2019-10-13T15:30:26.831 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:26.825+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.828419+0000) 2019-10-13T15:30:26.831 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:26.825+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.828585+0000) 2019-10-13T15:30:26.832 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:26.825+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.829141+0000) 2019-10-13T15:30:26.832 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:26.825+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.829468+0000) 2019-10-13T15:30:26.833 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:26.829+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.829929+0000) 2019-10-13T15:30:26.833 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:26.825+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.830515+0000) 2019-10-13T15:30:26.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:26.833+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:26.838702+0000) 2019-10-13T15:30:26.842 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:26.837+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228056 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:28.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:28.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:28.094857+0000) 2019-10-13T15:30:28.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:28.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:28.319242+0000) 2019-10-13T15:30:31.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:31.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.190444+0000) 2019-10-13T15:30:31.845 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:31.841+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.842068+0000) 2019-10-13T15:30:31.846 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:31.837+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.843827+0000) 2019-10-13T15:30:31.848 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:31.841+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.844824+0000) 2019-10-13T15:30:31.848 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:31.841+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.844990+0000) 2019-10-13T15:30:31.848 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:31.841+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.845320+0000) 2019-10-13T15:30:31.848 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:31.841+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.845626+0000) 2019-10-13T15:30:31.849 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:31.845+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.845944+0000) 2019-10-13T15:30:31.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:31.845+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:31.850748+0000) 2019-10-13T15:30:31.856 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:31.849+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228084 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:32.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:32.357+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:32.357890+0000) 2019-10-13T15:30:32.744 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:32.741+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:32.742062+0000) 2019-10-13T15:30:34.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:34.721+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:34.725426+0000) 2019-10-13T15:30:35.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:35.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:35.606687+0000) 2019-10-13T15:30:36.859 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:36.853+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.856432+0000) 2019-10-13T15:30:36.859 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:36.853+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.856685+0000) 2019-10-13T15:30:36.860 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:36.857+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.858246+0000) 2019-10-13T15:30:36.861 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:36.857+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.858601+0000) 2019-10-13T15:30:36.863 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:36.857+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.859856+0000) 2019-10-13T15:30:36.863 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:36.857+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.861008+0000) 2019-10-13T15:30:36.866 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:36.861+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.863537+0000) 2019-10-13T15:30:36.870 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:36.865+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:36.868343+0000) 2019-10-13T15:30:36.871 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:36.865+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228116 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:38.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:38.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:38.095061+0000) 2019-10-13T15:30:38.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:38.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:38.319429+0000) 2019-10-13T15:30:38.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:38.589+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:38.593355+0000) 2019-10-13T15:30:41.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:41.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.190718+0000) 2019-10-13T15:30:41.877 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:41.873+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.874015+0000) 2019-10-13T15:30:41.877 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:41.873+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.874808+0000) 2019-10-13T15:30:41.879 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:41.873+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.876690+0000) 2019-10-13T15:30:41.880 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:41.873+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.877065+0000) 2019-10-13T15:30:41.880 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:41.873+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.877241+0000) 2019-10-13T15:30:41.880 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:41.877+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.877886+0000) 2019-10-13T15:30:41.881 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:41.877+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.878591+0000) 2019-10-13T15:30:41.884 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:41.877+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:41.882277+0000) 2019-10-13T15:30:41.886 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:41.881+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228146 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:42.360 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:42.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:42.358062+0000) 2019-10-13T15:30:42.755 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:42.749+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:42.752699+0000) 2019-10-13T15:30:44.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:44.725+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:44.725655+0000) 2019-10-13T15:30:45.609 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:45.605+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:45.606880+0000) 2019-10-13T15:30:46.890 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:46.881+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.887637+0000) 2019-10-13T15:30:46.891 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:46.885+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.888572+0000) 2019-10-13T15:30:46.893 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:46.889+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.890775+0000) 2019-10-13T15:30:46.893 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:46.889+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.890929+0000) 2019-10-13T15:30:46.895 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:46.889+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.892445+0000) 2019-10-13T15:30:46.895 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:46.889+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.892713+0000) 2019-10-13T15:30:46.896 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:46.889+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.892821+0000) 2019-10-13T15:30:46.896 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:46.893+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:46.893841+0000) 2019-10-13T15:30:46.902 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:46.897+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:48.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:48.093+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:48.095265+0000) 2019-10-13T15:30:48.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:48.317+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:48.319650+0000) 2019-10-13T15:30:48.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:48.589+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:48.593578+0000) 2019-10-13T15:30:51.193 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:51.189+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.190938+0000) 2019-10-13T15:30:51.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:51.901+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.902569+0000) 2019-10-13T15:30:51.906 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:51.901+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.903338+0000) 2019-10-13T15:30:51.906 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:51.901+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.903746+0000) 2019-10-13T15:30:51.907 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:51.901+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.904458+0000) 2019-10-13T15:30:51.907 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:51.901+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.905153+0000) 2019-10-13T15:30:51.908 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:51.905+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.905993+0000) 2019-10-13T15:30:51.909 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:51.905+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.906533+0000) 2019-10-13T15:30:51.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:51.905+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:51.912076+0000) 2019-10-13T15:30:51.917 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:51.913+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228206 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:52.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:52.357+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:52.358238+0000) 2019-10-13T15:30:52.386 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:30:52.387 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:52.391 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:30:52.391 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:30:52.762 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:52.757+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:52.759899+0000) 2019-10-13T15:30:54.728 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:54.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:54.725846+0000) 2019-10-13T15:30:55.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:55.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:55.607072+0000) 2019-10-13T15:30:56.920 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:30:56.913+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.917765+0000) 2019-10-13T15:30:56.921 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:56.916+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.918316+0000) 2019-10-13T15:30:56.921 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:56.916+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.918648+0000) 2019-10-13T15:30:56.923 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:30:56.916+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.919472+0000) 2019-10-13T15:30:56.924 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:30:56.916+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.920097+0000) 2019-10-13T15:30:56.924 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:30:56.916+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.920257+0000) 2019-10-13T15:30:56.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:56.916+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.920732+0000) 2019-10-13T15:30:56.930 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:30:56.921+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:56.927702+0000) 2019-10-13T15:30:56.930 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:30:56.924+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228237 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:30:58.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:30:58.092+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:58.095476+0000) 2019-10-13T15:30:58.322 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:30:58.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:58.319864+0000) 2019-10-13T15:30:58.596 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:30:58.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:30:58.593790+0000) 2019-10-13T15:31:01.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:01.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.191153+0000) 2019-10-13T15:31:01.934 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:01.928+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.931665+0000) 2019-10-13T15:31:01.937 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:01.929+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.935134+0000) 2019-10-13T15:31:01.939 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:01.932+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.936634+0000) 2019-10-13T15:31:01.940 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:01.933+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.937636+0000) 2019-10-13T15:31:01.940 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:01.936+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.938219+0000) 2019-10-13T15:31:01.941 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:01.936+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.938829+0000) 2019-10-13T15:31:01.942 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:01.936+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.939717+0000) 2019-10-13T15:31:01.943 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:01.936+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:01.940388+0000) 2019-10-13T15:31:01.945 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:01.940+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228268 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:02.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:02.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:02.358406+0000) 2019-10-13T15:31:02.769 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:02.761+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:02.767113+0000) 2019-10-13T15:31:04.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:04.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:04.726069+0000) 2019-10-13T15:31:05.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:05.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:05.607303+0000) 2019-10-13T15:31:06.950 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:06.944+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.947199+0000) 2019-10-13T15:31:06.950 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:06.944+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.947999+0000) 2019-10-13T15:31:06.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:06.944+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.948525+0000) 2019-10-13T15:31:06.951 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:06.944+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.948648+0000) 2019-10-13T15:31:06.952 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:06.945+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.949881+0000) 2019-10-13T15:31:06.952 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:06.948+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.950284+0000) 2019-10-13T15:31:06.953 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:06.948+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.950539+0000) 2019-10-13T15:31:06.958 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:06.949+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:06.955550+0000) 2019-10-13T15:31:06.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:06.952+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:08.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:08.092+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:08.095680+0000) 2019-10-13T15:31:08.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:08.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:08.320088+0000) 2019-10-13T15:31:08.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:08.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:08.593985+0000) 2019-10-13T15:31:11.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:11.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.191356+0000) 2019-10-13T15:31:11.964 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:11.956+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.961560+0000) 2019-10-13T15:31:11.968 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:11.957+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.963588+0000) 2019-10-13T15:31:11.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:11.960+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.962575+0000) 2019-10-13T15:31:11.968 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:11.960+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.962678+0000) 2019-10-13T15:31:11.969 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:11.960+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.962982+0000) 2019-10-13T15:31:11.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:11.960+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.963864+0000) 2019-10-13T15:31:11.969 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:11.960+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.964659+0000) 2019-10-13T15:31:11.970 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:11.965+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:11.968349+0000) 2019-10-13T15:31:11.974 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:11.968+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228328 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:12.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:12.353+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:12.358560+0000) 2019-10-13T15:31:12.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:12.773+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:12.774388+0000) 2019-10-13T15:31:14.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:14.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:14.726289+0000) 2019-10-13T15:31:15.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:15.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:15.607525+0000) 2019-10-13T15:31:16.978 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:16.972+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.975047+0000) 2019-10-13T15:31:16.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:16.972+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.976023+0000) 2019-10-13T15:31:16.979 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:16.972+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.976044+0000) 2019-10-13T15:31:16.979 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:16.972+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.976927+0000) 2019-10-13T15:31:16.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:16.972+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.977295+0000) 2019-10-13T15:31:16.980 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:16.976+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.978083+0000) 2019-10-13T15:31:16.981 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:16.973+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.978965+0000) 2019-10-13T15:31:16.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:16.981+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:16.984943+0000) 2019-10-13T15:31:16.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:16.980+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228359 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:18.098 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:18.092+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:18.095912+0000) 2019-10-13T15:31:18.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:18.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:18.320283+0000) 2019-10-13T15:31:18.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:18.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:18.594195+0000) 2019-10-13T15:31:21.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:21.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.191554+0000) 2019-10-13T15:31:21.993 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:21.988+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.990100+0000) 2019-10-13T15:31:21.993 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:21.988+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.989825+0000) 2019-10-13T15:31:21.994 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:21.988+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.990703+0000) 2019-10-13T15:31:21.994 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:21.988+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.991187+0000) 2019-10-13T15:31:21.994 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:21.988+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.991639+0000) 2019-10-13T15:31:21.995 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:21.988+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.993254+0000) 2019-10-13T15:31:21.996 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:21.989+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.993940+0000) 2019-10-13T15:31:22.000 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:21.993+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:21.997132+0000) 2019-10-13T15:31:22.003 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:21.996+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228390 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:22.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:22.357+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:22.358763+0000) 2019-10-13T15:31:22.508 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:31:22.508 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:22.511 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:31:22.512 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:22.784 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:22.777+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:22.781616+0000) 2019-10-13T15:31:24.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:24.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:24.726504+0000) 2019-10-13T15:31:25.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:25.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:25.607742+0000) 2019-10-13T15:31:27.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:27.000+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.004821+0000) 2019-10-13T15:31:27.008 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:27.000+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.005304+0000) 2019-10-13T15:31:27.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:27.004+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.006558+0000) 2019-10-13T15:31:27.009 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:27.001+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.006883+0000) 2019-10-13T15:31:27.010 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:27.004+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.007336+0000) 2019-10-13T15:31:27.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:27.004+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.007678+0000) 2019-10-13T15:31:27.013 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:27.005+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.011067+0000) 2019-10-13T15:31:27.018 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:27.012+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228421 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:27.076 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:27.000+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:27.004042+0000) 2019-10-13T15:31:28.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:28.092+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:28.096132+0000) 2019-10-13T15:31:28.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:28.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:28.320502+0000) 2019-10-13T15:31:28.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:28.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:28.594414+0000) 2019-10-13T15:31:31.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:31.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:31.191785+0000) 2019-10-13T15:31:32.024 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:32.016+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.019944+0000) 2019-10-13T15:31:32.024 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:32.016+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.020933+0000) 2019-10-13T15:31:32.025 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:32.016+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.021317+0000) 2019-10-13T15:31:32.025 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:32.020+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.021890+0000) 2019-10-13T15:31:32.026 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:32.016+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.021729+0000) 2019-10-13T15:31:32.026 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:32.020+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.022115+0000) 2019-10-13T15:31:32.026 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:32.020+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.022993+0000) 2019-10-13T15:31:32.029 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:32.020+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.026419+0000) 2019-10-13T15:31:32.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:32.028+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228450 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:32.361 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:32.356+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.358974+0000) 2019-10-13T15:31:32.791 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:32.784+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:32.788770+0000) 2019-10-13T15:31:34.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:34.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:34.726741+0000) 2019-10-13T15:31:35.610 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:35.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:35.607938+0000) 2019-10-13T15:31:37.036 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:37.028+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.033583+0000) 2019-10-13T15:31:37.037 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:37.032+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.034417+0000) 2019-10-13T15:31:37.037 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:37.032+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.034777+0000) 2019-10-13T15:31:37.038 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:37.032+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.035446+0000) 2019-10-13T15:31:37.038 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:37.028+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.035867+0000) 2019-10-13T15:31:37.038 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:37.032+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.036192+0000) 2019-10-13T15:31:37.039 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:37.032+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.036511+0000) 2019-10-13T15:31:37.045 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:37.036+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:37.042559+0000) 2019-10-13T15:31:37.048 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:37.040+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:38.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:38.092+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:38.096284+0000) 2019-10-13T15:31:38.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:38.320+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:38.320707+0000) 2019-10-13T15:31:38.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:38.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:38.594650+0000) 2019-10-13T15:31:41.194 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:41.188+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:41.191983+0000) 2019-10-13T15:31:42.051 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:42.044+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.048951+0000) 2019-10-13T15:31:42.053 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:42.048+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.050945+0000) 2019-10-13T15:31:42.054 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:42.048+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.051612+0000) 2019-10-13T15:31:42.054 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:42.048+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.051639+0000) 2019-10-13T15:31:42.055 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:42.048+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.052156+0000) 2019-10-13T15:31:42.055 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:42.048+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.052790+0000) 2019-10-13T15:31:42.056 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:42.048+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.053159+0000) 2019-10-13T15:31:42.062 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:42.052+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.059521+0000) 2019-10-13T15:31:42.063 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:42.056+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228512 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:42.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:42.356+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.359177+0000) 2019-10-13T15:31:42.798 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:42.792+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:42.795977+0000) 2019-10-13T15:31:44.729 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:44.724+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:44.726973+0000) 2019-10-13T15:31:45.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:45.604+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:45.608126+0000) 2019-10-13T15:31:47.067 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:47.060+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.064314+0000) 2019-10-13T15:31:47.067 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:47.060+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.064828+0000) 2019-10-13T15:31:47.070 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:47.064+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.067359+0000) 2019-10-13T15:31:47.070 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:47.064+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.067541+0000) 2019-10-13T15:31:47.070 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:47.064+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.068217+0000) 2019-10-13T15:31:47.071 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:47.064+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.068886+0000) 2019-10-13T15:31:47.072 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:47.064+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.069560+0000) 2019-10-13T15:31:47.074 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:47.064+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:47.071590+0000) 2019-10-13T15:31:47.078 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:47.072+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228543 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:48.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:48.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:48.096485+0000) 2019-10-13T15:31:48.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:48.316+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:48.320906+0000) 2019-10-13T15:31:48.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:48.592+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:48.594802+0000) 2019-10-13T15:31:51.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:51.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:51.192250+0000) 2019-10-13T15:31:52.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:52.075+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.080430+0000) 2019-10-13T15:31:52.085 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:52.079+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.082922+0000) 2019-10-13T15:31:52.086 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:52.076+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.083106+0000) 2019-10-13T15:31:52.086 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:52.079+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.083548+0000) 2019-10-13T15:31:52.086 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:52.079+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.083890+0000) 2019-10-13T15:31:52.088 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:52.079+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.085456+0000) 2019-10-13T15:31:52.088 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:52.079+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.085717+0000) 2019-10-13T15:31:52.089 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:52.080+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.086972+0000) 2019-10-13T15:31:52.092 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:52.087+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:52.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:52.356+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.359345+0000) 2019-10-13T15:31:52.548 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:31:52.549 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:52.552 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:31:52.552 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:31:52.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:52.796+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:52.803159+0000) 2019-10-13T15:31:54.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:54.723+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:54.727203+0000) 2019-10-13T15:31:55.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:55.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:55.608318+0000) 2019-10-13T15:31:57.095 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:57.087+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.093001+0000) 2019-10-13T15:31:57.096 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:57.087+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.093214+0000) 2019-10-13T15:31:57.097 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:57.091+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.095122+0000) 2019-10-13T15:31:57.099 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:31:57.092+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.096080+0000) 2019-10-13T15:31:57.099 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:31:57.091+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.095596+0000) 2019-10-13T15:31:57.099 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:31:57.091+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.095677+0000) 2019-10-13T15:31:57.099 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:31:57.091+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.097031+0000) 2019-10-13T15:31:57.105 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:31:57.096+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:57.103138+0000) 2019-10-13T15:31:57.107 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:31:57.099+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228602 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:31:58.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:31:58.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:58.096742+0000) 2019-10-13T15:31:58.323 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:31:58.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:58.321098+0000) 2019-10-13T15:31:58.597 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:31:58.591+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:31:58.595002+0000) 2019-10-13T15:32:01.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:01.187+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:01.192447+0000) 2019-10-13T15:32:02.111 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:02.103+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.107791+0000) 2019-10-13T15:32:02.111 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:02.103+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.108450+0000) 2019-10-13T15:32:02.112 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:02.103+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.109338+0000) 2019-10-13T15:32:02.113 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:02.104+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.111193+0000) 2019-10-13T15:32:02.114 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:02.107+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.111316+0000) 2019-10-13T15:32:02.114 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:02.107+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.111921+0000) 2019-10-13T15:32:02.115 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:02.107+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.112634+0000) 2019-10-13T15:32:02.119 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:02.112+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.116912+0000) 2019-10-13T15:32:02.123 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:02.115+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228633 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:02.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:02.356+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.359518+0000) 2019-10-13T15:32:02.814 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:02.808+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:02.811519+0000) 2019-10-13T15:32:04.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:04.723+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:04.727401+0000) 2019-10-13T15:32:05.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:05.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:05.608524+0000) 2019-10-13T15:32:07.126 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:07.119+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.123937+0000) 2019-10-13T15:32:07.127 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:07.119+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.124813+0000) 2019-10-13T15:32:07.127 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:07.119+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.124581+0000) 2019-10-13T15:32:07.128 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:07.119+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.125443+0000) 2019-10-13T15:32:07.129 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:07.120+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.126634+0000) 2019-10-13T15:32:07.131 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:07.123+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.127619+0000) 2019-10-13T15:32:07.132 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:07.123+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.127102+0000) 2019-10-13T15:32:07.136 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:07.128+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:07.133546+0000) 2019-10-13T15:32:07.137 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:07.131+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228663 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:08.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:08.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:08.096898+0000) 2019-10-13T15:32:08.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:08.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:08.321316+0000) 2019-10-13T15:32:08.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:08.591+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:08.595201+0000) 2019-10-13T15:32:11.195 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:11.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:11.192685+0000) 2019-10-13T15:32:12.140 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:12.131+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.137068+0000) 2019-10-13T15:32:12.141 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:12.135+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.138503+0000) 2019-10-13T15:32:12.141 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:12.132+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.138945+0000) 2019-10-13T15:32:12.142 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:12.135+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.139412+0000) 2019-10-13T15:32:12.143 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:12.135+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.140362+0000) 2019-10-13T15:32:12.143 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:12.135+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.139985+0000) 2019-10-13T15:32:12.143 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:12.135+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.140686+0000) 2019-10-13T15:32:12.152 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:12.143+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:12.155 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:12.148+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.152633+0000) 2019-10-13T15:32:12.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:12.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.359683+0000) 2019-10-13T15:32:12.825 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:12.812+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:12.818434+0000) 2019-10-13T15:32:14.730 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:14.723+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:14.727626+0000) 2019-10-13T15:32:15.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:15.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:15.608721+0000) 2019-10-13T15:32:17.155 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:17.147+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.152794+0000) 2019-10-13T15:32:17.156 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:17.147+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.153110+0000) 2019-10-13T15:32:17.156 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:17.147+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.153489+0000) 2019-10-13T15:32:17.157 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:17.151+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.153951+0000) 2019-10-13T15:32:17.157 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:17.151+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.155296+0000) 2019-10-13T15:32:17.158 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:17.151+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.155658+0000) 2019-10-13T15:32:17.158 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:17.152+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.156169+0000) 2019-10-13T15:32:17.164 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:17.156+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:17.162121+0000) 2019-10-13T15:32:17.165 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:17.159+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228722 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:18.099 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:18.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:18.097087+0000) 2019-10-13T15:32:18.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:18.315+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:18.321517+0000) 2019-10-13T15:32:18.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:18.591+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:18.595407+0000) 2019-10-13T15:32:21.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:21.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:21.192904+0000) 2019-10-13T15:32:22.174 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:22.163+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.166616+0000) 2019-10-13T15:32:22.174 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:22.163+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.166999+0000) 2019-10-13T15:32:22.174 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:22.163+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.167292+0000) 2019-10-13T15:32:22.175 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:22.163+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.169484+0000) 2019-10-13T15:32:22.175 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:22.167+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.170032+0000) 2019-10-13T15:32:22.175 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:22.167+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.171393+0000) 2019-10-13T15:32:22.175 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:22.168+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.172374+0000) 2019-10-13T15:32:22.179 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:22.172+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.176520+0000) 2019-10-13T15:32:22.180 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:22.171+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228753 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:22.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:22.352+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.359882+0000) 2019-10-13T15:32:22.775 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:32:22.775 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:22.779 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:32:22.779 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:22.827 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:22.820+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:22.825046+0000) 2019-10-13T15:32:24.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:24.727+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:24.727828+0000) 2019-10-13T15:32:25.611 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:25.603+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:25.608913+0000) 2019-10-13T15:32:27.184 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:27.175+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.181120+0000) 2019-10-13T15:32:27.184 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:27.179+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.181831+0000) 2019-10-13T15:32:27.184 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:27.179+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.182242+0000) 2019-10-13T15:32:27.185 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:27.179+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.182834+0000) 2019-10-13T15:32:27.186 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:27.176+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.183714+0000) 2019-10-13T15:32:27.187 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:27.179+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.184395+0000) 2019-10-13T15:32:27.187 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:27.179+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.184583+0000) 2019-10-13T15:32:27.189 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:27.180+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:27.187014+0000) 2019-10-13T15:32:27.194 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:27.187+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228782 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:28.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:28.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:28.097255+0000) 2019-10-13T15:32:28.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:28.319+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:28.321774+0000) 2019-10-13T15:32:28.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:28.591+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:28.595595+0000) 2019-10-13T15:32:31.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:31.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:31.193130+0000) 2019-10-13T15:32:32.199 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:32.191+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.196511+0000) 2019-10-13T15:32:32.200 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:32.191+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.197177+0000) 2019-10-13T15:32:32.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:32.195+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.197918+0000) 2019-10-13T15:32:32.201 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:32.195+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.198273+0000) 2019-10-13T15:32:32.201 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:32.195+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.198800+0000) 2019-10-13T15:32:32.201 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:32.196+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.199447+0000) 2019-10-13T15:32:32.203 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:32.195+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.200324+0000) 2019-10-13T15:32:32.208 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:32.204+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.206373+0000) 2019-10-13T15:32:32.209 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:32.203+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:32.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:32.356+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.360052+0000) 2019-10-13T15:32:32.834 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:32.828+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:32.831688+0000) 2019-10-13T15:32:34.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:34.723+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:34.728058+0000) 2019-10-13T15:32:35.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:35.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:35.609126+0000) 2019-10-13T15:32:37.213 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:37.207+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.210578+0000) 2019-10-13T15:32:37.214 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:37.207+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.211128+0000) 2019-10-13T15:32:37.214 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:37.207+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.211794+0000) 2019-10-13T15:32:37.214 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:37.207+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.212137+0000) 2019-10-13T15:32:37.215 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:37.207+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.212485+0000) 2019-10-13T15:32:37.216 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:37.207+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.213584+0000) 2019-10-13T15:32:37.217 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:37.211+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.214826+0000) 2019-10-13T15:32:37.221 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:37.215+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:37.219071+0000) 2019-10-13T15:32:37.223 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:37.215+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228841 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:38.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:38.091+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:38.097430+0000) 2019-10-13T15:32:38.324 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:38.319+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:38.321923+0000) 2019-10-13T15:32:38.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:38.591+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:38.595831+0000) 2019-10-13T15:32:41.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:41.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:41.193345+0000) 2019-10-13T15:32:42.228 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:42.223+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.225893+0000) 2019-10-13T15:32:42.229 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:42.223+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.226328+0000) 2019-10-13T15:32:42.230 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:42.223+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.227369+0000) 2019-10-13T15:32:42.230 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:42.223+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.227992+0000) 2019-10-13T15:32:42.231 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:42.223+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.228074+0000) 2019-10-13T15:32:42.231 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:42.223+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.228463+0000) 2019-10-13T15:32:42.231 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:42.227+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.229368+0000) 2019-10-13T15:32:42.235 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:42.231+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.232785+0000) 2019-10-13T15:32:42.239 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:42.231+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228874 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:42.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:42.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.360315+0000) 2019-10-13T15:32:42.841 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:42.835+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:42.838213+0000) 2019-10-13T15:32:44.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:44.727+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:44.728252+0000) 2019-10-13T15:32:45.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:45.602+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:45.609328+0000) 2019-10-13T15:32:47.242 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:47.234+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.239129+0000) 2019-10-13T15:32:47.243 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:47.234+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.240335+0000) 2019-10-13T15:32:47.243 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:47.234+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.240892+0000) 2019-10-13T15:32:47.244 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:47.234+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.241267+0000) 2019-10-13T15:32:47.244 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:47.238+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.242021+0000) 2019-10-13T15:32:47.245 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:47.238+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.242509+0000) 2019-10-13T15:32:47.245 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:47.239+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.243115+0000) 2019-10-13T15:32:47.251 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:47.247+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:47.248955+0000) 2019-10-13T15:32:47.253 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:47.246+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228903 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:48.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:48.094+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:48.097626+0000) 2019-10-13T15:32:48.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:48.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:48.322125+0000) 2019-10-13T15:32:48.598 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:48.594+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:48.596025+0000) 2019-10-13T15:32:51.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:51.186+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:51.193543+0000) 2019-10-13T15:32:52.256 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:52.246+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.253512+0000) 2019-10-13T15:32:52.257 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:52.250+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.254846+0000) 2019-10-13T15:32:52.258 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:52.250+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.255248+0000) 2019-10-13T15:32:52.258 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:52.250+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.255578+0000) 2019-10-13T15:32:52.259 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:52.251+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.255925+0000) 2019-10-13T15:32:52.259 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:52.250+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.256480+0000) 2019-10-13T15:32:52.260 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:52.250+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.257254+0000) 2019-10-13T15:32:52.266 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:52.259+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.263665+0000) 2019-10-13T15:32:52.268 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:52.262+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228934 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:52.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:52.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.360487+0000) 2019-10-13T15:32:52.812 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:32:52.812 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:52.816 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:32:52.817 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:32:52.847 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:52.843+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:52.844698+0000) 2019-10-13T15:32:54.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:54.722+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:54.728472+0000) 2019-10-13T15:32:55.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:55.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:55.609532+0000) 2019-10-13T15:32:57.271 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:32:57.267+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.269005+0000) 2019-10-13T15:32:57.272 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:57.262+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.269200+0000) 2019-10-13T15:32:57.273 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:32:57.262+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.269616+0000) 2019-10-13T15:32:57.273 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:57.266+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.269890+0000) 2019-10-13T15:32:57.274 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:32:57.266+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.271315+0000) 2019-10-13T15:32:57.274 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:32:57.266+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.271136+0000) 2019-10-13T15:32:57.274 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:57.266+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.271702+0000) 2019-10-13T15:32:57.281 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:32:57.275+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:57.278625+0000) 2019-10-13T15:32:57.282 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:32:57.274+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228963 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:32:58.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:32:58.094+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:58.097828+0000) 2019-10-13T15:32:58.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:32:58.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:58.322363+0000) 2019-10-13T15:32:58.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:32:58.590+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:32:58.596259+0000) 2019-10-13T15:33:01.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:01.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:01.193778+0000) 2019-10-13T15:33:02.286 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:02.278+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.283660+0000) 2019-10-13T15:33:02.287 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:02.283+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.284460+0000) 2019-10-13T15:33:02.287 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:02.278+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.285011+0000) 2019-10-13T15:33:02.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:02.278+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.285065+0000) 2019-10-13T15:33:02.288 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:02.278+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.285620+0000) 2019-10-13T15:33:02.288 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:02.278+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.285399+0000) 2019-10-13T15:33:02.289 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:02.282+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.286240+0000) 2019-10-13T15:33:02.296 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:02.286+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228996 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:02.296 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:02.291+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.293463+0000) 2019-10-13T15:33:02.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:02.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.360668+0000) 2019-10-13T15:33:02.854 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:02.847+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:02.851241+0000) 2019-10-13T15:33:04.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:04.726+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:04.728673+0000) 2019-10-13T15:33:05.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:05.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:05.609771+0000) 2019-10-13T15:33:07.301 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:07.294+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.298395+0000) 2019-10-13T15:33:07.301 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:07.294+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.298626+0000) 2019-10-13T15:33:07.303 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:07.299+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.300471+0000) 2019-10-13T15:33:07.304 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:07.298+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.302108+0000) 2019-10-13T15:33:07.305 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:07.298+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.302303+0000) 2019-10-13T15:33:07.305 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:07.298+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.303133+0000) 2019-10-13T15:33:07.306 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:07.298+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.303542+0000) 2019-10-13T15:33:07.310 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:07.302+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229025 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:07.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:07.307+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:07.310212+0000) 2019-10-13T15:33:08.100 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:08.094+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:08.097966+0000) 2019-10-13T15:33:08.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:08.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:08.322593+0000) 2019-10-13T15:33:08.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:08.590+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:08.596482+0000) 2019-10-13T15:33:11.196 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:11.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:11.193964+0000) 2019-10-13T15:33:12.314 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:12.306+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.311119+0000) 2019-10-13T15:33:12.314 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:12.306+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.311359+0000) 2019-10-13T15:33:12.314 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:12.306+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.311970+0000) 2019-10-13T15:33:12.316 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:12.311+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.314000+0000) 2019-10-13T15:33:12.317 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:12.310+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.314039+0000) 2019-10-13T15:33:12.317 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:12.310+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.314407+0000) 2019-10-13T15:33:12.317 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:12.310+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.314884+0000) 2019-10-13T15:33:12.321 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:12.315+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.319413+0000) 2019-10-13T15:33:12.326 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:12.318+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229055 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:12.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:12.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.360838+0000) 2019-10-13T15:33:12.860 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:12.855+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:12.857695+0000) 2019-10-13T15:33:14.731 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:14.726+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:14.728898+0000) 2019-10-13T15:33:15.612 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:15.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:15.609938+0000) 2019-10-13T15:33:17.330 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:17.322+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.327498+0000) 2019-10-13T15:33:17.330 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:17.327+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.328208+0000) 2019-10-13T15:33:17.331 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:17.322+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.328374+0000) 2019-10-13T15:33:17.331 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:17.322+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.328834+0000) 2019-10-13T15:33:17.332 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:17.322+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.329673+0000) 2019-10-13T15:33:17.332 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:17.326+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.329941+0000) 2019-10-13T15:33:17.333 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:17.326+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.330312+0000) 2019-10-13T15:33:17.339 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:17.335+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:17.336966+0000) 2019-10-13T15:33:17.341 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:17.334+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229085 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:18.108 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:18.094+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:18.098138+0000) 2019-10-13T15:33:18.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:18.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:18.322817+0000) 2019-10-13T15:33:18.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:18.590+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:18.596712+0000) 2019-10-13T15:33:21.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:21.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:21.194131+0000) 2019-10-13T15:33:22.344 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:22.338+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.341601+0000) 2019-10-13T15:33:22.345 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:22.338+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.343165+0000) 2019-10-13T15:33:22.346 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:22.338+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.343527+0000) 2019-10-13T15:33:22.347 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:22.338+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.344555+0000) 2019-10-13T15:33:22.347 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:22.343+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.345011+0000) 2019-10-13T15:33:22.348 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:22.338+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.345194+0000) 2019-10-13T15:33:22.348 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:22.342+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.345835+0000) 2019-10-13T15:33:22.354 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:22.347+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.351853+0000) 2019-10-13T15:33:22.355 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:22.346+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229117 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:22.363 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:22.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.361024+0000) 2019-10-13T15:33:22.866 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:22.863+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:22.864181+0000) 2019-10-13T15:33:22.869 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:33:22.869 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:22.873 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:33:22.873 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:24.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:24.726+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:24.729136+0000) 2019-10-13T15:33:25.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:25.606+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:25.610126+0000) 2019-10-13T15:33:27.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:27.350+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.356086+0000) 2019-10-13T15:33:27.359 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:27.350+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.356783+0000) 2019-10-13T15:33:27.361 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:27.354+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.358183+0000) 2019-10-13T15:33:27.361 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:27.354+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.358038+0000) 2019-10-13T15:33:27.361 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:27.354+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.358605+0000) 2019-10-13T15:33:27.362 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:27.354+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.359310+0000) 2019-10-13T15:33:27.362 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:27.355+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.359722+0000) 2019-10-13T15:33:27.368 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:27.363+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:27.366300+0000) 2019-10-13T15:33:27.372 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:27.362+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229147 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:28.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:28.094+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:28.098322+0000) 2019-10-13T15:33:28.325 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:28.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:28.322995+0000) 2019-10-13T15:33:28.599 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:28.590+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:28.596868+0000) 2019-10-13T15:33:31.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:31.190+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:31.194305+0000) 2019-10-13T15:33:32.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:32.359+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.361201+0000) 2019-10-13T15:33:32.374 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:32.366+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.371505+0000) 2019-10-13T15:33:32.375 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:32.366+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.372935+0000) 2019-10-13T15:33:32.376 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:32.370+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.373723+0000) 2019-10-13T15:33:32.377 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:32.366+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.373566+0000) 2019-10-13T15:33:32.377 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:32.370+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.375198+0000) 2019-10-13T15:33:32.379 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:32.375+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.376735+0000) 2019-10-13T15:33:32.383 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:32.374+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.380381+0000) 2019-10-13T15:33:32.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:32.378+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229177 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:32.391 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:32.387+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.388870+0000) 2019-10-13T15:33:32.873 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:32.867+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:32.870724+0000) 2019-10-13T15:33:34.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:34.726+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:34.729369+0000) 2019-10-13T15:33:35.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:35.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:35.610347+0000) 2019-10-13T15:33:37.393 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:37.386+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.389936+0000) 2019-10-13T15:33:37.393 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:37.386+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.390210+0000) 2019-10-13T15:33:37.393 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:37.386+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.390574+0000) 2019-10-13T15:33:37.395 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:37.386+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.392366+0000) 2019-10-13T15:33:37.395 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:37.386+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.392856+0000) 2019-10-13T15:33:37.397 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:37.390+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.394887+0000) 2019-10-13T15:33:37.398 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:37.391+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.395480+0000) 2019-10-13T15:33:37.402 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:37.394+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229208 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:37.403 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:37.399+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:37.400739+0000) 2019-10-13T15:33:38.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:38.098+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:38.098551+0000) 2019-10-13T15:33:38.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:38.318+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:38.323134+0000) 2019-10-13T15:33:38.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:38.590+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:38.597061+0000) 2019-10-13T15:33:41.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:41.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:41.194539+0000) 2019-10-13T15:33:42.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:42.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.361437+0000) 2019-10-13T15:33:42.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:42.398+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.403620+0000) 2019-10-13T15:33:42.406 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:42.397+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.403925+0000) 2019-10-13T15:33:42.407 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:42.397+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.404338+0000) 2019-10-13T15:33:42.407 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:42.397+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.404608+0000) 2019-10-13T15:33:42.408 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:42.397+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.405288+0000) 2019-10-13T15:33:42.408 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:42.397+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.405690+0000) 2019-10-13T15:33:42.409 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:42.401+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.407179+0000) 2019-10-13T15:33:42.415 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:42.410+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.413040+0000) 2019-10-13T15:33:42.418 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:42.409+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229239 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:42.880 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:42.874+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:42.877333+0000) 2019-10-13T15:33:44.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:44.725+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:44.729618+0000) 2019-10-13T15:33:45.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:45.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:45.610538+0000) 2019-10-13T15:33:47.421 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:47.413+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.418339+0000) 2019-10-13T15:33:47.422 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:47.413+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.419993+0000) 2019-10-13T15:33:47.424 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:47.413+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.420910+0000) 2019-10-13T15:33:47.424 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:47.418+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.420814+0000) 2019-10-13T15:33:47.425 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:47.413+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.421395+0000) 2019-10-13T15:33:47.425 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:47.417+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.421732+0000) 2019-10-13T15:33:47.425 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:47.417+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.422430+0000) 2019-10-13T15:33:47.429 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:47.422+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:47.426704+0000) 2019-10-13T15:33:47.432 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:47.425+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229269 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:48.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:48.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:48.098769+0000) 2019-10-13T15:33:48.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:48.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:48.323333+0000) 2019-10-13T15:33:48.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:48.589+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:48.597261+0000) 2019-10-13T15:33:51.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:51.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:51.194746+0000) 2019-10-13T15:33:52.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:52.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.361634+0000) 2019-10-13T15:33:52.435 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:52.429+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.432540+0000) 2019-10-13T15:33:52.436 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:52.429+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.434239+0000) 2019-10-13T15:33:52.437 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:52.429+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.434471+0000) 2019-10-13T15:33:52.438 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:52.429+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.435627+0000) 2019-10-13T15:33:52.439 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:52.434+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.436122+0000) 2019-10-13T15:33:52.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:52.429+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.436270+0000) 2019-10-13T15:33:52.439 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:52.429+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.436556+0000) 2019-10-13T15:33:52.444 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:52.438+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.441807+0000) 2019-10-13T15:33:52.447 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:52.437+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229299 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:52.886 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:52.882+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:52.884015+0000) 2019-10-13T15:33:53.000 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:33:53.000 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:53.003 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:33:53.004 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:33:54.732 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:54.729+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:54.729822+0000) 2019-10-13T15:33:55.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:55.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:55.610744+0000) 2019-10-13T15:33:57.450 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:57.445+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.447568+0000) 2019-10-13T15:33:57.451 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:57.445+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.448617+0000) 2019-10-13T15:33:57.451 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:33:57.445+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.449117+0000) 2019-10-13T15:33:57.452 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:57.445+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.449533+0000) 2019-10-13T15:33:57.453 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:33:57.446+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.451387+0000) 2019-10-13T15:33:57.454 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:33:57.449+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.451385+0000) 2019-10-13T15:33:57.454 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:33:57.449+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.452156+0000) 2019-10-13T15:33:57.457 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:33:57.450+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:57.454573+0000) 2019-10-13T15:33:57.462 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:33:57.457+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229331 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:33:58.101 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:33:58.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:58.098977+0000) 2019-10-13T15:33:58.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:33:58.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:58.323542+0000) 2019-10-13T15:33:58.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:33:58.593+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:33:58.597460+0000) 2019-10-13T15:34:01.197 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:01.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:01.194926+0000) 2019-10-13T15:34:02.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:02.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.361828+0000) 2019-10-13T15:34:02.465 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:02.461+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.462473+0000) 2019-10-13T15:34:02.512 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:02.462+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.466037+0000) 2019-10-13T15:34:02.512 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:02.466+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.469715+0000) 2019-10-13T15:34:02.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:02.461+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.463129+0000) 2019-10-13T15:34:02.512 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:02.461+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.464593+0000) 2019-10-13T15:34:02.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:02.461+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.465115+0000) 2019-10-13T15:34:02.513 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:02.461+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.465462+0000) 2019-10-13T15:34:02.513 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:02.465+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.466648+0000) 2019-10-13T15:34:02.513 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:02.473+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229361 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:02.893 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:02.886+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:02.890666+0000) 2019-10-13T15:34:04.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:04.729+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:04.730040+0000) 2019-10-13T15:34:05.613 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:05.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:05.610942+0000) 2019-10-13T15:34:07.481 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:07.474+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.479079+0000) 2019-10-13T15:34:07.482 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:07.477+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.480089+0000) 2019-10-13T15:34:07.483 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:07.477+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.480832+0000) 2019-10-13T15:34:07.483 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:07.477+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.480951+0000) 2019-10-13T15:34:07.484 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:07.477+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.481464+0000) 2019-10-13T15:34:07.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:07.481+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.483394+0000) 2019-10-13T15:34:07.488 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:07.485+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.485833+0000) 2019-10-13T15:34:07.492 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:07.485+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229391 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:07.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:07.490+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:07.493423+0000) 2019-10-13T15:34:08.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:08.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:08.099171+0000) 2019-10-13T15:34:08.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:08.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:08.323733+0000) 2019-10-13T15:34:08.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:08.593+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:08.597660+0000) 2019-10-13T15:34:11.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:11.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:11.195130+0000) 2019-10-13T15:34:12.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:12.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.362050+0000) 2019-10-13T15:34:12.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:12.489+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.492946+0000) 2019-10-13T15:34:12.496 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:12.489+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.493258+0000) 2019-10-13T15:34:12.496 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:12.493+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.493977+0000) 2019-10-13T15:34:12.497 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:12.493+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.494819+0000) 2019-10-13T15:34:12.497 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:12.490+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.495360+0000) 2019-10-13T15:34:12.498 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:12.493+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.495514+0000) 2019-10-13T15:34:12.499 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:12.493+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.495689+0000) 2019-10-13T15:34:12.503 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:12.498+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.501063+0000) 2019-10-13T15:34:12.507 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:12.501+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229420 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:12.900 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:12.894+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:12.897304+0000) 2019-10-13T15:34:14.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:14.729+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:14.730238+0000) 2019-10-13T15:34:15.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:15.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:15.611139+0000) 2019-10-13T15:34:17.510 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:17.505+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.508037+0000) 2019-10-13T15:34:17.511 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:17.506+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.508948+0000) 2019-10-13T15:34:17.512 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:17.505+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.509571+0000) 2019-10-13T15:34:17.512 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:17.509+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.509989+0000) 2019-10-13T15:34:17.513 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:17.509+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.510951+0000) 2019-10-13T15:34:17.514 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:17.509+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.512047+0000) 2019-10-13T15:34:17.516 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:17.509+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.513682+0000) 2019-10-13T15:34:17.517 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:17.510+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:17.514091+0000) 2019-10-13T15:34:17.522 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:17.517+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229452 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:18.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:18.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:18.099386+0000) 2019-10-13T15:34:18.326 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:18.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:18.323940+0000) 2019-10-13T15:34:18.600 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:18.597+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:18.597896+0000) 2019-10-13T15:34:21.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:21.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:21.195337+0000) 2019-10-13T15:34:22.364 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:22.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.362251+0000) 2019-10-13T15:34:22.525 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:22.521+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.522587+0000) 2019-10-13T15:34:22.526 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:22.521+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.523085+0000) 2019-10-13T15:34:22.528 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:22.525+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.525885+0000) 2019-10-13T15:34:22.529 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:22.525+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.526958+0000) 2019-10-13T15:34:22.530 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:22.525+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.527351+0000) 2019-10-13T15:34:22.530 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:22.522+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.527857+0000) 2019-10-13T15:34:22.530 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:22.525+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.528057+0000) 2019-10-13T15:34:22.535 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:22.530+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.533194+0000) 2019-10-13T15:34:22.538 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:22.533+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229481 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:22.906 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:22.898+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:22.904011+0000) 2019-10-13T15:34:23.081 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:34:23.081 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:23.085 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:34:23.085 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:24.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:24.729+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:24.730395+0000) 2019-10-13T15:34:25.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:25.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:25.611330+0000) 2019-10-13T15:34:27.540 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:27.533+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.537326+0000) 2019-10-13T15:34:27.540 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:27.537+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.538043+0000) 2019-10-13T15:34:27.541 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:27.537+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.538643+0000) 2019-10-13T15:34:27.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:27.537+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.539361+0000) 2019-10-13T15:34:27.543 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:27.538+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.540976+0000) 2019-10-13T15:34:27.543 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:27.537+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.541314+0000) 2019-10-13T15:34:27.544 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:27.541+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.542067+0000) 2019-10-13T15:34:27.551 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:27.546+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:27.549081+0000) 2019-10-13T15:34:27.552 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:27.545+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229511 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:28.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:28.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:28.099557+0000) 2019-10-13T15:34:28.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:28.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:28.324116+0000) 2019-10-13T15:34:28.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:28.597+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:28.598055+0000) 2019-10-13T15:34:31.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:31.193+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:31.195550+0000) 2019-10-13T15:34:32.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:32.358+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.362460+0000) 2019-10-13T15:34:32.561 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:32.554+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.556163+0000) 2019-10-13T15:34:32.562 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:32.549+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.553099+0000) 2019-10-13T15:34:32.562 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:32.549+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.553299+0000) 2019-10-13T15:34:32.562 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:32.553+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.554128+0000) 2019-10-13T15:34:32.562 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:32.553+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.554876+0000) 2019-10-13T15:34:32.563 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:32.553+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.555167+0000) 2019-10-13T15:34:32.563 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:32.553+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.556149+0000) 2019-10-13T15:34:32.566 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:32.561+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229540 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:32.567 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:32.562+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.564633+0000) 2019-10-13T15:34:32.913 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:32.906+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:32.910590+0000) 2019-10-13T15:34:34.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:34.729+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:34.730619+0000) 2019-10-13T15:34:35.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:35.609+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:35.611521+0000) 2019-10-13T15:34:37.570 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:37.565+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.567139+0000) 2019-10-13T15:34:37.570 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:37.565+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.567987+0000) 2019-10-13T15:34:37.571 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:37.565+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.568371+0000) 2019-10-13T15:34:37.572 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:37.565+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.568721+0000) 2019-10-13T15:34:37.573 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:37.569+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.569859+0000) 2019-10-13T15:34:37.573 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:37.569+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.570462+0000) 2019-10-13T15:34:37.574 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:37.566+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.570455+0000) 2019-10-13T15:34:37.578 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:37.570+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:37.575905+0000) 2019-10-13T15:34:37.582 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:37.577+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229572 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:38.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:38.097+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:38.099707+0000) 2019-10-13T15:34:38.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:38.321+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:38.324293+0000) 2019-10-13T15:34:38.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:38.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:38.598271+0000) 2019-10-13T15:34:41.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:41.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:41.195729+0000) 2019-10-13T15:34:42.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:42.362+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.362649+0000) 2019-10-13T15:34:42.585 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:42.578+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.582601+0000) 2019-10-13T15:34:42.586 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:42.580+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.582998+0000) 2019-10-13T15:34:42.586 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:42.580+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.583677+0000) 2019-10-13T15:34:42.588 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:42.580+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.585356+0000) 2019-10-13T15:34:42.588 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:42.580+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.585717+0000) 2019-10-13T15:34:42.590 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:42.584+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.587126+0000) 2019-10-13T15:34:42.591 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:42.584+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.588720+0000) 2019-10-13T15:34:42.595 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:42.590+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.592486+0000) 2019-10-13T15:34:42.597 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:42.592+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229601 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:42.919 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:42.914+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:42.917077+0000) 2019-10-13T15:34:44.733 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:44.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:44.730818+0000) 2019-10-13T15:34:45.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:45.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:45.611710+0000) 2019-10-13T15:34:47.600 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:47.592+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.597469+0000) 2019-10-13T15:34:47.601 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:47.596+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.598198+0000) 2019-10-13T15:34:47.601 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:47.596+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.598500+0000) 2019-10-13T15:34:47.602 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:47.593+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.599705+0000) 2019-10-13T15:34:47.603 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:47.596+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.600591+0000) 2019-10-13T15:34:47.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:47.596+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.600927+0000) 2019-10-13T15:34:47.604 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:47.596+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.601303+0000) 2019-10-13T15:34:47.608 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:47.601+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:47.605592+0000) 2019-10-13T15:34:47.613 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:47.608+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229631 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:48.102 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:48.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:48.099944+0000) 2019-10-13T15:34:48.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:48.320+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:48.324489+0000) 2019-10-13T15:34:48.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:48.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:48.598492+0000) 2019-10-13T15:34:51.198 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:51.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:51.195942+0000) 2019-10-13T15:34:52.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:52.361+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.362850+0000) 2019-10-13T15:34:52.617 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:52.612+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.614441+0000) 2019-10-13T15:34:52.617 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:52.612+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.614657+0000) 2019-10-13T15:34:52.619 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:52.612+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.616671+0000) 2019-10-13T15:34:52.619 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:52.612+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.616867+0000) 2019-10-13T15:34:52.623 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:52.617+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.620747+0000) 2019-10-13T15:34:52.625 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:52.620+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.622360+0000) 2019-10-13T15:34:52.625 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:52.620+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.622878+0000) 2019-10-13T15:34:52.627 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:52.621+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.624800+0000) 2019-10-13T15:34:52.628 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:52.620+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229659 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:52.926 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:52.921+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:52.923584+0000) 2019-10-13T15:34:53.214 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:34:53.214 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:53.218 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:34:53.218 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:34:54.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:54.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:54.731020+0000) 2019-10-13T15:34:55.614 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:55.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:55.611901+0000) 2019-10-13T15:34:57.631 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:57.624+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.629098+0000) 2019-10-13T15:34:57.632 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:34:57.624+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.629444+0000) 2019-10-13T15:34:57.632 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:34:57.625+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.630388+0000) 2019-10-13T15:34:57.634 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:34:57.628+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.632043+0000) 2019-10-13T15:34:57.636 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:57.632+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.634090+0000) 2019-10-13T15:34:57.637 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:34:57.632+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.634644+0000) 2019-10-13T15:34:57.637 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:57.632+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.635138+0000) 2019-10-13T15:34:57.642 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:34:57.636+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229692 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:34:57.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:34:57.637+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:57.641758+0000) 2019-10-13T15:34:58.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:34:58.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:58.100152+0000) 2019-10-13T15:34:58.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:34:58.320+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:58.324646+0000) 2019-10-13T15:34:58.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:34:58.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:34:58.598662+0000) 2019-10-13T15:35:01.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:01.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:01.196142+0000) 2019-10-13T15:35:02.365 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:02.361+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.363070+0000) 2019-10-13T15:35:02.646 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:02.640+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.643324+0000) 2019-10-13T15:35:02.647 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:02.640+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.643688+0000) 2019-10-13T15:35:02.647 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:02.641+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.644415+0000) 2019-10-13T15:35:02.647 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:02.640+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.645187+0000) 2019-10-13T15:35:02.650 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:02.644+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.647369+0000) 2019-10-13T15:35:02.650 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:02.644+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.647645+0000) 2019-10-13T15:35:02.650 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:02.644+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.648208+0000) 2019-10-13T15:35:02.656 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:02.649+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.653640+0000) 2019-10-13T15:35:02.658 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:02.652+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229720 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:02.935 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:02.929+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:02.932748+0000) 2019-10-13T15:35:04.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:04.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:04.731218+0000) 2019-10-13T15:35:05.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:05.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:05.612059+0000) 2019-10-13T15:35:07.661 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:07.653+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.658715+0000) 2019-10-13T15:35:07.662 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:07.656+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.659579+0000) 2019-10-13T15:35:07.662 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:07.656+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.660313+0000) 2019-10-13T15:35:07.665 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:07.660+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.662502+0000) 2019-10-13T15:35:07.665 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:07.660+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.662827+0000) 2019-10-13T15:35:07.667 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:07.660+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.664741+0000) 2019-10-13T15:35:07.668 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:07.664+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.665891+0000) 2019-10-13T15:35:07.674 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:07.668+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229751 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:07.674 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:07.665+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:07.671990+0000) 2019-10-13T15:35:08.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:08.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:08.100330+0000) 2019-10-13T15:35:08.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:08.324+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:08.324842+0000) 2019-10-13T15:35:08.601 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:08.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:08.598879+0000) 2019-10-13T15:35:11.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:11.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:11.196341+0000) 2019-10-13T15:35:12.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:12.361+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.363299+0000) 2019-10-13T15:35:12.677 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:12.672+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.673963+0000) 2019-10-13T15:35:12.678 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:12.672+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.675920+0000) 2019-10-13T15:35:12.680 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:12.673+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.678283+0000) 2019-10-13T15:35:12.681 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:12.676+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.678595+0000) 2019-10-13T15:35:12.682 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:12.676+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.679287+0000) 2019-10-13T15:35:12.682 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:12.676+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.679970+0000) 2019-10-13T15:35:12.684 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:12.680+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.682082+0000) 2019-10-13T15:35:12.689 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:12.684+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229779 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:12.691 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:12.685+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.688784+0000) 2019-10-13T15:35:12.942 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:12.937+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:12.939355+0000) 2019-10-13T15:35:14.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:14.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:14.731455+0000) 2019-10-13T15:35:15.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:15.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:15.612278+0000) 2019-10-13T15:35:17.693 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:17.688+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.690730+0000) 2019-10-13T15:35:17.694 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:17.688+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.691185+0000) 2019-10-13T15:35:17.694 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:17.688+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.691618+0000) 2019-10-13T15:35:17.694 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:17.688+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.692054+0000) 2019-10-13T15:35:17.695 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:17.689+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.692618+0000) 2019-10-13T15:35:17.695 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:17.688+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.693088+0000) 2019-10-13T15:35:17.697 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:17.692+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.694668+0000) 2019-10-13T15:35:17.702 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:17.697+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:17.700282+0000) 2019-10-13T15:35:17.703 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:17.696+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229812 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:18.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:18.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:18.100509+0000) 2019-10-13T15:35:18.327 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:18.320+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:18.325054+0000) 2019-10-13T15:35:18.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:18.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:18.599072+0000) 2019-10-13T15:35:21.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:21.196+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:21.196554+0000) 2019-10-13T15:35:22.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:22.361+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.363496+0000) 2019-10-13T15:35:22.707 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:22.701+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.704727+0000) 2019-10-13T15:35:22.708 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:22.700+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.705170+0000) 2019-10-13T15:35:22.708 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:22.700+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.705576+0000) 2019-10-13T15:35:22.710 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:22.704+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.708114+0000) 2019-10-13T15:35:22.711 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:22.704+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.708316+0000) 2019-10-13T15:35:22.711 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:22.704+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.709136+0000) 2019-10-13T15:35:22.712 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:22.708+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.710133+0000) 2019-10-13T15:35:22.716 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:22.709+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.714095+0000) 2019-10-13T15:35:22.718 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:22.712+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229839 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:22.948 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:22.945+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:22.945761+0000) 2019-10-13T15:35:23.296 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:35:23.296 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:23.300 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:35:23.300 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:24.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:24.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:24.731680+0000) 2019-10-13T15:35:25.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:25.608+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:25.612464+0000) 2019-10-13T15:35:27.721 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:27.713+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.718991+0000) 2019-10-13T15:35:27.722 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:27.716+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.719487+0000) 2019-10-13T15:35:27.722 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:27.716+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.720025+0000) 2019-10-13T15:35:27.723 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:27.716+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.720322+0000) 2019-10-13T15:35:27.724 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:27.720+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.722289+0000) 2019-10-13T15:35:27.726 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:27.720+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.723857+0000) 2019-10-13T15:35:27.728 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:27.720+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.725650+0000) 2019-10-13T15:35:27.732 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:27.725+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:27.729699+0000) 2019-10-13T15:35:27.733 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:27.728+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229870 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:28.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:28.096+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:28.100682+0000) 2019-10-13T15:35:28.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:28.324+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:28.325233+0000) 2019-10-13T15:35:28.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:28.596+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:28.599250+0000) 2019-10-13T15:35:31.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:31.192+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:31.196767+0000) 2019-10-13T15:35:32.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:32.361+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.363693+0000) 2019-10-13T15:35:32.736 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:32.732+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.733953+0000) 2019-10-13T15:35:32.737 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:32.732+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.734743+0000) 2019-10-13T15:35:32.738 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:32.729+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.735273+0000) 2019-10-13T15:35:32.738 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:32.732+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.735883+0000) 2019-10-13T15:35:32.738 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:32.732+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.736086+0000) 2019-10-13T15:35:32.739 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:32.732+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.736743+0000) 2019-10-13T15:35:32.739 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:32.732+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.737085+0000) 2019-10-13T15:35:32.747 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:32.741+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.744484+0000) 2019-10-13T15:35:32.748 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:32.744+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229898 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:32.955 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:32.949+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:32.952278+0000) 2019-10-13T15:35:34.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:34.728+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:34.731887+0000) 2019-10-13T15:35:35.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:35.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:35.612663+0000) 2019-10-13T15:35:37.751 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:37.743+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.748698+0000) 2019-10-13T15:35:37.752 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:37.743+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.749098+0000) 2019-10-13T15:35:37.752 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:37.743+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.749526+0000) 2019-10-13T15:35:37.753 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:37.747+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.750580+0000) 2019-10-13T15:35:37.754 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:37.747+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.750725+0000) 2019-10-13T15:35:37.755 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:37.749+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.752707+0000) 2019-10-13T15:35:37.755 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:37.747+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.753088+0000) 2019-10-13T15:35:37.763 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:37.757+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:37.760949+0000) 2019-10-13T15:35:37.764 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:37.755+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229930 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:38.103 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:38.099+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:38.100878+0000) 2019-10-13T15:35:38.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:38.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:38.325454+0000) 2019-10-13T15:35:38.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:38.595+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:38.599409+0000) 2019-10-13T15:35:41.199 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:41.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:41.196962+0000) 2019-10-13T15:35:42.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:42.357+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.363922+0000) 2019-10-13T15:35:42.767 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:42.759+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.764686+0000) 2019-10-13T15:35:42.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:42.759+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.765243+0000) 2019-10-13T15:35:42.768 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:42.759+0000 7f6c9111b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.765415+0000) 2019-10-13T15:35:42.770 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:42.763+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.767871+0000) 2019-10-13T15:35:42.772 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:42.765+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.770430+0000) 2019-10-13T15:35:42.775 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:42.767+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.772367+0000) 2019-10-13T15:35:42.776 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:42.771+0000 7f14097c9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.774187+0000) 2019-10-13T15:35:42.779 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:42.771+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229959 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:42.783 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:42.777+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.781060+0000) 2019-10-13T15:35:42.961 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:42.953+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:42.958776+0000) 2019-10-13T15:35:44.734 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:44.727+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:44.732088+0000) 2019-10-13T15:35:45.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:45.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:45.612894+0000) 2019-10-13T15:35:47.782 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:47.777+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.780152+0000) 2019-10-13T15:35:47.783 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:47.775+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.780374+0000) 2019-10-13T15:35:47.783 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:47.775+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.781304+0000) 2019-10-13T15:35:47.784 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:47.779+0000 7f292df83700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.782069+0000) 2019-10-13T15:35:47.785 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:47.779+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.782573+0000) 2019-10-13T15:35:47.785 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:47.779+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.782848+0000) 2019-10-13T15:35:47.786 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:47.779+0000 7f89ffe7f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.783158+0000) 2019-10-13T15:35:47.789 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:47.781+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:47.787427+0000) 2019-10-13T15:35:47.794 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:47.787+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229989 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:48.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:48.095+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:48.101098+0000) 2019-10-13T15:35:48.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:48.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:48.325654+0000) 2019-10-13T15:35:48.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:48.595+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:48.599541+0000) 2019-10-13T15:35:51.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:51.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:51.197155+0000) 2019-10-13T15:35:52.366 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:52.360+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.364120+0000) 2019-10-13T15:35:52.798 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:52.791+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.795508+0000) 2019-10-13T15:35:52.799 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:52.788+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.795897+0000) 2019-10-13T15:35:52.799 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:52.791+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.796153+0000) 2019-10-13T15:35:52.799 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:52.791+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.797022+0000) 2019-10-13T15:35:52.800 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:52.791+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.797554+0000) 2019-10-13T15:35:52.801 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:52.795+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.797725+0000) 2019-10-13T15:35:52.801 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:52.795+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.798172+0000) 2019-10-13T15:35:52.805 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:52.796+0000 7f237ee50700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.803114+0000) 2019-10-13T15:35:52.809 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:52.803+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230018 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:52.968 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:52.964+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:52.965336+0000) 2019-10-13T15:35:53.351 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:35:53.352 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:53.355 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:35:53.356 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:35:54.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:54.727+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:54.732313+0000) 2019-10-13T15:35:55.615 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:55.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:55.613085+0000) 2019-10-13T15:35:57.813 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:57.807+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.811142+0000) 2019-10-13T15:35:57.815 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:35:57.808+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.813320+0000) 2019-10-13T15:35:57.816 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:35:57.807+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.813295+0000) 2019-10-13T15:35:57.816 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:57.811+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.813825+0000) 2019-10-13T15:35:57.817 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:35:57.811+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.814544+0000) 2019-10-13T15:35:57.818 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:57.811+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.815541+0000) 2019-10-13T15:35:57.818 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:35:57.811+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.815924+0000) 2019-10-13T15:35:57.820 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:35:57.812+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:57.817481+0000) 2019-10-13T15:35:57.824 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:35:57.815+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230050 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:35:58.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:35:58.095+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:58.101276+0000) 2019-10-13T15:35:58.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:35:58.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:58.325826+0000) 2019-10-13T15:35:58.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:35:58.595+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:35:58.599717+0000) 2019-10-13T15:36:01.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:01.191+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:01.197355+0000) 2019-10-13T15:36:02.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:02.360+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.364386+0000) 2019-10-13T15:36:02.827 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:02.819+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.824395+0000) 2019-10-13T15:36:02.828 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:02.819+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.825310+0000) 2019-10-13T15:36:02.829 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:02.823+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.826185+0000) 2019-10-13T15:36:02.829 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:02.823+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.826425+0000) 2019-10-13T15:36:02.829 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:02.820+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.827239+0000) 2019-10-13T15:36:02.830 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:02.823+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.826710+0000) 2019-10-13T15:36:02.830 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:02.823+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.828150+0000) 2019-10-13T15:36:02.837 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:02.828+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.835009+0000) 2019-10-13T15:36:02.839 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:02.831+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230079 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:02.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:02.968+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:02.972007+0000) 2019-10-13T15:36:04.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:04.731+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:04.732538+0000) 2019-10-13T15:36:05.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:05.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:05.613279+0000) 2019-10-13T15:36:07.842 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:07.835+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.839395+0000) 2019-10-13T15:36:07.843 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:07.835+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.840385+0000) 2019-10-13T15:36:07.843 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:07.835+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.840454+0000) 2019-10-13T15:36:07.844 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:07.836+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.841233+0000) 2019-10-13T15:36:07.844 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:07.839+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.842084+0000) 2019-10-13T15:36:07.845 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:07.839+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.842383+0000) 2019-10-13T15:36:07.846 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:07.839+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.844163+0000) 2019-10-13T15:36:07.852 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:07.844+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:07.849564+0000) 2019-10-13T15:36:07.854 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:07.847+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230109 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:08.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:08.099+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:08.101470+0000) 2019-10-13T15:36:08.328 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:08.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:08.326018+0000) 2019-10-13T15:36:08.602 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:08.599+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:08.599912+0000) 2019-10-13T15:36:11.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:11.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:11.197556+0000) 2019-10-13T15:36:12.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:12.360+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.364585+0000) 2019-10-13T15:36:12.857 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:12.851+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.854787+0000) 2019-10-13T15:36:12.858 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:12.851+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.855228+0000) 2019-10-13T15:36:12.860 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:12.851+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.857229+0000) 2019-10-13T15:36:12.861 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:12.852+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.858847+0000) 2019-10-13T15:36:12.862 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:12.855+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.859292+0000) 2019-10-13T15:36:12.862 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:12.855+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.859842+0000) 2019-10-13T15:36:12.864 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:12.855+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.861568+0000) 2019-10-13T15:36:12.868 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:12.860+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.866069+0000) 2019-10-13T15:36:12.870 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:12.863+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230138 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:12.986 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:12.980+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:12.983342+0000) 2019-10-13T15:36:14.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:14.727+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:14.732692+0000) 2019-10-13T15:36:15.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:15.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:15.613501+0000) 2019-10-13T15:36:17.873 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:17.867+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.870873+0000) 2019-10-13T15:36:17.874 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:17.867+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.871577+0000) 2019-10-13T15:36:17.874 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:17.867+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.871866+0000) 2019-10-13T15:36:17.875 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:17.867+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.872447+0000) 2019-10-13T15:36:17.875 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:17.867+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.872562+0000) 2019-10-13T15:36:17.875 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:17.868+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.873050+0000) 2019-10-13T15:36:17.876 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:17.871+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.874232+0000) 2019-10-13T15:36:17.883 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:17.876+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:17.880451+0000) 2019-10-13T15:36:17.885 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:17.879+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230170 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:18.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:18.095+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:18.101649+0000) 2019-10-13T15:36:18.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:18.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:18.326223+0000) 2019-10-13T15:36:18.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:18.599+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:18.600111+0000) 2019-10-13T15:36:21.200 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:21.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:21.197797+0000) 2019-10-13T15:36:22.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:22.364+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.364786+0000) 2019-10-13T15:36:22.888 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:22.883+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.885798+0000) 2019-10-13T15:36:22.889 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:22.883+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.886388+0000) 2019-10-13T15:36:22.889 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:22.883+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.886878+0000) 2019-10-13T15:36:22.890 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:22.883+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.887892+0000) 2019-10-13T15:36:22.891 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:22.883+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.888375+0000) 2019-10-13T15:36:22.891 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:22.884+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.889471+0000) 2019-10-13T15:36:22.892 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:22.887+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.889815+0000) 2019-10-13T15:36:22.896 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:22.888+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.893624+0000) 2019-10-13T15:36:22.899 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:22.891+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230199 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:22.993 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:22.984+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:22.990103+0000) 2019-10-13T15:36:23.393 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:36:23.393 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:23.397 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:36:23.397 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:24.735 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:24.731+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:24.732893+0000) 2019-10-13T15:36:25.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:25.607+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:25.613696+0000) 2019-10-13T15:36:27.903 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:27.895+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.900367+0000) 2019-10-13T15:36:27.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:27.895+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.901258+0000) 2019-10-13T15:36:27.904 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:27.895+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.901461+0000) 2019-10-13T15:36:27.904 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:27.899+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.902180+0000) 2019-10-13T15:36:27.905 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:27.899+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.902531+0000) 2019-10-13T15:36:27.905 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:27.899+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.902893+0000) 2019-10-13T15:36:27.906 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:27.896+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.903472+0000) 2019-10-13T15:36:27.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:27.904+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:27.911423+0000) 2019-10-13T15:36:27.916 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:27.907+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230229 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:28.104 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:28.099+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:28.101857+0000) 2019-10-13T15:36:28.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:28.323+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:28.326402+0000) 2019-10-13T15:36:28.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:28.595+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:28.600287+0000) 2019-10-13T15:36:31.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:31.195+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:31.198024+0000) 2019-10-13T15:36:32.367 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:32.364+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.364994+0000) 2019-10-13T15:36:32.920 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:32.910+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.917488+0000) 2019-10-13T15:36:32.921 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:32.914+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.918190+0000) 2019-10-13T15:36:32.921 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:32.914+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.918702+0000) 2019-10-13T15:36:32.922 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:32.914+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.920145+0000) 2019-10-13T15:36:32.923 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:32.914+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.920507+0000) 2019-10-13T15:36:32.924 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:32.916+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.921339+0000) 2019-10-13T15:36:32.924 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:32.916+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.922148+0000) 2019-10-13T15:36:32.925 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:32.918+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:32.922372+0000) 2019-10-13T15:36:32.928 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:32.922+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230258 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:33.003 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:33.000+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:33.000842+0000) 2019-10-13T15:36:34.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:34.730+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:34.733090+0000) 2019-10-13T15:36:35.616 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:35.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:35.613878+0000) 2019-10-13T15:36:37.932 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:37.922+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.929223+0000) 2019-10-13T15:36:37.934 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:37.926+0000 7f140a7cb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.931326+0000) 2019-10-13T15:36:37.934 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:37.926+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.931988+0000) 2019-10-13T15:36:37.935 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:37.926+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.932434+0000) 2019-10-13T15:36:37.935 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:37.926+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.932789+0000) 2019-10-13T15:36:37.936 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:37.926+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.933439+0000) 2019-10-13T15:36:37.936 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:37.928+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.934153+0000) 2019-10-13T15:36:37.943 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:37.936+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:37.940382+0000) 2019-10-13T15:36:37.943 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:37.934+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230290 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:38.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:38.098+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:38.102051+0000) 2019-10-13T15:36:38.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:38.322+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:38.326572+0000) 2019-10-13T15:36:38.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:38.598+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:38.600463+0000) 2019-10-13T15:36:41.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:41.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:41.198235+0000) 2019-10-13T15:36:42.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:42.364+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.365187+0000) 2019-10-13T15:36:42.947 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:42.938+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.944472+0000) 2019-10-13T15:36:42.948 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:42.938+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.944867+0000) 2019-10-13T15:36:42.948 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:42.940+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.945669+0000) 2019-10-13T15:36:42.949 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:42.942+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.946122+0000) 2019-10-13T15:36:42.949 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:42.942+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.946726+0000) 2019-10-13T15:36:42.949 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:42.942+0000 7f89ff67e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.947264+0000) 2019-10-13T15:36:42.950 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:42.942+0000 7f6c9191c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.947814+0000) 2019-10-13T15:36:42.952 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:42.944+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:42.950365+0000) 2019-10-13T15:36:42.959 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:42.950+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230321 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:43.009 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:43.004+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:43.006264+0000) 2019-10-13T15:36:44.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:44.730+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:44.733286+0000) 2019-10-13T15:36:45.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:45.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:45.614079+0000) 2019-10-13T15:36:47.962 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:47.954+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.959756+0000) 2019-10-13T15:36:47.963 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:47.954+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.960380+0000) 2019-10-13T15:36:47.963 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:47.954+0000 7fda8634e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.960874+0000) 2019-10-13T15:36:47.964 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:47.954+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.961596+0000) 2019-10-13T15:36:47.965 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:47.958+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.962989+0000) 2019-10-13T15:36:47.967 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:47.960+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.964431+0000) 2019-10-13T15:36:47.969 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:47.962+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.966847+0000) 2019-10-13T15:36:47.973 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:47.966+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230349 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:47.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:47.964+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:47.971484+0000) 2019-10-13T15:36:48.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:48.098+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:48.102221+0000) 2019-10-13T15:36:48.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:48.322+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:48.326751+0000) 2019-10-13T15:36:48.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:48.598+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:48.600664+0000) 2019-10-13T15:36:51.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:51.194+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:51.198449+0000) 2019-10-13T15:36:52.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:52.364+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.365402+0000) 2019-10-13T15:36:52.977 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:52.972+0000 7ff33ba6d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.975086+0000) 2019-10-13T15:36:52.978 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:52.970+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.974763+0000) 2019-10-13T15:36:52.978 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:52.970+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.975628+0000) 2019-10-13T15:36:52.979 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:52.970+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.975394+0000) 2019-10-13T15:36:52.979 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:52.970+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.976221+0000) 2019-10-13T15:36:52.980 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:52.974+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.977783+0000) 2019-10-13T15:36:52.980 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:52.974+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.978011+0000) 2019-10-13T15:36:52.987 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:52.984+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:52.985159+0000) 2019-10-13T15:36:52.988 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:52.978+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230379 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:53.015 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:53.012+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:53.012845+0000) 2019-10-13T15:36:53.456 INFO:teuthology.orchestra.run.smithi110:Running: 2019-10-13T15:36:53.457 INFO:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:53.460 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-13T15:36:53.460 INFO:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-10-13T15:36:54.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:54.726+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:54.733489+0000) 2019-10-13T15:36:55.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:55.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:55.614270+0000) 2019-10-13T15:36:57.992 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:57.982+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.989119+0000) 2019-10-13T15:36:57.992 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:36:57.982+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.989367+0000) 2019-10-13T15:36:57.993 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:57.986+0000 7f292e784700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.990227+0000) 2019-10-13T15:36:57.994 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:57.986+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.991306+0000) 2019-10-13T15:36:57.994 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:36:57.988+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.991190+0000) 2019-10-13T15:36:57.995 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:36:57.986+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.992056+0000) 2019-10-13T15:36:57.995 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:36:57.986+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.992826+0000) 2019-10-13T15:36:57.999 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:36:57.996+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:57.997215+0000) 2019-10-13T15:36:58.002 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:36:57.994+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230409 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:36:58.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:36:58.098+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:58.102432+0000) 2019-10-13T15:36:58.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:36:58.326+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:58.326936+0000) 2019-10-13T15:36:58.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:36:58.594+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:36:58.600875+0000) 2019-10-13T15:37:01.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:37:01.198+0000 7f0ed7512700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:01.198667+0000) 2019-10-13T15:37:02.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:37:02.363+0000 7ff338a67700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:02.365614+0000) 2019-10-13T15:37:03.006 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:37:02.998+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.003652+0000) 2019-10-13T15:37:03.007 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:37:02.998+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.004036+0000) 2019-10-13T15:37:03.007 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:37:02.998+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.004748+0000) 2019-10-13T15:37:03.009 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:37:03.002+0000 7f0edcd1d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.007091+0000) 2019-10-13T15:37:03.010 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:37:03.003+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.007394+0000) 2019-10-13T15:37:03.010 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:37:03.002+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.007708+0000) 2019-10-13T15:37:03.011 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:37:03.002+0000 7fda87350700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.008002+0000) 2019-10-13T15:37:03.017 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:37:03.010+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230439 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:37:03.018 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:37:03.015+0000 7f237f651700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.016322+0000) 2019-10-13T15:37:03.021 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:37:03.015+0000 7f2379e46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:03.019295+0000) 2019-10-13T15:37:04.736 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:37:04.730+0000 7f13fb355700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:04.733689+0000) 2019-10-13T15:37:05.617 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:37:05.610+0000 7fda83348700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:05.614424+0000) 2019-10-13T15:37:08.021 INFO:tasks.ceph.osd.2.smithi110.stderr:2019-10-13T15:37:08.014+0000 7f1409fca700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.018004+0000) 2019-10-13T15:37:08.021 INFO:tasks.ceph.mds.b.smithi204.stderr:2019-10-13T15:37:08.015+0000 7ff33c26e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.018763+0000) 2019-10-13T15:37:08.022 INFO:tasks.ceph.mgr.y.smithi110.stderr:2019-10-13T15:37:08.014+0000 7f0edd51e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.019735+0000) 2019-10-13T15:37:08.023 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:37:08.014+0000 7f292d782700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.020575+0000) 2019-10-13T15:37:08.023 INFO:tasks.ceph.mds.a.smithi110.stderr:2019-10-13T15:37:08.014+0000 7fda86b4f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.021072+0000) 2019-10-13T15:37:08.024 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:37:08.014+0000 7f89fee7d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.021636+0000) 2019-10-13T15:37:08.024 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:37:08.014+0000 7f6c9091a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.020102+0000) 2019-10-13T15:37:08.030 INFO:tasks.ceph.mgr.x.smithi204.stderr:2019-10-13T15:37:08.023+0000 7f237fe52700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.027690+0000) 2019-10-13T15:37:08.032 INFO:tasks.ceph.mon.a.smithi110.stderr:2019-10-13T15:37:08.022+0000 7fe528211700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230466 slow ops, oldest is log(5 entries from seq 504 at 2019-10-13T04:21:19.524572+0000) 2019-10-13T15:37:08.105 INFO:tasks.ceph.osd.3.smithi110.stderr:2019-10-13T15:37:08.102+0000 7f291f30e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.102613+0000) 2019-10-13T15:37:08.329 INFO:tasks.ceph.osd.1.smithi110.stderr:2019-10-13T15:37:08.322+0000 7f6c824a6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.327107+0000) 2019-10-13T15:37:08.603 INFO:tasks.ceph.osd.0.smithi110.stderr:2019-10-13T15:37:08.598+0000 7f89f0a09700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-13T14:37:08.601081+0000) 2019-10-13T15:37:10.903 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-13T15:37:10.903 DEBUG:teuthology.task.console_log:Killing console logger for smithi204 2019-10-13T15:37:10.903 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2019-10-13T15:37:10.904 DEBUG:teuthology.task.console_log:Killing console logger for smithi204 2019-10-13T15:37:10.904 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2019-10-13T15:37:10.904 DEBUG:teuthology.exit:Finished running handlers